In order to assess the accuracy of our league, we will revisit five recent opinions, and compare the predicted outcomes with the actual outcomes. We have selected cases that did not necessarily fall along ideological lines to test the capability of our league to detect nuance in judgments. In this week’s 10th Justice, we will take a look at Hemi Group, Briscoe, Powell, Hertz Corp, and Shatzer, all of which were decided at the end of January and February. We also consider how cheating impacted the results.

For each case, we have recorded our outcome statistics and SMRs (standard majority ratio). The SMR provides a method to test whether or not users perceive the Court as dominated by conservative ideology.

In Hemi Group v. City of New York, the Supreme Court reversed the 2nd Circuit’s decision that the non-payment of taxes was an injury flowing from a third party’s actions for the purpose of the RICO Act. Overall, only 39% of total predictions found that the Supreme Court would reverse, and the confidence interval was 10.25% (at the 95% level), indicating that the results were a decently accurate representation of predictions about the case (and their general incorrectness). As for the specific split, 18 users guessed that the outcome would have less than 9 Justices voting. These members were astute, as Sotomayor recusing herself from an opinion involving the Second Circuit should have been expected. But only half of those predictions, 10% of total predictions, were reverse predictions. No member correctly predicted the votes of each Justice. The SMRs indicate that most predictions were counting on liberal Justices joining the majority, since Stevens, Ginsburg, and Breyer all have SMRs above one at a statistically significant level. Of course, this could also be representative of a complex case or issue, as Kennedy was in the minority for this case. This fact would also explain how the majority of predictions got the general outcome wrong.
In Briscoe v. Virginia, the Supreme Court reversed the Virginia Supreme Court’s. The Court held that a Defendant does not waive his 6th Amendment rights to the confrontation of forensic analysts who prepared forensic evident used in court by failing to call them as witnesses. 63% of predictions correctly guessed that the Supreme Court would reverse. The results were statistically significant at a 99% confidence level, with a confidence interval of 11.86. For the specific split, only 3 users, 4.6% of total predictions, predicted a unanimous outcome. These members probably correctly assumed a per curiam reversal would be issued in light of Melendez-Diaz v. Massachusetts. The SMRs further support this conclusion since the “liberal minority” all had SMRs above 1 at a statistically significant level. Sotomayor’s SMR of 2.17 indicated that she was highly likely to join in a majority decisions, and Ginsburg’s SMR of 1.71 was also a solid indicator. Overall, it is interesting to note that while predictions produced a statistically sound correct outcome, a smaller proportion of users predicted the correct split than in Hemi.
In Florida v. Powell, the Court reversed a Florida Supreme Court decision that held that informing a defendant that they had a right to “talk to an attorney” was insufficient to inform them they had a right to have counsel present. Overall, 51% of members predicted that the Court would reverse, but with such a tight margin, the results were not statistically significant. Only three users, approximately 2% of total predictions, predicted the correct split, while only one user predicted the correct votes. In conjunction with the indeterminate nature of the general outcome predictions, the SMRs did not reveal any additional information since all Justices had SMRs that were not significantly different from 1. This indicates that predictions mainly sorted along ideological lines.

But how did the FantasySCOTUS cheaters affect the results? This installment of the 10th Justice continues at JoshBlackman.com, after the jump.

In Hertz v. Friend, the Court reversed a 9th Circuit holding which held that the “place of operations” test was the appropriate test for determining a corporation’s citizenship for purposes of diversity jurisdiction. 69% of all predictions held that the Court would reverse the decision at a confidence level of 99%, with a confidence interval of 11.83. Additionally, 33 members, approximately 30% of all predictions, predicted that the decision to reverse would be unanimous. The SMRs of Stevens, Ginsburg, Breyer, and Sotomayor also reflect this outcome, since all four have SMRs significantly above 1 (all of them are well above 2), which indicates a strong tendency for there to be a large majority with Justices joining the majority across ideological lines. As in this case, clear procedural rules and issues are easier to predict and do not rely heavily on ideology to reach an outcome.

In Maryland v. Shatzer, the Court reversed a Maryland Court of Appeals decision which held that once a suspect invokes the right to counsel, the suspect may not be re-interrogated until they are given counsel or voluntarily initiate communication, even if the new interrogation takes place three years later. The results in the above table take into account the issues encountered with certain people who sought to cheat at the league. 53% of the predictions concerning this case held that the Court would reverse, but the result was not statistically significant even in the context of the 637 total predictions. However, 27 users, 4.2% of total predictions, guessed the unanimous outcome correctly. The SMRs once again further supports this conclusion, although not as strongly as they did in Hertz. All of the Justices have SMRs significantly above 1 (except for Ginsburg), which indicates a likelihood of an unanimous opinion. Another interesting factor is that the conservative Justices had higher SMRs than the liberal Justices. This shows that they were more likely to join in a majority across ideological lines. Although the overall statistics did not provide significant clarity, the SMRs were useful for giving a richer explanation of the outcome.

Overall, this set of recently decided cases is a good cross section of the different situations that can arise with predictions and statistics. Although they are not always perfect or determinative, these statistics do provide insights into the prediction process and perceptions of certain cases. Additionally, the diversity of issues and cases such as the ones outlined above also provide feedback on where and when the statistics provide more or less information. In the end however, the disposition of each Justice can only be predicted in so much as they allow their actions to be predictable.

Many thanks to Corey Carpenter for his excellent assistance with this post.