Who will be the next Supreme Court Justice? Vote Now!

April 7th, 2010

The conventional wisdom in the SCOTUSphere is that Justice John Paul Stevens will step down at the end of the term. But who will replace him? calls upon the wisdom of our crowds, and invites you to vote, and rank your favorite picks for the High Court. As a special Article III treat, we have waived the registration fee. Sign up now for FREE at!

This week’s 10th Justice will discuss whether or not oral arguments have a significant effect on case predictions. However, we have chosen six cases, Alabama v. North Carolina, Briscoe v. Virginia, Mac’s v. Shell Oil, American Needle v. NFL, U.S. v. Comstock, and McDonald v. Chicago and will compare user perception of these cases before and after oral arguments.

Hypothesis testing is an important statistical method for determining whether or not characteristics of samples are due to randomization or other factors. A hypothesis test is a question couched in the terms of a null and alternate hypothesis. The null hypothesis is what would be true if the results were due to randomization, while the alternative hypothesis is what the researcher wants to show. As such the testing results either reject or fail to reject the null (random) hypothesis rather than conclusively “prove” the alternate hypothesis.

Because we want to know whether our pre-argument percentages are significantly different from the post-argument percentages, we will be using a two-tailed test based on a standard normal (Z) distribution, in which we compare a test statistic (test Z-score) against a Z-score derived from our confidence level. For our purposes, the null hypothesis is that there is no significant difference between proportions (indicated by a Z-score within specific values determined by our confidence level).

The following table lists the total number of predictions before arguments, the percentage of votes before arguments, the number of predictions after arguments, the percetnage of votes after arguments, the test’s Z-score, and whether or not we can reject the null hypothesis at the 90%, 95%, or 99% confidence level. Additionally, the Z-score corresponding to the confidence interval will also be listed, but because it is a two-tailed test, our test value must either be above the positive confidence value or below the negative confidence value in order for the proportions to be significantly different.

An explanation of the results, after the jump at

The post-argument percentages were lower than pre-argument percentages for Alabama, Mac’s, American Needle, and Comstock, but not for Briscoe and McDonald. This fact indicates that oral arguments generally introduce more uncertainty.

In Alabama, the Z-score of 1.607 fell short of the 1.645 threshold to be significant at a 90% confidence level, and we could not reject the null hypothesis. Briscoe and Mac’s also fell short of the same standard with Z-scores of 1.355 and 1.223 respectively.

American Needle was the only case where we rejected the null hypothesis of no significance with a z-score of 3.127, above the 2.576 required for the 99% confidence level.

Out of the six cases, Comstock and McDonald had the least potential for significance with respective Z-scores of 0.826 and 0.919, once again failing to reject the null hypothesis. Notice that even for McDonald, which has substantially more predictions than the other cases, we still failed to reject the null hypothesis of no significant difference.

These results do not indicate that oral arguments do not affect user perception of the outcome of the case. Rather, we cannot confidently state that the differences are significant. Overall, the statistical significance of oral arguments for user predictions is an exception, not a rule.

Since SCOTUS followers tend to be familiar with the facts and rulings of the case, the oral argument adds less value unless a Justice unexpectedly telegraphs that he or she is unexpectedly leaning towards a different outcome. This may explain the results in American Needle, as the predictions took a decisive change of course after oral arguments. Also, it is important to note that most of the cases above did not have a marginal majority of predictions for affirm or reversal except for Briscoe, which indicates that oral arguments do not resolve a significant portion of uncertainty in cases. Due to our failure to reject the null hypothesis in most of the cases, factors such as the passage of time and randomization are possible explanations for the difference in our proportions.