Blog

In our article, Dan Katz, Mike Bommarito, and I rely on the machine learning technology, known as “extremely randomized trees,” to build a model that predicts decisions with a 69.7% accuracy,and individual justice votes with a 70.9% accuracy, over all cases from 1953-2013.

One of the most common questions I get, in various forms, is how can your model predict cases that were already decided. This often followed by a critique that our model was backwards looking, and attempted to fit a model with what happened. This isn’t how our model works, so let me try to use an analogy, with law bloggers we all know, to explain it.

On Tuesday, the Supreme Court announces that Thursday will be the final day of the term. We know that there are five outstanding cases. On Wednesday, Orin at VC offers his predictions for the five cases. He makes this prediction based on everything he knows about the Court from this term, and from the last few terms. He also considers factors like what the lower court did, how out of line it was with the Court’s precedents, and perhaps how often that Circuit had been reversed. He also considers how specific Justices have treated specific cases involving this particular issue. Based on all of these factors, which he assigns varying weights, he posts his predictions on Wednesday.

The Court makes a decision on Thursday. On Friday, Orin takes a retrospective, analyzes his predictions, and tries to figure out what went right and what went wrong. Maybe he weighted some factors too much, and other factors too little. He notes that next term he will try to use a slightly different framework.

Come the following June, Orin, based on the good and bad steps he took the previous term, approaches the decision-making process slightly differently, and comes up with better, or worse results. Before each day the Court is in session to hand down opinions, Orin takes a look back, figures what works and didn’t, and revises his methodology. Then, he offers a new set of predictions. After the cases are decided, he again looks back and revises his approach, and offers new predictions. Tedious for sure.

After doing this for, let’s say 60 years, Orin has a pretty good sense of what the right weight of factors to balance are. This becomes his standard model. With this standard formula, he goes back and runs predictions in each of the 60 terms, limiting himself only to information he knew at the time the case is decided (as hard as this may be, I’m positive Orin can do this). In other words, the ideologies of the Justices in 2014 will be different than the ideologies of the Justices in 2074. The model is the same, but the variables are time-sensitive. After Orin runs this 60 year experiment in the year 2074, he finds that the finely calibrated weights of the variably achieved an overall accuracy rate of (let’s say) 70%.

Now, imagine instead of just Orin doing this 60-year process, every member of the VC did it. And each member came up with different weightings of the variable. Will went one way, David went another, Ilya a third, Randy a fourth, etc. Each of their models, when run over 60 years, achieved accuracy rates around 70–some higher, some lower.

Now, imagine thousands of legal bloggers on the legal blogosphere attempt the same 60-year experiment, each designing different weights that may work better or worse.

The king of the blogosphere then averages together all of the different models to figure out the “gold standard” that can be used to predict any case during the previous 60 years, or for the upcoming October 2074 term (where the Court will decide whether robots have a right to privacy, or something like that).

This is effecitvely how our machine-learning model works. When designing the method, we attempted to model how hundreds or thousands of law professors who, if glued to their computers for 60 years, could have designed a prediction methodology. Instead of working with an individual blogger, or even a group of bloggers, our model generates thousands of extremely randomized decision trees. Each tree assigns different weights to a series of over 95 variables. Then, based on these predictions, we figure which weights work better than others, and refine our methodology. But all the predictions are made before the case was decided–as if we were using a time machine. When going through the 60-year history, we only look at information known prior to the case decision. As I explained to Vox:

 Instead of making this backward-looking, we wanted to try to be forward looking. Say a case was decided on March 12th 1954. What information was known to the world on March 11th 1954, the day before it was decided (which was actually Antonin Scalia’s eighteenth birthday — sorry, I’m a nerd). What did we know on March 11th 1954? Based only on that data we started using a machine learning process called extremely randomized trees.

In the same way that other studies generate a decision tree, our algorithms basically spit out lots of trees, randomly designed. Each tree put different weights on different variables. Then we checked the trees. Some trees happen to work better than others. The weights of the variables are calculated to four or five decimal places. They use very precise weights. By creating enough trees, we were able to figure out which ones did best.

Once we make our predictions, we have to test them against the ultimate outcome to see what worked and didn’t. Then we can go back with the revised weights of variables.

In the end, our model will be put to the test this year. We will be predicting cases in real time, and post all of our predictions online. It’s hard to predict in advance how we will do. Some years we were around 80%. Some years we were around 60%. There is a lot of variability in the Court’s docket. And, the recent unanimity of the Court has thrown a wrench in things a bit. But we are really excited to see how we do this year. We are even more excited for the players in our tournament to compete against–and beat–our model. This will give us far more insights into where humans excel, and where machines excel.

While I am a fan of Justice Ginsburg (in particular her neck doilies), I am getting somewhat tired of the unflagging adulation of her. Her interview with Katie Couric had enough softballs to field an entire little league team. For example, RBG said that Hobby Lobby “have no constitutional right to foist that belief on the hundreds and hundreds of women.” She later discussed the Free Exercise Clause. She knows that Hobby Lobby was about RFRA, and not the First Amendment. Any competent interviewer would have pushed back. But Katie Couric was too busy gazing into RBG’s closet. I can’t imagine a similar interview with Justice Scalia going the same way–think of her grilling of Sarah Palin back in 2008.

But all of this hero worship may be having a deleterious impact on Notorious RBG herself. This stuff goes to your head, or jabot. I was stunned that she  took Couric’s bait, and said that her five male colleagues have a “blind spot” when it comes to woman.

Couric: “All three women Justices were in the minority in the Hobby Lobby decision. Do you believe that the five male Justices truly understood the ramifications of their decision?”

Ginsburg: “I would have to say no, but I am ever hopeful that if the Court has a blind spot today, its eyes will be open tomorrow.”

Couric: “But you do in fact feel these five Justices had a bit of a blind spot?”

Ginsburg: “In Hobby Lobby? Yes. Yes, I did.”

Couric: “And why was that?”

Ginsburg: “The same kind of blind spot the majority had in the Ledbetter case”

That is a stunning rebuke to her colleagues. Not only does she refuse agree to disagree about the interpretation of RFRA, but she said that they are *blind* to the plight of the woman–putting aside the law, because whatever. This is akin to Justice Sotomayor’s dissent in Schuette, who explained that Chief Justice Roberts can’t understand how racial minorities feel, and that he is “out of touch.” That is a blind spot. Such rhetoric is par for the course for the chattering class, but it is unseemly from the Justices themselves. RBG joined Sotomayor’s opinion, but I thought she was better than that. I guess not.

For further evidence of her own visions of grandeur, Consider Justice Ginsburg’s interview with Joan Biskupic on Thursday. She literally thinks she is the best, and no one can replace her:

Referring to the political polarization in Washington and the unlikelihood that another liberal in her mold could be confirmed by the Senate, Ginsburg, the senior liberal on the nine-member bench, asked rhetorically, “So tell me who the president could have nominated this spring that you would rather see on the court than me?”

That is some hubris. Oh I’m sure I could find hundreds of law professors who would rather see someone half her age on the Court, even someone more moderate. What RBG maybe doesn’t realize is the non-starstruck lawyers care less about who sits on the Court than who is casting votes for years to come. I wonder if she would have deigned to say that in 2009 when Judge Sotomayor and Solicitor General Elena Kagan were cozying up the short list.

In the past she explained that she wants to stick around until she can no longer do the job. But now, we see the ulterior motive–she doesn’t think anyone else can do the job better than her. And all of the praise of her no doubt heightens this. I still think she should decide when she retires, but now we know why she’s sticking around.

After a certain point, it becomes difficult to separate Justice Ruth Bader Ginsburg and the Notorious RBG. As a cause célèbre, she is now beyond the reach of normal commentary on the Court. Criticizing her opinions amounts to criticizing women’s rights more broadly. For example, when Justice Alito responded to charges (many extremely exaggerated) in RBG’s Hobby Lobby dissent, the sense on the left was that he was attacking women, and RBG in particular. When I write a post about her decisions, I find myself double-checking adjectives for sensitivity, which is something I would not think twice about if I was writing about a Breyer or Stevens opinion. It becomes very dangerous when the law transcends the judicial opinions, and the Justices themselves become the locus of the constitutional discourse. For all the talk about polarization on the Court by the conservatives, it is the Court’s liberal wing–RBG and Sotomayor in particular–who are fragmenting the unity, both in their opinions and in public.

In an earlier post, I noted that Erwin Chemerinsky incorrectly labelled Bush v. Gore as case by the Roberts Court. Bush v. Gore was decided in 2000 and CJ Roberts was confirmed in 2005. But maybe I was wrong.

David Cole, in his review of several recent books about the Supreme Court argues that the Roberts Court began with Bush v. Gore:

The Roberts Court, which just concluded its ninth term, was officially launched on September 29, 2005, when Justice John Paul Stevens administered the oath to the newly confirmed chief justice, John Roberts. In a more significant sense, however, the Roberts Court’s birth—or at least conception—should be dated five years earlier, to December 12, 2000. That’s the day the Supreme Court decided Bush v. Gore, ending a recount of the too-close-to-call Florida presidential vote, and ensuring that George W. Bush would become president with half a million fewer popular votes than Al Gore. The Court’s five conservatives—William Rehnquist, Anthony Kennedy, Antonin Scalia, Sandra Day O’Connor, and Clarence Thomas—relied on a wholly unprecedented theory of the Constitution’s guarantee of “equal protection of the laws,” which they announced would apply this one time only, to block the Florida recount, and President Bush took office.

Bush was reelected in 2004, this time without needing the Supreme Court’s help, and that meant that when Justice O’Connor announced her retirement and Chief Justice Rehnquist died in office in 2005, President Bush, not Al Gore or a successor, had the privilege of appointing two new justices and shaping the Court for years to come.

Cole then lists a parade of non-horribles that would have happened if President Al Gore replaced Rehnquist, with, I don’t know, Chief Justice Ruthe Bader Ginsburg, and added, why not, Associate Justice Diane Wood. And, to be sure, Justices Souter and Stevens probably would have retired sooner, giving Gore more choices. So let’s add Justices Kagan and Sotomayor for good measure. But would O’Connor have been so willing to retire early? Anyway, here’s the alternate reality:

Had a Democratic president been able to replace Rehnquist and O’Connor, constitutional law today would be dramatically different. Affirmative action would be on firm constitutional ground. The Voting Rights Act would remain in place. The Second Amendment would protect only the state’s authority to raise militias, not private individuals’ right to own guns. Women’s right to terminate a pregnancy would be robustly protected. The validity of Obamacare would never have been in doubt. Consumers and employees would be able to challenge abusive corporate action in class action lawsuits. And Citizens United v. Federal Election Commission, which struck down regulations on corporate political campaign expenditures and called into question a range of campaign spending rules, would have come out the other way.

But it was not to be.

Of course, a final recount  of all ballots would most likely given Florida to Bush, by a razor-thin margin. As the Washington Post reported in 2001:

In all likelihood, George W. Bush still would have won Florida and the presidency last year if either of two limited recounts — one requested by Al Gore, the other ordered by the Florida Supreme Court — had been completed, according to a study commissioned by The Washington Post and other news organizations.

But if Gore had found a way to trigger a statewide recount of all disputed ballots, or if the courts had required it, the result likely would have been different. An examination of uncounted ballots throughout Florida found enough where voter intent was clear to give Gore the narrowest of margins.

Interestingly, the litigation strategies of both Bush and Gore were probably counterproductive:

But an examination of the disputed ballots suggests that in hindsight the battalions of lawyers and election experts who descended on Florida pursued strategies that ended up working against the interests of their candidates.

The study indicates, for example, that Bush had less to fear from the recounts underway than he thought. Under any standard used to judge the ballots in the four counties where Gore lawyers had sought a recount — Palm Beach, Broward, Miami-Dade and Volusia — Bush still ended up with more votes than Gore, according to the study. Bush also would have had more votes if the limited statewide recount ordered by the Florida Supreme Court and then stopped by the U.S. Supreme Court had been carried through.

Had Bush not been party to short-circuiting those recounts, he might have escaped criticism that his victory hinged on legal maneuvering rather than on counting the votes.

In Gore’s case, the decision to ask for recounts in four counties rather than seek a statewide recount ultimately had far greater impact. But in the chaos of the early days of the recount battle, when Gore needed additional votes as quickly as possible and recounts in the four heavily Democratic counties offered him that possibility, that was not so obvious.

One of the biggest criticisms of the D.C. Circuit’s opinion in Halbig, is that it would have been ridiculous for the Affordable Care Act, which aims to provide “near-universal” health insurance, to deny tax credits to residents of uncooperative states. Why, critics argue, would the drafters of the law allow recalcitrant Republican governors to sabotage its implementation and deny benefits to their poorest residents?

In a new article in the American Spectator, titled “Obamacare Was Designed to Punish Uncooperative States,” I show how another key portion of the Affordable Care Act was designed to operate in just this fashion–the Medicaid expansion. Recall that in NFIB v. Sebelius, one of the main issues was whether HHS would take away all of a state’s Medicaid budget if it did not expand under the ACA. The Justices pressed Solicitor General Verrilli on this point, based on a letter HHS sent to Arizona Governor Jan Bewer threatening to gut her entire Medicaid budget.

In 2010, Arizona inquired about what would happen if it declined to expand its Medicaid coverage under Obamacare. The federal government replied that it would eliminate its contribution to the state’s Medicaid budget entirely. The Department of Health and Human Services sent Arizona Governor Jan Brewer an ominous and pointed letter: “In order to retain the current level of existing funding, the state would need to comply with the new conditions under the ACA.” This observation was followed by a stark warning: “We want you to be aware that it appears that your request…would result in a loss of [all] Medicaid funding for Arizona.”

Arizona stood to lose almost $8 billion. That would have obliterated the Grand Canyon State’s budget, eliminated health insurance for the poorest of Arizonans, and potentially thrown the entire market into an adverse-selection death spiral. The feds could not force Arizona to expand Medicaid, but if Arizona declined to play ball, they would gut the state’s program entirely. It was Brewer’s choice.

Ultimately, the Supreme Court held that this was no meaningful choice at all—like putting a gun to someone’s head and saying “your money or your life.” Though conservatives were ultimately disappointed with Obamacare’s trip to the high court, this was their silver lining. During oral arguments, the justices pressed Solicitor General Donald Verrilli, the top government lawyer, about the letter to Arizona. They wanted to know whether the federal government would in fact take away all of Arizona’s funding if it declined to participate in the expansion. Verrilli refused to answer whether HHS Secretary Kathleen Sebelius had the power to revoke all funds, saying only that she would not exercise it.

The SG refused to answer the questions of the Justices, because he knew that Secretary Sebelius would not disclaim that power (I provide a lengthy background discussion in Part VI of Unprecedented).

Justice Scalia ribbed: “I wouldn’t think that is a surprise question.” The solicitor general was not unprepared. Verrilli told the justices, “I’m trying to be careful about the authority of the secretary of health and human services and how it will apply in the future.”

The Obama administration made a conscious choice, as I discuss in Unprecedented: The Constitutional Challenge of Obamacare. Sebelius was not willing to relinquish the power to deprive uncooperative states of all of their Medicaid funding. She wanted to preserve this ultimate authority to punish a state that disobeyed federal dictates. Verrilli’s evasion inflamed the concerns of Justices Roberts, Breyer, and Kagan, the very justices who would soon vote against him. The Obama administration would not disclaim that power, so the court took it from them.

Chief Justice Roberts wrote for the court, “What Congress is not free to do is to penalize states that choose not to participate in that new program by taking away their existing Medicaid funding.” It was coercive and unconstitutional. Yet this was the original design of the law, which no Democrats in Congress found objectionable.

Obamacare’s history, including the Solicitor General’s position in NFIB v. Sebelius, suggest that a core feature of the ACA would have punished noncompliant states, and their residents. While we may never know how those four words wound up in the statute,  as further evidence of legislators’ state of mind, we could take notice of the fact that the Affordable Care Act’s Medicaid expansion worked exactly on this theory of carrots and sticks. Uncooperative states, and their residents, would be punished.

Please note that my piece has nothing to do with Chevron. Rather, in light of the reasoning behind the Medicaid expansion, as both the D.C. and Fourth Circuit concluded, it is at least “plausible” that these four words were deliberately placed in the statute for this purpose.

Both courts acknowledged that legislative history does not say how these four words made their way into the statute. With this silence, the Fourth Circuit noted that “it is at least plausiblethat Congress would have wanted to ensure state involvement in the creation and operation of the Exchanges.” The D.C. Circuit likewise observed, “the most that can be said of [the challenger’s] theory is that it is plausible.” It’s telling that both courts independently came up with the exact same description: “plausible.”

That no one found this Medicaid regime objectionable in 2009 suggests that the tax credit scheme may not have been objectionable as it seems today.

Here is my conclusion:

Keeping the history of the law’s Medicaid provisions in mind, consider again whether it is indeed “plausible” that someone in the Senate—or maybe even an influential lobbyist or academic helping to draft the bill—intended with these four words to dangle similar carrots to induce unwilling states to establish health insurance exchanges. Perhaps some states would have initially declined to build exchanges, and in those cases HHS had the authority to operate a federal exchange as a backup. Could red-state governors have long handled the backlash from their citizens being punished with unaffordable insurance premiums? Maybe. Maybe not.

But such an incentive structure is at least consistent with the thinking behind other provisions of Obamacare: that states can be made to swallow bitter pills.

In considering this law, we should never lose sight of the fact that this was a rushed, draft piece of legislation that was never meant to be final. But this is the final law we have.

No one knows for sure how these four words made their way into the statute—which is to be expected. The 3,000-page bill was drafted quickly behind closed doors. Almost no one in the Senate bothered to read it. The version of the bill that passed the Senate on a straight party-line vote on Christmas Eve 2009 was only a draft that was never intended to become law. However, once Democrats lost their filibuster-proof majority after the election of Senator Scott Brown in Massachusetts, they decided to foist this incomplete, rushed, and unfinished draft on the American people.

And in January 2010, 51 health policy experts–including Tim Jost and Jon Gruber–urged the President to sign the incomplete Senate bill, warts and all.

These bills are imperfect. Yet they represent a huge step forward in creating a more humane, effective, and sustainable health care system for every American. We have come further than we have ever come before. Only two steps remain. The House must adopt the Senate bill, and the President must sign it.

So here we are, with the imperfect bill.

Update: Peter Suderman weighs in with similar thoughts:

Indeed, that explains Obamacare’s original Medicaid expansion mechanism, which, prior to Supreme Court intervention, threatened states that did not participate with the loss of all existing federal Medicaid funding. This would have been a huge price, and would have effectively gutted one of the nation’s two major health programs had a large number of states chosen not to comply.

If conditioning subsidies on the state establishment of exchanges was “legislative nihilism,” then so too was Obamacare’s original treatment of its Medicaid expansion.

The reason that Congress made its Medicaid threat so easily is that it was assumed that every state would eventually play ball and expand the program. Funding for the expansion was generous, and resistance would be catastrophically expensive.

The federal exchange system was such an afterthought that the law provided no funding whatsoever to create it. Federal health authorities had to scramble to rewire funding in order to get it built. In contrast, Obamacare provided nearly unlimited funds for states to set up their own exchanges. The thinking was that no state would turn the government down.

The total lack of funding for the federal exchange strongly suggests that Congress didn’t intend any subsidies to flow through the federal exchanges, because Congress didn’t really intend for them to exist.

 

In 2009, I created FantasySCOTUS. What started off as a joke, and began as a hastily-put-together website, has grown beyond my wildest imagination. Now, we have over 20,000 players who make predictions about how the Justices will decide cases. In an article I co-authored in 2011, we found that our prediction market was strikingly accurate, with the power predictors hitting a 75% accuracy rate in a given year.

Today, I am proud to announce the next evolution in Supreme Court prediction. Rather than relying on human predictions, my colleagues and I have developed an algorithm that can predict any case decided by the Supreme Court, since 1953, using only information available at the time of the cert grant.

In a new article, my co-authors Daniel Martin Katz, Michael J. Bommarito II, and I have designed a general approach to predicting the behavior of the Supreme Court of the United States. Using only data available prior to the date of decision, our model correctly identifies 69.7% of the Court’s overall affirm and reverse decisions and correctly forecasts 70.9% of the votes of individual justices across 7,700 cases and more than 68,000 justice votes.

While other models have achieved comparable accuracy rates, they were only designed to work at a single point in time with a single set of nine justices. Our model has proven consistently accurate at predicting six decades of behavior of thirty Justices appointed by thirteen Presidents. It works for the Roberts Court as well as it does for the Rehnquist, Burger, and Warren Courts. It works for Scalia, Thomas, and Alito as well as it does for Douglas, Brennan, and Marshall. Plus, we can predict Harlan, Powell, O’Connor, and Kennedy.

69.7% Accuracy for Case Outcomes

We begin making forward prediction starting with the first case of the Warren Court in 1953, through the end of the 2012-2013 term. For each of the predictions, offered over 60 years–7,700 cases and in excess of 68,000 individual justice votes–we only rely on data that would have been available prior to the Court’s decision. In effect, we generated a new round of predictions every day of every Supreme Court term since 1953. With this data, through a method of machine-learning known as “extremely randomized trees,” and a process known as feature-engineering, we were able to build a model that can generate Justice-by-Justice predictions for any case. Applying the extremely randomized trees approach to each case from 1953-2013, our model correctly forecasts 69.7% of Case Outcomes and 70.9% of Justice Level Vote Outcomes over the sixty year period.

This graph illustrates our accuracy rate over the past six decades. Although our accurate rate fluctuates year-to-year–as low as 60% and as high as 80%–the best fit line hovers right around 71%. We tended to be a bit more accurate during the Warren and Burger courts than during the Rehnquist and Roberts Courts.Recent courts have had much more variability.

70.9% Accuracy for Justice Predictions

Overall, we have a 70.9% accuracy rate for justice predictions. Some justices were harder to predict than others. To illustrate the “predictability” of a Justice, we generated a heat map.  On this map, we’ve plotted each Justice who has served on the Court, and for each year added a shaded box. The more green the cell, the more predictable the Justice in that year. Our method performs well at predicting certain Justices and not as well on others.

For example, Justices Harlan, Frankfurter, and Burton prove comparatively difficult to predict. All of these Justices are closer to the ideological center. By contrast, our method is fairly accurate at predicting the behavior of Justices Douglas, Brennan, and Thomas. These justices are quite far from the ideological center. There are, of course, notable exceptions. Justice Stevens begins as a difficult to predict justice but over time becomes increasingly easier to predict. In his years as an Associate Justice our performance in predicting William Rehnquist is relatively strong. This changes almost immediately following his elevation to Chief Justice in 1986 when our performance begins to decline. Our model learns, and can track a Justice’s shifting throughout his or her tenure from appointment till retirement.

Predicting Affirmances and Reversals

We are also able to break down our accuracy based on the vote configurations. Our model tracks the commonplace intuition that 9-0 reversals are easier to forecast than 5-4 reversals. While our performance between these categories is somewhat close in certain years, we consistently perform better in unanimous reversal cases than in cases which feature disagreement between justices. We also perform better on cases with a vote of 9-0 to affirm than in cases that affirm through a divided court.

Our model struggles to identify in advance cases that the Court ultimately decides to affirm–especially unanimous affirmances. Since 1953, the Court has affirmed 2,623 cases or 34.1% of its fully argued cases. On this subset of cases, our model does not perform particularly well. In some years, we are able to forecast less than 25% of these cases correctly.  This graph represents the overall reverse rate.

Machine Learning Model – Extremely Randomized Trees

So how does our algorithm work? Our model generates many randomized decision trees that try to predict the outcome of the cases, with different variables receiving different weights. This is known as the “extremely randomized trees” method. Then, the model compares the predictions of the trees to what actually happened, and learns what works, and what doesn’t. This process is repeated process many, many times, to calculate the weights that should be afforded to different variables. In the end, the model creates a general model to predict all cases across all courts. You can download all of our source code here.

This general model is represented by this graph, which lists the 90+ variables we consider for each case, and their relevant weights.

Collectively, individual case features account for approximately 23% of predictive power while Justice and Court level background information account for just 4.4%. Much of the predictive power of our model is driven by tracking a variety of behavioral trends. This includes tracking the ideological direction of overall voting trends as well as the voting behavior of various justices. Differences in these trends prove particular useful for prediction. These include general and issue specific differences between individual justices and the balance of the Court as well as ideological differences between the Supreme Court and lower courts. Contrary to what many may think, it’s not all about ideology. The identity of the petitioner, respondent, what the cause of action is, what Circuit the case arises from, and other case-specific features are very significant.

The Tournament

So what’s next? I’m sure you’re wondering how our model will do with future cases decided in the upcoming term. That’s the plan.

This year we will be hosting a tournament where the players of FantasySCOTUS will compete against our algorithm. What IBM’s Watson did on Jeopardy, our model aims to do for the Supreme Court. Stay tuned for more details.