What happens when an empirical piece makes a big mistake? From Paul Caron, we see a “clarification” essay from Professor Katherine Y. Barnes (Arizona) in Is Affirmative Action Responsible for the Achievement Gap Between Black and White Law Students, where she acknowledges some errors in her methodology from a 2007 article on the so-called “mismatch theory” of Affirmative Action. I don’t have any comments on her methodology, but her mea culpa is somewhat interesting, and should serve as a cautionary note about the reliance on empirical legal studies by law profs:
In 2008, Professors Doug Williams and Richard Sander contacted me regarding replication of my results. Unfortunately, I had changed institu- tions between the time the essay was slated for publication and this contact. Due to my own negligence, although I thought I had transferred all of my files to my computer at my new institution, I had not. Thus, I did not have the original programs that I used to analyze the data. I reconstructed the programs for Professors Williams and Sander and their colleagues Dr. Rog- er Bolus and Dr. Marc Luppino but was unable to replicate the same results as presented in the original essay. Because my first commitment is to the truth, or as much thereof as the limits of logic, method, data, and human ca- pacities allow, this Revision followed.
Research is a process of formulating and reformulating theories on the basis of new information. Empirical research, in particular, involves the of- ten public debate regarding the appropriate methods, analysis, and conclu- sions to be drawn from data. By its nature all empirical research is imperfect in some way. Some imperfections are correctible, and although all empirical researchers hope that mistakes in analysis are infrequent, the academic process of replication, further investigation, and debate (like the methods of science more generally) is built to find flaws in current research in order to improve knowledge.
I am very grateful to Professor Williams, Professor Sander, Dr. Luppi- no, and Dr. Bolus for their effort in replicating my results and their dili- gence in helping to advance our understanding of the empirical validity of the mismatch hypothesis. I mix a sense of embarrassment at the problems with my original analysis with the pleasure of seeing the methods of science and scholarly discourse work the way they should.
Before revisiting the main arguments and results of my 2007 essay, I would like to take a moment to reflect on the process by which law profes- sors publish their research and how others can avoid my error in the future. The preferred method to avoid errors is, of course, not to make them. This, however, is not always possible.
For myself, my practice for submitted papers has changed. The best practice to avoid mistakes in coding is to double code every program,6 and I now do so. I also keep a pristine copy of the program used for the results in the submitted version of the paper in a separate “read-only” directory, where future changes in the program will not be confused with the analysis from the submitted paper.
Beyond the personal responsibility of researchers, law reviews also have the responsibility to facilitate replication. Journals should have a sys- tematic policy that programs that support results must be submitted with the article,7 that programs and datasets must be posted on the law journal’s website, and that technical appendices must be published on the law jour- nal’s website to explain more of the details of the analysis. Because of the large number of law journals and the fact that their editorships change every year, long-term policies are more difficult to implement and retain. Al- though difficult, law reviews should commit to these best practices when evaluating and publishing empirical research. Indeed, it is optimal to have a more comprehensive clearinghouse, such as SSRN, which would alleviate these structural pressures by providing one well-maintained location to house programs and datasets. Although I believe such policies would help prevent future errors and should therefore be implemented widely, I take full responsibility for my error in not maintaining my program appropriate- ly, as well as any errors in analysis which that program would have illuminated. Nonetheless, I am pleased that Northwestern University Law Review has agreed to make available a copy of all my data and programs on their website to aid future research and policy on this topic.8
Now what happens if no one caught this error? It would have continued to be cited (I presume). Does the fault lay with the author? With the journal? With the lack of peer-review for legal publications? With law professors who are trained in the law (and statistics only secondarily) trying to do empirical studies?
And what if policy-makers or legislators passed laws on the basis of this now-refuted article?
This passage in the conclusion is somewhat telling:
Professors Williams, Sander, Luppino, and Bolus write that my con- clusions are “exactly opposite” to the conclusions in my 2007 essay, sug- gesting that my revised results support mismatch.36 This is incorrect. Their first argument is that ending affirmative action would increase the percen- tage of black law students who pass the bar by 27%.37 This is irrelevant to mismatch. Their second argument is that I have miscoded bar passage in this Revision.38 I fixed this coding but was not permitted to publish it here. Nonetheless, the recoding did not change the conclusions.39