A U.S. Supreme Court mystery drew them together: the Harvard 3L, the engineer, the Jenner & Block associate, the Massachusetts Institute of Technology professor and a team of MIT doctoral students. The mystery: Who actually wrote the joint dissent in last year's health care blockbuster?
Immediately after the June 2012 ruling in National Federation of Independent Business v. Sebelius, intense speculation surrounded the authorship of the joint dissent. Was it primarily the handiwork of Chief Justice John Roberts Jr., who allegedly began writing the dissent and subsequently changed his mind? Was it Justice Anthony Kennedy? How about Justice Antonin Scalia? Could it have been both Kennedy and Scalia or another dissenter?
Words and phrases common to each justice's writing style offered the clues to unraveling, with what the investigators claim is a high degree of accuracy, that mystery and eventually all of the per curiam — unsigned — opinions of the Roberts Court.
William Li, a doctoral student in MIT's computer-science and artificial-intelligence laboratory, approached his professor and principal investigator at the laboratory, Andrew Lo, about whether they could resolve the speculation using natural language-processing tools. Lo sent out a call for students interested in pursuing the project.
"For me, I did my master's degree in technology and policy, and this seemed at the junction of an important policy issue and then on the technology side as well," Li said. "I have a background in natural-language processing."
The important policy issue was what they later would describe in an article as the "shocking number" of unsigned judicial opinions published each year by U.S. courts. Traditionally used to make quick work of noncontroversial cases, per curiams today frequently involve controversial issues and include dissenting opinions.
"Obscuring authorship removes the sense of accountability for each decision's outcome and the reasoning that led to it," Li and his colleagues wrote in "Using Algorithmic Attribution Tech­niques to Determine Authorship in Unsigned Judicial Opinions," 16 Stan. Tech. L. Rev. 503 (2013).
"Anonymity also makes it more difficult for scholars, historians, practitioners, political commentators, and — in the thirty-nine states with elected judges and justices — the electorate, to glean valuable information about legal decision-makers and the way they make their decisions, " they wrote.
The researchers found that the Warren Court used per curiam opinions 28.7 percent of the time; the Burger Court 17.7 percent; the Rehnquist Court 10.3 percent; and the Roberts Court 13.3 percent. In 2011, the federal courts of appeal issued per curiam opinions 7.6 percent of the time, but the rate varied significantly across circuits. For example, the D.C. Circuit relied on per curiam opinions only 0.3 percent of the time, while the Fifth Circuit used them 15.9 percent of the time. "The problem is direr in some state courts, where per curiam opinions constitute more than half of an elected court's decisions," the team wrote.
BUILDING A MODEL
Li, Lo and another MIT doctoral student, Pablo Azar, began discussions last August about how to structure a research project to discover the authors of the health care ruling. James Cox, at the time a Jenner & Block associate, came on board as legal consultant. Cox's role later was taken over by Phil Hill, the Harvard 3L and a fellow at Harvard's Berkman Center for Internet and Society. Rounding out the team were Berkman Center engineer David Larochelle and MIT professor Robert Berwick.
"I was in charge of researching whether unsigned opinions were a bigger problem than just Obamacare," Harvard Law's Hill recalled. "Along the way, I found a great deal of scholarship criticizing unsigned opinions. I saw it was a large issue both in quantity and quality, and I approached them about expanding the project to this highly controversial issue."
The researchers downloaded all Supreme Court decisions written by the nine justices now serving on the court between 2005 and 2011, during Roberts' tenure — a total of 568 opinions. They also obtained the majority and dissenting opinions in the health care case as well as 65 per curiam decisions issued by the Roberts Court before November 2012.
Using algorithms, they worked through the fall of 2012 to identify features of the justices' individual writing style based upon opinions known to have been written by each one. Those characteristics — one-, two- and three-word phrases known as n-grams — they encoded in a statistical prediction model that described the writing styles of the justices under consideration. For example, Justice Clarence Thomas' highly predictive, one-word n-grams include "Therefore," "However" and "explaining." Justice Stephen Breyer's highly predictive three-word n-grams include "in respect to," "For one thing" and "That is because."
Their article notes that their approach has been applied "to a wide range of literary, historical, and contemporary domains, including studies on the Fed­eralist Papers, Shakespeare's plays, and more recently, on large numbers of authors in online blogs or forums."
By Thanksgiving, the model was ready to run on opinions, Li said. "That's when we started to get fairly good accuracy levels," he said. "We continued our discussions, and Phil was involved in where else this could be applied. We were relatively new to studying legal opinions and we were especially interested in the health care decision. But there is a lot of interest in these other unsigned opinions. From November onwards, we decided to focus more on unsigned opinions."
Although their model did not explicitly consider the role of the justices' clerks in the writing process, the researchers assumed, and then tested and validated, that the features of a justice's writing are similar from year to year.
Overall, their model is 81.2 percent accurate, according to Hill. "That is, after training our model on 90 percent of the Roberts Court's signed opinions, we applied the model to the remaining 10 percent of the signed opinions with obvious identifiers like the justice's name removed," he said. "Our predictions matched the signed justice 95 out of 117 times."
In the end, the model predicted that Kennedy and Scalia had the highest probability of authorship and of being prime actors in writing the joint dissent in the health care case. The researchers also identified in their article (see the chart on page 529 of the review) the probable authors of the Roberts Court's 65 per curiam opinions.
"It has been a very interesting endeavor for all of us," said Hill, who recently graduated from Harvard and will join Kirkland & Ellis. "Right after publication, we did an addendum on this term's three big decisions to verify our model: Shelby, Windsor and Perry [Shelby County, Ala. v. Holder; U.S. v. Windsor; Hollingsworth v. Perry ]. For all of those, our results ended up being in line with the actual justices. For the [Defense of Marriage Act] decision, it was 99.7 percent for Kennedy. It seems the model is still working pretty well."
Both Hill and Li believe more work can be done with the machine-learning approach in understanding the dynamics of judicial opinion writing in general.
"For me personally, I'm very interested in text analysis applied to the legal domain," Li said. "One of the ideas I'm working on now is understanding the U.S. legal code — authorship, relationships between different sections."
He acknowledged that it would be nice to be able to test their results against what truly happened in the unsigned opinions.
"We can only rely on models we've trained on past signed opinions," he said. "Hopefully, our article has been able to provide some interesting ideas to people. I don't think we're advocating that all operations of the Supreme Court should be open or public, but to help legal scholars or the public better understand the dynamics of opinion writing, hopefully this would be useful."
And the computer side and the law side even learned how to talk to each other.
"A lot of innovative work comes from this collaboration," Li said. "At our first meetings, we might be speaking different languages and have different conceptions, but at the end, it sort of melded. It was really a lot of fun."
Contact Marcia Coyle at email@example.com.