Conventional wisdom holds that access to legal representation increases the chances of securing a favorable outcome.
Harvard Law School professor James Greiner and statistics graduate student Cassandra Wolos Pattanayak teamed up three years ago to test that assumption through a rigorous methodological study involving people fighting for unemployment benefits. They arrived at a surprising conclusion: Bringing in a lawyer made no difference to the claimant’s odds of victory and could even harm his or her interests by delaying the process.
Law and statistics might seem like strange bedfellows, but more legal scholars appear to be embracing the data-driven approach to research that has long been the norm in political science, economics and the social sciences. Legal scholarship traditionally involved analyzing cases, but some law professors are using statistics to examine everything from judicial ideology to the effects of tort reform.
Proponents of this movement — dubbed empirical legal studies — view it as a major trend in legal academia. Harnessing data to answer legal questions gives their work credibility, and empirical research tends to reach a wider audience than traditional legal scholarship, they say.
“Outside the walls of law schools, almost no one cares about my doctrinal work,” said Theodore Eisenberg, a professor at Cornell Law School who has published empirical studies of settlement rates, attorney fees in class actions and the effects of defendants’ criminal records in trial outcomes. “But policymakers and the media have paid attention to my empirical work on what’s going on in the legal system.”
Not every law professor is ready to grab a calculator and start running regressions. Critics deride empiricism as a fad that might actually be detracting from the quality of classroom instruction.
“One of the concerns I have is on the pedagogical side,” said University of Chicago Law School professor Brian Leiter. “There are some very smart and talented people doing this, but what does it add in the classroom, and how does it help teach analytical reasoning?”
The empirical study of law is not necessarily a new phenomenon — law professors did this type of work from about 1910 through the 1930s, said Herbert Kritzer, a professor at the University of Minnesota Law School. Scholars continued to conduct empirical legal research, but they typically were academics from other fields — say, economics and sociology — not law professors. Only during the past 10 years have empirical legal studies become more mainstream within the legal academy, Kritzer said. Debate over the merits and limitations of empirical legal scholarship have become staples of the legal blogosphere during the past year.
It’s hard to quantify this movement — there is little evidence tracking its growth, although a 2005 article by Vanderbilt University Law School professor Tracey George reported that mentions of “statistical significance” among Westlaw documents grew nearly every year between 1990 and 2003.
Cornell Law School and Eisenberg in 2004 founded the Journal of Empirical Legal Studies, a peer-reviewed journal devoted specifically to empirical research in the law. The first Conference on Empirical Legal Studies was held two years later with about 200 attendees. The annual conference now draws more than 400, according to Eisenberg.
The University of California at Los Angeles School of Law, Washington University in St. Louis School of Law, the University of California, Berkeley School of Law and Northwestern University School of Law are among the institutions that have established centers for empirical legal studies.
Empirical legal studies “has always been on the outskirts, but now law schools are looking for it,” Eisenberg said. “The more people we get doing it, the faster acceptance will come.”
The real value in having lawyers conduct empirical work is that they tend to ask different questions than do academics in other fields, said Andrew Martin, director of Washington University’s Center for Empirical Research in the Law. Political scientists tend to focus on the disposition of cases, while lawyers care more about how winners and losers are determined, he said. Understanding that process clearly benefits law students and professors, he said.
A persistent critique of law professors conducting empirical research is that they often aren’t very good at it. The old joke goes that people land at law school because they don’t like math.
Law schools are addressing this knowledge deficit in two ways. Elite schools are hiring holders of doctorates from other academic areas who have training in data-based research. Some of those doctorates also hold J.D. degrees, while others don’t. At Northwestern, about one-third of the law faculty hold a doctorate.
Several schools hold workshops and classes on empirical legal studies geared toward law professors and students. Martin has teamed up with Northwestern law professor Lee Epstein since 2002 to offer an annual two-day empirical legal studies boot camp, and 450 people have participated thus far. Both Northwestern and George Mason University School of Law plan training sessions in May.
“We see two kinds of people in the workshop,” Epstein said. “There are the people who really want to be able to read and understand the empirical articles, and then there are the people who really want to do this type of work. The workshop is a way to stick their toe into the water.”
UCLA law professor Stephen Bainbridge disdains the trend. “A lot of the people I see who are empiricists, often with doctorates in the social sciences, aren’t very good lawyers,” he said. “I’ve read numerous papers that just got the law wrong. The problem is that we’re hiring people with Ph.D.s in other fields, but their law credentials are middling at best. Someone who is a brilliant economist wants to be in a economics department, so we get second-rate lawyers who are second-rate in their academic field.”
Empiricists acknowledge that not all data-driven work is excellent. Epstein co-authored a scathing critique of the accuracy of empirical legal studies in 2002, but said the quality of the work has improved significantly since then.
One of the most high-profile examples of flawed empirical scholarship was a 2008 article by Tulane University Law School professor Vernon Palmer and Loyola University New Orleans economics professor John Levendis. They concluded that Louisiana Supreme Court judges overwhelmingly ruled in favor of parties that had made campaign contributions. However, problems with the data were discovered after the article was published in the Tulane Law Review, and it was retracted.
The student-edited journal system presents a hurdle for empirical legal studies because so few law students have the methodological background to review empirical work, several researchers said. Student journal editors have started sending those submissions for outside review by people with training in empirical-research methods. Law professors, meanwhile, are collaborating with statisticians or scholars in other fields. “Traditional doctrinal research is a one-person event usually,” Eisenberg said. “So-called social science research is much more of a team effort.”
One reason for collaborations of this sort is that gathering data is complicated and time-consuming. For example, before Greiner and Pattanayak could begin their study on legal representation, they had to convince the Harvard Legal Aid Bureau to supply cases and offer information regarding its assistance to people facing administrative hearings on unemployment benefits. Then they collected a randomized sample of those being offered legal representation though the clinic — focusing only on the outcomes of those represented by the clinic would have skewed the sample. They collected case and offer information for 18 months before coding and analyzing the data.
“I thought we’d find a 15[%] to 20% improvement among those with offers, maybe more,” Greiner said. “In fact, our study found that it probably causes a mild harm,” delaying benefits payouts by several weeks.
Greiner cautioned that the study was limited in scope and was not an indictment of the work of Harvard students or of legal aid efforts (Harvard Legal Aid Bureau leaders circulated a letter pointing out that the study looked at only a small subset of clients and examined outcomes of offers of assistance by the clinic rather than outcomes of those it ultimately represented). However, Greiner hopes that the results will help the Harvard Legal Aid Bureau improve its allocation of resources and services.
The rise of online databases is yet another reason that empirical legal studies is growing in popularity, Martin said. His center maintains databases covering civil rights litigation, U.S. Supreme Court decisions and Equal Employment Opportunity Commission litigation that are available to researchers. “We have a lot more data we can mine on the Internet and we have much more computing power than we did 20 years ago,” he said.
Data also are available through government-funded entities such as the Administrative Office of the U.S. Courts and the Federal Bureau of Investigation’s Uniform Crime Reporting Program.
Empirical scholars have been knocked for conducting research simply because a data set is available, not because they have an important subject to explore. “They can only answer questions when they can get numbers,” Bainbridge said. “If they can’t count something, it doesn’t exist for their purposes.”
That criticism rings hollow to Eisenberg, who said that empirical researchers are wise to apply the data that the government or other organizations have invested in assembling.
The rise of empirical legal studies has created tension between believers and those who see it as a trendy distraction. That level of tension varies from school to school, depending on the priorities of individual administrations, Epstein said. “Everybody plays to their competitive advantage. If your background is in doctrinal work, than that’s the most important type of work. If it’s in empirical work, that that’s most important. Eventually, the idea is that the two will inform each other. Things will settle down.”
Several empirical scholars said that some doctrinal scholars simply don’t understand their work. Bainbridge insisted that the skepticism is justified. “I think ELS is pretty much getting a free ride,” he said. “There are a lot of people who have decided, for whatever reason, that this is the coming thing. We need to have a much more vigorous debate about the place of empirical legal studies in the legal academy.”
Doctrinal legal scholarship is in no danger of being eclipsed and replaced by empirical work, Eisenberg said. Far more research based on case analysis is coming out of law schools than is empirical work, and many professors choose to do both.
“What’s the right balance? Is it 50-50? We’re nowhere near that,” Eisenberg said. “Right now, there’s no threat of people doing empirical outnumbering the doctrinal faculty.”
Karen Sloan can be contacted at email@example.com.