In late October, The Washington Post Magazine did a fine job discussing the law school graduate (under)employment controversy. So fine, in fact, that the dean of American University’s law school, Claudio Grossman, felt compelled to fire off a letter to the editor criticizing the Post for noting that only 35 percent of his school’s 2011 graduates were employed in full-time, long-term positions as of February 2012. Dean Grossman wrote:

“The Post focused on a single employment statistic that is grossly misleading and relied on a number taken from only one of 16 primary employment categories collected by the American Bar Association. In the case of American University Washington College of Law, a far more appropriate statistic than the one the Post cited is 79.6 percent, which reflects the true employment data and career choices of our 2011 graduates. These graduates are employed in positions requiring bar passage, in positions in which a law degree provides a distinct employment advantage or in other professional positions where developed legal skills are highly valued, or they are pursuing advanced degrees.”

 Is the ABA’s “employed—bar passage required full-time/long-term” category, which the Post used to assess American University, actually misleading? More urgently, what do the various terms used in the “ABA Standard 509 Questionnaire” mean? And are there differences between the terms’ stated definitions and what they mean in practice? Statistical tools can help answer these questions.

The definitions for the terms used in the questionnaire appear on the ABA’s website in a proposed document titled, “ABA Employment Questionnaire Definitions and Instructions” [PDF], one realizes that graduates are asked three primary questions:

(1)    What is the graduate’s employment status? (e.g. “employed – bar passage required, full-time/long-term,” or, “employed – JD advantage, part-time/short-term,” etc.) Employment status includes all of the law school’s graduates, even those who are unemployed or did not return the questionnaire.
(2)    What is the graduate’s employment type? (e.g. “law firm – 2-10, part-time/long-term,” or, “judicial clerkship – state, part-time/short-term,” etc.) Employment type excludes all graduates who are pursuing a graduate degree, are unemployed for any reason, or did not return the survey.
(3)    Is the graduate employed by the law school? Graduates counted in this category are also counted in both the employment status and employment type categories.

The immediate problem is that what the ABA says the terms mean and what the graduates completing the survey say in their responses do not necessarily coincide. For example, the ABA states that the employment type “business and industry” can include “accounting firms, investment banking and financial institutions, entertainment/sports management companies, insurance companies,” etc. But it might also mean service sector jobs that don’t require even a college education. Can we tell the difference from the questionnaire?

Yes, by dividing the employment statuses, employment types, and law school–funded positions by the number of graduates from each law school and then correlating them to one another, we can find commonalities among law schools whose graduating classes found themselves in those categories. The bedrock assumption is that graduate employment outcomes tend to clump together at most law schools, based on the fact that law schools are stratified according to prestige. For example, few, if any, law schools will have some of their graduates working full-time/long-term at 501-plus attorney law firms while others are unemployed or don’t reply to the survey.

One of the benefits of this methodology is that comparisons can be made within the three primary questions in the employment questionnaire, e.g., whether there’s a relationship between “employed – bar passage required” and “employed – JD advantage,” and comparisons are also possible between the three questions, such as between “pursuing graduate degree” and “501-plus attorney law firms.” Here are a few choice examples.

“Employed – Bar Passage Required Full-Time/Long-Term”

Returning to dean Grossman’s criticism, is “employed – bar passage required full-time/long-term” a “grossly misleading” standard for measuring law graduate employment outcomes? I don’t believe so.

It’s understandable that “employed – bar passage required full-time/long-term” correlates to the broader “employed – bar passage required” employment status, but a few more observations can be made about the correlation shown above. One, it establishes that full-time/long-term employment at lawyer jobs corresponds inversely to unemployment. Two, it challenges the claim that the juris doctor is versatile: if it were, we would expect a positive (or at least a less negative) correlation with “employed – JD advantage” and “employed – professional position.” Apparently, law graduates prioritize lawyer positions.

Here’s the correlation between “employed –bar passage required full-time/long-term” and the employment types.

What’s striking about this data is the negative correlation between full-time long-term employment as a lawyer and working as a solo practitioner or in a two-to-10-attorney law firm. You’d think the relationship would be positive. However, the correlation here is between an employment status and the employment types. That means the higher the percentage of a law school’s graduates working in jobs requiring bar passage full-time/long-term, the lower the percentage of those graduates working in small firms, even though solos checked the “employed – bar passage required” box in the questionnaire nonetheless. (Note also the strong connection with federal clerkships.)

Although it appears that The Washington Post Magazine‘s decision to use “employed – bar passage required full-time/long-term” as the standard metric for determining a law school’s success is not “grossly misleading,” we must take care not to commit the “ecological fallacy,” which means believing an average result applies to an entire population when there might be outliers. Indeed, it’s plausible that because they come from a D.C. law school, a higher proportion of American University’s graduates use their legal educations in federal government jobs or lobbying firms, but that’s something the law school will have to demonstrate for itself.

“Law School–Funded Positions”

Meanwhile, the question that should be on everyone’s minds is, what happens to those graduates (and their classmates) who are rehired by their own law schools? Behold:

As we might expect, there is a negative correlation between law school–funded employment and underemployment, and graduates from law schools that have employment programs typically do well. The important fact to consider about law school–funded positions is that they weaken the meaning of the “public interest,” “academia,” and “government,” employment types. This is the consequence of separating out law school–funded work from the other two employment questions: It makes being a law professorís postgraduate research assistant sound more like charity work than it really is because most law schools are nonprofit or public entities.

“Employed – Professional Position” and “Business and Industry”

But what about those who graduate from law school and wind up in “professional positions” and “business and industry”?

We find a strong negative correlation in law schools whose graduates work in professional jobs with graduates who work as licensed attorneys. There’s also a positive correlation with graduates working in nonprofessional positions and unemployment. This is an inauspicious result for anyone thinking that legal education leads to professional nonlawyer work, at least immediately after graduation. The correlation between “employed – professional position” and the employment types in the questionnaire is similarly discouraging.

Law schools whose graduates tend to work in professional positions have graduates who state on the ABA questionnaire that they work in academia, small firms, state governments, and “business and industry.” How does this employment type compare to the other categories?

Like “employed – professional position,” “business and industry” corresponds to graduates in small-law, nonprofessional positions, and unemployment.

I have two more.

The Underemployed

Combining the percentages of “unemployed – start date deferred,” “unemployed – not seeking,” “unemployed – seeking,” and “employment status unknown,” and correlating them to the other categories, we obtain the following results.

I think these results speak for themselves.

Accreditation Year

Rather than using U.S. News‘s rankings, which wouldn’t be very sporting, here’s the correlation between graduate employment outcomes and the year their law schools received ABA accreditation. In this situation, negative correlations mean the older a law school is, the more likely its graduates find themselves in the listed categories.

Graduates from more recently accredited law schools are significantly more likely to be underemployed than their peers who graduated from schools that have been accredited for longer.

To conclude: The correlations here are tendencies and do not apply to all law schools equally, and they only apply to 2011 graduates, but I suspect these tendencies are permanent. Because the distributions of graduates among the categories aren’t equal, the ultimate significance of the correlations might differ. Then again, the point is to get an understanding of what the terms in the employment questionnaire mean in practice, and I believe we can make the following claims:

(1)    Using “employed – bar passage required full-time/long-term” is a reasonable measure of law schools’ job placement success rates. Law schools claiming they are exceptions must demonstrate that their degrees greatly enhanced their graduates’ employment outcomes above what they could have achieved had they not gone to law school, especially because the “business and industry” employment type corresponds strongly to nonprofessional work.
(2)    Law school–funded positions probably dilute the value of the “public interest,” “academia,” and “government” employment types.
(3)    “Employed – professional position” and “business and industry” are both unhelpful terms. More descriptive terms would be something like “jobs that the graduate could have obtained without going to law school.”
(4)    Graduates of more recently accredited law schools tend to have poorer outcomes.

Hopefully, going forward, applicants, law schools, and everyone else will consider the employment questionnaire’s terms’ practical meanings rather than their stated ones.

Matt Leichter is a writer and attorney licensed in Wisconsin and New York, and he holds a master’s degree in International Affairs from Marquette University. He operates The Law School Tuition Bubble, which archives, chronicles, and analyzes the deteriorating American legal education system. It is also a platform for higher education and student debt reform.