Many lawyers shy away from statistical analysis precisely because math wasn’t their strong suit in school, but the good news is that what sometimes appears to be a complex math problem actually has little to do with mathematics. Instead, it’s a problem of labeling and categorizing.
Statistical analysis is powerfully explanatory, counterintuitive, misleading and readily exploitable—all at once. It can reveal patterns and associations that otherwise are undetectable, yet it can blind people to deeper and more fundamental patterns and associations. Its reliance on probability provides a logical foundation for inferences while threatening to trigger cognitive biases responsible for numerous logical fallacies.
Fortunately, sorting it all out depends more on a skill common to lawyers—the ability to interrogate evidence and arguments—than the skills of mathematicians.
We lawyers come by our blind spot for statistics honestly. This dawned on me while looking over my youngest child’s homework. One of the problems went like this: Tom’s class went to the berry patch and picked 18 berries. On the way home he ate seven berries. How many berries did Tom have when he got home?
I realized that teachers instruct us on this business of labeling and numbering things almost from the moment we start school, so it appears sensible and unremarkable. It all seems innocent enough, but imagine that Tom’s 18 berries included 10 blueberries, three raspberries, three blackberries and one strawberry.
“They’re just trying to make a math problem for little kids, not teach them the different sorts of berries,” you might object.
OK, let’s say they’re all blueberries, but Tom ate the seven plump, ripe ones and none of the smaller ones.
“Same objection,” you might retort.
But the point remains. If we really are taking Tom’s data and trying to draw some inference from it, the failure to remove the berry label and dig into the data allows Tom’s real preferences to remain hidden. Any conclusions drawn from the data are thus shaky at best.
Getting at what the data actually reveals means questioning labels. Converting the data into graphics is a great way to interrogate labels. Take, for instance, the following example taken from Ken Ross’ book, A Mathematician at the Ballpark: Odds and Probabilities for Baseball Fans.
Dave Justice had a better batting average than Derek Jeter in both 1995 and 1996, yet Jeter’s combined 1995-’96 batting average was higher than Justice’s combined 1995-’96 batting average. How can that be?
Looking at the actual numbers confirms the math but compounds the confusion: Justice batted .253 and .321 in ’95 and ’96, respectively. Jeter batted .250 and .314 those same seasons.
How could Justice do better each year but worse for both? The key is to convert the data into pie charts scaled to the number of at-bats. It’s suddenly obvious that the players had very different sorts of seasons.
In the first year, Justice played full time, getting 104 hits out of 411 at-bats, while Jeter rode the bench, getting 12 hits out of 48 at-bats.
In the second year, their situations were reversed, with Justice getting 45 hits out of 140 at-bats and Jeter getting 183 hits out of 582 at-bats.
Jeter is clearly the better full-time batter (.314 versus .253), and that’s what drove his combined two-season average past Justice’s. Tellingly, their career batting averages wound up being almost identical to their 1995-’96 combined averages.
Does that mean aggregating or combining data is always the way to go? No, that misses the point. Take another example of Simpson’s Paradox, which occurs when the conclusion drawn from some set of data is completely reversed when that data is either aggregated or disaggregated.
This example is presented in a 1978 statistics textbook by David Freedman, Robert Pisani and Roger Purves. Consider graduate admissions at a university. Combining all the admissions data for the various graduate schools revealed that the university admitted 44 percent of all men who applied compared to only 35 percent of all women who applied.
Yet, something unexpected emerges when looking at the department-by-department data. The university was more likely to admit women than men across the departments.
For example, despite the fact that Department X had a higher rate of admission for men than women (37 percent of men versus 34 percent of women), it still admitted more women (201 women compared to 120 men). The reason? A disproportionately large number of women applied to the department; thus, the department inevitably had to reject a larger number of women (593 women applied compared to 325 men).
Again, these pie charts help us understand how it could be that women’s rate of admissions could be lower overall but higher in almost every department. The point, then, is that seemingly ordinary and obvious labels like “berries,” “baseball season” and “graduate school” can hide differences at least as important as those between apples and oranges.
Let’s consider another hypothetical example involving labelling and risk, this time in the medical context. Imagine that a patient’s mammogram come back positive. The patient asks her doctor if she has breast cancer.
Let’s assume three things: A mammogram detects breast cancer 95 percent of the time if a patient has it, a mammogram falsely indicates breast cancer only 5 percent of the time, and one in 100 women the patient’s age has breast cancer. What should the doctor tell her?
It may not look like an apples-and-oranges problem, but it is. Remember that the 95 percent detection rate goes with the 1 percent of women who have breast cancer, while the 5 percent false-positive rate goes with the 99 percent of women who don’t.
She has a five-in-six chance of not having breast cancer despite having a positive mammogram. As I hope you can see from these examples, which required nothing more challenging than multiplication and division, it’s not always the math that causes the confusion when it comes to statistical analysis; often it’s the language.
Remember that, while you can’t do science unless things are labeled, labeling often hides truths and misdirects listeners, thereby presenting an opportunity both for mischief-making and discovery.
David A. Oliver is a partner in Vorys Sater Seymour and Pease in Houston. He is board-certified in personal-injury trial law by the Texas Board of Legal Specialization. He litigates allegations of injuries due to exposure to chemicals or pharmaceuticals. He is the editor of the blog Mass Torts: State of the Art.