As far back as 1835, when the New York Sun reported that there was life on the moon,1 false claims have been presented as scientific fact. In modern times, the principal remedy to such misrepresentations is the injunction of “peer review and publication” stressed in the National Academy of Sciences (NAS) report, “Strengthening Forensic Science in the United States: A Path Forward.” However, due to an uptick of dubious scientific data appearing in scholarly journals, another level of validation is required—citation analysis.

Opposites Retract

“Law and science have their own patterns of information creation, storage, indexing and dissemination. While the scientific method has little in common with legal analysis, the former has been struggling to make peace with the latter in the courtroom for a long time. As a result, it is within the precincts of Frye, Daubert and Khumo that forensic evidence is weighed and judged.”2 Indeed, a key element in the evaluative process of forensic evidence is the literature review.

Scientific articles that form the basis for expertise and expert opinions are mainly published through a peer review process that should assure reliability. However, research misconduct does occur, and when it is uncovered, editors and publishers of scholarly journals have an uphill battle in redressing its impact.

A recent article in the Wall Street Journal highlights the current state of the problem:

Since 2001, while the number of papers published in research journals has risen 44%, the number retracted has leapt more than 15-fold, data compiled for The Wall Street Journal by Thomson Reuters reveal. Just 22 retraction notices appeared in 2001, but 139 in 2006 and 339 last year. Through seven months of this year, there have been 210, according to Thomson Reuters Web of Science, an index of 11,600 peer-reviewed journals world-wide.3

According to David E. Wright, a professor at Michigan State University, there are four reasons that scientists might take short cuts:

[1] some sort of mental disorder; [2] foreign nationals who learned somewhat different scientific standards; [3] inadequate mentoring; and, most commonly, [4] tremendous and increasing professional pressure to publish studies.4

While the scientific community addresses the causes of research misconduct, the publishing world is grappling with its outcomes.

Tainted findings can pollute the information stream and mislead researchers and forensic experts. Thus, the problems created by research misconduct raise an important question for the litigator, what measures can be taken to assure the validity of published scientific research?

After dealing with the problem firsthand, the editors at the Annals of Internal Medicine, published by the American College of Physicians, explored the issues in a notable case.5

In 2003, the University of Vermont notified three professional journals that one of their former faculty members, Eric Poehlman, PhD, had submitted articles for publication that were based on false and fabricated research. Later, the Office of Research Integrity completed its own investigation finding that 10 articles by Poehlman in as many journals, as well as NIH grant applications, relied on fabricated research.

Eventually, he was prosecuted for supplying false data in a grant application and pled guilty. Now the journals were left with the fallout. The Annals, one of the three journals notified by the University of Vermont, published a retraction; the others did not. Most disconcerting of all, “authors continued to cite Poehlman’s Annals article after the 2003 notice of retraction.”

Warnings and withdrawals can be triggered by a spectrum of reasons stemming from honest mistakes to full blown fraud.6 In the scientific publishing lexicon, three levels of caution, which resemble Shepard’s signals, are the most salient:

Retraction: What a journal issues when an investigation has shown that an article contains faked data or has been plagiarized. It tells the reader to ignore that article. Expression of Concern: What a journal issues when the editor is concerned that an article contains faked data or has been plagiarized but an investigation has either not begun or has begun but has not reached a conclusion about that article. Correction: What a journal issues to correct a mistake by substituting correct information or by asking the reader to disregard specified parts of an article (for example, a reference to a retracted article).

In describing their experiences with the Poehlman article, the editors at the Annals of Internal Medicine considered the approaches that can be taken by the scientific community and the Office of Research Integrity to forestall the impact of tainted research.

According to them, the current methods for making known fraudulent research are “faulty.” Their chief recommendation:

Journals should require authors to attest that they have checked their manuscript’s reference list against the National Library of Medicine [NLM] master list of retracted articles.

In light of the multiple modes for accessing an article, e.g., publishers’ websites, periodical databases, free sites, pay walls, etc., a comprehensive method of linking retractions to all access points must be found. Otherwise readers viewing a particular version of an article might not know that it has been called into question or withdrawn.

In addition to publicizing a notice of retraction, the article’s citation in other works also has to be addressed. Indeed, stemming the tide of misleading research is an enormous undertaking as noted by the Annals editors:

One hundred eighty-six of Poehlman’s articles are in the Science Citation Index database…; collectively, they have 3007 citations. Someone could use the Science Citation Index database to identify each author who cited fraudulent articles by Poehlman. Contacting the authors of 3007 articles is a formidable undertaking. Who will take responsibility?

Legal Informatics

Shepardizing is a lawyer’s art filled with subtlety and nuance and whose ultimate aim is information quality assurance. Still, even the best efforts of highly trained legal editors and the safeguards of time tested research practices are not beyond reproach.

In the 1930s an Ohio judge had been disbarred after being found

guilty of having written, and caused to be published in a legal journal, an opinion in a fictitious cause purporting to have been heard and decided by him when he well knew that its published report would be relied upon by lawyers and judges as the decision of a court in a litigated controversy.7

In that same era, a case reached the U.S. Supreme Court based on a rule that did not exist; and in our time, there was an instance of incomplete research in a High Court death penalty case.8

Notwithstanding the limitations of human researchers, legal informatics has made great strides and those experiences can provide guidance for quality assurance of published scientific research.

Forensic Bibliometrics

It was Frank Shepard’s methodology that paved the way for Eugene Garfield’s creation of the Science Citation Index (SCI), and ultimately, the page ranking protocols used by Internet search engines.9 Most legal opinions can be Shepardized, and along with a full court press of bibliometric analysis in multiple sources, this tool can provide a high level of quality assurance. The same is not easily accomplished in the scientific disciplines.10

Aside from the tools already noted, quality control of scholarly literature would benefit from something resembling a Shepard’s for scientific research. It would be a universal mechanism that flags retracted articles in peer review journals and treatises, in all formats and at all access points, clearly indicating which ones should not be cited or relied upon.

One tactic might be found in emerging information management tools, e.g., “a single search box that can search a library’s online and physical content including articles, books, journals, newspaper articles, e-books, specialist collections and more.”11 Thus warnings about questioned scholarship could be found and confirmed through multiple avenues and federated searches conducted at one time with linkage to granulated warnings.

In a comprehensive report, the Science and Technology Committee of the British Parliament made note of a Shepard’s type tool, described by Robert Campbell of Wiley-Blackwell publishing, which ought to make headway on data quality assurance:

The [publishing] industry is developing […] a new project called CrossMark. Every paper that has gone through the peer review process has the ongoing stewardship of the publisher picking up on retractions or corrections. By clicking on to the CrossMark logo, you can go to the metadata and find out if there have been any updates or even retractions. That is a technical solution which is being launched this year.12

Courting Disaster

While citation analysis tools undergo development, the consequences of uncovering questionable scholarly literature in the courtroom must be confronted now.

For example, there are over 5,000 cases pending in the Court of Federal Claims under the National Vaccine Injury Compensation Program (see Omnibus Autism Proceeding), the gravamen of which is the causal connection between vaccination and autism.

One federal judge, in an unrelated vaccine case, made this obiter dictum:

As demonstrated in the recent autism cases, expert medical opinions based on discredited medical theories or research may be found not to be reliable. However, in the autism cases, the key article upon which experts relied was discredited by the scientific community and co-authors later retracted its conclusion.13

In setting the record straight, the retraction of a journal article’s author is tantamount to the recantation of a witness or complainant.

Information quality assurance is an issue of overarching importance for litigants. Undoubtedly, attorneys preparing to cross-examine a witness with a 20-page curriculum vitae would benefit from the services of a bibliographic investigator (or librarian) as well as a forensics expert.

Whether such services will become the hallmark of effective assistance of counsel, such as the obligation to consult with a forensic expert in general, remains to be determined.14 And what are the testifying expert’s responsibilities in conducting a literature search? Shouldn’t they be ethically responsible for unearthing discredited research in their field?

Scientific studies are referenced in news articles, bench books, and available through a thousand portals on the Internet. Yet, it is not known how many warnings are likely to be posted in any given source when an article has been retracted or a study discredited. Thus a judge’s or juror’s knowledge might be founded on invalid assumptions from unvetted sources. Moreover, there is no citation analysis tool for the grey literature of the scientific world.

In light of the CSI effect and similar credibility inflation phenomena, the results of bibliometric analysis can offer a cautionary lesson for judges and jurors in their evaluation of forensic science and expert opinions.

Indeed, information experts have acknowledged the difficulty of establishing quality assurance: Although retraction of papers appears to be increasing, it is not a common occurrence, so even an experienced librarian or information specialist may not be familiar with how bibliographic databases deal with annotating records to indicate retraction.15

For these reasons, an expert in the citation analysis of scientific literature can play a crucial role in litigation.

Conclusion

Perhaps Jacksonian era Americans were credulous, but modern day people are willing to believe in spaghetti harvests, Piltdown Man, and an endless parade of myths, hoaxes, and amazing stories. Even articles appearing in satirical newspapers like The Onion are sometimes cited as fact. Without some kind of comprehensive quality assurance, all scientific publications might have to be labeled “Believe It or Not!”

Defendants facing trial or challenging their convictions should not have to wait behind bars hoping that bad research will be ferreted out. While it is remotely possible that the scientist who manufactures fraudulent data will be confronted in court, it is their legacy in the published world that is radioactive. Some experts might not be aware that they have relied on a retracted article in forming their opinions, but judges and lawyers should. In a landscape of scientific data landmines, the task of shepardizing forensic literature is no snipe hunt.

Ken Strutin is director of legal information services at the New York State Defenders Association.

Endnotes:

1. See Steven W. Ruskin, “A Newly-Discovered Letter of J.F.W. Herschel Concerning the ‘Great Moon Hoax,’” 33 J. Hist. Astronomy 71 (2002).

2. Ken Strutin, Chapter 10, “Research Resources in Technology, Science and the Law (Keeping Current)” in The Future of Evidence: How Science and Technology Will Change the Practice of Law (ABA 2011).

3. Gautam Naik, “Mistakes in Scientific Studies Surge,” Wall St. J., Aug. 10, 2011; see also National Library of Medicine Programs and Services 11 (NIH 2009) (“Previously indexed citations were updated to reflect 249 retractions, and 33,526 comments found in subsequently published notices or articles”).

4. “Charges of Fake Research Hit New High,” MSNBC, July 10, 2005; see David E. Wright et al., “Mentoring and Research Misconduct: An Analysis of Research Mentoring in Closed ORI Cases,” 14 Sci. Eng. Ethics 323, 335 (2008).

5. Harold C. Sox and Drummond Rennie, “Research Misconduct, Retraction, and Cleansing the Medical Literature: Lessons From the Poehlman Case,” 144 Annals Internal Med. 609 (2006). See also Misconduct Finding: Eric T. Poehlman (ORI Before 2007).

6. See generally “Errata, Retractions, Partial Retractions, Corrected and Republished Articles, Duplicate Publications, Comments (including Author Replies), Updates, Patient Summaries, and Republished (Reprinted) Articles Policy for MEDLINE” (NLM Fact Sheet).

7. In re Copland, 33 N.E.2d 857, 858 (Ohio Ct. App., Cuyahoga County 1940).

8. See Erwin N. Griswold, “Government in Ignorance of Law a Plea for Better Publication of Executive Legislation,” 48 Harv. L. Rev. 198, 204 (1934); Linda Greenhouse, “In Court Ruling on Executions, a Factual Flaw,” N.Y. Times, July 2, 2008.

9. See Members of the Clever Project, “Hypersearching the Web,” Sci. Am., June 1999, at 54.

10. See, e.g., Handling Misconduct—Case Summaries (ORI); Reference Point (NLM); Retraction Watch (eds. Adam Marcus & Ivan Oransky); The Scholarly Kitchen (Society for Scholarly Publishing).

11. Chris Keene, “Discovery Services: Next Generation of Searching Scholarly Information,” Serials, 24(2), July 2011, at 193.

12. Science and Technology Committee—Eighth Report: Peer Review in Scientific Publications par. 251 (House of Commons Rep. Jul. 18, 2011); see Carol Anne Meyer, Distinguishing Published Scholarly Content with Crossmark, 24 Learned Publishing 87 (April 2011).

13 Graves v. Sec’y of the Dept. of HHS, 2011 U.S. Claims LEXIS 1466 at 35 (Fed. Cl. July 5, 2011). See generally Steven J. Meyers, “Denying the Obvious,” 20 Fed. Cir. B.J. 633, 639-40 n.47 (2011).

14. See Harrington v. Richter, 131 S. Ct. 770 (2011). Accord People v. Hull, 71 A.D.3d 1336 (3rd Dept. 2010); People v. Richard R., 31 Misc. 3d 1212A (Westchester County Ct. 2011).

15. Kath Wright & Catriona McDaid, “Reporting of Article Retractions in Bibliographic Databases and Online Journals,” 99 J. Med. Libr. Assoc. 164 (2011).