Image: theseanster93
A pair of e-discovery research projects, intended to test new methods and processes, may also lead to customers receiving extra velocity on sales pitches this spring.
That's because in addition to the final results of the U.S. government's annual Text Retrieval Conference (TREC) Legal Track, due in February, there will also be a new round of the nonprofit Electronic Discovery Institute's own performance study, last conducted in 2006. In both cases, the contests pit teams of document reviewers, using their choice of software, all working on the same data set to learn which methods are the fastest and most thorough.
Such benchmarks may be the e-discovery equivalent of sabermetrics in professional baseball -- the concept that computers and measurements are more useful than a veteran leader's intuition.
Similarly, organizers of both projects know that real-world customers value cost savings, ease-of-use, and efficient processes, not laboratory results. As such, teams from both sides are discouraged from openly discussing test results and from using results for marketing and sales purposes. Some do anyway, especially after results begin to circulate in the legal technology community. For example, among predictive coding specialists, Equivio this month published advertisements touting its 2010 TREC performance, while its rival Recommind is already asserting high marks based on preliminary results from the 2011 edition. (Law Technology News viewed TREC documents from both years and found mixed evidence of both vendors' claims.)
TREC Legal Track formed in 2006, with most of its teams coming from universities. It remained a majority academic exercise until 2009 and 2010, when more companies entered than universities. The 2011 version reversed that trend, drawing 10 teams, of which only four were vendors -- search startup Helioid, content management expert OpenText, Recommind, and e-discovery provider TCDI.
TREC officials declined to comment ahead of the official results. But some participants, such as Tom McKenzie, vice president of marketing at TCDI, defended TREC's role. "It's important for people to understand that this is, for most of us, purely a research opportuntiy. In fact we do not use the same technology in TREC as we use for our clients. Stuff that we do in TREC is really our R&D group," he said, in Greensboro, N.C.
However, in 2011, many e-discovery companies appear to have saved their resources for the more pragmatic E-Discovery Institute project. The latter study is limited to businesses and is expected by organizers to field up to 20 teams. "I'm more interested in the much bigger policy decisions that the judiciary is making," EDI leader and LTN advisory board member Patrick Oot said. "When litigants choose one form of technology over the other, what are the ramifications of it? It's very difficult to apply real science to a loosey-goosey standard like reasonableness."
In addition, the EDI project is sponsored by Oracle, which may give contracts to top-performing teams, event staff said. The test will begin early in 2012 because of personnel changes after its announcement last February, Oot said. Beyond measuring document recall and precision, teams will also try to determine which documents are privileged, a step TREC does not take.
Regardless, TCDI's McKenzie said, no customer has ever asked him to show scores from the discovery challenges. Instead, he said, "Hopefuly at some point in time, [the experiments] boil down into the technology that we actually provide to our clients. But research is research."
Evan Koblentz is a reporter for Law Technology News. Send e-mail.
Subscribe to Law Technology News
