Secrets of Search – Part II

http://ow.ly/83OzV

This is the 2nd part of a now 3 part series (the author states that this originally was intended as a two part series but due to the amount of content it required expansion) on the secrets of search during the eDiscovery phase of litigation.

The author recaps his main two points of the first article, "I can quickly summarize the first two secrets with popular slang: keyword search sucks, and so does manual review (although not quite as bad), and because most manual review sucks, most so-calledobjective measurements of precision and recall are unreliable."

The third secret revealed is as follows, "The Third Search Secret (Known Only to a Very Few): e-Discovery Watson May Still Not Be Able to Beat Our Champions"

The author seeks a new "Gold" standard for attorney review.

The components of this new standard include:" The exact contours of the new gold are now under development in dozens of law firms, private companies, and universities around the world. Although we do not know all of the details, we know it will involve:

1. high quality technology assisted review, with smart software, very smart predictive coding type, and multiple expert review of key seed-set training documents;
2. constant expert involvement, both subject matter experts (attorneys) and AI experts (technologists);
3. direct supervision and feedback by the responsible lawyer(s) (merits counsel) signing under 26(g);
4. extensive quality control methods, including training and more training, sampling, positive feedback loops, clever batching, and sometimes, quick reassignment or firing of reviewers who are not working well on the project;
5. experienced, well motivated human reviewers who know and like the AI agents (software tools) they work with;
6. new tools and psychological techniques (e.g. game theory) to facilitate prolonged concentration (beyond just coffee, $, and fear) to keep attorney reviewers engaged and motivated to perform the complex legal judgment tasks required to correctly review thousands of usually boring documents for days on end (voyeurism will only take you so far);
7. top-notch project managers who know and understand their team, both human and computer, and the new tools and techniques under development to help coach the team;
8. strategic cooperation between opposing counsel with adequate disclosures to build trust and mutually acceptable relevancy standards; and,
9. final, last-chance review of a production set before going out the door by spot checking, judgmental sampling (i.e. search for those attorney domains one more time), and random sampling."

P.S. This is another very comprehensive and enlightening article by Mr. Losey.  It is certainly worth reading, and re-reading.  The darkest secret of searching is said to be coming in Part 3 of this excellent series.