Forensics, Statistics, and Law: Ten Years After “A Path Forward”

 

 

Volume 69 September 2019
Forensics, Statistics, and Law:
Ten Years After “A Path Forward”

Brandon Garrett L. Neil Williams, Jr. Professor of Law, Duke University School of Law, and Director, Duke Center for Science and Justice.

This year marks the tenth Anniversary of the 2009 National Research Council report, Strengthening Forensic Science in the United States: A Path Forward.[1][1].Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat’l Research Council of the Nat’l Acads., Strengthening Forensic Science in the United States: A Path Forward (2009), https://www.nap.edu/read/12589/chapter/2 [https://perma.cc/JG6Y-X8SN] [hereinafter Strengthening Forensic Science]. Judge Harry Edwards, who cochaired the Committee that authored the Report, noted that before starting the project, he had “no preconceived views about the forensic science community.” [2][2]. Strengthening Forensic Science in the United States: A Path Forward: Hearing Before the S, Comm. on the Judiciary, 110th Cong. 2 (2009) (statement of the Hon. Harry T. Edwards, Senior Circuit Judge, U.S. Court of Appeals for the D.C. Circuit). He “simply assumed, as I suspect many of my judicial colleagues do, that forensic science disciplines typically are well grounded in scientific methodology.”[3][3]. Id. The 300 page report had followed an elaborate and lengthy peer review process before its release. The Committee concluded that much of forensic evidence used in criminal trials was “without any meaningful scientific validation.”[4][4]. Strengthening Forensic Science, supra note 1, at 108. They described “major problems” in forensics, including where faulty forensic science lead to wrongful convictions.[5][5]. Id. at 5.

It has been an eventful ten years. That report prompted a renewed interest in supporting research to improve the statistical foundations of forensic evidence, new protocols and standards in crime laboratories, and a changing approach towards litigating expert evidence in criminal cases. This online symposium includes contributions by lawyers and scientists and policymakers working at the intersection of law, statistics, and forensic science to respond to these challenges. Each was part of a group who gathered for conversations at a March 2019 conference at Duke Law School, which brought together a group of twenty to twenty-five leading crime lab directors, forensic scientists, statisticians, legal scholars, judges, and practicing lawyers to discuss, in a roundtable setting, how to plan the next ten years of the path forward for forensics. The conference was made possible by the Innocence Project and the Center for Statistics and Applications in Forensic Science (“CSAFE”) collaboration, extending across five universities, including Duke, which has been working for several years, with generous support from the National Institute of Standards and Technology (“NIST”) to research these questions.

The three main topics at the conference were improvements in the crime lab, including a focus on the contents of law reports, the future of litigation surrounding forensic evidence, and education efforts both among forensic professionals and lawyers. We posed questions, such as:

• What are models for blind proficiency testing and verification in crime laboratories?
• What are models for incorporate statistics into crime lab procedures?
• How can forensic science be better regulated? What is the role of OSAC standards? What is the role of a forensic science commission?
• How can forensic reports be more clearly documented? What are best practices for the content of courtroom testimony by experts?
• What are models for best educating people, including forensic scientists, lawyers, judges, and jurors, about forensic methods, including the relevant statistics?

Rather than have panels, we had discussion sessions, including a design thinking session focused on how one might redesign a standard lab report. At lunch, we heard from Keith Harward, who was exonerated by post-conviction DNA testing, having been convicted and nearly sentenced to death based on faulty bite mark evidence, in a conversation with Peter Neufeld. Six different dentists said his teeth were a match when he convicted in 1982 and then at a second trial in 1986.

Our first session described what has changed and what has not since the 2009 Report was published. Sue Ballou, the program manager for forensic science at the Office of Law Enforcement Standards in the Office of Special Programs at NIST since 2000 and also the immediate past president of the American Academy of Forensic Science, spoke first. In this online symposium, Ballou describes the changing focus on forensics as NIST and at the federal level generally. [6][6]. Sue Ballou, Forensics at the Federal Level, 69 DLJ Online 1 (2019). Even before the NAS Report, NIST was aware that there was a deep need for consistent standards to be set for forensic disciplines. However, the focus of NIST changed more noticeably after the Report came out. New focus groups were created to work on problems. The Report made it easier to obtain funds for these efforts, and new and more difficult problems, such as human factors, were addressed. The FBI began to change its lab procedures. Other agencies started to follow suit and the Department of Justice created a National Commission on Forensic Science in 2012. NIST created CSAFE, with a focus on statistics and pattern evidence. Ballou concludes that, “In short, ten years later, we are strengthening forensic science in the United States.”

Upon its release, as Steven Kendall, Program Officer, Committee on Science, Technology, and Law, National Academies of Sciences, Engineering, and Medicine, describes in his piece, the NAS Report attracted sustained public and media attention. [7][7]. Steven Kendall, The Public Perception of the Path Forward Report, 69 DLJ Online 12 (2019). Commentators called the report a “blockbuster” and a “watershed.” The Report was featured on the New York Times Science Times cover, in National Geographic, in print media, television programs, and on Last Week Tonight (The John Oliver clip has 6.6 million youtube views). President Obama took notice of the Report in a Harvard Law Review article. Congress held hearings and legislation was introduced to regulate forensics along the lines that the NAS Report proposed—none of that legislation passed. The Supreme Court promptly cited the Report, with Justice Antonin Scalia noting that forensics have “serious deficiencies” and are not “immune from the risk of manipulation.” [8][8]. Melendez-Diaz v. Massachusetts, 557 U.S. 305, 318 (2009).

Peter Neufeld, who co-founded the Innocence Project, describes in his piece, The Trajectory of Forensics, the history and the culture surrounding forensic science before the report came out. [9][9]. Peter Neufeld, The Trajectory of Forensics, 69 DLJ Online 17 (2019). In the late 1970s and early 1980s, cases were being litigated at the intersection of law and science, but lawyers received very little information about forensic analysis. Forensic lab reports were not detailed, Neufeld describes; they were typically either “thumbs up” or “thumbs down.” No courts were seriously considering the validity and reliability of old methods. There was no regulation or legislation. And yet other labs were regulated. Unlike crime labs, clinical labs had regulations, legislation, quality control, and independent outside bodies. These differences were not scientific, but were rather caused by differing constituencies. The DNA revolution and wrongful conviction cases changed matters deeply. It was revealed that more than half of wrongful convictions brought to light by DNA testing were based on bad scientific evidence. There was a new constituency; more people started caring about forensic science. The NAS Report had a significant impact on innocence work. The Report created an opening to talk about these issues. Claims about changes in science affected the doctrine of finality. Five states enacted “change of science” laws (legislatively and judicially). Nevertheless, today we are still “at the beginning of that trajectory,” and there is much work to be done.

Karen Kafadar, who chairs statistics at the University of Virginia and is the President of the American Statistics Association, served on the Committee that drafted the Report. Kafadar described the work of the Committee and its impact, focusing in particular on the role of statistics. [10][10]. Karen Kafadar, Statistics and the Impact of the 2009 NAS Report, 69 DLJ Online 6 (2019). Kafadar points us to the most cited sentence of the report: “With the exception of nuclear DNA analysis, however, no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.” That sentence does not provide any answers, but it does pose a challenge: how can all of the non-nuclear DNA forensics be made reliable, or even tested to find out how reliable they are? Kafadar describes how there has been some progress. DOJ and NIST convened the National Commission on Forensic Science (“NCFS”), which was active from 2014–2017. Yet there is still no federal agency that meets all of NAS’s recommended criteria for regulating forensics. There have been other disappointments following the 2009 Report—the NCFS was disbanded in April of 2017. The focus of the working groups convened by NIST, the Organization of Scientific Area Committees, has been on approving existing standards rather than scrutinizing scientific validity. The White House PCAST Report was criticized more than it was appreciated. From the perspective of bringing statistics into forensics, Kafadar concludes that progress has been made but much remains to be done.

What is the path forward then years later? The challenge posted by the central sentence in the 2009 “A Path Forward” Report is still with us. We still need further research to support the use of much of forensics, in order to obtain the scientific validation that the Committee observed was lacking across so many disciplines. We still need more collaboration between scientists and practitioners across disciplines. We still need more research on error rates. We have much to do to improve the use of forensic evidence in public defender and prosecutors’ offices as well as in the courts. While the 2009 Report placed us on the path forward, we have a long distance to travel. At least we know where the road is, and we have new collaborators and resources to sustain us along this important journey toward improved science and justice.


L. Neil Williams, Jr. Professor of Law, Duke University School of Law, and Director, Duke Center for Science and Justice.

[1] Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat’l Research Council of the Nat’l Acads., Strengthening Forensic Science in the United States: A Path Forward (2009), https://www.nap.edu/read/12589/chapter/2 [https://perma.cc/JG6Y-X8SN] [hereinafter Strengthening Forensic Science].

[2] Strengthening Forensic Science in the United States: A Path Forward: Hearing Before the S., Comm. on the Judiciary, 110th Cong. 2 (2009) (statement of the Hon. Harry T. Edwards, Senior Circuit Judge, U.S. Court of Appeals for the D.C. Circuit).

[3] Id.

[4] Strengthening Forensic Science, supra note 1, at 108.

[5] Id. at 5.

[6] Sue Ballou, Forensics at the Federal Level, 69 DLJ Online 1 (2019).

[7] Steven Kendall, The Public Perception of the Path Forward Report, 69 DLJ Online 12 (2019).

[8] Melendez-Diaz v. Massachusetts, 557 U.S. 305, 318 (2009).

[9] Peter Neufeld, The Trajectory of Forensics, 69 DLJ Online 17 (2019).

[10] Karen Kafadar, Statistics and the Impact of the 2009 NAS Report, 69 DLJ Online 6 (2019).

Forensics, Statistics, and Law: Ten Years After “A Path Forward”

 

 

Volume 69 September 2019
Forensics, Statistics, and Law:
Ten Years After “A Path Forward”

Brandon Garrett L. Neil Williams, Jr. Professor of Law, Duke University School of Law, and Director, Duke Center for Science and Justice.

This year marks the tenth Anniversary of the 2009 National Research Council report, Strengthening Forensic Science in the United States: A Path Forward.[1][1].Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat’l Research Council of the Nat’l Acads., Strengthening Forensic Science in the United States: A Path Forward (2009), https://www.nap.edu/read/12589/chapter/2 [https://perma.cc/JG6Y-X8SN] [hereinafter Strengthening Forensic Science]. Judge Harry Edwards, who cochaired the Committee that authored the Report, noted that before starting the project, he had “no preconceived views about the forensic science community.” [2][2]. Strengthening Forensic Science in the United States: A Path Forward: Hearing Before the S, Comm. on the Judiciary, 110th Cong. 2 (2009) (statement of the Hon. Harry T. Edwards, Senior Circuit Judge, U.S. Court of Appeals for the D.C. Circuit). He “simply assumed, as I suspect many of my judicial colleagues do, that forensic science disciplines typically are well grounded in scientific methodology.”[3][3]. Id. The 300 page report had followed an elaborate and lengthy peer review process before its release. The Committee concluded that much of forensic evidence used in criminal trials was “without any meaningful scientific validation.”[4][4]. Strengthening Forensic Science, supra note 1, at 108. They described “major problems” in forensics, including where faulty forensic science lead to wrongful convictions.[5][5]. Id. at 5.

It has been an eventful ten years. That report prompted a renewed interest in supporting research to improve the statistical foundations of forensic evidence, new protocols and standards in crime laboratories, and a changing approach towards litigating expert evidence in criminal cases. This online symposium includes contributions by lawyers and scientists and policymakers working at the intersection of law, statistics, and forensic science to respond to these challenges. Each was part of a group who gathered for conversations at a March 2019 conference at Duke Law School, which brought together a group of twenty to twenty-five leading crime lab directors, forensic scientists, statisticians, legal scholars, judges, and practicing lawyers to discuss, in a roundtable setting, how to plan the next ten years of the path forward for forensics. The conference was made possible by the Innocence Project and the Center for Statistics and Applications in Forensic Science (“CSAFE”) collaboration, extending across five universities, including Duke, which has been working for several years, with generous support from the National Institute of Standards and Technology (“NIST”) to research these questions.

The three main topics at the conference were improvements in the crime lab, including a focus on the contents of law reports, the future of litigation surrounding forensic evidence, and education efforts both among forensic professionals and lawyers. We posed questions, such as:

• What are models for blind proficiency testing and verification in crime laboratories?
• What are models for incorporate statistics into crime lab procedures?
• How can forensic science be better regulated? What is the role of OSAC standards? What is the role of a forensic science commission?
• How can forensic reports be more clearly documented? What are best practices for the content of courtroom testimony by experts?
• What are models for best educating people, including forensic scientists, lawyers, judges, and jurors, about forensic methods, including the relevant statistics?

Rather than have panels, we had discussion sessions, including a design thinking session focused on how one might redesign a standard lab report. At lunch, we heard from Keith Harward, who was exonerated by post-conviction DNA testing, having been convicted and nearly sentenced to death based on faulty bite mark evidence, in a conversation with Peter Neufeld. Six different dentists said his teeth were a match when he convicted in 1982 and then at a second trial in 1986.

Our first session described what has changed and what has not since the 2009 Report was published. Sue Ballou, the program manager for forensic science at the Office of Law Enforcement Standards in the Office of Special Programs at NIST since 2000 and also the immediate past president of the American Academy of Forensic Science, spoke first. In this online symposium, Ballou describes the changing focus on forensics as NIST and at the federal level generally. [6][6]. Sue Ballou, Forensics at the Federal Level, 69 DLJ Online 1 (2019). Even before the NAS Report, NIST was aware that there was a deep need for consistent standards to be set for forensic disciplines. However, the focus of NIST changed more noticeably after the Report came out. New focus groups were created to work on problems. The Report made it easier to obtain funds for these efforts, and new and more difficult problems, such as human factors, were addressed. The FBI began to change its lab procedures. Other agencies started to follow suit and the Department of Justice created a National Commission on Forensic Science in 2012. NIST created CSAFE, with a focus on statistics and pattern evidence. Ballou concludes that, “In short, ten years later, we are strengthening forensic science in the United States.”

Upon its release, as Steven Kendall, Program Officer, Committee on Science, Technology, and Law, National Academies of Sciences, Engineering, and Medicine, describes in his piece, the NAS Report attracted sustained public and media attention. [7][7]. Steven Kendall, The Public Perception of the Path Forward Report, 69 DLJ Online 12 (2019). Commentators called the report a “blockbuster” and a “watershed.” The Report was featured on the New York Times Science Times cover, in National Geographic, in print media, television programs, and on Last Week Tonight (The John Oliver clip has 6.6 million youtube views). President Obama took notice of the Report in a Harvard Law Review article. Congress held hearings and legislation was introduced to regulate forensics along the lines that the NAS Report proposed—none of that legislation passed. The Supreme Court promptly cited the Report, with Justice Antonin Scalia noting that forensics have “serious deficiencies” and are not “immune from the risk of manipulation.” [8][8]. Melendez-Diaz v. Massachusetts, 557 U.S. 305, 318 (2009).

Peter Neufeld, who co-founded the Innocence Project, describes in his piece, The Trajectory of Forensics, the history and the culture surrounding forensic science before the report came out. [9][9]. Peter Neufeld, The Trajectory of Forensics, 69 DLJ Online 17 (2019). In the late 1970s and early 1980s, cases were being litigated at the intersection of law and science, but lawyers received very little information about forensic analysis. Forensic lab reports were not detailed, Neufeld describes; they were typically either “thumbs up” or “thumbs down.” No courts were seriously considering the validity and reliability of old methods. There was no regulation or legislation. And yet other labs were regulated. Unlike crime labs, clinical labs had regulations, legislation, quality control, and independent outside bodies. These differences were not scientific, but were rather caused by differing constituencies. The DNA revolution and wrongful conviction cases changed matters deeply. It was revealed that more than half of wrongful convictions brought to light by DNA testing were based on bad scientific evidence. There was a new constituency; more people started caring about forensic science. The NAS Report had a significant impact on innocence work. The Report created an opening to talk about these issues. Claims about changes in science affected the doctrine of finality. Five states enacted “change of science” laws (legislatively and judicially). Nevertheless, today we are still “at the beginning of that trajectory,” and there is much work to be done.

Karen Kafadar, who chairs statistics at the University of Virginia and is the President of the American Statistics Association, served on the Committee that drafted the Report. Kafadar described the work of the Committee and its impact, focusing in particular on the role of statistics. [10][10]. Karen Kafadar, Statistics and the Impact of the 2009 NAS Report, 69 DLJ Online 6 (2019). Kafadar points us to the most cited sentence of the report: “With the exception of nuclear DNA analysis, however, no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.” That sentence does not provide any answers, but it does pose a challenge: how can all of the non-nuclear DNA forensics be made reliable, or even tested to find out how reliable they are? Kafadar describes how there has been some progress. DOJ and NIST convened the National Commission on Forensic Science (“NCFS”), which was active from 2014–2017. Yet there is still no federal agency that meets all of NAS’s recommended criteria for regulating forensics. There have been other disappointments following the 2009 Report—the NCFS was disbanded in April of 2017. The focus of the working groups convened by NIST, the Organization of Scientific Area Committees, has been on approving existing standards rather than scrutinizing scientific validity. The White House PCAST Report was criticized more than it was appreciated. From the perspective of bringing statistics into forensics, Kafadar concludes that progress has been made but much remains to be done.

What is the path forward then years later? The challenge posted by the central sentence in the 2009 “A Path Forward” Report is still with us. We still need further research to support the use of much of forensics, in order to obtain the scientific validation that the Committee observed was lacking across so many disciplines. We still need more collaboration between scientists and practitioners across disciplines. We still need more research on error rates. We have much to do to improve the use of forensic evidence in public defender and prosecutors’ offices as well as in the courts. While the 2009 Report placed us on the path forward, we have a long distance to travel. At least we know where the road is, and we have new collaborators and resources to sustain us along this important journey toward improved science and justice.


L. Neil Williams, Jr. Professor of Law, Duke University School of Law, and Director, Duke Center for Science and Justice.

[1] Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat’l Research Council of the Nat’l Acads., Strengthening Forensic Science in the United States: A Path Forward (2009), https://www.nap.edu/read/12589/chapter/2 [https://perma.cc/JG6Y-X8SN] [hereinafter Strengthening Forensic Science].

[2] Strengthening Forensic Science in the United States: A Path Forward: Hearing Before the S., Comm. on the Judiciary, 110th Cong. 2 (2009) (statement of the Hon. Harry T. Edwards, Senior Circuit Judge, U.S. Court of Appeals for the D.C. Circuit).

[3] Id.

[4] Strengthening Forensic Science, supra note 1, at 108.

[5] Id. at 5.

[6] Sue Ballou, Forensics at the Federal Level, 69 DLJ Online 1 (2019).

[7] Steven Kendall, The Public Perception of the Path Forward Report, 69 DLJ Online 12 (2019).

[8] Melendez-Diaz v. Massachusetts, 557 U.S. 305, 318 (2009).

[9] Peter Neufeld, The Trajectory of Forensics, 69 DLJ Online 17 (2019).

[10] Karen Kafadar, Statistics and the Impact of the 2009 NAS Report, 69 DLJ Online 6 (2019).