Collaborative Construction Of A New Legal Field

Volume 68 September 2018
COLLABORATIVE CONSTRUCTION OF A NEW LEGAL FIELD

Ronald F. Wright & Mark A. Hall †. Needham Y. Gulley Professor of Criminal Law, and Fred & Elizabeth Turnage Professor of Law, respectively, both at Wake Forest University.

INTRODUCTION

Eric Ruben and Joseph Blocher have announced the birth of an “entire field of constitutional doctrine,” which they rightly note is “a rare challenge and opportunity for judges, lawyers, and scholars.” [1][1]. Eric Ruben & Joseph Blocher, From Theory to Doctrine: An Empirical Analysis of the Right to Keep and Bear Arms After Heller, 67 Duke L.J. 1433, 1435 (2018). The birth announcement of a new legal field is a familiar move, one that we’ve made ourselves (with limited success). [2][2]. See, e.g., Mark A. Hall, The History and Future of Health Care Law: An Essentialist View, 41 Wake Forest L. Rev. 347, 357–63 (2006); Ronald F. Wright, Sentencers, Bureaucrats, and the Administrative Law Perspective on the Federal Sentencing Commission, 79 Calif. L. Rev. 1, 4–5 (1991). One of us also declared the arrival of milk law, a new field that inexplicably failed to attract the attention of other scholars. See Ronald F. Wright & Paul Huck, Counting Cases About Milk, Our “Most Nearly Perfect Food,” 1860-1940, 36 Law & Soc’y Rev. 51, 51–57 (2002). What is more intriguing for us than learning of this new arrival is the way that Ruben and Blocher chose to midwife this new legal field: using the social science methodology of textual content analysis.

The time-honored method of discovering a new doctrinal field has been the ineffable, hard to replicate work of an individual scholar reading selected cases. For example, when William Prosser declared a new doctrine of tort-enforced privacy, he described a collection of published judicial opinions and highlighted their common elements and future implications. [3][3]. William L. Prosser, Privacy, 48 Calif. L. Rev. 383, 388–89 (1960) (declaring four distinct privacy torts). At the other end of the doctrinal life cycle, see generally Grant Gilmore, The Death of Contract (1974). This declaration singled out cases for scrutiny but did not identify the areas where Prosser searched for evidence or discuss the cases that did not fit his chosen pattern. Critics of the author’s descriptive framework must start from scratch—constructing their own caselaw searches and analyzing their competing examples—to explain where the author might have gone wrong. [4][4]. See generally Neil M. Richards & Daniel J. Solove, Prosser’s Privacy Law: A Mixed Legacy, 98 Calif. L. Rev. 1887 (2010) (describing the impact and critiques of Prosser’s doctrinal synthesis).

Unlike this classic “Lone Ranger” style of legal scholarship, Ruben and Blocher (who we’ll sometimes call R&B) have published more of an invitation than an announcement. They want to involve others in their construction of a new body of legal doctrine, using the established techniques of content analysis to study Second Amendment cases in ways that other researchers can replicate. They specify the questions they ask, their method of collecting cases, and their techniques for classifying those cases. While this methodology has many useful applications across legal scholarship, [5][5]. Mark A. Hall & Ronald F. Wright, Systematic Content Analysis of Judicial Opinions, 96 Calif. L. Rev. 63, 73 (2008). we are especially intrigued by its use in the earliest days of an emerging field, when scholars first start to see judicial decisions forming a connected doctrinal body of law.

In our post-truth era, [6][6]. See Yes, I’d Lie to You, Economist (Sept. 10, 2016), https://www.economist.com/briefing/2016/09/10/yes-id-lie-to-you [https://perma.cc/8BRX-VPCV] (documenting the rise of the “post-truth” era); see also Julie Beck, This Article Won’t Change Your Mind, Atlantic, (Mar. 13, 2017), https://www.theatlantic.com/science/archive/2017/03/this-article-wont-change-your-mind/519093 [https://perma.cc/BEA8-3YMX] (exploring the phenomenon by which the presentation of evidence contrary to one’s beliefs may in fact reinforce them). it is ever more important for legal scholars to develop and refine objective and verifiable ways to analyze legal doctrine. [7][7]. For further information on the post-truth era’s implications for the law, see generally Allison Larsen, Constitutional Law in an Age of Alternative Facts, 93 N.Y.U. L. Rev. 175 (2018). Thus, in this comment, we spotlight the mainstream methodological choices that R&B made at each stage of their content analysis. We also consider how their work resembles the “systematic literature review” method that social scientists use to synthesize large bodies of research in a field. [8][8]. See generally William Baude, Adam S. Chilton & Anup Malani, Making Doctrinal Work More Rigorous: Lessons from Systematic Reviews, 84 U. Chi. L. Rev. 37 (2017) (arguing that legal scholars should conduct systematic reviews). Finally, we consider the scholarly collaboration that this social science methodology makes possible for Second Amendment scholarship going forward. In particular, what sort of replication studies or meta-analysis could other scholars pursue, using R&B’s work as a starting point?

I. OPEN-SOURCE CODING FOR DOCTRINE

Blocher and Ruben’s use of content analysis is an exemplary model for others to follow. Critically, they explained their methods in enough detail to allow other scholars to replicate their study or to make informed choices about building a different study. Their doctrinal coding was open for users—and for later data collectors—to see.

Data selection was their first step. [9][9]. See Ruben & Blocher, supra note 1, at 1455–58. After setting the start and end dates of their search, the authors rejected databases assembled by advocates and chose instead a Westlaw database that included both trial and appellate decisions from the state and federal courts. Given the small number of state trial court opinions in the search results and the overrepresentation of New York in those results, they justifiably excluded state trial court cases. [10][10]. Id. at 1458. While reported opinions do not reliably reflect the activity of all courts, the use of reported opinions is well suited to the project here, which focuses on how precedent is building and forming a new doctrinal area. Happily, these data selection methods produced a reasonable number of cases, allowing the authors to code a “universal sample” of all relevant cases. They avoided hard choices about how to select a representative sample for coding. [11][11]. Id.

R&B’s most difficult judgment call in collecting cases was their exclusion of cases in which the Second Amendment outcome was “incidental” to the issue(s) being decided. [12][12]. Id. at 1461. Another exclusion that requires some legal judgment that might be difficult to replicate involved “nondecisional” cases, such as magistrate recommendations or opinions later vacated by the same judges. See id. at 1460. Beyond offering four convincing examples of this exclusion, [13][13]. Examples include brief doctrinal discussions of the Second Amendment in the course of determining the effectiveness of defense counsel or the qualified immunity of police officers. Id. at 1461. the authors stated this aspect of their selection criterion only in general terms, leaving ample room for later researchers to adopt confirmatory or alternative approaches. In fact, R&B noted that other researchers had previously made different judgments about including or excluding a few of the opinions, so they recoded seven opinions after comparing their data with this previous work. [14][14]. Id. at 1466.

Coding the selected cases came next, using 90 piloted questions provided in an appendix. [15][15]. Id. 1458; see also id. at app. A (documenting the survey by which R&B prepared their data). In their pilot study, three coders independently applied a draft questionnaire to the same ten cases, which identified points that required clarification. Again, this is a sound and settled practice in the field. Among the decisions the authors had to make was how to code for the outcome in each case. This is a perennial problem in case content analysis, considering the many possible forms of a litigation win, loss, or draw. Learning from the experience of other users of this methodology, [16][16]. See id. at 1462 n.136 (citing Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 85 tbl.5 (2002)). Ruben and Blocher also acknowledged the complexity of procedural posture in coding for litigation success but decided not to create any method of weighting litigant success based on procedural posture. See id. at 1470. It is surprising that Ruben and Blocher did not discuss the Priest-Klein selection hypothesis in their analysis of the actual and expected levels of litigant success. See generally George L. Priest & Benjamin Klein, The Selection of Disputes for Litigation, 13 J. Leg. Stud. 1 (1984). Ruben and Blocher explained their coding of litigation outcomes in enough detail to allow others to either replicate or revise those choices.

R&B also made sound decisions in how to categorize types of weapons laws. Rather than invent new categories from scratch, they smartly based their key categories on existing theoretical work. [17][17]. See Eugene Volokh, Implementing the Right to Keep and Bear Arms for Self-Defense: An Analytical Framework and a Research Agenda, 56 UCLA L. Rev. 1443, 1475–1549 (2009). R&B published their iteration of the Volokh framework as Appendix B. As with their publication of their search terms and their coding questions, making this categorization framework transparent invites participation by later scholars. Relying on the judgment of other scholars about the most relevant categories increases the chances that their work will help to build a consensus on the content of emerging doctrine.

During the coding process, Ruben and Blocher again opted for preferred methods. They relied on student coders who each received standard training; students double coded some cases and the authors analyzed their level of agreement using the most widely accepted statistical measures. [18][18]. Ruben & Blocher, supra note 1, at 1464. At the end of the process, however, they departed from standard practice to categorize for themselves the nature of the regulation in question: while unorthodox, the choice is explained in enough detail for other scholars to make informed judgments. [19][19]. Id. at 1466.

Throughout the construction of their database, Ruben and Blocher followed the methodological lead of other disciplines, just as other legal scholars are starting to do. Their mainstream method promotes collaboration among legal scholars and across social science disciplines, just as it was designed to do in other academic disciplines.

II. THE SECOND AMENDMENT AND CONSENSUS

In a constitutional and regulatory field fraught with policy and theoretical conflict, Ruben and Blocher designed a study that seeks consensus by documenting and describing doctrine rather than interpreting it. Their methodological choices make it possible, looking back, to isolate areas of disagreement among scholars when they offer competing descriptions of the doctrine. Future scholars can connect back to their work, treating their survey as a starting point.

What form might this future descriptive scholarship take? In many social science fields, an initial novel finding prompts other researchers to replicate or refute the study. [20][20]. For instance, the controversy over whether gun control laws increase crime comes to mind. John Lott initially studied the subject in More Guns, Less Crime (1998). Scholars responded in Steven Levitt, Freakonomics, 130–34 (2005), and Nat’l Research Council of the Nat’l Acads., Firearms and Violence: A Critical Review, 72–102 (2005). Replication studies, however, would not serve the same purpose in descriptive Second Amendment scholarship. Because Ruben and Blocher coded the entire set of cases they collected, rather than a sample of those cases, replication studies are not necessary to test for sampling error.

Instead, future researchers might extend R&B’s methodology to later time periods, [21][21]. See, e.g., Robert B. Thompson, Piercing the Corporate Veil: An Empirical Study, 76 Cornell L. Rev. 1036, 1044–1070 (1991) (content analysis of judicial opinions on the topic of piercing the corporate veil); Lee C. Hodge & Andrew B. Sachs, Piercing the Mist: Bringing the Thompson Study into the 1990s, 43 Wake Forest L. Rev. 341, 341 (2008); Richmond McPherson & Nader Raja, Corporate Justice: An Empirical Study of Piercing Rates and Factors Courts Consider When Piercing the Corporate Veil, 45 Wake Forest L. Rev. 931, 937–41 (2010). or they could conduct “sensitivity testing,” which explores whether altering some of R&Bs methodological choices changes substantially their observed patterns. Future researchers might also add further topics to R&B’s questionnaire or recode the answers to certain questions in ways that they find more enlightening. Each of these new studies might reach some unique insights, but they are likely to build from a descriptive foundation that Ruben and Blocher created.

Finally, in most social science fields, scholars at some point turn to meta-analysis. That is, they combine the results from a group of studies, using larger sample sizes to resolve differences among the studies and to achieve better estimates of the effects that researchers studied. Meta-analysis, in this classic sense, matters less for studies of judicial opinions than for other topics of social science research because sampling error is normally not an issue. When every researcher analyzes the entire body of cases, a meta-analysis to achieve larger sample sizes for clearer insight is beside the point.

On the other hand, a “systematic” review of different doctrinal analyses of the Second Amendment might reveal interesting trends. [22][22]. A meta-analysis is one example of the broader category of systematic reviews. See David Moher et al., Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, 151 Annals Internal Med. 264, 264 (2009). A systematic review of doctrinal studies would identify the question the reviewer wants to answer (such as, “Who brings Second Amendment challenges to firearms regulations?”), the past studies that address this question (including the R&B study, the published and unpublished studies that predate their work, and any later systematic efforts to collect and describe the case law), the relative strengths of the methods each study used, and a method of synthesizing what each study reports (perhaps by weighting more heavily the studies that used the most reliable methodologies). This method offers a credible way of drawing insights from the entire body of scholarly work that might not be available from a single study, and it minimizes subjectivity and selection bias in characterizing what the scholarly literature says.

William Baude, Adam Chilton, and Anup Malani propose the use of systematic literature review methods at an earlier stage: to evaluate the judicial opinions themselves, rather than to synthesize scholarly studies of those opinions. [23][23]. See Baude et al., supra note 7, at 51–54. Systematic review, in this usage, becomes an alternative to content analysis. The two methods share many virtues, such as transparency in announcing the research questions and the search terms used during case collection. They are not, however, identical. Systematic literature review is somewhat more flexible than content analysis, allowing more room for researchers to weight certain judicial opinions more heavily, based on criteria that the authors announce and defend. On the other hand, content analysis tends to provide more uniform methods of analysis across different studies, allowing the field to draw conclusions across time and jurisdictions as results accumulate from studies of different collections of judicial opinions.

CONCLUSION

What enthuses us most about Ruben and Blocher’s use of content analysis is not just how well, but also when they chose to deploy this method—early in the development of an important new field of legal doctrine. Fields of law that are well established must grapple with a legacy of Lone Ranger-style interpretations of the caselaw. Scholars who arrive later, using content analysis or similar social science methods, can find it difficult to build consensus about how best to describe the doctrine. The relevant time frame for study in those established areas can be disputed, and the earlier scholarship sets up competing ways to frame the relevant questions. [24][24]. We catalogued examples—including some suboptimal uses of the method—a decade ago, in Hall & Wright, supra note 5, at 71 n.29. Here, R&B present an exemplary use of collaborative methods at the beginning stage for an emerging field. Future courts and scholars can build on this work, making possible a consensus—on the descriptive level—in this constitutional field where scholarly collaboration once seemed too much to ask.


Copyright© 2018 Ronald F. Wright & Mark A. Hall.

Needham Y. Gulley Professor of Criminal Law, and Fred & Elizabeth Turnage Professor of Law, respectively, both at Wake Forest University.

[1] Eric Ruben & Joseph Blocher, From Theory to Doctrine: An Empirical Analysis of the Right to Keep and Bear Arms After Heller, 67 Duke L.J. 1433, 1435 (2018).

[2] See, e.g., Mark A. Hall, The History and Future of Health Care Law: An Essentialist View, 41 Wake Forest L. Rev. 347, 357–63 (2006); Ronald F. Wright, Sentencers, Bureaucrats, and the Administrative Law Perspective on the Federal Sentencing Commission, 79 Calif. L. Rev. 1, 4–5 (1991). One of us also declared the arrival of milk law, a new field that inexplicably failed to attract the attention of other scholars. See Ronald F. Wright & Paul Huck, Counting Cases About Milk, Our “Most Nearly Perfect Food,” 1860-1940, 36 Law & Soc’y Rev. 51, 51–57 (2002).

[3] William L. Prosser, Privacy, 48 Calif. L. Rev. 383, 388–89 (1960) (declaring four distinct privacy torts). At the other end of the doctrinal life cycle, see generally Grant Gilmore, The Death of Contract (1974).

[4] See generally Neil M. Richards & Daniel J. Solove, Prosser’s Privacy Law: A Mixed Legacy, 98 Calif. L. Rev. 1887 (2010) (describing the impact and critiques of Prosser’s doctrinal synthesis).

[5] Mark A. Hall & Ronald F. Wright, Systematic Content Analysis of Judicial Opinions, 96 Calif. L. Rev. 63, 73 (2008).

[6] See Yes, I’d Lie to You, Economist (Sept. 10, 2016), https://www.economist.com/briefing/2016/09/10/yes-id-lie-to-you [https://perma.cc/8BRX-VPCV] (documenting the rise of the “post-truth” era); see also Julie Beck, This Article Won’t Change Your Mind, Atlantic, (Mar. 13, 2017), https://www.theatlantic.com/science/archive/2017/03/this-article-wont-change-your-mind/519093 [https://perma.cc/BEA8-3YMX] (exploring the phenomenon by which the presentation of evidence contrary to one’s beliefs may in fact reinforce them).

[7] For further information on the post-truth era’s implications for the law, see generally Allison Larsen, Constitutional Law in an Age of Alternative Facts, 93 N.Y.U. L. Rev. 175 (2018).

[8] See generally William Baude, Adam S. Chilton & Anup Malani, Making Doctrinal Work More Rigorous: Lessons from Systematic Reviews, 84 U. Chi. L. Rev. 37 (2017) (arguing that legal scholars should conduct systematic reviews).

[9] See Ruben & Blocher, supra note 1, at 1455–58.

[10] Id. at 1458.

[11] Id.

[12] Id. at 1461. Another exclusion that requires some legal judgment that might be difficult to replicate involved “nondecisional” cases, such as magistrate recommendations or opinions later vacated by the same judges. See id. at 1460.

[13] Examples include brief doctrinal discussions of the Second Amendment in the course of determining the effectiveness of defense counsel or the qualified immunity of police officers. Id. at 1461.

[14] Id. at 1466.

[15] Id. 1458; see also id. at app. A (documenting the survey by which R&B prepared their data). In their pilot study, three coders independently applied a draft questionnaire to the same ten cases, which identified points that required clarification. Again, this is a sound and settled practice in the field.

[16] See id. at 1462 n.136 (citing Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 85 tbl.5 (2002)). Ruben and Blocher also acknowledged the complexity of procedural posture in coding for litigation success but decided not to create any method of weighting litigant success based on procedural posture. See id. at 1470. It is surprising that Ruben and Blocher did not discuss the Priest-Klein selection hypothesis in their analysis of the actual and expected levels of litigant success. See generally George L. Priest & Benjamin Klein, The Selection of Disputes for Litigation, 13 J. Leg. Stud. 1 (1984).

[17] See Eugene Volokh, Implementing the Right to Keep and Bear Arms for Self-Defense: An Analytical Framework and a Research Agenda, 56 UCLA L. Rev. 1443, 1475–1549 (2009). R&B published their iteration of the Volokh framework as Appendix B. As with their publication of their search terms and their coding questions, making this categorization framework transparent invites participation by later scholars.

[18] Ruben & Blocher, supra note 1, at 1464.

[19] Id. at 1466.

[20] For instance, the controversy over whether gun control laws increase crime comes to mind. John Lott initially studied the subject in More Guns, Less Crime (1998). Scholars responded in Steven Levitt, Freakonomics, 130–34 (2005), and Nat’l Research Council of the Nat’l Acads., Firearms and Violence: A Critical Review, 72–102 (2005).

[21] See, e.g., Robert B. Thompson, Piercing the Corporate Veil: An Empirical Study, 76 Cornell L. Rev. 1036, 1044–1070 (1991) (content analysis of judicial opinions on the topic of piercing the corporate veil); Lee C. Hodge & Andrew B. Sachs, Piercing the Mist: Bringing the Thompson Study into the 1990s, 43 Wake Forest L. Rev. 341, 341 (2008); Richmond McPherson & Nader Raja, Corporate Justice: An Empirical Study of Piercing Rates and Factors Courts Consider When Piercing the Corporate Veil, 45 Wake Forest L. Rev. 931, 937–41 (2010).

[22] A meta-analysis is one example of the broader category of systematic reviews. See David Moher et al., Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, 151 Annals Internal Med. 264, 264 (2009).

[23] See Baude et al., supra note 7, at 51–54.

[24] We catalogued examples—including some suboptimal uses of the method—a decade ago, in Hall & Wright, supra note 5, at 71 n.29.

Collaborative Construction Of A New Legal Field

Volume 68 September 2018
COLLABORATIVE CONSTRUCTION OF A NEW LEGAL FIELD

Ronald F. Wright & Mark A. Hall †. Needham Y. Gulley Professor of Criminal Law, and Fred & Elizabeth Turnage Professor of Law, respectively, both at Wake Forest University.

INTRODUCTION

Eric Ruben and Joseph Blocher have announced the birth of an “entire field of constitutional doctrine,” which they rightly note is “a rare challenge and opportunity for judges, lawyers, and scholars.” [1][1]. Eric Ruben & Joseph Blocher, From Theory to Doctrine: An Empirical Analysis of the Right to Keep and Bear Arms After Heller, 67 Duke L.J. 1433, 1435 (2018). The birth announcement of a new legal field is a familiar move, one that we’ve made ourselves (with limited success). [2][2]. See, e.g., Mark A. Hall, The History and Future of Health Care Law: An Essentialist View, 41 Wake Forest L. Rev. 347, 357–63 (2006); Ronald F. Wright, Sentencers, Bureaucrats, and the Administrative Law Perspective on the Federal Sentencing Commission, 79 Calif. L. Rev. 1, 4–5 (1991). One of us also declared the arrival of milk law, a new field that inexplicably failed to attract the attention of other scholars. See Ronald F. Wright & Paul Huck, Counting Cases About Milk, Our “Most Nearly Perfect Food,” 1860-1940, 36 Law & Soc’y Rev. 51, 51–57 (2002). What is more intriguing for us than learning of this new arrival is the way that Ruben and Blocher chose to midwife this new legal field: using the social science methodology of textual content analysis.

The time-honored method of discovering a new doctrinal field has been the ineffable, hard to replicate work of an individual scholar reading selected cases. For example, when William Prosser declared a new doctrine of tort-enforced privacy, he described a collection of published judicial opinions and highlighted their common elements and future implications. [3][3]. William L. Prosser, Privacy, 48 Calif. L. Rev. 383, 388–89 (1960) (declaring four distinct privacy torts). At the other end of the doctrinal life cycle, see generally Grant Gilmore, The Death of Contract (1974). This declaration singled out cases for scrutiny but did not identify the areas where Prosser searched for evidence or discuss the cases that did not fit his chosen pattern. Critics of the author’s descriptive framework must start from scratch—constructing their own caselaw searches and analyzing their competing examples—to explain where the author might have gone wrong. [4][4]. See generally Neil M. Richards & Daniel J. Solove, Prosser’s Privacy Law: A Mixed Legacy, 98 Calif. L. Rev. 1887 (2010) (describing the impact and critiques of Prosser’s doctrinal synthesis).

Unlike this classic “Lone Ranger” style of legal scholarship, Ruben and Blocher (who we’ll sometimes call R&B) have published more of an invitation than an announcement. They want to involve others in their construction of a new body of legal doctrine, using the established techniques of content analysis to study Second Amendment cases in ways that other researchers can replicate. They specify the questions they ask, their method of collecting cases, and their techniques for classifying those cases. While this methodology has many useful applications across legal scholarship, [5][5]. Mark A. Hall & Ronald F. Wright, Systematic Content Analysis of Judicial Opinions, 96 Calif. L. Rev. 63, 73 (2008). we are especially intrigued by its use in the earliest days of an emerging field, when scholars first start to see judicial decisions forming a connected doctrinal body of law.

In our post-truth era, [6][6]. See Yes, I’d Lie to You, Economist (Sept. 10, 2016), https://www.economist.com/briefing/2016/09/10/yes-id-lie-to-you [https://perma.cc/8BRX-VPCV] (documenting the rise of the “post-truth” era); see also Julie Beck, This Article Won’t Change Your Mind, Atlantic, (Mar. 13, 2017), https://www.theatlantic.com/science/archive/2017/03/this-article-wont-change-your-mind/519093 [https://perma.cc/BEA8-3YMX] (exploring the phenomenon by which the presentation of evidence contrary to one’s beliefs may in fact reinforce them). it is ever more important for legal scholars to develop and refine objective and verifiable ways to analyze legal doctrine. [7][7]. For further information on the post-truth era’s implications for the law, see generally Allison Larsen, Constitutional Law in an Age of Alternative Facts, 93 N.Y.U. L. Rev. 175 (2018). Thus, in this comment, we spotlight the mainstream methodological choices that R&B made at each stage of their content analysis. We also consider how their work resembles the “systematic literature review” method that social scientists use to synthesize large bodies of research in a field. [8][8]. See generally William Baude, Adam S. Chilton & Anup Malani, Making Doctrinal Work More Rigorous: Lessons from Systematic Reviews, 84 U. Chi. L. Rev. 37 (2017) (arguing that legal scholars should conduct systematic reviews). Finally, we consider the scholarly collaboration that this social science methodology makes possible for Second Amendment scholarship going forward. In particular, what sort of replication studies or meta-analysis could other scholars pursue, using R&B’s work as a starting point?

I. OPEN-SOURCE CODING FOR DOCTRINE

Blocher and Ruben’s use of content analysis is an exemplary model for others to follow. Critically, they explained their methods in enough detail to allow other scholars to replicate their study or to make informed choices about building a different study. Their doctrinal coding was open for users—and for later data collectors—to see.

Data selection was their first step. [9][9]. See Ruben & Blocher, supra note 1, at 1455–58. After setting the start and end dates of their search, the authors rejected databases assembled by advocates and chose instead a Westlaw database that included both trial and appellate decisions from the state and federal courts. Given the small number of state trial court opinions in the search results and the overrepresentation of New York in those results, they justifiably excluded state trial court cases. [10][10]. Id. at 1458. While reported opinions do not reliably reflect the activity of all courts, the use of reported opinions is well suited to the project here, which focuses on how precedent is building and forming a new doctrinal area. Happily, these data selection methods produced a reasonable number of cases, allowing the authors to code a “universal sample” of all relevant cases. They avoided hard choices about how to select a representative sample for coding. [11][11]. Id.

R&B’s most difficult judgment call in collecting cases was their exclusion of cases in which the Second Amendment outcome was “incidental” to the issue(s) being decided. [12][12]. Id. at 1461. Another exclusion that requires some legal judgment that might be difficult to replicate involved “nondecisional” cases, such as magistrate recommendations or opinions later vacated by the same judges. See id. at 1460. Beyond offering four convincing examples of this exclusion, [13][13]. Examples include brief doctrinal discussions of the Second Amendment in the course of determining the effectiveness of defense counsel or the qualified immunity of police officers. Id. at 1461. the authors stated this aspect of their selection criterion only in general terms, leaving ample room for later researchers to adopt confirmatory or alternative approaches. In fact, R&B noted that other researchers had previously made different judgments about including or excluding a few of the opinions, so they recoded seven opinions after comparing their data with this previous work. [14][14]. Id. at 1466.

Coding the selected cases came next, using 90 piloted questions provided in an appendix. [15][15]. Id. 1458; see also id. at app. A (documenting the survey by which R&B prepared their data). In their pilot study, three coders independently applied a draft questionnaire to the same ten cases, which identified points that required clarification. Again, this is a sound and settled practice in the field. Among the decisions the authors had to make was how to code for the outcome in each case. This is a perennial problem in case content analysis, considering the many possible forms of a litigation win, loss, or draw. Learning from the experience of other users of this methodology, [16][16]. See id. at 1462 n.136 (citing Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 85 tbl.5 (2002)). Ruben and Blocher also acknowledged the complexity of procedural posture in coding for litigation success but decided not to create any method of weighting litigant success based on procedural posture. See id. at 1470. It is surprising that Ruben and Blocher did not discuss the Priest-Klein selection hypothesis in their analysis of the actual and expected levels of litigant success. See generally George L. Priest & Benjamin Klein, The Selection of Disputes for Litigation, 13 J. Leg. Stud. 1 (1984). Ruben and Blocher explained their coding of litigation outcomes in enough detail to allow others to either replicate or revise those choices.

R&B also made sound decisions in how to categorize types of weapons laws. Rather than invent new categories from scratch, they smartly based their key categories on existing theoretical work. [17][17]. See Eugene Volokh, Implementing the Right to Keep and Bear Arms for Self-Defense: An Analytical Framework and a Research Agenda, 56 UCLA L. Rev. 1443, 1475–1549 (2009). R&B published their iteration of the Volokh framework as Appendix B. As with their publication of their search terms and their coding questions, making this categorization framework transparent invites participation by later scholars. Relying on the judgment of other scholars about the most relevant categories increases the chances that their work will help to build a consensus on the content of emerging doctrine.

During the coding process, Ruben and Blocher again opted for preferred methods. They relied on student coders who each received standard training; students double coded some cases and the authors analyzed their level of agreement using the most widely accepted statistical measures. [18][18]. Ruben & Blocher, supra note 1, at 1464. At the end of the process, however, they departed from standard practice to categorize for themselves the nature of the regulation in question: while unorthodox, the choice is explained in enough detail for other scholars to make informed judgments. [19][19]. Id. at 1466.

Throughout the construction of their database, Ruben and Blocher followed the methodological lead of other disciplines, just as other legal scholars are starting to do. Their mainstream method promotes collaboration among legal scholars and across social science disciplines, just as it was designed to do in other academic disciplines.

II. THE SECOND AMENDMENT AND CONSENSUS

In a constitutional and regulatory field fraught with policy and theoretical conflict, Ruben and Blocher designed a study that seeks consensus by documenting and describing doctrine rather than interpreting it. Their methodological choices make it possible, looking back, to isolate areas of disagreement among scholars when they offer competing descriptions of the doctrine. Future scholars can connect back to their work, treating their survey as a starting point.

What form might this future descriptive scholarship take? In many social science fields, an initial novel finding prompts other researchers to replicate or refute the study. [20][20]. For instance, the controversy over whether gun control laws increase crime comes to mind. John Lott initially studied the subject in More Guns, Less Crime (1998). Scholars responded in Steven Levitt, Freakonomics, 130–34 (2005), and Nat’l Research Council of the Nat’l Acads., Firearms and Violence: A Critical Review, 72–102 (2005). Replication studies, however, would not serve the same purpose in descriptive Second Amendment scholarship. Because Ruben and Blocher coded the entire set of cases they collected, rather than a sample of those cases, replication studies are not necessary to test for sampling error.

Instead, future researchers might extend R&B’s methodology to later time periods, [21][21]. See, e.g., Robert B. Thompson, Piercing the Corporate Veil: An Empirical Study, 76 Cornell L. Rev. 1036, 1044–1070 (1991) (content analysis of judicial opinions on the topic of piercing the corporate veil); Lee C. Hodge & Andrew B. Sachs, Piercing the Mist: Bringing the Thompson Study into the 1990s, 43 Wake Forest L. Rev. 341, 341 (2008); Richmond McPherson & Nader Raja, Corporate Justice: An Empirical Study of Piercing Rates and Factors Courts Consider When Piercing the Corporate Veil, 45 Wake Forest L. Rev. 931, 937–41 (2010). or they could conduct “sensitivity testing,” which explores whether altering some of R&Bs methodological choices changes substantially their observed patterns. Future researchers might also add further topics to R&B’s questionnaire or recode the answers to certain questions in ways that they find more enlightening. Each of these new studies might reach some unique insights, but they are likely to build from a descriptive foundation that Ruben and Blocher created.

Finally, in most social science fields, scholars at some point turn to meta-analysis. That is, they combine the results from a group of studies, using larger sample sizes to resolve differences among the studies and to achieve better estimates of the effects that researchers studied. Meta-analysis, in this classic sense, matters less for studies of judicial opinions than for other topics of social science research because sampling error is normally not an issue. When every researcher analyzes the entire body of cases, a meta-analysis to achieve larger sample sizes for clearer insight is beside the point.

On the other hand, a “systematic” review of different doctrinal analyses of the Second Amendment might reveal interesting trends. [22][22]. A meta-analysis is one example of the broader category of systematic reviews. See David Moher et al., Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, 151 Annals Internal Med. 264, 264 (2009). A systematic review of doctrinal studies would identify the question the reviewer wants to answer (such as, “Who brings Second Amendment challenges to firearms regulations?”), the past studies that address this question (including the R&B study, the published and unpublished studies that predate their work, and any later systematic efforts to collect and describe the case law), the relative strengths of the methods each study used, and a method of synthesizing what each study reports (perhaps by weighting more heavily the studies that used the most reliable methodologies). This method offers a credible way of drawing insights from the entire body of scholarly work that might not be available from a single study, and it minimizes subjectivity and selection bias in characterizing what the scholarly literature says.

William Baude, Adam Chilton, and Anup Malani propose the use of systematic literature review methods at an earlier stage: to evaluate the judicial opinions themselves, rather than to synthesize scholarly studies of those opinions. [23][23]. See Baude et al., supra note 7, at 51–54. Systematic review, in this usage, becomes an alternative to content analysis. The two methods share many virtues, such as transparency in announcing the research questions and the search terms used during case collection. They are not, however, identical. Systematic literature review is somewhat more flexible than content analysis, allowing more room for researchers to weight certain judicial opinions more heavily, based on criteria that the authors announce and defend. On the other hand, content analysis tends to provide more uniform methods of analysis across different studies, allowing the field to draw conclusions across time and jurisdictions as results accumulate from studies of different collections of judicial opinions.

CONCLUSION

What enthuses us most about Ruben and Blocher’s use of content analysis is not just how well, but also when they chose to deploy this method—early in the development of an important new field of legal doctrine. Fields of law that are well established must grapple with a legacy of Lone Ranger-style interpretations of the caselaw. Scholars who arrive later, using content analysis or similar social science methods, can find it difficult to build consensus about how best to describe the doctrine. The relevant time frame for study in those established areas can be disputed, and the earlier scholarship sets up competing ways to frame the relevant questions. [24][24]. We catalogued examples—including some suboptimal uses of the method—a decade ago, in Hall & Wright, supra note 5, at 71 n.29. Here, R&B present an exemplary use of collaborative methods at the beginning stage for an emerging field. Future courts and scholars can build on this work, making possible a consensus—on the descriptive level—in this constitutional field where scholarly collaboration once seemed too much to ask.


Copyright© 2018 Ronald F. Wright & Mark A. Hall.

Needham Y. Gulley Professor of Criminal Law, and Fred & Elizabeth Turnage Professor of Law, respectively, both at Wake Forest University.

[1] Eric Ruben & Joseph Blocher, From Theory to Doctrine: An Empirical Analysis of the Right to Keep and Bear Arms After Heller, 67 Duke L.J. 1433, 1435 (2018).

[2] See, e.g., Mark A. Hall, The History and Future of Health Care Law: An Essentialist View, 41 Wake Forest L. Rev. 347, 357–63 (2006); Ronald F. Wright, Sentencers, Bureaucrats, and the Administrative Law Perspective on the Federal Sentencing Commission, 79 Calif. L. Rev. 1, 4–5 (1991). One of us also declared the arrival of milk law, a new field that inexplicably failed to attract the attention of other scholars. See Ronald F. Wright & Paul Huck, Counting Cases About Milk, Our “Most Nearly Perfect Food,” 1860-1940, 36 Law & Soc’y Rev. 51, 51–57 (2002).

[3] William L. Prosser, Privacy, 48 Calif. L. Rev. 383, 388–89 (1960) (declaring four distinct privacy torts). At the other end of the doctrinal life cycle, see generally Grant Gilmore, The Death of Contract (1974).

[4] See generally Neil M. Richards & Daniel J. Solove, Prosser’s Privacy Law: A Mixed Legacy, 98 Calif. L. Rev. 1887 (2010) (describing the impact and critiques of Prosser’s doctrinal synthesis).

[5] Mark A. Hall & Ronald F. Wright, Systematic Content Analysis of Judicial Opinions, 96 Calif. L. Rev. 63, 73 (2008).

[6] See Yes, I’d Lie to You, Economist (Sept. 10, 2016), https://www.economist.com/briefing/2016/09/10/yes-id-lie-to-you [https://perma.cc/8BRX-VPCV] (documenting the rise of the “post-truth” era); see also Julie Beck, This Article Won’t Change Your Mind, Atlantic, (Mar. 13, 2017), https://www.theatlantic.com/science/archive/2017/03/this-article-wont-change-your-mind/519093 [https://perma.cc/BEA8-3YMX] (exploring the phenomenon by which the presentation of evidence contrary to one’s beliefs may in fact reinforce them).

[7] For further information on the post-truth era’s implications for the law, see generally Allison Larsen, Constitutional Law in an Age of Alternative Facts, 93 N.Y.U. L. Rev. 175 (2018).

[8] See generally William Baude, Adam S. Chilton & Anup Malani, Making Doctrinal Work More Rigorous: Lessons from Systematic Reviews, 84 U. Chi. L. Rev. 37 (2017) (arguing that legal scholars should conduct systematic reviews).

[9] See Ruben & Blocher, supra note 1, at 1455–58.

[10] Id. at 1458.

[11] Id.

[12] Id. at 1461. Another exclusion that requires some legal judgment that might be difficult to replicate involved “nondecisional” cases, such as magistrate recommendations or opinions later vacated by the same judges. See id. at 1460.

[13] Examples include brief doctrinal discussions of the Second Amendment in the course of determining the effectiveness of defense counsel or the qualified immunity of police officers. Id. at 1461.

[14] Id. at 1466.

[15] Id. 1458; see also id. at app. A (documenting the survey by which R&B prepared their data). In their pilot study, three coders independently applied a draft questionnaire to the same ten cases, which identified points that required clarification. Again, this is a sound and settled practice in the field.

[16] See id. at 1462 n.136 (citing Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 85 tbl.5 (2002)). Ruben and Blocher also acknowledged the complexity of procedural posture in coding for litigation success but decided not to create any method of weighting litigant success based on procedural posture. See id. at 1470. It is surprising that Ruben and Blocher did not discuss the Priest-Klein selection hypothesis in their analysis of the actual and expected levels of litigant success. See generally George L. Priest & Benjamin Klein, The Selection of Disputes for Litigation, 13 J. Leg. Stud. 1 (1984).

[17] See Eugene Volokh, Implementing the Right to Keep and Bear Arms for Self-Defense: An Analytical Framework and a Research Agenda, 56 UCLA L. Rev. 1443, 1475–1549 (2009). R&B published their iteration of the Volokh framework as Appendix B. As with their publication of their search terms and their coding questions, making this categorization framework transparent invites participation by later scholars.

[18] Ruben & Blocher, supra note 1, at 1464.

[19] Id. at 1466.

[20] For instance, the controversy over whether gun control laws increase crime comes to mind. John Lott initially studied the subject in More Guns, Less Crime (1998). Scholars responded in Steven Levitt, Freakonomics, 130–34 (2005), and Nat’l Research Council of the Nat’l Acads., Firearms and Violence: A Critical Review, 72–102 (2005).

[21] See, e.g., Robert B. Thompson, Piercing the Corporate Veil: An Empirical Study, 76 Cornell L. Rev. 1036, 1044–1070 (1991) (content analysis of judicial opinions on the topic of piercing the corporate veil); Lee C. Hodge & Andrew B. Sachs, Piercing the Mist: Bringing the Thompson Study into the 1990s, 43 Wake Forest L. Rev. 341, 341 (2008); Richmond McPherson & Nader Raja, Corporate Justice: An Empirical Study of Piercing Rates and Factors Courts Consider When Piercing the Corporate Veil, 45 Wake Forest L. Rev. 931, 937–41 (2010).

[22] A meta-analysis is one example of the broader category of systematic reviews. See David Moher et al., Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement, 151 Annals Internal Med. 264, 264 (2009).

[23] See Baude et al., supra note 7, at 51–54.

[24] We catalogued examples—including some suboptimal uses of the method—a decade ago, in Hall & Wright, supra note 5, at 71 n.29.