Data for: The structure of description: Evaluating historical description and its role in theorizing

Résumé 0

This is an Annotation for Transparent Inquiry (ATI) data project. The annotated article can be viewed on the publisher's website. This data overview first briefly discusses the paper’s data generation and analysis elements that Annotation for Transparency Inquiry (ATI) tries to foreground. It then elaborates more fully on the paper’s logic of annotation by identifying two broad annotation categories that encompass seven distinct annotation sub-categories. 1.1. Data Generation: The generation of data can take several forms. A conventional hypothesis testing paper will generate data to test the empirical implications derived from its theoretical predictions as well as those of its main challengers. Such data is generated by drawing on existing data sets, conducting experiments, or doing original field or archival research. This paper takes a slightly different path given that it does not test hypotheses. It pursues what could be call a quasi meta-analysis and uses as its data the book reviews, scholarly exchanges, historiographical assessments, and archival “replications” that together comprised the so-called Goldhagen controversy. It treats the qualitative findings of other scholars as evidence and thus can be said to resemble a meta-analysis. As a result, my date generation involved little more than downloading PDF or articles or checking books from the library. The paper’s reliance on such secondary, easily accessible, and copy-righted material also means that it makes little sense to included original textual passages as is the standard in the ATI. However, such a meta-analysis involves a sufficient number of different research judgments that should be made transparent because their complexity makes them unsuitable for conventional footnotes. For those instances, I choose to add ATI. The logic of annotation section below elaborates more fully on the nature of those judgements. 1.3. Data Analysis: I reviewed 45 distinct contributions to the debate that varied in length from short book reviews to detailed, longish journal articles. Of those 45 works, I ended up citing nine (20%). The selection was largely dictated by what I deemed to be their relevance in helping me evaluate Goldhagen and Browning’s descriptive inferences. It might be worth recalling that I am not trying to settle every last facet of the Goldhagen debate nor am I interested in the wider public reception of his work which received a lot of attention. I am strictly interested in using the debate to illustrate a narrow, strictly methodological point about the nature of description. I used citation scores as a second criterion for their selection. I used those scores as a proxy measure for the scholarly consensus on the validity of the assessment of Goldhagen and Browning’s work. Using the term in strictly figurative sense, these selection criteria constituted my “sampling” strategy. It should add that the works from which I could sample from where heavily skewed in favor of Browning and against Goldhagen. Of the 39 works (excluding Goldhagens self-defenses), only one scholar defended Goldhagen (Markovits 1998) and one article remained neutral. (Mahoney and Ellsberg 1999) I attribute this skewedness to the scholarly consensus on the flaws in Goldhagen’s analysis. I cannot confirm this consensus because I did not analyze the primary materials myself. Goldhagen critics, for example, could have been motivated less by his factual errors and more by his oftentimes condescending tone of earlier scholars’ work, professional animosity towards a non-historian (Goldhagen was a political scientist and most of his critics historians), or sheer envy of the book’s commercial success, and very positive public (i.e. none scholarly) reception, particularly in Germany. (Deak 1997, 297) 1.4. Logic of Annotation: Broadly speaking, this paper’s ATI tries to increase confidence in its claims by foregrounding two sets of research judgments that are usually not acknowledged in conventional footnotes. The first set of judgments relate to the selection, and interpretation of empirical evidence used to support my claims. These are the very judgments that the conventional ATI is trying to foreground. I will refer to the foregrounding of those judgements as research transparency. The second set of judgments veers a bit farther from this more conventional use of ATI. These judgments related to broader elements of knowledge production that also influence the confidence in our research findings independently on how the evidence was handled. I will refer to the foregrounding of those judgments as research integrity. The rest of this section elaborates on these two logics and identifies the specific annotation sub-categories it uses. Each bold-faced term in the text indicates a distinct sub-category of either research transparency or research integrity. Each subsequent annotation in turn is assigned on of these sub-categories. Parenthetically, I should point out that this paper does not aspire to a comprehensive ATI annotation in which every element of research transparency and research integrity are addressed and foregrounded. The invitation to annotate this paper came after it was completed, thus making it impossible to retrace each of its research judgments. I instead selected those judgements that were both important for the paper, and where I still could retrace in my notes the original judgements. The paper thus practices only part of what it preaches. But I still hope that those annotations will both increase the confidence in my findings, and more importantly, illustrate the range of possible ATI applications. Research Transparency: The vast majority of my annotations follow the standard logic of using ATI to make transparent three key judgments related the evidence used to evaluate theoretical claims. These claims follow closely the three categories outlined by the Data Access and Research Transparency (DA-RT) initiative: data access, production transparency, and analytical transparency. It places particular attention on analytical transparency because those criteria are still evolving with respect to qualitative textual or ethnographic evidence. (Buthe & Jacobs, forthcoming, 2015; Golden & Golden 2016) This paper’s effort to specify criteria to evaluate descriptive inferences is part of this broader effort to make more explicit the criteria for qualitative evidence that analytical transparency should address. For example, the current specifications of analytical transparency don’t ask authors to be explicit about the conceptualization, ontological parameter, epistemological presuppositions, or cross-level inferences, four element this paper highlights. In another paper, I tried to be more specific elements of analytical transparency that apply to causal as opposed to descriptive inferences. Those elements include assessment of prior research, the specificity of theoretical predictions, units of analysis, and ontological assumptions. (Kreuzer, no date) Based on this prior work, I would like to add conceptual transparency as an additional ATI category to create the possibility to elaborate more fully on terminological or conceptual ambiguities. Research Integrity: The paper looks at research integrity as broader category than research transparency that affects confidence in scholarly findings. Research integrity is not as clearly established as research transparency. The term was first proposed by Peter Hall (2016) to underscore the limitations of the narrower, more test-centric DA-RT research transparency guidelines. It has been espoused by scholars in the recent Qualitative Transparency Deliberations (QTD) to point to a wider range of research practices that affect the integrity of the research process even though they are distinct from the final testing stage. (Buthe Jacobs, forthcoming) Research integrity sees research transparency as one but not the exclusive focus for Annotation for Transparency Inquiry. (ATI) As Figure 1 shows, research integrity raises the possibility that ATI also address question related to research ethics, knowledge production, and epistemological priors. Various journals require scholars to disclose conflicts of interests or funding sources to make potential ethical conflicts more transparent. ATI would permit more substantial discussions in instances where such conflicts exist. I do not include any ATI related to ethical conflicts. Production of knowledge is a broad umbrella term used in history of science or sociology of knowledge to explore how the personal, professional, economic, and political circumstances shape the research process independently of the final, empirical testing stage. This category is fairly broad but could include elements like the publication process, disciplinary variations in standards for research transparency, professional rank of authors relative to each other, the positionality of the scholar, the broader, public reception of the work, or funding agencies. Finally, epistemological priors try to explicate broader philosophical presuppositions that might cast doubt on research findings because they are not widely shared or incommensurate with other works that a scholar engages. For this paper, these epistemological priors were sufficiently important that it addresses them in section 1.2. rather than relegating them to an ATI. Sources Cited: Büthe, Tim and Alan Jacobs, eds. “Qualitative Transparency Deliberations Report,” Perspectives on Politics, forthcoming. Büthe, Tim, and Alan Jacobs, eds. 2015. “Symposium: Transparency in Qualitative and Multi-Method Research.” Qualitative and Multi-Method Research 13(1). Deák, István. 1997. “Holocaust Views: The Goldhagen Controversy in Retrospect.” Central European History 30(2): 295–3 Golden, Matt, and Sona Golden, eds. 2016. “Symposium: Data Access and Research Transparency.” Comparative Politics Newsletter 26(1). Kreuzer, Marcus. Unpublished paper. “DATA FILES, FOOTNOTES, AND EDITORS: Bridging Quantitative, Qualitative, and Editorial Transparency Practice Hall, Peter. 2016. “Transparency, Research Integrity, and Multiple Methods.” Comparative Politics Newsletter 26(1 (Spring)): 28–32 Mahoney, James, and Michael Ellsberg. 1999. “Goldhagen’s Hitler’s Willing Executioners: A Clarification and Methodological Critique.” Journal of Historical Sociology 12(4): 422–436 Markovits, Andrei. 1998. “Discomposure in History’s Final Resting Place.” In Unwilling Germans? The Goldhagen Debate, ed. Robert R. Shandley. Minneapolis: University of Minnesota Press.