Empirical Research and Writing: A Political Science Student’s Practical Guide [illustrated, revised] 1483369633, 9781483369631

Citation preview

Empirical Research and Writing

To my parents, who had to put up with my “why” questions for all those years, and to my husband, whose curiosity and geekiness are matched only by my own.

Empirical Research and Writing A Political Science Student’s Practical Guide

Leanne C. Powner

For information:

CQ Press An Imprint of SAGE Publications, Inc. 2455 Teller Road Thousand Oaks, California 91320 E-mail: [email protected] SAGE Publications Ltd.

Copyright  2015 by CQ Press, an Imprint of SAGE Publications, Inc. CQ Press is a registered trademark of Congressional Quarterly Inc. All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.

1 Oliver’s Yard 55 City Road London EC1Y 1SP United Kingdom SAGE Publications India Pvt. Ltd. B 1/I 1 Mohan Cooperative Industrial Area Mathura Road, New Delhi 110 044 India

Printed in the United States of America A catalog record of this book is available from the Library of Congress. ISBN 978-1-4833-6963-1

SAGE Publications Asia-Pacific Pte. Ltd. 3 Church Street #10-04 Samsung Hub Singapore 049483

This book is printed on acid-free paper.

Acquisitions Editor:  Sarah Calabi Editorial Assistant:  Davia Grant Production Editor:  Bennie Clark Allen Copy Editor:  Lana Todorovic-Arndt Typesetter:  C&M Digitals (P) Ltd. Proofreader:  Scott Oney Indexer: Maria Sosnowski Cover Designer:  Glenn Vogel Marketing Manager: Amy Whitaker

14 15 16 17 18 10 9 8 7 6 5 4 3 2 1

C o n t e n t s

Tables and Figures

x

Notes to Students and Instructors

xi

The Thank-You’s

xv

About the Author

xvii

Part I: The Preliminaries Chapter 1: From Research Topic to Research Question

1

Doing Social Science 3 Research Questions and This Course’s Research Project 3 From Research Topic to Research Question 5 Crafting a Research Question 6 Finding and Refining a Research Question 9 Where Do Research Questions Come From? 9 How to Phrase a Research Question 12 Writing Your Paper 18 Summary 19 Key Terms 19 Chapter 2: From Research Question to Theory to Hypothesis

21

What Is a Theory? 22 The Parts of a Theory 23 Where Theories Come From 25 From Question to Theory 28 Tools for Developing Your Theory 31 Parsing a Theory: The Causal Mechanism 36 Specifying Assumptions and Scope Conditions 41 From Theory to Hypothesis 44 The Parts of a Hypothesis 44 Hypotheses and Evidence 46 Concepts, Indicators, and Validity 48 Writing Your Theory Section 51 Summary 53 Key Terms 54

vi

Contents

Part II: The Practicalities Chapter 3: Doing Pre-research

55

The Parts of an Empirical Paper 55 How to Think About Literature(s) 56 What a Literature Review Is Not 57 How to Find Literature(s) 58 How to Organize Literature(s) 63 Managing the Reading and Information 63 Managing the Reference List 65 How to Write the Literature Review 68 Strategies for Finding an Organizational Structure 69 Organizing a Literature 71 Actually Writing the [********] Thing 72 Summary 80 Key Terms 80 Chapter 4: Choosing a Design That Fits Your Question

81

Types of Hypotheses 82 Four Probabilistic Hypothesis Types 82 Two Deterministic Hypothesis Types 89 What Type of Analysis Should I Conduct? 95 Similarities and Differences in Qualitative and Quantitative Methods 95 Strengths and Weaknesses of Qualitative and Quantitative Methods 97 Overview of Techniques 100 Major Forms of Quantitative Analysis 100 Major Forms of Qualitative Analysis 104 Summary 108 Key Terms 108 Chapter 5: Case Selection and Study Design for Qualitative Research

109

Qualitative Study Design 110 How Many Cases Should I Study? 110 Which Cases Should I Study? 112 A Note on Background Research 115 Hypothesis-Testing Techniques and Case Selection 116 Content Analysis 118 Analytic Narratives 122 Case Control/Controlled Comparison Method 124 Structured Focused Comparison 129 Process Tracing 130

Contents

Writing Your Methodology Section 132 Summary 134 Key Terms 134 Chapter 6: Qualitative Data Collection and Management

135

Information, Data, and Evidence 135 Information versus Data 136 Measurement 137 Maximizing Leverage Over Your Hypotheses 139 Kinds of Qualitative Data Sources 140 Data and Counterfactuals 141 Data Collection Techniques 144 Research From Sources 144 Human Subjects Research 148 Sources and Resources 152 Data Management Options 153 Writing About Data Collection 155 Summary 156 Key Terms 156 Chapter 7: Quantitative Data Collection and Management

157

Identifying Data Needs: What Cases? 157 Units of Analysis 158 Populations and Samples 161 Identifying Data Needs: What Variables? 162 Control Variables 162 Multiple Indicators and Robustness Checks 167 Measurement: Matching Concepts to Indicators 168 Operationalization: Validity 168 Matching Concepts and Indicators 170 Measurement: Reliability 170 Getting Ready-to-Use Data 171 Replication Datasets 172 Data Archives 174 Citing Datasets 174 Collecting and Managing Your Own Data 175 A Brief Discussion of Coding 177 Summary 179 Key Terms 179 Chapter 8: Preparing Quantitative Data for Analysis

181

Transferring Data into Your Stats Program 181 Getting Started 182 Dealing With Missing Data 183

vii

viii

Contents

Variables, Variable Names, and Variable Labels 184 Entering Variable Labels and Value Labels 184 Preparing to Analyze Your Data 185 Checking for Nonlinearity 185 Dealing With Nonlinearities 187 Detecting and Addressing Colinearity 190 Other Data Manipulations 192 Generating New Composite Variables 193 Recoding Variables 198 The Theory-Data Danger Zone: Endogeneity, Simultaneity, and Omitted Variable Bias 200 Endogeneity and Simultaneity 201 Omitted Variable Bias 203 Nonlinear Models 204 Summary 205 Key Terms 205 Chapter 9: Writing Up Your Research

207

The Abstract 207 The Bookends 209 The Introduction 209 The Conclusion 210 The Results: Conventions of Reporting and Discussing Quantitative Analysis 210 Tables and Figures 211 Conventions of Reporting Results 211 Discussing Qualitative Evidence and Claims 217 Focus on the Evidence 217 Be Explicit About the Study’s Limitations 219 Confine Your Conclusions 220 Summary 221 Key Terms 221

Part III: Post-paper Processes Chapter 10: Practicing Peer Review

223

Writing Without Whining 223 Self-Editing 226 Strategies of Self-Editing 228 Editing for Style and Tone: Social Science Writing 231 A Note on How Much Background Is Enough 234 Practicing Peer Review 234 Peer Review in the Social Sciences 236 Procedures for Peer Review 237

Contents

Writing Your Review of Peer Research 243 Summary 244 Key Terms 244 Chapter 11: Posters, Presentations, and Publishing

245

Presentations 245 Logistics 245 Preparing Your Presentation 247 Presenting Your Paper 249 Participating as a Discussant 251 Posters 252 Logistics 253 Preparing Your Poster 253 Preparing to Present Your Poster 256 Slip ’n’ Slide 257 Graphing Quantitative Results 259 Post-paper Possibilities 259 Presentation 260 Publication 263 Brief Remarks on Graduate Study in Political Science and International Relations 265 Summary 267 Glossary 269 References 281 Index 289

ix

T A B L ES A N D F I G U R ES

Tables   2.1  Inductive and Deductive Theorizing   2.2  US States by Presidential Vote, 2000–2004   2.3  Potential Explanations of Swinging States   3.1  Sample Literature Matrix   3.2  Bibliographic Management Software Options   4.1  Data Showing a Necessary Condition   4.2  Data Showing a Sufficient Condition   4.3  Data Showing a Necessary and Sufficient Condition   4.4  How Desirable Research Characteristics Are Achieved   4.5  Summary of Quantitative Tools   8.1  Examples of Variables, Names, and Labels  8.2 Realism’s Predictions   8.3  Power Transition Theory’s Predictions   8.4  Predicted Effects for EU Leadership    Hypothesis, Example 8.1

26 34 36 64 66 90 91 92 96 103 185 194 195 196

Figures   2.1  US States by Voting Consistency, 2000–2004   2.2  Typical Partisanship Scale for US Politics   8.1  A Nonlinear Relationship: Gross Domestic    Product per Capita and Birth Rate   8.2  A Log-Transformed Relationship 11.1  Graphical Presentation of Significant    and Insignificant Findings

x

32 49 188 189 260

N o t e s t o S t ud e n t s a n d I n s t ruc t o r s

To the Student This is not your typical textbook. This is the handbook I wish I had when I first learned to conduct empirical research. It’s also the book my students have asked for as they learned to conduct empirical research, and the original draft of this book came from handouts that I prepared at their request. It contains, in a nutshell, the nitty-gritty stuff of empirical paper writing that faculty know but don’t always have the time or ability to convey to students. How do I find a paper topic? Where do I find literature review sources, and how many do I need? How do I choose cases for my qualitative research design, and where can I find primary sources? What kinds of data are out there, how do I get them, and how do I get them into my stats program? How do I talk about my findings? How do I review a peer’s paper when I know nothing about his or her topic? How do I write an abstract? In short, it’s the questions you’ve always wanted to ask, but either didn’t feel comfortable asking, didn’t have a chance to ask, or didn’t realize you wanted to ask. Empirical Research and Writing: A Political Science Student’s Practical Guide (ERW) gives you all the benefits of a faculty member’s experience in doing empirical research, even if you’ve never written an empirical paper before. It’s practical and pragmatic, full of personto-person advice for specific tools, techniques, and strategies to help you succeed. You’ll find helpful tips from faculty (Insider Insights), other students (Peer Pointers), and Writing Center experts and subject matter specialists (Talking Tips) about all stages of the paper process. Empirical Research and Writing includes a number of examples and exercises that will help you prepare for writing your own paper. Preview activities are usually detailed examples of research design issues in the real world. Practice exercises give you a chance to develop your skills before you turn to the Progress activities, which help you move forward on your own research paper. Even if your professor doesn’t require you to do these activities, I strongly encourage you to at least think through them. They’re designed to help you be ready to do your own work. Rome wasn’t built in a day; research skills likewise take some time and experience to develop. Good papers come out of good preparation, and preparation takes practice.

xi

xii

Notes to Students and Instructors

Why do we need an entire book to tell us how to write a single paper? Knowledge is power. I believe in the value of understanding why you’re being asked to do something. As a faculty member, I’ve learned that when students understand the principles and objectives behind a particular task, they’re more likely to achieve those objectives and to retain the skills and knowledge gained. From your perspective as a student, understanding the overall goal helps you to make better decisions about your own work by giving you criteria for making those decisions. This usually leads to better grades because you’re able to make choices that are consistent with the same rationale the instructor is using (and expecting), and you can support your decisions in those terms. When you know the goals and the expectations for an assignment, you are more confident about your work, and the assignment seems a lot easier, too. For this particular assignment—an empirical research paper—the answer to a lot of questions you and your classmates have will be, “It depends.” Part of the reason it takes a book to tell you how to write a single paper is that this book isn’t about writing just your particular paper. It’s about writing your paper and all the other papers your classmates are writing, too. The underlying principles are the same for everyone, but the details differ quite a bit across specific applications. Advice that’s right for you may not be right for your classmates, and vice versa. So this book contains both general advice and guidance about expectations for this type of work, and a lot of more specific advice for particular situations. You’ll want to read and think carefully about which advice applies to you. When in doubt, consult your own instructor. After all, she or he will be grading your paper, not me!

How do I use this book? The first time you write an empirical paper or use this book, you should read the relevant chapter(s) as you progress through the assignment. It’s designed to deliver information “just in time,” right at the point when you’re likely to need it. Everyone will probably want to read all of Chapters 1 through 4. Use sticky notes or other tools to flag things that are especially relevant for your project. After that, your needs will diverge based on whether you are using qualitative or quantitative methods. For Chapters 5 through 8, only read the chapters that are relevant to your design, and read selectively in Chapter 9 based on your needs. Chapters 10 and 11 will also probably be relevant only in sections based on how your instructor has configured the course and the paper requirements. Don’t skip the footnotes; they often contain helpful additional explanations, examples, or citations. In addition, the website contains a ton of additional material—nearly half again as much as the book contains—on more specialized topics that couldn’t fit in the book. You’ll find “Web Extra” icons in the book indicating what additional material is available; go to http://study .sagepub.com/powner1e to access these sections.



Notes to Students and Instructors

After that initial read-through, use this book as a reference manual. The publishing team and I have put a lot of care and attention into the index to help you find things quickly, either when you need to review during that first empirical paper-writing experience or when you run across other issues or new problems in a later paper. We don’t expect you to memorize this book. It’s a how-to guide and reference manual. No one has all of this information in their heads for immediate consultation, not even me. (Empirical Research and Writing has an extensive bibliography for a reason!) Think of it as having your own personal professor to consult. This book might not have all of the answers, but it will give you a toolbox for asking questions and a set of principles to use to make your own research design decisions. One final note: This book is not intended as a substitute for advice from a qualified methodologist or specialist in your particular technique. I have deliberately omitted nuances, variants, technical jargon, and the like in the interest of length and readability. You should never, ever cite this book in a published paper. This is particularly true for MA and PhD students. Use the footnotes and references to go find the original sources, and cite those.

Why are you making us do this? And why do you have such high expectations? Can’t we just do it the easy way? Well, technically, I’m not making you do it. Your own professor is making you do this paper. My job in this book is to coach you through the process. One of the reasons that I chose to write a book to coach students through the empirical research process is because I have such high standards for my students. I believe that even though you are students, you are also junior researchers. I see no reason that undergraduates (and MA students) cannot produce quality, publication-worthy research. In fact, every year I see plenty of evidence to the contrary in the papers my own students produce and in the papers I see presented at the Midwest Political Science Association’s undergraduate research poster sessions. I believe that students are smart, motivated, creative people who—with a little guidance on the how-to’s and norms of this particular paper form—can conduct outstanding research. And every time I teach, my students show me that my beliefs are right. So, I wrote Empirical Research and Writing to give other students that same how-to knowledge and the same information about norms and conventions that I give my own students. As for why your instructor is making you write this kind of paper, I don’t know for certain, but I can tell you some of the reasons that I personally assign them. First, my course objectives usually explicitly include items that help students learn to think like social scientists. To me, the ability to think in general terms, locate and investigate patterns, and deploy concepts to explain the social world are crucial value-added components of an education in political science as opposed to an education in politics, public policy, policy analysis, or history.

xiii

xiv

Notes to Students and Instructors

Second, empirical research helps to produce individuals who are prepared to be knowledgeable, engaged citizens. Learning to evaluate claims using data, learning how and when to make comparisons, and knowing how to locate data that are comparable across time and space are important skills for anyone who wants to be an informed citizen, and they’re even more crucial for those who aspire to positions of public trust. You might never need to do “an empirical paper” in this particular format again, but you will engage in empirical research of various degrees of formality for the rest of your life. Finally, annual surveys of employers show that every year, the most indemand skill sets remain the same: critical thinking, strong writing, and quantitative literacy (in various orders, depending on the year). What primary skills do empirical research papers develop and show off? Critical thinking, strong writing, and for some, quantitative literacy. So asking students to do this kind of paper—which seems like an esoteric skill that you won’t need in the “real world” outside the ivory tower—is actually a very good job preparation exercise. At the end of the term, students have a compact, revised, and high-quality document that they can submit as a writing sample. I can’t think of any other kind of paper assignment that does such a good job of providing students the opportunity to hone and display their skills in these high-demand areas. As for why we demand that you produce such high-quality work instead of taking an easier route, well, we hope you see that this is for your own benefit. This book is designed to ease your route to an outstanding final product by answering many of the questions you may have along the way.

To the Instructor This is not a methodology book. It is not even a methodology reference book. It is a handbook that summarizes a good amount of scholarly wisdom and collective knowledge about research design and methodology. In the interests of clarity and accessibility, I have kept in-text citations to research methodology literature to the absolute minimum. Further Fun lists in the Instructor’s Manual direct students toward more complete and/or advanced treatments that I deem accessible to the average advanced undergraduate or beginning master’s student. Since faculty backgrounds and familiarity with various research techniques differ, the Instructor’s Manual includes some additional annotated references for your own reference. They may also be helpful for advanced students who may need to discuss these issues in their work, but students at a level where they need such guidance probably should be consulting with their own advisers and methodology professors. I welcome input and insights into the techniques presented here from anyone with expertise in those areas. I am not an expert in all of these tools, and my experience with some, particularly the qualitative techniques, is somewhat limited. Please feel free to send thoughts, suggestions, insights, or citations to me at [email protected].

T h e T h a n k Y o u ’ s

M

y students at American University’s School of International Service first encouraged me to make something more of my paper-writing handouts and then survived the first draft. Their name for the book was Professor Powner’s Purely Pragmatic ’Pirical Paper Planner, or P7 for short, but early reviewers thought that was too cheesy, so we went for the more staid title you actually see on the book. My students are bummed about the title change, but they’re happy this has turned into a real book. A number of my former students from American, Wooster, and Michigan also volunteered their time and their experiences to read and comment on later drafts. Their feedback was invaluable to this book and greatly enhanced its forthe-student focus. They also provided the many Peer Pointer and Insider Insight boxes found throughout the chapters. A mind-boggling number of faculty colleagues have been part of this book’s past. They have used drafts of this book, in whole or in part, in their classrooms; they have read it and given tons of amazing, thoughtful feedback; they have responded readily and thoughtfully to my frequent Facebook and email requests for examples and articles. There are so many, in fact, that I cannot even name them all. Those who read chapters or segments include Sarah Croco, Michelle Allendoerfer, Mary Durfee, Thomas Flores, Shanna Kirschner, Jane Lawrence, Allison Nau, Anca Turcu, Sam Snideman, Elizabeth Mann, Steven Campbell, Cecile Vigour, Manuel Teodoro, Erika Forsberg, Patrick Thaddeus Jackson, Jessica A. Grieser, Brittany Y. Davis, and Heather M. McIntosh, Unislawa Williams, Jean Gabriel Jolivet, Peter Brusoe, Trevor Thrall, and Daniel Masters. Chapter 10 benefited enormously from consultations with English, Language Arts, and Composition instructors from the elementary to collegiate levels. These include Janis Powner, Kristine Leonardo, Sarah Clark Hendess, Laura Larkin, and two others. The CQ Press team has been just as fantastic with this book as with my previous two. Charisse Kiino, Elise Frasier, and Sarah Calabi got the book in; I’m certain that editorial assistant Davia Grant did far more work on this



xv

xvi

The Thank-You’s

book than I know about. The production team—copyeditor Lana Arndt and production editor Bennie Clark Allen—bludgeoned my ugly mess of files through the process and turned it into a book. Allison Hughes put together a fantastic website to complement the book. All remaining errors are, of course, my own.

A b o u t t h e A u t h o r

Leanne C. Powner holds a PhD in political science from the University of Michigan, where she won just about every teaching award available. She was Juliana Wilson Thompson Visiting Assistant Professor at the College of Wooster and has also taught at American University (her undergraduate alma mater) and the Johns Hopkins University School of Advanced International Studies. Between the three schools, she taught research methods to BA, MA, and PhD students. She’s got a lot of practice explaining highly abstract stuff to undergraduates; her two previous books with CQ Press, Applying the Strategic Perspective (third and fourth edition), did for formal modeling what this book does for research design. She and her husband, an officer in the US Air Force (no, he’s not a pilot), live in southeastern Virginia with their two beagles. In her spare time (Spare time? What is that?), she enjoys cooking and crocheting. She regrets the paucity of Princess Bride and Monty Python references in this book but hopes that the pervasive puns and advanced alliteration will compensate.



xvii

T

From Research Topic to Research Question

1

he world is a systematic place. It’s full of patterns that make sense, patterns that we can discover and explain and use to predict things. Most of us accept this statement as a matter of course in the natural world. If we put a frying pan on a gas stove and light the burner, heat will transfer from the burner’s flame to the pan, in proportion to the size of and duration of exposure to the heat source, and then into any food we’ve placed in the pan. This happens on any day of the week, at any time of the day or year, under any weather circumstances, and for any person who performs this task. We can predict that if we increase the size of the flame or the duration of the pan’s exposure to the flame, the pan will become hotter and the food will cook faster. The social world—and the subset of it that we know as the political world—is also a systematic place with patterns and predictability. Most people greet this statement with at least a little bit of skepticism, if not outright incredulity. We’ve all heard some casual observer of American politics grumble about how “nothing that happens in Washington makes any sense.” Some politicians vote one way, others vote another way; some of them don’t vote the way we expected. This one causes a scandal, that one makes a horrible public gaffe, another one inexplicably loses a primary election. From day to day, there’s no telling what bit of nonsense is going to emerge next, at least according to that casual grumbler, and on the surface of things, that grumble seems to hold a lot of truth. If your source of that grumble is anything like mine, though, it’s older, sounds awfully like one of your grandparents, and is usually followed by something like “anymore” and a reference to “back in my day.” That statement—far from asserting the unpredictability of politics—is actually a profound claim in support of the predictability of the political world. Your grumbler holds beliefs and expectations formed over a long period of time and many observations: which way legislators should vote, based on various characteristics; how they should behave (i.e., what constitutes a gaffe or scandal); that incumbents are usually reelected. The fact that we can form expectations and make predictions about political actions or outcomes suggests that patterns do exist and that, subconsciously at least, we recognize those



1

2

Empirical Research and Writing

patterns. Humans are by nature pattern seekers; patterns help us make sense of the world around us. The patterns we find in the social world are not usually as strong or consistent as those in the natural world—certainly not as strong as the first and second laws of thermodynamics, which govern our frying pan example—but they definitely exist, and with a little bit of digging, we can find them.1 As a social science, the field of political science is committed to discovering and explaining these patterns, in the domestic politics of both the United States and other countries, and in politics between and across countries. Like our fellow social sciences, economics, social psychology, sociology, and anthropology, we are committed to making sense out of the world that we observe—the empirical world—by seeking patterns and explanations for general phenomena as well as for specific cases. Unlike the natural sciences, our patterns are generally contingent on other circumstances. Patterns of legislator behavior, for example, usually differ by country, though we can definitely find other patterns that extend across countries as well. Part of the challenge, and so part of the fascination and interestingness, of the social sciences lies in figuring out what those contingencies and mitigating circumstances are and in determining just how broadly some of our explanations and patterns stretch. This requires looking at many cases and many contexts; one observation does not make a pattern. Why Do Your challenge, if you’re reading this book, is to join social scienPoliticians Dislike Political Science? tists in our effort to make sense of the social or political world. You’ve been assigned a paper that asks you to identify a puzzle or pattern in the political world, to craft an explanation for that puzzle or pattern, and then to test that explanation against the evidence. In short, your goal here is to discover new knowledge: to figure out something that we as a society collectively didn’t know before. It’s a bit of a daunting idea, but at the end of the course, you’ll know something that no one knew before. A little intimidating, yes, but it’s also intriguing and enticing and fascinating and a bit exciting. In this chapter, we’ll briefly discuss political science as a social science and what the social sciences like in a “good” theory. We then examine how the paper you’re being asked to write for this class differs from other types of papers you may have written before. We’ll look at research topics, research questions, and characteristics of good—meaning feasible for an empirical paper like the one you’ve been assigned—research questions. We end with a discussion of how to create your own research questions, including sources for ideas and ways to phrase your question. You’ll even have a chance to practice your skills and prepare for your own paper with the chapter activities. We conclude with a list of terms introduced in this chapter. 1Most

modern social scientists reject the idea that we can find laws in the social world that are as strong as those in the natural world. The idea of scientific laws is most prominently associated with Carl Hempel’s famous (1966) book, Philosophy of Natural Science.

Chapter 1   From Research Topic to Research Question

Doing Social Science Social science values several things in theory and research. Four characteristics of theory that we particularly value are parsimony, predictiveness, falsifiability, and fertility. Parsimony means that the theory explains more using less. Measuring concepts is very difficult, and so the fewer variables or pieces of information that we need to use, the smaller our chances of getting findings that are actually caused by our errors. Because of this, we deem a theory using fewer pieces of information the “better” theory, and we reject ones that require more but produce the same results. Predictiveness means that the theory can help us to understand cases beyond those from which we derived it. Understanding what has occurred is useful, but the most useful theory also helps us to predict things that hadn’t yet happened at the time the theory was created, or that weren’t part of the theorizer’s original dataset. We also value falsifiability. If our ultimate goal is to understand how the world works, then we need to be able to reject and discard theories that do not explain the world well. To do this, we need to be able to identify what evidence would convince us that the theory is incorrect, even if we have not actually observed that evidence. If we cannot logically identify types of evidence that are inconsistent with a particular theory, we will never be able to determine if the theory is indeed valid. Finally, we value fertility in a theory. Theories that suggest other observable implications or other novel hypotheses are valuable because they prompt us to do further research and to build a cluster of knowledge around that theory and research question. In this way, our knowledge cumulates—builds on itself—rather than remaining as isolated relationships and discoveries. This cumulation helps us move forward as a field rather than persist in reinventing the wheel.2 We share with the natural sciences the value of replicability of research. We believe that science (the development of new knowledge) should proceed in a public manner, with data and analyses freely available to others, and with the sources of our conclusions clearly explained. Because of this, we place great emphasis on explaining in our research reports exactly how we did things: what we measured and how we measured it, the sources we used for those data, what specific analytical techniques and software we used, why we made certain choices and not others, etc. Ideally, another researcher should be able to get that same data himself or herself and reproduce your results.

Research Questions and This Course’s Research Project Research in political science can take many forms, but four types of research questions are particularly prominent. Normative questions ask how things 2This is part of the reason why we review the literature on our question in each research project we write.

3

4

Empirical Research and Writing

should be, or what policies are best. The key feature of a normative question is some element of evaluation against subjective criteria: the “should” or “better” component. What is the best way to address climate change? Well, that depends entirely on your (subjective) definition of “best.” Does it mean mitigating the overall cost of management? Does it mean spreading the cost of action equitably—or perhaps proportionately—across countries? Hypothetical (or theoretical) questions address issues of what if, or what might be. How might a resurgence of Zapatista guerrillas in southern Mexico affect the Mexican government’s ability to wage war against the drug cartels that dominate the north? This is an interesting question; it requires the researcher to find out how the Mexican government is dealing with the drug cartels now, and then to project what would occur if the Zapatista rebellion were to reassert itself. This might never happen in real life, but it is certainly worth thinking about. Factual/procedural questions, on the other hand, ask questions about the facts of the world. What is the United Nations’ policy on population control measures? How does the federal government regulate nuclear power stations? The answers to these questions are objectively verifiable facts of the kind that one might be able to find on Wikipedia or by asking a knowledgeable person. Empirical3 questions, finally, ask about how the world is, or how the world does work. Their concern is with actual events and phenomena, ones that have occurred or are occurring; they care about the whole of a type of event or phenomenon, rather than about specific instances. Why and how does the entry of a third-party candidate into a Congressional race cause either candidate to change his or her campaign strategy? Why do democracies never fight each other, even though they fight wars overall just as frequently as any other type of state? Normative, hypothetical, factual, and empirical questions are all valid forms of research, and they serve very important purposes in creating our overall understanding of how the world works. Procedural research often forms the basis for identifying empirical puzzles, and it can provide some of the raw data on which more systematic and generalized empirical research relies. General empirical explanations are necessary for reaching hypothetical conclusions; without understanding the usual effects of certain variables or factors, we cannot reach reasonable conclusions about the effects of changing those variables’ values in specific cases.4 Understanding the possible things that might happen—the hypotheticals—gives us grounds to evaluate our potential futures from a normative position. Those normative positions, then, raise other questions about whether the world actually is the way we want it to be (factual or empirical), and how we might get it there (hypothetical). This course’s research project asks you to conduct empirical research. Empirical research typically seeks a general story or explanation, one that 3The term empirical simply means that it is guided by scientific evidence and/or experiment, that it uses real-world evidence in examining its claims. 4Most hypothetical research does this intuitively, on the basis of deep background knowledge of the case and/or others like it, rather than relying on explicit empirical research.

Chapter 1   From Research Topic to Research Question

applies across cases and across time. It creates new knowledge about the way the world actually works. This is different than many other research papers you’ve written before for other classes. For those, you usually had a thesis, a central argument around which you marshaled evidence. For this paper, on the other hand, you’ll have a hypothesis, a statement of the relationship that you expect to exist. In your other papers, you definitely had evidence to support your thesis—or at least you did if you wanted to get an A! You might have discussed evidence that didn’t support your argument, but you did so mostly in an effort to show that it wasn’t very damaging to your claim. In empirical research, you’ll have evidence, but it will take different forms. You may very well find that your evidence does not support your hypothesis, and that’s totally okay. A finding of “no relationship” is important, especially if theory expected that we would find a relationship there. We don’t go into an empirical research project already knowing the answer to the question. We enter the project to answer the question, and sometimes the answer is not what we expected. What constitutes evidence is different in this kind of paper as well. In thesisbased papers, you could cite another author’s argument as evidence for your own, and you could get all of your evidence from other published works. In fact, you were usually encouraged to do this. Good thesis-based papers took existing arguments and evidence and marshaled them in an innovative manner, or introduced evidence that previously hadn’t been associated with this argument. In empirical research, on the other hand, you are contributing to our knowledge, not simply reanalyzing others’ information or collecting it in a new format or structure or for a new purpose. We do this by collecting raw data (information) and analyzing them using some highly specified, rather rigid techniques. The use of specified, rigid analysis techniques makes our research—our transformation of data into findings—replicable by other scholars. These techniques produce certain kinds of conclusions that emerge entirely from the data. The conclusions are not statements of opinion, nor are they particularly influenced by our own personal opinions. This makes the conclusions more credible to others. Even if others disagree with our opinions, if they agree that we collected the appropriate data and analyzed them correctly, then they must accept the conclusions that follow.5

From Research Topic to Research Question Most people begin a paper with a research topic—some single noun or expression that interests you. Your current answer to the question “So, what are you interested in?” is almost certainly a research topic: the Middle East,

5In their classic text Designing Social Inquiry, King, Keohane, and Verba describe the situation quite bluntly: “To put it most directly but quite indelicately, no one cares what we think—the scholarly community only cares what we can demonstrate” (1994, 15). Using collectively accepted empirical methods and reliable, public data makes our conclusions more convincing to others, even if they disagree with us. This is why knowledge of—and adherence to—professional norms of research design and analysis is so important.

5

6

Empirical Research and Writing

Congressional midterm elections, microcredit, arms control, state education policy, climate change, AIDS, the European Union, etc. The list is literally endless. Research topics, however, are not the most helpful things in helping you to shape a specific research paper. They lack any direction or any guidance for what to do: They simply state a topic. They don’t focus our attention onto any part of it, or help us to find an entry point into the material. Imagine that some (mildly sadistic and probably somewhat inebriated) relative asked you about your interests, and then challenged you by saying, “Oh yeah? Tell me everything you know about [insert your interest here].” Where would you start? What are the main parts? What is interesting about it? Simply having a topic of interest does not give us any help to answer that [insert your choice of adjective here] relative’s question. A research question differs crucially from a research topic in that its question format provides guidance. It defines a limited scope or boundary for the topic; it tells us what constitutes an end to the answer to your relative’s question. By being open-ended, though, it gives us guidance for what to do next: Answer it. In other words, it allows us to bite off a section of the topic that’s big enough to chew and swallow. It directs our attention to a manageable, defined, bounded problem and then tells us what constitutes a complete response.

Crafting a Research Question Crafting a good research question for your research project is a bit like Goldilocks looking for a bed in the home of the three bears. You need a question that is not too broad and not too narrow. A “just right” question has an answer that takes about as much space to answer as you have for your paper.6 The only way to get good at generating research questions, unfortunately, is to practice. We have no magic formula for creating a good research question. Below I present a list of some basic principles to help you formulate your own research questions.

Good questions . . . •• Usually draw on background knowledge. Start with something you know. Think about what you’ve studied in other classes that interested you or that made you wonder. What made you stick around after class to ask the professor more questions? •• Often identify or begin from empirical puzzles or anomalous outcomes. What things make you go “hmm”? Have you ever noticed two very similar initial situations that end up with radically different outcomes? Have 6In

graduate school, this gets turned on its head: A paper is as long as you need to answer your question properly. Faculty will usually give some expectations to help you establish an appropriately bounded research question, but answering your question completely and well is usually more important.

Chapter 1   From Research Topic to Research Question

you ever noticed something that just didn’t seem to make sense, or seen behaviors that don’t seem to be achieving the actors’ goals but they keep doing them anyway? These types of things usually make really good research questions. Your background knowledge can be useful here in helping you identify them. •• Often use “reporter questions”: who, what, when, where, why, how. Good empirical questions are actual questions—they end with a question mark and express some uncertainty—that go beyond asking about basic facts. They ask things such as, “Under what conditions?” “How much?” and “What relationship?” As we discussed above, questions are better than statements because they provide both direction and boundaries. •• May link to or draw on theories from the field. The field of political science is littered with theories that you can use as inspiration for your own arguments: pluralism, federalism, corporatism, rational choice, constructivism, judicial activism—again, the list is nearly endless.7 Much empirical research has ties to questions raised by these schools of thought, but not all. It is possible to theorize without relying on a major theory. Theorizing is simply imagining a logically plausible connection or path between a cause and an effect, and an -ism is not essential to this. -Isms can suggest potential paths and potential causes, but so can your imagination.8 •• Are usually about the outcomes rather than about the causes. A question that is narrow enough for a single paper is usually about explaining some outcome (“Y”) rather than about some independent variable or cause (“X”). Answering the question “What causes Y?” is a more specific and focused task than trying to answer the question “What are the effects of X?” “What are the effects of X” is an incredibly broad question—the list of potential dependent variables (outcomes) is again infinite.9

Good questions usually do not . . . •• Have one-sentence or factual answers. Unfortunately, this eliminates many who, what, and when questions. Good empirical research questions are not ones where you could get the answer from a publicly available source like Wikipedia, or by asking one knowledgeable person.

7International relations (IR) as a field is unlike almost all other fields of social science, including comparative politics, in that it has Grand Theories that purport to explain all behaviors that we observe. In IR, these are realism, liberalism, and, to a certain extent, constructivism. 8We’ll 9An

explore theory and theorizing in Chapter 2.

even better research question than “What causes Y?” is “How does X affect Y?” Notice how the second question narrows the scope of the paper to a focus on one particular independent (X) variable.

7

8

Empirical Research and Writing

•• Contain a lot of proper nouns. Proper nouns, by definition, refer to specific cases, events, things, or people.10 The point of empirical research is usually to look for general causes or patterns across many cases, and using proper nouns limits the set of cases. For example, the question “Why did the United States invade Iraq?” is too narrow for good empirical research. A better question might ask, “Why do states initiate wars?” Notice how the proper nouns became common ones: I determined what the specific case was an instance of—here, of a state initiating a war—and then asked my question about that. My new question’s answers will explain not only why the United States attacked Iraq, but also why other states have started other wars. •• Have a single correct answer. A good empirical research question is one where reasonable, educated people could disagree about the answer, at least before the analysis occurs. How does economic development affect the environment? One answer might be that development harms the natural environment because industrialization usually involves pollution, and economic growth usually involves increased numbers of personal automobiles (with their accompanying emissions) and increased consumption. Another answer might be that development is good for the environment, because as people’s basic needs are increasingly met—as their personal economic situation improves—they have time and energy and money to address things like the environment. Until we actually do the research, we have no way to know which of those equally plausible answers is actually correct. It’s also entirely possible that neither of them is correct, or that both are correct. •• Focus on specific measures or indicators of a concept. Good empirical questions have theory behind them, and theories are about concepts. “How does birth rate affect GDP?” is about specific measures of concepts. What the question is really asking is, “How does maternal health [or women’s status or something similar] affect development?” We prefer research questions about concepts because these types of questions allow us to generate additional observable implications—in other words, they’re more fertile (no pun intended) than questions about measures. If the real underlying research question here relates maternal health to development, then we should expect birth rate to predict other measures of development, and we should also expect other maternal health indicators such as fertility rate, access to contraception, female HIV/AIDS rate (and mother-to-child HIV/AIDS transmission rates), etc., to also predict various measures of development.

10In English, a proper noun is a word that begins with a capital letter—for example, English, Greek, and Latin are examples of specific languages.

Chapter 1   From Research Topic to Research Question

For your research paper for this class, you should think broadly and explore your interests. Think creatively, but also think—at least a little—in terms of doableness. Data on the effects of fair trade on individual rural Nicaraguan villages, on rural school boards’ attitudes toward federal educational reform efforts, or on the safe-sex practices of male sex workers in India are not generally going to be available to you from your current location. They’re great ideas for your bachelor’s or master’s thesis or dissertation, though, or any other project where you’ve got a longer time period and access to funding opportunities.

Peer Pointer

“One of the best pieces of advice I received from my adviser was that the best research comes from the best research questions. The part of the process that I expected to move most quickly—that is, creating a specific, meaningful research question that addressed a gap in the literature—proved to be the most challenging and probably the most important step.” —Elizabeth M., University of Michigan

Finding and Refining a Research Question Most people—experienced and beginning researchers alike—complain that finding a good research question is often the hardest part of the research process. Getting a research question alone is not particularly hard; the (more) difficult part is narrowing or broadening it as necessary to fit within the available time or paper space. The biggest piece of advice that I can provide for finding and refining a research question is that good questions are the result of lots of brainstorming and multiple rounds of revision and feedback. Don’t plan on using the first one that pops into your mind. Take the time to generate several options. Keep your need for a research question in the back of your head, and jot down ideas as they occur to you over the course of a week or so—when you’re reading, when you’re in other classes, when you hear something intriguing on the news, etc. The more potential starting points you have, the better your chances will be to find a question that will work.

Where Do Research Questions Come From? As I suggested above, research questions come from many places. The most common place to “find” a research question is in something else that you read—for another paper, for another class, or even that you read in the news. Most published (empirical) papers contain at least a brief review of the literature, where the author briefly summarizes existing knowledge about a question

9

10

Empirical Research and Writing

(or family of related questions) and places his or her own question into that scholarly context. In doing so, the author usually highlights several gaps or weaknesses in the literature. These “holes” are great places for other authors to situate their own work; bridging gaps between related ideas or findings is a very important contribution to the scholarly enterprise and to cumulating knowledge in a particular research program.11 So, a first place to look is in the literature review of an (empirical) article that really intrigued you. Sometimes the authors will identify these areas for future research explicitly, often in the conclusion of the article. A second and related place to look is in literature reviews. Some journals publish stand-alone article-length literature reviews that explore and elucidate the range of our knowledge on a particular research program. Two journals, the Annual Review of Political Science (ARPS) and International Studies Review (ISR), publish exclusively literature reviews (and in the case of ISR, book reviews as well).12 The Oxford Handbook of Political Science is a 10-volume collection of stand-alone literature reviews that compensate for infrequent updating by being incredibly comprehensive in their coverage. As Reading and Understanding Political Science (Powner 2007) discusses, literature review articles impose order (or at least try to, anyway) on the assorted literature on a question and illuminate areas of that research program where theory and/or empirics are particularly weak. The purpose of these articles, then, is twofold: to organize existing knowledge and to indicate where we need additional knowledge to enhance our understanding. An article-length literature review will almost certainly include calls for further research on particular unilluminated corners of the research program. Sometimes these are embedded in the body of the article; other times, they’re in a separate section at the end of the piece. 11A

research program is a set of interrelated research questions focusing on a fairly well-defined topic. One popular research program, to which many scholars have contributed, addresses the “democratic peace”—the empirical observation that no two democracies have ever fought a war with each other. Some scholars have questioned (or sought to demonstrate) that such a peace even exists. Others then tried to establish correlates or causes for the observed relationship (or nonrelationship, depending on your position in the debate). Yet another group of scholars has worked to arbitrate between the competing (and sometimes incompatible) sets of explanations to see which family of mechanisms (normative, cultural, leader-incentive-based, institutional, etc.) had more support. The list can go on, but these are some of the most common research questions associated with the democratic peace research program. 12Other journals, especially World Politics (WP) and International Organization (IO), increasingly publish stand-alone literature reviews. Much like the reviews in ARPS, these pieces are solicited: The editors explicitly ask top scholars in the field to write them, so that the review reflects, as much as possible, the cutting edge and/or state of the art in that research program. ISR publishes a much broader set of authors, and while the journal’s peer review process provides some measure of quality control, the standards of IO, ARPS, and WP are significantly higher than those of ISR. Unfortunately, these types of article-length reviews are relatively less common outside of journals of international affairs; ARPS appears to be the only source that regularly publishes them.

Chapter 1   From Research Topic to Research Question

Sometimes students feel that taking a research question idea from another article is somehow “cheating,” or even “stealing” someone else’s idea; they feel that using a question suggested by someone else is somehow being intellectually dishonest. Others feel that piggy-backing on someone else’s research question is “not making a contribution,” or is not making “enough” of a contribution to be worth their time and effort. Neither of these is even remotely true. If you are concerned about the ethics of using a research question first suggested somewhere else, simply cite the source in your work, and then include the source in the references as you would any other cited piece. Not having “invented” the idea itself does not undermine your work or otherwise decrease its value, nor is it academically dishonest. Talking Tips

To attribute a research question to another author or source, you might say something like this: •• “This paper responds to Smith and Jones’s (2001) call for further investigation of the relationship between chickens and eggs.” •• “As Smith and Jones (2001, 24) note, the absence of research on the chicken-egg relationship constitutes a significant gap in our understanding of this phenomenon. This paper addresses this gap by . . .”

If your concern is about the potential magnitude of the contribution, your concern is misplaced. Most pieces of research are not earth-shattering; the earth-shattering, in fact, are few and far between. Building on the work of others contributes to the literature more than most people otherwise think. By doing spin-off projects, you are demonstrating that the original work is fertile, and that is a highly desirable quality in a theory (see earlier in this chapter, on the qualities of theory). You are also showing that the original findings are replicable—another desirable quality in research—and that the original piece was done correctly. It also builds cumulative knowledge in a research program. Think about it: If every scholar pursued his or her own research program, working entirely independently of other scholars (even those in closely related research programs), we would duplicate a lot of work without gaining any additional knowledge. We would have islands of knowledge with no bridges or connections between them, and this is not how the sciences—social or otherwise—progress. Doing work suggested by others who also work in the field is not beneath you or otherwise denigrating or inappropriate; it is a valuable and absolutely essential part of making progress as a field.

11

12

Empirical Research and Writing

Even more, responding explicitly to another piece of scholarship can give you a leg up on the data collection phase of a project. The original author would have noted his or her data sources in the cited paper, meaning that you could easily re-create the dataset from the designated sources and focus your energy on collecting that one new variable you need to test your new part of the question. Sometimes, authors will even post the (quantitative) datasets that they used in the paper. These replication datasets may be on the authors’ own personal websites, or they may be on the journal’s common data web page.13 In a one-term project like the one you are probably writing for this course, having such a well-developed starting point is a big help. Besides coming from others’ suggestions, research questions can also come from your own personal experiences or observations: something you read in the newspaper, something you experienced while traveling abroad, or any of a host of other sources. These are valid sources of ideas, too. They may be a bit more difficult to articulate than ones where other scholars have already blazed a trail, or they may require a bit more original data collection on your part, but they are very appropriate sources of ideas. Just be sure to be in touch with your instructor(s). You will eventually need to ground your argument in existing literatures, and your instructor(s) can help by suggesting related, parallel, or supporting lines of argument for you to investigate.

How to Phrase a Research Question There is no one correct way to state a research question. Some ways are better than others, but which is better depends heavily on the question. As the Note to Students suggested, this is the part where the answer to many questions about this project will be, “It depends.” We can, however, make a few generalizations about ways of phrasing research questions that are frequently better than others. First, good research questions are open-minded. They don’t go in with a biased or predetermined notion of what the research should find. “How can we resolve the intractable problems of the Arab-Israeli conflict?” is not a good research question. First, it assumes “problems”—a word with a distinctly negative connotation that already sets a tone for the argument that would follow. Second, it declares—again by assumption—that these problems are intractable. If the problems are intractable, then by definition, they cannot be resolved.14 Likewise, “How much does conflict reduce trade between states?” presumes (before the author even completes the research) that the effect of conflict on trade is actually negative. Second, good empirical research questions allow for the possibility of several potential answers. “How does conflict affect trade?” is a better version of the question, because conflict could possibly have any of several relationships 13We

revisit replication datasets, their organization and use, and their ethics in Chapter 7.

14This

is also not a very good empirical research question; it’s closer to a hypothetical one.

Chapter 1   From Research Topic to Research Question

13

(positive, negative, no relationship) with the outcome of interest, trade. “Why do Americans vote for third-party candidates?” also could have multiple potential answers. How or why questions often work better here than “does” questions—“Does conflict affect trade?” has only two possible answers, yes or no, and we could easily broaden it to provide information about the direction of that relationship without adding additional work to the research. Finally, most people generally find that narrowing a question is easier than broadening it. When in doubt, I suggest you phrase your question in the broadest way possible. If necessary, you can then narrow down the scope of the investigation—even if the theory and research question themselves are more expansive—to something that is doable for a particular project. Empirical research is typically about producing generalizable knowledge, that is to say, making claims that have support (or at least potential applicability) beyond any one time and space, and so one can usually make a case for testing on a sample of convenience. For example, you may have an argument about the effect of disasters on birth rates. We have reliable data for both of these things for virtually all countries of the world, but only for the mid-1960s to the present. We have little reason to believe that people in this era have a fundamentally different response to disasters than people in previous periods,15 so testing simply on the sample of convenience does not impede the development of knowledge. On a related note, however, one main caveat about the breadth of empirical research questions requires discussion here. We cannot limit the scope of a research question to, say, one election or one geographic region or one time period, simply because that one case personally interests us. An empirical research question asks about a general phenomenon. If we artificially limit the range of cases we consider by imposing some type of arbitrary geographic (or other) criterion, we risk biasing our pool of cases and getting spurious (and perhaps more dangerously, incorrect) findings. We should theorize about—and plan to test our theories on—the entire pool of cases where the phenomenon of interest occurs. Of course, using the full population of cases is not always doable in practice. The biggest reason is data availability. US politics is one of Examples of the few areas where most of the data we could want are available, but Bounded Theories even there, some significant gaps exist. For example, until 1940, US and Bounded census forms did not ask any questions other than name, age, gender, Investigations and a crude coding of race. We thus lack sufficient data to study the effects of literacy and income on women and minority voters in the interwar period; that’s unfortunate, because that window—between when women won the right to vote and when voting access became much easier for women but contested for minorities—could provide valuable insights about politics at the intersection of class, race, and gender. As another example, no reliable and 15Survival rates for the infants might differ from 1960 onward, but this does not affect the raw birth rate (live births per 1,000 women). So inferences made on the convenience sample are unlikely to be a function of improvements in health care or other post–World War II phenomena.

14

Empirical Research and Writing

consistent national gross domestic product (GDP) data exist before the early 1940s (for the United States or anywhere else). Few important economic variables such as GDP and trade penetration are available at the level of US states. These are circumstances where the data simply don’t exist and are nearly impossible to construct from available information. In these situations, we must find other ways to investigate the issues of interest. For the race, class, and gender question, for example, we might construct a qualitative investigation using a carefully selected set of oral history interviews from the Library of Congress and the National Archives and diaries written by women of that era as our primary data sources.16 Sometimes, data availability is limited by the deliberate choice of the data-collecting agencies, which often choose to focus on cases that are “important” or “interesting” for their agenda. The prevalence of the AIDS disease in Africa has led to the collection of much more detailed (and accurate) statistics for that region than for anywhere else. Only the EuroBarometer and Latinobarometro surveys ask a battery of questions about individual attitudes toward economic integration that are both cross-nationally comparable and asked in a consistent time series; of those, only the Latinobarometro series focuses on issues relevant to developing countries. These types of limitations are artificial and are products of choices by the data-collecting agencies, and using just these cases without acknowledgement of their unusualness risks very serious problems. Where relevant and where possible, we should attempt to expand the pool of data to include other relevant cases. These cases are “interesting” for a reason that probably correlates with the same thing we’re interested in, and so using only “interesting” cases—even if they’re the only ones for which data are readily available—leads to some possible problems.17 Other times, some phenomena only happen in certain geographic regions. The only countries undergoing transitions from communist economies to purely market-based ones are in central and eastern Europe;18 only southeastern US states have regular experiences with hurricanes. These types of limitations are part of the natural world—they are constraints on the scope of comparable phenomena or cases that arise independently of any action or intervention by the investigator. Regardless of the source of the constraints,

16We

know that widespread differences in literacy rates existed at this time, so using diaries alone would bias our sample. The use of oral history interviews as a complementary data source helps to mitigate the effects of literacy. 17The classic exposition of these types of “selection bias” issues is Geddes (2003), especially Chapter

3 (“How the Cases You Choose Affect the Results You Get”). We return to these issues in Chapters 5, 6, and 7. 18Afro-socialism

did not, for example, involve extensive efforts at heavy industrialization and regionwide central planning in the same manner that Soviet-governed economies did. Cuba and China are, arguably, still mostly communist economies, or at least are not making explicit transitions to capitalist market structures, so neither of these sets of states are relevant to the research question.

Chapter 1   From Research Topic to Research Question

though, the pool of available data limits the set of cases about which we can draw conclusions. In both cases, either natural limitations or imposed ones, the researcher must be aware of the limitations that data availability can cause and be careful to express his or her findings in a way that reflects these limits on inference. In short, any boundary you place on an empirical investigation, geographic, chronologic, or otherwise, has to be theoretically defensible, or at a minimum (in the case of data availability or manageability constraints) empirically justifiable. If you have one of these relatively rare bounded questions, or are concerned that you may need to limit your study to a range narrower than that of your theory, you should plan to consult with your instructor often. Preview 1.1 Sample Research Questions Previous students have examined the questions below in their papers. I explain here why these questions were good and suggest variants of them that would not have worked well.

A. How does a midterm recession affect presidential economic policy? This question worked really well. The student wanted to know whether presiding over a recession that starts during a term makes a president become more centrist or more extreme in his preferred economic policy. The student studied all post–World War II US presidents who experienced a marked decline in economic growth. By looking at how frequently the presidents used specific key words in their annual State of the Union speeches before and after the recessions, the student could identify trends in economic policy preferences that were distinct from enacted policies (some requiring central bank or Congressional consent) and economic outcomes (which depend on many nonpolicy forces). Studying a single president and a single recession would not give a representative sample; we can’t generalize from that.

B. How do disasters affect the birth rate? This question is mostly viable, but as it’s currently phrased, it’s rather broad and the outcome of interest, birth rate, is a specific indicator instead of a concept. What larger idea does the birth rate represent here—population trends? Social response to external threat? What kind of disasters: only natural ones, or do man-made ones (like wars) count too? Answering that question depends fundamentally on your theory—the argument you have for why you expect disasters to influence birth rates. If you expect disasters

(Continued)

15

16

Empirical Research and Writing

(Continued) to influence birth rates because citizens realize the effect of the disaster on the population’s age distribution, and so a bulge in the birth rate is intended to repopulate the country, then yes, including man-made disasters in the research probably makes sense. If you have a different type of argument to link the cause (independent) and outcome (dependent) variables—perhaps that it’s a religious or faith-motivated response against the injustice of the (deity-inflicted) disaster—then including man-made disasters does not make sense on a theoretical basis. Adding a geographic or chronological boundary to this question likewise does not make sense from a theoretically justifiable point of view.

C. Why does the political left become social democratic in some Latin American countries and populist in others? This is a viable research question. The geographic limitation here is acceptable because the phenomenon of interest—populist political movements—only occurs in Latin America, at least in a form that is comparable to the other cases. Using only Latin American cases removes the need to control for (make an adjustment to accommodate) potential background variables that may vary across cases but affect their values on the dependent variable—colonial history, geographic region, previous experience with both popular and authoritarian rule, etc. Practice 1.1 Evaluating Research Questions Consider each of the questions below. Determine if each is clearly specified enough to be the basis for a research project (i.e., a single paper, about 20–25 pages). If it is not, identify what parts are too broad, or too narrow, and find a way to rephrase it to make the question viable for a single research project. A. Why do presidents spend so much time talking about foreign policy during the campaign when only 1–2% of voters base their votes on it?

B. How do countries’ economic interests affect their positions on the conflict in Syria?

Chapter 1   From Research Topic to Research Question

C. Why did the British election of 2010 produce a Conservative minority government?

D. Does international law cause states to change their behavior?

Progress 1.1 Brainstorming Research Questions Use this activity to begin brainstorming some potential research questions for your research project. You should plan to generate at least two or three ideas, in case one turns out not to be viable for some reason or another. A. On a clean sheet of paper, take 2 to 3 minutes to make a list of any political science or international relations terms that come to mind. Use a timer or other device (one song on the radio or your MP3 player, for example) to manage the time. Just write anything that comes to mind. B. Take another 2 to 3 minutes to review your list. Using a pencil, put an x by anything that is less attractive to you. Put a check mark by anything that is particularly interesting to you. Put a question mark by any term that is a proper noun (starts with a capital letter). Again, manage the time. Don’t spend too long on this. C. Get out a second sheet of paper. Make a second pass through the list, and this time, write (on the new page) any pairs of checkmarked words that seem to fit together. If you suspect they appear to have a relationship—for example, household wealth and educational outcomes—put an arrow between them showing the direction of the suspected relationship; don’t worry about whether the relationship increases or decreases for now. Try to avoid the words marked with a question mark. You have about 5 or 6 minutes for this stage—one longer track or two shorter (radio version) songs. D. Return to your new list and consider the pairs that you’ve associated. Can you formulate them into research questions? Play around with them, reversing the order in which the terms appear, perhaps adding another term that you forgot in the initial brainstorm, or by adding some more specific terms or ideas to them. Again, you probably shouldn’t spend more than 5 or 6 minutes here. Try to write two to three potentially viable research questions. Consider how and where you might need to narrow them to make them doable, and where you might need to broaden them to think in terms of generalizable explanations.

17

18

Empirical Research and Writing

Writing Your Paper As we continue through the book, you’ll find sections like this at the end of most chapters. They are designed to help you with the actual writing of the paper, section by section. Most researchers write the final draft of their paper in a short period around the completion of the actual research. This is normal, even for faculty.19 To write the paper successfully at the end, though, you will need two important documents: a research notebook and a Leftovers file. Your research notebook can be any old notebook or even a computer file. I personally prefer using a paper notebook because I can work simultaneously on the paper and on my computer screen more easily. This may be a generational handicap, so your mileage may vary. The voice of experience, however, strongly implores you not to settle for scribbles on scraps of paper. They will get lost. Use a notebook or something else with pages that are all attached together. Your research notebook is the place to scribble answers to any Progress activities that your instructor doesn’t ask you to turn in. It’s also a place to jot your thoughts, questions, and realizations as you work your way through the paper. Tendencies toward procrastination aside, I encourage you to begin sketching out sections of your paper in writing as you encounter these sections in the book. Like any piece of good writing, a quality research paper goes through a number of drafts. Very little of this initial writing will appear in your final draft, but the writing itself will be invaluable for several reasons. First, you can ensure that you actually know all the things you need to know to write the section—and if you don’t know them, it will identify exactly what you don’t know so you can reread, do research, or otherwise figure out that information before the paper deadline. It forces you to articulate everything explicitly rather than letting it slide past when you can’t easily find words. Second, it provides you a record of what you know and what you were thinking at various stages of your research project. Don’t rely on your memory for keeping track of everything. You won’t remember, and you’ll find yourself missing key things when you go to write. Making a note now takes a lot less time and effort than trying to reverse-engineer your decisions when you’re under the pressure of a deadline. Third, and perhaps most important, you can get the ideas started so they can marinate. Your argument will evolve, especially as you delve deeper into the project. Do not expect that your earliest versions will actually end up in the paper; most will end up in your Leftovers file. This is where we come to our second necessary document: the Leftovers file. Much like a good spaghetti sauce, some ideas improve with a bit of time chilling in another container. The Leftovers file is where you stick all those bits 19One of the reasons we apply to and attend conferences is so that we have deadlines for writing our papers. And even then, most of us don’t seriously start writing until about a month before, which allows about 2 weeks to write before we have to send off the paper. (This is a dirty little professional secret; don’t tell your professor that I told you.)

Chapter 1   From Research Topic to Research Question

of ideas from your early drafts that don’t make the final cut. Sometimes, this is because your ideas have changed; other times, you’ve found better ways to express them. Occasionally, you get a really cool idea for something else that just doesn’t go into this particular paper. But the key thing about a Leftovers file is that it’s a single place to dump this stuff, where you can find it again later when you need it. For me, this is usually a single running Word document, but again, your mileage may vary.

Summary In this chapter, we explored the idea of the social world as a collection of patterned phenomena, and how the social sciences attempt to make sense of those patterns. We value the characteristics of parsimony, predictiveness, falsifiability, fertility, and replicability in research. Research questions are one of four types—normative, hypothetical, factual/procedural, or empirical— depending on the goal or purpose of the investigation. Empirical research questions deal with the world “as it is,” seeking general explanations for patterns of outcomes or classes of phenomena. A good research question is one whose answer takes as much space as your paper has length.

Key Terms •• •• •• •• •• ••

Parsimony Predictiveness Falsifiability Fertility Replicability Normative

•• •• •• •• ••

Hypothetical Factual/procedural Empirical Research topic Research question

19

I

From Research Question to Theory to Hypothesis

2

remember an incident from my own undergraduate years that caused me some confusion both when it occurred and for some time thereafter. I took a class with a professor I admired, a well-spoken older man who seemingly knew everything. I don’t remember what we were discussing, but one day, a particular guy with a less than stellar reputation (in my mind at least) raised his hand in the middle of class and asked, “Professor, where do theories come from?” The professor froze. I was sitting to the side of the room, and I could see his face clearly. It had the deer-in-the-headlights look of panic for a moment, and then it was gone. The professor turned to the questioner and said firmly, “Books. Theories come from books.” And then he led the class discussion back onto the topic. Now, random questions from this particular guy—and professorial efforts to lead the discussion elsewhere after such questions—were not at all uncommon. What was uncommon, though, was the professor’s reaction. I had never seen this professor hesitate, not even for the heartbeat that this had lasted. In fact, this was the faculty member who encouraged me to ask the most questions and who always had the answer, and if you haven’t guessed, I was one of those students who always had a ton of questions. How could such a simple question throw him for a loop like that? He regularly handled all my off-thewall inquiries with aplomb; to think that he didn’t know the answer was just mind-boggling. I think I eventually attributed the professor’s response to being tired that day or something like that, but it nagged me for quite a while. What I understand now, from the other side of the lectern, is that my classmate’s question is the intellectual equivalent of that parentally dreaded question, “Daddy, where do babies come from?” We faculty know what theories are and where they come from, and we have them in our own research, and we can identify them when we see them, and we can deride others for not having them in their research . . . but most of us can’t really articulate well what a theory actually is or how we get them. Many methods textbooks fumble awkwardly around the topic, flailing for a few brief pages before abandoning the attempt. Others skip discussing theory altogether except for noting its necessity; they deem it the purview of “substantive” classes, not a methods course such as theirs.



21

22

Empirical Research and Writing

This lack of theory is a serious problem for a methods textbook, because theory motivates the entire research project. Trying to teach research methods without the role of theory is like trying to teach someone to drive in a car with no brakes: By definition, the entire enterprise will rapidly get out of control. A novice driver with no brakes will either go nowhere fast, or will go fast until he or she comes to a fiery crashing end. A novice researcher who lacks a theory, or lacks an understanding of theory, will have many research decisions to make and no basis for making them. With no theory, many students freeze up and find that their research goes practically nowhere; students who make decisions without theory usually crash and burn. This chapter is dedicated to theories: what they are, how to get them, and what to do with them once you have them. By taking the time to develop a well-articulated theory, you will position yourself for research success by giving yourself a basis for making your future research design decisions. Even though it’s theory, theory is, in this case, entirely practical. Chapter 1 introduced different types of research and research questions in political science, and you practiced evaluating and developing research questions of your own. In Chapter 2, we take the next logical step: helping you to formulate answers to those questions, which is to say, we enter the world of theory. We begin by discussing what theory is and where it comes from. The second section helps you to develop your own theory and introduces tools to help you with this. The third section builds the bridge between theories and hypotheses, and it gives you opportunities to begin moving from your theory to your hypotheses.

What Is a Theory? A classic text on research design defines a social scientific theory as “a reasoned and precise speculation about the answer to a research question, including a statement about why the proposed answer is correct” (King, Keohane, and Verba 1994, 19). In other words, it proposes an answer to the research question and tells why that answer is expected by specifying a mechanism, some chain of events or reactions or changes, that connects the independent (cause) variable(s) with the dependent (outcome) variable. Theories are simplifications of reality. The purpose of a theory is not to give a complete detailed retelling of an event or phenomenon, or even a “complete” explanation. Theories direct our attention to specific elements of a causal story. They isolate one particular outcome or part of the outcome, and then they explain that outcome by highlighting some parts of the story that the author feels are important while downplaying other details that are less important to the author’s story. Understanding this is important: No theory is trying to explain every single detail and every facet of an event or phenomenon. Each theory bites off a small bit of the larger story—isolated from among other events and related phenomena by scope conditions—and uses assumptions to simplify the set of moving parts down to something manageable. From this

Chapter 2   From Research Question to Theory to Hypothesis

more manageable set of parts, the author makes a prediction about how the moving parts cause the outcome. We’ll begin this chapter by considering all these parts of a theory and then discuss where theories come from, at least in a general sense.

The Parts of a Theory A typical theory has four parts: the expectation or prediction, the causal mechanism, the assumptions, and scope conditions. The first part of a theory is the expectation or prediction. This is the part of the theory statement that says what the author expects to happen. The prediction is the short form of the answer to the research question, the outcome of the interaction. This is the easiest part of the theory, the one you most immediately think of when you try to explain some event. The second part of a theory is, to many social science practitioners, the most important: the causal mechanism. It provides a specific chain of steps, series of links, or other specific accounting of how changes in the cause variable affect the outcome variable. Without a causal mechanism, a theory is nothing more than a statement of expectations, a vague prediction. Some scholars believe that the causal mechanism itself is the heart of the theory. The causal mechanism tells why something happened, which allows us to go from predicting or expecting an outcome to explaining it. Since explanation is the goal of social science research, this why factor assumes a crucial position in our research. The specific steps of the causal mechanism allow us to differentiate between theories that predict the same outcome, as we discuss below. The third component of a theory is its assumptions. All theories make assumptions. Assumptions are claims or beliefs, typically implicit ones, about how the world operates. Some common assumptions in political science are, for example, that elected officials seek to stay in office, that individuals are equally motivated to vote, or that similarly situated firms benefit identically from policies. Other seemingly simple assumptions are actually more complex, such as the widespread assumption that individuals are rational actors who maximize their personal utility. This is really three assumptions in one: The actors are individuals, the individuals are rational, and their objective is utility maximization. But many more assumptions are quite insidious, such as the assumption that gravity operates. After all, theories about which side will win a war rely on armies firing projectiles that will injure or kill the other side, and that’s four instances of the necessity of gravity right there: armies, firing things that will hit the other side and will have enough force to injure or kill. One of the hardest steps in theorizing is articulating the set of assumptions you have about your event, outcome, or phenomenon of interest that actually matter for your theory. The assumption about gravity, for example, is a necessary assumption for most theories of war, but natural science gives us reason to believe that gravity holds in all places on Earth. Since I don’t know of any current research on extraplanetary political science, the assumption of continued

23

24

Empirical Research and Writing

and constant gravity seems safe to make without stating it explicitly. One of the reasons scholars do not explicitly articulate all of their assumptions is that some, such as the persistence-of-gravity claim, are true for all hypotheses and sets of claims; stating them for every hypothesis or even every research question would be painfully redundant. Other times, scholars make several assumptions at once through a single statement. Invoking some terms or concepts from rational choice theory, for example, tends to imply that the author is accepting all of the assumptions of rational choice theory in her paper. Many of these statements and their corresponding assumptions are rather abstruse; they emerge from extended debates in the scholarly literature about various -isms.1 As a beginning researcher, you’re not expected to be familiar with all of these, but you should consider a variety of assumptions and determine which, if any, are relevant to your project. The specific set of assumptions you are making varies widely across subfields and between research questions; we’ll consider some options below. The fourth and final component of a theory is the theory’s scope conditions. Scope conditions identify the theory’s domain—the set of cases over which the theory is expected to operate. This term is somewhat less familiar to many scholars of American politics, for whom the domain is usually implicit: all (American) voters, all Congresses (and Congress members), all bureaucratic decisions, etc. By constraining their analysis to American politics, and/or theorizing about behaviors that are fairly unique to the US context (e.g., Supreme Court decisions, impeachment, US policy processes and laws), scholars have already defined the domain. The same is true for studies of politics or unique political processes within other countries or units as well. Adopting resolutions at the UN General Assembly, for example, provides both a temporal scope (time frame) and a substantive scope, as do studies of French legislative behavior. 1One set of assumptions that we all make relates to ontology, or beliefs about how the world works. Researchers working in the positivist empirical tradition (the one described in this book) generally believe that the social world is composed of systematic and predictable components, and unsystematic and unpredictable (random or stochastic) components. The social world also exists as a distinct entity that we can study, independent and outside of ourselves. The entire methodology of positive empirical research revolves around these often unstated assumptions. Other scholars outside of the positivist empirical tradition hold different beliefs, which are often associated with -isms in the same way that positivism has its own assumptions. These alternate assumptions include beliefs that all human institutions are mutually constituted by the observers rather than existing outside the observer, and/or that the act of observing a phenomenon changes it. These scholars have different norms and desired characteristics for research. Empirical Research and Writing (ERW) assumes that you are working in the positivist empirical tradition, at least for the purpose of this particular assignment and/or course. If you have strong intellectual ties to a nonpositivist tradition—usually indicated by knowing what the word ontology means and/or knowing what you assume about the world—then you should plan to communicate regularly with your instructor during the research design process to ensure that you are working within a framework that your instructor can or will evaluate.

Chapter 2   From Research Question to Theory to Hypothesis

For scholars of cross-national comparative politics and international politics, on the other hand, domains are often not so clear-cut. Does a theory of revolution apply to all revolutions? Revolutions in countries with no history of democracy? With colonial history? As explanations for anticolonial revolutions (wars of national independence)? In richer countries? In countries with no significant social or ethnic cleavages? Does it explain massive social revolutions as well as more mundane civil revolutions? Does it explain bloodless ones as well as bloody ones? Do coups count as revolutions? This is when we return to the question from Chapter 1, “What is this a case of?” and turn it on its head. This is a case of revolution, yes, but what kind of revolution? Good scope conditions identify a set of general conditions that could exist in multiple times and places, rather than identifying a specific time and place (Mahoney and Goertz 2004, 660). Ideally, one should think of scope as having a temporal dimension—a time frame or range—and a spatial dimension that defines the units of analysis. When you can identify the period and the cases within that period, you’re on track to have developed solid scope conditions.

Insider Insight

Don’t worry if you can’t articulate all of your assumptions or scope conditions right now. Many will surface as you begin discussing your theory, research design, and evidence with classmates and colleagues. Just keep track of them in your research notebook as they surface so you have them for later.

To summarize, a complete theory of an event or phenomenon requires all four components: a prediction or expectation, a causal mechanism, clearly articulated assumptions, and scope conditions. In other words, we must know what set of events we are explaining (scope), what we expect to happen (prediction), under what conditions (assumptions), and why (mechanism).

Where Theories Come From So where do theories come from? I’ll give you a hint: The answer is not “books,” no matter what my professor said. Or at least, it’s not only books. We generate theories—potential answers to our research questions, our hunches about what the relationship looks like—from a wide range of sources and through a range of different methods. These methods generally take one of two forms: inductive theorizing, or deductive theorizing. In inductive theorizing, the potential explanation emerges from a study of specific cases, and then it is generalized outward from there. We might develop a theory of Supreme Court

25

26

Empirical Research and Writing

behavior, for example, by studying a handful of Court decisions and drawing conclusions from the set of data we examined. Likewise, we might theorize about the outbreak of major war based on an examination of events around World War I and World War II, or about the role of ethnic cleavages in developmental politics by studying Indonesia and South Africa. In all these cases, we use the specific cases to help develop general explanations. In deductive theorizing, on the other hand, potential explanations emerge from abstract analysis outside of the context of any specific case. We step back from the data and think, in general, about ways in which something could occur or what could cause a certain outcome. Essentially, we try to tell a story about how or why an outcome happened without using any proper nouns. Thinking about the role of gatekeepers in a legislature leads to general predictions that we can test Why Theorizing against the role of, say, committee chairmen in Congress or party leadby Induction Is ership in most European parliaments. Considering the process by Problematic which US presidents incorporate public opinion into decision making likewise produces testable hypotheses about when public opinion is most likely to influence policy. Deductive theorizing produces broad, generalized hypotheses that apply to large numbers of cases or contexts. In short, while deductive and inductive theorizing conduct the same tasks in the research process, they do them in a very different order. The order in which we do these tasks matters a lot to the type of research output we produce. Social scientists follow natural scientists in believing that the ideal research design is the experiment, in which one devises a treatment or intervention based on some theory, applies the treatment to the case, collects data on the result, and sees if the data patterns match the hypothesis. The idea and expectations come before the data. Crucially, this implies that the theory is independent of the data—that the theory wasn’t crafted to “explain” data we already had. Think of it this way: If we use the event sequence A  B  C to form predictions that A leads to B and B leads to C, and then we “test” our theory on the same data, we will inevitably find that the theory is supported. This is bad science. Good science establishes a theory and expectations before conducting the research and then does the research according to well-established procedures and

Table 2.1  Inductive and Deductive Theorizing Research Process With Inductive Theorizing

Research Process With Deductive Theorizing

1. Get data

1. Develop theory

2. Poke around in it, looking for patterns

2. Project theory to cases (hypothesize)

3. Develop theory

3. Get data

4. Project theory to (additional) cases

4. Poke around in it, looking for patterns

Chapter 2   From Research Question to Theory to Hypothesis

accepted norms of practice (and publishes this information). Only then do we obtain data to analyze—again using well-accepted and publicly shared techniques—and draw conclusions. Theorizing is not a hard task in and of itself.2 In fact, you theorize every day, as part of your ordinary daily behavior. You do it without even thinking about the fact that you’re doing it. For instance, consider your route across campus to your noon class. Your research question is something like, “what is the fastest way to get to class from here?” One potential answer is, “by cutting straight through the student union, since that’s the shortest route.” Another potential answer is, “by going around the student union rather than through it, because that will avoid fighting against the crowd headed into the cafeteria for lunch.” Congratulations! You just constructed two theories about getting around on campus. You provided an answer to the question and a rationale—a causal mechanism—for it. If you formed these expectations on the first day of class based on your general knowledge of campus and its traffic patterns, you engaged in deductive theorizing. If you spent the first month or so trying different routes and comparing the timing, then you engaged in inductive theorizing. In either case, you would want to test your theory by gathering new data—from yourself and possibly others—and comparing your predictions to your expectations. To finish the theory, let’s consider the two remaining components: assumptions and scope conditions. In this particular theory, your assumptions included that your schedule would remain the same and your starting and ending locations would not change during the semester. If either of those changed, your theory would no longer apply. Likewise, this theory’s scope conditions mean that it is relevant only to your schedule this semester at your particular institution. The “hunches” or ideas that underlie much deductive theorizing come from a multitude of sources or approaches. We often draw heavily on background knowledge of one or a few cases to identify a mechanism, and then we see if that mechanism generalizes across cases. Sometimes, related literatures are a help, even if they’re only tangentially related.3 Mechanisms can also come 2In this book, I assume that you are taking a deductive approach to your research puzzle, whether by your own choice or inclination or because your instructor has asked you to. The rest of the guidance in this chapter and beyond leads you through the practicalities of developing a deductive theory and hypotheses, designing a research project to test your hypotheses, collecting and analyzing the data, and discussing the results. If you feel your project is more of an inductive one, you should plan to consult with your instructor as soon as possible to determine whether you can pursue your current line of inquiry in the context of your course’s assignment requirements, or whether you should reshape it into one that can be more easily pursued through a deductive approach. 3I once used theories of cognitive dissonance (political psychology, but familiar to me from studies of US foreign policy), explanations of interest group mobilization (from comparative and American politics), and Benedict Anderson’s (1983) concept of “imagined communities” to explain the timing of the French revolution. Seriously. In a PhD-level French history course. It was one of the coolest (and funnest) papers I ever wrote. The result? The chair of the History Department tried to get me to switch PhD programs. Lesson? Intriguing ideas can come from the most unexpected sources.

27

28

Empirical Research and Writing

from common concepts in the social sciences, such as collective action problems, positive and negative incentives, principal-agent problems, transaction costs, and path dependency. Researchers working in international relations can also draw on Grand Theory—realism, neoliberalism, Marxism, constructivism, insert-your-own-ism—as a source for potential answers. Many researchers employ tools such as formal theory and its most popular variant, game theory, to develop deductive theories of events.4 We can also use our own common sense or a suggestion from another scholar as a source for theory. At its heart, a theory is nothing more than a potential answer to your research question: a statement of the relationship you expect to find and a “because” clause to give the basic reasoning. It does not need to be something novel or something that you personally dreamed up or discovered. All theory is based on other research, so don’t be afraid to look around you for theory inspirations. Often, anything you can phrase as an answer to a why or how question can form the basis of a theory.

“

Quips and Quotes

“Brilliant insights can contribute to understanding by yielding interesting new hypotheses, but brilliance is not a method of empirical research.” —King, Keohane, and Verba 1994, 16

To be clear, theorizing is not about having brilliant insights. Theories don’t have to be earth-shattering or make seminal contributions to the field. They simply have to explain some phenomenon, and often the most obvious or commonsense explanation is the correct one. Don’t worry about trying to do something new that no one has done before. To be honest, most theory in political science is of the “well, yeah,” or “no, duh” level of brilliance: It’s an obvious, common-sense answer to a question. But all theories, even commonsense ones, need to be stated and tested before we can accept them. Take a question that interests you, think about an answer, and go from there.

From Question to Theory A proposed answer to a research question is a theory, or at least the bones of one. But as we discussed in Chapter 1, good research questions often have more than one possible answer. The process of developing a theory for your question is one of identifying numerous potential explanations and then sifting through 4For a good introduction to formal theory, see Kellstedt and Whitten (2009, 31–38), or Shepsle and

Bonchek (1997).

Chapter 2   From Research Question to Theory to Hypothesis

Preview 2.1 Good Question, Bad Answer The examples below show good research questions with weak theories— that is, poor attempts at potential answers that are unlikely to yield good empirical research papers.1 Our goal is to craft better, more satisfying answers that can form the basis of a theory. A. Q: Why do states start wars? A: Because they wanted to fight. This is a good research question, as we discussed in Chapter 1. The answer, though, is somewhat lacking. The response most people would have to that potential answer is, “Why did they want to fight?” Better answers might be, “because they felt threatened by their adversary,” “because they wanted something their adversary had,” “because they believed they could win,” etc. These are all straightforward, no-duh kinds of answers, but all are testable and go beyond the superficial. B. Q: Why do some countries have parliamentary systems of government, while others have presidential systems? A: Because that’s what their Constitution says they should have. Again, why do their Constitutions say different things? Some potential answers might be found in differing patterns of social cleavages (countries with multiple social cleavages may have wanted a system that allowed for multiparty systems, which tend to work best with parliamentary systems), historical evolution/path dependency (countries that formerly were monarchies may have evolved into constitutional monarchies), or historical experience (countries that had bad experiences with presidents-turned-dictators may want to avoid concentrating power in one person again). Practice 2.1 More Good Questions For each otherwise-good research question below, create at least two better answers. A. Q: Why do some incumbents lose elections? A: Because they got fewer votes than another candidate.

1If these sound a little like the bad joke, “Why did the chicken cross the road? To get to the other

side,” you’re not far off. The point is that these answers, while technically accurate, fail to get at (Continued) the underlying issues in a satisfying manner.

(Continued)

29

30

Empirical Research and Writing

(Continued) B. Q:  Why do countries create new international organizations? A: Because they want to.

C. Q: Why do some marginal political groups turn to violence (terrorism) and others don’t? A: Because they think violence is the answer.

Progress 2.1 Theorizing Your Own Research Now it’s your turn to begin theorizing about your own research questions. Return to your work from Progress 1.1 or other potential research questions you’ve generated since then. Write each research question at the top of a sheet of paper. Underneath it, write a ridiculous answer to the question along the lines of the samples in Preview 2.1 and Practice 2.1. Then, ask yourself why about your ridiculous answer, like we did above. Try to craft three or four possible explanations—just a brief sentence each—for your question. Do this for at least three draft research questions.

them to find the one that you feel is the most likely or strongest explanation of the phenomenon.5 Making this determination usually requires you to rely on background reading or prior knowledge. In many cases, your proposed answer(s) will shift as you read the academic literature about your question— that is, as you do your literature review. And remember, it’s okay if you guess “wrong” and the theory you investigate doesn’t find support in your data. The potential answers you generate or find will differ in the direction of predicted effect, and in the mechanism that links cause to effect. Good research then compares the various theories to see whether the one you argued for is indeed the strongest or most complete explanation, or whether it does poorly in explaining the phenomenon.6 At this point in the research process, your goal is simply to identify several competing theories (explanations) for your phenomenon of interest, and to 5Notice

that I did not say “best” or “only” explanation of the phenomenon/class of events under consideration. That’s not our goal. 6In the social sciences, alas, even our strongest theories seldom do a phenomenal or complete job of

explaining the outcome of interest, so we’re looking for relative improvement here. Good research aims for a parsimonious, fertile theory that explains more than existing theories.

Chapter 2   From Research Question to Theory to Hypothesis

begin to home in on a particular theory that you think is worthy of testing. It could be one of your own devising or it could be one you identified in the literature. Regardless of its source, you need to be able to identify the dependent and independent variables and the causal mechanism that makes changes in the independent or cause variable lead to changes in the dependent or outcome variable. In the first part of this section, we’ll look at tools for developing your own theory. The second part presents a tool for specifying causal mechanisms, and the third part provides some guidance on specifying assumptions and scope conditions.

Tools for Developing Your Theory Developing a good theory begins with developing a clear research question. Many different tools can help you in your quest for an explanation, by helping you clarify your question, by helping you articulate linkages and patterns, or both. Some of my favorite are arrow diagrams, Venn diagrams, and two-by-two tables. Arrow diagrams are great for diagramming causal mechanisms. Since most of empirical political science is concerned with these types of causal chains, you should ultimately want to end up with an arrow diagram of your theory. We’ll spend more time on those below, but before that, let’s discuss the uses of other tools. Venn diagrams, two-by-two tables, and other similar graphic organizers are fantastic tools for theorizing early on in the research process. They can be especially helpful with focusing from a broad research question to a narrower, more focused dependent variable with explainable variation, and for most novice researchers, this is one of the biggest hurdles in the early research process. Venn diagrams are overlapping circles (or squares or triangles or whatever) that allow you to group cases by their values on some characteristic(s) of interest (i.e., some variable[s]). Areas of overlap between the shapes then show where cases have characteristics in common. Sometimes, simply sorting your cases by values like this can illuminate interesting patterns that you might not have spotted otherwise, and exploring those patterns can help to suggest explanations. Interesting research questions and theories can emerge from simply observing that no cases fall in the overlapping region, or that the distribution of cases in the two circles is very lopsided. Consider, for example, a research question asking about causes of swing voting, in which we want to explain why some states do not consistently vote for the same political party in US presidential elections. The Venn diagram in Figure 2.1 shows data about state vote consistency from the 2000 and 2004 US presidential elections using the standard two-letter US Postal Service state abbreviations.7 States that voted Democratic in both elections are on the left, 7The 2000 election pitted then-vice president Al Gore (Democrat) against then-governor George W. Bush (Texas); Bush won. The 2004 election pitted incumbent George W. Bush against thensenator John Kerry (Massachusetts); Bush won. A state’s allocation of votes in the Electoral College is equal to the number of senators and representatives that it has in Congress, ranging from 3 to 55.

31

32

Empirical Research and Writing

Figure 2.1  US States by Voting Consistency, 2000–2004

CA DE IL

CO IN NC VA AK

CT HI ME

MD MA MI MN NJ NY

OR RI WA

PA VT WI

IA NM NH

FL NV OH AL AR

GA KS LA MO NE SC TN UT WY

ID KY MS MT ND SD TX WV

Voted Republican

Voted Democratic

SOURCE: US Federal Election Commission.

and states voting Republican in both elections are on the right. The area of overlap shows states that did not vote consistently between these two elections—that is, the state went to one party in 2000 and the other party in 2004. This region is very sparsely populated; it suggests that our best predictor for which way a state would go in 2004 was which way it went in 2000. For all the fuss that commentators made about swing states, very few states actually swung. In fact, only a grand total of three states—accounting for a measly 16 Electoral College votes—actually changed hands between parties. This represents only 3% of the total votes of the Electoral College, and had they gone the other way, they would not have affected the outcome of the election. So from a general research question about swing states, we’ve now got a couple more specific forms of the question to choose from, and each would have its own set of answers. First, if so many states are at risk of swinging, why do so few actually swing? This asks about the proportion of states that might swing during any particular election. Another more specific version of the research question might ask about why some particular states become swingers and others only remain at risk. This is a story about explaining the behavior of particular states during a single election instead of a story about explaining the proportion of states (or Electoral College votes) swinging in each election.8 If Despite its lack of Congressional votes, Washington, DC, has received three votes in the Electoral College since the 1964 presidential election. 8You

might have noticed that these questions are about different dependent variables, proportions of states versus individual states. Our theories would thus be about predicting different things; we can’t just simply use one question’s theories to answer the other question. These questions thus have different units of analysis. We’ll revisit the concept of unit of analysis in Chapters 5 and 7 for

Chapter 2   From Research Question to Theory to Hypothesis

we hadn’t sat down and sorted the cases, though, we are unlikely to have noticed either of these patterns, and without noticing these specific patterns, we don’t have a beginning point for theorizing. While we used a specific set of cases here for the purpose of example, this theorizing process is largely deductive. Our research questions are not about explaining the specific states that swung, nor did we use the specific cases that swung as the basis for generating theory; our research questions are about explaining swing behavior in general. Venn diagrams are great tools, but my favorite one of all is the simple twoby-two matrix. The two-by-two matrix allows us to develop a more refined categorization (typology) of cases than the Venn diagram typically does. At the theorizing stage, we have two possible ways to use such a matrix. The first, which is less common for novice researchers, is to work in the abstract. We begin by identifying two concepts or variables and imagining hypothetical high and low values for them (or present/absent, or whatever is appropriate). Then we hypothesize in the abstract about what the likely outcomes are, enter those in the box, and test those predictions against data. The hard part of this is identifying the concepts or variables that go on the axes. Doing this without explicit reference to cases can be difficult in the absence of extensive background knowledge, which most novice researchers lack, but it’s often worth a try if you feel you’ve got a good grip on your variable(s) and theory. The second approach, which is more common for novice researchers, is to identify two variables as the axes, sort cases into the boxes made by crossing the variables, and then try to make sense of the patterns that emerge. The data themselves will help to suggest potential theories to explain the variation in outcomes. For an example, let’s continue with the US presidential elections data we used above in the Venn diagrams. Figure 2.1 shows US states grouped by electoral outcome in the 2000 and 2004 presidential elections. In each election, a state could cast its electoral votes for the Democratic candidate, or for the Republican candidate.9 Those two options label my rows and columns. Sorting states by outcome in both elections reveals another puzzling pattern. As in the Venn diagram, we see that only three states swung between these elections. Intriguingly, though, those three states did not swing in the same direction. Iowa and New Mexico swung from Democratic to Republican, while New Hampshire swung from Republican to Democratic. This observation clarifies our research question further: What explains divergent swings? We’d begin by trying to figure out what factors Iowa and New Mexico had in common with New Hampshire, to trigger their swinging qualitative and quantitative research, respectively. 9Maine (ME) and Nebraska (NE) are unusual cases. These states spread their electoral votes across their Congressional (House) districts. The Electoral College members vote independently, in accordance with the district’s results. In 2008, two of Nebraska’s district votes went to John McCain and the third went to Barack Obama. McCain thus got both of Nebraska’s statewide Electoral College votes and two of the district Electoral College voters, while Obama got the remaining district voter. In the 2000 and 2004 elections, all five of Nebraska’s Electoral College votes went to the same party. Despite the possibility, Maine has never split its votes.

33

34

Empirical Research and Writing

TABLE 2.2  US States by Presidential Vote, 2000–2004 2004 2000 Democratic

Democratic CA CT DE DC HI IL ME

Republican

Republican MD MA MI MN NJ NY OR NH

PA RI VT WA WI

IA NM

CO FL IN NV NC OH VA AL AK AZ

AR GA ID KS KY LA MS MO MT NE

ND OK SC SD TN TX UT WV WY

SOURCE: US Federal Election Commission

at all. But then in our second stage, however, we’d also look for characteristics that Iowa and New Mexico share—but that New Hampshire does not have—to explain why two swung one way and the third swung the other way.10 We would want to be careful, though, to phrase the theory in terms of general phenomena or characteristics so that the theory is portable to other elections, where other states are of interest. Ultimately, we must test the theory outside the data we used to create it to ascertain the theory’s generality. Ok, we’ve successfully narrowed down our research question into a specific puzzle. Now what do we do? For this example, take the question of predicting which states are most likely to change hands in a particular election, and let’s think through theorizing about it together. Scopewise, we can 10A little additional probing of the data shows that Iowa and New Mexico were the only states to swing between the 2004 and 2008 elections, with both states switching their support away from the Republican party and casting their votes for Democratic candidate Barack Obama. That observation raises even more puzzles about swing state behavior.

Chapter 2   From Research Question to Theory to Hypothesis

r­ easonably restrict our analysis to post–World War II presidential elections.11 By this point, the outlines of modern campaign infrastructure existed, with parties, polling, television, electoral law, and salient left/right cleavages all similar to those we have now. While the details of each of these change over time, their existence themselves is constant, and that makes all of these elections comparable in ways that the wartime and Depression era elections would not be. Because we know that which specific states swing changes from election to election, we want to focus on characteristics—variables—that are not specific to this particular set of states. Overly specific variables would be, for example, the role of Howard Dean, the Vermont governor whose unorthodox campaign (and behavior) significantly influenced the 2004 Democratic primary results, or the role of a specific public figure’s endorsement. Both are too particular to the 2004 election. So we need to think of things that can vary over time and space that might explain why some—few—states swing but others do not. Essentially, we’re looking for anything that might cause the share of voters for any particular candidate to change. Off the top of my head, I can think of a few things: time candidate spends in the state, state demographic shifts (age, race, income, etc.), changes in voter laws, state economic conditions, candidate or vice-presidential ties to the state, endorsement of state party or governor, media buying in that state, and overall candidate spending. All of these things could, in some way, affect which way a state goes. Looking at that list, I’m starting to see a pattern. Some of the variables I identified are about individuals, some are about states, and some are about candidates. Let’s re-sort the variables into those groups and see if this suggests any other variables that we missed on the first pass. Table 2.3 shows one individual-level variable and three variables for state and candidate levels. While we don’t have to have the same number of variables in each category, the table does prompt me to think a bit more closely about whether any additional factors affecting individual voters’ propensity to turn out might be in play here. Hmm . . . Oh—one more possible idea, voter mobilization drives like get-out-the-vote initiatives. That might play a role; it’s worth adding to the table, so go ahead and do that. Add anything else you can think of, too—we left you some room in all of the columns. For our next step, we need to shift gears from brainstorming to evaluating. Before I can choose one theory that I think will work best, I need to go through them and figure out how each would work. In other words, I need to sketch a causal mechanism for each potential independent variable. So let’s take a look at tools for doing that. 11We

could even make a case for restricting the study to only those elections after Hawaii and Alaska joined the Union in 1959, or after the District of Columbia received electoral votes in 1964, but for simplicity and to increase the number of elections available to test the theory on, I’m going with the earliest reasonable date. (If you make a decision like this in your own research, you will need to justify it in a similar manner to this.)

35

36

Empirical Research and Writing

TABLE 2.3  Potential Explanations of Swinging States Individual

State

Candidate

Voter laws

Demographic changes

Time spent in state

Endorsement by state party leadership

Ties to state

Economic conditions

Media buying Overall candidate spending

Parsing a Theory: The Causal Mechanism As we’ve discussed, any research question can have multiple theories about its answer. Those theories can differ in three primary ways: the independent (causal) variable, the direction of the relationship they expect, and the mechanism producing the expected effect. The arrow diagram is a great way to break a theory into its component parts, and this process helps to generate hypotheses about outcomes that we can test.

Different Variables, Different Theories The most straightforward way that theories can differ is in their independent variable. Our example above about predicting which states will swing is fundamentally a story of considering many different independent variables until we find a theory that we are willing to test more broadly. Each of the independent variables we identified above has a mechanism that connects it to the dependent variable (DV), and before we choose a theory to test against the broader dataset, we should at least think through the causal mechanisms and see what we’ve got. At the individual level, voter laws influence Electoral College outcomes indirectly, by affecting who can vote. Voter laws that either inhibit and/or facilitate participation by a party’s supporters could affect the total tally of the popular vote and so affect the state outcome. We’ve got a bit of a problem there, though. Voter laws are usually the same from election to election, so that would be hard to use to predict changes in state outcomes across elections. But that suggests that what we’re interested in here is not just the laws themselves, but changes in voter laws that would affect turnout. So perhaps my theory here is that changes in voter laws affect turnout, which affects state outcomes. That certainly sounds plausible.

Chapter 2   From Research Question to Theory to Hypothesis

Demographic changes are a little easier to work with—possibly because we’ve already expressed them in terms of changes. (Hmm . . . that’s probably something I should keep thinking about). Changes in the composition of the state’s residents could shift the share of the population who are inclined to prefer each candidate, which could influence the outcome of the election at the state level. The collapse of the steel industry in the 1970s and 1980s made jobs hard to get in what we now call the Rust Belt, including my home state of Pennsylvania. The result was fewer union members as the steel mills shed jobs (so less organized support for Democratic candidates) and a mass exodus of younger people (also a hit to the traditional Democratic partisan base) to regions with more jobs.12 I definitely think this sounds like it could work. But what about total candidate spending? When I originally suggested that variable, I thought that maybe increased candidate spending overall would increase the chance of states swinging toward them because the more money a candidate spends, the more reasons the press has to discuss them, which generates additional media time alongside all the media time and mail ads and anything else the candidate spent money on. The more I think about it, though, I’m not sure that total candidate spending is a good predictor variable for whether a specific state swings in a particular election. It’s definitely a candidate-level variable, but it’s a characteristic of the candidate at the national level, and our research question is about a phenomenon—Electoral College voting— that happens at the state level of analysis. The variable Total Candidate Spending would have the same value for a particular candidate for all states in the same election. It might be a better predictor for overall vote share—a dependent variable measured at the national level of analysis—than it is for state vote direction. That’s something important to consider. Still, our dataset contains multiple elections, so even though candidate spending is a constant within an election, it will vary across elections. That should be enough variation for us to analyze the data if we really wanted to. In my opinion, though, this is a pretty weak causal mechanism; I had to work to come up with a story for this variable, though it definitely seemed plausible when I initially wrote it on the list. I’d probably want to include Total Candidate Spending as a control variable (see Chapters 5 and 7) in my study, especially if I were using quantitative analysis, but for right now, I’m pretty sure this would not be the theory I focused on in my paper.13 12Ever wonder why there are so many Pittsburgh Steelers fans outside of the Pittsburgh area? Baby boomers grew up watching the team be successful and win Super Bowls, but the collapse of the regional economy as they entered the labor market drove them to move elsewhere. They took their loyalty with them, as is common for fans of winning teams. (This explains, for example, the lack of widespread fans of the Cleveland Browns—an equally steel-dependent city with a vastly less successful football team.) The result is a national following for what is, essentially, a relatively smallmarket team. Kind of neat, huh? 13In the interests of space, I’m only going to do a couple of these here. I strongly encourage you, though, to at least think through the causal mechanisms behind the individual level variable that you proposed, as well as one or two of the others.

37

38

Empirical Research and Writing

Same Variables, Different Theories We can also have multiple theories with the same independent and dependent variables. These theories differ in their mechanisms and predicted outcomes. As an example of differing mechanisms, let’s consider the democratic peace, which claims that democracies never engage in military conflict with one another but are just as likely to fight other types of regimes—in other words, that democracy causes interstate peace. One reason why democracies might be more likely to experience peace between themselves is because they had domestic norms of peaceful conflict resolution and project them to each other. A second mechanism would be that leaders have to keep their publics happy, and publics don’t like to be involved in expensive, usually deadly, and losing wars. Both predict the same observable outcome, absence of fighting between a pair of democracies, but for very different reasons.14 Other times, two theories with the same independent variable (IV) and DV may predict entirely opposite effects. Let’s consider the research question “How does economic development affect the environment?” One possible answer is that economic development spurs resource overuse and results in declining environmental quality. We might write this theory as Environment (decline)    Resource overuse    Economic development

The causal story here reads from right to left, thanks to a quirky convention imported from quantitative analysis: Economic development leads to resource overuse, which leads to a decline in the quality of the natural environment. This is a fairly straightforward theory, one that appears in a variety of guises across a wide range of literatures. This isn’t the only possible answer, though. An equally plausible argument might be this:

Environmental Citizen interest Economic Economic Environment  issues in politics  in nonmaterial  (material)  development (improvement) (increase) issues (rise) security (rise) (increase)

In this story, as a country reaches higher levels of development, its citizens experience higher levels of material security. Since they no longer need to worry about their daily needs of food and shelter, they begin to have time and energy to care about other nonmaterial matters, such as the environment and human rights. Since these issues interest citizens, they become attractive to

14To test these particular arguments, then, we would need to identify other observable implications that would be true for one of the theories but not for the other, and test those hypotheses as well. We return to this example in more detail in Preview 2.1.

Chapter 2   From Research Question to Theory to Hypothesis

policy makers, and ultimately, the politicians’ need to obtain votes causes them to adopt environmentally friendly policy that restores past damage and limits further abuse.15 This second theory predicts a relationship opposite to the first one: Here, we expect development to appear along with better environmental quality, and in the first, we expected economic development to appear with poor environmental quality. Both are equally valid as theories. The second is more complex, with intervening steps and a more detailed causal mechanism, but both theories are equally valid and generate testable hypotheses. In most arrow diagrams, the theory itself often generates hypotheses directly. The predictions we made at each step of the causal chain are testable hypotheses. That’s why we often indicate the (increase) or (decrease) below the variable/step in the chain. Other ways to indicate these proto-hypotheses include + or – signs before the variable name, and up or down arrows in front of the variable name. Good theories are carefully articulated and account for all decisions or stages between the independent and dependent variables; they give us a great head start on hypothesizing. Talking Tip

Be careful with your terminology; these terms are not direct opposites of one other in their conversational sense. Positive and negative are correct ways to refer to opposite effects, but the antonym for direct is inverse, not indirect. Inverse and indirect are not substitutes. An inverse relationship is negative; the DV goes down as the IV goes up. An indirect relationship, on the other hand, means that some other variable intervenes between the IV and the DV: The IV affects some third variable, Z, which then affects our outcome.

Preview 2.2 Theorizing the Democratic Peace A phenomenally large number of scholars have proposed an incredibly diverse range of theories to account for the democratic peace: the empirical fact that democracies never fight each other, even though they are equally

(Continued) 15This “postmaterialist” argument is most closely associated with the work of Ronald Inglehart and

his colleagues (Inglehart and Norris 2003, 2004; Inglehart and Welzel 2005). He and his colleagues direct the World Values Survey (WVS), a cross-national survey repeated at 5-year intervals in a range of countries, which allows them to test their hypotheses about postmaterial values in politics. Links to the WVS data, as well as to all of the other publicly available datasets mentioned in this book, are on the ERW website, http://study.sagepub.com/powner1e.

39

40

Empirical Research and Writing

(Continued) likely to fight wars with other types of states (nondemocracies and mixed regimes). I present several of these below.16 Remember to read the arguments from right to left. Peace            Democracy A. Domestic norms of peaceful conflict resolution One of the key domestic norms associated with democracy is the peaceful resolution of conflicts by legal or other negotiated means; seizure of one’s goals or objectives or desires by force is not socially acceptable in democracies. Democracies recognize this predilection in other democracies and so are willing to negotiate with others like themselves, who they know also consider the use of force to be generally unacceptable. B. Selective use of threats  Incentives for opposition signaling Leaders in democracies face an institutionalized domestic political opposition, and the opposition has incentives to capitalize on the leader’s “bad” policy decisions. This includes making threats (like starting a war) that the leader has no intent to carry out. As a result, democratic leaders are very selective in making threats of war because the opposition’s behavior will credibly signal the leader’s intentions to the potential adversary. When both sides are democracies, threats of war will be very rare, but when they are made, they will be very credible; parties thus have incentives to settle their conflicts by means short of war. C. Avoid losing wars  Leader incentives for “good” foreign policy  Leaders face reelection Leaders of democracies face regular reelection, and reelection creates incentives for them to pursue “good” public policy, including foreign policy. This leads them to avoid situations where they could become involved in costly wars that they might lose, and so they’re very picky about what wars they join or start. This leads democracies to be less likely to initiate wars against other democracies. Practice 2.2 What’s Missing The items below represent common theories, or at least common phenomena, in political science. Your task is to theorize about what connects the 16These

three theories are associated with the work of Doyle (1986), Schultz (1998), and Bueno de Mesquita et al. (2004), respectively.

Chapter 2   From Research Question to Theory to Hypothesis

independent—causal—variable and the dependent—outcome—variable. Write this in the space provided. Some causal chains may take more than one step; just add arrows in the middle as appropriate. Feel free to use other paper. A.

Amount of aid sent to country (US$)





Natural disaster in a poor country

B.

Financial or banking  crisis



Government corruption

C.

Probability of war





Unequal distribution of power between two states

D.

Economic development





Female educational empowerment

E.

Success of a thirdparty Congressional candidate





Economic inequality in a district

Progress 2.2 Fill In the Blank Return to some of the research questions you generated in Practice 1.2 or in class during the Research Question Game (if your class played it). Choose three of those, and write them in the form of Practice 2.1 above. Then, formulate a theory for each of those research questions. What do you think is the answer to the research question, and why do you think that’s the case? Just a sentence or two will do. Please use the format of Practice 2.2.

Specifying Assumptions and Scope Conditions Once you have a research question, a theory about its answer, some ideas about hypotheses, and a rough draft of a causal mechanism, you’ll be ready to begin thinking about the assumptions and scope conditions that go along with your theory. To be clear: I do not expect that you have all of those things right now, at this point in the term, on your first read-through of this chapter. You will, however, need to have all of them eventually, and preferably relatively soon. You should continue to read through the rest of this chapter, even if you don’t have all those parts in place yet, so that you will know what to do next. Then you can work through this section once you are ready. You may even find that reading through this helps you articulate your main theoretical claims by helping you identify the boundaries of your idea.

41

42

Empirical Research and Writing

Assumptions are things we take for granted about how the world operates. Avoiding assumptions is impossible in the social sciences (or anywhere else) because we as humans need to interpret the things we see to make sense of them. We make a lot of assumptions in everyday life, and we make even more when we start to theorize abstractly about the social world. To give one example, consider voting. We commonly assume that since a person voted for candidate A, candidate A is that voter’s most preferred candidate. On the surface, this is a reasonable assumption—but it is definitely an assumption. In technical terms, this is an assumption of sincere voting, that a person votes for the candidate she prefers the most regardless of anything else, like the candidate’s chances of winning. A competing assumption is that voters cast ballots strategically—that is, they cast their vote in such a manner that their most preferred credible candidate wins.17 Both assumptions have empirical support; each has incredibly different implications for theory. If we are trying to theorize about how people form preferences for candidates, we may not be able to use revealed preferences—votes—as evidence of preferences if we assume that voters behave strategically. Assumptions come in many flavors. The most common in the social sciences are about actors, their motivations, and the choices available to them at any given point. Economics, for example, assumes that firms are the primary actors and that they seek profit; that the behaviors of consumers and producers are interdependent and a function of price; and that governments are able to manipulate the economy through taxing and spending policies. Because the actors in political science are more diverse than those whose behavior economic theory traditionally considers, we can’t make such a simple list of assumptions. But when you’re working on your own project, you might consider some of the following kinds of questions. They may not all be appropriate for your particular research project, but they’re good places to start. Who or what are the actors or decision makers in your theory? Focus on just the relevant parts. You might believe that corporate interest groups matter a lot in politics, but if you’re studying voter response to emotional appeals, corporations may not be worth mentioning. Scholars of international relations may need to decide whether the state is a billiard ball (in Waltz’s [1979] terms) that simply responds to external forces acting on it, or whether some actor in the state—a leader or other figure—purposively makes a decision to pursue international conflict. Not all theories have actors or decision makers; again, whether this is a relevant question to ask about your assumptions depends entirely on your research question and your theory. 17In

the 2000 presidential election, some left-leaning voters cast ballots for independent candidate (and sometime Green Party candidate) Ralph Nader over Democratic Party nominee Vice President Al Gore. By voting sincerely here—casting their votes for Nader even though he had no realistic chance to win—these individuals split the left vote across two candidates. The Republican nominee, Texas governor George W. Bush, had a plurality (the largest share) of votes, even though more people in crucial jurisdictions preferred the left-leaning candidates. Similar dynamics occurred in 1992, when Texas businessman H. Ross Perot ran as an independent conservative candidate and split the Republican vote between then-vice president George H. W. Bush and the Democratic nominee, Arkansas governor Bill Clinton. Clinton was elected to the first of two terms as a result.

Chapter 2   From Research Question to Theory to Hypothesis

What motivates actors or their decisions? Once you know the actors (if any), you can begin to consider why they make the choices they do. Are they motivated by personal gain? By policy preferences? By staying in office? By social or societal benefit? All of these are possible answers to what motivates politicians, judges, activists, bureaucrats, and others. Do actors know all the possible options and their outcomes, or are they uncertain about some of those things? If your theory is not about actors—if it’s about conditions or contexts that facilitate or inhibit something, for example—then you may need to think about whether how those conditions came about matters. What underlying conditions are necessary for events predicted by your theory to occur? Does your theory require a particular constellation of interests, capabilities (power), or rules? Veto authority and decision-making procedures can greatly affect outcomes. A researcher studying comparative policy making in parliamentary systems might find that the role of party discipline or the number of parties matters a lot, both of which are products of the country’s electoral and party systems. An entire branch of Grand Theory in international relations, structural realism, focuses on the effects of power distribution among the members of the international system by studying relationships between the number and relative size of “great powers” in the system and various outcomes such as war occurrence. Again, though, focus on the major ones: Skip the gravity, water, air, and similar assumptions and focus on the ones that are necessary for your theory to operate. Does the outcome or phenomenon of interest limit the population or set of relevant cases? If so, why and how? If you’re studying wars or parliaments, for example, by definition, those can only occur between (wars) or within (parliaments) states, and referring to states means you’re focusing on the world after roughly the Peace of Westphalia in 1648. Similarly, studying European Union legislation, television campaign advertising, or nuclear proliferation restricts the appropriate pool of comparison cases to the period in which these things exist.18 These particular examples sit at the boundary where assumptions bleed into scope conditions. Scope conditions are boundaries for your theory. They differ from assumptions in that they delimit the range of cases in which your theory could be expected to operate; they do not directly explain why the theory operates in the manner it does. Theories do not apply in all places and at all times, usually because some theoretically necessary characteristic or some assumption is not met in cases outside of the theory’s scope. Most studies of US Congressional behavior, for example, typically do not try to explain behavior before the Watergate era, when the current system of committees, seniority, and such was installed. These features of the contemporary Congress are necessary to any explanation of its behavior, so including cases that lack these criteria doesn’t make any sense. Again, most scholars don’t know all of their assumptions by this early a stage in their research. Your goal at this point is to begin thinking about the why’s and how’s behind your theory: the assumptions and scope conditions 18This particular shade of assumption would typically appear in your paper where you describe the empirical research design rather than as an explicit assumption.

43

44

Empirical Research and Writing

that make your argument work. You’ll want to revisit this section (of the book and of your paper) repeatedly as you work your way through your research process and keep thinking about these things.

From Theory to Hypothesis As we discussed in Chapter 1, good research questions are usually about how to explain variation in outcomes. The research question asks about the relationship between concepts, and the theory provides an argument about the relationship between the concepts. The next step is to derive testable hypotheses from the theory. Testable hypotheses rely on identifying observable implications of the theory that would occur within the scope and under the assumptions you’ve specified in your theory. Observable implications are exactly what they sound like: They are patterns or trends that you would expect to see if your hypothesis was correct. An observable implication is phrased in terms of indicators, not concepts—the characteristics of the cases that you will be able to observe and measure to test your argument.19 Peer Pointer

“I had plenty of ideas for topics that I wanted to study, but it took me a long time to decide what causal relationship I actually wanted to test. The sooner you can learn to think in terms of [independent and dependent] variables, the easier your research process will be.” —Michael C., College of Wooster

The Parts of a Hypothesis More specifically, a hypothesis is a statement of the relationship that you expect to find between your dependent (or outcome) variable and your independent (or cause) variable. A hypothesis contains three important pieces of information: It identifies the dependent variable, the independent variable, and the direction of the expected relationship. By convention, a hypothesis is usually written as ± DV  + IV20 19We’ll 20An

revisit indicators and measurement in the last section of this chapter.

“official” working hypothesis, as opposed to this generic form, can only contain a plus or a minus in front of the DV; it cannot contain both. If increases in the same IV predicted both increases in the DV and decreases in the same DV, then we would not be able to falsify the hypothesis. No matter what we observed, it would always support the hypothesis. As we discussed in Chapter 1, falsifiability is a desirable characteristic so that we can reject incorrect theories.

Chapter 2   From Research Question to Theory to Hypothesis

Placing the DV on the left seems a bit odd, but this is how statistical software understands these things. We use statistical tools in quantitative research approaches, so this has become convention for researchers of both qualitative and quantitative orientations as well. The + in front of the IV indicates that we’re making predictions about what happens as the value of the independent variable increases. This could be increasing on a “regular” counting kind of scale (from 1,035 dead in battle to 2,436 dead), or from no to yes (0 to 1 in math notation), or from some to a lot (2 to 3 using numbers as shorthand). By convention (again one imported from the logic of statistical analysis), we usually make predictions about increasing values of the IV. The key part of the hypothesis is the sign in front of the DV. This can be + if you think the two variables are directly or positively related, that is to say, if you think that increases in the value of the IV cause increases in the value of the DV. Two variables that have a positive relationship, for example, are age and income: Older people tend to make more money than younger ones, on average. We’d write that as + Income  + Age The sign in front of the DV can also be – if you think that the two variables are inversely or negatively related, that is to say, that increases in the value of the IV cause decreases in the value of the DV. Two variables that are inversely related are amount of free time and number of classes you’re taking: People who are taking more classes have, on average, less free time than those taking fewer classes. We’d write that as – Free time  + Classes A hypothesis does not, however, contain a “because” clause. The sentence with the “because” clause is your theory: It relates concepts to one another and provides an explanation or justification for the expected relationship. A theory for our free time and classes relationship, for example, might be, “Busy people relax less.” The hypothesis, on the other hand, makes a prediction about the relationship between observable indicators of your concepts. Talking Tip

Weak discussions of causal mechanisms will say things like “After . . .” and simply string events along. A stronger presentation would be, “Faced with a choice between policies A and B, leaders who are interested in reelection will choose B because it satisfies the interests of their core partisan supporters.” This latter presentation is better because it integrates justifications for the proposed mechanism based on your assumptions and/or scope conditions.

45

46

Empirical Research and Writing

Hypotheses and Evidence The next step is to think about evidence for your hypotheses, and in particular, to identify two very important types of evidence, observable implications and falsifiers. Observable implications are empirical patterns that you’d expect to see if your hypothesis is correct. In the case of the hypothesis above about age and income, we’d expect to see that older people had higher salaries than younger ones. An important thing to note here is the “on average” part of the verbal hypothesis. Just because we find one example that doesn’t fit the hypothesis doesn’t mean that we’re wrong. An important characteristic of the social world is that it is not deterministic: We do not always observe the same outcome from the same value of an independent variable. Much of the physical world is (largely) deterministic. In chemistry, if you mix a specific quantity of chemical A and a specific quantity of chemical B together, you will always get a reaction that produces a specific quantity of some compounds C and D.21 In the social world, situations when a particular value of an independent variable always results in a particular value of the dependent variable are incredibly rare in the social sciences. Just because two states have highly unequal levels of capabilities does not mean that the stronger one will attack the weaker one, as balance of power theory would suggest—balance of power theory says only that war is more likely under these circumstances than others. A single theory may also have multiple observable implications, and so it may have multiple hypotheses associated with it. This is great—in fact, it’s ideal. This situation allows us to subject the theory to repeated attempts at falsification. If it withstands multiple tests, we can be more confident that it’s right. For example, consider the graduate school admission process. Most graduate programs start with a theory that a student is qualified for graduate admission. If this theory is true, we should expect to see a range of different types of supporting evidence. A student who is qualified for graduate study would have, for example, high scores on the GRE and strong writing skills. The student would also have a high GPA and a rigorous undergraduate course load. The student would have faculty members who both know the student and his or her work well, and who are willing to write about it on the student’s behalf. All of these are observable implications of a theory that claims a student is qualified for graduate study. If we see all of them, we know that the student is qualified; seeing all but one is also strong support for the theory, though not as strong as seeing all of them. A case (a student) where the theory is not well supported across the different observable implications is probably one where the theory does not hold—where the student is not qualified for graduate study. Thus, graduate programs ask you to provide a range of types of evidence

21As a chemist friend takes great pains to remind me, this is subject, of course, to some limits on the degree of precision with which we can measure the inputs and outputs. We discuss some measurement issues below, and again in Chapter 6 (for qualitative techniques) and Chapter 7 (for quantitative techniques).

Chapter 2   From Research Question to Theory to Hypothesis

47

in your admissions packet: GRE scores, and transcripts, and letters of recommendation, and a writing sample. Falsifiers, on the other hand, are pieces of evidence that you would expect to find if your hypothesis is incorrect. Because you expect your Probabilistic hypothesis to be right on average, across a large number of cases, small Theorizing, numbers of exceptions are possible. Let’s consider the hypothesis Falsifiers, and expressed above that income increases with age. The cases of a younger Bill Gates and Michael Phelps (the Olympic swimmer who had a gazil- Counter-Examples lion dollars in endorsement deals before he was even 18), for example, clearly do not support our hypothesis. But across the larger population, this relationship generally holds. For example, senior professors make more money than junior ones, who make more money than graduate students, who make more money than undergraduate students, who make more money than (most) high school students, who make more money than junior high students, etc. In the case of this particular hypothesis, falsifying evidence would be finding that, on average and across a large number of cases, younger people make more than older ones. Our argument is a generalized, probabilistic one—that generally, on average, income will increase with age. Our hypothesis does not claim that this will always be true in all cases. Another piece of partially falsifying evidence might be that the relationship only holds up to a certain point: Perhaps increasing age explains increasing income up to a certain point, but after that point—say, after retirement—age continues to increase but income begins to decrease again. In this case, our hypothesis would be partially right, but also partially wrong.22 Just to be clear, falsifiers are types of evidence (or pieces of potential evidence) that would indicate that the hypothesis is incorrect. They are not counterexplanations (hypotheses that would explain adverse findings), or counterexamples (single or rare cases that do not fit the hypothesis—the Michael Phelps of the social world).

A Caveat about Hypotheses One main caveat about hypotheses deserves discussion here. We cannot explain a constant (something whose value is the same across observations or cases) with a variable (something whose value changes across cases or observations). The converse is also true: We cannot explain a variable with a constant. To make this more concrete, imagine the following scenario. It’s the end of term, and you have two final exams today, back to back, for classes in your major. You didn’t sleep well last night, and you didn’t eat breakfast this morning, but you had a Red Bull immediately before each exam. You fail one exam, 22Such a finding might cause us to propose a revised theory in our conclusions and to test that new theory in a follow-on paper. We would not simply alter our theory for this paper so that our evidence supports it. As we discussed in Chapter 1, we go into an empirical research project with a question and a proposed answer and a set of tests; we do not go in knowing that our answer is correct. Finding that your theory is wrong is quite common, and it’s often just as important as finding that your theory is supported.

48

Empirical Research and Writing

but pass the other. What caused this variation in outcomes? We can’t really tell from the information given. All three potential causes—poor sleep, the lack of breakfast, and the effects of Red Bull—are the same (constant) across both cases, but the outcomes are different. Something other than one of these three things had to cause the variation in outcomes. Likewise, making an argument that colonialism or colonization explains democratic outcomes in Latin America is quite difficult. All Latin American states were colonized around the same time, almost all were colonized by Spain, and they all achieved independence around the same time and in the same manner. Their current levels of democracy and their histories of democratic experiences vary widely from country to country. Costa Rica has had a fairly stable democratic government for most of the post–World War II period. Venezuela, Peru, Argentina, and Chile have oscillated back and forth between democracy and nondemocracy, with Argentina becoming pretty democratic during the 1990s and Peru becoming significantly less so during this period while Fujimora was in office. Since colonial history is the same—or pretty close to it—across all of these cases, we cannot use it to explain variation in democracy outcomes. Not all instances of this problem are quite so obvious or explicit. All sorts of situations can lead to research that lacks variation—or lacks the full range of variation—on key variables. We’ll revisit some of the more common forms of these problems in our discussions of selection effects and selection bias (in Chapters 5 and 7 for qualitative and quantitative designs respectively), but you should be aware of the problem now. It’s a very common trap for novice researchers, and one you would do well to avoid.

Concepts, Indicators, and Validity We’ve briefly touched on the idea that we need to find observable indicators of our (usually unobservable) concepts. Our theories are about concepts; our hypotheses are about observable relationships between indicators of those concepts. This section introduces one of the major ideas in measurement: matching indicators with concepts, that is, validity.23 Creating workable hypotheses goes hand-in-hand with identifying viable indicators for each of the concepts. At this stage in your research project, identifying specific indicators isn’t entirely necessary, but you should be thinking about ways to measure the concepts in your theory so that you can construct testable hypotheses.24 The most fundamental issue in measurement is one of validity: Does our indicator actually capture the concept we’re interested in, and nothing else? If we haven’t actually captured the concept, then the mechanics of how we 23Chapters

6 and 7 will revisit validity and introduce the other major issue in measurement, reliability, in the context of qualitative- and quantitative-specific applications. 24The next step in your research project, conducting a review of the literature, will provide you with more information about specific indicators and data sources used by scholars working in your area of interest. We’ll discuss literature reviews in Chapter 3.

Chapter 2   From Research Question to Theory to Hypothesis

49

measured it are irrelevant. We may have a highly reliable, highly accurate, and highly precise indicator—but if it doesn’t actually reflect the concept we want, it’s really not useful at all. If our measures don’t capture our concepts, we have no idea what our test is actually telling us. It might show a relationship between our indicators, but if those indicators don’t capture their underlying concepts well, then we can’t draw any conclusions about our theory—which is, after all, about concepts as represented by indicators. Operationalization is the process of identifying a valid observable indicator for your unobservable concept. For some concepts, operationalization is generally straightforward. We can usually measure sex or gender by asking survey respondents whether they are male or female.25 Some concepts are not so simple to measure. Racial identity in the United States provides a good example. Survey researchers (and the US Census) used to treat race as dichotomous—you were white or you were Black, which was to say, not-white. The category of not-white, though, actually captures several groups who are not Black, such as those of Asian or American Indian descent. We’ve since expanded our conception of race to include a variety of categories, including Asian-, African-, and Native American. In addition, many survey organizations, including the US Census, also allow for respondents to indicate “two or more races” as an option. For other concepts, political scientists have developed standard indicators that we can deploy in surveys or other data collection efforts. We normally examine the intensity of an international or domestic conflict, for example, by counting battle deaths. Partisanship in American politics is often measured on a 7-point scale (see Figure 2.2). This scale was first developed by researchers working on the American National Elections Studies (ANES), the longestrunning survey series in US politics. Before its adoption for these purposes, though, ANES investigators did pretests to be sure that the measure was valid—that asking respondents to summarize all of their political preferences in one single label actually captured the concept the researchers wanted. They did this by asking the individuals to state their policy preferences on a series of policy issues that had clear left-right implications, and then later in the same Figure 2.2  Typical Partisanship Scale for US Politics

Strong Democrat

25Even

Moderate Democrat

Independentleaning Democrat

Independent

Independentleaning Republican

Moderate Republican

Strong Republican

this is usually a bit more complex than these two choices. Offering “male” and “female” as the only two choices is still generally the practice in most survey organizations. Depending on the survey’s purpose and needs, however, you may see a choice of “no answer” and/or “other” to accommodate individuals who are transsexual, whose gender expression and biological sex may differ, or who otherwise do not wish to disclose this information.

50

Empirical Research and Writing

survey, asked the individual to choose one value from the choices shown in the figure. They then used several statistical tools to ensure that the values individuals chose on the single consolidated scale were consistent with the policy preferences the respondents had identified—that, for example, individuals who wanted extensive government support for the poor and substantial state intervention in the economy were more likely to identify themselves as some form of Democrat, rather than “Strong Republican.” The scale would be invalid if they had found many instances of Strong Republicans wanting state intervention and a large welfare state, or worse, if they had found no relationship at all between partisan policy preferences and their potential measure of ideology. This kind of pretesting for validity is an important feature of most highquality large-scale datasets in political science. Because the creators of this scale carefully tested it, we can be sure that it actually captures partisanship, and not something different such as ideology, which is a different concept altogether. For example, a Tea Party conservative in the United States might identify ideologically as being strongly conservative, but would not identify strongly with the mainstream Republican Party; a similar story could exist for an American who identifies with the Green Party but ideologically shares many preferences with the Democratic Party. The key thing is that we need to choose a valid indicator that captures the concept we want, and only that concept, and to be able to do that, we need to be very clear on what that concept is. Researchers in both the qualitative and quantitative traditions face substantial challenges in creating valid measures of concepts. We’ll address those specific challenges in Chapters 6 and 7, but in a more general sense, those problems are part of a larger issue of confusion between concepts and indicators. You—and a lot of other researchers, including some faculty members and article authors—can avoid a lot of trouble with measurement later by thinking carefully and intensively now about what your concepts are. Identifying those underlying concepts now, at the beginning of the research, and framing your research question and your theory entirely around those concepts, helps you avoid reaching the end of the project with results that you can’t link to your theory. When you have a theory about one thing and an indicator measuring something else, you end up trying to put a square peg (evidence) into a round hole (theory). You might be able to force those two together, but they won’t fit well; gaps will always exist around the edges. That slippage—the gap between concept and indicator—can ruin even otherwise well-designed research. Your best bet for avoiding slippage is to be very, very clear on your underlying concepts, and that often entails refining your research question. Try to frame your research question and theory in terms of concepts as much as possible. For example, suppose you begin this chapter with the research question “How does trade influence wars?” You think—you theorize—that trade makes wars less likely because it integrates nations’ economies and makes them dependent on one another. That, on the surface, sounds like a pretty good start

Chapter 2   From Research Question to Theory to Hypothesis

of a theory . . . until you step back and realize that trade itself isn’t really the cause. Trade is an indicator of the real cause of decreased war: integration. So your question really is, “How does (economic) integration influence wars?” Trade is just one indicator of economic integration. Increased financial flows, similar business cycles, and many other indicators exist, all of which should also be associated with decreased conflict.

Writing Your Theory Section The theory section is typically written toward the end of the research process, while data collection and analysis are in progress. In terms of the structure of the paper, it usually falls between the literature review (which we discuss in Chapter 3) and the research design section (the subject of Chapters 4–8). Like almost all other sections of empirical papers, the shape of a theory section depends heavily on the specific theory it’s presenting, and some of that information may not be available to you right now, this early in the research process.26 What I can share with you instead are things to think about when writing your theory section and questions that may help you to clarify the things you need to discuss in it. Because of that, you should read all the way through this now, and you should write a rough draft of the section. As the previous chapter made clear, successful papers require much writing prior to the actual final draft, and there’s no time like the present to get started. Make a dry run now, then revisit it regularly as you continue to work. The point of the theory section is to present your story—your theory, the causal mechanism behind your hypotheses—to the reader. Without a clear understanding of why you think various things should happen in particular ways or under particular circumstances—that is, without a good understanding of the theory—readers have a very difficult time understanding the empirical analysis, let alone evaluating it. Because the theory is so important to the Talking Tip

Theory sections tend to link closely to hypotheses and research design sections, particularly if you bring out observable implications in your discussion. Statements like “If this argument is true, we should observe this thing and another thing” are entirely appropriate for this section. Potential falsifiers for both your theory and extant alternate theories are also good to discuss, too.

26The abstract, introduction, and conclusion are relatively formulaic and do not depend on the content of the paper or its arguments. All other sections are at least partially dependent.

51

52

Empirical Research and Writing

paper, from both the authorial and reader perspectives, it deserves more attention from the author than it usually gets in novice researchers’ work. Successful theory sections typically build a bridge between the gaps identified in the literature and the author’s own hypothesis. You’ll identify those gaps in your literature review process, so for now, focus on the latter stages of explaining what you think happens and why. Typically, you’ll want to begin by establishing the sequence of events or changes that lead from the IV to the DV, that is, explicating the causal mechanism. Good expositions here not only say what happens but why—they start to build some of the assumptions that underlie the mechanism. Resist the temptation to restructure your theory to account for your results. Remember that you designed all of this research to test a particular theory and its associated hypotheses. If you go changing things after the fact, you will now have a theory whose research design demands will probably be different from the design you executed. Findings of nonsupport are useful, and they are just as valid as examples of research as findings of support. Early on in the project, the most important thing is that you can clearly articulate exactly what you think is going on in that causal mechanism. This may take several iterations of trying to explain things. When you hit a wall and feel stuck, that’s a sign that you need to try a different tactic to explain it. I usually just jump right out of whatever document I was working in and try again from the beginning; “CTRL+N” or COMMAND-N produces a new blank document quickly and allows me to keep going without pause. It sounds a little ridiculous, but never doubt the psychological power of starting over with a clean slate.27 At this stage, I also usually try to talk about my ideas with anyone who will listen. The act of talking about it to someone, just like the act of writing, forces me to choose words and articulate my thoughts rather than just letting them flounder around as ill-defined ideas. When I go to write the actual draft of the theory section that I intend to submit, I usually start immediately, without looking back at my previous drafts, unless I’ve written a good draft of it very recently (within the last week or so). I don’t often use or even consult the old drafts at this point because they represent how I was thinking about my theory at the time that I wrote them, and I know from experience that my theory—and my ability to articulate it—evolve over time. The best rendition of the theory that I can produce is almost always the one I produce at the end of the research process. I wouldn’t be able to produce that without all the prior drafts, because I learned to write about my theory by writing about it, but the prior drafts themselves are usually vastly inferior in quality. The prior drafts are useful, though, when discussing assumptions. The most important assumptions of my theory are usually explained in the causal mechanism discussion, but some of the secondary ones are also quite important.

27When I’m done for the day, I’ll review all the other bits and pieces and dump anything that feels even vaguely useful into the Leftovers file. Err on the side of caution; you can throw things out later, but if you never save them, you can’t ever recover them.

Chapter 2   From Research Question to Theory to Hypothesis

Because they’re not quite so crucial, they’re easy to forget, which is why I usually go back and reference the older drafts at this point. In general, you should always articulate any nonobvious assumptions and any assumptions that drive your result, and note these in your paper. Try to avoid a string of statements all beginning with “I assume” or “It is important to assume” or the like. Presenting a bunch of assumptions in a way where they each don’t include a form of the word assume is a skill, but it’s worth developing.28 The assumptions are in the paper because they matter to your argument, so try to embed them in the argument. Dealing with scope conditions is a bit trickier. Sometimes they’ll go here with the rest of the assumptions. Other times, you’ll find that you can discuss them more smoothly within the methods section. When you discuss the set of cases on which you plan to test your hypotheses, you can also discuss why or if this differs from the larger population to which the theory applies. Most theory sections can benefit from some form of graphic, even just a simple arrow diagram or flow chart, that shows the order of events or which variables affect one another. This is particularly true for multistep process hypotheses and conditional hypotheses. Learning to build diagrams in your word processing program now is a good use of time; this way the skills will be available later when you’re under time pressure. In general, though, theory sections follow a much less rigid structure than other parts of the paper. You will want to remain open to alternative orders of presentation, especially as you develop expertise and confidence in your research skills. Use published papers as models. They’re your best source for ideas about how to put the pieces together. You may find you need to try several different arrangements before you find one that you really like or that works for your particular set of claims and needs.

Summary The transition from research question to theory to hypothesis is a crucial part of producing a good empirical research paper. A good theory explains patterns in data with a well-articulated “because” clause that specifies a causal mechanism linking the independent variable to the dependent variable. A good theory also identifies the scope conditions and assumptions under which it operates. Good hypotheses then identify observable implications of the theory—things we would observe if the theory were correct—and make predictions about relationships between measurable indicators of the theory’s concepts. Developing your theory and fully explicating its causal mechanism are key components of this process, as is using valid indicators of concepts. This is part of why the theory is such an important part of empirical research: Without a carefully thought-out theory, empirical research doesn’t make much sense. 28Again, though, when doing an early draft, go ahead and write “I assume” at the start of every sentence if you need to; the important part is just to get the ideas out of your head and onto the paper.

53

54

Empirical Research and Writing

Key Terms •• •• •• •• •• •• •• ••

Theory Causal mechanism Assumption Scope conditions Domain Inductive theorizing Deductive theorizing Hypothesis

•• Dependent (or outcome) variable •• Independent (or cause) variable •• Observable implications •• Falsifiers •• Validity •• Operationalization

3 P

Doing Pre-research

re-research is what one does before collecting data and estimating statistics. The pre-research phase of an empirical project is crucial to the project’s success. In this phase, you investigate the status of extant knowledge in the scholarly community about your research question, including finding out what data exist and/or are commonly used for your concepts. You also develop a sense of where your proposed answer to the research question—your theory—fits into the literature. These important steps help you avoid unnecessary duplication of work. No one wants to reinvent the wheel, but the only way to avoid it is to go find out what we already know. This chapter begins by reviewing the sections of a standard empirical paper, of the type that you see published in peer-reviewed academic journals and the type that this book helps you produce. Knowing your way around an article can help make your literature review time more productive in the long run by giving you clues to important things like what indicators, control variables, and analytical techniques are commonly used in answering your research question and others related to it. We then move into a discussion of how to think about, find, and navigate an academic literature. Finally, I provide strategies for the most challenging parts of the literature review: organizing it all and actually writing it.

The Parts of an Empirical Paper An empirical paper usually contains six distinct sections. The introduction presents the research question and summarizes the paper’s argument and findings. It usually follows a very predictable template, which we’ll discuss in Chapter 9. The second section, the literature review, places the research question, the proposed approach, and the proposed argument in the context of prior scholarship. Its job is to set the stage for the author’s own argument. This section is the primary focus of Chapter 3. In the third section of an empirical paper, authors explain their theory and explicitly identify their hypotheses. This section both provides the intellectual supports behind the author’s theory and states what evidence would support that claim. It specifies the expected relationship between concepts,



55

56

Empirical Research and Writing

the mechanisms producing these links, and the observable expectations about coefficients or statistical tests. The research design section is fourth. It establishes the scope or domain of the project, explains how the author operationalizes and measures variables, identifies data sources, and explains the data analysis technique. In the fifth section, the author analyses the data: This is the section where you actually do the research and present your results. Finally, the conclusion summarizes the theory, evidence, and findings, and specifies avenues for related future research (unresolved or newly raised questions, etc.), policy implications, or similar extensions of the work. It, too, follows a pretty standard formula that we’ll discuss in Chapter 9. These parts generally appear in the order listed. Sometimes authors will merge the theory/hypotheses and research design sections, or split them up somewhat differently—theory, then hypotheses and research design together. Occasionally, an author will derive the hypotheses directly from the various literatures reviewed, and so the hypotheses will appear in the literature review section. That said, these are fairly rare. The vast majority of papers will include these six sections, usually with these same names (or close variants). Subheadings should be short and descriptive of the section’s content. They are not places to get creative.

How to Think About Literature(s) A scholarly literature is a body of research about a particular research question or research theme. The various pieces or strands or elements of a literature are the individual articles or books that compose it. We can define a literature quite broadly, as in the literature on political institutions or political culture, or quite narrowly, as in the literature on the normative foundations of the democratic peace. The various elements of a literature are often connected by reference to one or a few foundational works, such as the literature on transnational advocacy networks that sprang from Keck and Sikkink’s (1998) Activists Beyond Borders, or that on campaign finance stemming from Jacobson’s (1980) Money in Congressional Elections. These foundational works often have the function of defining a research program, or series of related research questions, whose scholars regularly cite one another and draw on each other’s work. Once you’ve found a couple of pieces on a topic, comparing their reference lists is a good way to identify foundational or core pieces of scholarship. Items that multiple authors cite are probably fairly central in the scholarly discussion. Authors who have multiple items cited in the bibliography are often worth researching further.1 1Googling the author’s name and “CV” will usually produce his or her academic résumé (a curriculum vita), which will contain full citations for all the author’s publications. Many faculty have not only their CVs but also some of their papers online, so be sure to check the author’s web page for published pieces and working papers, which contain research the author is working on but hasn’t yet published. We discuss scholarly norms and etiquette surrounding working papers below.

Chapter 3  Doing Pre-research

The goal of the literature review is to situate your own research in the broader context. Where do you fit in terms of the specific theory you employ? Is it similar to a theory others have used by focusing, for example, on the role of identity, or the role of political parties, or the role of entrenched social interests? Where do you fit in terms of the specific methods and methodologies, or data sources, you’re employing? Does most of the literature use one set of tools or assumptions, and you’re making a crucial or radical change? Has anyone else used the dataset, or theory, or model type, etc., that you’re using? To answer these questions, you need to know what’s out there already. We turn now to the very practical side of learning what’s out there: how to search the university’s holdings and get scholarship.

What a Literature Review Is Not We can clear up some final misconceptions about the literature review by briefly discussing what a literature review is not. First, a literature review is not an article-by-article summary of the pieces you read, with one paragraph per article or item. I refer to this as the “beads-on-a-string” approach: It treats each article as a separate or distinct entity, connected to the previous and following pieces only by the string someone forced through the middle. Early-stage drafts may read like this; they sound like little more than strung-together annotated bibliographies rather than overarching summaries of the literature as a literature.2 The literature review is also not a historical background to your topic or cases. You can presume that your reader is at least reasonably well informed about geography and basic characteristics. “The United States has a presidential system of government with two political parties” is not a necessary statement, nor are statements like “Israel is a country that was founded in 1948.” Any specialized information that is directly relevant to your research can go in other sections as appropriate. Israel’s unusual political institutional experiment of having a directly elected prime minister with a parliamentary legislature means that it is an interesting case for testing some types of hypotheses; this probably goes in your research design section as a justification for choosing this case. Your literature review should not contain every single piece you read, no matter how tempting that may be. If it’s not directly relevant to your theory or data, it doesn’t go in the paper. Many students (and to be honest, some faculty) have a strong urge to cite everything they possibly can so the paper looks “better,” but lots of irrelevant information and citations can undermine your central claim by cluttering up the paper and obscuring the important stuff. Be picky (“discriminating”). If it’s not directly in the line of your argument, or in a line necessary to support your argument, you don’t cite it. And if you don’t 2A big warning flag for beads-on-a-string literature reviews is that the sentences all start with authors’ names or citations. It’s all about individual items, and the author-as-grammatical-subject structures are a clear sign of this.

57

58

Empirical Research and Writing

cite it, it doesn’t go in the bibliography. Some of that material may end up falling into that necessary but somewhat distasteful category of “background reading,” stuff you read to help make your research design decisions but which isn’t directly related to your research. Background reading is assumed by your reader rather than documented and trumpeted by you. Finally, I should note that a literature review is not a literary review. In English, these two terms are not interchangeable. A literary review is a particular type of critical review of fiction (or occasionally, creative nonfiction like a memoir). These two genres of writing—literature reviews and literary reviews— have very little in common except a focus on evaluating others’ work.

How to Find Literature(s) Let’s start with the big don’t: the card catalog. Scholarly books (sometimes called monographs) are not the best place to start to find literature for two reasons. First, academic books can often take several years to appear in print, and so they’re quite out of date by the time they appear; and second, most people publish their book chapters as articles well before the book itself is in print. The book form, then, is often not much more than a collection of someone’s recent research papers. A published literature review that includes only one or two books is quite common in the social sciences.3 Most useful literature for literature reviews is published in peer-reviewed scholarly journals. A peer-reviewed scholarly journal is one that publishes quality academic research; we know that the items it publishes have been vetted by at least three, and sometimes as many as five or six, other very qualified academics. The authors of scholarly articles (and the reviewers) are almost all college or university faculty, meaning that they hold earned research doctorates from high-quality institutions.4 This system of self-policing helps to ensure that published work is indeed done correctly and that its methods and conclusions are believable. The vetting process is very rigorous at many journals. Most top journals publish less than 10% of the pieces that authors submit to them, meaning that

3It’s

less common in the humanities; turn-around to publication is often faster, and authors are much less likely to have published the components as separate articles. So books are actually usually a novel contribution of scholarship, rather than the social science model of “expanded collection of my recent research papers plus some case studies.” 4Sometimes

you’ll find a few articles from PhD candidates, who are in the last stages of their doctoral degrees. This is not really an important piece of information; the vetting process for publication is the same for everyone, no matter their status or rank. The key thing with scholarly journals is that everyone associated with the process has a college or university affiliation. These are usually listed under the author’s name or on a separate Contributors page at the back of the journal. Getting a faculty position and/or getting into a PhD program also involve a process of peer review and screening, which adds an additional layer of vetting to the person’s overall body of work; in short, it helps to ensure that the reviewer is well-trained, knowledgeable, and reputable.

Chapter 3  Doing Pre-research

90% or more get rejected during peer review.5 The peer review process is usually double-blind, which is to say that the author does not know who the reviewers are, and the reviewers do not know the name of the author of the piece they’re reviewing. This enables everyone involved to be fully critical without fear of repercussion. It also prevents favoritism or bias from reviewers, who might say or think, even subconsciously, that just because a piece is from a well-known or highly regarded academic, it must be good and done correctly. A small handful of academic publishers dominates the scholarly literature in each field. In political science and international relations, most journals are published by Cambridge University Press, Taylor and Francis, Routledge, SAGE, or Blackwell.6 Those publishers have websites where they post all the articles from each journal for access by subscription. Most colleges and universities pay for institutional subscriptions to virtually all of the relevant journals in political science and international relations. You can usually access these publisher archives, or several other nonpublisher archives, from any computer on campus. Off-campus access protocols differ by institution, but they’re usually clearly indicated on the “Search Journals” page of your institution’s electronic card catalog. Typically, you must access the source through the library’s web page; search the catalog or the database listing for the desired title and enter your login credentials as prompted. You should never be paying out-ofpocket for access to a journal article. If your school does not own a subscription to that particular journal, then you can use interlibrary loan (ILL) to request a copy of a book or article from a broader pool of schools. Both of these tasks are usually available online through your library’s website; search the catalog for the name of the journal, then use the links on the catalog entry page to ILL it. That said, ILL takes time, sometimes up to 2 to 3 weeks for books or older journal articles. If you plan to use these sources, you should allow plenty of time between your searching and your due date. Several other important sources for journal articles exist. In the social sciences, JSTOR (short for the Journal Storage Project, www.jstor.org) is an extensive database of full-text journal articles. Its holdings contain hundreds of journals—some 90 in political science alone, and more in sociology, history, 5The

American Political Science Review (APSR) and International Organization are two of the top journals in US political science. The APSR’s 2007–2008 acceptance rate was 10.5% (87/829, all accepted after revision; calculated from Rogowski and Triesman 2009); the 2011–2012 rate was 6.1% (all after revision; Ishiyama 2013, 421). 6To be clear: Most journals are sponsored by scholarly associations or societies (such as the American Political Science Association or International Studies Association), which usually have substantive responsibility for the journal’s contents. The society nominates and approves the editor and editorial board, who are always senior scholars in the field. A small number of journals, such as World Politics and International Organization, are independent in that they are not association journals. The editors and editorial board of these journals are self-selected and self-perpetuating, but again, it’s through a public process that incorporates and includes many of the field’s senior scholars. The publisher of a journal has nothing to do with the peer review process. It simply manages the physical logistics of hard-copy journals: typesetting, printing, mailing, archiving, etc.

59

60

Empirical Research and Writing

and law—and are fully digitized, often back to the very first volume of each title. They are full-text and fully searchable, and you can download PDFs of articles very easily. The disadvantage of JSTOR is that each journal has a “moving wall,” a window between its most recent published issue and the most recent issue available in JSTOR.7 This can be up to five years for some journals, though most are typically about three years. Recently, JSTOR has added a feature that will search the contents of a journal after the moving wall and provide links to it in your JSTOR search results; be sure to check or uncheck the box for that on the search form if you want that feature. In international affairs, CIAO (Columbia International Affairs Online) and Project MUSE also are significant collections of journals. Most of the CIAO and Project MUSE holdings are duplicated in JSTOR, though, so for most scholars, JSTOR is usually the first stop on the journal search. A few words of caution here: Just because a source says review or journal in the title doesn’t mean it’s a peer-reviewed scholarly publication that you should use in your literature review. Many journals of policy scholarship or policy commentary exist, and they’re often peer-reviewed (at least partially), but they do not include the kinds of research/scholarship that we discuss in literature reviews. In international affairs, Foreign Affairs and Foreign Policy, for example, are almost exclusively about proper noun topics or policy advice, and as we discussed in Chapter 1, proper nouns are not good for research questions (or their answers), and policy questions are not empirical. Those two sources, along with Millennium and Orbis and a few other similar publications, are not things you should include in your literature review. Similar titles exist in comparative politics and American politics, including The Atlantic, the Economist, and New Republic. Many of these publications are included in JSTOR and other similar databases, so you will need to be alert for them in your search results. As Chapter 1 also mentioned, several journals also publish article-length literature reviews. In political science and international relations these are Annual Review of Political Science, International Studies Review (which also contains book reviews), and to a much lesser extent, World Politics (WP) and International Organization (IO).8 The latter three publish only solicited reviews, meaning that the editors approached some senior member of the field who works in that area and asked him or her to write a review. This solicited review then goes through the normal peer review process. The result is that literature reviews published in ARPS, WP, and IO are generally of very high quality and are as current as possible given journal publication cycles 7The

purpose of the moving wall is so that archive subscriptions don’t compete with current subscriptions; you can’t get to the most recent content without having a recent or current membership or subscription to the publication. 8The

Oxford Handbook of Political Science is a 10-volume collection of essay-length literature reviews, with very solid coverage of research programs in American politics. The Journal of Public Policy Processes and Policy Studies also frequently contain literature reviews on issues of public policy analysis.

Chapter 3  Doing Pre-research

(i.e., usually about 6–8 months). These stand-alone literature reviews will have extensive bibliographies that also serve as good starting points for you to locate central pieces of research in that field.9 Bibliography-hopping is usually one of the better strategies for finding related literature, especially once you’ve found one or two core or recent pieces. Use the bibliography or reference list in each item you read to identify other good candidates. Bibliographies, however, only go backward in time. We have a fantastic tool for going forward in time—to find pieces that cite the item in question, things that came after and possibly build on the piece we’ve just read. This is the Social Sciences Citation Index (SSCI), sometimes known as the Web of Science. It creates lists and even visual maps of things that cite one another. You can use this to go forward one or more generations of scholarship. For example, let’s say you’ve found Jim Morrow’s (1994) article on information and distribution problems, and you want to find other works that have built on this—ones published later that cite the article of interest. The SSCI can then help you see that Morrow (1994) is cited by Koremenos et al. (2001). Likewise, you can use that feature again to find that Koremenos et al. (2001) is then cited by Gilligan (2004), and so on. The major limitation of the SSCI is that it focuses almost entirely on articles and does not generally do cross-indexing for books, either books cited in SSCI-indexed articles or books referencing SSCI-indexed articles. Also, the SSCI has very limited access; you may only be able to access it from computers that are directly on the campus network (including your personal laptop connected to the campus wireless network), not via remote login. Check with your own library about off-campus access to the SSCI and other databases. Finally, if you plan to use the SSCI, its search tools can be a bit tricky; a few minutes’ orientation with a reference librarian will save you a significant amount of time and frustration. Finally, we owe a few minutes to that most pervasive search tool: Google. Google is a wonderful tool, and I use it frequently—just not in the early stages of identifying literature. I use Google for two reasons in my pre-research. First, if I find an item that I really want to read but to which my own library doesn’t appear to have immediate electronic access, I’ll try a basic Google search to see if it’s available online for free—on the author’s website, on someone else’s e-syllabus or website, or even on the journal’s website as a free sample. This can help reduce the need for ILL requests, which saves you time and your library money. Second, I will use Google Scholar at some point, often later in the process, to see if any conference or working papers exist on my topic. Google Scholar searches several major working-paper repositories, including the Social Science Research Network (SSRN) and the archives of the Annual Meetings of the American Political Science Association and several other 9We do not normally cite book reviews and generally do not cite literature reviews directly. We go back to the source materials directly. The exception to this would be if a literature review or book review raises a good critique that you want to echo in your own discussion of the source work, or if a literature review raises the question or points out the gap that your research fills.

61

62

Empirical Research and Writing

major professional organizations. The Annual Meeting archives include all papers or posters presented at these conferences, and since most journal articles start their public lives as conference papers, it’s a great way to find cuttingedge, not-even-published-yet research. Prepublished work such as that on the SSRN or other conference/working paper archives comes with its own set of risks, rewards, and etiquette. First, papers on these sites have not been peer-reviewed in the way that published pieces are. In fact, one major reason to take a paper to a conference is to get public feedback on your methods, theory, and the like. This isn’t to say that these works aren’t credible—they’ve often been reviewed informally by at least a few people before they reach the conference—but they shouldn’t form the foundation of your literature review. Second, working with prepublished research is often very beneficial to both you and the paper’s author. You get cutting-edge results to discuss, but sometimes almost as important, that paper will contain a very recent literature review to help you find more ideas and more pieces. Most authors always want more feedback on their work, especially if it hasn’t yet been sent out for review and publication. That leads us to the third issue: etiquette. Prepublished manuscripts, even if they’re in a conference archive or on a person’s website, are often not “finished” documents. They’re being circulated for feedback, not put out there as conclusive findings, and so you should not cite them without the author’s permission. Most prepublished papers will say this explicitly on them, with a remark like “Please do not cite without permission from the author” right on the cover page. Even if the paper does not say this, you should email the author and politely ask for permission to cite it in your literature review.10 Most scholars will gladly grant permission, often with the condition that you send them a copy of your completed paper and/or some feedback on the working paper. Insider Insight

Try using concepts as search terms for your literature review. The concept form of your dependent variable, and any key concepts in your theory or causal mechanisms, are more likely to produce usable results than searching with proper nouns.

10The only time you do not need to ask for permission is when the paper’s cover page says “forthcoming in (some journal or edited book).” First, check the publisher’s website to see if the book/ article has indeed been published; if so, cite as if you had read it in the book or journal. If it has not yet been published, you can cite the journal or book in question as something like Smith, Peter. (forthcoming). “Title of the Paper.” Title of the Journal. Since it hasn’t yet been published, no year or page numbers are included; the in-text citation is (Smith forthcoming). (Double-check with your professor’s preferred style manual for punctuation and/or URL requirements.)

Chapter 3  Doing Pre-research

How to Organize Literature(s) Organizing a literature review is a tricky process. The default mode that most authors use in their first draft, at least, is a sort of beads-on-a-string approach, characterized by a lot of sentences or paragraphs that start with an author’s name. The author’s focus on individual pieces within the literature tends to manifest itself by having the pieces as the focus of each sentence or paragraph—that is, the authors’ names serve as the grammatical subject. This emphasis on the trees often omits important characteristics of the forest, such as gaps or inconsistencies between pieces rather than within them. Ultimately, you want to identify gaps or weaknesses in the literature. If you’re reviewing the correct literatures for your particular paper, your project should fit neatly into those gaps or weaknesses. This does not mean to say that you need to go through each item like a laundry list and make some kind of critique against it. In fact, that’s the least useful way to conduct a literature review. You are reviewing and critiquing the literature, not the individual pieces in it. Look for gaps and weaknesses and lacunae at the broader level of the whole body of scholarship. You can discuss critical weaknesses in pivotal works or in pieces very close to your own approach, but the focus should be on looking at the broader level—the bird’s-eye view, so to speak. Look across pieces more than down into individual ones. Your structure of information and reading management may not mirror the system I describe here and use myself, but you should keep in mind that this is ultimately the goal of the literature review.

Managing the Reading and Information Writing a literature review is one of those tasks in which a small investment in organization up front has large payoffs later. The point of a literature review is to integrate and evaluate a substantial number of different pieces, so any strategy that facilitates keeping track of all of them is useful. Two common strategies are tables and annotated bibliographies; those who are visual learners/ thinkers often prefer the former, and those with strong textual orientations prefer the latter, but some people do both, and a small fraction uses neither. The first strategy is to make a table (technically, a matrix) to organize your reading. I usually do this in Excel so that I can maximize the sort functions, but Word or any similar program will work as well. I usually include the columns listed in Table 3.1 to start; as I go, I may find myself adding columns such as theory family (see below), responding to, primary data source, etc. As I read each (empirical) article, I complete its row in the table.11 This process takes just a minute or two. “Source” is the abbreviation for the journal where the article is published. “Findings” is usually some indication of support for the hypotheses; the anticipated direction of the relationship is indicated by a 11Of course, not all items you read will be empirical articles. Some theoretical or model-based articles may well be relevant to your research question, and you’ll need to be flexible with how you complete the columns in these circumstances.

63

64

Empirical Research and Writing

TABLE 3.1  Sample Literature Matrix

Author

Year

Source

DV

Key IV (Predict?)

Findings

Quick Summary

Thoughts

+ or – in the “Key IV (Predict?)” column. “Quick Summary” is just a one- or two-sentence explanation of the theory or of anything else important to the main argument. “Thoughts” includes any reaction I had to the piece, such as preliminary thoughts about a critique of it or how it relates to my own work. The advantage to this type of organizational strategy is that it allows you to sort easily on any of these characteristics, such as findings or dependent variable. It allows you to see a substantial number of entries at a single glance, since each one doesn’t take up much room on the screen. The primary disadvantage of this strategy is that in some ways it discourages thinking critically about the pieces and seeing them as parts of a whole. That’s why the Thoughts column is so important—as you begin to expand the number of entries in your table, go back and keep making notes in the Thoughts column for each article. These thoughts form the basis of your structure and your critical approach to the literature (see below). If you aren’t comfortable going back like this, or you don’t have a sufficiently comfortable grasp on the material to start commenting very early, consider taking notes on the cover pages of the articles instead and building a matrix a little later in the process, after you’ve read a few pieces. The second strategy is to use an annotated bibliography. An annotated bibliography, as its name suggests, consists of two parts: the bibliography and the annotations. Each entry in your annotated bibliography is one piece of literature. It always begins with the complete citation, in the preferred format that you’ll be using for the paper itself. (Doing it this way saves time in the long run.) These then can get pasted directly into your bibliography for the final paper if you’re not using bibliographic management software (see below). The paragraph-long annotations then do two crucial things. They provide a brief summary of the key points, and they offer some type of critique or evaluation or contextualization of the piece. The latter is actually the most crucial part of the process, since this is what later allows you to build your commentary into a coherent review. Avoid the temptation to copy and paste the paper’s abstract into your annotated bibliography. Yes, that’s the type of information you need, in about the length and level of detail that you need it, but if you copy and paste, you’ll miss the benefits of having to read it and process it and articulate the ideas for yourself. Copy and paste fundamentally defeats the purpose of the annotated bibliography; this is not to mention the

Chapter 3  Doing Pre-research

fact that it constitutes a violation of most schools’ academic integrity codes (if you have to turn in the annotated bibliography) and leaves you with no basis to write a critique or anything else in your actual literature review.

A Few Last Thoughts Students often ask whether printing the articles or reading them in an electronic form is better. This is entirely a personal preference, for the most part, but I recommend that you always print the cover page. Sometimes it seems like a waste of paper or printing, but it ensures that you have all the necessary citation information. Having to go back and re-find the article because you’re missing crucial citation information will be more costly in the long run. The cover page also gives you room to write your notes about the article. Even if you’re not printing the article itself, you may want to print the cover page and make your notes on there. Many people find having hard copy pages to move around and physically regroup is a useful way to try out different potential organizations of the literature. A few words of caution: Don’t put off making your notes in your bibliography software, annotated bibliography file, and/or on the article cover page. Doing it now only takes a couple of minutes. Just do it. If you put it off until later, you’ll forget what the article was about and have to go back through it, which will take longer—if, that is, you ever get around to doing it. Articles needing notes have a tendency to pile up, and no, I can tell you from experience, waiting and doing it all at once will not be faster than doing it now. If you wait, you will need to find an empty hour or two, and who has those? If you do it now, you can do it in just a couple of minutes, and we can all manage to find those. Trust me on this one. The task is not faster if you try to do it all at once.

Managing the Reference List The task of mentally managing the references is fairly complex and deserves effort up front, by making a matrix or annotated bibliography. But the task of physically managing the reference list is at least as complicated. Modern researchers rely on electronic bibliography management tools, with a wide range of bibliography software and plugins available. Table 3.2 lists several popular tools roughly in order of popularity in political science. All of these tools track which items from your reference list are cited in any given document, and they generate bibliographies of cited items in the format of your choice. (The number and choice of formats available varies; the most common in political science are APA and APSA.12) You can also change between bibliography formats with ease—just choose the format you want from the software’s list, and it will do the rest. Each of these software options has significant 12APA is the American Psychological Association and APSA is the American Political Science Association. Both APA and APSA are in-text citation formats, as is the less-common MLA (Modern Language Association, which dominates humanities scholarship). Among footnote/endnote reference systems, Chicago/Turabian is the most widely supported.

65

66

Free

JabRef + BibTex13

13I

Included in MS Word software

MS Word

Personal computer

Personal computer (or personal profile on school network)

Web based

Personal computer or web based

Personal computer or web based

Network, web, or personal computer

thank Mike Kristufek for helping with the JabRef + BibTex entries.

www.bibtex.org

http://jabref .sourceforge.net/

School-paid subscription

www.zotero.org

Free, but requires (free) Mozilla Firefox browser

RefWorks

Zotero (plug-in)

Free from

Zotero (standalone)

www.zotero.org

School-paid subscription or personal expense

EndNote

How Do I Get It?

Where Are My References Stored?

Integrates with BibTex to auto-format bibliography for LaTeX documents only

Built into software; no additional integration required

Plugin download required; supports Office & OpenOffice (via *.rtf)

Plugin is a separate (free) download; Office & OpenOffice supported

Installs with software; Office & OpenOffice supported

Installs with software

How Does It Integrate With Word Processing?

TABLE 3.2  Bibliographic Management Software Options

Packages for many common formats, or custom user-defined formats

APA, MLA, Chicago; limited options; does not support APSA

All, including APSA

All, including APSA

All, including APSA

All, including APSA

What Common Formats Does It Support?

All entries must be hand coded; highly flexible for unusual cite formats and item types; can create shortcut links from citation to source document

Limited search function; all items must be manually entered

Metadata grab from supported platforms; some library databases support citation export

Tag search; notes field; screen grab; metadata grab (equivalent to citation export)

Tag search; notes field; screen grab; metadata grab (equivalent to citation export)

Key word search; large notes field; many library databases export citations

What Other Special Features Does It Have?

Chapter 3  Doing Pre-research

pluses and minuses. EndNote is powerful but expensive. BibTex is equally powerful, if not more so, but it is a programming toolkit for use with the LaTeX typesetting language; if you don’t have a background in computer languages or a need to write math equations, it’s probably not for you. The Zotero options include a screen capture tool that is great for those doing research on Internet and web-based phenomena; its integration with the Firefox web browser not only captures metadata embedded in most library database items, but also will grab appropriate information from the headers of any web page. Microsoft Word’s bibliography tool debuted in the Office 2007/2008 package; it integrates well with the base word processing software, though it lacks the direct-dataimport functionalities of Zotero, RefWorks, and EndNote. Which of these options is right (write?) for you really will depend on what you have (personal computer? Mac? OpenOffice?) and what you need (citation formats, library database or other software integration). The table gives an overview, but you may want to discuss your options with the library staff at your school. They can tell you about which, if any, of these your school provides to students, and they can help you determine which platform best fits your personal research needs and computing situation. Remember that the library staff are information specialists. They can tell you how to find it and how to manage it, no matter what it is.

How Many Sources Do I Need? You’re probably going to hate me, but the answer is, as with so many research design questions, “It depends.” This question has no set answer. You need to capture the main lines of argument in your research area, well enough to contextualize your own research. In some areas, this means only five or seven key works; in others, you may need 20 or more. I can only offer a couple of general guidelines. First, for an empirical paper, the literature review is typically about 20% to 25% of the paper’s total length. For a 20-ish page complete empirical paper, you’re probably looking at four to five pages of literature review. For a research design paper, in which you simply describe the project you’d do without actually executing the data collection and analysis, the literature review typically is closer to about 40% of the total length. Second, most people find that they can estimate the number of sources they need by multiplying the estimated literature review length by 3 (i.e., having an average of three sources for every page of literature review). This is only a rough estimate, and as I discuss more below, we do not simply write a paragraph on each item so that three items per page means three paragraphs per page. This beads-on-a-string approach is not acceptable. You’ll need to integrate the sources in the review itself. But for estimating the number of sources, it’s a good but rough guideline.14 14Graduate students and advanced literature review writers will probably want to estimate four to five sources per page.

67

68

Empirical Research and Writing

Finally, the reduction factor at each stage of screening is about one half for beginning researchers; at the MA thesis and PhD levels, it can be two– thirds or more. That means that about half (two–thirds) of the items you have will get thrown out or rejected at each phase of locating and preparing to write about a literature. If you search in JSTOR for a key word, for example, you might get 60 citations. That’s enough, right? Wrong. Once you take a closer look at those citations, just reading the titles and source journal names alone will cause you to reject about half (30) of them. They’re policy stuff, they’re book reviews, they’re single occurrences of the word or phrase in a context that makes no sense or is irrelevant to you. After this first cut, you’re down to about 30 items, and you’d normally click through and read the abstract for each of these. Reading the abstracts will also eliminate about half of the sources, since they’ll turn out to be about other research questions or to be only tangentially connected to your topic. So now you’re down to 15 sources, and that’s the bare minimum—and probably not even that—for a good four- to five-page literature review. Writing a literature review on that alone would be possible only if every single piece were totally on the right topic. Only about a quarter of what you turn up in an initial search will even be worth reading, and some of those won’t be worth citing or will be offtarget in some way that makes them not appropriate to cite. Always err on the high side; eliminating or omitting items from the final write-up is easier than desperately hunting for one more thing that is “just right” to fill the gap you found in your essay.

How to Write the Literature Review Writing a literature review is typically not a one-night affair. It’s not something you can crank out in an all-nighter. Since the point of it is to think critically about the pieces and integrate them, being sleep deprived usually results in a very poor piece since your critical thinking shuts down. It is best done in stages, with several sessions of preparatory work and organizing prior to the actual sitting-down-and-typing part. I typically find that my first round of writing is really just a very rough draft. No matter how much thinking I’ve done about it already (see below), the process of trying to write it makes me find more connections and reconsider my structure and organization. So going through two or three complete drafts of a literature review is not uncommon for me or most of the other scholars I know.15 Be prepared to put time into this process; it is a process, not a single-try situation. 15I know very few people—a total of perhaps two—who can reliably get a literature review “right” the first time they sit down and type. The one who can do it most reliably has been writing in this field for nearly 30 years and is personally responsible for most of the key works in his corner of it. Most of us need a couple of iterations, including at least one outright starting over, before we get something that works for our purposes. Don’t ever delete or throw away those drafts, though; you

Chapter 3  Doing Pre-research

Strategies for Finding an Organizational Structure A “good” organizational scheme will include all the critical pieces and a large majority of the other items you want to include. The categories will be large enough to contain a reasonable number of citations (i.e., a category isn’t just one or two pieces), but small enough to still be meaningful divisions of the literature. Some potential organizational strategies might include sorting by •• •• •• ••

Dependent variable Key independent variable Theory family16 Strength of support for your/their hypothesis(ses) or direction of findings •• Measurement strategy/data source for a key variable •• Field of origin of the piece (for projects that draw on, e.g., legal scholarship, philosophy, and political science) Which of these you choose is often a function of how your work relates to the existing literature. You will need to find a structure that allows you to highlight how your work fits into our current understanding of the field and how it fills a gap or responds to a need. If your argument is about a particular independent variable that other scholars have not addressed, then strength of support is probably not a good structure, nor is theory family. If you have one dependent variable, then an independent-variablesbased organization might make sense; if you have multiple dependent variables, then a dependent-variable-based organization might make more sense. The “right” organizational structure very much depends on your particular question, the state of research in that field, and the relationship between the two. In general, we do not sort by research method or technique (qualitative versus quantitative, formal model versus empirical, regression versus other, etc.). These are not typically useful ways to categorize the literature. The whole point of the literature review is to place your work in context by giving the reader some sense of what we already know. That is best done by focusing on strands of knowledge within the literature, not by focusing on the technical details of how the authors reached those conclusions.

might find the ideas in them useful at another point in the project. Just drop them in your Leftovers file for later. This is particularly true for larger projects like theses and dissertations, where ideas and arguments will recur repeatedly over time and pages. 16A “theory family” is a cluster of related answers to a research question. For the democratic peace literature, these might include normative explanations, institutional explanations, and Kantian explanations.

69

70

Empirical Research and Writing

Insider Insights

Prof. Hilton Oberzinger (2005) has identified several popular rhetorical patterns for literature reviews. These include competing lines of argument (“Battlebots”), history of developments in the field (“Road Map”), related-but-not-directly-on-target materials (“Guilt by Association”), the application of a new theoretical or methodological lens to a wellresearched topic (“Eyeball Switch”), holes in current knowledge (“Swiss Cheese”), and current research methodologies requiring replication (“Déjà Vu All Over Again”). Check out his article online for more on these and other common literature review organizational strategies.

No matter what structure you adopt, you should at all costs avoid the beads-on-a-string approach. Your goal, as I’ve said, is to integrate the literature, to look for patterns and ideas that go beyond individual pieces. A paper that simply takes a one-paragraph-per-article approach to the literature review is typically not going to be very successful. The process of organizing and integrating the various pieces is what actually provides the context for your own work: It organizes the material into categories or groups that are relevant to your own work. We turn next to tools and techniques for making sense of a literature.

Insider Insight

“The hardest part about writing a literature review is looking at the forest, not the trees.” —Prof. R. Paschal, George Mason University Law School

Organizing a Literature Most people find that the best way to organize a literature review is to use some sort of graphic organizer. This can be anything from a Venn diagram to a concept web to a table to . . . Well, pretty much anything can be a graphic organizer. The key part of a graphic organizer is that it’s graphic. It uses visual tools to display information. An outline is a form of graphic organizer, though a

Chapter 3  Doing Pre-research

fairly weak one: It shows, by how far items are indented, how far away they are from the main idea or how subordinate they are in the argument. Alas, many authors, though by no means all, find that outlines are not the most helpful of tools in the early stage of organizing a literature review. Outlines force you to think in a linear fashion—this, then this, then that, then the other thing. Graphic organizers like webs or Venn diagrams aren’t linear. Venn diagrams allow you to make groups; concept webs and mind maps explicitly push you to hunt for nonlinear relationships among the elements.17 In most of my papers, I go through several different stages of organizing in my literature. The first stage, as I discussed above, is simply keeping track of what’s happening in all of the different pieces and looking for patterns there. The second stage usually involves something like a Venn diagram, or at least separate piles/groups, usually using a technique like the sticky note one I describe below. Once I’ve figured out what authors and/or pieces go together, I usually then make something more like a web to get a better look at the relationships between the pieces in each group. The web will show which pieces or ideas are more central to the groups than others, because key ideas or arguments will be at nodes in the web and have lots of other things connected to them. Finally, I use what I learned from the web to put together a quick outline that will actually guide my writing. Sometimes this consists of simply stacking the sticky notes, index cards, or cover pages in the desired order, with an occasional sticky note to remind me of transitions to use, linkages to point out, or concepts I wanted to introduce. By the time I sit down to write, I’ve done so much thinking about the literature and how the pieces relate to one another that the actual writing is a pretty quick and painless affair. I’ve already figured out how all the parts fit together and what the connections are between them; all I have to do is simply capture it in essay form.18 My usual strategy for working through the organizational process is a variant on the time-honored tradition of using index cards for your bibliography and note taking. Back when I went to school, we were taught to put all our notes on separate index cards, and each bibliography entry on another index card, and then you organized your paper by making piles of related cards, or by laying them out in the order you wanted to address them. Technology is a bit more developed now—back then computers still used DOS, not Windows, and the average family didn’t have one at home—but I still tend to do this part of my thinking using old-fashioned tools. I’m very much a visual learner and thinker, so actually seeing all of it spread out is a big help for me. Your mileage from the technique I describe below may vary. Any method or technique that 17For more information on concept mapping, mind webs, and other forms of graphic organizers, see http://www.saskschools.ca/curr_content/biology20/concept_web.html or http://www.graphic .org/goindex.html. Both are intended for K–12 audiences and are very simplistic, but they will give you the basic idea.

71

72

Empirical Research and Writing

works for you is a valid tool to use, so don’t feel like you have to do this if you know this isn’t how you think. Begin by sticking a big piece of paper on the wall as a base—an old poster board, a chunk of cardboard, two cut-open-and-taped-together paper grocery bags, whatever.19 Write the (proposed) main sections or themes of your literature review on a bigger size (and/or different color) of sticky and put these on the base. Then, write the authors (usually authors, year, and perhaps a bit of the title) of each piece onto a smaller sticky note. If it’s a crucial piece of literature, you might want to put a star on it, or if you have several distinct literatures that you’re drawing together, you might put each of those on a different color of note or use a different color of ink. If you think of other key concepts linking groups of items together, or other ideas you need to include as transitions between pieces, grab another sticky. You can move the sticky notes around on your base and regroup them in different manners until you find one that you’re happy with. Changing the organizational structure is a snap—simply write another sticky note or two with the new headings and regroup the articles. If you’re concerned that you might want to return to a structure after you play with a new one, snap a couple of photos of it before you begin dismantling to help you restore it if necessary.

Actually Writing the [********]20 Thing Ultimately, no one can tell you how to write your literature review. It is, after all, yours, and your research question and your theory drive it. What I can do—and what your college or university’s writing center can help you do—is provide tips to help you avoid some common pitfalls. A good literature review clearly adheres to all three of these, and poor ones usually fail at all of them. First and foremost, allow yourself enough time to write and rewrite it. This is not a one-all-nighter assignment. You might have something done to turn in by the deadline if you do this, but chances are pretty good that you’ll have done all that work without really gaining much understanding, or making much progress on your final paper, as a result. And that defeats the purpose of having you write the literature review early on: The whole point of making you write a literature review prior to turning in your final paper is to help you make progress on that final paper. Sometimes—rarely—that progress is the words 18That said, it’s not quite that easy! As I mentioned above, I usually need two or three complete from-scratch drafts to get something I’m happy with. 19Use painter’s tape (looks like blue masking tape) to avoid damaging your wall. Most sticky notes adhere poorly to painted drywall or cinder block, so the base is an important part of keeping all the ideas up there at once. If you can’t or don’t want to put something on the wall as a base, use a table, patch of tile or linoleum floor, or something else of that ilk. The advantage of a stick-up base is that you can take it down and move it into another room to keep working; floors aren’t so portable. 20Insert

your choice of adjective here.

Chapter 3  Doing Pre-research

you wrote in that first submitted literature review. More often, it’s the ideas you develop during that drafting process that help the most. Second, redrafting often works better than editing. One of the hardest things for me to accept when I started writing literature reviews and other professional papers was that sometimes, you can do a lot better by starting over than by trying to force new ideas or information into an existing paper or section. I had a hard time accepting that I wasn’t “wasting” those pages by discarding them and starting again. Those particular 800 or 1,500 or however many words were being discarded, yes, but the important stuff— the ideas from them that I developed during that writing—were still definitely alive and kicking around in my brain, waiting for a chance to come out in a better fashion than they did in the previous edition.21 Think of it this way: When you wrote the initial draft, you had one particular set of points, one particular outline and set of main claims, in mind. As your thinking changes, as you come up with new ideas and realize the importance of other points and claims, the original structure no longer works best to convey those new ideas and points. Forcing the new ideas into the old structure undermines the points the original structure was trying to convey because too much other information—the new stuff—is crammed in between those main points. But it also undermines the ideas in the new material because those ideas are not being supported by an appropriate structure and supporting information. Redrafting is better than editing because it allows those more refined newer ideas to have a supporting structure and supporting information of their own rather than being forced into a structure that’s not really designed for them.22 Third, good literature reviews are about the literature—the forest—rather than the “trees” (individual pieces) within it. Strong literature reviews are about the overarching shape of the literature, and in particular where relevant holes, weaknesses, points of contention, or points of consensus are. Strong literature reviewers are also able to link those holes, weaknesses, or whatever to their own research and show how their research will fit into that literature. In other words, they provide a true scholarly context for the research, rather than simply indicating the state of current knowledge.

21Never throw out (or delete) drafts! They always contain interesting and useful nuggets of ideas for later, and things that you might be able to bring up at other points in the paper but not necessarily where you initially put them. Don’t delete all copies of the discarded text; simply dump it from your main paper into your Leftovers file for later reference. 22This

is often one of the first things faculty readers notice about a student empirical paper: We can tell whether the literature review was edited, or redrafted, based on how smoothly it reads. Edited ones are jagged and often jump around as the new information tries to shoehorn its way into a section not designed for it. Redrafted ones, on the other hand, read much more smoothly and often show other forms of evidence of having been through multiple rounds of thought and writing.

73

74

Empirical Research and Writing

A Sample Literature Review Excerpt Consider, for example, the following “excerpt” of a fictitious literature review for a paper on the age-old but still vexing question of “Public Perceptions and Scientific ‘Knowledge’: Which Came First, the Chicken or the Egg?”23 In their 1980 study, Smith and Jones prove the temporal primacy of eggs by considering evolutionary evidence from the fields of biology, paleontology, and biological archaeology. Their regression-based meta-analysis yields strong evidence for the primacy of eggs over chickens. They note in particular that chickens are unlikely to have evolved from non-egg-laying to egg-laying; such an evolution “would be inconsistent with all other evidence from all other species” (183). Thus, eggs must have existed for the first chickens to hatch from, because chickens that did not lay eggs but reproduced through some other manner are unlikely to have then evolved into egg-layers. Jones and Smith (1987) replicate and extend the Smith and Jones (1980) study. In this paper, they collect and analyze original data from a 3-year random sample of egg-producing and poultry-producing farms in the United States and Canada. Their analyses use both regression and cross-tabulation, and they continue to find strong support for the temporal primacy of eggs. Jones and Smith (1987) also present a replication of their (Smith and Jones, 1980) meta-analysis that includes research published between 1979 and 1986. It too continues to produce the same findings. Steven George, a professor at Harvard University, reaches a similar conclusion but from a very different starting point: the science of cloning (George 1993). In particular, he places special emphasis on the need for unusual germinating conditions for the cloning of avians such as chickens that mimic the conditions of an egg. No other type of animal requires such adjustments to the cloning process to achieve a successful clone. Other birds like ducks and ostriches require these special conditions too. This, George suggests, strongly implies that the egg must have existed prior to the emergence of the first chickens. Even under the best possible scientific conditions, avian evolution from stem and other cells requires an egg-like shell for successful reproduction. Finally, using data from the Producers of European Eggs and Poultry (PEEP), Little (1999) determines that Jones and Smith’s 23Of course, all names and citations in this section are fictitious as well. Focus on the form, not the content.

Chapter 3  Doing Pre-research

(1987) study, while methodologically flawed, nonetheless reached the correct conclusion. Little raises several important concerns about Jones and Smith’s (1987) methodology for their 3-year random sample. He argues instead that Jones and Smith misconceptualized the sample frame by including only farms producing chickens or eggs, but not those producing both. Little also claims that the use of three separate random samples, rather than a single 3-year panel model, weakens Jones and Smith’s ability to support their claims. He also criticizes their use of regression and cross-tabulation as being “unable to capture or correct adequately for the degree of endogeneity and simultaneity inherent in this research question” (1999, 1132–33). Little (1999) thus studies the entire population of egg and/or poultry producing farms, rather than a sample, across 4 years, and he analyzes the data using both negative binomial regression and factor analysis. His conclusions continue to support the claim that eggs existed prior to chickens. Smith, Jones, and Charles (2000) published a short rejoinder to Little’s (1999) critiques of their (1980, 1987) methodology. In their response, they take issue with some of Little’s claims, including the point about the conception of the sampling frame. They reanalyze their previous data using Little’s (1999) techniques, and find that evidence for the primacy of eggs is even stronger under this method. The coefficient is significant at 0.001, rather than 0.01 as in Smith and Jones’s 1980 study (Smith, Jones, and Charles 2000, 269). Finally, Smith, Jones, and Charles note that Little’s (1999) methodology itself does not fully correct for simultaneity or autoregression in the data. In a brief response, Little (2000, 278–83) reanalyzes his data in response to Smith, Jones, and Charles’s concerns, and he shows that his findings continue to hold and have better “diagnostic statistics” than Smith, Jones, and Charles’s (2000) own reanalysis. Finally, Chalmers considers the role of scientific uncertainty in his forthcoming book What Do We Know and How Do We Know It? His research finds that a majority of Americans (53%) accept the doctrine of egg primacy. Only 12%, however, are aware of scientific disputes about this topic, and less than 1% can name a reason for why the egg primacy argument or findings might be wrong (Chalmers forthcoming, 213). To most students, this looks like a fully adequate literature review: It reviews in some detail each of the major pieces of literature in the field. To an experienced reader (or writer) of literature reviews, on the other hand, this looks a lot like an early draft. What gives it away?

75

76

Empirical Research and Writing

Preview 3.1 The First Revision Imagine that you (or one of your classmates) wrote the literature review above. Consider the characteristics of a good literature review and evaluate this draft against those criteria. A. Allowed enough time to write and rewrite it. An experienced reader probably suspects that this is a first draft. The organization of this literature review follows the beads-on-a-string structure, with each paragraph generally devoted to a single piece of research. The organization is also chronological: Pieces appear in the order they were written. The result, in this case, is that the George article—which uses a totally different approach—is dropped in the middle of a discussion of four other pieces whose approach is all the same and that, in fact, are in dialogue with one another. B. Redrafted rather than edited. To the extent that this student has revised at all, the most likely approach was editing. The statement about ducks and ostriches in the George (1993) discussion is just stuck in there, not attached theoretically, thematically, or in any other way to the sentences before or after it. In the final paragraph, the statement about the significance levels sounds like someone besides the author read over the draft and asked, “how did they know” or “how much stronger” or something similar at that point in the paragraph. In response the author simply stuck in one sentence of evidence that is not particularly tied into the rest of the paragraph. C. Looks at the forest instead of the trees. To an experienced reader, this draft is 100% trees and 0% forest. Each paragraph is about another piece, which is a classic “tree” signal. Most sentences start with an author’s name or at least have the author as the subject or actor of the sentence—again, a classic indicator of a “trees” approach. This literature review presents a lot of information about the particular methods and findings of each paper, but it doesn’t provide a lot of information about the overall strengths and/or weaknesses of the literature, where the author’s own research fits into this discussion, or even what other literatures touch on this subject or closely related subjects.

The author of this draft is also unaware of some conventions in social science literature reviews. Authors’ university affiliations and book/article titles do not go in the body of the paper, nor do technical details of methodology. We

Chapter 3  Doing Pre-research

Talking Tips

Avoid using the word prove. By definition in the social sciences, we can never be totally sure that our answer is “right.” Some probability always exists that it might be wrong, no matter how strong the available evidence looks. Instead, try the following: • “Smith and Jones (2001) demonstrate that the relationship between chickens and eggs fails to hold when controlling for the presence of roosters.” • “Scholars have uniformly concluded that the relationship between chickens and eggs is strongly conditioned on the presence of roosters (Nichols 1980, 1998; Thompson 2003; Watson 2007).” Other useful verbs include show, note, identify, assert, and find. We talk about finding support for a hypothesis, or the data supporting the hypothesis; we might also have evidence that casts doubt on, counters, or undermines existing arguments or claims about a relationship.

also cannot prove anything in the social sciences—see the Talking Tip for alternate ways to express certainty and uncertainty. So, now that we know what the weaknesses of this initial draft are, let’s see what we can do to fix them.

A Good Revision Smith and Jones (1980) brought the scientific discussion of chickens and eggs into mainstream debate (Chalmers forthcoming). Their analysis of dozens of other studies from a range of fields provided strong evidence for egg primacy. Smith and Jones conducted several follow-up analyses, including those by Jones and Smith (1987) and Smith, Jones, and Charles (2000), with similar results. Little (2000) disagrees with Jones and Smith on a number of points, including methods and data. Nonetheless, his (2000) study of European egg and poultry data provides results that continue to corroborate Smith and Jones’s claims. George (1993) reaches similar conclusions based on the science of cloning. In particular, he emphasizes the need for unusual germinating conditions for the cloning of avians such as chickens. Even under the best possible scientific conditions, avian evolution from stem and other cells requires an egg-like shell for successful reproduction. No other type of animal requires such adjustments to the cloning process

77

78

Empirical Research and Writing

to achieve a successful clone. This, George suggests, strongly implies that the egg must have existed prior to the emergence of the first chickens because the emergence of chickens is otherwise inexplicable by current science. George’s (1993) theory and claims are logically sound, though the absence of empirical evidence explicitly supporting his argument weakens it substantially. Chalmers considers the role of scientific uncertainty in his forthcoming book What Do We Know and How Do We Know It? His research finds that a slim majority of Americans (53%) accept the doctrine of egg primacy; 46% accept chicken primacy and 1% are not sure. Only 12%, however, are aware of scientific disputes about this topic, and less than 1% can name a reason for why the egg primacy argument or findings might be wrong (Chalmers forthcoming, 213). To an experienced reader, this is definitely not a first draft. This literature review indicates that the author recognizes some patterns and groupings among the items discussed—for example, that the articles from Jones and Smith, and from Little, are connected to one another by more than just topic. George (1993) and Chalmers (forthcoming) are not part of that group, though, and the author is still uncertain how to handle them. They’ve been given “tree” treatment, each with a paragraph of its own, and those paragraphs are really just choppy reedits of the originals. The paragraphs—and most sentences—still have authors as subjects. Chalmers’s book title doesn’t go in the body, either. We’re also not getting much context here, especially about how this literature relates to the author’s research question about public perception and scientific knowledge. This is a lot better than the first draft, but it’s still got room for improvement. With that, let’s look at . . .

A Better Revision Proponents of egg primacy draw on arguments made by scholars in the fields of biology, paleontology, and biological archaeology (Chalmers forthcoming; George 1993; Smith and Jones 1980, 1987). The pioneering meta-analysis of Smith and Jones (1980) assembled what had been a disparate set of unrelated studies into a single coherent research question, and it provided clear evidence in favor of egg primacy. Later analysis by them and others continued to find strong and consistent support for egg primacy across a wide range of datasets and methods of analysis (Jones and Smith 1987; Little 1999, 2000; Smith, Jones, and Charles 2000). Despite disagreement among scholars themselves on methodology (see especially Smith, Jones, and Charles 2000 and Little 2000), findings in this line of research have led to a general consensus among the public that eggs definitely came first (Chalmers forthcoming).

Chapter 3  Doing Pre-research

79

How does the “better” version differ from the original and “good” versions? First, it’s very succinct. It hits the key ideas—broad range of source disciplines, no significant disputes about the findings, persistent disputes about methodology—in a single focused paragraph. It contains no extraneous information about specific datasets, particular methodological points, or even individual articles’ findings. What matters here is not the individual articles themselves but their overall findings as a literature, a set of related research claims and findings. The writer of this paragraph has put considerable thought into •• locating the similarities and differences among the articles, as evidenced by the comments about consistent findings across datasets and analysis methods, and about substantive agreement but methodological dispute; •• understanding how the pieces relate to one another and to broader fields of research, as evidenced by the remarks contextualizing this literature as drawing from a range of disciplines and having broad public support, and identifying that Smith and Jones’s (1980) work was pioneering; and •• setting up a link to his or her own research question, as evidenced in the remark about the achieving of public consensus without scientific consensus; we presume this is a transition to another paragraph moving in that direction. In short, this paragraph does an awful lot in a very small amount of space. It gives us a very good sense of how the “forest” looks, without especial focus on the trees. Yes, this is the third draft, but believe me, no words here are wasted—not even the ones we “discarded” from the earlier drafts.24 Is this the best possible draft of the literature review? Of course not. No such thing really exists. In a full paper on public perception and scientific knowledge, we’d want to know much more about both the breadth of public perception of this supposed “fact” and its controversy, and More Practice we’d probably want to know about other literatures that relate to these. Literature Reviews You have probably noticed, however, that this literature review is for American, substantially shorter than the starting draft. This is always true. Alas, Comparative, and this means that a good literature review requires a lot of individual International pieces to achieve any significant length. The starting point of “how Politics many pieces do I need” is only a starting point. As your expertise with literature reviews (and with the literatures of interest to you) grows, you will become less concerned with “how many” and focus more on how well you’ve managed to integrate the pieces into a whole and then refocus the discussion of the literature on your particular concerns or interests. Quality here definitely trumps quantity, but as always, quality takes time. 24Again,

don’t delete! Drop the excised text into your Leftovers file and continue.

80

“

Empirical Research and Writing

Quips and Quotes

“I would not have made this [letter] so long except that I did not have the leisure to make it shorter.” —Blaise Pascal, French mathematician, 1660

Practice 3.1 Chicken and Egg Redux Read the (fictitious) literature review titled “In Defense of the Chicken” from the ERW Companion Website http://study.sagepub.com/powner1e. Repeat the ‘good’ and ‘better’ drafting process outlined in the Chicken and Egg example.

Summary This chapter introduced the basic parts of a standard empirical paper— introduction, literature review, theory and hypotheses, analysis, and conclusions. We then homed in on the first one that most people write, the literature review. The literature review is a unique genre of writing with its own conventions and expectations. Good literature reviews are multistage processes that show evidence of redrafting and that place literature in a context that allows the reader not only to understand the shape of the forest—the entire body of literature—but also to locate the author’s claims within in that literature.

Key Terms •• Scholarly literature •• Peer review process (doubleblind) •• Interlibrary Loan (ILL) •• JSTOR

•• Bibliography-hopping •• Social Sciences Citation Index (SSCI) •• Annotated bibliography •• Graphic organizer

B

Choosing a Design That Fits Your Question

4

y now, you have established a research question, created a theory to answer your question, identified observable implications, and developed testable hypotheses to see if your answer is right. This chapter moves into the next stage of going from idea to paper: identifying an appropriate research design to test your hypotheses. We have two main groups of empirical research tools, qualitative and quantitative. The choice between these, and between different techniques within these groups, depends on your research question and hypotheses. So let’s get two common misperceptions out of the way. First, very few research questions explicitly need one form or another. We can test virtually all hypotheses using both quantitative methods and qualitative methods. If you’ve already got a particular method or research design in mind—for example, you’ve already convinced yourself that you need to do a case study—you should discard that thought immediately and hold off on making a decision until you know more about both forms. Second, there is no such thing as “a math person.” Even if there was, it would have no bearing on your choice of methods or techniques.1 The appropriate technique to choose is a function of your research question and the specific hypotheses you intend to test from it. To choose a tool—a particular dataset, a specific case—and then go find a research question to answer with it, is bad science. This chapter begins, then, by discussing the different types of hypotheses that we might have as potential answers to a research question. We then consider 1Americans insist on treating math as a talent. The rest of the world treats math as a skill that all people can learn, much the way we treat reading. All individuals are expected to be able to achieve a rather high standard of proficiency in it unless they have some significant mental impairment. (In fact, the global standard for illiteracy is reading below an eighth-grade level.) But Americans think that math is something you have to have a talent or gift for—we talk about “not being a math type of person” or “not being good at math.” This is a large pile of baloney. Get over it now. Yes, ladies, I’m talking to you. Yes, I’m also talking to the guys, but we women are much more likely to believe this about ourselves, partly as a function of how math is taught in the United States and partly as a function of natural gender differences in brain development through the school years. We’d never let someone choose a job or an assignment or a course because they’re “just not a reading type of person,” or because they’re “not good at reading,” meaning that they think they can’t and/or refuse to try. Our insistence on treating math this way just exposes the ridiculousness of how we Americans think about it. Numeracy is just as fundamental as—and no more difficult than—literacy.



81

82

Empirical Research and Writing

the differences between, and advantages of, qualitative and quantitative designs. Only after we’ve completed these last preliminary steps can we make a decision about which type of research design we need. No hard-and-fast rules exist for choosing designs, because we can test virtually all theories with both. The final section previews a range of different research designs, highlights their strengths and weaknesses, and offers some general guidelines for helping you decide which will be a good fit for your particular theory and hypotheses. The next several chapters provide more specific guidance for each family of techniques.

Types of Hypotheses Hypotheses form the basic cogs of social science research. Research questions, literature reviews, analyses, and all the rest ultimately occur because the author has some hypothesis or set of hypotheses that she or he wants to test. Hypothesis testing is the business of social science. Virtually all types of potential causal explanations fall into one of six hypothesis types. As you theorize about your own paper’s argument, you will want to keep these types in mind. Try phrasing your argument as each of these. Which sounds right?

Four Probabilistic Hypothesis Types Four types of hypotheses make claims about general, on-average-acrossmultiple-cases relationships: directional hypotheses, hypotheses of no effect, relative hypotheses, and interactive hypotheses. Let’s look at each in turn.

Directional Hypotheses Directional hypotheses make a claim about the effect of some cause variable, X or the independent variable, on some outcome or dependent variable Y. More specifically, a directional hypothesis articulates the effect on Y of increases in X; these are the type of hypotheses we examined in Chapter 2, in our discussion of moving from a theory to a hypothesis. Directional hypotheses usually take the form of H: The effect of X on Y is positive. [or] As X increases, Y also increases. —or— H: The effect of X on Y is negative. [or] As X increases, Y decreases. This is the most common type of hypothesis. We are trying to understand how changes in X (or different values of X) produce some type of change in (or different values of) Y.2

2If

this sounds awfully like the idea behind the slope of a line to you, you’re on the right track.

Chapter 4  Choosing a Design That Fits Your Question

Normally, the X’s that we think of first for these types of hypotheses are continuous variables, of the interval-ratio type. Interval-ratio variables are measured at the highest (most precise) level of measurement—they contain, implicitly or explicitly, some type of unit that makes an increase in the variable a one-unit increase. We can talk meaningfully of “a one-unit increase” for variables measured at this level.3 Some examples of interval-ratio level variables might include percentage of the popular vote for a particular presidential candidate, number of parliamentary seats reserved for a particular social group (like women or a religious or ethnic minority), or duration of a war in months. In all three cases, we can talk meaningfully of increases or decreases because the unit provides a constant benchmark for the interval between the numbers. One percent, for example, tells us “one more per 100 people,” whether that additional percentage is from 17% to 18% or 93% to 94%. The difference between three seats and four seats is the same as the difference between 15 and 16 seats: one seat. So creating hypotheses about interval-ratio variables is relatively straightforward: We can picture a one-unit increase in an interval-ratio independent variable (IV) and think about its effect on an interval-ratio dependent variable (DV). Most variables that students think of off the tops of their heads are measured at the interval-ratio level. Any time you can say “number of,” or attach a unit to the measurement (dollars, deaths, days, etc.), you’re in the realm of interval-ratio measurement. This is not the only way to measure variables, though. Some concepts of interest do not come in nice, countable, scalable units. They come in a more amorphous shape like “more” and “less” of a characteristic. For example, a survey question about the president’s handling of the economy might have as possible answers “strongly disapprove, disapprove, neither approve nor disapprove, approve, strongly approve”—a set of responses known as a Likert scale. These five possible answers can clearly be ranked in an order from least to most (or most to least) approval; there is a definite sense of increase in the ordering. The same is true for a variable measuring frequency of occurrence for certain war crimes: no record, minimal or incidental, frequent, widespread. Because the variables’ values can be ranked, or ordered, we call them ordinal variables. Ordinal variables are also pretty easy to hypothesize about because we can still meaningfully talk about increases or decreases in the variable’s value, even if we can’t make specific claims about the magnitude of the increase or decrease.4

3Interval-level and ratio-level are different, but for most social scientists, this distinction is irrelevant because we treat both types of variables identically in our analyses. 4In quantitative work, these kinds of variables’ values often receive numeric indicators—“strongly disapprove” might get a code of 1 and “strongly approve” a code of 5. These values are simply shorthand—codes—for the true value of the response. The numbers themselves don’t mean anything. Someone with a code of 4 does not exhibit precisely twice as much approval as someone with a code of 2.

83

84

Empirical Research and Writing

Our final level of measurement, our final type of variable, is the nominal level of measurement. Nominal variables are categorical in nature. These can be many categories (race or ethnicity in the United States, for example), or they can be yes/no or on/off variables such as “the European Union issued a declaration” or “aid recipient is a former colony of donor.” When a variable of this type has just two possible outcomes, we often refer to it as dichotomous or binary. For dichotomous (“divided into two parts”) variables, a value of “1” (the standard “yes” or “true” code for most quantitative work and qualitative shorthand) simply means that this category applies or the yes/no thing occurred. The number value, if a number value is used, has no numeric meaning; it just means “yes” or “true” or, for categorical variables, “this category.” For multivalue or polychotomous indicators, it simply indicates which value the variable takes in that case. Because the numeric values are placeholders rather than actual values, we cannot sensibly talk about “increases” in the variable’s value. We can talk about with or without the presence of X, or when there is or isn’t X, or X’s presence or absence, but we can’t talk about increasing or decreasing X. A hypothesis about a dichotomous IV might sound like “The presence of X increases the value of Y.” A hypothesis about a dichotomous DV normally talks in terms of increasing the probability of observing Y: “As X increases, observing Y is more likely.” For polychotomous IVs, we can talk about categories: “When X is blue, then Y is higher.” For polychotomous DVs, we can talk about movement between categories or the probability of a certain category: “As X increases, we are more likely to observe instances of Y1,” or “As X increases, cases of Y1 should outnumber cases of Y2.”

Preview 4.1 Being Level-Headed A. Education. Consider a hypothesis about how level of education affects a respondent’s income. We could measure education as a nominal variable: college graduate/not a college graduate, for example. This is a yes/no categorical variable. We could also measure education as an ordinal variable: less than high school, high school diploma, some college, associate’s degree, bachelor’s degree, some graduate school, graduate degree. Finally, we could measure education as an interval-ratio level variable, that is, a specific number of years (23, in my case). Which should we choose? For most purposes, what matters is attainment of a degree, which may take varying numbers of years. Most scholars using respondent’s level of education as a variable thus use an ordinal measure rather than something more fine-grained like the interval-ratio version.

Chapter 4  Choosing a Design That Fits Your Question

B. Commission of war crimes. War crimes are violations of the laws of war established by the various Geneva Conventions. These include, for example, killing civilians, mistreating prisoners of war, and using chemical or biological weapons. Consider a hypothesis about how violations by one party in a war affect violations by the other party (Morrow 2007; Morrow and Jo 2006). We could measure this as a nominal variable: violations did/did not occur. We could also measure this using an ordinal scale: no recorded violations, minimal or incidental violations, frequent violations, widespread violations. Finally, we could measure this with an exact count of recorded violations. Which is best for this case? The nominal variable could be problematic since most wars exhibit at least a low level of war crimes. The war crimes variable would thus be “yes/1” for all cases; “explaining” violations by all parties against all other parties would lack variation and be a rather boring (and short) story. Measuring with an exact count is also problematic. States don’t announce or publicize their war crimes violations, particularly not in most 20th century cases (before the Internet). So we would not have a full list to consult, and since history is written by the winners, we are likely to find a very biased set of reports. Also, most researchers agree that war crimes violations are not a linear phenomenon. The difference between 3 and 10 civilian deaths, for example, is very different than the difference between 5,003 and 5,010 deaths. An interval-ratio scale would thus be biased in its counts, and it would miss the distinction that 3 to 10 deaths is likely to have been one single incident whereas 5,003 to 5,010 deaths likely indicates widespread commission of violations and a national policy of using violations to achieve policy goals. Practice 4.1 Level of Measurement For each hypothesis below, identify the DV and IV, and then indicate the level of measurement (LOM) for each. If more than one way of measuring a variable exists, identify one and its level of measurement, and jot a brief note about how and why you operationalized the variable as you did. A. States that previously abused human rights are more likely to condemn human rights violations in other countries than states with no history of human rights abuse. DV: LOM: IV: LOM:

(Continued)

85

86

Empirical Research and Writing

(Continued) B. Highly ambitious bureaucrats are more likely to pursue innovative policies than are less-ambitious bureaucrats (Teodoro 2011). DV: LOM: IV: LOM: C. Countries that use different types of electoral systems for national and subnational elections are more likely to experience citizen perceptions of low government accountability than states using a single electoral framework. DV: LOM: IV: LOM: D. As education level increases, the probability of voting for Democratic candidates decreases. DV: LOM: IV: LOM: E. Countries with many veto points in their policy-making systems are less likely to be able to respond to policy crises successfully than countries with fewer veto points (Tsebelis 2002). DV: LOM: IV: LOM: F. Members of Congress who sit on powerful regulatory committees will receive more campaign donations from corporations than those who do not sit on powerful committees. DV: LOM: IV: LOM:

Talking Tips

Good directional hypotheses clearly identify a than—more likely than what. Lower than what? Direction is always relative to something. Few things irritate knowledgeable readers more than directional hypotheses where the reference (“relative to”) group is not clear! Be sure that the discussion around your hypothesis clearly indicates this reference category.

Chapter 4  Choosing a Design That Fits Your Question

87

Hypotheses of No Relationship A “hypothesis of no relationship” is a bit of a strange thing. Usually, a hypothesis of no relationship is the “null hypothesis,” especially for researchers working in a quantitative framework. We may, however, have reasons to believe that errors in prior research—especially failing to consider an alternative explanation—led to researchers finding spurious (artificial) relationships that do not actually exist. In these cases, we would hypothesize that no relationship exists between the initially posited variables—that X does not systematically affect Y. In quantitative terms, a hypothesis of no relationship implies that the coefficient on the variable of interest is not significant—that is, that the observed relationship could have occurred by chance.5 In the literature on the democratic peace, for example, Joanne Gowa (1999) and Examples of others have argued that the absence of wars between democracies is Hypotheses of No simply a statistical fluke, not a true causal relationship. The relative Relationship from rarity of democracies in the international system before about 1920— Comparative, that is, for more than half of the years our most common datasets American, and cover—combines with the relative rarity of wars themselves to make International war between democracies incredibly unlikely in any given year. Any Politics observed relationship between wars and democracies is thus spurious: The joint probability of two low probabilities is an incredibly low probability, so the absence of conflict between democracies could easily be the fault of random chance rather than some systematic effect. Relative Hypotheses A relative hypothesis makes a claim not about the variables themselves but about the relationship between their effects, relative to one another. The direction of each X’s relationship with Y matters, but the magnitude of those effects is also of interest. One common relative hypothesis is that the effect of a particular IV, let’s call it x1, is greater than the effect of a second IV, x2. Other versions might be that the effect of x1 is less than, or equal to, the effect of x2. In these cases, we are interested not only in the direction of the effect, but also in the relative magnitude of those effects. Explicit relative hypotheses are not very common in the quantitative literature, but they do exist. In a study of political advertising and candidate support, for example, Valentino et al. (2002) consider whether narrative cues have a larger effect than simply visual cues in priming respondents to consider race. Explicit relative hypotheses are somewhat more common in the qualitative literature. Researchers have considered the effects of different kinds of social structures on local public goods provision in China (Tsai 2007), and patterns of decentralization on subnational power in Latin America (Falleti 2005). Careful process tracing, the use of counterfactuals, and other techniques allow researchers to gauge the relative magnitude of effects even without explicit numeric effect sizes. 5More

on significance and coefficients below.

88

Empirical Research and Writing

Conditional Hypotheses All of the hypothesis types we have considered thus far are ones about the influence of single independent variables directly on the outcome variable. In an interactive or conditional hypothesis, the researcher argues instead that the effect of one IV, x1, is dependent on the value of a second IV, x2. Conditional hypotheses involve three variables instead of our usual two. For a practical example, consider the following hypothesis: When individuals get caught in the rain, they open their umbrellas. Our outcome of interest, the deployment of an open umbrella, results from experiencing rain. That statement appears, at least on the surface, to be a complete and accurate description of events. Imagine, though, a warm summer day with a sunny forecast. Not a cloud is in sight . . . until about 4 PM, when some dark ugly clouds form on the horizon, and one of those pop-up thunderstorms brings drenching rain. Our hypothesis says we should observe umbrellas opening in the hands of all those folks caught outside in the storm. Will we see umbrellas now? The answer to that is, probably not. Our hypothesis, alas, lacks a very important factor that conditions the effect of rain on umbrella appearance. We forgot about the part where nonappearance of umbrellas could result from not having an umbrella to deploy. With a nice forecast, very few people have umbrellas with them. So the choice to open an umbrella is conditional on having an umbrella to open. An individual would only open an umbrella if he or she was caught in the rain and also had an umbrella. In a parallel manner, simply having an umbrella would not trigger an individual to open it unless rain also occurred. Another way to state this effect is that the presence of rain and the availability of an umbrella interact to produce the outcome of interest, an opened umbrella during the rain. The outcome of interest only occurs if both of the causes are present. An easy way to write this in hypothesis notation is to use multiplication. Since both presence of rain (Rain) and availability of an umbrella (Umbrella) are yes/no variables, we can treat values of “no” as 0 and “yes” as 1. When we multiply them together, their product is 1 only if both variables equal 1. + open umbrella  + (Rain * Umbrella) If either of the components is not true—it’s not raining, or the individual doesn’t have an umbrella—we predict that we won’t observe an open umbrella (i.e., the value of open umbrella won’t increase; it will remain 0). The righthand side of our hypothesis model doesn’t increase unless both are true, so that’s the only condition under which we’d expect to observe an open umbrella. This term on the right, Rain * Umbrella, is known as an interaction term. Conditional hypotheses and the interaction terms used to test them are quite common in every subfield of political science. Citizens will not pay taxes

Chapter 4  Choosing a Design That Fits Your Question

unless they feel that they risk some punishment for failing to do so and that the government demanding taxes and doing the punishing is legitimate (Levi 1997). Choosing judges by election in a state with the death penalty, and having public support for the death penalty, results in judges who have a substantially more conservative ideology than we would otherwise expect (Brace and Boyea 2008). Despite predictions that citizens of wealthier democracies should be more tolerant, Andersen and Fetner (2008) find that the effect of national wealth on attitudes toward homosexuality is conditional on income distribution within the society. As countries get wealthier, tolerance generally increases, but as economic inequality increases within wealthy states, tolerance decreases. Leaders are more likely to be punished for the outcome of a war if they both lose the war and were responsible for the state’s involvement in the war (Croco 2011). Talking Tip

Avoid using to impact as a verb. (My high school English teacher always said, “The only things that are impacted are teeth.”) Alternative verbs include influence, affect, cause, induce, depress, increase, produce, etc. For interactive hypotheses, you could also say that one variable magnifies, enhances, modulates, or modifies the effect of another variable. Note that effect with an “e” is usually a noun.

Two Deterministic Hypothesis Types The hypotheses we discussed above were probabilistic—that is, they assumed that on average across a large population, we would observe a particular relationship. These probabilistic hypotheses recognize that some few cases will be anomalous and will not fit our hypothesis. Deterministic hypotheses, on the other hand, seek conditions that always hold. Such absolutes are relatively rare in most of the social world, but they do exist, as do hypotheses about them.

Necessary Conditions A necessary condition asserts that some outcome Y cannot happen without the occurrence of some causal or prior variable X. The easiest way to think about what this means is to look at a simple two-by-two table. Imagine an X variable and a Y variable, each of which is dichotomous (i.e., has only yes/no, on/off, 1/0 values). The values in the cells show the number of cases in which the X variable was or was not present, and the corresponding value of Y.6 6In accordance with convention, I list the X variable on the horizontal axis. In a table, this means the columns, with the lowest value farthest left. The Y value is on the vertical axis, which means the rows, again with the lowest value at the “origin” of the table.

89

90

Empirical Research and Writing

We hypothesize that X is necessary for Y. If X is a necessary condition for Y, then we should not observe any cases of success without also observing X. TABLE 4.1  Data Showing a Necessary Condition X variable   Y variable

0

1

0

17

6

1

0

12

Let’s think about what the data should look like if our hypothesis is correct. If X is necessary for Y, and we observe Y = 1, then we should also observe X = 1. Conversely, if we observe X = 0, then we should not observe any cases of Y = 1; the cell for (X = 0, Y = 1) has to be empty if X really is necessary for Y. Looking at Table 4.1, we see that for the data shown here, X is indeed necessary for Y. If we look at the bottom row of the table, where Y = 1, we observe no cases of Y = 1 where X = 0. We only observe cases of Y = 1 where X = 1.7 The key thing about a necessary hypothesis is that it’s interested in predicting cases of Y = 1; it’s a story about the DV. One of the best-known examples of necessary conditions in the political science literature is Theda Skocpol’s (1979) work States and Social Revolutions. She argues that state breakdown and peasant revolt are both necessary conditions for the occurrence of a social revolution, that is, a revolution that changes the entire order of society, not just the government structure. Skocpol tests her argument with process-tracing analyses of three major social revolutions (French, Chinese, and Russian), sprinkling it liberally with other brief cases to present counterfactual analyses and cases of alternate variable values. While later scholars have cast some doubt on Skocpol’s work, at least in part because of her case selection (e.g., Collier and Mahoney 1996; Geddes 2003; Mahoney 1999; Sekhon 2004), States and Social Revolutions remains one of the few major works to tackle hypotheses of necessity. As we’ve discussed before, the social world is a messy place, with many factors contributing to any specific outcome. Most social phenomena exhibit equifinality, meaning that more than one route to a particular outcome exists. Arguments of necessity are thus difficult to support, but nonetheless, they do exist and scholars do test them.

Sufficient Conditions A sufficient condition asserts that some cause X always leads to the occurrence of some outcome Y. Again, let’s picture a two-by-two table of 7We don’t need to look at the top row of data for a “necessary” hypothesis. The distribution of cases

where Y = 0 is entirely not of interest; we’re only concerned about cases where Y occurred (Y = 1) (Braumoeller and Goertz 2000, 846).

Chapter 4  Choosing a Design That Fits Your Question

dichotomous variables, with outcome counts as the cell entries. We hypothesize that X is a sufficient condition for Y. TABLE 4.2  Data Showing a Sufficient Condition X variable   Y variable

0

1

0

6

0

1

3

4

If X is, indeed, a sufficient condition for Y, then we should expect to find no cases of X = 1, Y = 0, that is, no cases should exist where X was present but Y did not occur. Table 4.2 shows this; the cell for X = 1, Y = 0 is empty. A hypothesis of sufficiency is, like its necessity counterpart, a fairly rare thing. Unlike necessary conditions, though, sufficient conditions allow for multiple routes to the outcome of interest. Observing X will automatically produce Y, but the possibility of observing Y without X exists. A hypothesis of a necessary condition precludes alternate routes to observing Y; Y can only happen if X happens. To think about this distinction a different way, a sufficient condition is a story about an IV, specifically about its presence. We are not interested, in this case, in cases where X = 0. For a hypothesis of sufficiency, Y may occur even without X’s presence; the hypothesis says nothing about other ways that Y may or may not occur. The key thing of interest is that when X = 1, Y always occurs.

Necessary and Sufficient Conditions Hypotheses of joint necessity and sufficiency are possible. In these hypotheses, X is both necessary and sufficient to produce Y. Where X occurs, Y will always occur, and no other way exists to obtain Y without X. In terms of our two-by-two table, a necessary and sufficient condition has no cases where X = 1, Y = 0, and no cases where X = 0, Y = 1. Events are observed only in the main diagonal of the table, where X = 0, Y = 0, and X = 1, Y = 1.8 Table 4.3 shows data that support a hypothesis of a necessary and sufficient condition. For hypotheses of joint necessity and sufficiency, both the IV and DV typically must be dichotomous. When considering hypotheses of necessary or sufficient conditions, however, we can easily expand this discussion from dichotomous variables to ones of degree—high, medium, low, and no value of some characteristic, for example, or few, some, and many. In theory, we can 8The main diagonal of a (square) table always runs from top left to bottom right, where X = Y. If you do not have a square table—if the number of rows is not equal to the number of columns—you cannot use the language of the main diagonal and the off diagonal (the diagonal going from top right to bottom left) to describe your table’s contents.

91

92

Empirical Research and Writing

TABLE 4.3  Data Showing a Necessary and Sufficient Condition X variable   Y variable

0

1

0

11

0

1

0

9

expand it all the way to interval-ratio variables, where it turns into a threshold effect. In this, we identify through inductive or deductive means some threshold over which we consider X to be “present” and any values below that, X is effectively absent. For example, the international policy community has a widely held belief, supported by quite a bit of research, that a national income of at least $5,000 per capita (in 1990s US$) is a necessary condition for democracy to be stable. This is not, however, a sufficient condition; a wide range of countries have GDP over US$5,000 but lack democracy. Talking Tip

Avoid using the terms necessary, sufficient, and/or significant in your hypotheses unless you really actually mean that and intend to demonstrate that in your paper.

Preview 4.2 Characterizing Hypotheses H: As a political candidate shifts resources from policy-based advertising to emotion-based advertising, overall support for the candidate will not change, but supporters and opponents alike will become more extreme in their support for the candidate. This is a tricky question—three separate testable hypotheses are in here. First, let’s consider that support is a dichotomous ordinal variable and that advertising spending is measured in the difference in percentage of the advertising budget directed toward ads of each type. Then we have Support  (PolicyAd$% – EmotAd$%), meaning, we expect no significant change in support even as our X variable, the difference in policy ad spending and emotional ad spending, becomes

Chapter 4  Choosing a Design That Fits Your Question

negative. (Negative values of the IV here indicate that emotional ad spending is greater than policy advertising spending.) This is a hypothesis of no relationship. We’re not done there, though. The second and third hypotheses are + StrengthofSupport  (PolicyAd$% – EmotAd$%) if R (Respondent) is supporter, and – StrengthofSupport  (PolicyAd$% – EmotAd$%) if R is opposed. This presumes that Strength of Support is an ordinal variable, of the “strongly opposed, opposed, neutral, support, strongly support” (i.e., Likert scale) type. Hypotheses 2 and 3 are thus directional hypotheses.9 Practice 4.2 Characterizing Hypotheses Write each of the following word-hypotheses in our standard hypothesis format. Identify the IV, DV, and LOM for each. (You may need to indicate operationalization.) Then, determine which of the six types of hypotheses each represents. Again, if you perceive more than one way to characterize the hypothesis based on operationalization or anything similar, jot a brief note to explain your reasoning. (HINT: More than one DV or IV may be involved in each hypothesis.) A. Mass-movement-based “revolutions” such as the Color Revolutions and the Arab Spring only emerge after the occurrence of national electoral fraud (including rigged elections). DV: LOM: IV: LOM: Hypothesis notation: Hypothesis type:

9These two hypotheses could be written as a single conditional hypothesis, + (PolAd$ – EmoAd$) + RSupport + (RSupport*(PolAd$ – EmoAd$)), but this formulation is not as intuitive as the two-part version presented above. If you already understand the logic of regression, I recommend you verify for yourself that the two statements (the one in the text and the one in this note) are equivalent.

(Continued)

93

94

Empirical Research and Writing

(Continued) B. Political corruption cannot occur in the presence of robust civil society. DV: LOM: IV: LOM: Hypothesis notation: Hypothesis type: C. System-changing interstate war is most likely when a challenger state is both dissatisfied with the current international status quo and at or near parity in military capabilities with the dominant state (Organski and Kugler 1981). DV: LOM: IV: LOM: Hypothesis notation: Hypothesis type: D. World War I occurred because policy makers perceived that offensive military technology was dominant, when in reality defensive technology was dominant (Christensen and Snyder 1990). DV: LOM: IV: LOM: Hypothesis notation: Hypothesis type: E. The presence of divided government should not affect the proportion of references to liberal and conservative economic policy in US presidential State of the Union addresses when compared with unified government under the same president.

Chapter 4  Choosing a Design That Fits Your Question

DV: LOM: IV: LOM: Hypothesis notation: Hypothesis type: Progress 4.2 Characterizing Your Own Hypotheses Return to the hypotheses you developed in Chapter 2 and refined during your literature review reading. Analyze and characterize each of your hypotheses as we did in Practice 4.2.

What Type of Analysis Should I Conduct? To be clear, qualitative and quantitative refer primarily to data types. We use the terms as shorthand to reference classes of analytical techniques. The first big question to answer is which of those families or classes of analysis tools is the right one for your question; the second task is then to decide what specific analytical tool you will use to analyze your data.10 This latter question almost always involves consultation with instructors, classmates, or other colleagues— even for seasoned researchers with lengthy publication histories. Two heads are better than one. Don’t be afraid to talk about tool selection with your professor(s), classmates, or other available and reasonably knowledgeable people. The ability to know what you don’t know, and to seek the input of others, is a key skill for success in the real world.

Similarities and Differences in Qualitative and Quantitative Methods Let’s begin by dispelling a few basic misconceptions. Both quantitative and qualitative methods are “scientific,” whatever you might mean by that (Jackson 2010). Both can—and have—been used to great effect to make substantial 10If an instructor or classmate asks you what kind of analysis you are conducting, “quantitative” or “qualitative” are not viable answers. The inquirer wants to know the specific tool from within these families that you intend to use. If you cannot actually specify which technique you are using, you are not yet ready to begin analysis. This is particularly true for researchers using qualitative tools. For quantitative research, case selection and data collection are (usually) the same no matter what particular analytic technique you use. For qualitative techniques, case selection principles differ radically across the various techniques, and the data needs are quite diverse as well. This means that data collection cannot start—and analysis certainly cannot start—until the researcher has clearly identified the analytical technique he or she intends to use.

95

96

Empirical Research and Writing

contributions to scholarly knowledge about systematic patterns in politics and international affairs. Both are capable of supporting claims of causality, though their mechanisms for doing so differ. In short, neither is inherently “better” than the other in some overall sense. Each has some strengths and some shortcomings, and each is better for supporting some types of claims than others. The trick is matching the particular hypotheses you have to the strengths and weaknesses of the methods. You want to choose a method that is strong in the type(s) of hypotheses you have, and whose weaknesses you can address through an additional component of research (even if hypothetical) or via a secondary analysis of your existing data. Both qualitative and quantitative (positivist) methods work by marshaling carefully selected pieces of information—data—in a particular format specified by the particular technique at hand. Both sets of techniques include provisions that prohibit or impede cherry-picking the data, that is, selecting only those pieces that support your claim and ignoring the rest. Both are equally credible when researchers follow their approach’s prescriptions and they clearly articulate and document their research decisions. These crucial similarities mean that, generally, we can apply the same standards for evaluation, and by extension, conclusions about credibility, to both qualitative and quantitative research (King, Keohane, and Verba 1994). Table 4.4 considers the three characteristics of desirable social-scientific research that we discussed in Chapter 1: The research is conducted in a transparent manner, its claims are generalizable, and it often speaks to claims about causality. Both qualitative and quantitative analytical approaches can meet all of these criteria when conducted in adherence with scholarly norms about data collection, analysis, and presentation. A major purpose of the later chapters of this book, then, is to introduce those norms and to help you become comfortable with them so that you can produce strong research. TABLE 4.4  How Desirable Research Characteristics Are Achieved Characteristic

Qualitative

Quantitative

Transparency of data collection and analysis

Primary sources, extensive bibliographies, counterfactual reasoning

Publicly available datasets, codebooks, documentation of sources, replication materials

Generalizability

Careful selection of cases to provide hard tests, depending on the specific analytical technique

Inclusion of all available or possible cases, with special techniques for rare, missing, censored, or irregular outcomes

Demonstrating causality

Counterfactuals, counterexamples, thought experiments, clear articulation and demonstration of causal chain

Ruling out other possible causes, including random chance, simultaneously

Chapter 4  Choosing a Design That Fits Your Question

Differences between qualitative and quantitative positivist approaches are, in many ways, largely superficial, especially when compared with nonpositivist approaches (Jackson 2010). The largest difference among positivist scholars is what some have termed the study of “causes of effects” versus the study of “effects of causes.” As Bennett and Elman (2006, 457–58) note, quantitative scholarship generally tries to establish the average effect of some causal variable, putting it in the latter camp. This type of work is rather clearly focused on outcomes as a subject of inquiry. Contemporary qualitative methodologists, on the other hand, typically gravitate toward a “causes of effects” framework, in which they work to detangle the mechanics behind how outcomes emerge— that is, about the causal processes leading from cause to effect. In practice, the biggest difference between qualitative and quantitative techniques comes from the training that faculty receive in graduate school. This training is very formative; it colors how we frame research questions for the rest of our careers. Most scholars have a preferred method family based on their current research questions, interests, and expertise; those, in turn, are often shaped by our graduate school experiences. While that initial training affects how we frame questions, however, it does not typically leave us blinded to the potentials of other approaches. Almost all of us are capable of reading and evaluating research conducted using both sets of techniques (we can, after all, grade all of your papers), and many of us can conduct at least basic research in both styles as well. Your particular instructor may not have expertise in the specific technique you end up using, but she or he can give you general guidance about design, point you to other helpful resources (including other faculty who may have more expertise in that technique), and show you tools to help you understand methodology scholarship.11

Strengths and Weaknesses of Qualitative and Quantitative Methods Practicing social scientists agree that both qualitative and quantitative methods have strengths and weaknesses, and they generally agree that the two sets of techniques are complementary rather than substitutes or even antithetical.12 11Methodology

is the study of research design and the study of (and creation of new) analysis techniques. Scholars who specialize in this field are known as methodologists; they differ from the rest of us because they are producers of research techniques rather than simply users of them. Most professors of research methods are not methodologists per se—we are simply well-trained in research methods. I, for example, would not self-identify as a methodologist, even though I’m writing a textbook on research design and methodology. 12Some

debate on the particular strengths and weaknesses of qualitative approaches continues in the literature. King, Keohane, and Verba’s (1994) Designing Social Inquiry (known colloquially as KKV after the authors) provided concrete advice for conducting qualitative research in the quantitative mold, and it is the foundation of what scholars call the “second generation” of qualitative methods guidance. While KKV is now considered a classic text on research design, its prescriptions have increasingly come under fire from the “third generation” of post-KKV qualitative methods scholarship. Mahoney (2010) provides a good overview of third-generation scholarship and its differences from second-generation work, and many core insights are collected in Brady and

97

98

Empirical Research and Writing

The strengths and weaknesses of each tool type are roughly complementary. Qualitative methods are good at testing hypotheses on a micro level though generalizing can be difficult, and quantitative methods are good at testing hypotheses on a macro level, but their generality can leave them lacking in specific cases. The trick is figuring out where your hypotheses lie, crossing that with information or intuition about available data, and then choosing a research technique whose strengths fit nicely with your hypothesis type(s) and data. The rest of this section speaks in very general terms about the perceived strengths and weaknesses of the two sets of tools, but remember, these are just general guidelines. Ways exist to do nearly anything with either tool set. Most hypotheses are about one of two things: outcomes, or processes that produce outcomes. Generally speaking, these two categories map onto our types of analysis, with quantitative tools usually used to study outcomes and qualitative ones for processes. The quantitative approach typically conceptualizes the world as having a systematic part and a nonsystematic part; the goal is to explain as much of the systematic part as possible. This leads to a focus on generalizability across cases. Most of the causes that quantitative work studies are more distant from the event or outcome of interest, or are generalized pressures rather than specific tangible factors or discrete phenomena. These underlying causes include things like the balance of power or economic integration as a cause of war in international relations, for example, or the role of income in partisan affiliation or emotional appeals in vote choice. Qualitative work, on the other hand, perceives the world as being composed of “patterned diversity” (Bennett and George 1997). The world still has systematic and nonsystematic parts, but often, the interest is in exploring the nonsystematic parts. Sometimes scholars are in search of more subtle patterns or similarities within the currently perceived-as-non-systematic part— that is, they’re seeking additional variables that may be undiscovered parts of the story. Sometimes, though, we seek to explain within-class variation. Where a quantitative researcher sees a lot of cases of “revolution,” a qualitative researcher might see a variety of types of revolutions: social revolutions, palace revolutions, coups, mass or grassroots revolutions, or more.13 The question for qualitative researchers here is not “when and where will a revolution occur,” but “what type of revolution will occur where, and under what conditions?” Scholars using primarily qualitative tools often have hypotheses about causes that are temporally or spatially near the outcome of interest— that is, they frequently seek immediate or proximate causes for phenomena or events. The ability to seek and test within-case claims about causal mechanisms is a big asset for this type of qualitative work. These are not hard-and-fast rules. Testing process hypotheses with quantitative tools is definitely possible; it just requires different (and often more

Collier (2004). Treatment of qualitative methods in Empirical Research and Writing blends insights from the second and third generations, but you should be aware that debates are still ongoing. 13Collier

and Levitsky’s (1997) piece makes its point in the title “Democracy with Adjectives.”

Chapter 4  Choosing a Design That Fits Your Question

difficult to collect) data. These data run into major problems of comparability across time and space, and so we typically don’t do it—but it can be done; Jennifer Widner’s (2008) analysis of constitutional drafting processes is a good example of this. Likewise, scholars sometimes study outcomes with qualitative tools. These are often extremely rare outcomes or ones where only one instance exists. We study them not because we are interested in that particular event or outcome itself, or because of its sheer rarity, but because it’s clearly identifiable as an instance of a larger class of some phenomenon or event. We conduct qualitative analyses of the development of the European Union, for instance, not because the European Union itself is rare, but because it’s the best-developed instance of regional economic integration and our only example of substantial political integration. No comparable units exist for cross-unit study, so we must use a single-case or within-case technique (Gerring 2004). In terms of the framework we discussed earlier in this chapter, underlying and proximate causes can come in all the types of hypotheses above. Simply knowing the type of hypothesis you have is not, unfortunately, enough to know what kind of design you need. Even deterministic hypotheses and conditional hypotheses, long thought to be the domain of qualitative and quantitative testing respectively, have tools in either family of approaches (Braumoeller and Goertz 2000; Ragin 1987, 2000). For most student research projects, data availability ultimately plays a strong role in the choice of techniques. When we know very little about a topic, and/or when the population of interest is not very clearly defined, most scholars prefer to use qualitative methods because they are more conducive to exploratory analysis and inductive theorizing. Quantitative techniques, on the other hand, are usually the tool of choice when the population is both large and well-defined, when access to research subjects is limited, and/or when limited time and/or resources constrain the project. Clearly defined large populations allow easy sampling. Access to research subjects is limited in many cases, such as in the study of historical events where all participants are now dead, or surveys where the potential respondents are geographically dispersed. As I’ve alluded to in this chapter and elsewhere, many existing datasets in political and social science are publicly available and easily accessible through public data portals or university subscriptions.14

“

Quips and Quotes

“Research designs invariably face a choice between knowing more about less, or less about more.” —Gerring 2004, 348

14The

ERW website, http://study.sagepub.com/powner1e, contains links to many of the datasets mentioned in this book.

99

100

Empirical Research and Writing

Overview of Techniques In the two subsections that follow, we’ll consider some major tools of quantitative and qualitative analysis, respectively. The descriptions here are necessarily broad-brush. They aim to give you a bird’s-eye view of the choices available to you, and they point out some things that particular techniques are good (and not so good) for. Before you read this section, return to Progress Activity 4.2 above and review your response there. Keep your answer in mind as you read these descriptions, and mark any techniques whose descriptions match your hypothesis types. When you are done reading this section, review the techniques you’ve marked and consider your preliminary decision. If none of them seem to fit well, don’t worry. Hundreds of techniques exist, and this is just a small sample of the most common ones. Feel free to ask your instructor about others that might fit your question better based on your hypothesis types.

Major Forms of Quantitative Analysis The key factor in determining what type of quantitative analysis one conducts is the form of the dependent variable, primarily the level of measurement. Nominal, ordinal, and interval-ratio variables require different tools, with quantitative tools being most powerful when the variables involved are at the interval-ratio level of measurement.15 This section gives you a preview of some major quantitative tools so that you can make a preliminary decision among them. Chapter 7 reviews common issues related to obtaining and managing quantitative data that are common to all of these tools, and Chapter 8 discusses steps you’ll need to take to prepare your data for analysis. For more details on how to actually use these tools for your research (how to obtain the results, what the values mean, etc.), you’ll want to consult a textbook on quantitative methods.16 Table 4.5 summarizes the discussion. For nominal variables, introductory-level quantitative options are fairly limited. Provided that your nominal variables of interest meet the basic criterion of exhaustive and mutually exclusive values,17 your best tool is typically cross-tabulation, which sorts and counts cases by category on two variables. The chi-squared (χ2) value is the crucial indicator of whether 15This is largely because we are using math-based techniques. In math, numbers have a very particular meaning, as in interval-ratio variables: 3 is half as big as 6, the difference between 1 and 2 is the same as between 8 and 9, etc. Most basic models, such as regression (which you probably will/have/are learn[ing] in this course), are designed to handle interval-ratio DVs. When we have noninterval-ratio DVs, we must select a different type of model, one that understands what our different DV values represent: increments, levels, categories, durations, etc. 16My favorite strictly quantitative methods book is Pollock (2012). For methods through regression, that is, primarily bivariate methods, you can also try Healey (2009). 17Each observation receives one and only one value for each variable. All variables used in quantitative analysis must meet this criterion.

Chapter 4  Choosing a Design That Fits Your Question

you’ve found a not-likely-by-random-chance relationship.18 The biggest problem with using cross-tabulation is that it only considers bivariate relationships, that is, one DV and one IV, and that IV must also be nominal (or, at worst, ordinal). Since we know that the social world is a complex place and that many variables affect one another, bivariate testing strategies aren’t that convincing to readers because we know they’re omitting important parts of the story (other variables). Still, cross-tabulation is a useful tool, especially in exploratory analysis. It can handle multiple categories (values) for each variable, so long as the categories are mutually exclusive and exhaustive. If your IV and DV are both nominal, and you definitely have enough cases (30+), consider using cross-tabulation. If one or more of your key variables is not nominal, if you have a hypothesis of necessity, sufficiency, conditionality, or no relationship, or if you have fewer than 30 cases, you should consider qualitative designs or the intermediate-N designs discussed below. Probit, logit, and their associated variants are also options if you have a nominal DV and are more confident in your quantitative skills; more on them below. For ordinal DVs, we have a couple of options at varying degrees of complexity. First, we can still use cross-tabulation if we also have a nominal or ordinal IV. For ordinal IVs, we have the option of using something like the gamma statistic or Spearman’s rho, which provide estimates of the strength and direction of association between the variables. Alas, all three of these techniques are bivariate methods, and they suffer from the same problem of ignoring confounding variables that we had with nominal DVs. Again, though, for the purposes of student research, they may be entirely adequate; check with your instructor. Depending on the number of categories in an ordered DV, you may be able to treat it as if it were an ordinary interval-ratio variable and use regression (see below). Typically, if you have an ordered variable with more than about five categories, regression is an option worth exploring with your instructor. Examples of ordered DVs that scholars often treat as continuous interval-ratio variables include the Freedom House and Polity democracy scores and the American National Election Studies (ANES) ideology and partisanship scales. Our major tool of quantitative analysis, the regression, works with intervalratio DVs. This tool, which is sometimes called ordinary least squares or OLS regression, is the basic workhorse of quantitative social science, and it serves as the base for a wide range of options, adaptations, spinoffs, and variants. For our purposes right now, the chief thing to know is that the DV must be 18That’s the whole idea behind using statistical techniques. The notion of statistical “significance” is that the pattern of data is unlikely to have been produced by nothing other than random chance. A statistically significant coefficient or test statistic means that the computer can identify a systematic pattern in the data that is unlikely to have been produced if the cases were distributed randomly across the possible values of each variable.

101

102

Empirical Research and Writing

interval-ratio; the independent variables (including any control variables) may be at any level or combination of levels of measurement. Indeed, this flexibility is one of the reasons regression is such a common tool. OLS assumes that a linear relationship exists between the IV(s) and the DV: As X goes up, Y goes up too (or as X goes up, Y goes down) at a smooth, constant rate across all values of the IV. The key piece of output for a regression, then, are the coefficients on the IVs, which tell us the slope of the line depicting that relationship.19 Ultimately we’re interested in the sign of a coefficient, its statistical significance, and its size. Those pieces of information allow us to test hypotheses—particularly directional hypotheses and conditional hypotheses—against the data. Like most quantitative tools, regression requires a substantial number of observations to work. Typically, for regression, you want at least 20 to start and 10 more for every independent or control variable you add. If you’re able to get at least some variables from common data sources (see Chapter 7), then regression is a good tool for you. It’s probably not a viable strategy if you have to collect all of your data yourself, but fortunately, the times when this is necessary are few and far between. Scholars have already collected data about more things than you can probably imagine, so you’ve got a pretty good shot at having most if not all of your data collection already done. If your DV is not interval-ratio, but you do have a lot of observations, a lot of data from published datasets, and a little sense of adventure, you can consider using some slightly more complicated quantitative tools for dummy dependent variables, polychotomous nominal variables, or ordinal variables. Logit and probit models are our primary tools for dependent variables that are dichotomous (or dummy) variables. Again, the key output here is the coefficient on the independent variable(s) of interest. Unlike regression, though, we can’t interpret the coefficient values themselves quite as easily for reasons that we discuss in Chapter 8. As with regression, logit and probit are best suited to the most common kinds of hypotheses, namely directional, conditional, and no-significance hypotheses. Probit and logit are a bit more complex than regression, but fortunately, they are not our only tools for 0–1 DVs. We can use regression on dummy DVs in what’s known as a linear probability model. We simply interpret the predicted values of the regression as predicted probabilities and coefficients as effects. This method violates some statistical assumptions of regression, so published work doesn’t use it.20 For student research, though, it’s usually 19Yes, those kinds of coefficients, the ones you remember from algebra class, with that whole y = mx + b thing, where m is the slope and b is a constant and x and y are independent and dependent variables. The whole idea of regression is based on that simple middle school algebra formula. And you thought you’d never need that again.  20In particular, predicted probabilities that are greater than 1 or less than 0—“out of bounds,” in statistical speak—are very common and quite problematic.

Chapter 4  Choosing a Design That Fits Your Question

entirely adequate; we’re more interested in getting you to try your hand at the research process than in getting all the statistical bits right. If you’ve got a dummy DV, ask your instructor if a linear probability model is acceptable. If your DV contains multiple categories, you have a more complex situation. Probit and logit have variants that accommodate ordinal DVs, known as ordered logit and ordered probit, and other variants for categorical (nominal) DVs, known as multinomial logit and polychotomous probit. These generally work on the same basic idea as the plain probit and logit models; we’re interested in the coefficients, and the relationship is assumed to be nonlinear. The combination of multiple outcomes and nonlinear models make these somewhat challenging to interpret, though, so discuss their use with your instructor before committing to a quantitative approach for these DVs. Depending on other characteristics of your DV, your instructor may encourage you to use another quantitative approach, such as regression for an ordinal DV with many categories or cross-tabulation for a categorical DV. Finally, you may have a DV that doesn’t easily fit into any of these categories. It might be a duration, such as how long a bill sits in a Congressional committee before a vote or how long a civil war ceasefire lasts. It might be a count variable, such as the number of Congressional committees to whom a bill is referred, or the number of candidates or parties on the ballot in a given district. If you have one of these types of variables, consult with your instructor on the appropriate model to use. Specific types of models exist for counts, durations, and a wide range of other types of dependent variables. Most of these are rather complicated, however. Depending on your particular situation, you and your instructor may collectively decide that the most appropriate approach is using a simpler quantitative strategy, reframing your question to a different form of DV whose tools are more accessible, or adopting a qualitative strategy.

TABLE 4.5  Summary of Quantitative Tools DV

IV

Use

Look for

Nominal

Nominal

Cross-tabulation

Chi-squared

Interval-ratio

Multinominal logit or polychotomous probit

Coefficient

Nominal or ordinal

Cross-tabulation, gamma, or Spearman’s rho

Test statistic value

Interval-ratio

Ordered logit or ordered probit

Coefficient

Any

Regression

Coefficient

Ordinal

Interval-ratio

103

104

Empirical Research and Writing

Major Forms of Qualitative Analysis Most DVs of interest in qualitative work are nominal or ordinal. It’s no coincidence that these are the two levels of measurement where quantitative tools tend to be weaker. This section gives an overview of commonly used qualitative research tools so that you can make a preliminary decision among them. The techniques presented here are all variable-centered designs. Depending on your instructor’s preferences, a variety of other forms, such as descriptive and typological designs, may be appropriate for you. If your arguments don’t seem to fit with the types of hypotheses described above, and the approaches described below don’t seem to fit either, you should definitely plan to consult with your instructor about your options. We’ll discuss each of these techniques in more detail in Chapter 5, with particular attention to case selection criteria for each. Chapter 6 considers issues of collecting and managing qualitative data. The final subsection here briefly presents several techniques for so-called intermediate-N designs.

Within-Case Designs As with all within-case techniques, process tracing relies on multiplying observations within a single larger “case.” As its name implies, it is particularly useful when studying hypotheses about processes, that is, sequences of related or connected decisions. The researcher typically has a series of hypotheses (observable implications) of the theory, with DVs at different points in the process, which should all be supported if the theory is correct. Process tracing is also useful for explicit tests of causality in the sense that it allows the researcher to determine whether changes in variable values actually occurred in the order that a theory projects. In other words, it can help to detangle correlation from causation by testing a theory’s propositions at a micro level instead of at a macro level. The model of an analytic narrative is similar in some ways to process tracing. Process tracing focuses on individual observable implications—specific facts or patterns of facts that we should expect to see if the theory is correct. Analytic narratives, in contrast, tend to focus much more on the value(s) of variables within a case, particularly variables that are difficult to measure in a systematic way that is suitable for quantitative analysis. This might include things like “concern for international reputation” or “obstacles posed by gender.” We can definitely identify things that would increase or decrease values of this variable, but we would typically need to gather these data by, for example, interviewing participants/subjects, consulting diaries and private papers, and the like. While analytic narratives can in theory be used for a wide range of purposes, their use is most common in testing hypotheses and other propositions derived from a branch of theoretical research known as formal modeling.21 21In

formal modeling, researchers create abstract models of the relationship between concepts using forms of algebra. They then can derive propositions about the relationship between the variables and particular outcomes or other variables of interest by manipulating the model and solving for specific variables in terms of the rest. Powner (2007) describes these types of articles in more detail; Kellstedt and Whitten (2009, 31-38) provide an introduction to modeling.

Chapter 4  Choosing a Design That Fits Your Question

Between-Case Designs Three common methods of between-case analysis include the case control method, the method of structured focused comparison, and content analysis. The case control method builds on the methods of agreement, difference, and congruence developed by John Stuart Mill in A System of Logic (1843), and it depends fundamentally on logic to produce its conclusions. In the most common version of the case control method, we seek two cases that vary on the outcome of interest but are otherwise alike—that is, which control for all other possible causes—and then we look for the one independent variable that differs between the two cases. Logically speaking, we cannot explain constants with variables and variables with constants. So if we have two cases that are identical other than their outcomes and which vary only on one causal variable, then the only possible cause for the variation in outcomes is the varying causal variable.22 Because this approach finds most of its support from logic rather than from the strict evidence, the further we move from the “ideal” comparable case, the weaker our design becomes. The irregularity of the social world and the rarity with which some of our events of interest occur means that finding perfectly matched cases is nearly impossible. Its focus on outcomes also means that this is not as strong a technique for testing hypotheses about causal processes. Nonetheless, this general approach of finding “comparable” cases remains a very common format for many qualitative between-case studies; it is the foundation for an entire branch of research techniques using “matching” designs. The method of structured focused comparison (SFC), pioneered by Alexander George, follows a similar logic of implicit regression but in an even more detailed format. Structured focused comparison uses a preestablished set of questions to interrogate each case in search of evidence, and it applies this same set of questions to multiple cases. By using the same set of questions— collecting comparable and parallel data—about each case, we can develop cumulative knowledge about the phenomenon of interest. Having consistent and systematically gathered data points also facilitates making comparisons across cases in search of causal effects (i.e., the implicit regression model). SFC is a very useful tool for analysis because it helps the researcher focus on obtaining data and evidence from the case and reduces the risk of losing sight of the research objective amid the case details. SFC works for almost any type of hypothesis, though the implicit regression model underlying it means that it

22Consider an example from a friend with twin daughters. His children had eaten the same meals, with the exception that one had milk with lunch and the other had juice. When the milk drinker developed a mild case of food poisoning that afternoon, he concluded that the milk must be the culprit. How could he do this? It was the only food item that differed between the two children. Logically, no other food could be the culprit. If any other food were triggering the illness, both children should be sick because they had both consumed the same things other than their lunch beverages. Whether the milk had actually spoiled was not really relevant here; the sheer fact of that being the only variation in their food consumption was sufficient evidence to draw the conclusion.

105

106

Empirical Research and Writing

tends to work best for probabilistic hypotheses.23 Scholars have also used it effectively for process hypotheses as well as outcome hypotheses (Milner 1997). Finally, content analysis is a research technique that studies patterns in written or spoken sources. Its goal is usually to make inferences about things such as conceptual frames, mental images, or other beliefs of the authors based on their choices of words, analogies, negative and positive references, and similar information. One might study State of the Union speeches for evolving views on a particular policy issue over time or across parties, for example, or the tone of media coverage of political campaigns with an eye to evaluating whether and/or when coverage is biased. Normally, the textual data are converted into some type of numeric data (such as reference counts), which is then the focus of the analysis. Because it reduces a large quantity of data into a small set of manageable numbers, scholars often use it to make comparisons, implicitly or explicitly, across multiple cases, though single-case uses do sometimes occur as well.

Intermediate-N Designs Finally, we should briefly discuss some options for data analysis when you have more than “a few” but less than “a lot” of cases—that is, situations where you have between 5 and 20 cases, or even 5 and 50 cases. In these situations, you don’t have enough observations to do any effective quantitative analysis, but you have too many cases for traditional qualitative techniques. Scholars have developed a small suite of tools, mostly expanded from the standard qualitative toolkit, for such “intermediate-N” situations.24 These methods take advantage of the ability to locate countable patterns in the data even if they lack sufficient statistical power to determine statistical significance. Content analysis, which we briefly discussed above, has variants that expand into larger samples. With the aid of computers, counting word frequency in text files can be done rapidly and accurately, which enables researchers to code larger batches of texts. This then allows the researcher to look for changes over time or across categories via quantitative tests that are reasonably accurate with relatively low N, such as difference of means tests, χ2 tests, and the like. Qualitative Comparative Analysis (QCA) and its cousin Fuzzy Set Analysis (FSA) sit squarely on the border between qualitative and quantitative techniques; they combine features of both in-depth case knowledge and crosscase comparison (Ragin 1987). Both techniques test primarily deterministic hypotheses, that is, claims of necessity and/or sufficiency. Fuzzy Set Analysis is the more flexible (and more recent) tool of the two. The researcher codes cases on a series of variables and assigns them scores between 0 and 1, representing 23The implicit regression model combines with the structured data collection to make SFC one of the strongest qualitative tools available for testing relative hypotheses. 24N

is the statistical shorthand for “number of observations” (cases) used for analysis.

Chapter 4  Choosing a Design That Fits Your Question

how well or how much they exhibit that characteristic. The analysis then uses a form of Boolean algebra (the logical AND, OR, and NOT)25 to seek combinations of causes that are necessary and/or sufficient to produce the outcome. Intermediate-N techniques are a bit of an odd category among methodology tools because the factors driving their use are somewhat different than most of the previous tools we discussed in the qualitative and quantitative sections above. Qualitative and quantitative approaches are defined by characteristics of the data they analyze and the hypotheses they work best to test. In contrast, intermediate-N approaches operate quite differently. The choice to use these tools emerges only partially from the nature of the hypotheses and data. More than any other tools, the choice here depends very heavily on the number of available observations rather than the type of data. Many scholars are thus uncomfortable with intermediate-N designs for several reasons. First, some object that intermediate-N users artificially limit the number of cases in their population by defining the population very narrowly. Defining the population is a very important step in research design because it helps to ensure a fair test of the hypotheses; cherry-picking one’s cases to get desired results is one of the worst violations of scholarly research ethics.26 Second, some scholars object to these methods because they typically do not adhere to best practices in transparency for either qualitative or quantitative data. In short, coding schemes for many intermediate-N designs are often not transparent to or replicable by outsiders, which raises doubts about the credibility of the research. Intermediate-N techniques in published articles are thus relatively rare, and some methodologists are skeptical about their use. That said, sometimes these intermediate-N techniques are the only appropriate tools available, and so we must look for best practices in their use. When using intermediate-N techniques, you should construct numerous testable hypotheses for your data, just as you would with most qualitative or quantitative techniques. Consider different subgroups of cases, different combinations of characteristics or variables, or other things like that. Discuss your coding scheme explicitly in the methodology or measurement section of your paper. When writing up your results, you will probably need to choose selectively between the qualitative and quantitative advice offered in Chapter 9 based on your particular design. I encourage you to consult with your instructor if you feel that one of these tools is best for your research question and hypothesis. Your instructor may provide guidance for you directly, may be able to identify other resources to help you, or may encourage you to adopt a more common technique on a 25The two-by-two matrices we used when discussing necessary and sufficient conditions above demonstrated a hypothesis about single independent variable. Boolean algebra is a branch of set theory that allows us to work with multiple independent variables by combining them in a variety of ways to examine conjunctions between the categories. 26The only worse things are plagiarism and faking your data. It’s that serious of an academic crime.

107

108

Empirical Research and Writing

subset of your data or hypotheses. In the interests of manageability, however, Empirical Research and Writing will not provide additional guidance for intermediate-N techniques in Chapters 5 and 6. If you want to use one of these intermediate-N techniques, you should locate a couple of articles using the technique of interest—even if they’re not anywhere near your substantive field—and use those as templates for designing your study and discussing your findings.

Summary Probabilistic hypotheses include directional, relative, no-effect, and conditional types; these hypotheses make claims that they expect to be true, on average, across many cases. Deterministic hypotheses, on the other hand, make claims that should always hold; these include claims of necessity and/or sufficiency. Both qualitative and quantitative methods provide rigorous ways to establish and assess causality, though their understandings of causality and techniques for doing so differ. Qualitative and quantitative techniques are generally complementary rather than substitutes or opposites. Common quantitative tools include cross-tabulation, regression, and logit/probit models; which is appropriate typically depends on the level of measurement of the dependent variable. Common qualitative tools include two within-case techniques, process tracing and analytic narratives, and three between-case techniques, case control, structured focused comparison, and content analysis. Intermediate-N techniques exist for research questions whose number of cases is greater than that typically used for qualitative analysis but below the threshold for successful quantitative analysis.

Key Terms •• •• •• •• •• •• •• •• ••

Directional hypotheses Interval-ratio Level of measurement Likert scale Ordinal Nominal Dichotomous Hypothesis of no relationship Relative hypothesis

•• •• •• •• •• •• •• ••

Conditional hypothesis Deterministic hypothesis Necessary condition Sufficient condition Main diagonal Underlying causes Proximate causes Content analysis

T

Case Selection and Study Design for Qualitative Research

5

his chapter begins our foray into the practicalities—the nitty-gritty details—of performing qualitative analysis. A lot of beginning researchers think that qualitative research is somehow easier because you don’t have to do statistics, by which I assume they mean estimate relationships between variables. Since hypothesis testing is at its core about estimating and comparing relationships among variables, I’m not really sure where that belief comes from. I think it might stem from ideas that since we can all read and understand words, we can all therefore use words effectively as evidence, and so we can all do qualitative analysis without any special training. This belief really isn’t correct, either. Good qualitative research design is at least as much work as good quantitative research design, and I think that most scholars who practice both techniques would say that qualitative research is generally more work and requires more skill. Quantitative work is somewhat mechanical, in the sense that it has very firm and clear rules for determining what kind of data to get, how to analyze it, and how to interpret the output. Qualitative analysis has much more room for leeway—and so much more room for error—on the researcher’s part because it lacks these clear rules. This chapter summarizes the available best practices in research design for a handful of commonly used qualitative research techniques, and Chapter 6 does the same for some common qualitative data collection processes. Qualitative research, in short, is neither easy nor intuitive, and I’m going to talk very frankly here. I know that a lot of students choose qualitative research techniques because they are unwilling to deal with numbers. Another big group of students choose qualitative techniques because they have a particular case that they want to study or talk about. Neither of these is a valid reason for choosing qualitative research. Attempting to shoehorn a quantitative-test-demanding theory and hypothesis into a qualitative research design typically results in both more work for the researcher and a less successful paper. Choosing the correct design is a time-saving mechanism in the end. So please, think carefully about whether you should use qualitative techniques, not whether you want to use them. They are not a time-saver; if anything, they are often more time-consuming than quantitative methods.



109

110

Empirical Research and Writing

With that said, we’ll begin this chapter by looking at probably the most important but most underappreciated stage of qualitative research: the research design process. This process involves answering two crucial and interrelated questions, how many and which cases to study. We first discuss these issues in the abstract by reviewing general principles for good qualitative research design. Then, we consider each of the featured techniques separately, with a focus on specific case selection advice for that particular technique. Because instruction on specific qualitative techniques tends to be rather scarce in political science research methods textbooks, I also include a somewhat more extended introduction to each of these techniques along with citations to other items worth reading. The final section provides guidance for writing the research design section of your paper.

Qualitative Study Design Designing a qualitative research project takes effort and time, just as designing a quantitative one does. We cannot simply pick a case of interest and apply some theory to it, “just because.” When the number of cases is very small, we must make every one of them count toward testing our theory. Effective design for qualitative studies requires answering two closely related questions: How many cases should I study, and which ones? Only after this can we begin the process of gathering data.

How Many Cases Should I Study? The answer to this is, unfortunately, it depends. It is usually more than one, but rarely more than five or six for qualitative work. Even the so-called single-case or within-case designs are really implicitly comparative, and that’s the important thing to understand how they work. The key component to successful small-n testing is overcoming the so-called degrees of freedom problem.1 Degrees of freedom are a bit hard to define outside of the quantitative context where the term originated. Roughly speaking, they refer to how many independent pieces of information we have to use in calculating a statistic, or, in a qualitative context, explaining an outcome. Generally, we need one more piece of information than we have hypotheses to be able to draw conclusions. Precisely what those pieces of information are varies across types of research designs. 1Some

debate exists over whether degrees of freedom really are “the problem” with qualitative work. Some scholars believe that degrees of freedom are only relevant when case studies—by which they usually mean intensive studies of single instances of the core phenomenon of interest—are being used for theory testing rather than theory development. Since this book is primarily concerned with testing hypotheses, we’re going to accept that degrees of freedom matter, to at least some extent.

Chapter 5  Case Selection and Study Design for Qualitative Research

Insider Insight

The concept of degrees of freedom is pervasive in research design but often misunderstood. Imagine that I give you three numbers—10, 5, and 6—and ask you to calculate the mean. You add them up and get a sum of 21, and then divide by 3 to get a mean (average) of 7. A mean calculated with three input numbers—three data points—has two degrees of freedom. Once we compute a mean, we know that the sum of the three inputs has to equal three times the mean. With a sum of 21, knowing two of the addends (inputs) means that the third addend is totally determined: 3(7) = 21  10 + 5 + x = 21  x = 21 – 15  x = 6. This fully determined third addend occurs for any mean calculated with three digits: Once we know the mean of three values, and two of the values are determined, then the third can only be one possible number—it is not free to take on other values because the mean restricts it. So the mean of any three numbers has two degrees of freedom— two addends can vary to whatever values they want, and the mean restricts the third. As a general rule, for every statistic you calculate on a set of data, you lose one degree of freedom, so you always need at least one more data point than things you need to estimate or calculate. In most within-case designs, the goal is to multiply observations within that single “case” or instance of the main phenomenon of interest. Each observation becomes a piece of information, in the degrees of freedom sense, that we can use to draw conclusions. For most within-case designs, one observation constitutes a single hypothesis test—comparing one particular sub-outcome or data point within the case to two or more theories to see which one(s) predict that outcome. If the overall case is a hard test for the theory, and the individual hypotheses are also difficult tests, then the hypothesis test results can be implicitly summed to create an overall test of the theory.2 To determine a “winner” in this sense—that is, to draw conclusions about which theory works better—we need one more test than we have theories. If we have two theories and two tests, each theory could win one and we still wouldn’t have a winner. So we always need one more test to discriminate between them. In a within-case design, the comparison claims and cases are sometimes implicit. They may be counterfactual discussions (including comparison against competing predictions), before-and-after observations, or any of several other techniques. 2Think of it like a basketball playoff—which team is better is determined by a best-of-five or bestof-seven series. In within-case designs, which theory is stronger is determined by which one explains or predicts the most sub-outcomes within that case. Theories that correctly predict three of four events are better than ones that predict one or two of the four, and so on.

111

112

Empirical Research and Writing

For between-case design, the same general idea applies: We need at least one more observation than independent variables. In a between-case design, one observation is one “case,” one specific instance of the outcome of interest. Let’s imagine that we have three hypotheses, about the timing, magnitude, and political alignments around some proposed policy change, and they’re all drawn from the same underlying theory. With three hypotheses, we have three independent variables, so we need at least four cases—four data points—to ensure that we have enough information to draw accurate conclusions. Our hypothesis tests may be written out one at a time, but the variable values for each case were determined simultaneously. Some disagreement exists about whether control variables count as independent variables for the purposes of determining how many cases to study. In a quantitative analysis, control variables enter the model on the right-hand side of the equation, just like any other independent variable, and the model estimates coefficients for them. So for quantitative analysis, we always need to count controls as variables for the purpose of determining degrees of freedom. Most qualitative researchers generally disagree that we need to do this for between-case studies. The number of potential control variables is infinite, and unlike quantitative techniques, doing the analysis does not require simultaneously determining the effects of the controls along with the effects of the independent variables. So even if we deliberately selected cases to control for values on (a) particular variable(s), we don’t need extra cases in the study to counter it. If your cases’ values on the controls are only roughly comparable, however, you should consider a brief discussion of whether that control variable could actually contribute to variation in the outcomes. If time and space permit, you might consider a brief vignette on a case that shares values on the control variable in question but varies on some other crucial control variable to demonstrate that the control’s value is unlikely to have produced the result in question.

Which Cases Should I Study? Again, alas, the answer is, it depends. No firm rule exists for choosing cases. That said, for any given research technique, some case selection strategies are vastly more effective (and widely accepted) than others. As with so many things, we can identify “not good” much more easily than we can identify “best.” The sections below on the featured techniques give some more specific advice for case selection, but some general overarching principles apply to all qualitative designs. First, cases should provide variation on key variables. As we’ve discussed in several chapters, we cannot explain a constant with a variable, or a variable with a constant. Which variables must vary, and which should be held constant across cases, depends on the specific technique you’re using and your research question. Variables that are held constant across cases are usually described as control variables, or controls for short. Controls are variables that we are reasonably sure affect our outcome of interest in some manner, and because of this, we want to make sure that changes in these variables do not affect our

Chapter 5  Case Selection and Study Design for Qualitative Research

113

outcome of interest. We do this, in between-case designs, by choosing cases whose values are identical or similar on these characteristics. Because constants (same control variable values) cannot explain variables (different dependent variable values), we can ensure that the value of the controlled variable (or changes in that value) do not affect our outcome of interest. Determining your control variables is an important stage of theorizing for qualitative research, and identifying appropriately controlled cases is an important step in research design. Second, cases should be broadly representative of the underlying population. If your cases do not, in some way, reflect the population of Representative interest, then you will not be able to draw appropriate inferences about Cases, Purposive the population from your purposive sample. Since the ultimate goal of Sampling, and empirical research is explaining patterns across cases, an inability to Inference to generalize would constitute a major obstacle, and so this is a huge Populations weakness in a paper. The key thing here is carefully identifying your population of interest. Your ability to do this is largely contingent on whether you successfully developed a theory about concepts in your preliminary work. If you have a story about concepts, then incidents of your concept—or at least incidents where your concept could have occurred even if it didn’t—constitute your population. If you’re having trouble identifying your population, chances are very good that you haven’t clearly articulated the underlying concepts of your theory yet; you’re trapped on indicators rather than concepts. Ask yourself, “what is this an example of,” revisit your work in Chapter 2, consult with classmates or colleagues, and try again. If that fails, consult with your instructor. For example, let’s consider research questions about presidential language. State of the Union addresses are very useful and important policy statements, but at the same time, they’re not typical presidential statements. They’re major prime-time events that receive national and international attention. If you want to study presidential policy preferences or the national policy agenda, these addresses are a very good data source. But if what you want to study is changing images and metaphors for the state of the economy or perceptions of potentially hostile countries, then State of the Union addresses are not a representative sample of presidential speech. You’d probably be better off studying press conferences or informal interviews—things that are more frequent, lower stakes, and likely to be direct presidential products rather than speechwriter written. Likewise, if you’re studying something about wars as a general class of events, you would probably want to pick wars of average duration, average casualties, and average salience of the disputed issue. You would not want to choose the World Wars, which are of extremely long duration, incredibly high casualties, and national-survival-invoking salience. Trying to generalize about all wars from a study of the World Wars would almost certainly lead to incorrect conclusions because the cases are very extreme examples of the phenomenon. If you want to draw conclusions about wars of national survival, major system-changing wars, or multilateral conflict, though, then the World Wars are good choices.

114

Empirical Research and Writing

Third, cases that pose difficult tests are preferable to those that pose easy tests for your hypotheses. The logic for this is pretty straightforward. Imagine that your professor gave an exam with the following questions: “1. Write your instructor’s first and last names with correct spelling. 2. Name five people in this class. 3. Identify the building name and room number where our class meets.” All of these represent an accurate assessment of your learning in this course—you learned all of these things as a result of your participation in this course. Virtually everyone would ace this exam, and therein lies the rub. A test that everyone aces doesn’t actually tell anyone anything about your knowledge. It certainly doesn’t help the instructor (and employers and graduate schools) determine how much you have learned, either in an absolute sense or relative to your classmates. A passing score on this “exam” is thus a constant—not a variable that provides useful information about individual cases (students) or across them. Hypothesis tests conform to the same logic. A hypothesis of “the event happened on Earth” is an easy test. Any theory at all predicts this; it is an assumption or precondition of most contemporary theory. All cases pass it with ease, and so the test doesn’t provide us any power for discriminating between competing theories. A hard test is one that provides the most difficult situation for a theory to be successful. In a hard test, your theory is least likely to be able to explain the outcome. To give one example, theories of international cooperation regularly claim that areas of national security are least likely to see cooperation because states prefer to maintain unilateral control over their security rather than share it with partners whose interests may change or not align with their own. To test my theory about when interstate cooperation should occur (Powner 2008), I studied cooperation in foreign and security policy precisely because these issue areas were expected to be unusual and hard to explain. Using a hard test allows you (and your reader) to make the “Sinatra Inference”—if the theory can make it here, in a challenging context, it can make it anywhere.3 Ideally, the hardest test is one where no current theory can successfully explain an outcome—except for your theory. If a theory passes a hard test, you can conclude that it is generally correct and more useful than other theories. Passing an easy test is uninformative, but passing a hard test tells us a lot about the theory’s overall usefulness. Finally, you, the researcher, have a responsibility to justify your case selection to readers in terms of concepts and variables. Did you pick your cases to maximize variation on your independent variable (IV), or on your dependent variable (DV)? Were cases selected for study on the basis of their value(s) on (a) particular variable(s)? If so, which variable(s) and value(s), and why? Did you choose two cases from the same country in different time periods to control for culture and geography, or did you choose two countries at the same time period to control for period technology and global distribution of resources? Either choice is valid, but you only chose one. Each strategy has 3Cf.

Bennett and Elman 2006. This is a reference to Ol’ Blue Eyes’ famous show-stopping hit, “New York, New York,” and its memorable line, “If I can make it here/I’ll make it anywhere/It’s up to you, New York, New York.” Think you don’t know it? Look it up online—I can just about guarantee you know it.

Chapter 5  Case Selection and Study Design for Qualitative Research

advantages and disadvantages, and you need to share those with your reader. Many possible cases could have been chosen but weren’t. You thus need to convince your reader that your case selection represents an appropriately difficult test for your theory—that you did not simply cherry-pick cases where your theory was most likely to find support. This justification typically constitutes one or two paragraphs in the methodology/research design section of your paper, which we discuss more in the final section of this chapter. Your ability to convince your reader that you chose your cases in an appropriate manner, and constructed a sufficiently difficult test, are crucial early steps in convincing your reader that your theory is correct.

A Note on Background Research Background research is almost always necessary during the design stage of a qualitative research project. This research can take two forms: identifying the population of cases from which you should sample, and determining preliminary variable values across cases to make your case selections. First, you need to figure out what the set of cases is that you’re theorizing about—that is, what the relevant population is for your study. Depending on your particular research question, you may be able to use a quantitative dataset or other established list of cases to identify your population. For example, comprehensive lists exist of wars, disputes, and militarized crises; elections and electoral results in most semi- and fully democratic states; natural disasters in the United States and world; and members of Congress and Congressional votes. Many, many lists like this exist. Once you determine your unit of analysis (see below), you can search for published lists. Starting from a list will make your job much easier. Having to make your own population census or comprehensive listing is not the end of the world, but it will add a bit of extra time to your study. Second, you need to choose cases from among this population. For survey research and other forms of scholarship, we would typically want an unbiased sample, and we would achieve this by taking a random sample of the cases. Because the cases are chosen randomly—without respect to their values on any of the variables—the techniques of statistical inference allow us to accept that the sample is unbiased and representative of the population. This strategy does not work for qualitative work. We may be interested in rare or outlier events, such as treaties, revolutions, or very close House races, and a random sample of the population might fail to capture any cases of our phenomenon of interest. While these events are infrequent or rare, they are clearly socially significant phenomena that deserve scholarly attention and explanation. As a result, we must take a purposive sample from the population, or from as much of a population census as we are reasonably able to achieve.4

4For topics where no scholarly list exists from a published dataset, Wikipedia might be a second potential source. The existence, completeness, and quality of its lists vary widely across topics: The list of recall elections in US states is pretty solid; the list of economic sanctions is spotty at best. But it may be a viable starting place for some topics.

115

116

Empirical Research and Writing

A purposive sample is just as it sounds: We choose cases deliberately, based on their values on certain key variables, to enable us to generalize or make inferences about a particular category of cases. Taking an effective purposive sample—that is, choosing cases for a qualitative research project—typically requires determining provisional values for key variables across cases to ensure appropriate variation. During this stage, I typically keep a spreadsheet or table with quick (often number-coded) summaries of variable values so that I can sort cases by variable. Which variables you assess, and which you use for a selection basis, will differ by research question. At a minimum, you will want to note values for the IV and DV, and for any crucial control variables. (This is why having your theory and hypotheses ironed out before you start designing your research is so important.) At this stage of the research, credible (i.e., thoroughly referenced) Wikipedia and encyclopedia articles are an appropriate source for preliminary variable values, though of course you should never be citing Wikipedia as a research source.5 Be sure to keep a log of these research design decisions, and your rationale for them, in your research notebook for later reference.

Hypothesis-Testing Techniques and Case Selection One of the key steps in sampling of any sort—purposive or random—is creating a population census, or list of all members of the population. If we do not know anything about the population, then we are unable to tell whether our sample is (or is not) representative. Identifying the population is not as straightforward as it sounds. Not all cases leave similar types of tracks in the historical record, and this is unfortunate because the same factors that influence the occurrence of our phenomenon of interest also affect the probability that such cases will be noticed and described in the record. No matter which technique you are applying, you need to be aware of the causes and effects of selection bias. Selection bias is a major risk in social research, where we must rely on the historical record both to identify cases and to collect data on them. Think of it this way. How many wars can you name? Take a moment to make a quick list. (Really, seriously. Pause and make a list.) Then ask yourself, how many nonwars can you name? (Again, pause and make a list.) What does a non-war even look like? The same things that cause some crises to become wars also affect whether an issue becomes a dispute and a dispute becomes a crisis. And if it never becomes a dispute, then we have no way of even knowing if it was an issue. For example, the Great Canadian-American Water War of 2012 never occurred, nor did the 2012 lumber crisis between the same countries (nor did the 2011 or 2010 or other previous wars, for that matter)—those issues never became militarized disputes, so they never escalated into wars. 5Unless, that is, Wikipedia itself is the text of interest in a content analysis of, for example, bias and

framing in writing about controversial subjects.

Chapter 5  Case Selection and Study Design for Qualitative Research

The population of cases that actually become wars (or recall elections, or whatever), then, is a biased subset of the population that could have become wars (or whatever). We know that these actual-war cases are very high in salience, they affect vital national interests, and they presumably carry large potential benefits to winning since states are willing to incur large costs to fight. If we study only the cases in which war occurred, then we have artificially limited the range of values that these and many other important variables can take. This is one example of data that are subject to a selection effect—somehow, natural selection processes screen out or filter cases whose values on key variables are above or below some implicit threshold, and the result is a pool of observed cases whose values are abnormal when compared with the true underlying population. Selection bias occurs when we analyze data that, knowingly or unknowingly, contain data subject to selection effects. Our conclusions are biased because our case sample is biased—their mean is different than the mean of the underlying population. The biggest risk of selection bias in qualitative research, at least in my opinion and experience, is the urge to focus only on instances where the event or phenomenon of interest does occur and to ignore those where it could have occurred but didn’t. The logic behind this urge is quite straightforward and very strong. If one is interested in explaining the occurrence of wars, recall elections, or constitutional amendments, then studying instances of these makes sense on some intuitive level. If we want to know why those things happen, our gut reaction is to focus on cases where they happened, not on cases where they didn’t. This is where we come back to the discussion above about selection bias and the importance of having cases that vary on the key variables. If we only study cases where the phenomenon of interest has occurred, then we have no variation on our dependent variable. Without that variation, we can—and will—easily find cases where differing values on the IV led to the same value on the DV. In other words, we’re back to trying to explain constants—the occurrence of the phenomenon of interest—with variables. Without variation, we can’t really ever be sure that we have a trend; we need two points to determine a line. If we don’t study cases where the phenomenon of interest could have happened but didn’t, then we lose the variation we need to make a credible claim about covariation between the IV and DV. Negative evidence—cases whose significance lies in the absence of the phenomenon of interest (Lewis and Lewis 1980)—is absolutely crucial to testing almost any type of probabilistic or deterministic hypothesis. Often these nonevents leave little or no trace in the empirical record; they certainly don’t leave the kinds of traces that the events themselves do. A very famous example from international relations concerns whether deterrence— actions or positions that states take to discourage other states from threatening the deterring state’s interests—is a successful way for states to protect themselves and their interests (Achen and Snidal 1989). Cases of failed deterrence are easy to spot: They typically become dramatic crises that escalate into military conflict.

117

118

Empirical Research and Writing

Cases of successful deterrence, on the other hand, are nearly impossible to spot. When deterrence succeeds, by definition, nothing happens. The whole point of deterrence is to keep anything from happening that adversely affects your interests. So to identify an instance of deterrence success, you’d need to find a period of time or specific instance of nothing occurring that you can trace back to specific actions by the deterring state. Imagining a case of this is difficult, and actually finding one in the historic record is even harder. Successfully drawing conclusions from qualitative data—or from quantitative data—requires being aware of the risks of selection effects in the data. Our case selection process needs to explicitly identify and acknowledge routes through which cases end up in the observed sample, and ways in which cases of potential interest can fall through the cracks of history. Careful and appropriate inclusion of negative evidence is an essential component of successful qualitative research. These cases are just as important to our hypothesis testing as the positive cases are, and we must exercise just as much care in choosing them as we exercise in choosing our positive cases. Fortunately, we can use the possibility principle as a guide for identifying negative cases for inclusion in a study. The possibility principle says that we can consider as a negative case those instances in which the phenomenon of interest could plausibly have occurred. At least one IV takes a value that theory claims is crucial for or predictive of the outcome of interest, and no IVs have values that would predict against the outcome of interest (Mahoney and Goertz 2004). If our variables’ values are within these ranges, the phenomenon of interest could reasonably have occurred, even though it didn’t. Cases meeting these criteria thus make good sources of negative evidence.

Content Analysis Content analysis is a technique used to analyze and infer a variety of things from what people say and write. We do this, primarily, by examining the words they choose to use to discuss the topic of interest. The most common forms of content analysis involve count analysis and tracing. In count analysis, the researcher typically establishes a dictionary of key words associated with the topic in question, counts their use (usually across several documents or speakers), and uses those to make inferences. Frequency of word or term use here serves as a proxy for the importance of the concept that term describes. For example, we might have a research question about how US presidents’ economic policy views change when they experience a midterm recession. One hypothesis about this could be that we expect Democratic presidents to become more centrist, but Republican presidents to adopt more conservative positions. We would expect the frequency of terms associated with each position—“fighting inflation,” “reducing taxes”—to shift into patterns consistent with our hypothesis.6 6Another

well-known application of content analysis is in the attribution of the various Federalist papers to their authors. Scholars coded these unsigned texts and other texts that the Federalist authors wrote, and used content analysis techniques to attribute papers to their authors on the basis of patterns of word use, sentence structure and style, and imagery. Mosteller and Wallace (1964) is the canon study; Rudman (2012) contains more recent citations.

Chapter 5  Case Selection and Study Design for Qualitative Research

In the tracing approach, the focus shifts to changes over time to things that we can’t really count quite so efficiently, such as the kinds of analogies that leaders make about their adversaries. Iranian leaders’ public use of expressions other than “the Great Satan” to refer to the United States might reveal interesting dynamics in Iranian domestic politics, for example. Other scholars have examined former US national security adviser Zbigniew Brzezinski’s beliefs about the Soviet Union by considering the kinds of negative and positive frames he used to describe Soviet intentions. On the basis of significant shifts in the tone and type of references over time, Campbell (2011) argues convincingly that in contrast to popular wisdom, Brzezinski’s attitudes toward the Soviet Union did actually change during his time in office. Here, the emphasis is less on individual terms or items occurring in a preestablished dictionary, and more on the overall tone and shape of the references. In the count approach, we’re more interested in the relative balance of terms or concepts, with less interest in the time element; in the tracing approach, we’re more interested in changes over time. Because the count approach is both more frequent and a bit more persnickety to implement, I’ll focus more on it here. But whichever you choose, be sure to look at examples and appropriate resources for this research technique to ensure that you’re doing it correctly.

Study Design and Case Selection Common targets for content analysis are ones where the phenomenon of interest can be traced through texts. These might include, for example, Supreme Court opinions, speeches such as State of the Union messages or the annual Queen’s Speech in the United Kingdom, declarations and conclusions from international summits like the European Council’s quarterly meetings, statements made on the floor of the UN General Assembly or national legislatures, government policy documents on a topic, etc. With content analysis, we are primarily interested in the frequency of word use, the invocation of specific images, frames, or metaphors, the quality of the reference (positive/negative connotations), or other things of that nature—ones where word choice itself is the indicator of interest. Because of this, the ideal source material for content analysis has the speaker or person of interest doing the writing or speaking himself or herself, without the intervention of a speechwriter or translator. Unscripted speech is sometimes harder to obtain in full text than summaries or scripted text, but it is definitely worth using if it’s available for your question. Because the motivation for content analysis is to understand patterns of word use in a certain subject or speaker’s corpus, most researchers planning to use content analysis already have a good idea of the population they want to study. These could be amendments offered on the floor of the House during budget debates, Ronald Reagan’s scripted versus unscripted remarks about the Soviet Union, race or gender of individuals pictured in magazine stories about poverty, or Anglo- versus French-Canadian media coverage of the conflict in Iraq. In some of these contexts, the researcher could have any of several different hypotheses that require consulting different types of texts that are located

119

120

Empirical Research and Writing

in different databases. Knowing your hypotheses ahead of time—before you start your data collection—can save a lot of effort and time. Many full-text databases exist; consult specialist literature about your topic to begin the search for appropriate text repositories. Once you’ve decided on a general set of sources to study, however, you then need to decide which specific items within the population will be part of your sample. This is where the going can get dicey. Selection bias is an insidious thing with pernicious effects on results, and the effects are often magnified if it’s inadvertent or unrecognized. Your best bet, if possible, is to use the entire population of items within your time frame. Some source types come with catalogues of all entries: Government agencies and nongovernmental organizations usually have lists, for example, of all their publications; major newspapers are well indexed, so identifying the entire population of stories about a particular topic is easy. For some source types, though, the population census may not be complete. This is particularly true for things that don’t occur on a documented basis, such as presidential (and presidential spokesperson) press availabilities. If these occur in the White House Press Room, transcripts are available. If they occur on Air Force One, or while traveling, transcripts may or may not be available—and no record may exist that any press availability occurred, so a researcher may not even realize that his or her population census is incomplete. Other sources have similar odd biases. The Congressional Record contains all speeches on the floors of the US House and Senate, and committee hearings are available in transcript form as well. Unfortunately, in both cases, members of Congress can edit their remarks before publication, and in hearings, speakers can introduce prepared testimony into the record without actually reading it out. The only way to know about biases in your population list is to dig: Look for others who have used that source, find out about how the source was compiled, and ask questions. To reemphasize that point, at this stage, your most important task is to make sure you know how your population census—if one exists—was created, and whether it is complete. For some research questions, this task is very easy; State of the Union speeches occur once a year and the texts are archived in several credible locations. For other research questions, this may require corresponding with the agency or organization that created the population census. Don’t be afraid to reach out to these places and ask them about their methods if they don’t clearly specify criteria for inclusion, or if you have any other concerns about how they compiled the list. Sometimes methodology is corporate proprietary information, but most places can discuss at least general issues with you. Searching Google Scholar for other work using that data source is also good because you can find out what issues other scholars have raised about the data. You are primarily interested, right now, in identifying reasons that the list may be biased by the exclusion or omission of some systematic subset of cases. If you determine that the list is reasonably complete, you can proceed to extracting data and conducting analysis. If your list is not complete, or you have reason to suspect that omissions are systematic in a way that potentially biases your findings, you should definitely plan to consult with your instructor.

Chapter 5  Case Selection and Study Design for Qualitative Research

Once you’ve identified your population, the next step is to identify the appropriate sample for analysis. For most content analysis work, the population in question is fairly easy to identify and also reasonably small: a couple of decades of inaugural addresses, UN Security Council resolutions on Syria’s 2012–2014 crisis, New York Times articles about abortion politics in other countries. In these types of situations, you should take the ideal approach and include the entire population of items in your analysis. Colleagues who use content analysis recommend analyzing the entire population if the population has fewer than about 50 items. The advent of machine coding strategies—the simplest of which is the “find” command, available in any word processor or web browser by pressing CTRL+F—means that larger batches of text are easier to code than ever before. If your population for the desired period is large and contains more cases than you can reasonably code, however, you may need to consider using a subset of the population. We have two primary approaches for doing this. The first is defining a discrete subset and using the entire subpopulation within that subset. For example, instead of studying all Congressional statements about national security, you might choose to study all House statements about homeland security or all Senate speeches about Department of Defense spending. As long as you have a theoretically justifiable reason to limit your test to this subsample, and you can argue convincingly that this subsample is representative of the population in ways that matter for your question (i.e., that using a subsample does not introduce selection bias), this is an appropriate strategy. The second approach is to sample from the population. If you do not have a theoretically justifiable reason to limit the material substantively, you may need to resort to sampling. Again, the major task is not to introduce bias into the sample, so the best strategy is usually to take a random sample or a stratified random sample.7 Which of these approaches you choose will depend heavily on your research question. When I queried content analysis users about this issue, the consensus was that if possible, using a subpopulation is preferable to sampling, but all of the respondents agreed that it rarely is possible. If you’re planning to use content analysis in your research, some useful references on the method include Burnham et al. (2004, 236–42), Manheim et al. (2001, chap. 10), General Accounting Office (1989), Writing@CSU (n.d.), Escalada (2008), particularly the “Specific Procedures” link), and Sommer (n.d.). Elo and Kyngäs (2008) offer a somewhat more advanced treatment of content analysis in medicine but still provide much helpful information on the how-to process. The Comparative Manifesto Project and related publications 7In a stratified random sample, we subdivide the population and take a random sample (of either fixed size or proportional size) from each subgroup. For a content analysis of newspaper articles, we might stratify by length of the article in word count (easily obtained from the database that provides the population census), and then take an equal number of randomly selected cases from each group. We could also take a proportional sample, so that if articles of 1,000 words or more compose 15% of the population, then we take 15% of the total desired number of cases from the subset of 1,000+ word articles.

121

122

Empirical Research and Writing

(Budge et al. 2001; Klingemann et al. 2006) are a well-known example in comparative politics; van Doorn (2012) and Gilens (1999) apply this technique in studies of media, race, and poverty in American politics. A Sample Paper Using Content Analysis: The Influence of Historical Stereotypes on Contemporary Media Consumption

Analytic Narratives

The purpose of an analytic narrative is to provide a qualitative but still very rigorous test for hypotheses derived from an abstract, formalized model. In the words of one well-known advocate of this approach, analytic narratives “respect the specifics of time and place but within a framework that both disciplines detail and appropriates it for purposes that transcend a particular story” (Levi 1999, 155). The analyst allows the case’s context to color the particular types of observable implications sought or accepted as evidence; to a certain extent, the analyst also remains open to the possibility of the case itself suggesting refinements of the model or additional outcomes that support the underlying model (Levi 1999). In general, though, the particular case is interesting not on its own merits, but because it provides an appropriately difficult test for a broader, generalizable hypothesis. To use a very simple example of a formal model, consider the Prisoners’ Dilemma (PD). In PD, both parties have incentives to cooperate with one another, but they also have incentives to defect. If a specific historical context has the characteristics of PD, we should expect to observe particular types of behavior. Policy makers should express concerns about the possible costs of unilateral reneging by their adversary. They should also be unwilling to trust their adversary’s intentions in the absence of some type of credible signaling mechanism or commitment device, and they should ultimately opt for the “safe” but less preferable “defecting” outcome. These are all observable implications of the formal model characterized by PD-type preferences, simultaneous decision making without communication, and single-shot play. This is a very simple formal model; a single case of PD is not enough to form a whole paper. But a before-and-after study could work (oh, sorry, an intertemporal design)—for example, the role of “national technical means” (spy satellites) that allowed the United States and Soviet Union to monitor each other’s behavior in achieving and complying with arms control treaties. For students with some modeling background (e.g., economics or political science courses where it was taught), a simple extensive form game with or without some uncertainty can generate many interesting hypotheses, particularly if you pay attention to the underlying assumptions of rational choice theory.8 In some ways, analytic narratives try to combine the very rigorous deductive framework of formal modeling with the very flexible inductive approach often taken by proponents of “soaking and poking” case research.9 As you 8Drezner

(2003) provides a great example of what a three-node tree can do.

9The term soaking and poking comes from Richard Fenno’s (1978) famous study of House members

in their districts, where he followed 18 of them around their home districts for several years.

Chapter 5  Case Selection and Study Design for Qualitative Research

might expect, some doubts exist about the successfulness of such a fusion.10 Arcane methodological debates notwithstanding, I feel that analytical narratives are a valid and viable tool for the novice researcher who wishes to make a careful test of a formally derived hypothesis. Your instructor’s position on this may vary, so check with him or her before committing to this approach.

Study Design and Case Selection In an analytic narrative, the research question focuses on behavior the model highlights and dissects. The case is important because it provides grounds for testing hypotheses from the model. This differs from some process tracing and method of similarity designs in that the cases themselves are sometimes of interest in those approaches. Support for the hypotheses in an analytic narrative comes from both the formal derivation of the hypotheses (which are often comparative statics of the model), and also from the difficulty of the test. Model rigor is not a substitute for hard tests, but it can certainly complement them. Moreover, because the goal of an analytic narrative is to test hypotheses that should extend outside of particular contexts, tests outside of the case the researcher used to build the model are particularly useful ways to demonstrate additional support for one’s hypotheses. These “out-of-sample” tests can be particularly challenging, however, in the amount and depth of data collection they require; we’ll discuss them a bit more below. In my understanding of analytic narratives, the most important part of the process is identifying observable implications for the formally derived arguments prior to beginning the systematic search for evidence. The author may enter the modeling process and/or hypothesis generating stages of the research with preliminary expectations. In fact, this is typically the norm since scholars working in this vein often have knowledge of particular cases that motivate the features of the model. In practice, the emphasis on deriving hypotheses formally means that the model assumes center place. Model development takes time, and it often requires several iterations of refinement until the author is happy with it. Unfortunately, in a one-semester (or even two-semester) research project, delaying data collection until the model settles is simply impossible. You’re going to need to begin case selection as soon as you have sufficiently stable results to identify some key variable values or combinations that your case will need to address.

By immersing himself in their contexts without prior hypotheses and asking many questions over the course of the research, he discovered a number of important and meaningful patterns. This kind of inductive research is incredibly time-consuming and expensive, and as Lewis and Lewis (1980) note, the risks of unconscious bias in evidence selection are highest with inductive approaches. For these and other reasons, inductively derived hypotheses and evidence are less common in contemporary empirical political science than they previously have been in this field and/or currently are in other fields. 10See,

for example, the critical exchange between Jon Elster (2000) and the coinvestigators of the Analytic Narrative project, Bates et al., in the American Political Science Review (2000).

123

124

Empirical Research and Writing

Most users of analytic narratives choose their core case at least in part because that case posed a puzzle that prompted them to develop a model to explain the puzzle. This feature of analytic narratives is part of the reason that some scholars are skeptical of them; one of the basic principles of qualitative research design—at least in second-generation qualitative methodological thinking (see Chapter 4)—is that theories should always be tested on cases other than those from which they were derived. To help overcome this, and/or to provide additional support for their claims, many users of analytic narratives often incorporate evidence from—or tests on data from—cases other than the one(s) they used to develop the model. These “out-of-sample” tests can provide strong confirmatory evidence for a theory if they are sufficiently independent of the originating case yet clearly remain comparable in the underlying concepts. Out-of-sample cases may not provide identical or even comparable types of data, however, and researchers should be alert to the need to generate additional observable implications for alternate testing contexts. As soon as you have preliminary results on your main case, you should step back and determine if you have sufficient support for your theory from the available evidence and model. If the answer is no, and you determine that your theory could benefit from additional support by means of out of sample testing, I recommend that you plan a strategizing session with a partner, teaching assistant, or instructor to craft a mini–research design for use on the out of sample case. This should include criteria for choosing that case, a context-appropriate idea of how you’re going to measure key variables, and an estimate of how much room you’ll have to devote to this case in the paper. If you’re planning to use analytic narratives in your research, some useful references on the method include Bates et al. (1998) and Levi (1999). Levi (1997), the individual chapters in Bates et al. (1998), Sample Paper and the chapters in Rodrik (2003) are good examples of the technique. Using an Analytic Narrative: Socioeconomic Inequality and Risk Attitude in Finite Iterated Games

Case Control/Controlled Comparison Method

A final class of between-case qualitative methods goes under a variety of names, including the case control method and the controlled comparison method. You may also see references to “Mill’s methods,” referring to John Stuart Mill’s (1843) Methods of Agreement and Disagreement and the like, “most similar systems” and “most different systems” methods,11 or simply “the comparative method.” These terms are all fundamentally referring to the same kind of approach, one comparing multiple cases that have been selected to be as similar as possible on as many variables as possible so as to logically preclude these variables as possible causes.

11Rather confusingly, Mill names his Methods of Agreement and Disagreement based on the values of the dependent variable; Przeworski and Teune (1970) name their most similar and most different systems designs after the values of the independent variables.

Chapter 5  Case Selection and Study Design for Qualitative Research

These approaches work on the same underlying logic as our oft-repeated mantra about constants and variables: You cannot explain a constant with a variable or a variable with a constant. In Mill’s Method of Agreement, for example, we take two cases that agree on the dependent variable (have the same outcome value) but which are different on every other possible causal variable save one. That one remaining variable, on which the two cases share a value, is a possible cause of the outcome—it alone covaries in the same pattern as the outcome of interest. The basic idea on which these methods function is intuitively straightforward, and that makes them attractive tools for research. As just that short explanation makes clear, though, Mill’s Methods of Agreement and Disagreement have a number of important flaws. In Mill’s original approach, the Methods of Agreement and Disagreement were generally intended for inductive theorizing, not testing deductively obtained theories (which is what we’re doing here). Many times, nature (history) has not produced perfectly matched cases that we can analyze, nor can we find cases that are even remotely close on key values or that have only one variable in common. The Methods are also very sensitive to case selection; the cases you choose to study could produce results that are not at all supported as the dataset expands. Mill’s Methods suffer from additional challenges, such as their inability to conclusively determine that something is a cause and their inability to accommodate conditional or conjunctural hypotheses. In short, these tools are logically sound when used for their original purposes, but as Mill himself noted, they are ill-suited for direct use in the social sciences. Despite these drawbacks, many scholars proceed with their research at least partially using the principles of these approaches as the basis for their case selection. Qualitative methodologists continue to use these flawed tools, with some modifications and caveats, because the basic logic underlying them persists. We can match variable values across cases as a means of controlling for that variable’s effect, and we should choose cases with an eye to controlling for as many alternate explanations as possible. We can use within-case methods such as process tracing to augment the logic provided by these case control methods and to verify that purported causes are actually causal and not simply covariant.

Study Design and Case Selection For most uses of these tools, case selection is the key to successful and convincing hypothesis tests. Your sample—usually two or three cases, with four at maximum if you have a two-by-two research design—must be purposively selected with care. You will usually want to obtain wide, or relatively wide, variation on the dependent variable, including some negative or partially negative evidence, as well as variation on the independent variable.12 12This is not a case of “selection on the dependent variable” in the pejorative sense of the term. Ideally, you are selecting here on the basis of values on the IV. In practice, however, avoiding knowledge of the DV value is almost impossible, so some risk of bias exists. The pejorative use usually

125

126

Empirical Research and Writing

Determining where your cases fall in the distribution is only possible if you are familiar with the distribution of outcomes in the population, or at least a reasonably large sample of it. The logic behind this method, or perhaps I should say this class of methods, is to identify covariation as a precursor to causal claims. Your goal is to conduct what some scholars have called an “intuitive regression”—you want to make claims about the direction and, to some extent, magnitude of a relationship by establishing covariation between some cause and its purported effect, in the same way that regression identifies covariation between causes and effects across larger sets of cases. As with all covariation-based techniques, both qualitative and quantitative, you will need to control for other potential causes of the outcome so you can (possibly) reject them as producing the outcome of interest. In this particular family of research designs, case selection plays a large role in controlling for other potential causes (besides your IV and competing theories’ IVs) that might influence the DV. Unfortunately, no one can tell you which variables need to be controlled, or even which to prioritize in your case selection. We can’t even tell you whether two cases’ values are “alike enough” to serve as a control on a particular variable. It all depends entirely on your particular research question. You’re going to need to make some tough decisions here, and I encourage you to use your research notebook frequently as a repository for your thoughts as you wade through these issues and make your case selections. You will find these notes helpful later on as you write the methods section of your paper. The best possible type of cases for this technique are ones that are identical in every way except for the independent variable value of interest and the outcome. Experimental designs are able to do this, or to come as close as possible in the social sciences by randomly assigning treatment to otherwise identical pools of cases. Alas, most social science questions are not entirely amenable to experiments. We cannot rerun history with different variable values; even if it were possible, it would often be unethical.13 The next best thing to an actual experiment is the “natural” or quasi-experimental design. Sometimes, nature gives us a break and hands us situations that are nearly identical except for some variable whose value is externally determined. Many cool examples of this exist in American and comparative politics. To give one natural experiment that I personally find fascinating, the Jamaat-e-Islami (“Islamic Party”) refers to studies that select only positive occurrences of the phenomenon of interest. Selection on the basis of the DV, with little or no variation on the DV, is acceptable in some extremely limited cases. Most of these use within-case analytical techniques as the primary source of leverage over their hypotheses, and so they are not studying multiple cases. If after reading this section you still think you need to study more than one case with the same DV value—that is, you need to select cases with little or no variation on the DV—you will definitely want to consult with your instructor about your hypotheses and design. 13The closest we can come to doing this is to use counterfactual “thought experiments”; Chapter 6 addresses these.

Chapter 5  Case Selection and Study Design for Qualitative Research

formed as a radical political organization in British-colonized India in 1941. It persisted after India gained independence in 1947 and partitioned into (West) Pakistan and East Pakistan (Bangladesh after 1971). Here’s the puzzle, though: In Pakistan and Bangladesh, the organization uses (or has used) force—what some would call terrorism—to pursue its goals, and in India it’s a peaceful member of the mainstream political system. What explains this variation? It’s the same organization, with the same goals and history, during the same time period, in similar economic and geographic contexts. . . . The only thing that differs, really, is the set of political institutions under which the organization operates. Scholars exploit this variation to understand why some groups turn to violence and others remain peaceful. True natural experiments are not that common, unfortunately, and no comprehensive list of them exists. They usually come to researchers’ attention because one researcher develops incredibly deep local knowledge, sometimes through dedicated searching for circumstances with certain characteristics but often through some serendipitous occurrence, and begins to exploit that in her research. You may be aware of a couple that are related to your particular area of interest. Don’t discount this possibility, but don’t put all your eggs in one basket. You’ll definitely want to plan for a traditional multicase design, in the kind of process I outline below, but if an appropriate natural experiment falls into your lap, by all means—take it! The most common approach to case selection for a controlled comparison design begins with considering the entire population of cases and then eliminating ones that fall outside the scope of your study based on your scope conditions and assumptions. If you’ve got a theory of how parties recruit candidates for House seats and need states with relatively weak state-level party apparatus, then no matter what DV and IV values Illinois has, it’s not a good test case. Likewise, if you’re studying electoral fraud as a cause of civil conflict in developing countries, you must first identify all elections in all qualifying states. You may find you need to eliminate some of the “color revolutions” or “Arab Spring” events that you’d been thinking about because the GDP per capita of those states is above your threshold for being a developing country. Once you have a thinned-out population list, sort it by DV and IV value, even if it’s just a scratch table drawn by hand in your research notebook. Yes/ no or high/low (or high/medium/low) are appropriate ways to “measure” your variables in this table right now. Use these cells to help establish the set of cases you need based on your hypotheses. Normally, you need one case for each value of the independent variable (or, if the DV has more categories than the IV, one per DV category). Rule out any more cases that do not fit into the table or that are in cells of the table that you don’t need to study to test your hypothesis.14 Again, which cases fall into this latter category depends entirely on your 14Cases that do not fit at this stage—that are within the scope of the theory but which for some reason do not seem to involve your DV or IV (or for which the DV and/or IV are not measurable, or . . .)—are worth revisiting at a later date. Are they really within the scope of your study? Do they

127

128

Empirical Research and Writing

research question. If you don’t know whether you should study a case from a particular cell, review the guidance earlier in this chapter on how many cases you need and consider consulting with your instructor. Once you have a preliminary set of cases, you can then construct a working data table for them. Give each case a row of its own, and then establish preliminary values for the DV, IV, and any crucial controls such as ones proposed by competing theories or your assumptions.15 Having this information written down in a single location, such as your research notebook, will help you to write your methods section later—you’ll be able to recall your justification for making certain choices and to discuss comparable or other potential cases much more easily than if you just tried to remember it for later. A completed data table will give you all the information you need to begin making decisions about which cases to study in detail. Failure to identify any cases that have the desired IV value(s) and/or desired DV value(s) can be an important piece of information, but before you use this lack as evidence for (or against) a claim, you need to do due diligence. First, verify that you have used a complete population sample, or at least as much of one as you can gather from available sources. Check with faculty or others in your school who may have expertise in your topic. Second, verify that you’ve operationalized the concepts of your study in such a way that your measurement strategy is not inadvertently limiting your observations or screening out cases. Does a more generous but still conceptually defensible operationalization provide cases? Third, consider whether the process that creates events or observations is screening out cases—that is, whether a selection effect is occurring. Does some natural part of the process of how events or outcomes happen result in a truncated distribution of values on the IV or DV? If you can verify that your population list was complete and that you are not misassigning cases because of highly restrictive measurement strategies or measurement errors, then you can draw conclusions about the empty cell. For some research projects, lack of cases in a cell could be important evidence for a claim. This is particularly true for studies about necessary or sufficient conditions, but it could also be Sample Paper true for studies that allege a threshold effect for some variable, or that Using Controlled make claims about interaction effects. Comparison: If you’re planning to use controlled comparison techniques in your Verbal and Nonverbal Cues research, some useful references on the method include Clark, Golder, in Mass Appeals and Golder (2008, chap. 216) and Burnham et al. (2008, chap. 3). Falleti constitute a theoretically interesting subclass of cases that deviate from the expected categorization? These are topics you can revisit later, perhaps in the conclusion of your paper. 15This

is another point where hidden assumptions will often pop up. When you find yourself excluding cases because “they don’t fit,” be sure you can articulate why they don’t fit—it’s often an assumption you are making in your research but haven’t yet managed to put into words. 16The

first edition of this book contains a much more extensive treatment of Mill’s methods than the second edition.

Chapter 5  Case Selection and Study Design for Qualitative Research

(2005, comparative politics), Louderback (2009, international relations), and Putnam (1994, comparative politics) are good examples of the technique.

Structured Focused Comparison The method of structured focused comparison (SFC) is a between-case design that focuses our attention on specific facets of the cases to help evaluate hypotheses. The “structured” component of the name refers to a carefully crafted set of general questions that the researcher creates before embarking on data collection, much the way a survey researcher would craft a questionnaire before beginning to interview individuals or a quantitative researcher would design a codebook. In this way, the researcher obtains systematic, consistent data on study-crucial variables that he can compare across cases. The “focused” component of the name refers to the organization of the final case essay, which typically limits itself to discussion of only those facets of the case (variables) that are immediately relevant to the hypotheses at hand. Additional background and context information typically comprise one or two paragraphs at most (of a 40-manuscript-page paper, so less for a 20-manuscript-page student paper), with references to more detailed accounts. When you decide to use SFC for your study, you will want to plan to spend some significant time developing a data collection plan. I strongly recommend that you have a questionnaire, either as a data collection working tool or as a data analysis and management tool, that clearly lists the pieces of information you need to obtain for each case and a summary of each. Do you need to know the preferences of a certain set of actors about a particular policy? If so, list them, so you don’t miss any when you’re doing your data collection. Do you need to know how certain actors interpreted another’s actions? Again, list them. Skipping over or omitting information because you don’t know it or forgot it is frowned on in any form of qualitative analysis, but in SFC, full disclosure is expected. Most SFC-using studies pick their cases in pairs that vary on one variable at a time; theories involving two or more variables thus involve two or more pairs of tests. Because the cases are studied and presented in such a tightly focused manner, however, adding cases is not quite as huge an undertaking as it might be using other techniques. The strength of SFC designs is often found in the combination of tightly paired cases for each particular variable or hypothesis and more diverse contexts across the tests. Milner (1997, 27–28, 131–33) discusses this particularly well.

Study Design and Case Selection SFC is a between-case design, meaning that it requires at least two, and sometimes more, cases to provide a credible hypothesis test. It generally works on the “implicit regression” model discussed above for controlled comparisons. The biggest difference between SFC and controlled-comparison designs is the manner of focused and highly structured data collection, so case selection for SFC works generally on the same principles as controlled-comparison designs.

129

130

Empirical Research and Writing

Most SFC designs contain at least a small element of within-case interrogation, though, simply as a function of the types of questions they ask of the case. This additional means of support for hypothesis tests slightly lessens the need for perfectly controlled cases. In return, it opens up the possibility of selecting lessperfectly-matched cases in favor of ones that provide more variation on key concepts. Taking advantage of this increased flexibility requires you to be more familiar with both the criteria that produce hard tests for theories and the values of key variables across a number of cases. Uncovering and familiarizing yourself with variable values is not something I can help you with, but the criteria for hard tests are fairly clear. In a basic form of a hard test, the context of the case—the values of control variables—should be within the boundaries of your scope conditions and other assumptions, but near the extremes of those boundaries. Something about the case needs to make your theory unlikely to work, or at a minimum, it needs to make the outcome of interest unlikely to happen. Picking a case where the deck is stacked against your theory (or hypothesis) makes for a more convincing test because the outcome of interest is so unlikely. Highly improbable events or explanations are persuasive in Sample Paper ways that more likely things are not.17 Using Structured If you’re planning to use structured focused comparison in your Focused research, the most useful reference on the method is George and Comparison: Bennett (2005). Milner (1997) is the best published example of the Economic technique. I highly recommend that you consult Milner and/or the Conditions and Diaspora sample paper provided on the ERW website to get a good sense of what Formation a structured, focused comparison looks like—and how it differs from other comparisons—before you start writing.

Process Tracing Process tracing is the primary within-case method that we use in political science. It operates by means of using multiple facets within a case as evidence for testing theories. As this might suggest, its primary uses are for testing hypotheses about processes (as opposed to hypotheses about outcomes) and especially for testing claims about causal mechanisms. Process tracing often studies series of links within an event—the processes that led to a particular outcome—to determine whether support for a theory exists. All intervening steps must be linked in the way that the theory proposes to constitute support for the theory (George and Bennett 2005, 207). Because it focuses on within-case variation instead of between-case variation, its modus operandi and case selection principles do not follow the same rules as between-case methods. In particular, the provisions about degrees of 17For example, if some random online “friend” ran off and joined the circus, you might briefly wonder if he or she is crazy. If both of your parents suddenly quit their jobs and ran off to join the circus, you would be much more concerned that they’ve gone crazy. (No offense intended to parents or to circus employees.)

Chapter 5  Case Selection and Study Design for Qualitative Research

freedom do not quite apply in the same way as in the between-case methods; they are not hard-and-fast rules. Campbell (1978) refers to that kind of tallying-up design as “keeping a box score.” While contemporary (third generation) qualitative methodologists concur on the value of testing multiple hypotheses from a single theory in the context of a within-case design, they also generally believe that a “smoking gun” test can in some way count as extra weight in evaluating the theory. In other words, when testing a theory using process tracing, not all hypothesis tests are created equal. This inequality of tests derives from the type of observations that we collect in process tracing, which differ in an important way from the standard type of observations that we collect for other techniques (qualitative and quantitative). Our other analysis techniques all rely on what third-generation qualitative methodologists call dataset observations (DSOs), which consist of a string of variables on which each case has one value. The set of data created— the set of variables—remains consistent across cases, and different hypotheses simply require different combinations of these variables. What kind of event or phenomenon constitutes an observation remains consistent across observations. You can imagine this as a rectangular matrix where each observation (case) is a row and each variable is a column—in fact, this is the exact structure of a quantitative dataset. We then conduct analysis by comparing values or statistics within and between columns. Process tracing does not do this. It relies on the collection and evaluation of causal process observations (CPOs), which capture sequences of events or phases or characteristics within a single larger “case.” CPOs don’t have the same structure, the same sense of the same variables across all the cases and comparing variable values to test hypotheses, that DSOs do. A CPO may require data on three or four distinct if-then stages to be complete—getting from X to Y may take the form of “If X, then A, then B, which causes C, which causes D, which results in Y.” This is fundamentally a test of the causal mechanism for a particular hypothesis, which establishes the series of links that should be operating within the case.

Study Design and Case Selection Despite the different approach to thinking about data and testing hypotheses, case selection for process tracing proceeds generally along the same lines as our other qualitative methods. We employ background research to establish a general list of potential cases that meet our basic criteria of displaying the event or phenomenon of interest and then use other background research to establish rough estimates of the value of any crucial control variables. We can then use this preliminary data table to select the case that is most likely to provide a challenging test for our theory, both on its own merits and by being “easy” tests for competing hypotheses.18 As usual, be prepared to be surprised 18See

the discussion above under Structured Focused Comparison for more details on identifying hard tests.

131

132

Empirical Research and Writing

by the case that turns out to be “best” for testing your theory. Cases you weren’t aware of when you started the process often pop up.19 The primary source of support for theories in process tracing is the comparison to expectations generated by alternate theories (George and Bennett 2005, 217–18). This requires that not only must you have a well-articulated theory of your own (drawn from the literature or elsewhere), but you must also have well-articulated alternative explanations. Social science theories that meet this criterion in their raw state are relatively few, so be prepared to work through the process implications for your competitor theories as well as your own. A hard test—an environment where either or both theories are least likely to be successful—also generates additional support for the theory.20 The best advice I have for actually doing analysis via process tracing is to go slowly. You will need to do a lot of theorizing about intermediate steps before you can even start hunting down data to support them, and theorizing about intermediate outcomes often requires much reliance on assumptions. So take your time and think it through before you get too into the data collection process. At the same time, you’ll want to be open to spotting data that are consistent with your (or with alternative) explanations that you hadn’t really expected. Again, this requires knowing and being comfortable with the assumptions and causal mechanisms in both/all the theories. Expect that this process of theorizing and data collection will take a solid amount of steady work on a regular basis—this is not a technique well suited to procrastination. If you’re planning to use process tracing in your research, the most useful reference on the method is van Evera (1997); Checkel (2006) is also helpful. Advanced students might consult George and Bennett Sample Paper (2005, chap. 10). Louderback (2009, process tracing in the context of Using Process Tracing: Individual controlled comparison cases; international relations), Checkel (1999, comparative politics), Odell (2009, international relations), and Hacker Necessity and Path Dependence (2004, American politics) are good examples of the technique.

Writing Your Methodology Section The standard empirical paper format has as its third section a discussion of research design or methodology (i.e., the methods by which you will test your hypothesis/es). In a qualitative research project, this typically entails addressing four things: choice of research technique, case or sample selection, source 19I

used process tracing in part of my dissertation. To my surprise, the best case for testing my theory turned out to be the collapse of Albania in 1997—not the breakup of Yugoslavia in the early and mid-1990s, as I had thought. So I spent almost a year studying the politics and economics of the Albanian crisis, and the reaction of other European states to these developments. But there are still times when I ask myself, “Albania?!?!” 20The third source of support, out-of-sample testing, can also be useful, but the depth of research needed for successful process tracing often limits these to superficial discussions at best (Levi 1997).

Chapter 5  Case Selection and Study Design for Qualitative Research

descriptions, and measurement, coding, and/or dictionary. They generally appear in that order though some merging and rearranging of the latter two sections is possible depending on your research design. The point of this section, as I suggested above, is to convince your reader that your design choices are deliberate, that appropriate theoretical and methodological considerations underpin them, and that you are not cherry-picking evidence to manufacture support for your hypotheses. First, you should justify your choice of analytical technique. In most cases, you do not need to explain why you are using qualitative techniques instead of quantitative ones.21 Instead, you should address the particular form you chose—analytic narrative, process tracing, etc.—in terms of its fit for the type of hypotheses or theory you have. This is rarely more than a couple of sentences. It should cite some appropriate literature on qualitative methodology, such as those items recommended in the technique sections above, and/or examples of research using the same technique for similar kinds of theories and hypotheses.22 In essence, you need to make clear that the choice of technique is a reasoned and deliberate one on your part. Second, you should continue this effort to convince readers that your decisions are methodologically sound by discussing case selection. How did you identify the cases you chose? What factors shaped your decision to construct the test in the manner you chose? Why did you choose the cases that you did? What cases did you consider and discard, and why? These and the questions above are important elements of research design that need to be transparent and explicit to help achieve our scientific goal of replicable research. Remember that honesty is the best policy. Hopefully, you were making notes on these decisions in your research notebook while you were engaged in the process, so that you have something to reference as you write. The content of the third part of the methods section differs based on the particular qualitative technique you use. Users of content analysis will need to discuss both the approach(es) used to develop their dictionaries and the methods of identification and aggregation (see Chapter 6). The latter may include machine coding, multiple trained coders, solo coding by the investigator, etc. Users of structured focused comparison and methods of similarity will need to discuss the operationalization and measurement of key variables, including any unusual controls that were influential in case selection. Users of analytical 21You

should explain the choice of qualitative methods, however, if a legitimate choice between qualitative and quantitative existed, or if quantitative is the more appropriate solution but feasibility or data availability or the like restricted you to a qualitative study.

22Quantitative

research has to do this step as well, though it’s often done much less explicitly. The criteria for using various quantitative tools are much more widely known and agreed-upon, to the point where most authors do not bother to note that, for example, regression is appropriate because the dependent variable is a continuous unbounded variable measured at the interval-ratio level. They simply do it and assume that readers recognize the rationale. Because of the great variety of qualitative techniques and the existence of competing or unclear rationales for technique choice and case selection, qualitative researchers must still discuss these topics explicitly.

133

134

Empirical Research and Writing

narratives and process tracing must also discuss operationalization and measurement, but because each of the hypotheses tested involves different variables, these discussions normally occur in the context of each particular hypothesis test.23

Summary This chapter has introduced decision-making criteria for the three core choices of qualitative research design: the number of cases to study, which cases to study, and which particular analysis technique to use. Background research is important in identifying potential cases to study, especially cases that represent negative evidence, and in making case selections. Negative evidence plays a key role in maximizing the credibility of hypothesis tests by creating variation in the values of key variables. Common techniques in qualitative analysis include process tracing, content analysis, analytic narratives, structured focused comparison, and methods of similarity; principles of research design and case selection vary across these techniques.

Key Terms •• •• •• •• •• ••

Control variable (CV) Hard test Purposive sample Population census Selection effect Selection bias

•• •• •• ••

Negative evidence Possibility principle Dataset observation (DSO) Causal process observation (CPO)

23Again, expectations are somewhat higher, or at least more explicit, for qualitative work than for quantitative. Quantitative scholarship has a long tradition of standard indicators for various concepts and widely used datasets that allows shorthand reference to data sources and coding rules. Saying “I draw my cases from the Correlates of War interstate conflict dataset” or “I employ the 2004 NES” implies a number of things about the cases in the dataset, their selection criteria, and measurement of certain key variables.

T

Qualitative Data Collection and Management

6

he point of collecting qualitative data—what a lay person would call “doing research”—for an empirical paper is to test hypotheses. This is a fundamentally different purpose from other types of research you might have done for research papers in high school or other college courses. In practice, this means you need a carefully considered approach to collecting data, rather than the broad-sweep collection of information that you may have used in the past. Data collection is always the most time-consuming step of any research project. Inefficient data collection is both boring and frustrating, and it typically leads to poor research products. If you don’t know what you’re looking for, you’ll collect a lot of information but very little data, and you’ll have to spend a lot of time hunting for data as you write the paper.1 Planning ahead improves the data collection experience by making it shorter and more productive, and it facilitates creating a quality research product. This chapter begins by briefly reviewing the distinction between data and information, and it discusses issues of measurement as they relate to qualitative research. Next, it considers several prominent forms of qualitative data collection in detail and provides guidance for implementing these techniques in your own research. The third section suggests ways to maximize your leverage over your hypotheses by means of careful and thoughtful data collection. The fourth section identifies several particular types of archival and primary-source resources that you may have access to. Fifth, I present several useful tools for managing qualitative data, including software and hard-copy formats, and I conclude with a brief discussion of writing about your data collection process.

Information, Data, and Evidence Good qualitative research often relies on two or more types of sources and two or more methods of data collection. This allows for triangulation, a process by which findings and evidence from one type of source or analysis are buttressed by findings or evidence from another. Making sense of qualitative data thus 1The

aphorism “If you fail to plan, you plan to fail” is quite à propos here.



135

136

Empirical Research and Writing

requires that you be familiar with the types and forms of qualitative data that exist, and how data differs from ordinary information.

Information versus Data Information is facts. Data are intentionally gathered facts. This is not a trivial distinction. The key thing about data is that they are gathered for a specific purpose. Data are usually defined by the variables in our hypotheses, which is to say, by predefined data needs. Data help us test our hypotheses, usually by providing values for specific variables in our theories and hypotheses.2 In qualitative empirical research, we are typically interested in just the values of relevant variables because these serve as evidence for our claims. What this means in practice is that in a given empirical investigation, you will collect an awful lot of information. Not all of that information is data relevant to testing your hypotheses.3 All pieces of data are pieces of information, but not all information is data. So, with that in mind, let’s define some key terms. An observation is a single instance of the phenomenon under investigation. A variable is a systematic characteristic of an observation. A dataset thus (ordinarily) contains one value of each variable for each observation. One of the big differences between qualitative and quantitative data is how each defines an observation. For quantitative studies, all of the observations are (usually) of the same phenomenon, or, at least, of instances where that phenomenon could have occurred but may or may not have actually happened. For qualitative studies, what constitutes an observation can differ across hypotheses within the investigation, and in fact, this is the driving idea and operating mechanism for the entire family of within-case designs. Within-case designs function on the principle that each “case”—each historical incident or occurrence of an event, such as a “social revolution” (Skocpol 1979)—consists of many separate observations about different hypotheses relating to various facets of the case.4 Between-case designs have datasets that often look more like the quantitative format, in that the researcher collects data on the same observations/cases/ events for all the hypotheses. I’ll complicate this discussion further by noting that all evidence is data, but not all data serve as evidence in the context of any given argument. One of the most challenging aspects of qualitative research for many novice researchers is wading through the morass of data and identifying the specific bits that can function as evidence. This is particularly tricky because at the outset, we don’t know all the details of a case, and so we can only make general guesses 2Note

that the term data is plural. The correct singular form is datum; you’ll more commonly see references to a data point.

3By

implication, then, not all of it belongs in your paper. Be selective in what you include.

4Most

of the time, when qualitative researchers are referring to a case, they mean it in this broader sense, rather than the more narrow sense that quantitative researchers use (in which the terms case and observation are equivalent).

Chapter 6   Qualitative Data Collection and Management

about the types of information that would serve as good evidence for or against our hypothesis. Once we get into the data, we may find other data that we weren’t expecting. We may not find what we were expecting, either. Learning to gather enough raw information to have sufficient evidence, but simultaneously filter out the irrelevant or unnecessary stuff, is a skill that takes time to develop. It requires that you learn to think broadly and carefully about observable implications—and falsifications—of your hypotheses both before and during the data collection process. This takes time to develop. We’ll talk more about these steps later, but the best thing you can do is to think twice and act once.

Measurement Converting a piece of information into a piece of data requires measuring the variable’s value. Measurement is the process by which information is converted into comparable, systematized values of specific variables for specific observations; it is the link between information and data. The most common way to think about variables and measurement is to consider the level of measurement, which indicates how fine-grained the measurement is. Items measured at higher levels typically have more precision and comparability than items at lower levels. The highest level of measurement that we usually use in the social sciences is the interval-ratio level. Items measured at the interval-ratio level are explicitly numeric and have a unit attached to them: 37 days, $12.8 billion, 67.1% of votes. While individual variables in your study may be measured at this level, for example, time between committee and floor vote on a piece of controversial legislation, the majority of data used in qualitative analysis is not measured at this level. Even in the case of the time-between-votes example, a researcher investigating leadership strategy is probably interested in whether that duration was relatively short or relatively long, not the exact duration of the delay itself. The two less-fine-grained measurement levels are more common in qualitative research. In fact, many scholars believe that one of the hallmarks or key features of qualitative research is its ability to work with variables and data that do not lend themselves well to interval-ratio measurement (cf. Collier and Levitsky 1997). Variables measured at this level are less precise, in the technical meaning of the word, but are often more nuanced. Instead of having to indicate yes/no or how much, we can measure the variable with ideas like “to what extent?” or “of what type?” The intermediate level of measurement, ordinal, responds to the former question, and nominal to the latter question. Ordinal measurement creates variable values in ranked categories. We can order the categories from most to least, lowest to highest, strongly disagree to strongly agree, or any of a variety of similar scales. The key thing is that the categories themselves have an ordering or ranking to them. This differentiates it from the lowest level of measurement, nominal, in which we have categories with no relationship to one another: race, continent/region, marital status, etc.

137

138

Empirical Research and Writing

Practice 6.1 Identifying Levels of Measurement For each variable or indicator below, indicate whether that variable is measured at the nominal (N), ordinal (O), or interval-ratio (I-R) level of measurement. If you think that more than one answer could be correct, jot a quick justification for the answer you chose. A. Support for affirmative action (Likert scale) B. Religion C. Number of years as a European Union member D. Religiosity E. First-time voter F. Age at birth of first child G. Former colonial metropole H. Date of entry into World War II

Just because a variable’s value is a number does not make it “quantitative data.” The number of Supreme Court justices voting for or against a particular decision, or the count of how many times broadcasters use the words ethnicity and race correctly—those are all forms of fundamentally qualitative data. We might be interested only in split decisions of the US Supreme Court—ones where the decision was 5–4—to study patterns of legal reasoning used in dissents and supplementary dissents. Do the supplements differ from the main dissents in their use of legal analogy or philosophy? Do the dissents of split court decisions differ from the dissents of clear-majority cases in the number of precedent cases they cite? The vote was 5–4, but what that number means to us is that the decision was “split,” which is the value of a variable—closeness of decision—of each observation.

Validity and Reliability in Qualitative Measurement Doing good research, qualitative or quantitative, requires measures that are both valid and reliable. Validity addresses whether an indicator captures the concept it is intended to capture and nothing more. Reliability refers to our measurement tool itself. A reliable measure is one that returns the same value for a given case even when multiple individuals evaluate the case according to our rules for converting information into data. Achieving and demonstrating reliability and validity in qualitative measurement is somewhat more challenging than in quantitative measurement. With interval-ratio data and a large number of cases, quantitative researchers can employ a range of statistical tests to verify that their measurement strategy is effective.

Chapter 6   Qualitative Data Collection and Management

Qualitative researchers, alas, have no such tools, and the inherent nature of qualitative evidence makes establishing validity and reliability more complicated. When the goal is to identify gradations, characteristics, or nuances, as it is in so much qualitative research, establishing values for data becomes substantially more subjective and open to interpretation. Speakers and authors rarely manage to tell us everything we need to establish the kinds of fine-grained distinctions and assessments that we’d like to make. Does a particular quote establish that a speaker is strongly in favor of a particular position, or only moderately in favor of it? Or is she strongly in favor of some aspects of the position and only moderately favorable (or even unfavorable) toward other aspects? In these cases, reliability is clearly a problem, and validity can be problematic as well, depending on how well the available evidence aligns with the research question and data needs. This is why identifying observable implications, expected values, and potential evidence before you start researching is so important. Essentially, you are developing a measurement strategy: determining what values of what characteristics constitute data for testing your theory.5 Be as explicit and articulate as possible about your measurement strategy, especially when you write your research methods section. If you have hypotheses about when actors should be concerned about the perceptions of others, what kinds of things would you expect them to say that indicate such a concern? Would public remarks carry more or less weight than private ones? How do you know the actor is really stating his or her true belief and not just posing? Every qualitative researcher needs to be able to answer these kinds of questions for every variable you collect. “I’ll know it when I see it” is not a valid answer for research design and measurement, especially not when the goals of research include transparency and replicability.

Maximizing Leverage over Your Hypotheses Maximizing leverage from your data requires having clear observable implications and expected values before beginning data collection. Obtaining those observable implications, however, and maximizing leverage over hypotheses, require you to have clearly articulated hypotheses first. You should not collect any data at all without clear hypotheses. Read this chapter for ideas, but do not even begin exploring the library or the Internet without meeting with your instructor to get yourself good, workable hypotheses. Without clear hypotheses and a research plan to match them, you do not know what data to collect, and you will simply waste a lot of time and energy. A meeting with your instructor, and a half hour to plan and strategize, will save you hours of research time and make the time you spend more productive. That said, you will have a much more successful (and less frustrating) data collection experience if you think ahead about the types of sources that may be useful to you, and think about ways to expand your observable implication set. 5You may need to cycle between developing observable implications and hunting for evidence to refine your ideas, but this is normal.

139

140

Empirical Research and Writing

Kinds of Qualitative Data Sources We can classify qualitative data and their sources on a number of different dimensions. The first dimension is the frequency and regularity with which data are produced. In the running record, reports or data are produced systematically on a pre-arranged schedule: daily, monthly, annually, or the like. The Congressional Record is an example of a running record, as are newspapers, statistical series, and ships’ logs. Well-maintained diaries, some government records, and church bulletins can also fall into this category. Running records are expensive to maintain, but they can contain the most useful data for the researcher if and where they exist. Episodic records, in contrast, are produced more sporadically. These may include letters, government or other reports, less frequently used diaries, and presidential speeches. The contents of the Foreign Relations of the United States and similar series from other countries are likewise episodic, even if the series is published regularly. Episodic records can provide deep and important insight into particular instances, particularly periods of great stress on or interest to the creator, but their ability to provide details about everyday experiences or less stressful or interesting situations can be a significant limitation.6 A second approach to primary data source classification is by origin: public, private, or media. Public data are generated by government entities, including international organizations. This includes data created by public entities that are not publicly available, such as classified documents. Privately generated data, in contrast, include information produced by individuals such as diaries and letters, and also information produced by private organizations such as businesses, nongovernmental organizations, and interest groups—even if these data are publicly available. Media sources are an in-between kind of category. They are produced by private entities for the explicit purpose of widespread public dissemination. Finally, the most commonly referenced form of data classification deserves a little discussion: primary, secondary, and tertiary sources. Primary sources have no analysis or other filter between the source’s creator and the reader. The classic examples of primary source documents are letters and diaries written by individuals involved in events, but other materials such as interview transcripts, posters and movies, and even newspaper reports can be primary sources. Secondary sources, on the other hand, have analysis between the original data and the reader. These often include other scholars’ research reports (papers and books) about some event for which they consulted the original primary sources; most books by historians fall into this category. Finally, tertiary sources are summaries of secondary scholarship; at least two 6Johnson and Reynolds (2014, chap. 9) provide an excellent discussion of advantages and disadvan-

tages of using written records as sources.

Chapter 6   Qualitative Data Collection and Management

layers of analysis intervene between the reader and the original data. The most common forms of tertiary sources include encyclopedia entries, textbooks, and literature reviews. Most qualitative empirical scholarship avoids citing or using tertiary sources for data collection. Ideally, data come from primary or at worst secondary sources; tertiary sources function primarily for background research.

Data and Counterfactuals Qualitative analysts are also disadvantaged vis-à-vis quantitative analysts in the precision of their estimates. Multivariate quantitative analysis generates, as a matter of course, both estimates of uncertainty about observed relationships and also predicted outcomes under unobserved combinations of variable values. Statistical significance and confidence intervals provide information about the certainty of the relationship; we can evaluate expected values of the dependent variable (DV) for any combination of independent variable (IV) values. Both of these devices play a key role in making convincing arguments in quantitative research. Achieving either of those objectives—uncertainty estimates or unobserved predictions—in qualitative analysis requires special effort on the researcher’s part. Estimating confidence (uncertainty) requires the author to subjectively estimate and explicitly discuss his or her degree of certainty about the observed relationship. This involves considering the entire mass of evidence for and against a particular claim, in toto, and summarizing and evaluating those data for your reader. This may involve stating that the historical record is silent on a particular issue, that the evidence is conflicting, that credible sources dispute the memoir’s claims, or a range of similar remarks. Of course, it’s not quite that simple in practice. One major complicating element is that qualitative analysis generally tests hypotheses one at a time, in a bivariate kind of manner, with case selection providing the primary controls for confounding variables. But perfectly matched cases are few and far between, and that makes claims about “all else being equal” difficult to make because, well, all else isn’t equal. Quantitative analysis tests multiple hypotheses in a single model and allows controls for all possible/plausible confounds at any observed or unobserved value; qualitative methods don’t. This makes ruling out competing hypotheses and alternative explanations incredibly difficult. The solution that qualitative methodologists currently recommend is the use of counterfactuals. A counterfactual claim is a statement of the form, “If X had (not) happened, then Y,” where X and Y represent an independent and dependent variable respectively. Most of the time, we use counterfactuals in qualitative research to argue that alternative values of (often poorly controlled) control variables (CVs) would have no effect on the DV. Scholars also use them extensively in process tracing, where the focus on a single case means that only one value of each variable is actually observed, and in tests of necessary and/or

141

142

Empirical Research and Writing

sufficient hypotheses, where what we would expect at unobserved values is just as important to testing the hypothesis as the observed values are. Counterfactual examples involve a thought experiment in which you reevaluate historical events or cases as if one of the IVs or CVs had a value other than the observed value. For example, if Europe was governed primarily by center-left parties in late 1989, and not by center-right ones, would the Cold War have ended the way it did? If the proposed necessary and/or sufficient cause were not present in a particular case, can we credibly argue that the outcome would have matched our prediction? Working through a counterfactual example requires careful reasoning about causal sequences and about the theoretical relationship between variables. Writing a good counterfactual is not easy. The most credible counterfactuals have three key characteristics. First, they change only one variable value at a time. Changing more than one variable at a time complicates the task of drawing conclusions about the effects of the change. Think about it from a case-control framework. If only one variable’s value differs between the two observations, then logically, no other variable could have caused the observed outcome. Second, the changed variable is temporally and/or causally proximate to the outcome of interest. A claim that “if colonialism hadn’t occurred, Africa would be the richest continent” would be very hard to make credibly because many other factors besides colonialism affect the global distribution of wealth. For instance, would Europe have developed and industrialized rapidly anyway if colonization had not occurred in Africa? European economic development predated colonization; perhaps it would have continued (maybe at a slower pace) without resource extraction from Africa. Economic (mis)management by African rulers themselves is a significant cause of continued poverty there; would an absence of colonization really lead us to believe that African rulers would be that much better at economic management? Discrete changes in meso- or micro-level variables (as opposed to macro-level variables like colonialism or development/wealth) lead to more plausible counterfactuals. If Al Gore had won the 2000 US presidential election, would education reform still have occurred, and would it have taken the same form? That’s a plausible set of changes whose effects could reasonably be projected in the context of the case. Finally, the alternate variable value in the thought experiment should be plausible. Ideally, it should fall within the range of observed values in the population, or at least be a reasonable projection of existing trends. “If the medieval Crusaders had nuclear weapons” is not a plausible value for a variable about sophistication of military technology, nor would “if Martin Luther King Jr. had been elected president” be a plausible value for political leadership in the US civil rights movement. In the context of their respective cases, these are just not believable possible values for those variables, and counterfactuals built on them would not be persuasive.

Chapter 6   Qualitative Data Collection and Management

Practice 6.2 Evaluating Counterfactuals Identify the DV, IV, hypothesis type, and original value of the changed IV (you may need to do a bit of research on Wikipedia for this). Determine whether each proposed counterfactual would be credible; briefly explain why or why not. A. Southern planters’ dissatisfaction with their indebtedness to British banking houses was a necessary condition for American independence. If southern plantation owners had been able to bankroll their own operations in the colonies (instead of through London), support for American independence would have been much weaker in the southern colonies. IV: DV: Original value: H type: Credible? B. 9/11 would not have happened if the CIA and FBI had been able to share information. IV: DV: Original value: H type: Credible? C. Had Hillary Clinton gone to the University of Chicago and studied conservative economics there, she would have been the first female president of the United States as a Republican. IV: DV: Original value: H type: Credible?

(Continued)

143

144

Empirical Research and Writing

(Continued) D. A Nigerian independence constitution drafted with Nigerian input would have made both civil war and military coups far less likely there. IV: DV: Original value: H type: Credible?

Data Collection Techniques Data can come from almost anywhere, and the number of ways in which we could obtain it is likewise quite large. Qualitative researchers have developed a battery of specific techniques on which they frequently rely to gather their data efficiently and with minimal bias or other undesirable influences. The next two subsections briefly review common techniques of qualitative data collection. For most social-scientific purposes, we can distinguish data as being from human subjects, or as being from sources (sometimes called “texts” in the humanities). Collecting data from human subjects by interacting with them in some way requires attention to very special protocols, which we discuss below. Most of our data collection, and almost all data collection that novice researchers do for class projects, is from sources—from documents, existing datasets, and similar text- or visual-based resource sets. We turn first to those before addressing human subjects research.

Research from Sources The most common form of qualitative data collection is document analysis, or the systematic review of primary, secondary, and sometimes tertiary sources. This is the form of data collection with which you are probably the most familiar. Effective data collection from secondary and tertiary sources requires several strategies depending on the types of sources used. First, use bibliographies to find new sources. This process of bibliography hopping allows you to take advantage of the work researchers before you did in finding and collating sources. If you’ve allowed enough time in your research plan, you can probably request any major sources or titles your own library doesn’t have via interlibrary loan (ILL) in time to use them for your own research. Second, smart researchers make active use of indices and tables of contents, especially detailed tables of contents. No one has time to read all of the

Chapter 6   Qualitative Data Collection and Management

document(s) or book(s) in search of specific information. Use these built-in tools to make finding information easier. Don’t forget to consider synonyms, alternate spellings, and related terms when you use the index to a scholarly book. Consider looking for concepts as well as proper nouns or specific cases as well. Older titles and collections of primary sources often lack indices. In these cases, use in-document markers such as subheadings, tables and figures, dates, and other prominent and easily-spotted-on-a-skim items as signposts in your hunt for relevant information. Most primary sources, unfortunately, do not contain useful features such as bibliographies, indices, or even subheadings. As a result, collecting data from primary sources is often a tedious process, particularly if the sources are not available in a digital format. Digitized documents are often searchable, if only via the CTRL+F function of your web browser or word processing program, which can speed the process of identifying relevant passages. Again, though, having a good sense of what you are looking for before you start reading documents allows you to skim more effectively.

Archival Research The sections below on sources and resources provide guidance on locating archives that might hold relevant materials. While most archives do not have their full collections digitized, most do have digital versions of their finding aids, which are guides to each special collection or archive containing information about the collection’s substantive and temporal scopes as well as the origins and types of materials included. Archival data collection is one of my favorite kinds of research tasks. Something about getting into the data and getting my hands dirty gives me great joy.7 My biggest advice for archival work is to be prepared. First, go in with a research plan. Often you must create and communicate an overall master plan to the archive staff in advance to allow them to pull or locate the relevant records in time for your visit; the increasing number of archival finding aids available online greatly facilitates this task. That said, even small tasks in the archive will take longer than you expect them to, so have an action plan ready. This should be a priority list of data you absolutely must obtain from these sources, followed by data that you would love to have from these sources but can construct from others if necessary. This way, you can maximize the effectiveness of your time in the archive.8 If the archive’s finding aids did not explain the organization of the specific files you are using, ask the archive staff

7Yes, I’m a nerd. If you didn’t realize this by now, I wonder how much attention you paid in the pre-

vious chapters. The dirt comment is also literal—less-well-tended archives can be incredibly dusty. Plan to take a dust or surgical mask if you’re allergic or sensitive.

8Take some other work to do in case you need to wait for items to be retrieved. (Wait times at the Library of Congress are notorious, though they’ve improved dramatically in recent years.)

145

146

Empirical Research and Writing

for an orientation when they deliver the files, and be sure to ask about how and where to return files. Second, be prepared to be flexible. The evidence you want may not be in the precise form you want. Be open to considering new forms of evidence— new observable implications—for the hypotheses and underlying theory that you are testing. Remember that word choices and expressions differ over time and space. Things you expect to find may not appear in the form you expected. Third, know the policies of the archive before you get there. Some allow you to photocopy items on their machines for a small fee; others charge a (usually rather hefty) fee for them to copy materials for you. Some allow you to take digital photographs of the sources rather than photocopy; others ban cameras, including those on cell phones, and/or they may ban phones altogether. Having a camera (or not) or enough cash (or a check or roll of quarters or a wrist watch . . . ), and knowing things like where to park or get lunch, can greatly reduce the stress of using an unfamiliar archive. For further information on accessing, using, and analyzing archival data, you can consult Lustic (1996), Trachtenberg (2006), Burnham et al. (2004, chap. 7), or Elder, Pavalko, and Clipp (1992).

Text Mining A second form of document analysis involves text mining, or combing texts to collect data for content analysis. Scholars who use text mining often then use content analysis to seek patterns or changes in the way that actors use language.9 The key to successful text mining is a good dictionary, or list of terms of interest. The terms in your dictionary are the ones you hunt for and count in the texts. So any text-mining data collection project is only as good as the list of terms it begins with. (Again, we see why theory and thinking ahead matter so much!) The term dictionary may be a bit of a misnomer. Your dictionary may consist of individual terms such as taxes and unemployment, or it may include more sophisticated and open-ended concepts such as the use of historical analogies, employing negative or positive frames about a specific issue, or instances of legal reasoning. Whatever you are going to be looking for, try to have a working list of as many entries as possible. You need to know, at least partially, what you are looking for before you start searching. Developing a successful dictionary requires several iterations and pretesting against a sample of the data. The ERW website has a longer discussion of pretesting practices, but in short, you begin by establishing a preliminary dictionary based on the observable implications of your hypotheses. Test this dictionary by coding a small random or semi-random sample of your texts. Make copious notes as you code to ensure consistency across the full sample. 9This strategy is sometimes described as content analysis. I reserve that term for analysis of data gathered using text mining or similar strategies.

Chapter 6   Qualitative Data Collection and Management

147

Adjust the dictionary as needed to accommodate additional terms and then retest the dictionary. When you are reasonably satisfied with your dictionary, begin the actual data collection process. Be aware though that any additions to the dictionary after you begin the formal coding will require going back and recoding anything done prior to the additions—which means keeping track of what was done when and what changes were made when. This is a tedious and time-consuming process. Fortunately, you can largely avoid it by thinking ahead and pretesting; more information on good pretesting practices is available on the ERW website. As you code your documents, always begin each session of coding by rereading your coding rules in their entirety. Even if you think you know them, force yourself to look at every word; don’t just skim. During the coding process, refer frequently to your examples, descriptions, and other notes. Depending on your materials and coding needs, you may want to be keeping notes in your research notebook about unusual or questionable classifications so you can adjust or explain them later. Pretesting Your When you encounter an item in your pile that you coded previData Collection ously in the pretest phase, do not treat it differently; code it normally, Procedure and without reference to the pretest copies. Then, compare your pretest Coding Rules and midprocess codings. This will allow you to conduct a crude interreading reliability test to ensure that you are coding consistently. Finding a few minor differences between codings is normal.10 If you find these, review your coding rules and examples, determine which coding is correct, and add to the notes/examples as needed to help ensure future consistency. If you conduct these reviews regularly, you should be able to avoid any major problems.11 If your midprocess review identifies substantial discrepancies between your pretest coding and your midprocess coding, however, you should stop immediately and plan to consult with your instructor or another experienced researcher on how to diagnose the extent of the problem and rectify it. For further information on and guidance about doing text mining, you can consult Weber (1990), Sommer and Sommer (2001), and the web links in Chapter 5; Schmidt’s (2010, 93–95) discussion of content analysis also provides a good process checklist. Hopkins and King (2010) provide more information on automated text mining, including a software package (implemented in R) that automates coding of large text samples (available at http://gking.harvard. edu/readme). 10No

firm criterion exists for the desired level of inter-coder (reader) reliability; Schmidt (2010, 93) cites a threshold of 80%. Personally, I feel that a 1-in-5 error rate is a little too high, even for undergraduate class research. When combined with a p < 0.1 significance threshold, or even a p < 0.05 level, the risk of a Type I error is simply too high. I’d want to target 90% or better for my own research. 11You can help to ensure this by shuffling your source pile and deliberately choosing/replacing pretest items from throughout the stack.

148

Empirical Research and Writing

Human Subjects Research Research involving human subjects is governed in the United States by a federal regulation. Participation must be voluntary and on the basis of informed consent, and researchers must ensure that they conduct their research—including managing the resulting data—in a way that minimizes potential risk or damage to research participants.12 Each college or university manages its compliance with this regulation in part by requiring a review of research plans involving human subjects by a human subjects research committee.13 This committee ensures that research is conducted in ways that minimize subjects’ risk. The status of student research under this regulation is unclear, and policy toward undergraduate research especially varies across schools. Some schools automatically exempt all students conducting research in the context of a class assignment, while others only exempt undergraduates. Others require professors to apply for exemption on their students’ behalf, either individually or as a class; some require individual students to apply on their own behalf, even for undergraduate classroom research projects. Whether you must go through a human subjects clearance process, and the specifics of that process, vary by school. You the researcher are responsible for compliance with this regulation, whether you know about it or not. You are responsible for seeking information about compliance and review procedures; failure to do so carries significant risks for you, your instructor, and your school. So do yourself a favor and make sure you know what you need to do to be safe. A word of warning: The human subjects approval process can take a significant amount of time, so be sure to allow for this in your research timeline. You may wish to have a backup data collection plan in place so you can meet any deadlines. A number of forms of human subjects data collection exist. Elite interviewing, mass interviewing via survey, focus groups, and participant observation are a few of the most common. In the interests of space, I address elite interviewing here. Discussions of and resources for surveys, focus groups, and participant observation are on the ERW website.

Elite Interviewing Elite interviewing is a very effective way to get information for some types of hypotheses, particularly process ones, though the interviews themselves can be somewhat hard to obtain. Elite interviews are conducted with respondents 12Outside

of the United States, the specific names and mechanics vary but similar principles apply; Burnham et al. (2004, chap. 11) provide an excellent discussion of the situation in the United Kingdom. 13This body goes by a variety of names, including Institutional Review Board and Human Research Committee. Larger schools often have several different boards that focus on different types of research, such as medical research and social research. Your instructor will know more about specifics for your school.

Chapter 6   Qualitative Data Collection and Management

149

who are of interest to the researcher because of some position that they occupy, not because they are representative of some larger population.14 These positions might be elected, such as lawmakers, or appointed, such as bureaucrats; they need not be high-profile individuals to be “elites” for interviewing purposes. The most common way to identify elites for interviews is through documentary analysis or appropriate directories, such as the annual directories of members and staff published by the US Congress, or the various “Who’s Who” directories. Newspaper and magazine articles may quote appropriate individuals; interest or advocacy groups may also have relevant individuals on their staffs or in their networks of contacts. As you conduct your set of interviews, you may choose to engage in snowball sampling to identify other potential respondents. In a snowball sample (sometimes called respondent driven sampling), respondents are asked to provide names and possibly contact information for other individuals they feel would be useful to interview. This is particularly helpful when studying informal institutions and organizations such as social movements, protest groups, and the like, which do not have stable leadership structures, organizational directories, or even permanency that would allow you to locate the appropriate individuals in any other manner. One disadvantage of snowball sampling is the potential for bias as a result of the individuals sampled. The sample grows like a tree branch, from one base root (the initial interviewee) to further branches sometimes several nodes removed from the initial contact; you may find yourself “out on a limb” in an extreme or unrepresentative part of the organization without knowing it. So interviewing broadly, across individuals identified from as many different sources as possible, is a key step in getting trustable data from interviews. When requesting an interview with someone, especially a public figure, always initiate formal contact with a snail mail letter, and always end contact with a snail mail thank-you.15 For large offices, you may need to email a general request first to find out to whom an interview request should be addressed and where it should be sent. For example, some Congressional offices prefer that interview requests go through their local media liaison, who will then filter them to the member as appropriate; others will want all requests sent to the Washington office even if the interview would occur in the home district. If you Interviewing are interviewing individuals identified through online searches, you Representative may not have snail mail contact information available; in these cases, Samples 14Interviews

with representative samples can take the form of mass interviews, such as those conducted for survey research or focus groups; brief discussions of these issues are on the ERW website.

15Depending on your school’s policies, you may be able to use school letterhead for initial elite interview contacts. If you are trying to interview senior figures, letterhead is almost necessary to get your foot in the door as it shows you are conducting officially sanctioned research for a significant project. Using letterhead may require your faculty adviser or sponsor’s assistance; check with him or her for details if you think this may apply to you.

150

Empirical Research and Writing

a formally written email from your school account will suffice. Use “official” contact means such as your school email account (rather than a personal one) as much as possible to reinforce that this is a formal request for research.

Insider Insight

“To get interviews, pitch to people based on your shared interest in the particular topic. Also, use social media, alumni networks, LinkedIn, Facebook, etc., to see if you have any mutual friends or connections who can facilitate an introduction to a certain person or give you ideas about who you might talk to. I ended up reaching out to some alums from my undergrad for my MA research and one of them was shocked to hear that I was having trouble getting meetings with people at the nonprofit I was doing research on. [A]fter that conversation I sent emails and got replies. And I was able to get an interview that I never could’ve gotten before. If your topic is a hot-button issue or something in the news, get the local politicians on your list of interviewees. Even just saying that you’ve already talked to the city council member about it can help open doors.” —Prof. B. Davis, Allegheny College

Typically, elite interviews must include a statement of informed consent, especially if the interview is recorded. Some exceptions to this rule exist; your particular school’s human subjects board can provide more information. At a minimum, even if a formal written informed consent document isn’t required, you should obtain verbal consent from your subject at the beginning and end of the interview to use direct quotes from him or her in your research. The extent to which the individual can be quoted and his or her quotes can be attributed varies across the level of confidentiality at which the interview occurred. At a not-for-attribution level of confidentiality, you cannot even indicate that the information came from an interview; this severely limits the utility of information gained this way as evidence in your paper, since you have no way to cite a source unless you can triangulate the claim from another source. At the more common background level of attribution, you can cite the information as from an interview conducted with an individual identified by position or category (rather than by name), which allows you to use this information as evidence in your research. Depending on the extent of consent given by the interview subject, you may or may not be able to quote background interview material directly. At the fully attributed level, you can quote the interviewee directly and attribute the remarks by name to their speaker. This is uncommon in social

Chapter 6   Qualitative Data Collection and Management

science research since anonymity and confidentiality allow speakers to give honest opinions without fear of reprisal (in the case of negative opinions), and they lack incentives for interviewees to posture and take public positions that may not actually reflect their real preferences or motivations. That being said, we can never be sure that any one individual’s recall of events is factually correct, that he or she has correctly interpreted other actors’ positions or actions, or even that he is giving true and accurate information in the interview. Unless you are particularly interested in individual responses and attitudes to something, you should always try to triangulate your evidence and confirm dates, positions of other parties, etc., with other sources. You must be careful here to protect the confidentiality of your source, but as with journalism, independent confirmation of facts vastly increases our confidence in the reliability and accuracy of the information. Interviews—and your notes from them—are considered confidential information by most human subjects boards. This means you must take precautions to prevent unauthorized individuals, including people like your roommates, classmates, etc., from having access to them in forms where individual respondents (interviewees) are identified or identifiable. Depending on the conditions imposed by your own school’s human subjects board, this may entail having to keep your notes in a locked container when you are not currently working with them, maintaining notes or records in a particular manner or for a specific length of time after the project, and/or destroying them in a secure manner. At a minimum, you have an obligation to your sources to protect them and their confidentiality, which means that even if your human subjects board doesn’t impose conditions, you should still treat the notes with care. Successful interviewing is a learned skill bordering on an art form. As with archival research, the biggest tip is to be prepared. Learn as much background information about the individual and the situation as you possibly can before the interview. This allows you to ask specific and precise questions, which is good; the more specific the question, the more specific the answer is likely to be. Your time with your subject will be limited, so precise, targeted questions about the matters you need data on are best. If you’re interviewing a member of Congress about campaign advertising, “How important is your opponent’s use of negative ads in determining your own media strategy?” is usually a much better question than “So what kinds of things do you think about when you’re planning a media strategy?” An even better question, or a follow-up, might ask the respondent to rank several factors in terms of their influence on his or her media strategy; this ranking could then serve as a springboard for further questions and better discussion. For further information and guidance on using elite interviews, you can consult Hancock and Algozzine (2006, chap. 6) and Burnham et al. (2004, chap. 9). Kim and Sharman (2014), Kirschner (forthcoming, chap. 4-5, international relations), Lin (2002, US politics), and Art (2011, comparative politics) provide good examples of the use of interview data.

151

152

Empirical Research and Writing

Sources and Resources Qualitative research often requires more than basic library skills. If you have not yet done so, you should plan to sit down with a reference librarian (ideally the political science or social science subject matter specialist, if your library has one) for a few minutes to get an orientation to available resources. Each library differs in its electronic and print collections, and in peculiarities of the catalog system. Your own librarians are best equipped to help you find the best way to navigate the system and locate useful material. Remember, reference librarians are information specialists: their job is to know what’s available and to help you learn to access it. Broadly speaking, data for qualitative research can come from three sources: print collections, electronic collections, and special collections. Print collections include library-held manuscripts and monographs (“books” in common parlance), specialized reference volumes such as subject matter encyclopedias, and print collections such as the Foreign Relations of the United States series and Keesing’s Record of World Events. Printed collections of primary documents can be an invaluable asset, especially in process tracing research. Specialized reference volumes are usually in the reference section of the library rather than in the stacks. I’ve found them particularly helpful in establishing consistent values for variables across multiple cases, such as in structured focused comparison or case control designs. Your own librarians will know what resources are available to you for your particular topic. Special collections, in contrast, are typically archival. They are the direct papers of individuals or groups, rather than reproductions. Sometimes these collections are multimedia and may include artifacts from the event or period. A special collection of state election campaign materials may include posters, buttons, photographs of stump speeches, recordings of a debate, and the minutes of the state party nominating meeting. College and university libraries often maintain their own special collections and archives, which contain school history materials as well as special collections that alumni or public figures may have donated or entrusted to them for maintenance. Occasionally, local libraries and governments, historical societies, and other organizations will maintain their own archives. If you are doing research on local or state politics, you should definitely consider tapping into these resources. Most archives require permission and appointments for access, but if an appropriate collection is available, it is almost always worth using. Again, your own librarians will know what collections your school holds, and they may be aware of collections at other local schools or institutions that may be of use to you. The OCLC ArchiveGrid also contains a (partial) list of special collections maintained at member libraries. Special collections materials typically do not circulate; they require on-site use. Finally, electronic collections include items on microform and microfiche, such as historic newspaper archives, and databases of full text documents. Collections of papers and letters from historic figures are often available on

Chapter 6   Qualitative Data Collection and Management

microform (reels) or microfiche (slides), along with historic newspapers from the United States and elsewhere. Depending on your research question and your foreign language skills, these newspaper collections can be incredibly useful. Full-text electronic databases are also increasingly available. Some of the resources in the OCLC ArchiveGrid are full-text, and a number of other sources exist, particularly for government material. In the United States, for example, the Department of State’s daily briefings are all searchable online, as are presidential speeches. Legislative texts are all available on THOMAS through the Library of Congress’s website. The United Nations and European Union have all of their treaties, resolutions, and legislation (for the EU) online. Vanderbilt University maintains a collection of transcripts of network nightly news broadcasts. Some new projects, just as the University of Virginia’s Social Networks and Archival Context project, leverage recent developments in computing to produce some really interesting results. For people and events covered in their database, a search produces not only that person’s or event’s documents, but also documents by others about that person or event. Links to these and other sources are on the ERW website. Finally, the US National Science Foundation has recently launched a Qualitative Data Archive at https://qdr.syr.edu/, which contains an eclectic but growing set of qualitative data collections.

Data Management Options A number of approaches to data management exist, and which one is right for any given project depends heavily on the scale of the project, the nature of your data sources, and your own personal proclivities as a researcher. The two main strategies for data management are electronic and hard-copy. For small projects, you may be able to handle sticky notes or highlighting in the original sources (or photocopies) and then flip through while writing. Frankly, by the time you reach a 20+ page empirical research paper of the type this book addresses, you’re beyond the stage where this is an effective data management technique. If you have that few sources and that much time, then I suspect something is not quite right with your empirical strategy. Beyond about three or four sources, this strategy becomes highly time-consuming and inefficient. It increases the risk of inadvertent plagiarism, too, by tempting you with the original source’s exact wording. I generally try to stay out of my sources when writing for just this reason. A second hard-copy strategy is to take notes directly into your research notebook or other full-size sheets of paper. In my experience, this is not a good use of your research notebook. The research notebook is typically process oriented rather than data or content oriented; in this way, its use differs in the social sciences from what one often observes in the natural sciences. I also don’t recommend working in full-size sheets of paper. First, all that space makes writing long quotations highly appealing, and that strategy carries high risks for inadvertent plagiarism. Second, I rarely need an entire page of notes

153

154

Empirical Research and Writing

from a single source in one spot in my papers. Normally, I’m triangulating by pulling together different sources and references. Having multiple pieces of data on one sheet of paper also complicates my outlining/note organization strategy, which involves stacking things in the order I want to write about them. So I don’t like using full-size paper for notes. If I’ve got a short paper to write with qualitative data, or a short case in a larger paper, I most likely turn to index cards. Their small size deters long quotations; paraphrasing now reduces the risk of inadvertent plagiarism later. Having one data point or piece of information on each card allows me to shuffle the cards individually into the order I want to write about them, regardless of their sources. I can also color-code the tops of the cards with markers or highlighters to facilitate organizing them later, such as by section of the case discussion. Electronic note-taking strategies run some of the same risks as full-page note taking. In particular, we type faster than we write, so the risk for long quotations increases exponentially. Long quotes discourage you from paraphrasing, summarizing, or otherwise processing the information into its core elements. You essentially end up copying long passages without actually extracting the data from the information, which results in you ultimately doing more work for less value, and that’s never a fun, productive, or time-effective paper writing strategy. If you adopt an electronic strategy for note taking, you have several software options. Standard word-processing software allows you to keep a searchable running document; I find that clumsy for keeping data with their references. I spend a lot of time scrolling and not a lot of time writing. Spreadsheet software allows searchable text that can be associated directly with its sources; use one line per data point, much as you would with index cards, and include columns for source code and page. My current favorite for electronic note taking is Microsoft Office’s OneNote program, which comes bundled with most student and enterprise editions of MS Office. OneNote is a flexible program that allows you to click anywhere to make a text box, drag and drop text boxes across user-defined pages and sections, easily insert screen shots and other objects, and track original sources when copying and pasting text. Its biggest downsides are inconsistent availability (not everyone has it) and a bit of a learning curve at the outset. You’ll want to plan half an hour or so to figure out the interface and the tools before you start using it intensively, but it’s time well spent for the flexibility of the program. Other researchers recommend Scrivener. It’s highly popular among fiction writers and others with a nonlinear writing process, with separate sections and formatting for notes and document sections. The biggest downsides are that the Windows version was only recently developed (and so still has some bugs), the free trial only lasts 30 days, and it lacks features that scholarly research often exploits like integration with citation software. Finally, some scholars use EverNote to manage their note taking and data collection. It too is a highly flexible digital notepad tool; it allows users to organize information

Chapter 6   Qualitative Data Collection and Management

into fully searchable notebooks. It works on both computers and tablets/ smartphones, and it will automatically synch its data across platforms. Best of all, the standard version is free.

Peer Pointer

“Know when to stop. I always get caught in this, although I’ve gotten better at it: If you’re working under a deadline, know when to wrap up the research and start writing the thing. . . . When you’ve covered all the major parts of your [argument] and you’re just digging through the details, you can probably afford to stop researching and start writing.” —Andy T., College of Wooster

Writing about Data Collection In most empirical papers, a discussion of data collection comprises part of the research design section. The length and style of this discussion varies depending on the data collection approach, with primary source document analysis usually having the briefest discussion and more intricate techniques such as text mining receiving a more detailed treatment. Forms of writing about data collection differ as widely as the data collection approaches themselves. Your best bet for guidance on writing about your data collection process is to find another article that uses your data collection strategy and model your discussion on it. Very briefly, users of primary sources will often include a sentence or two indicating the sets of sources they used: presidential memoranda from the Public Papers of the President and Foreign Relations of the United States, oral history interviews archived with the Library of Congress, etc. Authors whose data come from primarily secondary sources and a small number of primary sources will often omit a general statement of the source of their data and instead discuss the diversity and origins of their source material in the body of the text itself. They might include a remark in their methods section saying, “Evidence drawn from a range of primary and secondary sources strongly supports these hypotheses.” Research relying on interviews will often include a statement about the number of interviews, when and where they were conducted, and the types of individuals involved. It might read something like “This analysis draws on more than fifteen interviews with Congressional staff, current and former executive branch regulators, and interest group lobbyists conducted between June and August 2013 in Washington, DC, and elsewhere.” Users of text mining techniques usually have the highest bar for discussing data collection methodology. In most cases, the expected discussion is closer

155

156

Empirical Research and Writing

to that found in quantitative papers than in qualitative ones. It should normally include some discussion on dictionary development, document population identification (and sampling, if relevant), coding processes and inter-coder reliability, and the like. Depending on the research question and the type of data collected, many will include a table of descriptive statistics and/or a brief sample of the codebook or dictionary in a print or online appendix.

Summary Data are systematically gathered information based around the concepts expressed in your theory. Identifying observable indicators for concepts in your theory and hypotheses greatly assists in achieving valid, reliable measurement and a satisfactory test of your hypotheses. Qualitative data generally derive from two sources, texts and human subjects. Data collection from texts frequently involves data mining or archival research, though data from strictly secondary sources can also be viable. Data from human subjects may involve elite interviewing, mass surveys, or participant observation. Your reference librarians are best positioned to help you locate relevant resources in your library and elsewhere.

Key Terms •• •• •• •• •• ••

Triangulation Observation Variable Dataset Measurement Level of measurement: Interval-ratio, ordinal, nominal •• Validity •• Reliability •• Running record

•• Episodic record •• Sources: primary, secondary, tertiary •• Counterfactual •• Document analysis •• Finding aid •• Text mining •• Dictionary •• Elite interviewing •• Snowball sample

I

Quantitative Data Collection and Management

7

n this chapter, we transition from pre-research and research design tasks to preparing for analysis. The crucial intermediate step in that process is assembling the data to analyze. That’s not a small task, but it needn’t be an overwhelming one. As with everything else in this book, thinking before doing can save you a lot of time, headache, and hassle. A lot of data exist in ready-touse form, thanks to the efforts of other researchers and data collection organizations. Knowing what you need and knowing where to get it is half the battle. If you do need to collect a variable or two of data for yourself, we have a lot of strategies and tips for making that necessary task as painless as possible. The first two sections of this chapter consider how to identify the data you need, in terms of both what cases and what variables. Thinking twice and acting once is a much more effective strategy than thinking once (or not at all) and having to do and re-do and probably re-re-do your data collection. The third section tackles the thorny issue of measurement and examines issues of validity and reliability, with a focus on readily available data. We then turn to practical issues of data gathering and data management. The fourth section discusses sources for ready-to-use data, and the final section provides guidance for collecting and managing new data. As with the other chapters in Part II of this book, there’s an awful lot to say and not a whole lot of room to say it. A number of additional sections are available on the book’s web page, and I encourage you to consult them.

Identifying Data Needs: What Cases? Collecting and managing quantitative data can be a somewhat overwhelming task without proper preparation. The biggest step in that preparation is to identify your data needs—that is, to figure out precisely what you need to gather. Having a “shopping list” of sorts will simplify the process immensely. This data needs list has two components: a domain of cases and a list of variables. The domain of a study is the spatial and temporal scope. It identifies what cases— individuals, countries, states, leaders, Congress members, etc.—will be included in the study (the spatial scope), and over what time period (the temporal scope). Identifying the domain of your study is the focus of this section.



157

158

Empirical Research and Writing

Units of Analysis The first issue to tackle in crafting your quantitative dataset is to determine the unit of analysis for your study. In quantitative analysis, an observation is a single instance of whatever constitutes your dependent variable. The unit of analysis is the thing that constitutes a single observation. If an individual’s opinion on the death penalty constitutes one value of the dependent variable (DV), then individuals are the unit of analysis. So your DV holds the answer to the question of your unit of analysis. Begin by asking yourself what constitutes one observation in your study. In American politics, common units of analysis are bills, elections, states, and individuals (in surveys). In comparative politics, options are more diverse. For those studying politics within other countries, the usual set of within-country options such as elections, policies, and individuals are available. For those studying politics across a set of states, the options include country-level variables such as growth rate, election results, and poverty levels as well as withincountry variables. In international relations, common units of analysis include states, wars, dyads (pairs of states), and combinations of those with years (i.e., country-years, dyad-years).1 Preview 7.1 Dependent Variables and Units of Analysis The examples below give three research questions and explain how to identify the unit of observation in each. A. Research question: How do attitudes toward gender equality differ across racial groups in different countries?

DV: attitude toward gender equality IV: race IV: country Race and attitudes are characteristics of individuals; individuals live in countries. Therefore, our units of analysis for this research question would be individuals. We would want survey data of some sort to investigate this question. B. Research question: How do natural disasters affect the birth rate?

DV: birth rate IV: natural disaster

1Many

research questions and hypotheses can be investigated on several levels of analysis. More than one may be appropriate or possible for your particular project, depending on the specifics of your theory and hypotheses. As always, if you’re not sure what unit of analysis you should use, consult your instructor to discuss your options.

Chapter 7   Quantitative Data Collection and Management

Birth rates are characteristics of individual countries. They change from year to year, and the occurrence of a natural disaster also varies from year to year, so we would want to capture multiple years. Therefore, our unit of analysis here is a country-year: Each country is observed once in a particular year.2 C. Research question: Are members of the EU becoming more alike in their UN voting records over time?

DV: similarity of voting record (among EU member states) IV: time This is a rather tricky question. An observation here is not a single UN vote by a single EU member state. We are not interested in the behavior of individual members; we are interested in the behavior of the group. One observation here is a single resolution or other voted-on thing in the UN. We measure our DV by, for example, counting how many EU members voted with the EU majority (so if 18 of the 27 members voted against a particular resolution, the DV value is 18). We might also express this as a proportion of the EU members voting together—18/27 or 2/3 or 0.67. Since the number of EU members changes over time, the latter measure is probably preferable. Practice 7.1 What Is the Unit of Analysis? Apply the same kind of logic from Example 7.1 to the research questions below. Start by identifying the DV and independent variable (IV) first, just so you’ve got them straight, and then think about what the unit of the DV is. Briefly explain your logic in the space provided (a sentence or two is enough). A. How does mayoral ideology affect whether US cities adopt minimum wages above the national minimum? DV: IV: Unit of analysis:

2We

could, in theory, record a measure of birth rate in a smaller unit than the year—say, quarterly. This would probably be better than an annual number for the particular research question here as it would allow us to be more specific about the link (or rather, the gap) between certain events and changes in the birth rate. Unfortunately, the global agencies that collect these data do so only on an annual basis.

(Continued)

159

160

Empirical Research and Writing

(Continued) B. Do highly restrictive gun control laws increase the number of firearms trafficked into the state across state lines (Coates and PearsonMerkowitz 2014)? DV: IV: Unit of analysis: C. Does invoking anger, but not disgust or fear, activate racial prejudice in individuals’ political decision making (Banks 2014)? DV: IV: Unit of analysis: D. Are insurgent groups whose resource bases are primarily economic more violent towards civilians than those whose resource bases are primarily social (Weinstein 2007)? DV: IV: Unit of analysis: E. How does the presence of a far-right political party in a national election influence the vote share for other rightist political parties in that election? DV: IV: Unit of analysis: F. Does an imbalance of military capabilities between a pair of states increase the probability that they will go to war? DV: IV: Unit of analysis:

Chapter 7   Quantitative Data Collection and Management

Populations and Samples Most statistical analyses make an important distinction between the population, or full set of relevant cases, and the sample, or subset of relevant cases. In the social sciences, we are normally interested in learning about the population of cases, and we’d usually prefer to analyze the entire population of cases so we can generalize about the population directly.3 Because gathering data on entire populations is difficult and expensive, we often turn to samples to draw our conclusions. How we choose that sample becomes a crucial factor. Most of the forms of analysis you’ve learned, such as regression analysis and difference of means tests, are tools of inferential statistics. This branch of statistics studies how to generalize about populations on the basis of samples, and these tools include assumptions about the relationship between the population and the sample. The biggest of these assumptions is that cases enter the sample in some manner that allows each member of the population to have an equal chance of entering the sample—the “equal probability of selection method” (EPSEM). A number of selection strategies meet this criterion, including simple random sampling with replacement. Sampling is a science that verges on an art form. Survey sampling is the most developed form of this, with a wide range of complicated algorithms available to generate samples that are highly representative of the population, or that reflect it in other key ways, or that obtain random samples of subgroups (Hispanic voters in Florida, for example), or anything you might like. Sampling is only effective if you have a clearly defined population. If you do not know the population of interest, making inferences back to it is nearly impossible. Defining the population is a theoretical exercise, so again, theorizing is important. Alas, complete population data are rarely available in international relations and comparative politics, and it happens in nonsurvey-based studies of American politics too. Most of the time, we must analyze incomplete populations, which are not the same thing as random samples. In these cases, we need to be concerned about whether our missing data are systematically different from our available data. Situations where all potential cases do not have equal chances of entering the sample experience selection effects. These can emerge from several sources, two in the data-generation process and two in the datacollection process. The key thing is that the missing or unobserved cases are systematic; the probability of an event entering the sample is correlated with some variable of interest. First, natural processes may weed out some cases so that not all cases that could potentially be observed are actually observed. States don’t go to war over everything; a process of escalation filters issues about which states are less resolute out of the sample of actually observed wars. Second, not all cases may leave equal amounts or types of evidence in the 3For bivariate analysis of relatively small numbers of cases, such as χ2 tests and Spearman’s rho, we generally want the population. We don’t have tools for handling missing data in these contexts.

161

162

Empirical Research and Writing

historical or social record. Some lawmakers will propose a bill that they don’t think will pass, but others may not even submit a bill if they think it will die in committee. The proposed bills actually appear in the record, but the ones that were drafted and not proposed don’t. The remaining two problems may be in the way that events are recorded. Things may have occurred but no evidence of them is available to you. Western news coverage of events in Africa is significantly less thorough than coverage of events in Europe, East Asia, and even Latin America, so an event (say, a landslide or instance of gross police brutality) in Africa is much less likely to come to a researcher’s attention than an identical event in Latin America. Finally, data collection practices or other limitations may cause some cases to drop from a sample in a way that is systematic. The poorest states do not have much money available for data collection. In a study of public goods provision and citizen satisfaction with democracy, for example, data are more likely to be missing for states at the lower end of the income distribution than at the higher, and since income correlates directly with public goods provision, our sample will be skewed toward higher-income cases or possibly artificially truncated by data availability. Analyzing data that suffer from selection effects produces coefficient estimates that suffer from selection bias. Ordinary least squares (OLS) coefficients, for example, can be incorrect in their size or significance, or even their sign; in short, nothing about them is correct. So you absolutely must be aware of and think about the process by which data get into your sample. Unless you are doing research that is explicitly based on a sample, such as research using survey data from nationally representative samples, you should be planning to collect and analyze population data. If you have a nonrandom sample or incomplete population, consult your instructor about ways to work around or compensate for this problem. Econometric techniques such as selection models can help in some cases.

Identifying Data Needs: What Variables? The data we need to conduct an analysis fall roughly into three groups: indicator(s) of the dependent variable(s); indicator(s) of the independent variable(s); and control variables. The need for indicators of the independent and dependent variables is rather obvious; if they are not in our dataset, we cannot look for a relationship between them! We’ll focus first, then, on the idea of control variables, and then revisit the dependent and independent variables to discuss the role of multiple indicators and robustness checks.

Control Variables We know that the basic idea behind a hypothesis is to propose a relationship between an independent variable and a dependent variable. We know too, though, that the world is a lot more complex than that. Very few social phenomena result from one single cause. Implicitly, that simple bivariate (one IV,

Chapter 7   Quantitative Data Collection and Management

one DV) hypothesis4 assumes that all other relevant variables are held at some constant value—that all these other factors remain the same while we compare the effects of lower and higher values of our IV on the DV. If the values of more than one IV are changing at any given time, we can’t determine whether the effect we observe is from changes in the value of IV1 or changes in the value of IV2 . This is called the ceteris paribus assumption, from the Latin term for “all other things being equal.” This key part of inference making—the need to keep all of the other variables at constant values—has important implications for how we conduct statistical tests. Unless we hold all the other relevant variables constant in our studies, we will not be able to tell whether changes in our IV of interest are the cause of changes in the DV. To do this, we include control variables in our statistical models. Control variables (CVs) are these other factors (other variables) that we think influence the outcome of interest, aside from our IV of interest. By putting them in the model alongside our IV of interest, we ensure that the computer program tells us the effect of our IV net of the effects of the CVs (i.e., after the CVs’ effects have been accounted for statistically). We do this by simply entering the CVs as other “right-hand-side” (independent) variables in our study. In practical terms, the need to control for other plausible causes of our DV means that we need to identify these control variables and collect data for them just as we would for any other IV. The most common question students have about CVs, though, is how many is enough? How far do we have to go in identifying “plausible potential causes” of something? In the social world, so many outcomes are related that this risks becoming an endless list of variables; the resulting statistical models end up containing pretty much everything except the kitchen sink (and sometimes that, too!). Unfortunately, one of the other things we know about statistical models is that throwing in a bunch of tangentially or distantly relevant variables—things that are potentially related but only in a very tenuous manner—also causes problems. So we have to strike a balance between including “all plausibly relevant causes” and including too much stuff, if only because ultimately we have to stop collecting data and start running analyses at some point. The answer to “How many CVs is enough?” is, of course, “It depends.” For most research questions, we have developed enough cumulative knowledge that we can readily identify three or four potential causes that other researchers have used as IVs or CVs in the past. This is another reason to pay attention to and make notes about variables while you’re reading for the lit review—other authors have quite frequently done most of the work for you on this already. Most research programs have a fairly standard set of four to five controls whose effect is well understood and which we should expect to see in

4Multivariate hypotheses do exist; they generally fall into the category of conditional hypotheses that we discussed in Chapter 2.

163

164

Empirical Research and Writing

any credible piece of research. Economists’ contributions to the study of economic outcomes produce a somewhat longer list of control variables for most political economy research.5 As a very rough rule of thumb, for most kinds of research questions, models with fewer than three to four right-hand-side (IV+CV) variables are somewhat suspect; with fewer variables than that, some critical causes are almost certainly missing, and then we run the risk of experiencing omitted variable bias in our findings.6 That said, models with more than eight to ten right-hand-side variables are also suspect. They run the risk of being “garbage can” models with so much junk in them that they’re really nothing but trash; they have too many moving parts to be confident about the results. In general, you should err on the side of caution when collecting data for CVs. Ignoring data that you’ve already got is a lot easier than having to go back and relocate and download and patch in data that you didn’t collect in your initial round. We have fairly effective statistical tests that can help us decide whether we should drop a variable that is not contributing much to our model (e.g., F tests, comparing Adjusted R2). Our methods to detect omitted variables are, unfortunately, nowhere near as good, since—as I’m sure you’ve guessed— what variables are “important” and should be included depends on your research question. Stats software doesn’t know or care what your research question is or what the variables stand for. All it sees is a bunch of numbers, and it will analyze any combination of variables you tell it to use, no matter how much sense they might (or might not) make in the theoretical story that lies behind the model. This is why theory is so important: It tells us what we should and should not be including in our models.

Preview 7.2 Identifying Control Variables A. Increased trade between a pair of states reduces the probability of conflict between them. DV:

Probability of conflict

IV:

Trade between states

Unit of analysis: By definition, trade occurs between pairs of states, so we are interested here in dyad-years.

5For

panel data, we know that both systematic cross-national forces—the things that become our IVs and CVs—cause the DV, but so do national (domestic-level) factors. To “control” for these idiosyncratic factors that vary from country to country, we often use a special set of control variables (called fixed effects) that effectively control for country. We revisit fixed effects in Chapter 8. 6Briefly,

omitted variable bias (OVB) is just what its name suggests. Because we have failed to control for important causes of our DV, some of the effects of those omitted variables get mistakenly attributed to other variables that we do include in the model. We revisit OVB in Chapter 8.

Chapter 7   Quantitative Data Collection and Management

Potential controls: Distance between the states (since we know that states tend to trade more—and fight more—with states in their geographic neighborhood), and whether the states had a conflict in the previous year (conflicts are likely to continue for more than one year; at a minimum, a conflict in a previous year signals that some tension or reason for hostilities exists between them). Practice 7.2 Identifying Control Variables For each of the hypotheses below, identify the level of analysis and the dependent and independent variables. Then, propose two control variables that we should include in a test of that hypothesis and give a brief idea of why and how you expect that CV to affect the DV. Pattern your responses on Preview 7.2. A. As the proportion of votes for the incumbent in a Congressional election increases, the number of challengers that incumbent faces in the next election should decrease. DV: IV: Unit of analysis: Potential control 1:

Potential control 2:

B. States that had support from international organizations (IOs) in preparing for their first post-transition election experience lower incidences of electoral irregularities (specific violations of rules designed to produce free and fair elections) than do states that did not have IO support. DV: IV:

(Continued)

165

166

Empirical Research and Writing

(Continued) Unit of analysis: Potential control 1:

Potential control 2:

C. People who have lost their jobs in the last year are less likely to support economic openness/regional trade integration than those who remain employed. DV: IV: Unit of analysis: Potential control 1:

Potential control 2:

D. Civil wars will last longer when insurgents commit atrocities (Kirschner forthcoming). DV: IV: Unit of analysis: Potential control 1:

Potential control 2:

Chapter 7   Quantitative Data Collection and Management

Multiple Indicators and Robustness Checks Under certain circumstances, you may want to collect multiple indicators for some concepts in your study. The primary reason for doing this is because you feel (or other scholars in the literature have concluded) that different indicators of these concepts capture different aspects of them and that no one measure captures all of the aspects. This multidimensional nature of most social science concepts means that only rarely do we have a single operational indicator that totally captures the concept. Take, for example, “development.” Most people have an instinct to operationalize development using the conventional “GDP per capita.” This is definitely one acceptable option, but it also uses a very narrow definition of development. In this example development is framed solely as “national economic development”; we omit here any questions of social development, political development, or improvement on other indicators besides income. The Human Development Index (HDI) attempts to rectify this somewhat by creating a measure that combines demographic and other characteristics to capture a population’s level of development in a way that does not rely on income. Gross domestic product (GDP) per capita provides an “average” income that, particularly in developing countries where high income inequality exists, is a very poor measure of the population’s standard of living. The two measures tend to track each other fairly well—countries that are high on one measure are generally high on the other measure—but some striking deviations exist, such as the oil-rich Middle Eastern and North African states, where populations are poor and standards of living are low, but where GDP per capita is quite high thanks to the oil revenues. A well-designed study would be aware of these types of discrepancies, and, rather than relying on just one indicator of a complex concept, it would check its results from the first model by substituting the alternate values instead. If the underlying theory is correct, the tests should support it no matter how we measure the concept (provided, that is, that the measurement is valid—that it actually captures the concept we are interested in, with few or no other concepts being captured by it as well). These “self-check” models using different indicators of key concepts are called robustness checks. They check to see if the findings are robust to different operationalizations of the variable. If the model collapses when we replace the POLITY institutional measure of democracy with the Freedom House rights-based measure of democracy, or when we replace GDP per capita with the HDI value, then clearly, the theory is not as strong as we might have thought from the initial results. We are typically most concerned about robustness checks on the measurements of the DV and IV. If reasonable, readily obtained data exist that would serve as a robustness check (alternative indicator) of one of your main concepts, you should seriously consider including it in your dataset and using it in that manner. Not only will it give you more confidence in your results from your main model, but it will also give you more to talk about in the paper itself.

167

168

Empirical Research and Writing

Measurement: Matching Concepts to Indicators We’ve briefly touched on the idea that we need to find observable indicators of our (usually unobservable) concepts. Our theories are about concepts; our hypotheses are about observable relationships between indicators of those concepts. This section introduces two major concepts in measurement: matching indicators with concepts—validity—and actually establishing values on those indicators for each of our cases—achieving reliability.

Operationalization: Validity The most fundamental issue in measurement is one of validity: Does our indicator actually capture the concept we’re interested in? If we haven’t actually captured the concept, then the mechanics of how we measured it are irrelevant. We may have a highly reliable, highly accurate, and highly precise indicator— but if it doesn’t reflect the concept we want, it’s not useful at all. Validity matters because if we don’t have measures that capture our concepts, we have no idea what our test is actually telling us. It might show a relationship between our indicators, but if those indicators don’t capture their underlying concepts well, then we can’t draw any conclusions about our theory—which is, after all, about concepts as represented by indicators. Operationalization is the process of identifying an observable indicator for your unobservable concept. For some concepts, operationalization is generally straightforward. We can usually measure sex or Validity Issues in gender by asking survey respondents whether they are male or female.7 Ethnic Politics For a slightly more complex example, scholars of international and comparative political economy are sometimes interested in the idea of economic interdependence, the idea that states’ national economies have become increasingly more closely linked. We usually operationalize this using an indicator of trade penetration, that is, what percentage of a state’s total economy is dependent on the international economy. To do this, we add the value of imports to the value of exports and divide by the country’s GDP. The resulting percentage tells us how much of the state’s annual income comes from other states (import and export partners), and so it is a valid measure of the extent to which one state’s economy is dependent on the economies of other states. For other concepts, political scientists have developed standard indicators, such as the partisanship measure introduced in Chapter 2. Not all “standard measures” have the kind of unquestioned, widespread acceptance of validity that the partisanship scale does. The most commonly used lists of wars and conflicts in international politics come from the Correlates of War (COW) project. Arguments about the validity of these lists, however, led to other 7Even

this is usually a bit more complex than these two choices. Offering “male” and “female” as the only two choices is still generally the practice in most survey organizations. Depending on the survey’s purpose and needs, however, you may see a choice of “no answer” and/or “other” to accommodate individuals who are transgender, whose gender expression and biological sex may differ, or who otherwise do not wish to disclose this information.

Chapter 7   Quantitative Data Collection and Management

scholars collecting competing lists of crises and conflicts. COW differentiates between wars and other forms of organized interstate violence. To qualify as a COW war, both parties must be recognized states in the international system, and conflict intensity must reach a level of 1,000 battle deaths a year, or at least 1,000 deaths total for conflicts lasting less than one year. The 1,000 deaths threshold is arbitrary. The COW researchers could have used any number, so long as they picked a single number, used it consistently, and had historically accurate data on deaths (that is, they constructed a reliable measure—more on that below). Debates over the threshold number of fatalities and over the list of “recognized states” were actually major motivations for some of the competing datasets such as the International Crisis Behavior (ICB) dataset and the Uppsala-PRIO conflict datasets.8 Practice 7.3 What Does This Indicator Capture? The examples below contain several proposed ways to measure concepts. Consider each carefully. What concept is the researcher trying to capture? Does this measurement capture it, and it alone—that is, is it valid? What else might our proposed measure capture besides the intended concept? Try to identify a revised research strategy that will capture just the desired concept and no more. Use other paper for your responses. A. Steve is interested in understanding how leaders’ assumptions about how the world works influence their foreign policy decision making. To identify leaders’ assumptions about how international relations works, he plans to read the foreign policy speeches of several world leaders (current and past) and count instances of phrases or ideas that reflect liberal and conservative worldviews. B. Sara wants to measure racial polarization in a large inner-city high school that is known to have ethnically based gangs and regular inschool and out-of-school violence. Her research budget does not extend to doing a student survey of the entire school (or even a substantially large sample). Instead, she plans to count instances of racially or ethnically charged graffiti in school bathrooms. C. Sue is researching the role of government structure (presidential and parliamentary) in determining public spending levels. Unaware of existing resources, Sue decides to code any system having a person with the title of “president” as a presidential system.

8This was a huge, and hugely expensive, undertaking; neither of these lists is yet as complete as the COW data. Depending on the researcher’s needs, though, using a more limited but more conceptually valid dataset may be preferable to using one that is more complete but not as valid for that researcher’s purposes.

169

170

Empirical Research and Writing

Matching Concepts and Indicators As the section above suggested, one of the most crucial steps in going from theory to hypothesis is the careful matching of concepts with indicators. One of the reasons that articulating your theory clearly is important is because without a clear theory—without some type of guide to what you think is happening and why—choosing an appropriate measurement of that concept is nearly impossible. This issue of validity permeates research of all types, both qualitative and quantitative, deductive and inductive, descriptive and inferential; it is worth some serious consideration even now in this very early stage of your research. For example, consider the question of partisanship in American politics and its relationship to social tolerance. What, exactly, do we mean by “partisanship”? It could mean party affiliation, or it could mean a more general sense of ideology. The American National Elections Study, like most surveys, measures ideology and party affiliation separately. Both are 7-point scales (replace conservative with Republican and liberal with Democrat) that we could use in our analyses. But which do we want? Someone who is a Democrat by affiliation might have an ideology value of “Strong liberal” through “Independent,” especially if she lives in a state with party-affiliated primaries (i.e., only those who are registered members of a party are allowed to vote in that party’s primary elections). Someone affiliated to the Green Party, on the other hand, would probably identify his ideology as “Strong liberal,” even though he does not associate with the Democratic Party per se. Likewise, Libertarian Party members probably identify their ideology as “Strong conservative.” Thinking carefully about which concept you mean—that is, having a theory that identifies a causal mechanism—allows you to select the correct indicator for that concept. By using standard measures, researchers can ensure that their findings are comparable with others’; they help ensure that we are comparing apples to apples and oranges to oranges across different studies. It also has the effect of saving a substantial amount of time and money. Reinventing the wheel is timeconsuming and expensive; so is duplicating datasets. Sometimes it’s necessary, as with the measures of what constitutes a war—the groups of scholars involved differed fundamentally on their understanding of the concept and so questioned the validity of others’ measurements. But frequently, scholars use and re-use others’ data, particularly in quantitative research. Whether we are using quantitative or qualitative forms of analysis, we must be careful to match our concepts with valid indicators. Using an invalid indicator would very likely produce inaccurate results, which would then lead us to draw incorrect conclusions about our theory.

Measurement: Reliability The concept of validity, as we just saw, refers to how closely the indicator captures the concept of interest and nothing more. Reliability, in contrast, refers to our measurement tool itself. A reliable measure is one that returns the same

Chapter 7   Quantitative Data Collection and Management

171

value for a given case even when multiple individuals evaluate the case according to our rules for converting information into data. One of the advantages that quantitative researchers have is that the reliability of most of their data is well documented. Most major data collection projects have sufficient funding to allow at least two different researchers to code each case, that is, to convert raw information into (qualitative or quantitative) data using a very specific set of rules and categories to establish variable values. These coding rules are established before data collection begins, and all coders are trained in their use so that coding is as reliable as humanly possible. Researchers will check the coding reliability of their assistants by computing inter-coder reliability ratings, which essentially measure how frequently coders disagree on a case’s value for a particular variable and how greatly they deviate. Depending on the degree of divergence, the primary investigator may convene a conference with both coders to agree on a code, or he or she may assign the case to a third coder. When data collection is complete, the coding rules are often made public, along with a bibliography of sources used to create the variable values. This way, other scholars can validate or replicate the coding. The scholarly community can publicly discuss the coding rules and variable values, and in fact, it does so rather frequently. In these ways, scholars work together to ensure the reliability of their data.

Talking Tip

Don’t worry about trying to vary the verb choice—coding is what you’re doing, so go ahead and say it. Try statements like “I code references to ______ or ______ as providing evidence in favor of a ___ viewpoint,” or “Actors expressing concern about ______ or frequently mentioning ____ are coded as ______.” I personally prefer the first form to the second. In my opinion, discussing coding is one of the few places where using the first person singular (“I”) is acceptable because it is explicitly about decisions that you as the researcher made; your instructor may differ.

Getting Ready-To-Use Data Ready-to-use data can come from a number of sources, including replication datasets, data archives, and publicly available data. Many nonprofit groups collect and publish data; the website contains an extensive though not exhaustive list of data sources for the kinds of data that students most often seek in political science. The data sources there include some of the most popular data used in the fields of American, comparative, and international politics, and are annotated with Data Sources

172

Empirical Research and Writing

descriptions and domains where possible. The rest of this section addresses other sources of data, such as replication datasets and data archives, and the norms surrounding the use of previously collected data.

Replication Datasets As Chapter 1 discussed, the sciences—social and natural—value replicability in research. Another scholar should be able to apply your techniques in exactly the same manner you did and obtain the same results that you did.9 As part of our collective effort to generate cumulative knowledge, scholars will often use each other’s datasets and arguments as building blocks. Someone who disagrees with methodological choices in a first published piece can obtain the replication dataset and try whatever she or he thinks is the correct form of analysis, or the right theory, or whatever.10 Because of this, a norm exists of making quantitative datasets freely available for others to analyze. In fact, an increasing number of journals require that their authors provide replication datasets and all the necessary code (statistical software instructions) to regenerate their results. Some journals centralize this in a data archive (see below); others allow authors to post the datasets on their own personal websites. One way to get a good head start on your dataset, then, is to use a replication dataset from some published piece. Even if the replication dataset is not a perfect match—it’s missing one or two variables that you want, for example— it’s still a lot more complete than if you had to collect and manage and merge all that data yourself! Adding a couple of variables is a lot less work than starting from scratch and reinventing the wheel. If you are using a replication dataset, you should always do several things. First, read the article that used this dataset. Be sure you understand the purpose for which these data were collected; data collected for some purposes may be coded in a manner that makes them significantly less useful for you. Second, always download the codebook when you download the data. A codebook explains the rules for establishing each variable’s value or score, the source of the data for that variable, and what all of the possible scores or values mean (i.e., what value indicates missing data). This is just as important to your research as the raw data itself! Without 9At

least one caveat to the norm of freely available data exists. Data that come from human subjects—survey respondents, experiment participants, etc.—are protected by federal law from dissemination except under very carefully controlled circumstances. Human subjects data have to be “cleaned” in such a way that respondents’ or participants’ confidentiality is fully protected, and this process is overseen in most schools by a Human Subjects Research board (sometimes called an Institutional Review Board—IRB). If the data involve questions about highly sensitive issues, or ones that might result in significant damage or harm to the participant/respondent, the IRB may prevent the data from being released at all without its explicit permission. 10A prominent example of this type of scholarly exchange in quantitative research are the articles of Simmons (2000), von Stein (2005), and Simmons and Hopkins (2005) on the politics of compliance with and commitment to the International Monetary Fund’s Article VIII; in qualitative research, see Christensen and Snyder (1990) and Morrow (1993) on alliance behavior.

Chapter 7   Quantitative Data Collection and Management

it, you will not have any idea what the variables mean. Was the FreedomHouse variable left in its original form (where 7 = least free), for example, or was it reversed to the more intuitive scale (where 7 = most free)? Take the time to review the codebook and familiarize yourself with the data and the way it was created. If the codebook is not clear, a polite email to the author is an appropriate way to request clarification. You can find most replication datasets by searching on Google or Google Scholar for the article’s citation and the term “dataset”; The Poli Sci Data site has a small but growing list of replication data by author. That said, not all authors post their data. If you find an article whose dataset sounds like it would be a good starting place for your research, you can ask the author(s) for their data. Because of the norm of replicability, most authors will provide the data. Some will not have a complete codebook available, especially if the dataset was not posted in a data archive.11 Most journals publish contact information for the author, or at least for a corresponding author, for each piece.12 You will usually find this contact information in the author biography statements on the first page of an article; a few journals include a separate page with contributor contact information at the end of each journal issue. Be aware that scholars are rather professionally mobile, especially in the top tier of academic institutions, and these institutions produce an overwhelming proportion of articles in the kinds of top journals that require replication datasets. Always Google the author or corresponding author first, or check the website of the school listed as the author’s affiliation, to be sure you’re sending your email to the right place. Use your school email account to contact authors. Authors are most likely to respond positively to polite requests that state your institutional affiliation, your status as a BA or MA or whatever student, and your interest in using the data for a class project this term. Noting that “it’s for a class project” is a subtle hint that the assignment has a deadline and that you would appreciate a prompt response, and it also helps to reassure authors that you are unlikely to try to mine their data and beat them to publishing results from it (known as “scooping”). Most academics are very flattered that someone else is reading their work and wants to build on it, so mentioning that you read the article is also a plus. You should generally avoid praising it much beyond a simple statement that it was particularly clear, that you liked the argument a lot, or something similar; statements beyond that are rather non-credible when they come from students. Despite our pleasure at being flattered, though, faculty are also incredibly busy with teaching and other obligations. Many times, we’ll need to do at least a little cleaning up on the dataset before we can send it, or we’ll need to find an hour or two to write up a codebook. Because of this, allow a week before re-contacting an author. Most of us are rather good 11Archives typically require that authors provide a codebook when they upload the data, and some will not even accept the data unless a codebook accompanies them. 12The corresponding author is the one the group of authors has designated as the central contact for inquiries related to the piece.

173

174

Empirical Research and Writing

about sending at least a quick response noting that it needs a bit of work and estimating when we can get the data to you. If you do not hear back from an author after a second email and a second week, assume that she or he will not be able to provide the data. In this as in many other things, you should avoid placing all of your eggs in one basket. Do not depend on someone else for your data. Have a contingency plan in place and work on building a dataset for yourself while you wait for a response.

Data Archives Prior to the emergence of the World Wide Web, most scholars deposited their data with a centralized data archive. The world’s largest and most comprehensive social science data archive is the Inter-University Consortium for Political and Social Research (ICPSR).13 ICPSR collects, stores, and disseminates datasets from large projects and individual papers alike. Datasets are usually searchable through a subscribing institution’s library catalogue; your school’s reference librarians can help you identify and access ICPSR data if you are unable to find appropriate data elsewhere or if you seek a historical dataset that is archived there. Prior editions of many common datasets are available through ICPSR, as are some major recurrent studies such as the General Social Survey (GSS) and American National Elections Studies (ANES or NES). With the emergence of easy-to-create personal websites, however, most authors now self-archive or archive with the journal that publishes their research. This has the effect of dispersing and fragmenting data across the Internet, but contemporary search engines can usually find at least the current version of most available data. Again, the Poli Sci Data website has a small but growing collection of links to these repositories. Finally, Paul Hensel of the University of North Texas has compiled an absolutely phenomenal collection of links to available IR-related datasets, organized by topic; he includes information on domain, variables, and data format. If you’re working in international relations (IR), his site is a great first stop to try to figure out what data might be out there on a topic you’re interested in. It primarily contains links to data collection projects.14 Most replication datasets aren’t listed, though he does link to some core publications or otherwise extensive replication datasets.

Citing Datasets Most of the major political science data collection projects have preferred means of citation. Usually, they ask you to cite a particular article in some journal where they announced the completion of the dataset, its contents and scope, etc. You should always check the dataset documentation (available on 13All of the websites referenced in this section are available on the Data page at http://study .sagepub.com/powner1e. 14I am unaware of any similar data source compilations for American or comparative politics beyond the (relatively limited, compared to Hensel’s) collection in the ERW Data page.

Chapter 7   Quantitative Data Collection and Management

175

the data project’s website) or the codebook for a preferred means of citation. Otherwise, if you are using data from a data collection project and it does not otherwise tell you how to cite it, you should include a citation to the data project home page in your paper (cite it to the dataset authors, usually identified in the codebook) and in the bibliography. If you are using replication data, you should cite the author’s published piece using that data—if you’re using the data from Smith and Jones (2001), then you should cite the Smith and Jones (2001) piece in the body and bibliography and mention in the text or a footnote that you obtained replication data from them via one of their websites (if so, provide the link and access date) or via personal correspondence (if so, provide the name of the individual who sent you the data and the date of the email). Personal correspondence and direct (website) access to replication datasets are not typically cited in the bibliography.

Collecting and Managing Your Own Data For many research questions, readily available data give a big head start on data collection. They can provide many common variables that often appear as controls in our studies. Depending on our research question, however, we often need to supplement the readily available data with a couple of variables collected especially for this research project: a new concept that scholars haven’t considered yet, a different operationalization of a concept, or the like.15 Collecting new data is often a time-consuming project, but it’s a necessary step both for any given research project and for the advancement of knowledge for the field as a whole. The secrets to (relatively) painless collection of novel data are much the same as the secrets for success at any other stage of paper writing: plan ahead, allow sufficient time, document everything, and back up frequently.16 First, when collecting novel data, having a well-defined data needs list is absolutely crucial. You should already know the spatial and temporal scope of the project by the time you start collecting novel data; you should also have verified that the data you want are not available in any publicly available dataset. Your next step, then, is to operationalize the concept(s) of interest. By articulating in advance what kind of information serves as evidence of your particular concept, you can focus your information acquisition process on the most likely sources. If you think ahead about how you’re going to code the variable for use in analysis, then you can hone your Managing Data in Excel data collection approach to be as efficient as possible. 15At

the professional and PhD levels, papers that contain no new data are rather rare. At the MA and BA levels, they are somewhat more common, but even there, collecting at least a small amount of novel data occurs more often than not. 16This discussion presumes that your novel data are coming from secondary sources (i.e., library research). In my experience, very few undergrads and even MA students conduct new surveys or lab experiments to gather novel quantitative data. The ERW website contains a brief discussion of procedures for collecting and generating quantitative data from these contexts.

176

Empirical Research and Writing

Second, allow sufficient time. Data collection—even if it’s primarily data collation from existing sources—takes time. In my experience, it takes about twice as long as I initially budget for it; several of my former students estimate five times as much time. Sources will be missing from the library, books will need to be recalled, the ILL system will go down; those types of things will aggravate the delays caused by mundane things like other assignments, library hours, and the like.17 That said, collecting information on what did happen is a relatively short process. The time-consuming part is collecting enough information to determine that nothing happened. When something exists or happens, finding records of it is a fairly straightforward task. To determine that nothing happened, however, or that the characteristic was absent, you have to do a lot of reading and hunting before you can conclude that no, it actually didn’t happen/wasn’t there. Knowing when you can safely cross the line from “none of the sources I consulted say anything about it” to “I’m confident that it didn’t happen/wasn’t there” is something you can only develop with time and practice. Third, document everything and keep duplicate copies. I cannot emphasize this enough. You need to know where you found each piece of information. That’s just standard citation stuff—but you need it for every single data point on every single variable. A column for “source” on your data collection form or spreadsheet can help you remember to get that information for every variable. In addition to that, though, you also need to know why you coded things in the way you did. Consistency in coding is absolutely crucial for establishing both measurement validity and measurement reliability, and the only way to do this is to keep track of what you are coding how. Many cases will be straightforward, but you’ll have quite a few odd or questionable ones, especially at the beginning. Thorough notes in your research notebook on coding, coding exceptions, and odd cases that require special handling are the only way to manage a data collection process if you want to have usable data at the end. Sloppy documentation makes an absolute mess. Poorly documented data violate scholarly norms about transparency in data and analysis, but they also simply put you in an unenviable position later. It might be faster today, but you’ll regret it when you have to go back and correct everything that you didn’t code properly the first time, and you find that you don’t have the right information and then have to go hunt it all down again.18 17Always remember to back up your work!!! Losing your data to file corruption or a computer crash is a horrible thing. You almost certainly have file storage space on the servers of your college or university that you can use. Many free commercial off-site file storage options also exist, including Dropbox, Google Drive, iCloud, and Microsoft SkyDrive. Some, such as Dropbox, allow you to save files on their remote servers, work on the file on your own computer, and have it automatically back up as you make changes. 18Noting what search engine or database you used is not typically enough documentation. Search engines and database collections change, so always note the search terms and any other specific search parameters you used in your research notebook. If possible, get the DOI (unique identifying number for electronic documents) from the source to help locate things later.

Chapter 7   Quantitative Data Collection and Management

177

Finally, back up your work. This includes your paper draft, but it also includes your working data spreadsheets, any notes files you have, and your dataset (and *.do file, if applicable). You may also want to “back up” your research notebook much as you back up your files; the best way to do that is simply to photocopy any pages that have been changed since your last copying spree. Insider Insight

“Keep photocopies/printouts of anything super-relevant or obscure that you might have trouble hunting down again later. I can’t tell you how many times I’ve gone back to those original sources to check on something that I never really thought I’d need to look at again but turns out my notes weren’t as clear or as detailed as I later needed. It’s much more efficient than having to figure out what book it was in and go hunt down and then skim the whole book again.” —Prof. S. Kirschner, Allegheny College

To summarize this discussion: Failing to plan is planning to fail. There is no other way to put it. This is especially true if you are collecting novel data, but it’s true even if you’re using ready data. Plan ahead for what data you need, allow a lot of time to get it, make sure you have all the information you need for citation and correct coding, and keep duplicate copies. You don’t have time to do it all twice, so think it through twice before you start and do it right the first time.

A Brief Discussion of Coding Coding is the process of converting raw information into analyzable data, whether qualitative or quantitative in form. The ERW website has a more detailed set of instructions for collecting novel data and developing a codebook, and if you are doing novel data collection, I strongly encourage you to read it. When collecting novel data, begin by recording the raw information and then apply your coding scheme right away. This will feel very awkward at first. You may need to Creating collect information for several cases before you begin trying to Codebooks code, so that you have comparison and reference values for ordinal to Collect Your or nominal variables. Don’t wait too long to start trying it out. You Own Data may find that you need to collect more information on certain aspects of the case to code it effectively, and you want to begin incorporating the search for that information into your data collection process as soon as possible.

178

Empirical Research and Writing

Most researchers find that they can combine the information acquisition phase and the coding phase into a single step by using a codebook, questionnaire, or other data collection form. By creating an explicit form or questionnaire to guide data collection, you can ensure that you are looking for all of the various information for each case (Did you answer all the questions?); that you’ve created consistent values across cases (Does each question have an answer from the list in the question?); that you have sources for each item you code (Did you write something in the “source” column?); and many other things. You can have separate questions prompting you to look for different facets of a concept, you can have a checklist of sources to consult for each case, or anything else that is helpful to you as you go through the process. This form can be updated and/or modified as you go through the data collection and coding process; just make sure you have a revision date or version number in the header or footer of the document for easy reference.19 In my experience, hard copy works well for coding the first few trial cases, so that you can make easy notes on changing layouts as well as on the data values and sources, but after that, electronic is the way to go. Most scholars I know use a template in Excel to manage the data collection process, with a new tab for each case. Use your research notebook as a place to note changes to questions on the form, new categories that you create, or things that are otherwise relevant to the overall coding process. Put any notes about why you coded specific cases as you did on the case coding form itself. Always record a source for each variable for each case. Whatever you choose to do, don’t just rely on scribbling all your notes in your research notebook in hopes of making sense of them later. That’s not a successful approach to data collection and coding. That’s a recipe for finding yourself with not all the right information to create your data and not enough time to hunt down the missing stuff. Generally, you should conduct novel data collection on a case-by-case basis rather than variable by variable. This allows you to identify and keep track of context-specific information that may matter for coding. Libraries are also usually organized this way, which helps in ensuring that you’ve consulted all the relevant sources. I usually keep a scratch list on the back or bottom of my data collection form of everything I looked at that I didn’t cite on the front so that I know I covered all Pretesting Your the sources. Data Collection Remember that each case can have one and only one value on a Procedure and particular variable.20 If you find yourself wanting to give cases more Coding Rules 19Many scholars use hard copy data collection sheets and take notes on them by hand. If you’re a handwritten-notes person, printing your data collection sheets (a few at a time, especially at the beginning, to allow for revisions) is definitely a worthwhile investment. If you’re an electronic-notes type of person, consider making a template in Word, OneNote, Access, or whatever program you’re using, to serve as your data collection form. 20When we analyze the data, we will treat nominal variables as a series of 0–1 “dummy” variables, with one dummy for each category (more on this in Chapter 8). Regardless of this, each category still represents a different value of a single variable (think partisan affiliation, regime type, etc.), and so a case can still only have one value on the variable.

Chapter 7   Quantitative Data Collection and Management

than one value on a particular variable, or if you find that none of your categories are fitting, you probably have an issue of concept-indicator mismatch. You may have conflated two different concepts into one question’s answers; if that’s the case, work to identify the two dimensions that you’ve merged and split them into two variables. Your instructor or teaching assistant can be a great help here. If none of your values seem to fit the actual cases, and no other pattern or grouping of observed values seems to be emerging, you should definitely talk to your instructor. You’ve likely got a definitional issue somewhere—your concept is probably either too narrowly or too broadly or vaguely defined. Either way, note this problem in your data collection form.

Summary This chapter introduced basic issues of data collection and data management for quantitative research. Use your theory to identify the unit of analysis for your study, and then determine the population and sample for your study. Be sure to capture appropriate variation in the DV and avoid selection bias. Issues of validity and reliability can potentially cause major problems with your analysis. Again, use your theory to carefully match indicators to concepts to minimize the risk of these problems. Think through the data collection process and plan ahead to maximize efficiency; gather all data for control variables and robustness checks in a single sweep, if possible. Consider using replication datasets published by other scholars as a base for your own research to avoid reinventing the wheel. Much data, particularly for standard indicators of common concepts, is freely available online through a variety of sources, and your institution’s library probably also subscribes to a number of other quantitative databases. Collecting new data is substantially more timeconsuming than using previously gathered data, but it is often necessary to test novel theories. Whether you use pregathered data or novel data, be sure to define your data needs list before beginning data collection, allow sufficient time, and document and back up everything.

Key Terms •• •• •• •• •• •• •• •• ••

Domain Observation Unit of analysis Population Sample Inferential statistics Selection effect Selection bias Ceteris paribus assumption

•• •• •• •• •• •• •• •• ••

Control variable (CV) Robustness checks Validity Operationalization Reliability Codes, coding Inter-coder reliability rating Codebook Data archive

179

I

Preparing Quantitative Data for Analysis

8

n this chapter, we move into the technical parts of paper writing: where you get your data into the statistics program and prepare to do something with them. This chapter includes a combination of very practical information about getting your data into the program and preparing them for analysis, and also pre-analysis steps you should take such as identifying and dealing with data irregularities and potential statistical problems. We begin with getting your data into the stats program, cleaning them so that they are ready for trustworthy analysis, and identifying and correcting any highly problematic aspects of the data. We then consider other ways to manipulate data, such as creating interaction terms, scales, and indices. Finally, we identify four common problems and solutions that arise when messy real-world data meet the too-clean world of theory: endogeneity, simultaneity bias, omitted variable bias, and fixed effects.

Transferring Data into Your Stats Program In practice, this should be one of your last steps of data preparation. In reality, you need to think ahead about it so that you have all the data in the right format and structure to merge them and transfer them with a minimum of fuss. This is another one of those places where a small amount of thinking and effort can save a lot of time in the later stages of the project. Of course, if you were lucky enough to find a replication dataset containing all your required variables that is already in the desired file format, then most of this section is irrelevant to you, and you should skip ahead. For simplicity, I discuss these procedures in the context of the statistical package Stata, which is popular among political science Great Statistics scholars and is increasingly taught in undergraduate methods courses. Resource Sites Other tools, including statistical software such as SPSS and MS Excel, and the computing and statistical language R, are also used (and taught) in political science. If you are using another platform for your analysis, you will need to consult materials provided by your instructor, the Help files of the software in question, and/or the Internet for assistance.



181

182

Empirical Research and Writing

Insider Insight

SPSS opens a wide range of data file types and will save them in a range of other formats, so you can use it as a transformation platform to open odd file types. It’s less effective with very old file formats, such as some archived at the Inter-University Consortium for Political and Social Research (ICPSR). A separate piece of software, Stat Transfer, does nothing but convert data files from one format to another. It’s pricey, though, so many schools don’t have it. If you are unable to open replication or similar files using SPSS or Excel, check with your school’s statistics department or math/stats lab to see if Stat Transfer is available to you.

Getting Started Ensure that all of your data are in a single Excel file, in the data structure you want them for your analysis (Country-year? Dyad-year? Election?). Do not plan to merge two partial datasets in Stata, or to manually input new data, or anything like that.1 If you need to generate new variables out of existing data, such as indices, dummy forms of some ordinal or nominal variable, interaction terms, or logged variables, you can do that in Stata more easily than in Excel, but everything else should be done. The import process is easier, too, if the final workbook contains only the worksheet you want. If you’ve got other sheets in that workbook, right-click on the worksheet tab at the bottom, be sure the “create a copy” box is not checked, and then select “new workbook” as the destination. This will move the selected sheet to the new workbook, where you can then save it and your dataset separately. The very top line—Row 1—of your worksheet should have the variable names (the short one-word names or abbreviations that appear in tables and in the variable list; see the discussion of variable names and labels below). Row 1 should not contain the variable labels, such as “Birth Rate per 1000 People.”2 Every column needs to have an entry in that top row, even the first one(s) where you’ve entered the country names or the years or whatever.3 The data 1Stata

does have the ability to merge datasets, but the process is cumbersome and requires certain kinds of preparation that are really only effective for power-users who have incredibly large data­ sets. In general, I avoid it. 2Most

statistical packages have rules concerning the naming of variables. In Stata and SPSS, variable names cannot start with a number or contain spaces. In Stata, variable names are casesensitive, so if you plan to type variable names into the command line, you may want to take that into consideration as you name your variables. 3If you are concerned about being able to reference certain observations later, you can create a variable with a unique observation number for each entry by giving Stata the command gen obsnum = _n. (Note the underscore before the n.) You can then tell Stata to browse varlist if obsnum==[#]. If no varlist is specified, Stata will display all variables for that observation.

Chapter 8  Preparing Quantitative Data for Analysis

should begin immediately in Row 2; do not put the variable labels there, or Stata will be very confused. If you have missing data, you may wish to use the “find and replace” command in Excel to replace all other missing data or notapplicable indicators with “.” or simply a blank (do not put a space in the replace with box; just leave it empty) to simplify things later. Finally, save the file. If you will be working with Stata 13 or later, you can save in the standard Excel format (*.xlsx); if your Stata version is earlier, you will need to save as file type *.csv (comma-separated values): In the Save As dialog box, click on the dropdown list under Save As File Type—immediately below the file name box—select *.csv, then name your file, and save it as usual. Once you have all of these preparatory steps completed, you are ready to import your data into Stata. The command is under the File menu. Follow the instructions and prompts. In Stata 13 and later, you will see Excel worksheet as an Import option; in earlier versions, you will need to select the *.csv file and indicate that the data are comma separated when asked. Don’t forget to save your newly created dataset as soon as you create it! Unlike Excel and Word, Stata does not auto-save or generate backup copies of the data.

Dealing with Missing Data Stata needs to know what values in your data represent actual data points and which represent missing data. It knows that “.” (a single period) is a missing data point; it also recognizes that an empty cell is missing data, and it will automatically insert a period into an empty cell. Some datasets, however, use other values to indicate missing data—particularly if data can be missing for multiple different reasons. For example, the commonly used POLITY dataset, an institutional measure of democracy, has state-year data for all states in the international system, dating back to 1815. It has four different missing values. -99 means that the state in question did not exist during the year specified, such as East Timor before 2005. -88 means that the state was in transition, usually meaning that it was engaged in the process of drafting and establishing a new political system. -77 denotes an interregnum, either a collapse of government during a civil war or foreign occupation with significant change between the pre- and postoccupation regimes. Finally, -66 indicates an interruption in a regime, such as foreign occupation, but the regime remains fundamentally the same before and after the occupation. Knowing which reason makes the data point missing is important for some uses of the data, such as studying regime transitions. For most purposes, however, the particular reason for missing data is irrelevant for the research question. Unless we inform Stata that these values actually indicate missing data, though, it will presume that -66, -77, -88, and -99 are just very large negative values in a dataset that otherwise ranges from -10 to 10. Your results would then be horribly off because the data are being skewed by a large cluster of negative values. You have two choices for handling missing data. First, you can simply recode the missing indicator in each variable into “.” by using the replace command. Second, if your dataset uses values other than “.” to designate missing data,

183

184

Empirical Research and Writing

and you need to preserve the information contained in those missing value codes (as in the POLITY example), you will need to inform Stata which of the values indicate missing data for each variable. To do this, go to “Data” in Stata’s GUI (the menus up top), and select Create or Change Variables > Other Variable Transformation Commands. Then, go to Other. Do this for each missing-data designator in that particular variable, and then press “OK.” You will then need to repeat this entire procedure for each variable that has non-missing indicators.4

Variables, Variable Names, and Variable Labels We’ve briefly introduced the idea that variables and their names and labels are different things. Variables have three possible references: the variable, the variable name, and the variable label. The variable is the name of the column of data in your dataset. This is what your statistics program knows as your variable, and your program may constrain the way you can name the variables (i.e., names can’t start with digits). Most programs restrict the length of variable names, so we often must abbreviate, perhaps with notes appended at the end like “01” for variables recoded as dichotomous indicators, or “ln” before the name to indicate that the variable has been log transformed. In larger studies, variables often become highly cryptic after multiple transformations and recodings, and an outsider (i.e., your readers) will not understand them. Always keep a log of what your variables mean and how you derived them, even if it’s just a running list in your research notebook saying things like “HiEmissions01 = 1 if Emissions > mean, = 0 if < mean.”5 Remember too that variable names are case-sensitive; Turnout, TurnOut, TURNOUT, and turnout are all different and cannot be interchanged in the command line. The variable name, on the other hand, is the form you reference in the text and discussion and list in your results table. This is the full name of the variable in a comprehensible form of English, written in title case and italic font within the paragraph.6 We normally introduce these names in the measurement section, where we describe how we operationalize concepts into indicators. Finally, the variable label is the complete descriptor of the variable that you (or really, your stats program) would use in creating graphs or tables. It is the most informative and longest of the names. Table 8.1 below shows two examples of variables, their names, and their labels.

Entering Variable Labels and Value Labels You can enter variable names and labels in most popular statistics packages. In the absence of labels, variables (what shows up in the variable list box) become the default title for axes, titles, and other graph components. This looks 4Variables that only use “.” or [ ] (empty cell) as the missing data indicator do not need this procedure. 5If

you are using a single .do file to manage all your data transformations, you can add remarks to the .do file by beginning the line with an asterisk. 6Some

specific outlets—journals or conferences or faculty members—may have different preferences; use this convention unless instructed otherwise. Title case in English involves capitalizing the first and last words, any nouns or action verbs, and prepositions of four letters or more.

Chapter 8  Preparing Quantitative Data for Analysis

185

TABLE 8.1  Examples of Variables, Names, and Labels Variable

Variable Name

Variable Label

BIRTHS

Birth Rate

Birth Rate per 1,000 Women

Lnfdiusd

FDI Inflows

Natural Log of FDI Inflows, US $

unprofessional; it will do for drafts, but final or circulated copies should always have correct labels. In Stata, the command is under Data > Labels > Label Variables.7 Your other option for graphs is to use the GUI to create the graphs; enter the desired axis titles on the appropriate tab. Generally, you will not want to put a title on the figure since it will require a caption. You can also enter value labels for ordinal and nominal variables. These then appear in tabulations instead of the variable value. This command, too, is under Stata’s Data menu: Data > Labels > Label Values > Define Value Labels. Most major data collection projects, Working With Survey and such as the World Values Survey and American National Elections Experimental Study, include both variable labels and value labels in their datasets. Data

Preparing to Analyze Your Data One of my stats professors once gave us some very valuable advice. “Buy two reams of paper for writing your dissertation,” he told us, “one for the analysis and chapter drafts, and another for all the plots you need to do before you analyze.” The secret to doing analysis right on the first try is to make sure that your data are all totally ready, meaning that you’ve checked for errors, located and addressed any irregularities in the data such as nonlinearities, and are totally familiar with the shape and content of your data. For our purposes on this paper, you can probably get by with doing the graphs and then saving them in a Word document for later examination; printing shouldn’t be as necessary. But the basic recommendation remains: Plot your data and examine them thoroughly before you do any analysis!

Checking for Nonlinearity If your dependent variable (DV) and independent variable (IV) are both interval-ratio variables, you essentially need two sets of plots to be sure everything is okay.8 First, you should do a histogram for each variable you will use 7Use 8If

the Variable View of SPSS’s data window.

your data are primarily ordinal, you can usually use these two sets of plots to check them, but you should probably ask your instructor if you need to do any other checks. If your data are primarily nominal, histograms and pie charts are the only graph types you can generate; crosstabulation, however, is often more helpful. A scatter plot relies on being able to identify “higher” and “lower” values of the variable to plot them on an axis, and nominal variable values do not have this characteristic.

186

Empirical Research and Writing

(IV, DV, and control variables), one at a time. In these histograms, you should make sure that no unusual values of the variable occur—states with birth rates of 0 per thousand people, states with birthrates of 51328 per thousand people, individuals with a gender of 7.3 (where 0 = male and 1= female), etc. Huge spikes of cases at a certain value—especially 0 or any values you have used as “missing” or “not applicable”—should also be suspect; if you get these, be sure that you have entered the missing data codes correctly (see previous section). You will also want to eyeball the histograms for significant issues of skew or bimodality, and make a note of these if they occur. Your second set of plots consists of scatter plots. These graphs are important no matter what, but they are especially crucial if you are using regression. Regression is the most common technique that students use for this type of paper, and its core underlying assumption is that the relationship between the DV and IV is linear. For this reason, you need to do a scatter plot of each IV that you expect to use in any model with the DV(s) you plan to use it with.9 The crucial thing to look at here is whether the relationship is at least plausibly linear, but you should also consider whether you have a theoretical reason to believe that the relationship is linear or nonlinear. For example, consider the relationship between annual income (DV) and age (IV). An individual’s annual income is typically flat (and very low) up through the age at which she gets her first job. After that, income rises substantially, with probably a big jump around the time she finishes college and gets a full-time job. As she reaches midlife and nears retirement, though, income will not change much until it plateaus at a stable annual level. After retirement, income will drop precipitously to a new lower level since she no longer earns a salary; it will not reach 0 until she dies, though, because she will still receive Social Security and/or investment income. We would not expect this annual income to have a linear relationship with age; if the scatter plot showed one, we should be suspicious. If we looked back at our histogram of income, we’d see additional hints that this variable is nonlinear—the histogram probably displays substantial left skew. As an example of a variable whose relationship with age (IV) is plausibly more-or-less linear, consider shoe size (or at least foot Nonlinear length, DV). Infants, babies, and toddlers have very small feet. Adults Relationships in have large feet, and feet don’t tend to become smaller even as adults Theory and Data age. We have no theoretical reason to believe that shoe size would be substantially nonlinear. It probably is not a perfect linear relationship; we know humans have growth spurts in their teenage years that cause shoe size to change rapidly in a short amount of time. Sometimes, too, some adults may lose a shoe size or two as they age and muscle mass and bones (for women) decrease. But at the population level in general, we have no reason to 9Remember that Stata takes the DV first; enter the command as scatter DV IV, and do not enter more than two variables. You may use an if function if necessary to condition the observations shown in the plot by some third variable.

Chapter 8  Preparing Quantitative Data for Analysis

believe that shoe size will take a sharp, consistent, and dramatic negative turn as people age; we expect the slope of the graph to be positive for all possible values of our IV, age. This is the key assumption we need to be able to make—that the sign of the slope remains the same across all possible values of the IV. If we have good reason to believe that the graph does change signs—goes from positive to negative (like age and income), or negative to positive,10 then regression is not a good tool unless we rescale or otherwise transform the variable so that the relationship between the transformed variable and the DV becomes linear.

Dealing with Nonlinearities The graphing that you did in the previous section was designed to determine if we’ve violated that key assumption of regression, that the relationship between each IV and the DV is plausibly linear. In part, this is a theoretical question; you should be sure to think about the expected relationship between the variables as you plot them. If you believe that the relationship should be nonlinear, then you should be prepared to take the steps below. If you don’t have a theoretical reason to expect nonlinearity, but you observe a fairly clear case of it in your plots, then you should also take the steps below. If you do not expect nonlinearity, but you observe an iffy case of it in the plots—it might show that, or it might not, it’s not clear—then you should consult with your instructor about whether transformation is necessary. The most common kind of transformation is the log transformation. The logarithm (log for short) relies on a mathematical constant, e, whose value is approximately 2.718. The log of a number, then, is the exponent to which we’d have to raise e to obtain the original number.11 So the log of 12 is 1.0792, because e1.0792 = 12. Two key things make a log helpful. First, the log of a number is always smaller than the original number. Second, the log of a big number is always much smaller (relative to that original value) than the log of a smaller number. Bigger original values get pulled down even more than their smaller counterparts do. This is why taking the log of a variable’s values will help to make its relationship more linear with the DV. Taking a log also helps with the analysis; not only are the high outlier values dramatically scaled back, but they become less influential in the analysis as well, and the histogram looks more like a normal distribution. 10Consider,

for example, age (IV) and the number of visits to the doctor a person makes each year (DV). Babies and small children see the doctor a lot. Healthy young adults and middle-aged individuals do not usually see a doctor more than once or twice a year. Older adults see doctors much more frequently as their bodies deteriorate and their risk of contracting many diseases or illnesses increases. The relationship between age and doctor visits has a “U” shape, with peaks on the extreme ends of the age range and a dip in the middle; somewhere around late middle age, the relationship changes from a negative relationship (as age increases, doctor’s visits decrease) to a positive relationship (as age increases, doctor’s visits increase). 11You

don’t need to understand the math mechanics behind this; it’s just FYI.

187

188

Empirical Research and Writing

The log transformation is appropriate when the plot of the original IV versus the DV shows a roughly exponential form. Figure 8.1 shows the scatter plot of an IV (against a DV) where the log transformation is appropriate. This exponential form curves steeply upward on the left, so that increasing the IV by one unit on the higher end of the scale involves a much larger change in the DV than a one-unit IV increase at the lower end of the scale. The opposite form of an exponential distribution also takes a log transformation. In these cases, a one-unit increase in the IV at the lower end of the scale involves a bigger change in the DV than a one-unit IV increase at higher values of the IV. To create a log-transformed variable in Stata, you should generate a new variable whose value is equal to the log of the old value. The syntax is gen newvarname=log(oldvarname); note that log( ) requires parentheses around the old variable name and only a single equals sign appears. Stata will take care of all of the math for you. Still, after you do this, you should generate a new scatter plot of the DV and the transformed IV just to check that it now has the desired shape. Figure 8.2 shows the log transformation of the IV against the original DV. The transformation is never a perfect fix—the data still do not form a perfect line—but the relationship is much closer to linear. If taking the log solved your nonlinearity problem, then you should use the transformed variable in all analyses. You should not touch the original value again; don’t drop it, though, in case you need it later. Figure 8.1  A  Nonlinear Relationship: Gross Domestic Product per Capita (GDPPC) and Birth Rate GDP per Capita vs. Birth Rate

60 50 40 30 20 10 0 –10

0

10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000

–20 GDPPC

Source: World Development Indicators, 2006 data.

Linear (GDPPC)

Chapter 8  Preparing Quantitative Data for Analysis

Figure 8.2  A Log-Transformed Relationship Log GDP per Capita vs. Birth Rate

60 50 40 30 20 10 0 0

1

2

Log GDPPC vs. Birth Rate

3

4

5

6

Linear (Log GDPPC vs. Birth Rate)

Source: World Development Indicators, 2006 data.

If a log transformation did not solve your nonlinearity problem, you should consult with your instructor to discuss other possible fixes. Log transformations work for many types of nonlinearities, but not all, and your instructor may know of a different, more appropriate strategy for your particular variable. The log transformation has two caveats. First, the log of 0 and negative numbers is undefined. You can only use the log transformation when the original values of the variable in question are positive. If you have a nonlinear variable with a substantial range of negative values, consult your instructor about appropriate options. Second, the log of values between 0 and 1 is negative. You may obtain negative values for the logged variable for completely legitimate reasons; the key message here is that you cannot take the log of negative numbers.12 In your pre-analysis plotting, you should have also created a histogram of the DV. The key thing to be alert for here is a long tail on the distribution, usually in the form of right skew (a few very high observation values, with most of the values much lower—think of putting Bill Gates and Warren Buffet on a histogram of net worth). If the nonlinear variable is the dependent variable, you should also work with your instructor to determine an appropriate correction. Many economic variables, such as gross domestic product (GDP) per capita and GDP, can simply take a log transformation of the dependent 12Most

decent software programs will report an error if you try to do this, but not all; Stata, for example, just returns blank spaces. This is why you check your histogram first to see what the lowest values are before you decide to log transform a variable.

189

190

Empirical Research and Writing

variable. In fact, these variables are rarely used in their raw forms as DVs or IVs; we almost always take the log because of the significant right skew in their distributions. This is generally true for almost all DVs measured in a unit of currency: FDI inflows, aid inflows, the value of trade, government budgets or spending, etc.13

“

Quips and Quotes

“The skew is where there are few.” —Prof. A. Conway, Princeton University

Detecting and Addressing Colinearity If we intend to use regression to analyze data, the final thing we need to check for in the preparation stage is colinearity among our IVs.14 To do this, create a correlation matrix for all of the independent variables and the DV. Include the transformed version of any transformed variable, not the original (untransformed) one. Use the pwcorr command so that you can obtain the number of observations; you do not need to obtain significance values for this. You should look for two particular things in this correlation matrix. First, examine all of the observation counts. Are any of them radically lower than the rest, and/or substantially lower than the total number of observations in your dataset? Consider especially the column or row showing each IV against the DV. Does the number of observations for which we have both IV and DV data drop precipitously for any particular IVs? Remember that any observation missing data on even a single variable will be dropped from the analysis. The maximum number of observations you can have in a particular model is the lowest number in the correlation matrix. If any of those cases in the lowestnumber-of-observations variable are missing data on other IVs, you’ll lose them too, and your N will be even smaller than that smallest-number-ofobservations. If one of your IVs is missing substantial amounts of data, you will lose a lot of observations in the analysis. Doing analysis on a subset of the data can be particularly problematic if the missing data is nonrandom (i.e., nonavailability of data is systematically related to some other variable or underlying 13For many (though not all) purposes, scholars will usually take the log of battle deaths, and some-

times war duration. The values for World War I and World War II are so much higher than all of the rest of the values that leaving the variable untransformed gives very misleading and incorrect results. 14If

you are not using regression, or you have primarily nominal variables, your needs here will probably differ. Please consult your instructor to ensure that you’ve done all the right steps.

Chapter 8  Preparing Quantitative Data for Analysis

factor, like poverty). If your sample is biased, your results will be biased. If you are in this situation, where one of your variables is costing you a substantial number of observations, you will need to think very carefully about how to address this. Do you really need to include that variable in your models? Are similar data available in another source that might have better coverage? Does another variable in your model capture the same concept, or can you identify another variable capturing the same concept that may have better coverage? The second thing to look for in your correlation matrix is excessive correlation between your IVs. This is called colinearity.15 The more closely one variable predicts the value of another, the closer they are to lying in the same line—that is to say, the more closely they are colinear. When our IVs are too closely correlated with one another, this violates a key assumption of the regression model. We can think of a pair of closely correlated variables as being somewhat like identical twins. Imagine that one of your identical twin 8-yearold neighbors ran through your yard and destroyed the flower bed. How can you tell which twin did it? The more those twins resemble one another, the more difficult discerning the culprit’s identity will be. Regression has the same problem. The more the variables look like one another—the more the deviations from the mean on one IV are matched by identical deviations from the mean on another IV—the harder a time the model will have in attributing variation in the DV to one or the other of the IVs. Ultimately, when two variables are perfectly correlated, the regression model will be entirely unable to distinguish one variable from the other. It will simply give up and drop one of them, and then proceed to analyze the rest. Cases of perfect colinearity are quite rare.16 The question then becomes, how much colinearity is too much? Remember that colinearity is a concern between independent variables only, not between IVs and the DV or between multiple DV indicators. No hard-and-fast rules apply, and different subfields have different standards. That said, a very rough but general rule of thumb is that you should begin to be concerned about the potential for colinearity when the bivariate correlations reach ±0.6. At this point you should deploy some appropriate tests for colinearity and its effects to determine if it’s a problem for your model.17 You should also consider whether the potentially 15This

is sometimes called multicolinearity. Most practitioners in political science treat the multi as redundant, though, since by definition, colinearity requires at least two variables (hence the co- prefix).

16Really, the only time this is likely for most of our data is if we accidentally enter the same variable

into the model twice. Stata will catch this and drop one of the duplicate entries. It can arise as well, though, if we enter two dummy variables that are, intentionally or not, perfectly correlated. If all the women in the sample agreed with a ballot proposition, and all the men did not, then the variables for gender and support for ballot proposition will be identical. The stats program will drop one of these, too; you’re left to figure out the problem. Once again, you can avoid this by checking your inter-IV correlations beforehand.

17Different types of models (regression, probit, etc.) require different tests, so I do not offer specifics

here. Your instructor is your best resource at this point.

191

192

Empirical Research and Writing

colinear variables are capturing the same concept. If they are, you should strongly consider dropping one or the other. In fact, I’d recommend running two more models, dropping each of the potentially colinear variables in turn, and examining the change in the goodness of fit statistics under each combination of variables.18 If the bivariate correlations reach ±0.75, you will almost certainly need to apply some fixes, up to and including dropping a problematic variable. You should take your correlation matrix, your data, and output from regressions that include all the problem variables and then all combinations of the problem variables to your instructor to discuss your options.19 Do not report results from models whose IVs have pairwise correlations greater than ±0.75 unless your instructor says you may do so. The single biggest cause of colinearity is an overloaded model. If you have six different measures of development, for example, you had better hope they’re correlated since they’re supposed to be measuring the same thing!! But you don’t need to have all of them in your model. You know that, for example, level of development is an important control variable for your theory. But you need one or at most two indicators of level of development, not six. Use the one indicator that best captures the idea: GDP per capita if you’re working with a strictly economic sense of development, and HDI if you’ve got a more social sense of it. Resist the temptation to run a “garbage can” model, where you throw in every variable that might matter along with multiple indicators for control variables and secondary concerns. The more variables you add, the greater your chances of colinearity problems.

Other Data Manipulations Sometimes, our research needs require us to transform data in other ways, beyond those required by our model type. We may need to convert ordinal variables into dummy variables. We might need to create groups out of continuous variables, or combine nominal or ordinal groups into larger groups. We might need to combine scores from several different survey questions or other variables into a single index. This section explores creating composite variables—those made by combining two or more variables—and other forms of transformation and recoding. 18For regression, the goodness of fit statistics are the R2 and Adjusted R2; in particular, pay attention

to the difference between these two values. Adding another variable always increases the R2, but if that variable is not too informative—if it’s too closely correlated with one of the others—then adding the variable will also decrease the Adjusted R2. Adding an informative variable increases R2; for the Adjusted R2, it typically does more for the explanatory power of the model (the R2) than it costs in degrees of freedom (the “adjusted” part of Adjusted R2). 19For

example, if you have three IVs and two (X2 and X3) are problematic, you’d do a model with both problem variables (Y = X1 + X2 + X3), then drop one problem variable (Y = X1 + X2), then drop the other problem variable (Y = X1 + X3), and finally drop both problem variables (Y = X1).

Chapter 8  Preparing Quantitative Data for Analysis

Generating New Composite Variables Most scholars prefer to generate composite variables—variables that are created by combining and/or manipulating other variables’ values—in their statistical software rather than in a spreadsheet program like Excel. Stata, like most software packages, can create a log of all the changes and commands you do, and it’s explicitly designed to treat each column of data separately. Three common types of composite variables appear in political science research: scales and indices, interaction terms, and lagged variables.

Making Scales and Indices: Cronbach’s Alpha Scales and indices (the plural of index) are common tools in data reduction, particularly in survey research. They allow us to take data from multiple questions or other measures and consolidate them into a single variable that captures more of the underlying concept than any one of the measures on its own can.20 The two most common ways of creating scales and indices are simple summation and averaging. In the summation approach, one simply adds all the response values to obtain a single summary value. The POLITY democracy measure works this way. It sums values on the executive recruitment, executive constraint, and political participation components to obtain a value that ranges from 0 (lowest value on all three components) to 10 (maximum value on all three components). The second approach to calculating a scale or index is to take the average of a set of components (i.e., sum the components, then divide by the number of components). This works best when the components are all scored on the same range, such as the 1–5 approach used with Likert scales. Items not scored on the same range should be rescaled to a common range, such as 0–1, to avoid giving any component undue weight in the composite measure. In all cases, be sure—especially with Likert-type responses—that you have coded all responses in the same direction before summing (i.e., that all the high values of the underlying concept are at the same end of the scale). This includes un-reversing any items that you reversed in the survey. The frequent use of this type of composite indicator has led to the development of a statistical tool for assessing how well the component variables track together—that is, whether they’re measuring the same underlying concept. This statistic is called Cronbach’s alpha (α), and all the major statistics packages compute it (Stata’s command is simply alpha [varlist]). Higher values of α indicate greater internal consistency, and the widely accepted cutoff for acceptable values of α is 0.7 (George & Mallery 2003, 231; Gliem & Gliem 2003, 87; Janda 2003; Cortina 1993). Be aware that a large number of items in the scale can artificially inflate α; this is more a problem for psychology than political science, but I’d be wary of more than 10 to 12 items. 20Researchers quibble over the difference between scales and indices (psychologists have one definition [Myers 2012]; sociologists have another [Babbie 2012]). For our purposes, they’re the same thing.

193

194

Empirical Research and Writing

Generating Interaction Terms Interaction terms are a special form of composite variable used to test conditional hypotheses, that is, hypotheses that the effect of one independent variable is contingent on the value of some other independent variable. We generate an interaction term by creating a new variable whose value is the product of the two separate independent variables. Interaction terms are actually pretty common in our theorizing, but any number of cases exist of scholars not including them in empirical tests. If you have a conditional hypothesis of the type we discussed in Chapter 2, you almost certainly need an interaction term in any quantitative analysis. You should not generate and insert interaction terms “just because,” however; you should have a clear theoretical argument (and related hypothesis) for why one is necessary. That said, they’re worth understanding, so even if this doesn’t particularly apply to your argument, read on through this section anyway. Let’s consider an example from a prominent branch of International Relations theorizing, that of power transition theory. Prior to the emergence of power transition theory (PTT), realists believed that war (particularly among great powers) was least likely when the two sides had relatively equal levels of capability—that is to say, when a balance of power existed between them. PTT, on the other hand, argues that the international system is dominated by one great power, a hegemon, which sets the norms and rules of the international system (or regional system). PTT says that (great power) war is most likely when a challenger state’s level of capabilities approaches, meets, and/or begins to exceed the hegemon’s capabilities and when that challenger state is dissatisfied with its position in the declining hegemon’s international order. To a realist, then, the best hypothesis to explain the probability of (great power) war is - Pr(War) = - Parity where Parity refers to the (absolute value of the) difference between the two states’ respective levels of capability; when the states are at parity with one another, the difference is 0, and as the difference in their relative capabilities grows, Parity’s value increases. Table 8.2 shows what a realist would TABLE 8.2  Realism’s Predictions

Parity

Pr(War)

Present

Absent

1 (lowest pr(war))

4 (highest pr(war))

Chapter 8  Preparing Quantitative Data for Analysis

predict about dyadic conflict, where 1 is the lowest predicted value and 4 is the highest predicted value. Power transition theorists, on the other hand, insist that a second variable, dissatisfaction (Grumpy), plays a key role. Their idea of the best hypothesis is - Pr(War) = + Parity + Grumpy + Parity*Grumpy Notice that in this case, the probability of war is determined by three separate components. Parity matters, in and of itself; the closer states are to equals, the more likely they are to fight (+ Parity)—neither can be sure beforehand who will win, so fighting seems like a reasonable gamble. Grumpyness also matters, in and of itself (+ Grumpy); a highly dissatisfied challenger state is more likely to fight anyway, on the rationale that perhaps a surprise attack, or a determined show of commitment and willingness to pay costs, will convince the other side to give it what it wants. What forms the real heart of power transition theory, though, is the last term, Parity*Grumpy. When a challenger state is both near Parity and also Grumpy, then this gives a special, additional boost to the probability of war. The state is able to mount a credible challenge—its capabilities are nearly equal to those of the dominant state—and it also has sufficient dissatisfaction to want to overthrow the leadership of the international system and establish a new system more favorable to it. This additional boost then means that we add an additional component to that predicted probability of war when Parity and Grumpy are both true. The total effect of each variable is contingent on the effect of the other. A power transition theorist, then, would predict the outcomes shown in Table 8.3.

TABLE 8.3  Power Transition Theory’s Predictions Parity Grumpy

Present

Absent

No

2

1 (lowest pr(war))

Yes

4 (highest pr(war))

3

The placement of 2 and 3 here reflects PTT’s additional argument that peaceful transitions are possible if the challenger comes from the satisfied coalition. War is less likely, then, when Grumpyness is 0, even when Parity exists, than it is if Parity is missing and Grumpyness exists.

195

196

Empirical Research and Writing

Preview 8.1 Identifying Interacting Variables

Hypothesis: The European Union uses its foreign policy mechanisms to respond to all sorts of global events. It is least likely to respond to a security crisis using its foreign policy mechanism if a historically neutral state holds its rotating presidency. DV:

EU response to an event using foreign policy tools (yes or no)

IV1: Event is a security crisis (not some other type of event like a humanitarian emergency) IV2: A historically neutral state holds the presidency (not an Atlanticist, Europeanist, or formerly Communist state) Complete hypothesis:

Pr(response) = + Security + Neutral – Security*Neutral

Explanation: We expect neutral presidencies to be more likely to respond in general; they tend to overcompensate for their nonparticipation in the security aspects of EU foreign policy by being very active in other aspects (No/ Yes = 4, see Table 8.4). As a result, the nonneutral states are often overactive in security matters (Yes/No = 3). But, we expect the combination of both neutral states and security issues to be less likely to obtain a response, so the sign on the interaction term is –. TABLE 8.4  P  redicted Effects for EU Leadership Hypothesis, Example 8.1 Neutral Pres Security

No

Yes

No

2

4 (highest pr(response))

Yes

3

1 (lowest pr(response))

Practice 8.1 Identifying Interacting Variables Identify the DV for each of the hypotheses listed below. Identify the unit of analysis and each of the two component variables, and then write the complete hypothesis. Be sure to show direction for each IV; use hypothesis notation throughout. Then make a table of predictions about the size of the effect. Indicate the lowest predicted DV value with 1 and the highest predicted DV value with 4. If possible, differentiate between the effects of the middle categories; if not, put 2.5 in each box. You might wish to jot yourself a note about why you ranked the cells the way you did, or why you predicted certain directions of effects, but this isn’t necessary.

Chapter 8  Preparing Quantitative Data for Analysis

A. White men are less likely to support affirmative action measures in public hiring than other social groups.

DV: IV1: IV2: Unit of analysis: Complete hypothesis:

B. Primary breadwinners who have lost their jobs are less likely to support regional integration.

DV: IV1: IV2: Unit of analysis: Complete hypothesis:

C. Leaders who are both responsible for getting their states involved in a war and who then lose the war are more likely to be punished than other types of leaders.

DV: IV1: IV2: Unit of analysis: Complete hypothesis:

D. States with high birth rates and low primary education completion rates are least likely to experience sustained economic growth.

DV: (Continued)

197

198

Empirical Research and Writing

(Continued)

IV1: IV2: Unit of analysis: Complete hypothesis:

E. Challenge: Conflict is most likely to occur when one member of a dyad has both the motive for conflict (a dispute of some sort) and the opportunity to act on that motivation (both geographic proximity and sufficient military forces). (HINT: How many variables are interacting here? How can you express that in tabular form?)

DV: IVs: Unit of analysis: Complete hypothesis:

Recoding Variables From time to time, you may need to revise how data values are represented in your dataset. For example, we might have a variable indicating the number of children a survey respondent has, but for our research interests—say, on funding for local schools—we only need to know whether the respondent has any children. We want to convert our variable from (discrete) interval-ratio measurement to nominal (yes/no categories) measurement, where 1 = has child/ children and 0 = no children. This process of revising the numerical values that represent different categories or values is called recoding. Recoding variables successfully requires two things: thinking it through and preserving the original data. First, think a recoding through entirely before you begin typing anything. Establish in advance which old values will map onto which new values, and be sure to account for all values of the old variable. You may even want to write out all the code first, before running anything, to ensure that you’ve got all your bases covered. This is a helpful place to use a .do file. These files are essentially strings of commands that you

Chapter 8  Preparing Quantitative Data for Analysis

want Stata to run. They allow you to write out all of your plans in advance, and then run them with a single command. You can also save .do files so you can see later what you ran, or use them to rerun your analysis in the future. Scholars who post replication datasets will often post a .do file as well that precisely replicates the analysis presented in the publication.21 Whether you use a .do file or not, planning ahead is still important. You will want to find out all of the values a particular variable can take before you begin, by running the tab or codebook commands. I usually make a work table for myself in my research notebook with old values in one column and new values in the other. This allows me to work out the rules for the recode command in advance. You always want to retain a copy of the instructions you used to recode or transform, in either your research notebook (via a table or something similar) or a log file, so that you or someone else can replicate or make sense of your data and analysis in the future. Think ahead about whether any of the new values you want to use are already used in the existing data for something else in a way that is problematic, and be sure that your process of recoding doesn’t accidentally change the wrong things. You need to plan as well for rules to handle missing and various sorts of not-applicable codes that are in the variable you’re working with. Replacing missing data or not-applicable codes with 0 is not usually appropriate since 0 is a meaningful value. In short, actually writing and executing the code to do your replacements should be the last (or next to last) step of this process, not the first. As with any other transformation or adjustment, you should run a quick histogram afterward to check for inaccuracies or oddities in the recoding, and/ or cross-tabulate the old and new variables. You’re probably sick of hearing me say this by now, but as with many other phases and stages of this project, an ounce of prevention is worth a pound or more of cure. Taking 2 minutes to plan and check now can save you hours in the end. Second, preserve your original data. Never, ever recode over your original data! Always generate a new variable that copies the old one (or begins the recoding process) and then recode in the copy. This way, if you make a mistake (probably because you find that you didn’t fully think it through; see the previous two paragraphs), you can still continue without losing any other work you did in that copy of the dataset. If you recode over the original data by accident, you should

21Other researchers like to use a .do file as a sort of replacement for a log. They putter around with figuring out what model specifications to run and what recoding they need to do in Stata’s main windows, and then when they are satisfied with a particular step, they transfer only those commands to the .do file. They don’t save the changes to the dataset or run a log at this stage. After they’ve gotten everything they need into a saved .do file, they reopen their data, start a log, and run the complete .do file on the clean data. This results in a log file that contains only the desired models instead of misspecifications and trial-and-error work as well. I do not generally recommend this approach. It’s too easy to forget to copy a particular command over, and I’m particularly paranoid about losing unsaved work. I usually run a log file during the initial putter session and copy the desired commands over into the do file. That way, I can regenerate the desired models again if I want or need them, but I’ve also still got a record of the other things that I tried and didn’t use. Your needs and preferences may differ. I will only note that you can always ignore a log file that you saved and don’t want or need to use, but you cannot go back and reference a file that you did not create.

199

200

Empirical Research and Writing

immediately exit the program without saving any changes to the dataset. This means that you’ll lose any other work you’ve done since your last save, but it’s usually the only way to ensure that you can recover your original data.22 If you make an error in your recoding process, and you’ve preserved the original data, you can simply drop the new-but-messed-up variable and start again. Preview 8.2 Recoding the Freedom House Rights Variables Freedom House produces good country-year data on levels of human rights and freedoms around the world. They have two main scales, one for political rights and one for civil rights. Each scale ranges from 1 to 7, with higher numbers, oddly, representing states that are less free. The highest, or best, score is a 1, which is rather counterintuitive and makes interpreting the results more difficult than it has to be. How would you flip this variable, so that higher scores indicate more free states? The simplest Stata command is recode, which recodes a series of old values to new values in a single step. Its general syntax is recode oldvar (old# = new#) (old# = new#), where old# is the previous value and new# is the value you want the variable to take. You can add as many (# = #) expressions as you need, or even recode two old values to the same new value with (old# old# = new#). Be aware that Stata’s default is to recode over existing data. I personally prefer to generate a copy of the variable first and recode on the copy to avoid all possibility of this, but if you are a lot less crazy than I am, you can also append , generate(newvar) to the end of your recode line. The command to recode the Freedom House data would thus be recode CivPolFrdm (1 = 7) (2 = 6) (3 = 5) (4 = 4) (5 = 3) (6 = 2) (7 = 1), generate(CivPolFlip) .

The Theory-Data Danger Zone: Endogeneity, Simultaneity, and Omitted Variable Bias As most of you have probably begun to realize by now, the reason “it depends” for most questions is because the appropriate methodological choices are always driven by your research question and your theory. This includes which variables you include and how you include them. Theory intersects with your data and variable choices in two other important ways that can be problematic 22Secondary lesson: Once you’ve satisfied yourself that a new or recoded variable is correctly trans-

formed, save your dataset. For that matter, just save frequently, period! Back up your data and analysis files regularly to an offsite storage site such as your university file storage space, Dropbox, Google Documents, or Microsoft SkyDrive, and/or email the files to yourself.

Chapter 8  Preparing Quantitative Data for Analysis

201

for analysis if you don’t think about them beforehand. Endogeneity refers to cases where your IVs cause one another—or worse, your DV causes one or more IVs. The latter is a special case of endogeneity, sometimes called simultaneity. Omitted variable bias occurs when IVs that are relevant to the theory are left out of your model; the result is that regression mistakenly attributes variation caused by the missing IVs to included IVs. The net result of all three types of problems is that your coefficients and their standard errors are not believable. I discuss each issue in turn.

Endogeneity and Simultaneity One of our more pernicious problems in the social sciences is that the social world is a complex place. Many variables cause or influence each other. We know that a certain level of GDP per capita is virtually necessary for a state to become and remain democratic (+Democracy  +GDPPC), but we also know that democracy as a political system promotes GDP growth, so that democracies are wealthier than other types of states (+GDPPC  +Democracy). Many other variables in political science exhibit this pattern, such as political activity and voting. Regression and many other models require a nice, neat, one-way-only causal story, so this kind of back-and-forth relationship between the variables gives us two big problems. The first is the one I described in the example, where the DV causes the IV. This is called simultaneity, and when it happens in a model, we describe the effects as simultaneity bias. Here, the X causes Y, but Y also causes X. The Y (DV) variable is thus represented simultaneously on both sides of the equation (model). Theoretically, this is problematic for the model because we know that the best predictor of a variable—say, the DV—is the DV itself. We have a number of ways to “fix” this empirically; none of them actually solve the theoretical problem of IV  DV, but they can at least alleviate the effects on the estimated coefficients and standard errors. The most common of these methods is to unhook the two variables in time by using a lagged variable, which we discussed above. This works theoretically because by observing the IV in a period (t—1) prior to the observation of the DV (in time t), we can ensure that the DV hasn’t caused the IV because the DV value hasn’t happened yet. This bit of theoretical sleight-of-hand solves the empirical problem of IV  DV in a way that we can live with. It’s not the best possible solution, but it is straightforward. Working With The second kind of problem that can emerge is endogeneity, or Lagged Variables when IVs cause other IVs. Consider the case of school completion rates. Education is a public good; its cost to an individual is more than the individual can typically pay, but the benefits accrue to all. Because of this, states usually collect taxes from all citizens and use the revenues to finance a public education system. In this case, we would reasonably expect that GDP per capita (GDPPC) predicts the primary school completion rate; wealthier states can afford to put more money toward education, and so they can afford to educate a larger share of eligible primary-school-aged children. We might also reasonably expect, though, that democracy predicts the primary school

202

Empirical Research and Writing

completion rate. Education is a public good; democratic leaders have more incentives to provide public goods than autocratic leaders because public goods distribute benefits to the entire population (and so increase the leader’s popularity at election time). Autocrats have very little reason to provide public goods; remaining in office requires the support of a small coterie of important supporters rather than the support of millions of average citizens. Here, we have a case where our conceptual relationship is +PrimarySchool  +GDPPC +Democracy. At first glance, this might look like a nice linear causal story, but we also know that +GDPPC  +Democracy and +Democracy  +GDPPC. This isn’t a pretty linear story; this is a big tangled causal mess. Regression can’t sort this out; it’s simply not powerful enough to detangle the effects of these variables on one another in their current forms.23 Our only way to solve this theoretical problem is to remove one of these variables, but if we simply drop one, we create an empirical problem because we then risk having omitted variable bias (see next section). Much as with the simultaneity “solution,” our empirical solution for endogeneity also involves a bit of sleight-of-hand. The theoretical problem, in a nutshell, is that democracy and GDPPC cause one another. So what we need to find is a sort of synonym or substitute term, a variable that predicts—but does not necessarily cause—one of the problem variables but not the other. We can use this instrumental variable as a tool (an instrument) to solve the problem. In our current example, scholars have figured out that a good instrument for GDPPC is, strangely enough, latitude. Distance to the equator and GDPPC have a positive relationship; states near the equator have some of the lowest GDPPCs in the world, and those farther from it have some of the highest. Latitude, however, is a poor predictor of democracy, so it works well for our purposes here. At a risk of being somewhat simplistic, what we do is use this instrument to predict values for the problem variable. We would estimate a regression for GDPPC  Latitude and use the constant and coefficients to calculate a predicted value and residual for each case in the dataset. We then replace GDPPC in the original PrimarySchooling model with the predicted values from the instrument model.24 We’ve removed the problem variable and replaced it with something that predicts the original values pretty well but that lacks the problematic theoretical relationship with the other variables. 23That

said, remember that all Stata sees is a bunch of columns of numbers. It does not know what they mean or what theoretical relationships exist between them. If you were to estimate this regression, Stata would happily give you coefficients and standard errors for all of the variables you entered. Unfortunately, those coefficients and standard errors would be totally unreliable. The data that went into them violate most of the assumptions of ordinary least squares regression. If you report these effects, you risk making a grave mistake in your estimates. You are the theorist—you must provide this knowledge and worry about the kinds of problems described in this section before you start to analyze the data! 24This approach of using instrumental variables also goes by the name “two-stage least squares,” or 2SLS, because of the two-step estimation process. See Kennedy (2008, chap. 9) for a concise introduction.

Chapter 8  Preparing Quantitative Data for Analysis

This process is actually more complicated than it sounds; identifying instruments can be devilishly difficult for most variables of interest. We would not expect that students in their first term of research design or statistics would do this, but you will find knowing the concept and the terminology useful in later coursework.

Omitted Variable Bias As the name suggests, omitted variable bias occurs when confounding variables—variables that are correlated with both the DV and IV of interest— are omitted from a regression. Leaving things out violates a key regression assumption that the DV is a linear function of the (included) IVs. When we omit confounding variables from a model, variation in the DV that should be attributed to the missing variables gets mistakenly attributed to included variables. The resulting coefficients are not trustworthy; in fact, they’re flatout wrong (biased). You could even get a totally spurious effect. For example, you might find that height is a strong and significant predictor of math skills. Is height really what’s acting here? Nope. The key causal variable, age (or years of schooling), is missing, and so the variation that should be attributed to age is instead being allocated to another variable that closely correlates with age, height. Once we add age to the model, the effect of height would disappear totally. What is perhaps worst about omitted variable bias is that we can’t even apply a mechanical correction to obtain the “true” values because the magnitude and direction of the bias in the coefficients is a function of the strength and direction of the correlation between the missing variable(s) and the included ones. This is why theorizing matters: It’s how you identify the potential confounds that need to be part of your model. Omitted variable bias can stem from omitted substantive variables, but it can also stem from failing to capture fixed effects (FE), or systematic variation across units in a study that is correlated with both the DV and the IV of interest. Panel studies are particularly prone to this problem. As a result, we normally include FE variables in a panel study to capture systematic variation across units (and/or time periods). The FE consist of a battery of dummy variables for each unit and/or time period, and we include the entire battery of FE variables in each model we run. What FE do, in practice, is make the coefficients tell about within-unit effects rather than across-unit effects. Imagine that I wanted to study population pattern shifts across US states over time. If I include FE for states, the resulting ordinary least squares (OLS) coefficients give the effect of the IVs across years. If I include FE for time (year), then the OLS coefficients give the effect of the IVs across states. One of the side effects of fixed effects is that panel studies require a large number of observations to overcome the degrees of freedom consumed by the FE variables. We can normally remedy this problem by including more cases or periods (i.e., increasing the temporal or spatial scope of the study). The FE dummies will also soak up a lot of the systematic variation in the DV and will

203

204

Empirical Research and Writing

make finding significant coefficients on substantive variables much more difficult. Using FE in a model also has implications for using lagged DVs and/or ordered models. The ERW website includes an additional discussion of omitted variable bias, including identifying it Identifying Omitted by using postestimation plots and ways in which it and FE can compliVariable Bias cate models. But for now, you need to be aware that these problems exist and consider carefully whether they’re at work in your models. Talking Tip

We do not normally discuss the coefficients of FE variables in a paper, or even list them in the results table. We simply remark in the table notes that FE were included for [whatever units or time periods], that they were omitted for space reasons, and that the full results are available from the author on request.

Nonlinear Models As we noted above, one of the key assumptions of OLS is that the relationship between the DV and IVs is linear—that it increases or decreases at a constant rate across the full range of IV values. This is not always the case, and it’s particularly untrue for situations in which our DV is a dichotomous (0–1) outcome. Things happen, or they don’t. With only two values of the DV, a linear relationship with any IV is simply not possible. For these types of situations, we use probit and logit models, which specifically address issues of dichotomous DVs by expressing effects in terms of the probability of observing a 1 or a 0 in the DV. We know that we cannot observe values of the DV that are lower than 0 or higher than 1, so the effect of the IV on the DV is expressed, effectively, as changing the probability of observing DV = 1. Since probability is also a value between 0 and 1, it works out very conveniently. Probit and logit models assume that the relationship between the IV and DV is not linear, that instead, it’s S-shaped. In general, the effect of IVs on the DV is assumed to be rather small at the extremes of the IV’s range, and much more pronounced in the middle. Tiny increases in the value of the IV above its lowest value shouldn’t greatly affect the DV value (no huge increase or decrease in the probability of observing the outcome). At some tipping point, though, the effect of increases in the IV begin to have a dramatic effect on the probability of observing the outcome, and the graph rises steeply. As the probability of observing the event nears 1, the effect of the IVs must by definition taper off since probabilities cannot exceed 1. Because this relationship is nonlinear, interpreting the coefficients is not as straightforward as it is for regression coefficients. Most statistical programs,

Chapter 8  Preparing Quantitative Data for Analysis

however, can estimate these effects with ease. If you’re willing to learn the extra step or two of stats stuff necessary to interpret the effects properly, probit and logit are incredibly powerful tools for working with 0–1 DVs. The Empirical Research and Writing website includes a brief discussion of probit and logit techniques that provides a quick overview of how these tools work so that you can determine if they’re something you need to consider. Your instructor can provide more guidance on using these if she thinks it’s appropriate for you; you might also consult Pollock (2012).

Probit and Logit Models

Summary Analyzing data requires doing more than simply running one model and reporting the results. Getting trustworthy results requires careful checking of the data, including checking for and addressing missing values, nonlinearities, and colinearity, and generating any necessary composite or recoded variables. These may include interaction terms, scales or indices, or even lagged variables. Finally, we must be aware of the problems that occur when messy real-world data meet assumption-laden models. Endogeneity, simultaneity, omitted variable bias, fixed effects, and dichotomous DVs all violate key assumptions of the OLS model and require appropriate statistical and/or theoretical adjustments to produce trustworthy results.

Key Terms •• •• •• •• •• •• •• •• ••

Variable Variable name Variable label Log transformation Colinearity (multicolinearity) Composite variables Index (scale) Cronbach’s alpha (α) Interaction terms

•• •• •• •• •• •• •• ••

Recoding Simultaneity (bias) Lagged variables Endogeneity Instrumental variable Omitted variable bias Confounding variable Fixed effects (FE)

205

9 T

Writing Up Your Research

his chapter’s focus is the writing of the paper itself. Many of the previous chapters have included sections on writing about their particular topics; this chapter fills in the gaps between those. Much of this chapter focuses on writing about your findings and on effective presentation of evidence, both qualitative and quantitative. But we also consider the crucially important but often overlooked introduction, conclusion, and abstract. The typical empirical research paper contains six sections, as Chapter 3 noted: introduction, literature review, theory, research design, analysis, and conclusion. These papers also commonly contain an abstract, which is a succinct, paragraph-length summary of the paper. The abstract is particularly useful as a framing device for writing the paper, so we begin there. The second section considers the “bookends” of the paper, the introduction and conclusion. In empirical research, these generally follow predictable (and similar) patterns, so we discuss them together. The majority of the chapter emphasizes presentation of analysis and evidence, with the third section considering quantitative conventions and the fourth section considering qualitative work.1

The Abstract I generally recommend that students start and end by writing an abstract for their paper. The abstract is a succinct overview the paper. The great thing about an abstract is that it requires you to conceptualize and contextualize your proposal in a way that then facilitates writing the paper, which is why I recommend starting the paper process by writing one. By the time you’ve finished writing the paper, you’ll often find that you need to rewrite the abstract to be consistent with your actual findings. In political science, the standard research paper has six sections. The standard abstract has six sentences. This is unlikely to be a coincidence. I usually 1The

chapters on literature reviews, theorizing, and research design contain extended discussions of writing these sections. In the interests of space, I refer you back to those chapters and do not repeat those suggestions here.



207

208

Empirical Research and Writing

frame my abstracts using the following outline, responding to one or more of the questions. 1. The research question: What puzzle, gap, or hole motivates this research? 2. The scholarly context: What have others said—or not said—that matters to me? How does my work fit into the existing literature? What’s the current explanation for my dependent variable? 3. The alternative argument: What is my claim? What do I think is wrong with or missing from the literature? 4. The testing method: How will I demonstrate support for my alternative claim? 5. Summary of (anticipated) results: What do I expect to find? Or, once the research is complete, what did I actually find? 6. Importance/relationship to existing knowledge: Why should anyone care about this question? What did I learn that we didn’t know before I did the research? A good abstract can provide a one-sentence answer for each of those six elements. The questions are suggestions, not requirements; you should shape your abstract to respond to your own paper’s motivation. Students often ask about how long an abstract should be and how much detail they should provide. In my opinion, successful abstracts are pitched at a relatively broad level but with enough detail that no sentence could be removed and placed in another abstract and still have it make sense. Abstract Writing In other words, it needs to be specific but not detailed. The Empirical Worksheet Research and Writing website contains an abstract-writing worksheet that provides these six prompts and space to write. If you cannot handwrite your response in the space provided, you’re giving too much information.

Talking Tip

Students often ask about when the use of the first person (“I”) is appropriate. The social sciences do not care about you and your beliefs and your arguments; they care only about the evidence. You are not part of the evidence; therefore, you do not belong in the paper except in very limited places where you might need to draw a contrast to an existing piece of work or to clarify decisions you as a researcher made in the research process. By implication, then, the first person will often appear briefly in the abstract.

Chapter 9   Writing Up Your Research

The Bookends The introduction and the conclusion of the paper form the bookends to your argument. They follow a very similar structure, at least in part because their underlying purpose is the same. They summarize the research puzzle, your claims, your investigation, and your results. They diverge only in their final paragraph. The end of the introduction previews the rest of the paper, especially its structure. The end of the conclusion usually looks beyond this particular paper to future research, policy implications, or similar issues outside of the scope of the current research. Most scholars prefer not to write the introduction until they have done and written the analysis sections so that they know what their findings are; this saves time in the long run.

The Introduction For the typical 20-page student research paper, most introductions generally follow a two-paragraph framework. The first paragraph highlights or presents the research question and its current answer and then previews your answer and findings. In short, it identifies the puzzle and gives some idea of how you will solve that puzzle, in both the mechanics (methods) and outcomes (findings). Again, as with the abstract, these are typically one sentence each, or perhaps two. Remember that you’ll get to expand on all of these matters in other parts of the paper. Your goal in the introduction is simply to give the reader enough information to convince them to read the rest of the paper. It’s not a persuasive essay per se like you might have written in high school, but it should clarify the empirical puzzle or anomaly that you’re investigating and give a teaser of the findings. The second paragraph provides a road map to the rest of the paper. This structure paragraph is crucial to helping your reader contextualize the material you’re going to provide. Think of it as the photo on the front of the jigsaw puzzle box: Without it, the reader is left to guess how the various components of the paper fit together. Model this paragraph after those in papers you read for your literature review. It should end up sounding something like this: “Section two begins by situating this new framework in the context of existing arguments about A and B. Section three then presents the new hypotheses and establishes the strategy for testing them. The penultimate section presents the results of two/four/several regressions/probits/whatever models/tests, which [strongly] suggest support for the new framework while rejecting the original claims about the effects of A and B. Finally, I conclude with policy implications and suggestions for future work.” Yes, it sounds kind of hokey, but that preview of what is coming helps your reader immensely. If you have a strong, focused empirical puzzle at the heart of your research, or a phenomenally clear illustrative puzzle case, you may find that a threeparagraph introduction provides a better fit and contextualization for your research. In this case, the first paragraph establishes the conflicting or puzzling empirical facts, articulates the contradiction or puzzle, and briefly describes

209

210

Empirical Research and Writing

how others have attempted to solve it in the past. The second paragraph, then, focuses primarily on the theory and findings of the current piece of work, and on any ways in which it significantly improves over existing work. The third paragraph is a road map, as in paragraph two above. For a 30- to 40-page-ish professional journal length paper, the introduction is typically five to six paragraphs. Typically, these correspond to the sections of the paper, with each section summarized in a paragraph. The last paragraph is always a road map, as above.

The Conclusion The conclusion of an empirical research paper generally has a two- or threeparagraph framework, for a 20-page student paper, or a similar five- to sixparagraph framework for a professional length paper. The first paragraph consists of a highly concise summary of the research question, scholarly context, hypotheses (about three sentences total), and the findings (about three more sentences). The second (optional) paragraph typically expands on the findings and their implications for other matters. Sometimes this involves policy implications of the findings. Other times, it will refer back to the initial case or puzzle that motivated the research (from the introduction) and show how the new explanation tested here improves upon previous explanations or explains previously inexplicable behavior. It might also consist of a very brief application of the theory to a separate case that is related to the motivating case. Do not force yourself to find something for here. If you have something that requires this paragraph, you will know; otherwise, omit it. The final paragraph discusses open research questions and directions for future research. What puzzles did your work raise—did you have any strange and/or inexplicable results? What weaknesses did your research have, and how might you fix those in future research? Does the literature contain any questions related to yours that do not yet have an answer? Are there any obvious ways in which others could build on your research to answer related research questions? Do your findings conflict with the literature, or do they further confirm it? What is the importance of your work—do your findings fill a gap in the existing literature, or do they otherwise link disparate and previously unlinked literatures? The point of this paragraph is not to cast doubt on your own work. This paragraph places your new contribution to the literature in its proper context and helps others who work in this research field to understand what you’ve done and where they—and you, and others—can build.

The Results: Conventions of Reporting and Discussing Quantitative Analysis Social scientists have developed a number of conventions for reporting quantitative results. These conventions ensure that readers have enough information about the methods and findings to evaluate the credibility of the results.

Chapter 9   Writing Up Your Research

Transparency is an important component of making research credible, and these norms and conventions have evolved to support that goal. We begin by discussing tables and figures, which are common in all forms of quantitative research, and then consider conventions of reporting regression results in detail. Regression and its variants are the most common forms of quantitative analysis used in student research, and the same principles apply to reporting results from probit, logit, and other types of multivariate models. This section concludes with brief discussions of reporting conventions for χ2 and difference of means tests.

Tables and Figures A table displays information using a format of rows and columns. A figure displays information in any other form; most frequently, this is a graph or diagram. Tables and figures are numbered consecutively through the paper, with a new series of numbers for each (i.e., Table 1, . . . , n ; Figure 1, . . . , n). Figures and tables are not normally designated with letters (Table A, etc.). Every table or figure needs a title that conveys information about its contents. You can add titles to tables or figures by using the “caption” feature in Microsoft Word. Highlight the entire figure or table, right-click on it, and select “Caption.” Word has a drop box where you can select whether the object is a table or a figure and designate if you want the caption above or below the object. The caption text box should already display “Table 1”; all you need to do is type in the rest of the title. Word will automatically use the correct number, counting from the front of the document and only counting those objects that it has captioned like this, and it will adjust any table/figure numbers as necessary if you insert one later. Titles on figures and tables normally use title case and are not ordinarily written in bold or italic font.2 This type of caption/title replaces the title that you might include in a figure. Examples are at the end of this chapter.

Conventions of Reporting Results The sections below present conventions of reporting results for three common types of quantitative tests: regressions, difference of means tests, and chisquared (χ2) tests. These are, however, only conventions. Your professor may have different or more specific preferences than the options discussed here. Always consult with your own professor in case of uncertainty. Two key items must be present in all results tables, no matter the test or estimator used: the number of cases in the analysis (N) and table notes. Table notes are at the bottom of a table, usually in a separate row and in smaller font, and provide three important pieces of information on what the table contains. First, most results tables use some scheme of asterisks to denote statistical significance; if you use such a scheme, your notes must include an explanation— for example, *p < 0.05, **p < 0.01, ***p < 0.001. Second, table notes also 2In APA style, title case capitalizes the first and last words, any nouns or verbs, and prepositions longer than four letters.

211

212

Empirical Research and Writing

indicate what the entries in the table cells are. For a regression, for example, that might mean “Cell entries are unstandardized coefficients with standard errors in parentheses”; a χ2 test might say, “Cell entries are observed counts with expected counts in parentheses.” Finally, table notes indicate the specific test (or model/estimator) and software used to produce the results. By convention, we use variable names—not their labels—in all results tables. We also normally round to two or three digits after the decimal point.3 This means, as a practical matter, that simply copying and pasting your statistical output into your paper will not produce appropriate tables, even for draft papers. Creating clean, readable tables is more time-consuming than you might expect, so be sure to allow plenty of time for this before your deadline! Finally, in discussing your results, be careful of how you discuss p values. P values do not indicate the relative “significance” of the variables themselves. In fact, p values are characteristics of test statistics such as χ2 and regression coefficients, not of the variable itself. We cannot infer anything about the substantive importance of the variables from the p values; p tells us nothing more than the degree of statistical certainty. Also, the precise p value does not add much information once we have deemed a coefficient “statistically significant” using whatever arbitrary alpha threshold we selected. So we do not generally speak of one coefficient or other test statistic as being more or less significant than another.

Regression Regression is the basic workhorse of quantitative analysis. It provides the scaffold from which many other quantitative inference tools hang. The discussion here is framed in terms of regression, but the general principles work for all multivariate models. As a quick note, we estimate coefficients; we do not run regressions or other models. The expression regression test is redundant. Results Table Presentation When presenting regression results, each variable is a row; columns then show different sets of coefficients obtained under various model specifications. Not all models may contain all variables; if a model excludes a particular variable, that variable’s cell is usually left empty. We cannot report 0.000 as the coefficient for an excluded variable; excluded variables do not have coefficients, and so the cell remains empty. The constant is the last row of values reported above the descriptive statistics (N and [Adjusted] R2); see below.4 3In

some cases, more decimal places may be necessary for small-magnitude coefficients; consult with your instructor if you think this may apply to you. 4Constants

are also estimated values and are subject to the same statistical uncertainty as coefficients. Always report significance for the constant in the same manner that you reported significance for coefficients. If your model does not have a constant—if, for example, you are reporting standardized coefficients or you have suppressed the constant for theoretical reasons—then you simply omit that row.

Chapter 9   Writing Up Your Research

213

If all of the models reported in a table use the same dependent variable (DV), we normally list that variable in the top left cell, with the independent variables (IVs) below it. In this case, the DV’s name usually appears in the table title as well. If the table contains multiple models, we number models from left to right in the table and reference those model numbers in the text (e.g., “Model 3 introduces controls for economic conditions.”) If the table includes multiple models with different DVs, we normally identify the DV at the head of the column, above or below the model number, to be as clear as possible for the reader. The third subsection discusses presentation of multiple models in more detail. Cell entries for a regression results table are the coefficients estimated by the model, usually rounded to two or three digits after the decimal. Coefficients must also have some indication of their statistical significance, usually in one of three common formats. First, many journals require that authors indicate statistical sigExamples of nificance of coefficients (or any statistics) using asterisks. Results Tables Conventionally, one asterisk indicates that the coefficient is significant at at least the p < 0.05 (α > 0.95) level, two indicate at least p < 0.01 (α > 0.99), and three indicate p < 0.001 (α > 0.999). If you are using the p < 0.1 (α > 0.9) confidence level, your system will differ. Asterisks are the minimum necessary to indicate significance. If you choose this option, your table notes must include an explanation of your asterisk scheme. Second, the maximal way to report significance is to include all available information—coefficient, standard error, and p value—to allow the reader to judge significance for himself or herself. Significance is, after all, a matter of degree. We normally do this by making a separate column for each component, so that each model is three columns of information. The cleanest way to do this, visually, is to use no interior lines in the body of the table except for vertical ones separating the models. We do not normally use an asterisk scheme if we are reporting all three pieces of information. If you choose this approach, you must still include table notes indicating your model and software, etc. The third approach to denote significance is an intermediate one. In this, cell entries are coefficients, and the cells also include either the p value or the standard error in parentheses, with the parenthetical component often on a second line. We do not normally report both in parentheses as this gets clumsy and awkward.5 If you choose this option, your table notes must include a statement informing the reader what is in the parentheses, standard error or p value.6 5If you want to report all three pieces of information, choose the maximal option and make separate columns. 6You may also choose to use asterisks with this approach; if so, your table notes must include an asterisk explanation.

214

Empirical Research and Writing

As the introduction to this section noted, results tables must always contain diagnostics and notes. For regression and other multivariate models, the results table must always include diagnostic information about the model itself: how many observations it included (the N of the model) and a measure of goodness of fit.7 Stata reports the number of cases in the upper right-hand corner of its output tables. To obtain the N in SPSS, in the regression window click on “Statistics” and then select “Descriptive Statistics.” Most software reports both R2 and Adjusted R2 for multivariate models; since we rarely do or report bivariate regressions, we are normally reporting Adjusted R2.8 N and R2 are always the last two rows of the table, just below the constant.

Results Discussion In a regression or any other multivariate model, significant coefficients are the evidence for or against your hypothesis. The coefficient on a variable is significant, not the variable or the regression itself. You should normally discuss the coefficient(s) associated with each hypothesis you presented in the paper. Most authors prefer that the order of discussion and the order in which the variables appear in the table mirror the order in which the hypotheses were initially presented. At a minimum, table order and discussion order should match. Discussion of significant coefficients—particularly those associated with hypotheses rather than those associated with control variables—usually dominates the analysis section. We cannot interpret the direction or magnitude of an insignificant coefficient (that is, after all, a large part of what statistical significance means), so we do not typically discuss their values in the text. That said, insignificant coefficients can be valuable and informative if their lack of significance is substantively interesting—if, for example, most of the literature finds a significant relationship but you do not, or if you had a hypothesis that a particular coefficient should be insignificant in the presence of a particular control variable or interaction term. If you do not find support for your key hypothesis(es), you should consult your instructor about how to present these results. Since statistical inference is all about drawing conclusions under conditions of uncertainty, we are usually fairly cautious in how we state results; we normally eschew sweeping claims in favor of more limited ones. 7R2

is only a valid goodness of fit (GOF) statistic for OLS (ordinary least squares) regression. Models other than OLS regression have their own GOF statistics. For most models estimated by maximum likelihood estimation (MLE) (e.g., probit/logit, ordered or polychotomous probit/logit, hazard models, etc.), you should report the log likelihood as the GOF statistic, and, if a model produces cutpoints, you should report those as well. We normally list these at the bottom of the table in place of a constant. For models producing predicted probabilities, many instructors and journals also prefer that you report the number or percentage of cases correctly predicted and may also ask for an eta (proportional reduction in error) statistic. 8Some authors get lazy or sloppy and label the table row as “R2” when it is really the Adjusted R2. Always assume that if the author has estimated a multivariate model, she or he is correctly reporting the Adjusted R2.

Chapter 9   Writing Up Your Research

Your readers can generally infer that any coefficient you discuss is significant, and so you do not need to repeat that term every time. A common way to include the significance of a coefficient in your discussion is to indicate the p value in parentheses: “The model indicates that a 1% increase in Female Literacy results in 3.4 fewer births per 1,000 women (Birth Rate, p < 0.05).” The p is italicized, and we normally include the leading 0 before the decimal point.

Working With Multiple Models A typical paper contains several different models, usually as variations on a theme. They may involve different sets of cases, if the relevant population is somewhat questionable or different datasets exist. They may involve alternate indicators of some concepts when the measurement of a particular idea is contested. Finally, they may include different sets of IVs for reasons of colinearity or to demonstrate that adding or deleting a variable affects other coefficients, as with interaction terms or claims about missing variables. We usually estimate multiple models for any hypothesis of interest. One common way of doing this uses a baseline model framework. The first model we estimate—the initial specification, or combination of variables included in a model—is all of the control variables against the DV. This represents our original base prediction about relationships and variation explained and the like. The second specification is then all of the controls, plus our key independent variable(s) of interest. This shows us the explicit additional effect that our newest variable had: Any increase in the Adjusted R2, or some other measure of goodness of fit, or of sign or significance on another coefficient, can only be an effect of the new variable. We can then estimate that variable’s additional effect on the model(s) by comparing predicted values. Sometimes authors will take the opposite approach, offering a streamlined baseline model with just the most crucial controls and the IV of interest, and then add less critical or more questionable control variables in further models to demonstrate that their inclusion or exclusion does not affect the results on the variable(s) of interest. Many authors also choose to conduct robustness checks on their results by substituting other measures of contested concepts or retesting their hypothesis on a different dataset involving a different group of cases. This is a particularly important strategy if your key IV involves a concept for which the literature has multiple widely accepted measures: partisanship and ideology in domestic politics, democracy in comparative and international politics, etc. These alternate specifications help to establish that your findings are not simply a function of indicator choice in the presence of multiple plausible indicators. If your underlying theory is actually supported by the data, then the results should persist no matter what indicators of the concept you use. (And if they don’t, talk to your instructor.) Marginal Effects The term marginal effects describes the change in the DV that occurs as a result of increasing one specific IV by the amount indicated. In a regression,

215

216

Empirical Research and Writing

we can obtain marginal effects from the coefficients themselves, since the interpretation of a regression coefficient is the effect of increasing that IV by one unit. In the case of transformed or composite variables, including but not limited to logged variables, interaction terms, and indices, interpretation is not quite so straightforward. Check with your instructor about how to discuss marginal effects of transformed or composite variables. In probit/logit models, or their cousins ordered probit/logit, we cannot directly infer the marginal effects from the coefficients themselves. Because the underlying mathematical model is nonlinear, the marginal effect of any one variable is a function of the values of all of the other IVs. For these non-OLS models, coefficients can tell us only the sign (direction) and significance of the effects, not size. Calculating marginal effects for probit, logit, and other MLE models is possible, but it does require a few extra steps of math or an appropriate function in your stats program. Many STATA users like the CLARIFY program, available from Gary King’s page at Harvard (http://gking.harvard .edu/clarify).

Difference of Means Test When using t-tests, the test statistic of interest is the t statistic. (That shouldn’t be a surprise.) As with coefficients, you should report the p value associated with that test statistic along with the number of cases used to calculate each mean. For dependent or paired t-tests, such as those used in beforeand-after studies, report the means of both sets of scores as well as the difference in the means and its associated confidence interval. If possible, compute Cohen’s d as a measure of effect size. For independent sample t-tests, again, report the means of each sample, the difference in means, and the confidence interval associated with that difference. Cohen’s d is also an appropriate measure of effect size. Generally, one should also use Levene’s test to determine whether the variance of the samples is homogeneous; pooling the standard deviations is only appropriate if the variance of the two samples is equal.9 If Levene’s test is significant, then the assumption of homogeneity of variance is violated.10 Welch’s procedure will compensate for this by adjusting the degrees of freedom and p value. Alternatively, you could opt for a nonparametric test instead. I strongly recommend graphical presentation of t-test results as well as textual and tabular presentation. A simple bar or column plot showing the 95% confidence interval is sufficient. Tables should always include the N for each sample. Always indicate in the table notes whether you used a paired or independent sample t-test. 9Violating

this assumption increases the chance of a Type I error.

10For some research questions, this finding itself could produce an interesting discussion as most of

our theories presuppose that variance will be homogeneous. Finding that it is not suggests, among other things, separate causal processes in the two samples.

Chapter 9   Writing Up Your Research

Chi-Squared Test When presenting the results of a χ2 test, you should always provide a table showing observed counts and expected counts as well as marginal totals. The formula for calculating expected counts is available online and in most statistics textbooks. Remember that the convention is to report the DV on the vertical axis of the table—the rows—and the IV on the horizontal axis—the columns. If you enter it correctly in Stata (tab DV IV), then it will come out in the same format you need to report it. The test statistic here is the χ2 value. Again, report the associated p value and indicate the degrees of freedom. Many editors also like to see the percentage of cases correctly predicted by the theory.

Discussing Qualitative Evidence and Claims Good qualitative research often combines several different data-gathering strategies with one or more design elements. One author might use process tracing in multiple cases selected under case comparison rules using archival and secondary sources; another might use interview and participant observation data in a structured focused comparison format—all to answer the same research question. This nearly infinite number of combinations, and the unique evidentiary demands of each, means that qualitative research generally does not have the same kinds of standard reporting formats and evaluative criteria that quantitative analysis has. I can, however, offer some guidelines for the presentation of quantitative data and analysis.

Focus on the Evidence The evidence you have to support your theory is the most crucial part of the paper. Remember that ultimately this is a paper about a theory and hypothesis, not about the case; the case simply serves as fodder for testing the hypothesis. If you’re having trouble isolating evidence from your data pile, or if you feel that your paper has stopped being about the theory and started being about the case, consider revisiting your definitions of terms and concepts. Review your research notebook from your work in Chapter 2 where you identified observable indicators for key variables. Don’t be afraid to open a new document and restart the section to reorient the discussion back to the theory. Pick the version of your text that does a better job of focusing on concepts, and shift the other to your Leftovers file. The form this takes differs widely across research designs. In process tracing, for example, support for the argument relies on establishing an unbroken chain of changes in or values of variables from the IV to the DV. Refer to your arrow diagram to ensure that you have evidence for all the links in your causal chain; consider putting the causal chain in the paper to help your reader establish that all the links are complete. Your argument is only as strong as the weakest link in your chain. Be explicit about temporality. Don’t be afraid to include dates or to use expressions like first, then, after, and finally. It may feel a bit repetitive, but establishing a solid chain of causation requires clear sequencing of data.

217

218

Empirical Research and Writing

Talking Tip

Establishing temporal progression without being repetitive requires varying your sentence structure. Don’t start every sentence with an adverb. Instead of saying “Next, Senator Smith proposed an amendment on X. Then, he moved for an immediate vote,” try “Senator Smith next proposed an amendment on X, and then moved for an immediate vote.” For structured focused comparison and comparative case methods, focus on variable values. This is where a summary data table, which you may have created in the process of collecting data (see Chapter 6), is very helpful in helping you to focus your discussion just on the variables and their values (and ensuring that you’ve discussed all of them). Think about framing your evidence; most people find presenting it thematically (by variable) within each case rather than chronologically to be more effective unless the research design has a process tracing component. Counterfactuals, falsifiers, and negative evidence are useful tools for buttressing your argument, particularly in circumstances where actual evidence is scant. Lack of supporting evidence for competing explanations can help your case by discrediting those explanations, even in the face of only weak support for your own argument. Perhaps you found evidence that explicitly rules out alternate explanations. Maybe you can identify specific observable things that would have occurred if alternative hypotheses were correct, or if your hypothesis were definitely incorrect, and you didn’t see any of those. Remember that your own hypothesis is competing against alternative explanations; even if you can’t build up a mountain of support for your own, undermining the others’ support is a valid strategy. Don’t worry about providing background information. Remember that your audience is reasonably well educated, even if they aren’t knowledgeable about your specific topic. If you have a particularly unusual or obscure case, a couple of sentences of context are appropriate, but they often appear in the research design section as part of the justification for case selection. If your reader feels he or she needs more background than that to evaluate the evidence, well, that’s what Wikipedia is for. If he or she needs more background to justify the case selection, on the other hand, then that probably means you need another sentence or so in the research design section. If you’re concerned about needing more background, you can explicitly ask about it in a classroom peer review context or in feedback from colleagues or other presubmission readers. Keep direct quotations to a minimum—two sentences, maximum, unless the text itself is serving as evidence. This is particularly true when using primary sources. Direct quotations, and especially extensive direct quotations, are necessary when the identity of the speaker is important, or when elements of

Chapter 9   Writing Up Your Research

tone or word choice are of direct relevance to the research hypothesis (such as in content analysis). Beyond those purposes, you will serve your research better by rephrasing evidence into your own words. Paraphrasing, especially when it invokes the concepts of your hypotheses, helps to make clear that the quotes are serving as evidence rather than as facts. If you must use quotations, remember that quotations don’t stand alone. Always be sure to interpret them for your reader so that they link clearly to an evidentiary need. Insider Insight

“Avoid qualifiers and unnecessary descriptive words. Perhaps, possibly, maybe, supposedly . . . but also: clearly, obviously, unfortunately, strangely, and so on. It makes things more interesting to read but often you undercut yourself, and you’re inserting yourself into the story. . . . In some cases, of course, qualifiers like ‘perhaps’ and so on are necessary because you really don’t know, but if you’re using them that often, then there’s something lacking in your data!” —Dr. P. Epstein, New York City Department of Cultural Affairs

Be Explicit About the Study’s Limitations Every study has limitations. The regularity of quantitative analysis and its standards of consistent reporting mean that limitations in quantitative analysis are often much more obvious to the reader. The large number of possible combinations of techniques and designs in qualitative analysis, however, and the lower transparency of data collection processes, mean that the limitations of qualitative studies are typically less clear. Roughly speaking, the limitations of qualitative analysis fall into two categories: limitations on available evidence and structural limitations in the study’s overall design. Evidence and data sources are a tricky set of concerns to navigate. Sometimes the ILL book doesn’t come in; sometimes the key volume of documents is missing; sometimes the material you need most is not available in a language you read. Qualitative researchers also face the problem of determining when they’ve done “enough” research and data gathering and can begin to draw conclusions from the collected data. This can be particularly problematic when working with negative evidence or when looking for evidence of “no” values of variables. Things that policy makers were concerned about, for example, appear in their diaries and memos, but at what point can you declare that the absence of references to an issue meant that they were not concerned about it? Questions like this are ultimately a judgment call by the researcher. You have to do enough research in the right sources to make these calls confidently, and knowing when you’ve hit that point only comes with practice (and/or deadlines).

219

220

Empirical Research and Writing

In general, your best line of defense here is to be honest and forthright about limitations in your data and sources. If no evidence for or against a proposition exists in the available data, say so; lack of evidence can itself be evidence in certain circumstances. “The historical/documentary/etc. record is silent on the matter of [whatever]” is an entirely appropriate statement, provided that you then qualify your evaluation of any related hypotheses based on this limitation. If you were limited to data sources in English and this eliminated certain important potential sources, say so; discuss the implications for your evidence and findings. If something could have affected your data and/or conclusions, your reader deserves to know that he or she can evaluate your claims with this in mind. A second set of limitations on qualitative analysis stems from the limited pool of cases available in the real world. Sometimes the best possible (hardest) test for a theory requires a case with a combination of variable values that simply does not exist. Strong party machines like those of the 1920s do not exist in the postwar United States, let alone in heavily Republican states. Other times, the best possible case may be poorly documented in easily accessible sources. The Roman Catholic Church is one of the earliest examples of a transnational and/or international organization, but its archives are seldom available to the public or even to researchers, who must be able to read Medieval Latin to work with the sources anyway. These types of structural limitations on study design are frustrating to novice and experienced researchers alike.11 They are an unfortunate fact of life in qualitative research, however, and so we learn to live with it. Again, the use of brief counterfactuals in your analysis discussion, or a short (one- or twoparagraph) analysis of an additional case with a different almost-best combination of variable values, can go a long way in building additional support for your argument even in the face of less than ideal case selection or study design.

Confine Your Conclusions Remember that you made your case selection and other research design decisions on the basis of a particular theory. As we discussed in Chapter 2, good theories are bounded; they have a scope and spatial and temporal domains that served as the foundation for your research design. Good qualitative research, then, confines its conclusions to the range of cases about which the theory makes predictions. If necessary, it further restricts the applicability of the conclusions to the set of cases bounded by the variable values observed in the cases used for testing, or at least to the range of values observed in the real world. At a minimum, you should be cognizant of and open about the limitations of your sample. Are the observed values of some variable artificially truncated as an artifact of case selection? Is some other variable poorly controlled? If you 11Depending on how far your actual study design deviates from the ideal study design, a brief discussion of that ideal form and how yours differs (and why) may be an appropriate part of the methods discussion.

Chapter 9   Writing Up Your Research

are drawing conclusions from different time periods or different units of analysis, be clear about why and how those comparisons make sense. You should be able to clearly articulate, in writing, why you can compare these cases and draw conclusions based on that comparison. Even if that text ends up in the Leftovers file instead of the paper itself, it’s a valuable step toward understanding the limitations you need to impose on generalizations from your small sample.

Summary This chapter presents some of the basic conventions of writing empirical papers in political science. Abstracts, introductions, and conclusions are formulaic and follow a predictable pattern; they are often among the last parts of a paper to be written. Conventions for reporting quantitative results include indicating significance, goodness of fit, and N in tables, discussing the significance of coefficients rather than of variables, and using baseline and multiple models to support your findings. Conventions for reporting qualitative research vary by research design, but they include careful obfuscation of sources for interview data, clear sequencing and temporality indicators in process tracing, minimizing direct quotations, and providing estimates of uncertainty for all conclusions drawn from qualitative data.

Key Terms •• Baseline model •• Specification

•• Robustness checks •• Marginal effect

221

10 S

Practicing Peer Review

tudents often struggle with paper writing in general, and with some of the specific writing tasks associated with this kind of assignment. Chapter 9 focused on the content portion of writing this paper; this chapter focuses instead on the process of writing. In particular, it addresses the postwriting processes often associated with major papers—self-editing and peer review—and the specific ways these occur in empirical social science writing. The paper isn’t done when you write the last word of the conclusion, but most students find themselves at a loss for how to proceed with revising or rewriting their papers. This chapter has three major sections. The first section gives some general tips for successful paper writing, drawn from my own experience and the experience of many friends and colleagues. Every person’s paper writing experiences and preferences are unique, but learning about strategies that successful academic writers use can help you expand your own repertoire of tools and options. In the second section, I discuss self-editing. Self-editing encompasses much more than looking for typos, yet most students don’t have tools to perform this crucial task effectively. The final section of the chapter explores the purpose and conduct of peer review in the social sciences and provides guidance for students on doing effective peer review of empirical research papers.

Writing without Whining Writing is not easy. Sometimes it is downright hard. But you can make it easier. The key to successful and stress-free (or at least lowest-possiblestress) writing is to be flexible. You need to be aware of your own writing process at both macro and micro levels, and you need to recognize that the paper-writing experience is not the same as the paper-reading experience. With self-awareness and a set of simple strategies, you can control your writing process rather than letting it control you. Writing a paper does not have to be agonizing and miserable; it can, in fact, be a rather pleasant experience if you are adequately prepared and equipped for it. Three simple strategies, distilled from conversations with dozens of scholars at various stages of their



223

224

Empirical Research and Writing

careers from undergraduate to senior faculty, can greatly facilitate the paperwriting process. First, audit your own writing process. After each writing stint, take 2 minutes to reflect on that session in your research notebook. How much were you struggling with the writing, and how much of it flowed easily? What sections were particularly easy or challenging to write? What environment were you in when you wrote, and did that that environment help or hinder your writing? How did you feel when you finished—exhausted, satisfied, frustrated? These are some ideas for things you might comment on in your notes. This may seem like an odd step for a project like this, but often the emotional experience of the writing is a clue to what is going on subconsciously in your writing process. If I Do Not Want To Write This Dang It—if I find myself cleaning or working out or doing any other things that I usually only do under duress just to avoid writing—that’s usually a sign that I’m not ready to write that section yet.1 I haven’t done all the thinking or research that I needed; I haven’t yet figured the section out subconsciously. If writing is like pulling teeth, if it’s miserable and frustrating, or if you’re actively avoiding writing by doing other normally distasteful things, you need to pull back and reflect. Once you recognize that you are struggling with a section or actively avoiding writing, you can take steps to identify the root problem and deal with it. Does all of this information go here? Does all of it go in the paper? Do I even have all of it, and if not, what’s missing? Does it go in this order? Rather than sitting there for hours staring at a screen and getting more frustrated with your inability to write, stop trying to write and start trying to figure out why the writing is not working. Analyze the data you jotted down in your postwriting audits; reconsider your outline; sketch a table to match claims with evidence or sources. Take an active approach and problem-solve. It will feel odd the first few times, but believe me, it works. Second, you do not need to write the paper the same way the reader experiences it.2 This has two elements: sequence and format. First, the whole paper needs to be written, but it does not need to be written from page 1 to page n, in order. In fact, that’s often one of the least effective ways to write, especially for a paper like this. This paper has distinct sections with clearly demarcated roles and contents. If one section isn’t writing well, and you’re having trouble figuring out the problem, simply abandon it for now and go write something else. Check the appropriate chapter of this book for guidance on what goes in another section, and work on that instead. This doesn’t absolve you of meeting 1My

home was never cleaner than during the writing of Chapters 2 and 4. Or maybe I should say, “during the ‘writing’ of Chapters 2 and 4.” 2Yes, this sounds like common sense. But I needed to be told this, several times, before I was willing

to try it, but once I did, it was a writing-life-changing thing. Common sense is just not that common, and some of us won’t believe it until we hear it from some “authority” or read it in a book. Well, consider both of those criteria met.

Chapter 10  Practicing Peer Review

deadlines for assignments, but it does give you permission to jump around within the paper or section or assignment and work on whatever is coming easily at that particular moment. Similarly, you don’t need to write it in the format the reader experiences it, either. The single most common thing that faculty and PhD students mentioned when asked about their writing strategies was to use pen and paper. Quite a few of them seemed embarrassed about it, as if there’s something wrong with writing things out by hand and then typing them in later.3 Writing by hand is slower than typing; it engages different muscles and is a much more physical action than typing. Changing the format in these manners can help your brain to process information differently and so can help you get around or break through a logjam. Many of the people I interviewed for this chapter also noted that they feel a lot more comfortable outlining, moving things around, scribbling in the margins, and performing other crucial prewriting tasks on paper (with a distinct preference among interviewees for plain paper at this stage) than they do on the computer screen.4 For writing text longhand (as opposed to prewriting), write on every other line of lined paper, use only one side of the page, and leave yourself wide (1–1.5 -in.) margins on both sides. This trio of strategies ensures that you have enough room to edit and/or add text as you continue to work on it. Third, when it’s writing time, it’s writing time. “I’m going to work on my paper” and “I’m going to write on my paper” are very different things. When it’s time to write, you should do nothing but write. Much like (American) football, writing is a momentum sport, and momentum can shift in an instant. The best way to get momentum is to train your brain that writing is an easy and ordinary task, and the only way to do that is to write regularly. Try, if possible, to spend at least 15 minutes a day writing on this paper, or to produce 200 words a day (roughly a paragraph). Don’t know where to start? Feel free to begin by stating the obvious just to get yourself typing and to get words on the page. I’ve been known to write things like “I need to write a paragraph today on the role of NATO members on EU foreign policy decision making, but I’m not sure where I want to start. NATO members comprise a majority of EU states, but decision making is entirely separate. . . .” Getting something on the page, getting my brain used to thinking through my fingers, and starting to talk through the issue that I’m blocked on can usually break the block and get me into the stuff I really need to write. Once I’m done working, or at least am writing smoothly, I can delete the filler bits that I used to get started and add an appropriate transition.

3I’m not sure I have the right to say this here, since I felt quite relieved to know that I wasn’t the only

one who wrote in hard copy on occasion.

4It may be a generational difference—most of us didn’t grow up with a computer in the house (or the classroom), and word processing software was much more primitive, even through the late 1990s—but I’d encourage you to try it.

225

226

Empirical Research and Writing

Peer Pointer

“I always put on the same music when I write—for me, it’s the Bach cello suites. That tells my brain that it’s time to write.” —Jessi G., Georgetown University Once you’ve got momentum going, capitalize on it. Exploit it shamelessly. If you’re writing well and you get stuck on something, but you know where you need to go after that, simply write [TRANSITION HERE] or [ONE MORE EXAMPLE] or whatever and move on. Can’t think of a word? Simply put [WORD] or _________ where it needs to be and move on. Don’t stop to work on tables or figures, either; plan to do those separately. Give yourself 2 minutes tops to track down a missing reference. Stick another [TABLE 1 HERE] or [SOURCE!!] note in the text and move on. Insider Insight

Use square brackets ([ and ]) or braces ({ and }) to enclose your notes to yourself. This way, you can use the find command (CTRL+F) during your final editing to ensure that you’ve identified and addressed (and removed!) all of those notes. Standard parentheses don’t work as well for this because they’re used in citation and for other purposes.

This is also not the time to edit. Keep in-line editing to a minimum; focus on getting words onto the page. You can tweak or rewrite later, once you’ve gotten the ideas out of your head and into a more easily manipulated format. If you write something you don’t like as much, or you find yourself out on a tangent or otherwise deviating from where you feel you need to be heading, simply skip a few lines and restart where you want to be. You can figure out what to excise to your Leftovers file and/or how to bridge the points later. Just keep going. Exploit your momentum ruthlessly.

Self-Editing Learning to self-edit effectively requires that you know (or learn) something about your own strengths and weaknesses as a writer. Knowing what your weaknesses and strengths are allows you to select self-editing strategies that focus on your weaknesses and play up to your strengths. As with so many things, the same strategies don’t work equally well for all people.

Chapter 10  Practicing Peer Review

227

Many styles of writing are “good.”5 Some are socially deemed “more good” than others for various reasons; what styles are seen as good varies across disciplines. The social sciences value clear, concise writing that generally avoids flowery language and contorted constructions. The result is often writing that has a dry and distant tone rather than a specific individual “voice.” In contrast to the humanities, we prefer explicitly and promptly articulated theses, arguments, and claims, rather than gradually developed ones that evolve and emerge over the course of the essay. In short, just because you are a good writer in English courses doesn’t mean you’re a good writer in social science courses. Be prepared to refine or adapt your style to conform to your instructor’s interpretation of disciplinary standards.6 Common types of writing weaknesses include the following: •• Global-level problems such as paper organization. This particular type of global problem is less common with empirical papers because the sections of the paper are so well defined. The more common variant at this point is an apparent “thesis shift” between intro and conclusion, so that the front and back ends of the paper don’t line up. You can alleviate or avoid this, at least for this type of paper assignment, by writing the intro last (after you’ve written the bulk of the paper and know your conclusions), and then tweaking the literature review as needed to keep everything in line. •• Paragraph-level problems such as paragraph length. You should expect at least one, and often two, paragraph breaks per page. Paragraphs focus on one main claim, with individual sentences providing discrete pieces of evidence for that claim. The infamous “concluding sentence” reiterating the main point, and the “transition sentence” to bridge to the next paragraph, are rarely effective or necessary—no matter what your high school English teacher told you. You still need to connect ideas, but you should be doing that through careful sequencing and structure of content rather than relying on obvious gimmicks and crutches. Paragraphs should build on one another to support a central claim for the section that you establish in the introduction or first paragraph of that section. •• Sentence-level problems. These include run-on sentences (sometimes in the form of comma splices) and repetitive sentence structures along with some more classically “grammatical” issues such as sentence fragments. Sentences should contain no more than one semicolon each, and semicolons themselves should preferably appear no more than one or two per paragraph. (I’m personally guilty of overuse of the semicolon.) The “Nerd Words”—things like hence, however, thus, and due to—are big culprits in this.

Detecting and Avoiding Sentence-Level Problems

5I personally define good writing as any style in which the words do not interfere with the transmission of ideas from the writer to the reader. 6This is particularly true if you have a double major in another non–social science field. Disciplinary conventions differ, and you may find that your two fields require vastly different styles of writing.

228

Empirical Research and Writing

•• Surface errors. Most of what students identify in peer review and self-editing falls into this category, which includes most basic grammar and mechanical issues as well as typographical errors. Unfortunately, surface errors are the most micro level of problems; they are annoyances, for the most part, rather than significant impediments to communication. These include the correct use of commas and other punctuation, words that sound the same but are spelled differently (their/there/they’re, your/you’re), and insidious issues like parallelism and agreement. The website has guidance on identifying and fixing common but annoying errors that can impede your reader’s understanding. Issues of word choice and vagueness fall between sentence and surface errors. Most are issues of usage rather than grammar or writing per se. Unfortunately, a book like this can do little to help with those directly because they are by definition context-dependent. If you regularly receive feedback from instructors noting that your sentences are vague or awkward, or that you have word choice issues, you should plan to consult with a peer tutor or staff tutor at your school’s writing center—preferably one who has subject matter expertise in political science or another social science. These individuals normally understand enough of the context of your assignment to help with these issues.7 Sentence-level and surface errors are the most frequent problems I find in student writing, but fortunately, they are among the easiest to fix if you’re willing to invest a little time in learning the rules. If you are a nonnative speaker of English, or your English coursework in high school (or before) did not include significant instruction in grammar or mechanics, you should consider borrowing a writer’s handbook from the library or acquiring an inexpensive used one. A writer’s handbook includes quick reference sections on punctuation, frequently misspelled and/or misused words, block quotations, and the like. Unlike many languages, English has specific rules for the use of commas (for example), and spending 10 minutes reviewing these will dramatically improve your ability to avoid mechanics errors and increase the clarity of your writing. If you don’t know what parallelism, comma splices, or agreement in number are, then you could benefit from having a writer’s handbook handy. Heck, I know what those are, but the handbook I was forced to buy in freshman comp still sits on my desk. It’s a little older and pretty worn around the edges, but then, so am I.

Finding and Fixing Surface Errors

Strategies of Self-Editing As the other chapters of this book made clear, the best research strategy to use is the one that is best able to address your research problem. Self-editing works the same way: The best strategy to use is one that best addresses your writing 7If

your biggest issue is word choice, disable the thesaurus function on your word processing program. Do not use it. Vocabulary fishing is obvious and generally ineffective. One of my colleagues refers to it as the “Vizzini problem”—“Inconceivable!” “You keep using that word. I do not think it means what you think it means.”

Chapter 10  Practicing Peer Review

problems. The strategies I discuss here emerged from discussions with half a dozen English, literature arts, and composition instructors. Pick and choose from them as necessary, but be aware that your needs may change from paper to paper and as you grow as a writer. 1. Change the font. Pick something wide and somewhat challenging to read like Lucinda or Bradley Hand; avoid clean sans-serif fonts like Arial or Calibri. Wide, awkward fonts will force your eyes and brain to slow down and actually look at each word to read it. I generally find this strategy most effective for self-editing typos, surface errors, and sentence-level mechanical problems, especially after I’ve been doing a lot of line editing. 2. Read aloud to yourself. If your primary problems are sentence-level and paragraph-level matters such as comma usage, sentence length, and paragraph organization, then you can probably benefit from reading your paper aloud to yourself. Force yourself to breathe only at the commas and periods. Finding yourself out of breath before you get to a comma? Consult your writer’s handbook (or a similar online source such as the Purdue University OWL), and add commas or start splitting your current sentences into multiple sentences. Notice the order of those instructions: Find out first where commas are allowed. Don’t just start adding them willy-nilly or, worse, for decorative effect. 3. Read aloud with a peer. This strategy comes in two variants: Reading your own paper to your partner, and hearing your paper read to you by your partner. It’s not a strategy I personally use, but English teachers at all levels swear by it. It is their most common recommendation for students who have issues with awkwardness, word choice, and usage. The ideal combination of who reads and who listens seems to vary depending on student learning characteristics and the exact types of problems the student has; the specific recommendations I got about this were less consistent. It seems to be a popular and effective strategy, but identifying the ideal form of implementation for any given student, assignment, or student pair seems to be a bit of an art. So you’ll have to try some different combinations to find out what works for you, and again, be aware that this may change. 4. Edit on hard copy. I personally prefer this in conjunction with a change in font to something a bit unusual but perhaps not as strange as if I’m editing on-screen. This is my preferred strategy when I’m having issues with paragraph-level concerns such as paragraph organization and paragraph length, though I often use it to check sentence length as well. My first task on this approach is to look at the paragraph lengths, usually as the pages are coming out of the printer. Does every page have at least one paragraph break on it? Then I highlight every sentence end-mark (periods, normally) in one color, and every semicolon in another. How far apart are the sentence end-marks (i.e., how long are the sentences)?

229

230

Empirical Research and Writing

Am I overusing semicolons? Do any sentences have an overabundance of commas? If my problems with a particular paper are at the paragraph level and/or mildly at the section organization level, I’ll highlight the topic sentence from each paragraph and then read all of those in order. If the strung-together topic sentences don’t make any sense, then what evidence is between them doesn’t matter. The section isn’t in the order it needs to be, and/or I’m missing some important claims or evidence. Insider Insight

“What I love to do when I self-edit is to ‘hide’ the text from myself for a day or two, take a break, and then come back and see if I am still happy with what I read. And usually, I am not. When writing, people tend to be blind to their own errors, so resting one’s eyes and mind, even for a few hours, helps.” —Lana Arndt, professional copyeditor If you know you have a weakness with particular global issues such as organization (at the whole-paper or major-section level), the time for you to self-edit for that is not now. Your primary editing stage is during prewriting and early drafting, though this last technique of highlighting topic sentences can help you ensure that you haven’t drifted too far during the writing process. Learning to self-edit effectively is a difficult but immensely valuable task. Be open to trying new strategies. Your writing problems or weaknesses may differ across types of assignments and even fields of study. For example, reading my paper aloud does little for me in English, but I’ll catch errors every time in French. My biggest weaknesses in French are sentence-level grammar errors, which stick out horribly when I try to force them out of my mouth, and my weaknesses in English tend to be paragraph or global level. Ask around in your class if you need to find a buddy to read with. (If you’re reading this because it was assigned for class, I doubt you’re the only one in the class needing to use that strategy.) Your school’s writing center is also a great resource for learning to self-edit. If the strategies here don’t seem to be helping you much, try visiting them. Walking through a self-edit or two with a trained peer tutor or staff tutor can help you learn more about your own self-editing needs as well as additional strategies for addressing those needs. Self-editing should occur before you submit your paper for peer review. Do not ever submit a paper for peer review without having given it at least a once-over after you finish working on it. In-line editing during the writing process cannot substitute for a coherent—even if cursory—review of a finished document.

Chapter 10  Practicing Peer Review

Editing for Style and Tone: Social Science Writing This book is not a good model for academic writing; do not use it in that manner. I deliberately alter tone and style as needed to make points, often leaning toward ridiculous when the subject matter is particularly dry (e.g., the chickenand-egg literature reviews). This was a conscious decision on my part to increase readability and read-desirability. The preferred tone for standard social science writing, alas, is much drier and more matter of fact. Don’t aim for cute or funny, even with titles or section headings. Puns and pop culture references are not generally necessary, but if an appropriate opportunity presents itself, you may cautiously take it.8 Many students find themselves in trouble on tone and style because they try too hard to “sound academic.” Don’t. If you are discussing research design and/or presenting evidence, your tone will almost certainly be sufficiently academic. Attempting to fancy up your writing by overusing Nerd Words, adding semicolons, or abusing other rhetorical devices is typically self-defeating. The resulting document has writing that interferes with conveying ideas from the writer to the reader, and that’s just bad writing.

Talking Tip

The Nerd Words are a pernicious group of words that writers often use to try to sound “more academic.” Their overuse unfortunately tends to lead to long, convoluted sentences that impede understanding more than facilitate it. The most common Nerd Words are however, thus, due to, as such, in/with regard(s) to, and in accordance with. Most Nerd Words are cheap transition devices; you can do better with verb choice and other tools. Try to avoid using Nerd Words if you possibly can. My general rule is that any sentence beginning with a Nerd Word is suspect, and any sentence with two or more should be scrapped and rewritten.

8Bad

puns in the political science literature include Krasner’s (1972) “Are Bureaucracies Important? Or, Allison Wonderland,” a rebuttal of Graham Allison’s (1969) classic paper on bureaucracies in the Cuban Missile Crisis, Elman and Elman’s (2002) “How not to be Lakatos [la-kah-tosh] Intolerant,” and Signorino and Ritter’s (1999) “Tau-b or Not Tau-b,” about the appropriateness of a certain statistic. Pop culture references include “With a Little Help from My Friends” (Murillo and Schrank 2005) and “The Empire Strikes Back: The Evolution of the Eastern Bloc from a Soviet Asset to a Soviet Liability” (Bunce 1985). But again, these titles came about because an appropriate bit of word play happened to exist, not because the authors were desperately trying to be funny.

231

232

Empirical Research and Writing

Besides the Nerd Words, two common problems appear in student writing. They tend to be bad for two reasons. First, they obfuscate the main point of the sentence, and second, they add words without adding substantive value. Most students have an implicit plan to write using as many words as possible to make the paper as long as possible. This is a faulty strategy. Filler obscures your key points, and it means you did more work than you had to. The more efficiently you write, the less work you do.9 I have also never seen a faculty member object because a paper was under the The Nerd Words stated minimum for an assignment. The three big targets for improving efficiency and reducing word count are the passive voice, verb + preposition pairs, and unnecessary prepositional phrases. First, the passive voice is loved by you. The passive voice is loved by undergraduates to an extent never before seen by me. Formally, the passive voice is when the actor in the sentence is the sentence’s grammatical object rather than its grammatical subject—the actor is not acting, it is being acted upon. The preposition “by” and verb forms that end in –ed are big red flags for passivity. The sentence “The passive voice is loved by you” contains both of those flags. Converting a passive sentence to the active voice—“You love the passive voice”—both reduces the word count and reorients the sentence to focus on its main actor or idea. A special class of passive-voice-esque expressions includes sentences starting with expressions like it is, there are, and it has been found that. In these sentences, the grammatical subject and the sentence’s topic do not match, and the topic itself is relegated to the later part of the sentence. The grammatical subject in two of these examples is it, and the sentences do not specify what it is.10 In the third case, the “there is/are” sentences, the grammatical subject is buried somewhere later in the sentence, well after the verb and usually a bunch of descriptors. Eliminating these expressions is quite easy. Instead of saying “It is ___ that X does Y,” just say “X ___ly does Y.” Again, using fewer words to make the same point is more efficient and more direct. For the “there is/are” sentences, you need to find the topic and make that the grammatical subject. Instead of “There are four reasons for X,” say “X has four causes,” or “Scholars have identified four reasons for X.”

“

Quips and Quotes

“‘To be’ as a verb interests only Hamlet.” —Mrs. Sarah Clark Hendess, Seminole County Virtual School

9This

is an especially helpful skill for essay exams, where your goal is to get as much information into as short a response as possible. 10Formally,

this is a case of unclear antecedence.

Chapter 10  Practicing Peer Review

Second, beware verb + preposition combinations. You use them all the time, especially in casual spoken English: look for, look at, bring down, write up, talk about, etc. In these cases, you are using two weak words to say what you could do in one better word. Look for becomes seek, search, or explore. Look at becomes study, examine, research, or investigate. Bring down becomes reduces or decreases. Swapping out—I mean, exchanging—stronger words for weaker ones will both reduce your word count and add to a more professional tone in your paper.

Practice 10.1 Broaden Your Range Identify appropriate synonyms for the common verb + preposition expressions below. A. to get rid of B. to write up C. to turn into D. to care about E. to be interested in

You can also eliminate many prepositional phrases by converting the object of the preposition into an adjective. For example, the previous sentence could have read, “by converting the preposition’s object into an adjective.” The same is true for many clauses, especially those beginning with that. I normally do a search for any instance of of and that to see if I can convert any into adjectives. Along with avoiding Nerd Words, you should also limit your use of colloquialisms and eschew rhetorical questions. Colloquialisms are expressions of casual or informal language that are inappropriate in formal writing. The most common example is a lot, which in formal language should be best or very or much or frequently or something like that. Other common examples include to have no idea, to have to, etc. Contractions (such as don’t) also belong in this category. Much like the verb + preposition pairs, eliminating these will usually reduce word count and sharpen your writing. Rhetorical questions, on the other hand, are questions that you ask your reader with no intent of them answering; they exist merely as a stylistic device or transition tool. “What, then, should we use to measure this concept?” is a good example. They really don’t do anything for your writing style, nor do they prompt reader response in the way that the author hopes. Just avoid them altogether, and you’ll be fine.

233

234

Empirical Research and Writing

Practice 10.2 Editing for Wordiness Rewrite each of the sentences below to reduce its word count. Consider the efficiency strategies discussed in this section. (All citations are fictitious.) A. There are a multitude of explanations as for why rain brings down the turnout of Democratic voters, as argued by Johnson (2003). B. If it is necessary to look for the causes that underlie this conflict, argued Shelton (1986), then we have to look back all the way to the end of the Roman Empire. C. The need for him to write a book to answer that question, however, is entirely due to his own predilection for verbosity; it is also born of the sincerity of his belief in his own brilliance.

A Note on How Much Background Is Enough In general, for an assignment like this paper, you can presume your reader has a general background in the social sciences and a (minimum) bachelor’s level education. Lay out the evidence, interpret it as necessary, and then move on. Your reader does not need an extensive backstory of the causes of World War I if your hypothesis tests occur in comparative case studies of war termination decisions among the Triple Alliance powers, or a history of cognitive dissonance theory in political psychology research. If you find yourself asking yourself whether you should include a discussion of something “as background,” the answer is almost always no. Background is presumed, not provided. Likewise, avoid words that don’t add value to your argument. Every word you write takes time and effort; write as few as possible to get the job done. Statements like “Israel is a country that was founded in 1948” are unnecessary; they contribute no information that is either (a) not common knowledge or (b) being provided (and interpreted) as evidence for a claim. You also don’t (usually) need to define basic terms of research design and methodology that you learned in this course; that knowledge is presumed. If, however, you are introducing concepts from another course (a separate stats course, an economics course, philosophy of science, whatever) then yes, you should treat them as not-common knowledge—a one-sentence explanation and a citation are usually sufficient. In short, if a point does not directly contribute to the exposition of your theory or the testing of that theory, it almost certainly doesn’t belong in your paper.

Practicing Peer Review Common wisdom among faculty is that for undergraduates, peer review is a generally useless practice that does nearly nothing to improve student papers, increases student anxiety about assignments and grading, and only adds to the

Chapter 10  Practicing Peer Review

grading load. Among undergraduates, common wisdom is that peer review is a generally useless practice in which those who have strong grammar or writing skills are forced to edit their peers’ papers, while everyone else mouths useless platitudes and vague “critiques” at each other. These twin beliefs stem from fundamental misperceptions: a misperception for students on the purpose and intent of peer review, and a misperception for faculty on the causes of students’ poor performance on peer review tasks. Faculty misunderstand the poor feedback as a function of students having written off the assignment as useless before they even did it, or as something into which students put minimal effort. The net result is crappy “reviews” with little value: It’s a self-fulfilling prophecy. Students, on the other hand, are often unclear about the function of peer review outside of the only context that they’ve likely experienced it: English class. A composition class has as its primary goal producing “good writing,” so students review primarily writing matters. Attention to surface errors such as grammar and spelling predominates; a few advanced students perhaps grapple with issues of thesis, evidence, and, on a really good day, structure. Most students lack the substantive knowledge to do more, especially if the paper examines topics beyond shared readings or personal experiences. Nobody is really happy with this situation. Students usually feel like it’s unproductive at best, frustrating at most, and busywork at worst; professors see additional grading and hassle with little noticeable improvement in the final products. In short, everyone agrees that student peer review is not serving its intended purpose, but it keeps getting assigned anyway. I hear these complaints regularly from both students and faculty, and frankly, I find it exasperating. Of course most students don’t give useful feedback; we don’t teach them how to do that or give them any guidance on what constitutes useful feedback for a particular assignment. Students, lacking guidance on what kinds of substantive matters to consider, default to doing the only thing they know: editing. So suboptimal outcomes occur, and until something changes, this equilibrium will persist. This section’s goal is to change something. I can’t do much about faculty opinions, but I can help students understand this process better. Student anxiety stems from two sources: uncertainty about the function and role of peer review in a noncomposition (“substantive”) classroom, and uncertainty about how to review something where they themselves have no expertise.11 So, we’ll tackle some of the issues of student uncertainty and frustration in the hope that knowledge leads to empowerment. We’ve touched on the role and function of 11Faculty interpret questions about “what are we supposed to do or talk about in the peer review” as students trying to game the system and do the minimal necessary to get a good grade. These questions are, instead, often a (poorly articulated) request for clarity in the assignment. For faculty, peer review in the discipline is a regular practice with widely understood norms. Combined with the broad level of background knowledge we receive in graduate school and the more narrow knowledge of our substantive specializations, peer review is a relatively straightforward thing. We often forget just how much we know, and most of us have no idea how much of what we know goes into reviewing work from our peers and our students.

235

236

Empirical Research and Writing

peer review at various places in this book (notably in Chapter 3), but we’ll consider it in more detail in the first subsection. The second subsection provides guidance for reviewing empirical research. One of the great things about empirical social science is that the strict expectations for content, consistent criteria for research design decisions, and the like mean that anybody can review and critique anything. Substantive knowledge is nice, and it can definitely add to the value of a review, but it’s not necessary.12

Peer Review in the Social Sciences In the social sciences, peer review’s primary function is to ensure that scholarship produces valid and valuable conclusions. We do this mostly by determining whether it conforms to our collective norms of research. Is the research design sound? Are the data valid and reliable? Is the analysis executed correctly, and do the conclusions follow from it? Most of the scholarly norms we’ve discussed in this book are about making sure you communicate these elements to your readers so they can evaluate your research on these grounds, and attention to these matters usually occupies the majority of a peer review. At the professional level, peer review also serves a gatekeeping function. The level of substantive knowledge that faculty bring to each other’s research allows them to make judgments about whether a piece of well-executed scholarship is sufficiently novel or otherwise contributory that it is worth one of the limited number of publication slots available in that particular outlet. Reviewers for journals are asking themselves if the work is done correctly, and also if it is worth publishing in that particular outlet given that outlet’s stated publishing mission and audience. At the student level, peer review is less a process of gatekeeping and more a process of continued development of the paper’s argument and evidence. Much as faculty members’ areas of expertise differ, so do students’ areas of strengths and knowledge. Students bring different perspectives and different sets of background knowledge to any particular assignment, and a peer review process allows students to (formally) pick each other’s brains for ideas. Lack of “expertise” in an area is often as much an asset as a hindrance because it leads you to question things that readers who are more familiar with the literature might overlook, assume, or accept because “that’s the way it’s always been done.” To be absolutely clear, peer review is not about editing. Editing is the author’s responsibility and it should be done to the best of the author’s ability prior to submitting the paper for peer review. (That’s why this chapter distinguishes 12How do you think one professor can grade an entire class worth of research papers? But seriously, sometimes the best feedback comes from smart people who are novices in your particular subject area. Much of my best dissertation feedback, for example, came from a chemist and a linguist. They asked questions about things that my political science readers just accepted or assumed.

Chapter 10  Practicing Peer Review

between self-editing and peer review stages of the postdrafting process.) Generally, comments on editing—grammar, word choice, style—do not appear in a formal peer review unless they substantially impinge on a paper’s readability.13

Getting the Most out of Peer Review The biggest thing that you personally can do to help make the peer review process a success is to make sure your paper is ready when you turn it in. This means doing serious line edits beforehand. You are not submitting a “draft” of your paper in any significant sense of the word; you are submitting a version of it. Your paper should be totally complete, including bibliography, appendices, citations, graphs and figures, etc. You should edit it to the best of your ability and format it to conform to your instructor’s preferences.14 You might be handing in this particular version to a peer, but it should be of sufficient quality that you would be comfortable handing in to a professor as your only graded submission. Failure to submit a quality document constitutes both an expression of extreme disrespect for your peers—who, after all, will have to read the document you submit—and a wasted opportunity for you, since you rob yourself of quality feedback on missing or unpolished work and instead get feedback on draft material that you may well end up cutting anyway. Remember, too, though, that peer review is an educational experience on both the doing and receiving ends. You are expected to get some benefits out of receiving peer feedback on your work, and to improve your own paper as a consequence, but at the same time, conducting a peer review of another paper is a valuable learning opportunity as well. It’s a chance to learn to interrogate research and become a critical consumer of scholarship. The review that you write is also an opportunity for you to demonstrate that you have mastered elements of research design or analysis that you personally may not have used in your own research. Simultaneously, though, it is an opportunity to demonstrate that you understand and can abide by norms of collegiality and professionalism in written communication. Remember to focus on constructive criticism and be as specific as you can. The suggestions in the next section will help you do just that.

Procedures for Peer Review Peer review serves several functions that range from superficial to significant. The trick to successful peer reviewing at the student level is to weight the time and effort put into each function by the importance of the function. Generally, 13At the professional level, however, brief remarks about discontinuity of style between sections of the paper are acceptable; this often signals poor editing by coauthors and is worth bringing to the authors’ attention. 14The time to visit the campus writing center or writing lab is not normally between the peer review version and the final submission version. It’s between the drafting stages and the peer review submission. Depending on the extent of changes your peer reviewers request, you may need to visit the writing center again, but that’s never a bad thing.

237

238

Empirical Research and Writing

the lower the level of the problem (and comment), the less valuable it is to the author. Try to focus the bulk of your review on midrange theoretical, conceptual, and empirical issues rather than line editing or questioning the entire premise of the paper.15 Preview 10.1 Reviewing Beyond Your Comfort Zone: It Takes a Village

Student 1: I’m interested in international environmental treaties. What can I say about race riots and cross-cutting social groups in US urban politics? She’s got an argument about social capital facilitating groups working together to solve racial problems . . . Wait. Treaties are about actors working together to solve a problem. Most environmental cooperation is a mixed-motive situation, where actors have incentives to cooperate and to cheat. It’s a collective action problem, or sometimes a tragedy of the commons. Do the racial groups in this paper face a mixed motive situation? If so, does it look like a tragedy of the commons, or more of a standard collective action problem (consuming vs. producing a public good)? Either way, that’s another explanation for the outcome of interest, and it makes different predictions than the author’s argument. Preview 10.2 Reviewing Beyond Your Comfort Zone: An American in Paris

Student 2: I study the role of US Department of Justice staff—bureaucrats— in decisions to investigate and prosecute federal criminal cases, and somehow I’m supposed to comment on this paper about the electoral strategies of far right wing political parties in France. Look at this, some of the sources aren’t even in English! The paper is making an argument about how party elites choose which districts to compete in and which to ignore. Hang on . . . Some of this sounds familiar. This is a case of a small group making strategically motivated decisions, with an eye on how their decisions now will affect future behavior. That sounds an awful lot like the logic of why the DOJ chooses to pursue some cases—because it thinks action now will deter future behavior by the other side (criminals, not the left wing or other rightist parties). Are party elites functionally similar to my bureaucrats? Do they maybe face the same motivations and constraints? I argue that my bureaucrats operate largely free of political influence from higher-ups, even when the higherups declare a “war” or something or other. What’s the relationship between these regional and party-level operatives, and the senior party leadership? How much does that matter for electoral strategy? Are there small group decision-making pathologies operating here, like in the DOJ?

15Both of those things have their time and place, but for student papers during a semester, they are neither here nor now.

Chapter 10  Practicing Peer Review

The typical review consists of three or four parts: the executive summary, the overview, the substantive feedback, and for a paper being reviewed for a journal or conference, a recommendation to the editors (accept, reject, R&R, etc.). The executive summary is a one-paragraph abstract-esque rendition of the review: core claim/finding, reviewer’s biggest substantive concerns, and recommendation (if relevant). Generally, the length of your review should increase with the length of the paper. For a 10- or 15-page paper (all written sections but excluding all tables, graphs, and references), you should be expecting two or three paragraphs of substantive comments. These would normally entail two or perhaps three moderately well-developed specific concerns and potential solutions (or at least things the author could do to alleviate your concern), and possibly one paragraph of miscellaneous notes. A good guideline for beginners is to aim to produce about one paragraph of substantive commentary for every four or five pages of substantive text. At the superficial level, most students perform at least a cursory check that all of the sections of the paper (and other assignment components such as an abstract) are present, and that formatting is both consistent and correct. If your peer is using the wrong citation format, for example, make a one-sentence note about it in the later portion of your review rather than correcting every instance. Such “reviewing” is both cursory, customary, and an act of courtesy or collegiality. It is not a fundamental or core element of the reviewing process and should not constitute the majority of your feedback; this is the kind of remark that belongs in the paragraph of miscellaneous notes. Also remember that you are reviewing the paper, not editing it. You should not need to return a marked-up copy of the paper with your review; your review should stand alone, and it should not be focused on line edits in the body of the paper itself.16 This kind of micro-level feedback is generally not helpful to the author, especially since much of the text itself may change between versions of the paper. At a midrange level, students can generate valid and very helpful critiques and feedback for one another. Midrange concerns generally focus on specific aspects of the paper’s content: theory articulation, measurement, case selection, literatures reviewed, etc. Feedback at this level is normally about the substance of the paper rather than its form or organization. You can consider specific research design decisions the author made, raise questions about the adequacy or strength of evidence for or against claims, introduce competing explanations the author has not considered, or do any of a variety of things. The subsections below provide a string of potential questions to consider for the paper as a whole, various components of research design, and the execution of both qualitative and quantitative analysis. You should also feel free to review relevant sections in this book as you review each section of the paper, especially if the author is using techniques or other components that you don’t 16In fact, I often encourage students to read review copies electronically, in PDF form, to discourage marking up the paper in an editing fashion.

239

240

Empirical Research and Writing

know as well. Each chapter gives plenty of questions for things to consider in research design decisions, and you can use those to help determine whether the author has provided convincing justifications for his or her decisions. Most student-to-student feedback should be at this midrange level; the bulk of the suggestions in the sections below focus here. At the most significant global level, students often face their biggest challenges. This is where broader background knowledge can be the most helpful, and where its absence is most commonly and acutely felt. That’s not to say that students can’t make global comments—just that they typically find it more difficult, and that’s okay. Not all papers need global comments. At the global level, you might consider issues such as whether the paper is framing the underlying research question in the most effective manner given the direction of the theory or findings, or whether the theory is missing some fundamental mechanism or alternative explanation. Does a theory about individual behavior such as voting or protesting rely, for example, on macro-level social forces such as “inequality” or “power” to explain behavior? Feedback at this level generally entails encouraging the author to rethink significant portions—if not all—of the paper, rather than simply reworking sections. I encourage you to think about critiques at this level, but for a typical semester-length course, time constraints may simply not allow authors to respond to these types of concerns in any meaningful way. Some midrange substantive issues border on global concerns. These include whether the testing strategy is appropriate for the theory (either quantitative or qualitative, or specific techniques within those categories), whether the hypotheses follow from the theory, or similar things along those lines that question some underlying component of the paper rather than a specific element of the paper itself. Resolving or addressing these issues involves questioning and possibly altering significant elements of the research design. For a professional who has much more distant deadlines and multiple opportunities to rewrite and resubmit, this is a challenge that will improve the paper. We’ll just estimate the other models over the semester break, or collect the new data next summer, and then send it off again. For students on a deadline, though, this kind of feedback can be devastating—particularly if you find that you agree with the reviewer! Students who find themselves the recipients of such feedback should not despair, whether they agree with the reviewer or not. The reviewer’s concerns are almost certainty valid. They typically either indicate an evolution in your paper or argument so that your initial design decisions are no longer as well justified as they were, or they indicate that you (the author) failed to communicate your justifications for your choices clearly. The latter condition is easy to remedy, with some revisions to your research design section and a footnote or two to acknowledge and respond to your reviewer’s concerns. The former condition is an accepted and acknowledged part of the research process. It happens to faculty all the time, but students never see it because those early versions of papers make their way into “leftovers” or “back drafts” files rather than into journal pages. In this situation, you will want to rework your introduction and research design section to frame the paper clearly in your initial question, and then shape the discussion and conclusion sections to provide

Chapter 10  Practicing Peer Review

the revised argument and outline some directions for future research and further testing. Your instructor probably has plenty of experience with this to provide you with further guidance related to your specific situation.

Reviewing Research Preliminaries and Design In reviewing a paper, you should begin by considering all aspects of the paper broadly. Think about what makes an argument convincing: a solid theory based on concepts from prior work, hypotheses based on observable implications of the theory, evidence that clearly supports the author’s claims while rejecting alternative explanations, and conclusions that are only as strong as the evidence. The questions here are guides, ideas of things for you to consider as you read and critique your peers’ papers, and they are relevant for all papers, no matter their analytical strategy. •• Theory. Does the author make a convincing claim for why he or she expects the proposed relationship? Is the argument grounded in the literature? Does it fundamentally make sense and is it well explained? In other words, does it have a theory based in clearly defined concepts? •• Research design. Is the choice of analytical technique both appropriate for the research question and justified in the paper in some manner? Does the author make an effort to rule out competing explanations? Does the author credibly control for or remove all outside influences that might affect the dependent variable? Is the appropriate population represented in the data, either in toto or as a sample drawn on a recognized basis? •• Measurement. Are the indicators valid measurements for the concepts? That is, do they capture the concept the author intended, or are they perhaps measuring something else (besides or instead of the intended concept)? Is the measurement strategy (the sources consulted and the manner in which the data were gathered) sound, or does it perhaps produce biased measurements by over- or underrepresenting some set of cases or variable values?

Reviewing Qualitative Research Critiques of qualitative research typically revolve around issues of case selection, evidence, and conclusions. •• Case selection. Is the research design appropriate for the research question and hypothesis (i.e., causal mechanism arguments probably require process tracing, probabilistic hypotheses probably require paired cases, etc.)? Does the author adequately and convincingly justify his or her case selections in a manner that is consistent with the principles for case selection established in Chapters 5 and 6? Can you explicitly identify the author’s criteria for case selection? Is selection bias possibly a problem for the research? •• Evidence. Is all the evidence there? Draw an arrow diagram of the paper’s theory and identify on your figure the paper’s specific pieces of evidence for each link, or sketch a data table and try to fill in the cells. Does the

241

242

Empirical Research and Writing

author present the evidence for his or her claim in a logical manner? Does the evidence support the claim that the author makes from it, or is the author overstating the strength of the evidence? What other evidence might the author provide to make the argument more convincing? Are there other observable implications that the author could address? •• Conclusions. Does the author appropriately limit the domain of his or her conclusions to the domain of the study? Is the conclusion an accurate assessment of the evidence, or does it overstate? Does it express an appropriate amount of uncertainty about the strength of the conclusions?

Reviewing Quantitative Research When reviewing quantitative research, you should think carefully about the author’s case selection, measurement strategies, and awareness (and avoidance) of analytical pitfalls. •• Measurement. Are all indicators valid across the temporal and spatial domains used in the paper? Do common alternative indicators exist in published literature, and if so, has the author identified his or her source properly and justified the use of that particular measure? If the author has constructed the measurement for this research, does he or she provide sufficient information on the coding processes to allow you to determine whether the data are coded consistently? •• Selection bias. Does the author’s dataset contain negative evidence— that is, do cases exhibit the full range of variation on the dependent variable? Does the author’s theory depend on a sequence of events or stages that might result in some potential cases selecting or filtering out of the sample? Are some categories of relevant cases missing or potentially undercounted in the dataset? •• Specification. Did the author choose a form of analysis that was appropriate for his or her data? Are all important potential causes of the dependent variable controlled for in the analysis? Does the author provide a correlation matrix or other information about potentially colinear variables and take appropriate steps to address any cases of extreme colinearity? Does the theory match the statistical model: Are interaction terms, logs, and/or other transformations used when needed? (Remember: When in doubt, check the theory!) Can you think of other model specifications that may be useful for testing the theory? These questions barely brush the tip of the iceberg for things you can consider when reviewing social science research. Many, many other topics exist for you to consider in your review of your peers’ research. Feel free to think about any questions from these lists, or any other questions not on these lists—whatever moves you. Negative answers to these kinds of questions might form the basis of one or more of your critiques. Think carefully about these questions; their superficial answer is almost always yes, but deeper probing often produces things that make you go “hmmm.” If you’ve got a concern about the

Chapter 10  Practicing Peer Review

paper and you’re having trouble articulating it, look at the list of terms in each chapter of this book to see if any of them jog your memory or provide the missing term. When all else fails, talk about the paper with a third classmate until you’ve managed to hash out your critique. This is an appropriate and permissible thing to do, though you should generally preserve the paper author’s anonymity if at all possible when reviewing nonfinal versions of research. Creating knowledge and engaging in scholarship is a public process, but courtesy suggests that discussing others’ errors or problems in works in progress should occur circumspectly.

Writing Your Review of Peer Research For a student review of a student paper, you should expect that your review will be about five paragraphs long. Most reviewers follow a fairly standard three- or four-part format that I introduced above. After the executive summary, the first body paragraph summarizes the piece’s main hypotheses and findings and concludes by raising the one or two main points of concern that the review will highlight. If the review is for a journal, the reviewer will provide a summary evaluation for the editors (reject the manuscript, offer the opportunity to revise and resubmit [R&R], accept as is, etc.). A second paragraph, if included, highlights things that the paper does particularly well and identifies its contributions to the study of the topic. Most undergraduate reviews will not have this paragraph. The body of the review is contained in the substantive paragraphs. The third and fourth (or more) paragraphs normally raise the reviewer’s most salient criticisms and provide support for the concern, and often make concrete requests for things to change or suggestions for improvement. Successful and useful reviews are specific in their concerns. Some examples of good feedback might include the following: •• “On page 3, the author proposes a mechanism involving X1, but the primary hypothesis (H1) tests a mechanism involving X2.” •• “The authors consider the role of X1, X2, and X3 in explaining Y, but they fail to consider the literature on Z as a cause of both X2 and Y. The results may thus suffer from omitted variable bias and/or endogeneity.” •• “The Cuba case illustrates the author’s claim of mechanism M1, but it also supports a key alternative hypothesis of mechanism M2. The paper would benefit from a brief discussion of why M1 is the stronger of the two explanations and how/why the author was able to rule out M2 as the primary cause.” Comments like “The author could benefit from citing more literature” are not helpful unless they are qualified: more literature on the relationship between X1 and Y, more literature that supports the claim of whatever, more literature that uses method M or approach A to resolve a particular research design problem, etc. You should feel free to ask for more clarity on a particular

243

244

Empirical Research and Writing

issue, a table to present data or a figure to elucidate an argument, and the like, but again, be specific. The paper’s author should be able to consult your review letter and make a checklist of things needing to be done to make the paper acceptable. When reviewing quantitative work, you may request additional robustness checks involving specific additional explanations, indicators, or variables; the author is not obligated to execute these unless the journal editor insists, but most authors will at least make an attempt to do them or explain why they can’t.17 The final paragraph summarizes the key concerns and usually ranks them by importance to address. If the review is for a journal, this paragraph reiterates the recommendation to the editor to accept, reject, or do something in between, and if the decision is R&R or accept with revisions, it identifies which of the changes are most critical.

Summary Writing is a momentum sport. Write regularly, even if in small chunks, and pay attention to your personal writing processes. When it’s writing time, it’s writing time—on paper, on the screen, on the back of a cafeteria napkin, on the chalkboard of an empty classroom after hours. Save the self-editing for later. Choose a self-editing strategy that targets the kinds of writing problems you know you usually have surface at the sentence, paragraph, or global level. Don’t be afraid to change strategies or try multiple strategies; taking more looks at your paper is always better. Peer review as it’s practiced in political science is about reviewing research design and execution, not line editing. For all papers, look for theory grounded in concepts, hypotheses that are directly observable implications of the theory, and measurement that is valid in the context of the research. For qualitative research, consider case selection, measurement, and whether the conclusions are commensurate with the scope of the theory and tests. For quantitative research, consider issues of measurement and case selection, and be alert for specification errors and analytical pitfalls the author might have missed. Helpful peer reviews provide concrete guidance about weaknesses in the paper and often include specific suggestions or requests for additional development of the paper.

Key Terms •• Passive voice •• Colloquialisms

•• Rhetorical questions •• Revise and resubmit (R&R)

17An optional paragraph between the substantive concerns and the conclusion may list any other items of concern such as key uncited publications as well as miscellaneous informative remarks to the author such as incorrect citation format, missing assignment parts, etc. We do not normally remark on grammar or style unless the issues are so severe that they significantly interfere with the reader’s ability to follow the argument.

C

Posters, Presentations, and Publishing

11

ongratulations! You’ve now finished writing your paper. For most researchers, though, this isn’t the end of the process—it’s only the beginning. A complete finished draft of the paper, like the one you turned (or will turn) in, is the first step in a process of soliciting public feedback from scholarly peers, revision, and ultimately publication in a peer-reviewed journal. While that’s probably not your desired endpoint, you should know your options for what to do with an excellent finished paper. This chapter discusses several ways in which you might share your research with your class, your college or university, or the broader public. In particular, you’ll learn about successfully creating and delivering posters and scholarly presentations. The chapter then considers effective use of presentation software in these contexts. Finally, we explore some options for sharing your work beyond local presentation: presentation at regional and national conferences, and publication in student journals.

Presentations Your instructor may require you to do a poster or presentation of your work, so we’ll spend a bit of time on each of them here. Presentations are the most common form of sharing scholarly work. In a typical oral presentation, authors have a fixed (and usually far too short, in their opinion!) amount of time to present their paper, and a brief question and answer (Q&A) period with the audience follows. Presentation sessions can take two forms: a basic classroom one-after-the-other format, or a conference panel format of several (four to five) related papers followed by comments from a discussant and a collective Q&A session. In a panel format, each panel has a chair, who manages time and serves as moderator, and a discussant, whose job is to provide comments on all the papers both jointly as a set and singly.1

Logistics Before the session, your instructor (or the chair of the panel) will inform you of your allotted time. In a professional panel, which typically lasts 1 hour and 1Sometimes,

the same person will serve as both chair and discussant.



245

246

Empirical Research and Writing

45 minutes, the norm is to save half of the panel time for Q&A and discussion; all of the presenters have about an hour total. As a result, the maximum presentation time you should expect is about 12 minutes. The panel chair (or a fellow student or the instructor) normally keeps time and signals presenters when they near the end of their allotted time.2 In larger panels and class sessions, the chair must be strict with presentation time; you will get cut off if you go over time. This is not personal; it’s to ensure that each author has a fair chance to present and receive feedback. For a panel presentation, panelists usually sit at the front of the room behind a table, facing the audience, with a podium for the speaker. For classroom presentations, normally the students assume their regular seats and go up one at a time. Your instructor may ask you to sit in the order of presentation to facilitate transitions. Presenters typically dress well—suits are de rigueur at most major conferences for presenters, and most guests will be in business attire as well. If you are doing a classroom presentation or classroom panel, your instructor may inform you that the dress code is somewhat more relaxed. One of the most important things to know about a panel (or any kind of presentation) is that ultimately it’s a team effort. The panel succeeds if its members work together as good colleagues, even if their research work and presentations are all done separately. Being a good colleague implies adherence to several norms. If you are on a panel with a discussant, you have an obligation to get your paper to him or her at least a week in advance unless explicitly told otherwise. Sending in your paper late does not allow the discussant enough time to read and prepare comments, so ultimately it’s your loss, but it also makes the discussant look bad. The norm in most cases is to email your paper to the discussant and to copy the rest of the panel (including the chair, if this is a separate person) on the email. You should then read all the other papers on the panel as they’re emailed to you. You don’t necessarily need to have comments prepared for your fellow panelists (though they’re always appreciated), but you should be prepared to discuss them and ask each other questions if the audience Q&A sags. A second norm of collegiality is adherence to time constraints. Time is very, very limited in presentation sessions, so be prepared. The biggest mistake novice presenters make is providing far too much detail. As a result, they take much more than their allotted time, get cut off, and so don’t get the feedback they were looking for. Since novice presenters are often also novice researchers, this is a huge loss. The biggest thing you can do to help yourself get good feedback and to be a good colleague is to practice your presentation. We’ll talk more about strategies for the presentation itself below, but when you budget your time for working on your presentation, don’t forget to allow practice time! Getting to your session early enough to load all the presentations on the projector computer also helps to stay on time during the panel itself by facilitating transitions between presenters. 2One

of the best pieces of advice I ever got was to aim for 2 minutes less than my allocated time— that way you’re finished when you actually run out of time.

Chapter 11  Posters, Presentations, and Publishing

Preparing Your Presentation Most contemporary research presentations involve projecting a slide show onto a screen and speaking alongside it, and I assume that your presentation requires you to prepare such a slide show. The number of slides you have is somewhat more flexible for a presentation than with a poster. A good guideline, though, is to have no more slides than you have minutes of your presentation, including the title slide, references, and tables (but not reserve slides, which we discuss below). Presentation organization varies from paper to paper because different papers have different needs. That said, a typical delivery structure might look something like this list: 1. Title. Title of your paper, your name, affiliation (department and college or university), and some basic type of contact information such as your school email address. The title should clearly express your research question; if it does not, you probably need to change your title! This is always the first slide. At this point you can briefly discuss the puzzle or question motivating your research. You may have a separate “The Puzzle” kind of slide as your second slide if you have a good puzzle and/or graphic that can go here. 2. Literature Review/Scholarly Context. What are the main themes or key pieces in the field of your research? How does your research connect to them? Consider a graphic organizer (Venn diagram, table/matrix, arraying items on axes, etc.) or using very brief bullets to illustrate the primary schools of thought or literatures of relevance. Remember that your goal is to contextualize your research for your audience: What do we know about the answer to the research question, and how does your paper relate to that existing knowledge? You’re not trying to demonstrate how well you know the literature or teach the audience about the literature’s contents. 3. Theory/Hypotheses. Express these clearly in terms that relate concepts to one another and indicate expected relationships between indicators. You may wish to use arrow diagrams or other visual tools. 4. Measurement/Methodology/Research Design. How did you operationalize your concepts into variables for analysis? What type of analysis did you do? Consider showing descriptive statistics for key variables in quantitative research or a visual representation of case selection principles for qualitative work. If you used a particularly complex research design, or an unusual or novel data source or methodology, you may need a second slide here. 5. Evidence/Findings. What did you find? For most quantitative work this is one slide of the results table, and one slide of discussion. Your needs may differ. As a helpful hint, you should not try to paste output from your statistical software directly into PowerPoint. It just makes a mess. Retyping it really is faster. Depending on the size of the tables, you may be able to paste in the tables you retyped for the paper itself. If a table is more than about five independent variables (plus headings, constant, N, R2, etc.), consult with your

247

248

Empirical Research and Writing

instructor about ways to fit it on one slide without using tiny font.3 For qualitative research, keep text to the absolute minimum. Try to find a way to express your variable values and/or findings in a table or other figure. At a minimum, aim for bullets, not complete sentences. 6. Evidence/Findings, 2. See above. Depending on your question and findings, you may want a third slide for graphic illustration of the findings, which we discuss in more detail below; be sure that you’re adding new information with this slide, not just rehashing the same ideas from the previous slide(s). 7. Conclusions. What was the main takeaway point from your research— which, if any, of your hypotheses were supported? How do your findings relate to the literature? Do any particularly interesting questions emerge from your research that you’d like to extend further? Does your research have any major limitations, and if so, how might you correct for them in the future? You should definitely make sure you bring your conclusion around to talk about support (or lack thereof) for your hypotheses, but beyond that, you have some flexibility. You don’t have to address all of the questions I suggested here, nor are they the only things you can discuss, but they do serve as good jumping-off points. 8. References. If you cited any literature in your presentation, you should plan to have a bibliography slide. Check with your instructor to find out if you need to display it in your talk, or whether you can simply have it in reserve. If you’re not sure, simply leave it up behind you while you answer questions. The precise structure of slides will differ from paper to paper. If, for example, you have one hypothesis from each of three literature families, you might merge those slides to have three slides, each of which has literature, theory, and hypothesis on it. If you have two hypotheses, each of which was tested using a separate design, then perhaps you have two sets of slides with each hypothesis and research design on the first slide, and results on the second. Sometimes the puzzle itself links clearly to the literature review in a way that obviates inclusion of a separate “puzzle” slide. In short: Use this as a guide, and use your paper’s organization as a guide, but don’t feel bound by either. Remember that your slides should be a summary of your points. Use bullets with sentence fragments. You should generally not have complete sentences on your slides, and that almost certainly rules out paragraphs! The only real exception to this is for qualitative work where the evidence is almost entirely in the form of quotes from subjects (discourse and content analysis, etc.). In these cases, sentences are necessary. Keep quotes as short as possible; use ellipsis points ( . . . ) to remove irrelevant parts of sentences or sentences intervening between two important points. 3Slides

of ginormous tables with tiny font are often derisively called “railroad schedules.”

Chapter 11  Posters, Presentations, and Publishing

Talking Tip

Use three periods ( . . . ) for excisions that are contained within a single sentence. Use four periods (. . . . ) for excisions that extend past a sentence end-marker (period, exclamation point, or question mark), regardless of the number of such end-markers that you remove. So if you remove the end of one sentence and two more complete sentences before quoting the tail end of the fourth, you would have “John Jones is a wild and crazy man . . . whose behavior reflects badly on the department.” The second fragment is capitalized only if it is capitalized in the original text, that is, starts the sentence.

Presenting Your Paper First, the most important part of your presentation is the presentation delivery itself, and there’s only one way to do well on that: Practice makes perfect. Get a couple of friends or classmates to lend you a half hour and give your presentation a few times until you have it down pat, you can deliver it within the time limit to an audience, and the audience is satisfied with the amount of detail you’re providing. Buying them all a cup of coffee in exchange for a half hour or so of time is a good deal. If you plan to rehearse on a weekend or evening, you may be able to use an empty classroom for practice.4 This also allows you to get comfortable using the classroom facilities for presenting, which may include a wireless presentation system (a remote control that allows you to advance slides without being at the computer), a gigantic (or very limited) screen, etc. You can also practice speaking loudly enough to be heard throughout the room, which can be a challenge in some classroom spaces, and gain experience at confidently fielding questions from the audience. Second, even if you don’t do formal practice runs in a classroom with an audience, you need to be familiar enough with your presentation that you can do it without relying on a script. When I queried other faculty about advice to give students for presentations, they almost uniformly replied, “Don’t read your talk!” You may want a script for your first couple of runs, but aim to graduate to a single sheet of talking points. Remember that you’ll be able to see your slides on the screen as you progress, so you don’t need to memorize your presentation. That said, one of the best things you can do to capture your audience’s attention is to deliver the first few slides—motivation, research question, and theory basically—without reference to your notes. If you are comfortable 4Depending on the rules for reserving classrooms at your particular institution, you may want to ask your instructor if she or he will reserve a classroom for you and your classmates to practice in. For a faculty member, this usually involves one quick call or email, and the improvement in presentations is definitely worth that few minutes of effort. So don’t be afraid to ask.

249

250

Empirical Research and Writing

and confident enough to talk for those first 2 to 3 minutes without notes, you can use that time to make eye contact with the audience, communicate your interest in the topic to them, and engage them with your presentation. Insider Insight

“For the love of all that is good in this world, don’t read your talk!!” —Prof. S. Croco, University of Maryland Finally, be prepared for questions. Remember that the reason you are presenting is to get feedback. Not everyone will agree with all of your research design choices or conclusions, and that’s entirely all right. Having an audience at your practice runs is great at least in part because they can ask questions and you can practice responding. Questions and comments disagreeing with you are not personal attacks on you or your ideas.5 The trick to responding to questions well is to remain calm. Some people find that writing down the question as it’s asked helps them to keep track of the question, craft an answer, and deliver it in a level fashion. Pausing a moment to process the question, gather your thoughts, and frame a response is also totally normal. Talking Tips

Answering a question with “I don’t know” or “That’s beyond the scope of my research” is acceptable though you should do this rarely and only when absolutely necessary. Better responses include these: “I’m not certain, but I can offer some conjectures.” “That’s not something I’ve looked into, but I would guess that . . .” “That’s a good question and I need to think about a response. Can I follow up with you later?” If you use the latter, be sure to follow up! Most strong presenters prepare ahead for questions that they expect the audience to ask by making a stash of reserve slides, to be shown if and only if someone in the audience asks. These might include sensitivity (robustness) tests, the full model written out, the full results table, additional quotations from primary sources, other forms of qualitative evidence, an arrow diagram 5This

is easy to say, but not always easy to do or feel—even for faculty who are veteran presenters.

Chapter 11  Posters, Presentations, and Publishing

of the theory (if it’s not already in the presentation), tables of descriptive statistics or pairwise correlations for all variables, or really anything that you expect someone might ask. Reserve slides allow the presenter to show that she or he has thought about these issues and understands why the questioner is asking about them. Audiences appreciate this kind of preparation, and it shows that you are a thorough and serious researcher.

Participating as a Discussant If you are participating in a panel as a discussant, you have a key role in helping the panel succeed as a panel and not just as a bunch of disparate speakers. The role of the discussant is twofold: first, to provide commentary and questions that will spark a discussion between the panelists and the audience, and second, to provide the panel’s authors with quality feedback on their work. You should begin to prepare as a discussant by emailing the panelists well in advance of the panel itself to establish a deadline for receipt of the papers so you can provide good comments. Be sure you are aware of your own schedule around that time so that you don’t cut it too close. In my experience, reading and commenting on a typical 20-page student paper will take about an hour, and a 40 pager will take about 2.5 hours. Most discussants want papers 1 to 2 weeks before the panel, but for major professional conferences, you can reasonably ask for them up to a month in advance (with sufficient notice, of course, for authors to adjust their writing schedules), and you can accept them as late as a couple of days before. If you have to travel to the conference, consider your travel schedule carefully when planning due dates; you probably won’t have printer access at the conference site. Once you have the papers, you can begin work on your two main tasks: prompting discussion and providing feedback. As you read the papers, feel free to make marginal comments and note questions to ask the author. You should carefully consider theory, literature, research design, and findings; in short, treat the paper as if you were reviewing it for a journal (as we discussed in Chapter 10). Try to avoid grammar and writing issues unless they are sufficiently severe that they impede your ability to understand the paper’s arguments.6 Most discussants prefer working on hard copies of the paper because they can simply hand the author the marked-up paper and a page of global comments at the panel. At the end of each paper, you should prepare some global notes for the author about how well the paper hangs together as a whole and about specific issues that you think deserve particular attention from the author. These can be either strengths or weaknesses of the paper; both types of feedback are helpful to authors. Aim for constructive feedback. Feel free to ask questions, raise issues the author doesn’t seem to have addressed, and/or push the author to dig deeper into the theory and/or data. Remember, specific feedback is more helpful than general feedback. 6Reading and Understanding Political Science (Powner 2007) has more guidance on questions to ask

yourself as you evaluate and critique empirical papers.

251

252

Empirical Research and Writing

The second role of facilitating discussion is, to many scholars, the more challenging task. Providing feedback involves looking at each paper singly; facilitating discussion requires looking at the papers as a group. This can be particularly challenging when the paper topics are disparate and are grouped together simply for convenience.7 Many discussants begin by looking for a wide range of similarities, complementarities, and differences between the papers. What, if any, assumptions do the papers share about how the world works? Anarchy, rationality, the absence of transaction costs, unitary actors, and many other assumptions could be lurking and worth discussion; what additional implications do these assumptions have for the theory and research design? You can also consider the ways in which the papers complement each other. They may be speaking to different stages in the same decision-making or electoral process, or answering the same underlying question using different testing strategies and data sources. You can likewise consider differences and points of contrast between the papers: assumptions, background literatures, theory, hypotheses, data sources, findings. As always, you don’t need to do all of these things, but they’re good places to start. The discussant’s comments to the panel traditionally follow all the panelists’ presentations. Unless the panel chair advises you otherwise, you should limit yourself to the same amount of presentation time as the panelists. Discussants do not usually prepare slide presentations. The default order to discuss papers is the order in which they were presented, which itself usually defaults to the order in which they were listed in the conference program. You can, however, discuss them in any order that makes sense for the points you want to make (i.e., discuss complementary papers together). Most discussants give a very brief oral summary of each paper and then highlight some key questions, findings, or concerns raised in more extensive comments to the presenter. After this, they provide some global commentary and/or questions raised by the panel as a whole, some of which are directed to the panelists and some of which are directed to the audience. Other discussants embed remarks to each of the individual presenters in the context of their broader comments to the entire panel. Following the conclusion of the discussant’s remarks, the chair may give each author a few moments to respond to the discussant’s comments. The remaining time is devoted to Q&A moderated by the panel chair, with audience questions having priority.

Posters Poster sessions are an increasingly popular way for scholars to share work in progress, particularly work that is in a relatively early stage. In a poster session, scholars display highly visual summaries of their work, and attendees 7I

once served as discussant on a panel titled “Communities, Institutions, and Security.” It had— yep, you got it—three papers: one on communities, one on institutions, and one on security.

Chapter 11  Posters, Presentations, and Publishing

mill around to review the posters and discuss them with their authors. This environment affords all involved a level of flexibility and freedom that panels lack. That said, the freedom of discussion is linked with a very limited amount of presentation space, so a fairly standard poster format has evolved to maximize information transmission in minimal space.

Logistics Poster sessions usually occur in a large room filled with lines of bulletin boards. Presenters tack their posters to the boards and then remain near their posters while audience members drift in and out of the room and explore those posters that interest them.8 One of the real advantages to this format for authors is the opportunity to discuss their paper with a relatively large number of people who are actually interested in it, and to interact with them in a way that allows for more circulation of ideas than the typical panel session. This is incredibly valuable for research in its early stages. For attendees, poster sessions are great because they can seek out only those papers in the session that interest them, instead of sitting through all the papers in a panel when they’re only interested in one. The brief and primarily visual nature of posters also allows them to explore a large number of papers in a limited amount of time. The opportunity to interact and network is really useful for them as well. Because of the slightly less formal nature of the poster session, dress code for presenters is often just a little more relaxed than it is for panel presenters (coat and tie instead of a suit for men; slacks or skirt and a nice blouse for women). In a typical student poster session, the format differs just a little bit. Most schools don’t have access to a bunch of mobile bulletin boards, for one, so frequently students will just affix their posters to the walls.9 The general format persists, though: Presenters stay near their posters while attendees mill around and ask questions. If presenting a poster is a required part of the course, students may be required to visit and evaluate a certain number of student posters. Depending on the number of students in the class, your instructor may choose to split the class in two. Half of the students present in the first half of the session while the other half act as attendees; then the groups switch roles halfway through the session. Outside guests may arrive and depart at will, but most students will be there for the entire session.

Preparing Your Poster Generally, a poster follows the same format as a paper presentation, though the emphasis is on presenting information visually instead of textually and in as concise a manner as possible. In a formal poster session, most people would have a single large sheet of paper (normally in the 4- by 5-foot range) as their 8Presenters 9Blue

may or may not have a table and a chair associated with their posting spaces.

painters’ tape is great for this.

253

254

Empirical Research and Writing

poster. Printing in that format is a bit expensive for a class assignment,10 so if your instructor allows, I encourage you to use a simpler strategy of printing and mounting a PowerPoint presentation. A typical poster for a 20-page paper would be about eight to nine slides (or segments/units on a single large poster), and probably be organized like this: 1. Title. Title of your paper, your name, affiliation (department and college or university), and some basic type of contact information such as your school email address. The title should clearly express your research question; if it does not; you probably need to change your title! 2. Literature Review/Scholarly Context. What are the main themes or key pieces in the field of your research? Consider a graphic organizer such as a Venn diagram, table/matrix, arraying items on axes, etc., rather than text bullets. 3. Theory/Hypotheses. Express these clearly in terms that relate concepts to one another and indicate expected relationships between indicators. 4. Measurement/Methodology/Research Design. How did you operationalize your concepts into variables for analysis? What type of analysis did you do? Consider showing descriptive statistics for key variables in quantitative research or a visual representation of case selection principles for qualitative work. If you used a particularly complex research design, or an unusual or novel data source or methodology, you may need a second slide here. 5. Evidence/Findings. What did you find? For most quantitative work this is one slide of the results table, and one slide of discussion. Your needs may differ. As a helpful hint, you should not try to paste output from your statistical software directly into PowerPoint. It just makes a mess. Retyping it really is faster. Depending on the size of the tables, you may be able to paste in the tables you retyped for the paper itself. If a table is more than about five independent variables (plus headings, constant, N, R2), consult with your instructor about ways to fit it on one slide without using a tiny font. For qualitative research, keep text to the absolute minimum. Try to find a way to express your variable values and/or findings in a table or other figure. 6. Evidence/Findings, 2. See above. Depending on your question and findings, you may want a third slide for graphic depiction of the findings, which we discuss in more detail below. In short, though, your evidence and findings should occupy at least a quarter of the poster. 10I’m told that FedEx Office (formerly Kinko’s) will do a 4- by 3-foot black-and-white poster for about $15 plus tax. Apparently this service is only available in person, not through their online printing system. Your college or university may also have a plotter printer available in the library or in a computer lab used by science or engineering students; costs there can be much cheaper.

Chapter 11  Posters, Presentations, and Publishing

7. Conclusions. What was the main takeaway point from your research— which, if any, of your hypotheses were supported? How do your findings relate to the literature? Do any particularly interesting questions emerge from your research that you’d like to extend further? Does your research have any major limitations, and if so, how might you correct for them in the future? You don’t have to address all of those, nor are they the only things you can discuss here, but they do serve as good jumping-off points. 8. References. If you cited any literature in your poster, you should plan to have a bibliography slide. Check with your instructor to find out if you need to display it with your poster, or whether you can simply have it in reserve. You then mount the slides on an ordinary piece of poster board.11 If you used PowerPoint with its default slide orientation (“landscape,” long edge on top), the pages will usually fit with one on top, then two columns of three. If the slides are in portrait orientation (short edge on top; you can change this under the Design Tab > Slide Layout), you can normally do three columns of two slides. The title slide then sits on top. Having pages sticking off the sides of the poster board makes the board harder to carry/transport, but you can get to three-by-three that way, with the title as the top left slide. To make a single large poster in PowerPoint, you simply change the slide size. Go to the Design tab in the Ribbon and click the “Slide Layout” button. Then change the poster size to your desired dimensions. Remember that the paper size is in exact inches—24 by 36 in., 36 by 48 in., or whatever—so while you set the layout to those dimensions, you will need to make your poster content smaller than that. Typically, you should plan to leave a half inch of empty space on all sides of a poster that you intend to print to allow for printer margins. The poster’s title, along with your name, affiliation, and contact information, normally go at the top center of the poster. Other than the size, this poster now operates exactly like any other PowerPoint slide: Use text boxes for text, insert graphs and bullets in the normal manner, etc. You can add a ruler and grid lines to your screen to help with layout by going to the View tab in the Ribbon and selecting those options; I find these tools invaluable for keeping my layout square and even. Remember that the size of text on the screen is deceptive; check in every now and then on overall layout by setting the zoom magnification to 100%. Try to avoid fonts smaller than about 18 to 20 points. Be sure your figures, graphs, and text are readable at a distance of 5 feet. Taking a screen shot at 100% magnification and then printing it will help you to determine whether your font and figure sizes are appropriate.

11I recommend double-sided tape or glue dots, which are sold in the scrapbooking section of most craft stores. Most liquid glues contain too much water and make the slides wrinkle.

255

256

Empirical Research and Writing

Preparing to Present Your Poster You will need to be able to present your poster to your viewers in a brief, concise manner. Remember that they are capable of reading the poster for themselves; your verbal summary should just hit the highlights. Think of it as the “elevator” version—a 30-second or shorter summary that encapsulates the core findings. Remember that you’ve worked from a research question, not a topic, and you’ve investigated a limited number of specific hypotheses; focus accordingly. Pick one or two main findings associated with your key hypotheses and organize the poster around that. You should also be prepared, however, to talk about parts of your design and your findings that you did not put on the poster. I encourage you to be as graphical as possible with the informational content of your poster. In general, keep the text on the slides to a minimum and remember that you’ll be standing by your poster to answer questions from viewers. The capabilities of PowerPoint allow you to make a visually interesting—and therefore more effective—poster to present your research. First, exploit the fact that you are working in a visual medium. Consider whether your results can be expressed as a graph—for example, regression coefficients and t-tests can be turned into columns for comparisons, which we discuss more below. Consider whether you can turn your theory/hypotheses into an arrow diagram, or display parts of your methodology (case selection, expectations, measurement) in a table or figure. Second, actively consider visual design. Try to avoid cluttered backgrounds and overly vivid or detailed color schemes. These distract from the information you’re presenting. At the extreme, poor font or color choices can make posters nearly illegible or incomprehensible. Most people find that using a discreet, professional-looking color scheme, one with some but not a lot of color, is the most effective way to communicate information in this format. Color is most useful for highlighting key points in your figure(s) and text because it both draws the viewer’s attention and also helps him or her to clarify the information being presented.12 Finally, most poster presenters find that having a few hard copies of the paper at the poster session is a good idea. Sometimes, people you talk with will be interested enough that they’ll ask for a copy so they can read more. Other times, you may need to reference or look something up to answer a question. Always have at least one for your own reference, but I encourage several if you can. Most presenters also keep a notebook and pen handy to jot down visitors’ suggestions or ideas to follow up on.

12If

you’re printing regular slides and attaching them to poster board, this is one of those times when spending the extra few dollars to print your slides in color is almost certainly worth it.

Chapter 11  Posters, Presentations, and Publishing

Insider Insight

“Bring Hershey’s Kisses or something like that to poster sessions and leave them on your table. They’re a great way to entice people over to your poster!” —Prof. M. Allendoerfer, George Washington University

Peer Pointers

“I wasn’t prepared for tough questions. I knew there would be questions and I was prepared to answer fluff about why I chose this thesis, what were my variables, how was I going to run my test, etc. I was not prepared for the ease at which professors and grad students could look at my small poster board, grasp the ideas and testing, and come back with major issues I couldn’t answer.” —Kailee C., College of Wooster

Slip ‘n’ Slide Presentation software such as PowerPoint is convenient and powerful. It is also powerfully tempting, leading to much use and also much abuse. You should plan time to prepare your presentation just as if it were another brief paper. Don’t think of it as something you can throw together. You will almost certainly need figures and tables, even if you did qualitative research, and these take time to construct well.13 Allow yourself sufficient time to prepare quality slides that reflect the amount of work you put into your paper and your pride in your work. Before we go into specifics about doing presentations well, three resources are worth your attention. First, Rob Salmond and David T. Smith offer wonderful, succinct advice on preparing posters (Salmond and Smith 2011) and presentations (Smith and Salmond 2011) specifically tailored to presenting empirical research in political science. These items are worth skimming even if they’re not on your course’s required reading list. Second, http://betterposters.blogspot.com has a lot of good guidance on designing 13One

obvious way for experienced audience members to tell well-prepared presentations from hastily assembled ones is the presenter’s ability to use graphs and figures effectively in the context of his or her argument.

257

258

Empirical Research and Writing

and presenting science visually. While some of the advice is tailored to posters specifically, some of it is just plain good visual design that can help anyone using PowerPoint or similar software. The author’s regular critiques of actual posters that readers submit gives some great advice and a wonderful beforeand-after chance to see what even small changes in layout and design can do for a graphic. Third, the Cheating Death by PowerPoint series is a fantastic introduction to making effective presentations, both visually and organizationally. You can access all three of the video lectures at www.lauramfoley. com/cdbppt.html. Corporate audiences are its primary target, but again, much of the advice is equally valid for scholars. The Internet offers a wide range of sources for guidance on posters and presentations of all types. The resources above are only a small sampling of what’s out there. I offer here a few words of my own advice for effective posters and presentations in political science research. First, manage the amount of text of the slides carefully. Keep font sizes at or above 20 point—18 if you’re desperate on things like graph or table labels—so that your audience can read the slide from a distance without squinting or getting distracted. Anytime your audience is busy reading or deciphering your slides, they are not paying attention to you, and that’s bad. This critically goes for evidence/findings slides, which can turn into enormous block quotes or sprawling results tables if the author is not particularly attentive to the risk. Second, building off of Smith and Salmond (2011), your slides (or poster) should not give your talk for you. Make sure that you are doing more than simply reading the slides. Slides should be nothing more than a highlights map, showing just the crucial points of structure that help your audience contextualize your argument and talk. You don’t need to toe the corporate line of “three bullets per slide, three words per bullet,” but you should use text sparingly. Smith and Salmond (2011) give a great set of examples of “more effective” and “less effective” slides to give you some ideas. Finally, remember that your slides are visual aids. Emphasize the visual aspect of the slides by using figures wherever possible. Small, information-rich but text-sparse tables and figures—such as the two-by-two matrices and Venn diagrams we used for theorizing and hypothesizing back in Chapter 2—are a great way to convert sections from text to visual. That said, skip the flashy graphics, any form of slide transition effects, and most animation.14 Your presentation is not an opportunity to show off your PowerPoint skills; it’s a time to share your research with your colleagues, classmates, and other audience members. 14A

little animation is okay for the purposes of information flow-rate control. If, for example, you feel that you really need to define the variables on the top of the slide before presenting the hypothesis on the bottom—perhaps you measured them in a particularly unique way or are not interpreting the concepts in the standard manner that most of the field uses—then you might consider providing a click cue for the hypothesis to appear. No flying, no fades, no sound effects; just have it appear. This kind of animation is acceptable in limited quantities, and only where clearly needed.

Chapter 11  Posters, Presentations, and Publishing

Graphing Quantitative Results One of the most common questions students have about presenting their research is how to present quantitative results without simply displaying a table of coefficients. A bar or column graph with error bars is a great way to communicate information about the magnitude of the effect (the column or bar height) and also the statistical significance (the error bar length). Error bars that cross the zero axis denote insignificant coefficients—the “plus-or-minus” range that the confidence interval shows contains 0. To add error bars to a column graph, first, enter your coefficients and the upper end of the 95% confidence interval (which is usually displayed on your results output) in a spreadsheet. Use Excel to create another column showing the high end of the confidence interval minus the coefficient. Then, make a column graph where each coefficient is its own series. They should all be appearing in different colors. Begin by selecting only one coefficient and making a single column, then use the Edit Data menu to add another series. Once you’ve got a graph, go to Graph Layout tab and select Error Bars > Error Bars by Percentage (it’s just to get them on the graph). Then, right click on one of the error bars that appears and select “Format error bar.” You should continue to use the two preset layouts at the top—both upper and lower error bars and with cap. Then select the Fixed Value option and enter the value you calculated (high end of the CI—coefficient) in the box. Repeat this process for each series on the chart. You should end up with graphs where the error bar’s center rests on the top of the column. If the bars are not centered, something is not right. Figure 11.1 shows results presented in a column graph. You can also use Format Data Series to insert space between your bars. Finally, be sure to label your vertical axis before copying and pasting the graph into PowerPoint; you may want to consider changing the background color or bar colors to work better with the color scheme of your poster slides. For Figure 11.1, I added the series labels (Conflict Res, Small Pres, etc.) using transparent text boxes overlaid on the graphs. This is a good example of exploiting the strengths and weaknesses of both platforms simultaneously.

Post-paper Possibilities As we discussed at the start of this chapter, turning in the paper draft is just the first step for most good research. Good research deserves to be shared, not least because it has discovered some new knowledge that may be of interest to others. Opportunities for sharing your research beyond your campus include presentation in poster and/or panel form, and also publication. Both publishing and presenting opportunities exist at the student and professional levels. I briefly discuss some of the major outlets and opportunities here. If you are interested in continuing to pursue your research, I recommend that you consult with your professor after she or he returns comments on your paper to discuss your options and make a plan.

259

Empirical Research and Writing

Figure 11.1  G  raphical Presentation of Significant and Insignificant Findings

When Did the European Union Cooperate? A Test

Change in Pr (Cooperation)

60 50 40 30 20 10 0 Conflict Res

Regional Int

Common Strat

Salience (log)

What Doesn’t Affect Cooperation? 15 Change in Pr (Cooperation)

260

10 Neighborhood

5 0 –5 –10

Small Pres

Neutral Pres

–15 –20 –25

SOURCE: Powner 2008.

Presentation Finding out about a conference is probably the most difficult step of the process. No centralized repository of conferences exists in political science, and this is particularly true for student conferences. Many graduate student conferences are hosted by the graduate students of a particular department or program, and

Chapter 11  Posters, Presentations, and Publishing

these may or may not be open to students outside of that department. Some graduate programs are open to undergraduates if and only if graduate students fail to take all of the available spaces. I wouldn’t bank on that option, but it’s worth knowing. Most graduate conferences are thematic, such as the University of Pittsburgh’s annual graduate student conference on the European Union. These thematic conferences are often announced on discipline-specific email lists. Your professor may be your best source for hearing of opportunities like this, so discussing options with him or her well in advance is helpful as he or she may then notice and bring to your attention opportunities of potential interest. Your professor’s email traffic probably only represents his or her research interests, however, so that shouldn’t substitute for you doing some diligent Internet searching on your own behalf. Applying to present at a conference is usually a straightforward and relatively simple task. Typically, application requires an abstract, along with basic contact information. Be sure that you revise and update your abstract to reflect your conclusions! Always use your school email address in official correspondence. Some student conferences may require that you submit the completed paper as an application, to ensure that they don’t give limited presentation slots to papers that aren’t actually going to get written. They may also require you to have a faculty sponsor, mentor, or adviser. Like the requirement of a completed paper, this is also helps to ensure that spaces are allocated wisely. If a conference to which you wish to apply requires a faculty sponsor, be sure to contact your instructor or another appropriate faculty member well in advance of the deadline—especially if it requires a letter of recommendation! Most of us need at least 3 weeks to write a recommendation, and possibly more if we need to read, reread, or review your paper to write it. If the conference requires that the faculty sponsor attend, this gets dicier. Most of us have very limited travel funds for conferences and must use them for our own research. If we are already planning to attend, most of us are happy to act as your sponsor. If we aren’t planning to attend, we may be able to point you to someone else to ask. Ultimately, though, you must respect your faculty adviser’s time. If we aren’t going and don’t know anyone else you can ask, and the conference requires you to have a sponsor, then you will need to seek another conference venue. Most conferences have a nominal registration fee to cover facilities, supplies, and support, and none that I know cover lodging, travel, and expenses. Your own institution may have some support available, particularly if you are going to a national conference of some sort such as Midwest (see below). Talk to your professor about funding opportunities in the department for student research. An increasing number of schools also have offices of undergraduate research, undergraduate research opportunity programs, or some similarly titled body that may have funds available. If all else fails, check with the office of your school’s dean or assistant/associate provost responsible for undergraduate education. Even if these bodies can provide some funding, though, you should not expect them to cover all of your expenses. One increasingly popular option is for schools, departments, and even departmental student associations to sponsor their own conferences, for their

261

262

Empirical Research and Writing

own students and perhaps students from nearby schools. These mini-conferences offer many of the same features of public conferences—a mixture of panel and poster presentation opportunities, plenty of opportunities to become involved, and a nice line on your résumé—for a much lower price. If you’re serious about your research, and you know others at your school who are serious about research too, you might want to raise the idea with your professor, your student organization, and/or other appropriate administrators. Student organizations are most likely to be receptive to requests to try innovative programming initiatives such as a conference, but increased interest in undergraduate research at campuses nationwide means that many schools are beginning to invest in this type of event. Finally, I should mention the National Conference for Undergraduate Research (NCUR). Participation in this conference is open to any student in a wide range of disciplines; political science is usually one of them. The call for submissions typically opens in mid-November and closes in mid-December. Acceptances go out in January, and the conference is in mid-April. When students ask about this opportunity, my reaction is somewhat mixed. The primary reason to pursue these opportunities to present or publish is because you want more feedback on your research and to continue with it. The vast majority of the audience at NCUR are undergraduates, and while students certainly can offer one another good feedback, it’s typically not worth the costs of the conference.15 So I don’t generally encourage presentation at this type of forum because it usually turns into presentation just for the sake of presentation—a line on your résumé rather than a research-improving action. I don’t usually recommend conference-going at all for students who don’t seriously have the improvement of their research and continued careers in research as their underlying goals.

Professional Conferences The field of political science and its sister field of international relations have a number of major professional conferences. Faculty often present their own research in these forums, both on panels and as posters. The limited number of presentation slots at these conferences means that undergraduates are unlikely to be accepted on a sole-authored paper unless the conference explicitly has an undergraduate research session. The Midwest Political Science Association (MPSA) offers several undergraduate poster sessions at its annual meeting, held every April in Chicago. I have taken undergraduate students to Midwest several times. Only one has continued to a PhD program, but all of them agree that it was a valuable and useful experience. Midwest’s student poster sessions occur in the main exhibit hall, and the majority of the audience 15These costs are not just monetary. Conferences require a substantial time investment for preparation as well as travel time and the duration of the conference itself, which falls during a busy time of the semester for most people. They also include the hassle of missing several days of classes in midsemester, when you have other assignments due and other obligations. In short, conference travel during the term is not something to be undertaken lightly.

Chapter 11  Posters, Presentations, and Publishing

is faculty—including faculty from many prominent graduate programs, who will often seek out the undergraduate sessions to look for promising student researchers. MPSA issues its annual call for proposals on its website after September 1 each year; undergraduate paper proposals usually close in January, but this date is subject to change. Most important, undergraduate papers require faculty sponsorship, preferably from a faculty member who is planning to attend the conference. If you are thinking of applying, you should definitely be in discussions with a professor. Sponsoring a student’s paper does not affect a professor’s own chances of getting a paper accepted, nor does it obligate the professor to attend the conference or the student’s session (if accepted). If your potential sponsor has questions, please encourage him or her to contact the MPSA office or me personally. Getting onto a conference program in a regular panel as an undergraduate is very, very difficult.16 The only way I have seen this done is via coauthoring. Coauthoring is rare, and students should generally wait to be approached by a professor. (I’ve done it once for a student whose research overlapped my own work.) Faculty are generally reluctant to propose coauthoring because doing research outside our own areas of expertise is a significant time investment, as is the added time to coach a novice researcher through the process, and faculty research time is already very, very limited during the academic year. Peer Pointers

“If you’re a student, no one expects extremely high-quality work or a mind-blowing presentation out of you, and people genuinely want to help you improve. I would have stressed out less [before my first professional conference].” —Jane S., Emory University

Publication A final option worth considering for your completed research is publication. A number of quality student journals exist that you can consider as outlets for your research, including some generalist (non-discipline-specific) titles as well as some that are specific to political science and international relations. This section discusses some general things to know about submitting your work to journals, such as procedures and norms. Increasing attention to, and interest in, undergraduate research in the last decade or so means that more student 16One of my advanced MA students got into what he called a “grownup panel” at MPSA on his own.

Given his background and his work experience, though, he was much closer in skill level to a PhD student, so I hesitate to generalize from his experience.

263

264

Empirical Research and Writing

journals are popping up every year; it’s always wise to check with your own institution first before sending your work elsewhere.17 First, a few general remarks about student journals. In almost all ways that matter, these journals function just as professional journals do. Double-blind peer review occurs, and then editorial boards (usually composed of undergraduate student volunteers) make selection decisions under the guidance of a (student) editor in chief. The largest differences between professional and student journals are the frequency of publication— usually twice a year rather than quarterly—and the quality of feedback Journals from reviewers. Because reviewers are also undergraduate students, Publishing their ability to provide feedback is limited to what undergraduates Undergraduate know and learn. Because that varies from school to school and student Research to student, you might luck out and get someone with a lot of knowledge that you don’t have who gives you great feedback about things you never thought of and who helps you greatly to improve your paper. Most likely, though, your reviewer will not provide a lot of guidance. Unquestioned acceptances are much more common than they are at the professional level, and most revise and resubmit (R&R) decisions from undergraduate journals concern stylistic and/or length issues rather than substance. That said, as a second point, most student journals hold to the standard norms of professional publishing. The largest of these norms is that of sole consideration: You should submit your paper to no more than one journal at a time. Most journals will require you to attest that your submission is not under review elsewhere at the same time that it is under review with them. This helps to make efficient use of limited reviewer time and energy and journal space, and it also helps to avoid significant copyright issues down the road should both journals publish your work. The second norm relates to blind peer review. When you submit your work, you will normally need to send two (typically electronic) versions of your paper. One should be fully identified, with your name, institution, contact information, and acknowledgments on the cover page (i.e., pretty much the same way you turned it in). The second version must be anonymous. The cover page should contain only the paper’s title, and your name should not appear anywhere in the document. Be sure to check the header, if you used your last name and page number there, and the “author” and “last saved by” fields of the document’s properties. Be sure to check the Instructions for Submissions or similar document, often found on the journal’s website, for specific formatting instructions such as citation style, manuscript length, and preferred file format. Finally, use the journal’s mission or scope statement to identify the most likely outlets for your work. Since you can only submit to one journal at a time, and several months usually elapse between submission and decision, you 17Journals

publishing solely graduate student (MA-level) research do exist; some of the journals listed on the website are also open to graduate research.

Chapter 11  Posters, Presentations, and Publishing

should choose wisely. Some student journals are multidisciplinary; some are exclusively political science. Some are even more narrow than that, focusing only on political theory/philosophy or international relations. Always read the scope statement to make sure your paper is within the range of what the journal publishes. Finally, some journals are open to publishing work from students at any institution; others restrict themselves to pieces written at their own institution. This latter criterion is often subject to change over time, so be sure you have current information before you submit your work.

Brief Remarks on Graduate Study in Political Science and International Relations If the theme of this chapter has been words that start with P—posters, presentations, publishing—this section’s word is proselytizing. I really do sincerely hope that you found some part of the research process enjoyable. If you really enjoyed it and want to continue doing it, you may want to consider further study in political science. And if that’s the case, you have two options: MA programs and PhD programs. Most master’s degrees—MA in political science or international affairs or the like—are primarily applied degrees.18 They emphasize politics in the real world, with a heavy diet of proper nouns, specific cases, and lots of details. Most such programs include significant amounts of classroom time with faculty members; internships may be part of the program, but classroom work dominates. MA programs do not typically emphasize skills in conducting academic (or even applied) research, though most contain at least one required methodology course and a thesis or other major research project.19 Typically, these standard MAs in political science serve two purposes. First, they can provide additional substantive knowledge to those changing careers or those in specialized positions where more subject-specific material is needed. Second, they can serve a credentialing function for those in jobs, such as education or some government positions, where the act of obtaining a master’s degree makes the holder eligible for a higher pay grade, promotions, or the like. The key thing is that in both of these cases, one seeks the MA in the context of a particular job for which content knowledge or a credential is needed. Seeking an MA in political science in the absence of a job requiring 18This discussion applies primarily to American or American-style degrees. In the context of other

educational systems, the meanings and roles of various degrees differ. If you’re considering postgraduate study outside of the United States, consult with an expert in that country’s educational system prior to committing to it or setting your heart on it. 19A few research-oriented MAs exist; more on them below. The British higher education system refers to master’s degrees of this general classroom-based shape as “taught” programs. Some programs of this shape describe themselves as MS degrees—Master of Science—because they have a highly technical focus, such as international development economics or international business. The general shape of the program is the same, however, and the advice below continues to apply.

265

266

Empirical Research and Writing

specialized knowledge, and/or in the absence of 3 to 5 years of work experience, is generally not a recommended course of action. If you want to be a politician, a political strategist, a diplomat, a Euro-/bureaucrat, or something else along those lines, you may want to eventually consider seeking an MA, but the key word there is eventually. For those interested in a career spent conducting research, however, PhD programs provide an alternative to MA programs. Unlike some fields, such as English or the natural sciences, master’s degrees do not serve as PhD preparation courses in either substantive knowledge or preprofessional training. In political science, a master’s is thus neither necessary nor particularly recommended for application to PhD programs. The two types of degree programs are almost totally different. The focus of a PhD program, even at policy-oriented schools, is research. What form that research takes, and what the ultimate product or use of that research is, varies across types of schools, but all types of PhD programs share a commitment to training scholars. Like the MA application process, you typically need to begin your application packet a full year before you hope to start graduate study. Ideally, you should begin talking to faculty about your interests and intentions by the spring of your junior year. You will need at least two or three faculty members to write academic recommendations for you; these faculty should have taught you in class and should be familiar with your writing, your work ethic, and your academic history.20 So begin networking with your instructors now. They can help you identify programs for which you’re a strong candidate, and they may know of opportunities at your school and elsewhere to help you determine if this is the right path for you.21 They have been there and done this, and they often make incredible mentors as you progress through graduate school. You don’t need to be a 4.0 scholar or have amazing GRE scores to get into a PhD program. What you crucially need to have is intellectual curiosity—that driving need to figure things out and to poke at puzzles until you can solve them—and a strong set of critical thinking skills to help you approach those puzzles in systematic and rigorous ways. For most top PhD programs, the best indicator of that latter quality is the GRE math score; they’re looking to see at least 155, and more often 160. Admissions committees don’t care much about your ability to do algebra or remember specific geometry formulas. To them, 20Most

PhD programs strongly prefer that your letters come from PhD-holding faculty. These individuals have completed PhD programs themselves; they know what is necessary for success in such programs and are therefore well-placed to assess whether you display those characteristics. Teaching assistants and non-PhD-holding faculty (such as faculty with law degrees or those who are “ABD” [all but dissertation] in a PhD program) are not usually the strongest recommenders. 21In particular, the American Political Science Association offers the Ralph Bunche Summer Institute and the Minority Student Recruitment Project for minority students interested in pursuing a PhD in political science. The National Science Foundation also sponsors several Research Experience for Undergraduates (REU) programs in political science each year. Their sites change year to year based on funding, and no centralized information source exists, so you’ll need to research these on your own.

Chapter 11  Posters, Presentations, and Publishing

the math score represents an ability to apply a set of standardized logical tools to problems and reach logically correct solutions. Because PhD programs are about developing research skills, committees want to know that you have a strong grounding in systematic problem solving. I know, though, that math isn’t the strongest suit for most political science types; we tend to be more verbally inclined than mathematically inclined. The best way to improve your math skills is to prepare. I don’t recommend taking a test preparation course as they’re incredibly expensive. You can do just as well on your own with practice books borrowed from the library and a firm commitment to regular practice time. If you’re still concerned about demonstrating math ability outside of a test environment, consider taking calculus, a more advanced statistics course, or a basic econometrics course; all are offered regularly even at relatively small schools. Including with your application a quantitative research paper that you prepared for a class, published, or presented somewhere is a another good signaling tool. As for demonstrating intellectual curiosity, the best place to do that is in your writing sample(s) and/or personal statement(s). The combination and length of documents that you need varies from school to school, but you should definitely have a basic personal statement prepared. These statements differ from those you may have written for undergraduate applications in that while they are called personal statements, they’re really not about you that much. They’re more about your research interests, not why you personally are special and should be admitted. These statements can, if appropriate, delve briefly into the origins of your interest in this research question or your interest in pursuing a PhD, but they should make clear that a strong empirical puzzle motivates your interests in graduate school. Most successful application statements also connect their author’s puzzle to the work of specific faculty, faculty clusters, or research centers at the school to help demonstrate a good “fit” between the student’s interests and the faculty’s interests. This is important because ultimately you must have an adviser for your dissertation. Schools do not admit students whom they do not feel they can successfully graduate. If the department lacks a faculty member who could advise a dissertation in the field of your interest, or the potential adviser may be leaving the school, may be on leave, or has too many advisees already, then it won’t admit you. This is in the best interests of both you and the school, so don’t take it personally. Regardless of your future career path, I wish you the best of luck. I hope this book has helped to introduce you to the joy (and yes, the fun) of discovering knowledge. Research can be an absolutely fascinating endeavor, and I hope this book and I have convinced you to try it again sometime.

Summary Posters, presentations, and publications are the three main ways to circulate your research for feedback and dispersion. Each has strengths and weaknesses. Posters require you to be concise and graphical, but they allow extensive

267

268

Empirical Research and Writing

interaction with the audience. Fortunately, powerful software designed for presentations facilitates good graphic design even for novices. Presentations, on the other hand, can be somewhat longer and less graphic than posters, but they also come at the cost of more limited audience feedback. Finally, publication is the usual end state of a piece of research; its strengths are its permanence and circulation, but its costs for undergraduates often include significant time for revisions, delays in publication, and weak feedback from reviewers. Options for continued study and involvement in political science research include master’s degrees, which are typically more applied and policy oriented, and PhD (doctoral) programs, which are research oriented.

G l o s s a r y

Adjusted R2  Goodness of fit measure for multivariate OLS regression; adjusts baseline R2 to account for degrees of freedom lost to increased IVs; similar interpretation to R2 (Chapter 8) AJPS  American Journal of Political Science Analytic narrative  Tool for qualitative analysis of hypotheses derived from formal models; variable focused (Chapter 4) ANES  American National Elections Study Annotated bibliography  Preliterature review tool combining complete citations for sources with a brief summary and commentary on each piece (Chapter 3) Assumption  Part of a theory: a claim or belief about how the world operates (Chapter 2) APA  American Psychological Association (organization and citation format) APSA  American Political Science Association (organization and citation format) APSR  American Political Science Review Background attribution  Level of interview confidentiality generally allowing data from the interview to be cited to an interview provided that the individual speaker cannot be identified from the citation; direct quotes may or may not be permitted by the interviewee (Chapter 6) Baseline model  Initial or foundational specification to which refinements, additions, and adjustments are made (Chapter 9) Bibliographic management software  Electronic tools that manage, collate, and format citations in manuscripts (Chapter 3) Bibliography-hopping  Literature discovery process in which the researcher uses items cited in prominent or recent work to locate other relevant pieces and authors further back in the scholarly chain (Chapter 3) Case control method  Between-case tool for qualitative analysis of carefully matched cases; matching on key variables associated with alternate hypotheses allows logical exclusion of these variables as causes of outcomes (Chapter 4) Causal mechanism  Part of a theory: the specific chain of events that leads from the independent variable to the dependent variable (Chapter 2) Causal process observation (CPO)  Process-tracing observation (within a case) that captures a sequence of events, phases, or characteristics at a given point within a larger “case” (Chapter 5)



269

270

Empirical Research and Writing

Census, population  An exhaustive listing of all elements (individuals, cases, etc.) of the population (Chapter 5) Ceteris paribus assumption  Assumption, often implicit, that all other variables are held at constant values when we consider the effect of changing one IV’s value on the DV (Chapter 7) Codebook  Document accompanying a published or replication dataset explaining coding rules, variable values, and related information for that data; sometimes called documentation (Chapter 7) Codes, coding  Process of applying measurement rules to evidence to produce data (Chapter 7) Colinearity (multicolinearity)  Excessive correlation between IVs of a multivariate model; violates assumptions of regression and similar models; requires correction for values above about 0.6 (Chapter 8) Colloquialisms  Casual or informal terms that are inappropriate in formal writing 10 Comparative statics  Study of the effects of changing one variable from a given state or value to another; often used to refer to hypotheses derived from formal or similar models about the effects of changing variable values (Chapter 5) Composite variables  Variables created by combining and/or manipulating other variables’ values (Chapter 8) Concept web  Form of graphic organizer emphasizing nonlinear and multiple connections between concepts (Chapter 3) Conditional hypothesis  Claim that the effect of one IV is dependent (conditional) on the effect of another IV (Chapter 4) Confounding variable  Variables correlated with both the DV and IV of interest (Chapter 8) Constant  A variable whose value does not change across cases; OR another name for the intercept in regression (Chapter 5) Content analysis  Tool for examining patterns in written or spoken sources; data reduction occurs by quantifying speech elements (Chapter 4) Control variable (CV)  Variable that we believe influences the DV along with our IV(s) of interest; must be accounted for in any qualitative or quantitative analysis to obtain accurate results (Chapter 2, 5, 7) Corresponding author  Author designated in a coauthored piece as the contact point for inquiries (Chapter 7) Counterexamples  Single or rare cases that do not fit the hypothesis; possibly defined as outliers in some contexts (Chapter 2) Counter-explanations  Hypotheses that explain adverse findings (Chapter 2) Counterfactual  Thought experiments used in qualitative research to consider the potential outcomes of (unobserved) variable values or combinations (Chapter 6) Country-year  Dataset structure where observations are each country in each year; common in comparative and international politics research (Chapter 7)

Glossary

COW  Correlates of War data collection project CPO See Causal process observation (Chapter 5) Cronbach’s alpha (α)  Statistic indicating how closely the components of a composite indicator capture the same underlying concept; higher values indicate better internal consistency, with a commonly accepted cutoff around 0.7 (Chapter 8) Cross-tabulation  Bivariate analytical technique for nominal or ordinal level IVs and DVs; indicator of statistical significance is usually χ2 (Chapter 4) CV See Control variable (Chapter 5) Data  Intentionally gathered information, usually reflecting values of variables (Chapter 6) Data archive  Centralized storage facility for collected data; best known is ICPSR (Chapter 7) Dataset  Collection of intentionally gathered information (data) normally consisting of one value for each variable for each observation (Chapter 6) Dataset observation (DSO)  A standard qualitative or quantitative observation establishing one and only one value for every variable for each case in the dataset (Chapter 5) Deductive theorizing  Researcher derives hypotheses from generalized analysis outside the context of any specific case (Chapter 2) Degrees of freedom  Roughly speaking, the number of independent pieces of information available for analysis after various calculations constrain the data (Chapter 5) Dependent (or outcome) variable (DV)  The outcome variable in a hypothesis (Chapter 2) Descriptive statistics  Summarize and describe characteristics of data (mean, median, mode, standard deviation, etc.) without reference or generalization beyond the available data itself (Chapter 7) Deterministic hypothesis  Class of hypotheses arguing that a particular relationship should hold across all cases; includes primarily hypotheses of necessity, sufficiency, and necessity and sufficiency (Chapter 4) Dichotomous  Describes an indicator or variable that takes on only two possible values; usually framed as a “yes/no” item; sometimes called “dummy” variables (Chapter 4) Dictionary  List of terms of interest used in text mining (Chapter 6) Direct relationship  A positive association between two variables—they both increase or decrease together (Chapter 2) Directional hypotheses  Predict increases or decreases in a DV as a function of increases in one or more IVs (Chapter 4) .do file  Stata file containing a list of commands to be executed in a single batch (Chapter 8) Document analysis  Qualitative analytical technique: systematic review of primary, secondary, and occasionally tertiary sources (Chapter 6) Domain  Spatial and temporal scope of a study (Chapter 7)

271

272

Empirical Research and Writing

Domain (of a theory)  The set of cases over which a theory is expected to operate; identified in a theory’s scope conditions (Chapter 2) Double-blind  Form of peer review in which neither the author(s) nor the reviewer(s) know the other’s identity (Chapter 3) DSO See Dataset observation (Chapter 5) Dummy variable See Dichotomous (Chapter 4) DV  Dependent variable (Chapter 2) Dyad  Pair of states or other units; common unit of analysis in international relations (Chapter 7) Dyad  Pair of states or other actors (used as a unit of analysis) (Chapter 2) Elite interviewing  Qualitative data collection process: interviews conducted with individuals chosen because of positions they occupy, rather than as representatives of some larger class; individuals need not be high profile to qualify as elites (Chapter 6) Empirical  Type of research question: asks about how the world is, based on observable data and tested by scientific observation or experiment (Chapter 1) Endogeneity  Problematic circumstance where IVs cause one another; requires deployment of appropriate fixes to estimate relationships (Chapter 8) Endogenous  Occurring or determined by forces inside the model Episodic record  Qualitative data source: reports produced on an inconsistent basis, sporadically, or one time only (Chapter 6) EPSEM  Equal Probability of Selection Method (for drawing samples) Equifinality  Property of the social world where many causal routes to a single outcome exist (Chapter 4) Exogenous   Occurring or determined by forces outside of the model Factual/procedural  Type of research question: identifies the basic facts of a situation (Chapter 1) Falsifiability  Characteristic of a theory: We can identify observable implications that would occur if the theory were incorrect (Chapter 1) Falsifiers  Specific pieces of evidence that would falsify a theory; the research would expect to find these if the theory were incorrect (Chapter 2) FE See Fixed effects (Chapter 8) Fertility  Characteristic of a theory: suggests other observable implications or novel hypotheses (Chapter 1) Finding aid  Guides to special or archival collections detailing temporal and substantive scope, material origins, etc. (Chapter 6) Fixed effects (FE)  Systematic variation across units in a study that is correlated with both DV and IV of interest or battery of dummy variables included in a study to capture fixed effects (Chapter 8)

Glossary

Formal model  Form of deductive reasoning that uses game theory and similar tools to depict and analyze situations in the abstract (Chapter 5) FSA See Fuzzy Set Analysis (Chapter 4) Fully attributed  Level of interview confidentiality allowing direct quotes and identification of speaker by name; rarely used in social science (Chapter 6) Fuzzy Set Analysis (FSA)  Similar to Qualitative Comparative Analysis; analyzes data using Boolean logic where elements of the sets can have degrees of membership (i.e., cases can be evaluated as having or being more or less of some characteristic); (Chapter 4) Gamma statistic  Test statistic for strength and direction of association between ordinal variables (Chapter 4) General Social Survey (GSS)  Large-scale study of the American population conducted yearly (1972–1994) or in alternating years (1996–present) (Chapter 7) GDP  Gross domestic product GDPPC  GDP per capita Goodness of fit statistic  Summary value computed by models to indicate how well they explain the data; varies by model type, with OLS and variants using R2 or Adjusted R2 and MLE based models using log likelihood (Chapter 8, 9) Graphic organizer  Visual depiction of information and relationships (Chapter 3) GSS  General Social Survey (Chapter 7) GUI  Graphical user interface—drop-down menus of a software program Hard test  A context in which a theory is least likely to be successful (Chapter 5) HDI  Human Development Index Hypothesis  A statement of the relationship that the researcher expects to find between her dependent variable and independent variable(s); usually phrased in terms of indicators (Chapter 2) Hypothetical  Type of research question: tries to predict what might happen (Chapter 1) ICPSR  Inter-University Consortium for Political and Social Research (Chapter 7) ILL  Interlibrary loan (Chapter 3) Independent (or cause) variable (IV)  The causal or explanatory variable in a hypothesis (Chapter 2) Index (scale)  Tool of data reduction: combines multiple indicators into a single measure, often by averaging (pl: indices) (Chapter 8) Indicator  Observable characteristic; measurable version of a concept (Chapter 2) Inductive theorizing  Researcher identifies potential explanations from one or a few cases, and then generalizes to other cases (Chapter 2) Inferential statistics  Brach of statistics that generalizes from samples to populations (Chapter 7)

273

274

Empirical Research and Writing

Instrumental variable  One common “solution” to endogeneity problems; a variable correlated with one of the problem variables is used as a synonym of sorts; occasionally called two-stage least squares (2SLS) (Chapter 8) Interaction terms  Composite variables generated by multiplying two or more component variables together; used to test conditional hypotheses (Chapter 8) Interactive hypothesis See Conditional hypothesis (Chapter 4) Inter-coder reliability rating  Measure calculated on data where multiple coders review a given case; indicates how well the coders agree on the values of the variables; sometimes called inter-reader reliability rating (Chapter 7) Interlibrary loan (ILL)  System allowing researchers to request items not owned by their own library from other sources (Chapter 3) Intermediate-N designs  Techniques for analysis of between approximately 30 and 50 cases (Chapter 4) Interval-ratio  Level of measurement: continuous or discrete quantities with consistent units attached such as years or votes; most precise level used in political science (Chapter 4, 6) Inverse relationship  A negative association between two variables—one increases as the other decreases (Chapter 2) IO  International Organization (journal) IRB  Institutional Review Board (human subjects board), responsible for ensuring compliance with human subjects research rules (Chapter 6) IV  Independent variable (Chapter 2) Journal Storage Project (JSTOR)  Massive database of full-text journals in many disciplines, with extensive historical holdings (Chapter 3) JSTOR  Journal Storage Project (Chapter 3) Lagged variables  Variables observed in a period prior to the period of analysis—for example, GDP from a prior year (Chapter 8) Level of measurement  Degree of precision used in the operationalization of a variable; most common in political science are interval-ratio, ordinal, and nominal (Chapter 4, 6) Likert scale  Common survey response scale (ordinal measurement) differentiating between four or five points: strongly agree, agree, no opinion/neutral (sometimes omitted), disagree, strongly disagree; other verbs may be substituted for agree. (Chapter 4) Linear probability model  Uses regression to analyze data with dichotomous DVs; problematic because it violates several key regression assumptions, but usually viable for student work (Chapter 4) Literature, scholarly  Body of research about a research question or research theme (Chapter 3) Log transformation  Logarithmic transformation of a variable exhibiting significant skew or exponential distribution; log value of x is the value to which the mathematical constant e must be raised to obtain x (Chapter 8)

Glossary

Logit  Common tool, with probit, for quantitative analysis of dichotomous DV data; test statistic is significance of coefficients though interpretation of coefficients is more complex than in regression (Chapter 4) Main diagonal  Set of cells in a square table or matrix running from the top left corner to the bottom right corner; typically corresponds to cases where X = Y (Chapter 4) Marginal effect  Change in DV value resulting from changing IV value(s) in the manner specified; appropriate measure of effect size for OLS and other multivariate estimators (Chapter 9) Measurement  Process by which information is converted into systematized values of variables that are comparable across observations (Chapter 6) Methodology   The study of research design and study (and creation) of new analysis techniques; specialists in this field are methodologists (Chapter 4) MLA  Modern Language Association (organization and citation format) MLE  Maximum likelihood estimation (Chapter 9) Monograph  Scholarly book or other extended work, usually written by a single author (Chapter 3) Multicolinearity See Colinearity (Chapter 8) Multinomial logit  Quantitative tool for analyzing DVs with three or more nominal categories; functionally identical to polychotomous probit (Chapter 4) N  Number of observations (Chapter 8) Natural experiment  Situation in which cases assigned to experimental and control conditions are determined by nature or exogenous forces, but outcome is arguably random or very close to it (Chapter 5) Necessary condition  Asserts that some cause X is required for the outcome Y to occur; implies that Y cannot occur in the absence of X (Chapter 4) Negative evidence  Cases whose significance to hypothesis testing is the absence of the phenomenon of interest (Chapter 5) No relationship, hypothesis of  Claim that one or more IVs has no (usually statistically) discernible effect on another; implies that the coefficient is not statistically significant (Chapter 4) Nominal  Level of measurement: unrankable but discrete categories, with no implied direction or magnitude; lowest precision of measurement (Chapter 4, 6) Normative  Type of research question: focuses on what should happen (Chapter 1) Not-for-attribution  Level of interview confidentiality not allowing any direct reference to an interview (or interview subject) as the source; all information gathered at this level must be triangulated (Chapter 6) MLA  Modern Language Association (organization and citation format) Observable implications  Empirical patterns that should emerge if a theory is correct (Chapter 2)

275

276

Empirical Research and Writing

Observation  A single instance of the phenomenon under investigation (Chapter 6, 7) OLS  Ordinary least squares regression; see Regression (Chapter 4) Omitted variable bias (OVB)  Incorrect estimates of relationships (qualitative or quantitative) resulting from failure to consider a relevant variable (Chapter 8) Ontology  Beliefs about the nature of being: in social sciences, how the world is constituted (Chapter 2) Operationalization  Process of identifying a valid observable indicator for the concepts expressed in a theory (Chapter 2, 4, 7) Ordered logit  Quantitative tool for analyzing DVs with three or more ordinal categories; functionally identical to ordered probit (Chapter 4) Ordered probit  Quantitative tool for analyzing DVs with three or more ordinal categories; functionally identical to ordered logit (Chapter 4) Ordinal  Level of measurement: rankable categories, where the intervals between categories may or may not be equal or precisely definable; intermediate level of measurement (Chapter 4, 6) Ordinary least squares regression See Regression (Chapter 4) OVB See Omitted variable bias (Chapter 8) Panel data  Multiple units observed at multiple points in time; also called time series cross section data (Chapter 7) Parsimony  Characteristic of a theory: explains more while using less (Chapter 1) Passive voice  Grammatical structure in which sentence’s actor is the object of an action rather than the instigator of it (Chapter 10) Peer review process (double-blind)  Procedure by which scholars evaluate each other’s research for rigor and completeness prior to publishing; acts as a gatekeeping device for professional publications (Chapter 3, 10) Polychotomous  Describes an indicator or variable that takes on three or more unordered values (i.e., is at the nominal level of measurement) (Chapter 4) Polychotomous probit  Quantitative tool for analyzing DVs with three or more nominal categories; functionally identical to multinomial logit (Chapter 4) Population  The complete set of relevant cases for a theory (Chapter 7) Population  The set of all cases relevant to a theory (Chapter 5) Possibility principle  Guideline for choosing negative cases for qualitative analysis: at least one IV takes a value that theory claims is crucial for the DV to occur, and no IVs predict against the outcome of interest (Chapter 5) Predictiveness  Characteristic of a theory: theory helps explain cases other than those from which it was derived (Chapter 1) Pretesting  Evaluating coding instruments, including dictionaries, against a small sample of cases or sources prior to beginning full-scale data collection (Chapter 6)

Glossary

Primary source  Qualitative data source characteristic: no analysis separates the researcher from the source’s creator (Chapter 6) Probabilistic hypothesis  Class of hypotheses arguing that a relationship holds across a pool of cases even though individual cases may not support the relationship; four main types are directional, relative, no relationship, and conditional (interactive) (Chapter 4) Probit  Common tool, with logit, for quantitative analysis of dichotomous DV data; test statistic is significance of coefficients though interpretation of coefficients is more complex than in regression (Chapter 4) Process tracing  Within-case tool for qualitative analysis of causal mechanisms (Chapter 4) Proximate causes  Immediate and direct trigger of outcome of interest (Chapter 4) Purposive sample  A deliberately selected subset of cases chosen for their values on particular key variables (Chapter 6) QCA See Qualitative Comparative Analysis (Chapter 4) Qualitative Comparative Analysis (QCA)  Analytical technique for qualitative data using Boolean logic to test propositions of necessity and/or sufficiency; requires variables to be fully in or out of a set (i.e., all variables must be dichotomous) (Chapter 4) R&R See Revise and resubmit (Chapter 10) R2  Goodness of fit measure for bivariate OLS regression; interprets as percentage of DV variation explained by variations in IV (Chapter 8) Recoding  Applying a new scale or coding scheme to existing data to change its level of measurement or other characteristics (Chapter 8) Regression  Workhorse tool of quantitative analysis; requires interval-ratio level DV but IVs may be any level of analysis; test statistic is significance of coefficients (Chapter 4) Relative hypothesis  Claim comparing the magnitude of effect of two or more independent variables (Chapter 4) Reliability  Characteristic of a measurement tool: produces values that are consistent across cases and applications (Chapter 6, 7) Replicability   Characteristic of research: sufficient transparency in research practices and reporting to allow another researchers to re-create our analysis (Chapter 1) Research program  Set of related research questions often drawing on the same concepts and theories (Chapter 3) Research question  A bounded statement of the phenomenon under investigation, usually focused on explaining variation in an outcome (Chapter 1) Research topic  An unbounded statement of the phenomenon of interest to the researcher (Chapter 1)

277

278

Empirical Research and Writing

Revise and resubmit (R&R)  Journal or other publication decision returning a paper to its author for significant revision and often re-review as a precursor to acceptance for publication (Chapter 10) Rhetorical questions  Questions posed by authors with no intention that the reader will respond; cheap transition device worth avoiding (Chapter 10) Robustness checks  Additional model specifications estimated using alternate indicators of key concepts to determine that findings hold across different operationalizations of those concepts (Chapter 7, 9) Running record  Qualitative data source: reports produced on a systematic and recurrent basis: hourly, daily, annually, etc. (Chapter 6) Sample  Subset of the population obtained/used for analysis (Chapter 7) Sample  A subset of the population (Chapter 5) Scale  Tool of data reduction: combines multiple indicators into a single measure, usually by summation (Chapter 8) Scooping  Exploiting someone else’s data by publishing analysis based on it before the collector is able to do so; considered very inappropriate professional behavior (Chapter 7) Scope conditions  Part of a theory: defines the domain of the theory or any other restrictions or boundaries on what cases the theory should be expected to explain (Chapter 2) Secondary source  Qualitative data source characteristic: one layer of analysis separates the researcher from the source’s creator (Chapter 6) Selection bias  The result of analyzing data that suffer from a selection effect (Chapter 5, 7) Selection effect   Natural or man-made processes produce an observed sample that is a biased subset of the underlying population; all cases do not have an equal chance of entering the observed sample (Chapter 5, 7) Sensitivity tests See Robustness checks (Chapter 10) SFC See Structured focused comparison (Chapter 4) Simultaneity  Special case of endogeneity in which the DV causes one or more IVs; requires deployment of appropriate fixes to estimate relationships (Chapter 8) Simultaneity bias  Incorrect coefficients (or qualitative relationship estimates) obtained from neglecting to consider the effect of Y on X as well as the effect of X on Y (Chapter 8) Snowball sample  Sampling procedure in which interviewees are asked to name other relevant individuals to interview, who are then interviewed and asked to name other individuals, etc. (Chapter 6) Soaking and poking  Inductive qualitative research technique involving deep immersion in a social context (Chapter 5)

Glossary

Social Sciences Citation Index (SSCI)  Tool for discovering work citing central or prominent articles, forward in time from the starting piece; sometimes called Web of Science (Chapter 3) Spearman’s rho  Test statistic for strength and direction of association between ordinal variables (Chapter 4) Special collections  Library resources on particular topics, often including archival and nontextual material; special collections typically do not circulate (Chapter 6) Specification  Particular combination of variables included in a statistical model (Chapter 9) SSCI  Social Sciences Citation Index (Chapter 3) Structured focused comparison  Between-case tool for analysis of qualitative data using an implicit regression model; variable focused and usually uses paired cases (Chapter 4) Sufficient condition  Asserts that some cause X always leads to the occurrence of Y; the absence of X may or may not result in Y (Chapter 4) Tertiary source  Qualitative data source characteristic: two or more layers of analysis separate the researcher from the source’s creator(s) (Chapter 6) Text mining  Process of combing texts to count items or identify passages for content analysis (Chapter 6) Theory  A reasoned speculation of the answer to a research question, usually phrased in terms of concepts; includes the expectation, a causal mechanism, assumptions, and scope conditions(Chapter 2) Theory family  Cluster of related answers to a research question; typically a subset of a scholarly literature (Chapter 3) Transformation  Mathematical altering of the scale of a variable to create a more linear relationship Triangulation  Reinforcing conclusions drawn from (primarily) qualitative data by deploying findings or evidence from other types of data or sources (Chapter 6) Underlying causes  Characteristics of the environment that facilitate or contribute to the occurrence of some outcome of interest (Chapter 6) Unit of analysis  The item that constitutes one observation in a quantitative study: decision, individual opinion, country, dyad, state-year, etc. (Chapter 6) Validity  Characteristic of an indicator: the indicator captures the concept of interest and nothing else (Chapter 2, 4, 6, 7) Variable  Name of the column in your dataset; shortest and most abbreviated representation of the data contents (Example in Table 8.1) (Chapter 8) Variable label  Longest form of variable description, used in tables and graphs (Example in Table 8.1) (Chapter 8)

279

280

Empirical Research and Writing

Variable name  Form of data content referenced in textual discussion and listed in results table; uses words instead of abbreviations (Example in Table 8.1) (Chapter 8) Venn diagram  Graphical organizer used to show overlap between sets (Chapter 2) Working paper  Unpublished manuscript, usually still being worked on by the author (Chapter 3)

R e f e r e n c e s

Achen, Christopher H., and Duncan Snidal. 1989. “Rational Deterrence Theory and Comparative Case Studies.” World Politics 41(1): 80–122. Allison, Graham T. 1969. “Conceptual Models and the Cuban Missile Crisis.” American Political Science Review 63(3): 689–718. Andersen, Robert, and Tina Fetner. 2008. “Economic Inequality and Intolerance: Attitudes toward Homosexuality in 35 Democracies.” American Journal of Political Science 52(4): 942–58. Anderson, Benedict. 1983. Imagined Communities: Reflections on the Origin and Spread of Nationalism. London: Verso/New Left. Art, David. 2011. Inside the Radical Right: The Development of Anti-Immigrant Parties in Western Europe. Cambridge [UK]: Cambridge University Press. Babbie, Earl R. 2012. The Practice of Social Research. 13th ed. Belmont [CA]: Cengage. Banks, Antoine. 2014. Anger and Racial Politics: The Emotional Foundations of Racial Attitudes in America. Cambridge [UK]: Cambridge University Press. Bates, Robert H., Avner Greif, Margaret Levi, Jean-Laurent Rosenthal, and Barry R. Weingast. 1998. Analytic Narratives. Princeton [NJ]: Princeton University Press. Bates, Robert, Avner Greif, Margaret Levi, Jean-Laurent Rosenthal, and Barry Weingast. 2000. “The Analytical Narrative Project.” American Political Science Review 94(3): 696–702. Bennett, Andrew, and Colin Elman. 2006. “Qualitative Research: Recent Developments in Case Study Methods.” Annual Review of Political Science 9: 455–76. Bennett, Andrew, and Alexander L. George. 1997. “Process Tracing in Case Study Research.” MacArthur Foundation Workshop on Case Study Methods, Harvard University. Brace, Paul, and Brent D. Boyea. 2008. “State Public Opinion, the Death Penalty, and the Practice of Electing Judges.” American Journal of Political Science 52(2): 360–72. Brady, Henry E., and David Collier, eds. 2004. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Lanham [MD]: Rowman and Littlefield. Braumoeller, Bear, and Gary Goertz. 2000. “The Methodology of Necessary Conditions.” American Political Science Review 94(4): 844–58. Budge, Ian, Hans-Dieter Klingemann, Andrea Volkens, Judith Bara, and Eric Tanenbaum. 2001. Mapping Policy Preferences: Estimates for Parties, Electors and Governments, 1945–98. Oxford [UK]: Oxford University Press. Bueno de Mesquita, Bruce, Alistair Smith, Randolph Siverson, and James Morrow. 2004. The Logic of Political Survival. Cambridge [MA]: MIT Press. Bunce, Valerie. 1985. “The Empire Strikes Back: The Evolution of the Eastern Bloc from a Soviet Asset to a Soviet Liability.” International Organization 39(1): 1–46. Burnham, Peter, Karen Gilland, Wyn Grant, and Zig Layton-Henry. 2004. Research Methods in Politics. New York: Palgrave Macmillan.



281

282

Empirical Research and Writing

Campbell, Donald T. 1975. “‘Degrees of Freedom’” and the Case Study.” Comparative Political Studies 8,2: 178-93. Campbell, Steven J. 2011. “The Worldview of Carter’s National Security Adviser: Assessing the Controversy over Brzezinski’s Foreign Policy Beliefs.” Paper presented at the Annual Meeting of the Northeastern Political Science Association, Philadelphia, PA. Checkel, Jeffrey. 1999. “Norms, Institutions, and National Identity in Contemporary Europe.” International Studies Quarterly 43(1): 83–114. Checkel, Jeffrey. 2006. “Tracing Causal Mechanisms.” International Studies Review 8(2): 362–70. Christiansen, Thomas, and Jack Snyder. 1990. “Chain Gangs and Passed Bucks: Predicting Alliance Patterns in Multipolarity.” International Organization 44(2): 137–68. Clark, William Roberts, Matt Golder, and Sona Nadenichek Golder. 2008. Principles of Comparative Politics. 1st ed. Washington, DC: CQ Press. Coates, Michael, and Shanna Pearson-Merkowitz. 2014. “Guns of Fortune: State Gun Control Policies and the Laws of Supply and Demand.” State Policy and Politics Conference, Bloomington, IN. http://sppc2014.indiana.edu/Papers/Coates%20 and%20Pearson%20Merkowitz.pdf (Accessed 23 July 2014). Collier, David, and Steven Levitsky. 1997. “Democracy with Adjectives: Conceptual Innovation in Comparative Research.” World Politics 49(3): 430–51. Collier, David, and James Mahoney. 1996. “Insights and Pit-falls: Selection Bias in Qualitative Research.” World Politics 49(1): 56–91. Cortina, Jose M. 1993. “What Is Coefficient Alpha? An Examination of Theory and Applications.” Journal of Applied Psychology 78(1): 98–104. Croco, Sarah E. 2011. “The Decider’s Dilemma: Leader Culpability, War Outcomes, and Domestic Punishment.” American Political Science Review 105(3): 457–77. Doyle, Michael. 1986. “Liberalism and World Politics.” American Political Science Review 80(4): 1151–61 Drezner, Daniel. 2003. “The Hidden Hand of Economic Coercion.” International Organization 57(3): 643–59. Elder, Glen H., Eliza K. Pavalko, and Elizabeth Colerick Clipp. 1992. “Working with Archival Data: Studying Lives.” Quantitative Applications in the Social Sciences. Thousand Oaks [CA]: Sage. Elman, Colin, and Miriam Fendius Elman. 2002. “How Not to Be Lakatos Intolerant: Appraising Progress in IR Research.” International Studies Quarterly 46(2): 231–62. Elo, Satu, and Kyngäs, Helvi. 2008. “The qualitative content analysis process.” Journal of Advanced Nursing 62(1): 107–115. http://academic.csuohio.edu/ kneuendorf/c63309/ArticlesFromClassMembers/Amy.pdf (Accessed 6 Jan 2013). Elster, Jon. 2000. “Rational Choice History: A Case of Excessive Ambition.” American Political Science Review 94(3): 685–95. Escalada, Monina. 2008. “How to Do Content Analysis: A Step-by-Step Guide.” Devcompage.com. http://devcompage.com/?p=2862 (Accessed 20 July 2014). Falleti, Tulia G. 2005. “A Sequential Theory of Decentralization: Latin American Cases in Comparative Perspective.” American Political Science Review 99(3): 327–46. Fenno, Richard F. 1978. Home Style: House Members in Their Districts. New York: Little, Brown. Geddes, Barbara. 2003. Paradigms and Sand Castles: Theory Building and Research Design in Comparative Politics. Ann Arbor: University of Michigan Press.

References

General Accounting Office (GAO). 1989. Content Analysis: A Methodology for Structuring and Analyzing Written Material. Transfer Paper 10.1.3. http://archive.gao.gov/ d48t13/138426.pdf (Accessed 18 Feb 2014). George, Alexander L., and Andrew Bennett. 2005. Case Studies and Theory Development in the Social Sciences. Cambridge [MA]: MIT Press. George, Darren, and Paul Mallery. 2003. SPSS for Windows Step by Step: A Simple Guide and Reference. 11.0 update (4th ed.). Boston: Allyn and Bacon. Gerring, John. 2004. “What Is a Case Study and What Is It Good For?” American Political Science Review 98(2): 341–54. Gilens, Martin. 1999. Why Americans Hate Welfare: Race, Media, and the Politics of Antipoverty Policy. Chicago: University of Chicago Press. Gilligan, Michael. 2004. “Is There a Broader-Deeper Tradeoff in International Multilateral Agreements?” International Organization 58(3): 459–84. Gliem, Joseph A., and Rosemary R. Gliem. 2003. “Calculating, Interpreting, and Reporting Cronbach’s Alpha Reliability Coefficient for Likert-Type Scales.” Proceedings of the 2003 Midwest Research to Practice Conference in Adult, Continuing and Community Education, pp. 82–88. https://scholarworks.iupui.edu/bitstream/ handle/1805/344/Gliem+&+Gliem.pdf?sequence=1 (Accessed 12 Jan 2014). Gowa, Joanne. 1999. Ballots and Bullets: The Elusive Democratic Peace. Princeton [NJ]: Princeton University Press. Hacker, Jacob S. 2004. “Privatizing Risk without Privatizing the Welfare State: The Hidden Politics of Social Policy Retrenchment in the United States.” American Political Science Review 98(2): 243–60. Hancock, Dawson R., and Bob Algozzine. 2006. Doing Case Study Research. New York: Teachers College Press/Columbia University. Healey, Joseph F. 2009. The Essentials of Statistics: A Tool for Social Research. Belmont [CA]: Wadsworth/Cengage. Hempel, Carl. 1966. Philosophy of Natural Science. Englewood Cliffs [NJ]: PrenticeHall. Hopkins, Daniel, and Gary King. 2010. “A Method of Automated Nonparametric Content Analysis for Social Science.” American Journal of Political Science 54(1): 229–47. Huffbauer, Gary Clyde, Jeffrey J. Schott, and Kimberly Ann Elliott. 1990. Economic Sanctions Reconsidered: History and Current Policy. 2nd ed. Washington, DC: Institute for International Economics. Inglehart, Ronald F., and Pippa Norris. 2003. Rising Tide: Gender Equality and Culture Change around the World. Cambridge [UK]: Cambridge University Press. Inglehart, Ronald F., and Pippa Norris. 2004. Sacred and Secular: Religion and Politics Worldwide. Cambridge [UK]: Cambridge University Press. Inglehart, Ronald F., and Christian Welzel. 2005. Modernization, Cultural Change, and Democracy. Cambridge [UK]: Cambridge University Press. Ishiyama, John. 2013. “Report of the Editors of the American Political Science Review, 2011–2012.” PS: Political Science and Politics 107(2): 419–22. Jackson, Patrick Thaddeus. 2010. The Conduct of Inquiry in International Relations: Philosophy of Science and Its Implications for the Study of World Politics. New York: Routledge. Jacobson, Gary C. 1980. Money in Congressional Elections. New Haven [CT]: Yale University Press

283

284

Empirical Research and Writing

Janda, Kenneth. 2003. “‘Things You Should Have Learned in School’: Scale Construction.” http://janda.org/workshop/scale%20construction/Talk/talk03.htm (Accessed 29 Aug 2013). Jervis, Robert. 1976. Perception and Misperception in International Politics. Princeton [NJ]: Princeton University Press. Johnson, Janet Buttolph, and H.T. Reynolds. 2014. Political Science Research Methods. 7th ed. Washington, DC: Sage/CQ Press. Keck, Margaret E., and Kathryn Sikkink. 1998. Activists beyond Borders: Advocacy Networks in International Politics. Ithaca [NY]: Cornell University Press. Kellstedt, Paul M., and Guy D. Whitten. 2009. The Fundamentals of Political Science Research. Cambridge [UK]: Cambridge University Press. Kennedy, Peter. 2008. A Guide to Econometrics. 6th edition. Oxford [UK]: Blackwell. Kim, Hun Joon, and J. C. Sharman. 2014. “Accounts and Accountability: Corruption, Human Rights and Individual Accountability Norms.” International Organization 68(2): 417–48. King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry. Princeton [NJ]: Princeton University Press. Kirschner, Shanna A. (Forthcoming). Trust and Fear in Civil Wars: Ending Intrastate Conflicts. Lexington Books. Klingemann, Hans-Dieter, Andrea Volkens, Judith Bara, Ian Budge, and Michael MacDonald. 2006. Mapping Policy Preferences II. Estimates for Parties, Electors, and Governments in Eastern Europe, the European Union and the OECD, 1990–2003. Oxford [UK]: Oxford University Press. Koremenos, Barbara, Charles Lipson, and Duncan Snidal. 2001. “The Rational Design of International Institutions.” International Organization 55(4): 761–99. Krasner, Stephen D. 1972. “Are Bureaucracies Important? (Or Allison Wonderland).” Foreign Policy 7: 159–79. Levi, Margaret. 1999. “Producing an Analytic Narrative.” In Critical Comparisons in Politics and Culture, eds. John Bowen and Roger Petersen. Cambridge [UK]: Cambridge University Press, 152–72. Levi, Margaret. 1997. Consent, Dissent, and Patriotism. Cambridge [UK]: Cambridge University Press. Lewis, George H., and Jonathan F. Lewis. 1980. “The Dog in the Night-Time: Negative Evidence in Social Research.” British Journal of Sociology 31(4): 544–58. Lin, Ann Chih. 2002. Reform in the Making: The Implementation of Social Policy in Prison. Princeton: Princeton University Press. Louderback, Whitney E. 2009. “The Implementation of Negotiated Settlements after Intrastate Wars.” BA Thesis, College of Wooster, Wooster, OH. http://www3 .wooster.edu/polisci/Louderback.html (Accessed 18 Feb 2014). Lustic, Ian S. 1996. “History, Historiography, and Political Science: Multiple Historical Records and the Problem of Selection Bias.” American Political Science Review 90(3): 605–618. Mahoney, James. 1999. “Nominal, Ordinal, and Narrative Appraisal in Macrocausal Analysis.” American Journal of Sociology 104(4): 1154–96. Mahoney, James. 2010. “After KKV: The New Methodology of Qualitative Research.” World Politics 62(1): 120–47. Mahoney, James, and Gary Goertz. 2004. “The Possibility Principle: Choosing Negative Cases in Qualitative Research.” American Political Science Review 98(4): 653–70.

References

Manheim, Jarol B., Richard C. Rich, Lars Willnat, and Craig Leonard Brians. 2008. Empirical Political Analysis: Research Methods in Political Science. 7th ed. New York: Pearson Longman.  Mill, John Stuart. 1843. A System of Logic. London: John W. Parker. See Google Books, http://books.google.com/books?id=y4MEAAAAQAAJ. Milner, Helen V. 1997. Interests, Institutions, and Information. Princeton [NJ]: Princeton University Press. Morrow, James D. 1993. “Arms versus Allies: Trade-offs in the Search for Security.” International Organization 47(2): 207–33. Morrow, James D. 1994. “Modeling the Forms of International Cooperation: Coordination and Distribution versus Information.” International Organization 48(2): 387– 423. Morrow, James D. 2007. “When Do States Follow the Laws of War?” American Political Science Review 101(3): 559–72. Morrow, James D. 2007. “Officers King and Zeng and the Case of the Unsupported Counterfactual.” International Studies Quarterly 50(2): 227–29. Morrow, James D., and Hyeran Jo. 2006. “Compliance with the Laws of War: Dataset and Coding Rules.” Conflict Management and Peace Science 23: 91–113. Mosteller, Frederick, and David L. Wallace. 1964. Inference and Disputed Authorship: The Federalist. Reading [MA]: Addison-Wesley. Murillo, Victoria, and Andrew Scrhank. 2005. “With a Little Help from My Friends: Partisan Politics, Transnational Alliances, and Labor Rights in Latin America.” Comparative Political Studies 30: 971–99. Myers, D. G. 2012. Social Psychology. 11th ed. New York: McGraw-Hill. Obenzinger, Hilton. 2005. “‘What Can a Literature Review Do for Me?’ How to Research, Write, and Survive a Literature Review.” http://libguides.csufresno.edu/ literaturereview (Accessed 1 May 2014). Odell, John. 2009. “Deadlocks in International Institutional Negotiations.” International Studies Quarterly 53(2): 273–300. Organski, A.F.K., and Jacek Kugler. 1981. The War Ledger. Chicago: University of Chicago Press. Pollock III, Phillip H. 2012. The Essentials of Political Analysis. 4th ed. Washington, DC: CQ Press/Sage. Powner, Leanne C. 2007. Reading and Understanding Political Science. Unpublished manuscript. http://jayandleanne.com/lpowner/tchdocs/readingps.pdf (Accessed 24 Jul 2014). Powner, Leanne C. 2008. Consensus, Capacity, and the Choice to Cooperate. Ph.D. Dissertation: University of Michigan, Ann Arbor. Putnam, Robert. 1994. Making Democracy Work: Civic Traditions in Modern Italy. Princeton [NJ]: Princeton University Press. Przeworski, Adam, and Henry Teune. 1970. The Logic of Comparative Political Inquiry. New York: Wiley. Ragin, Charles C. 1987. The Comparative Method: Moving beyond Qualitative and Quantitative Strategies. Berkeley: University of California Press. Ragin, Charles C. 2000. Fuzzy-Set Social Science. Chicago: University of Chicago Press. Rodrik, Dani, ed. 2003. In Search of Prosperity: Analytic Narratives on Economic Growth. Princeton [NJ]: Princeton University Press. Rogowski, Ronald, and Daniel Triesman. 2009. “Report of the Editors of the American Political Science Review.” PS: Politcal Science and Politics 42(2): 426–28.

285

286

Empirical Research and Writing

Rudman, Joseph. 2012. “The Twelve Disputed ‘Federalist’ Papers: A Case for Collaboration.” In Proceedings of Digital Humanities 2012, University of Hamburg, Germany, 16–20 Jul 2012. http://www.dh2012.uni-hamburg.de/conference/programme/ abstracts/the-twelve-disputed-federalist-papers-a-case-for-collaboration/ (Accessed 18 Feb 2014). Salmond, Rob, and David T. Smith. 2011. “Cheating Death-by-PowerPoint: Effective Use of Visual Aids at Professional Conference,” PS: Political Science & Politics 44,3: 589–96. Schultz, Kenneth. 1998. “Domestic Opposition and Signaling in International Crises.” American Political Science Review 92(4): 829–44. Schmidt, Diane E. 2010. Writing in Political Science: A Practical Guide. 4e. Upper Saddle River [NJ]: Pearson-Longman. Sekhon, Jasjeet S. 2004. “Quality Meets Quantity: Case Studies, Conditional Probability, and Counterfactuals.” Perspectives on Politics 2(2): 281–93. Shepsle, Kenneth A., and Mark S. Bonchek. 1997. Analyzing Politics: Rationality, Behavior, and Institutions. New York: W. W. Norton. Signorino, Curtis S., and Jeffrey M. Ritter. 1999. “Tau-b or Not Tau-b: Measuring the Similarity of Foreign Policy Positions.” International Studies Quarterly 43(1): 115–44. Simmons, Beth A. 2000. “International Law and State Behavior: Commitment and Compliance in International Monetary Affairs.” American Political Science Review 94(4): 819–35. Simmons, Beth A., and Daniel J. Hopkins. 2005. “The Constraining Power of International Treaties: Theory and Methods.” American Political Science Review 94(4): 623–31. Skocpol, Theda. 1979. States and Social Revolutions: A Comparative Analysis of France, Russia, and China. Cambridge [UK]: Cambridge University Press. Smith, David T., and Rob Salmond. 2011. “Verbal Sticks and Rhetorical Stones: Improving Conference Presentations in Political Science,” PS: Political Science & Politics 44,3: 583–88. Sommer, Barbara. n.d. “Doing a Content Analysis.” http://psychology.ucdavis.edu/ faculty_sites/sommerb/sommerdemo/content/doing.htm (Accessed 18 Feb 2014). Sommer, Robert, and Barbara Sommer. 2001. A Practical Guide to Behavioral Research: Tools and Techniques. 5th ed. Oxford [UK]: Oxford University Press. Stam, Allan C. 1996. Win, Lose, or Draw: Domestic Politics and the Crucible of War. Ann Arbor: University of Michigan Press. Teodoro, Manuel P. 2011. Bureaucratic Ambition: Careers, Motives, and the Innovative Administrator. Baltimore: Johns Hopkins University Press. Trachtenberg, Marc. 2006. The Craft of International History: A Guide to Method. Princeton [NJ]: Princeton University Press. Tsai, Lily L. 2007. “Solidary Groups, Informal Accountability, and Local Public Goods Provision in Rural China.”American Political Science Review 101(2): 355–72. Tsebelis, George. 2002. Veto Players: How Political Institutions Work. Princeton [NJ]: Princeton University Press. Valentino, Nicholas A., Vincent L. Hutchings, and Ismail K. White. 2002. “Cues That Matter: How Political Ads Prime Racial Attitudes during Campaigns.” American Journal of Political Science 96(1): 75–90.

References

Van Doorn, Bas W. 2012. “Media Portrayals of Poverty and Race in Pre- and PostWelfare Reform America.” Presented at the 2012 Annual Meeting of the Western Political Science Association, Portland, OR. http://wpsa.research.pdx.edu/ meet/2012/vandoorn.PDF (Accessed 20 July 2014). Van Evera, Stephen. 1997. Guide to Methods for Students of Political Science. Ithaca [NY]: Cornell University Press. Von Stein, Jana. 2005. “Do Treaties Constrain or Screen? Selection Bias and Treaty Compliance.” American Political Science Review 99(4): 611–22. Waltz, Kenneth. 1979. Theory of International Politics. Reading [MA]: Addison-Wesley. Weber, Robert Philip. 1990. “Basic Content Analysis.” Quantitative Applications in the Social Sciences. Thousand Oaks [CA]: Sage. Weinstein, Jeremy M. 2007. Inside Rebellion: The Politics of Insurgent Violence. Cambridge [UK]: Cambridge University Press. Widner, Jennifer A. 2008. “Constitution Writing in Post-conflict Settings,” William & Mary Law Review 49(4). http://scholarship.law.wm.edu/wmlr/vol49/iss4/16 (Accessed 11 Mar 2014). Writing@CSU. n.d. “Content Analysis.” http://writing.colostate.edu/guides/guide .cfm?guideid=61 (Accessed 20 July 2014).

287

i n d e x

Note: Figures, previews, tables, and notes are indicated by f, p, t, or n following the page number. Abstracts, 207–208 Actors in theories, 42–43 Adjusted R², 192n18, 214, 214n8 Allendoerfer, M., 257 American National Elections Studies (ANES), 49 American Political Science Review (APSR), 59n5 Analysis section. See Data analysis section Analytic narratives, 104, 122–124 Animation, use of, 258n14 Annotated bibliographies, 64–65 Annotations, 64–65 Annual Meetings of the American Political Science Association, 61–62 Annual Review of Political Science (ARPS), 10, 10n12, 60 APA format, 65, 65n12 APSA format, 65, 65n12 Archival research data archives, 145–146, 145nn7–8, 174 replication datasets, 173, 173n11 special collections, 152–153 ArchiveGrid, 152–153 Arndt, L., 230 Arrow diagrams, 31, 38 Articles versus books, 58, 58n3 Association journals, 59n5 Assumptions, 23–24, 42–43, 53 Asterisks, 213, 213n6 Attribution, 11, 150–151 Audits of writing process, 224 Background knowledge, 6, 218, 234 Background research, 58, 115–116 Backups, 176n17, 177, 200n22

Baseline models, 215 Battlebots, 70 Beads-on-a-string approach, 57, 57n2, 63, 70 Between-case designs, 105–106, 105n22, 112 Bias omitted variable bias, 164n6, 203–204 range of cases limitations and, 13 simultaneity bias, 201 snowball samples, 149 See also Selection bias Bibliographies, 64–65, 248, 255 Bibliography-hopping, 61, 144 BibTex, 66t, 67 Bivariate relationships, 101 Blind review, 264 Bookends, 209–210 Book reviews, 61n9 Books versus articles, 58, 58n3 Boolean algebra, 107n25 Braces, 226 Brackets, 226 Brainstorming of research questions, 17p Brilliance, 28 Bush, George H. W., 42n17 Bush, George W., 42n17 Candidate spending, 37 Captions, 211 Case control method, 105, 105n22, 124–129, 127nn14–15, 218 Cases, what constitutes, 136 Case selection analytic narratives, 123–124



289

290

Empirical Research and Writing

case control method, 125–129, 125n12, 127nn14–15 cherry picking data, 96, 107 content analysis, 119–122 peer review about, 241 process tracing, 130–132 qualitative design, 112–118 qualitative versus quantitative, 95n10 quantitative design, 161–162 structured focused comparison method, 129–130 writing about, 133 Causality, 96t Causal mechanisms, 23, 36–39, 39–40p, 45, 52 Causal process observations (CPOs), 131 Causes of effects versus effects of causes, 97 Causes versus outcomes, 7, 7n9 Certainty versus uncertainty, 77 Ceteris paribus assumption, 163 Cheating Death by PowerPoint, 258 Cherry picking data, 96, 107 Chi-squared value, 100–101, 217 CIAO (Columbia International Affairs Online), 60 Citations, 62, 62n10, 174–176 Cleveland Browns fans, 37n12 Clinton, Bill, 42n17 Coauthoring, 263 Codebooks, 172–173 Coding of data defined, 171, 177 inter-coder reliability, 147n10, 171 quantitative design, 177–179 recoding variables, 198–200, 200p text mining and, 147, 147nn10–11 Coefficients, 102, 102n19, 162, 212–215 Colinearity, 190–192 Collegiality, 246 Colloquialisms, 233 Color, use of, 256 Commas, 229 Comparative method. See Case control method Comparative politics, domains and, 25 Composite variables, 192–200, 194–196t, 196p, 200p Concepts, 8, 48–51

Concept webs, 71 Conclusion section overview, 56, 210 peer review about, 242 for posters, 255 for presentations, 248 Conditional hypotheses, 88–89 Conferences, 260–263, 262n15 Confidentiality, 150–151, 172n9 Confounding variables, 203 Constants, 47–48, 113, 212, 212n4 Content analysis, 106, 118–122, 118n6, 146n9 Contractions, 233 Controlled comparison method. See Case control method Control variables (CV), 112–113, 162–164, 164–165p Conway, A., 190 Correlates of War (COW), 168–169 Correlation matrices, 190 Corresponding authors, 173, 173n12 Count analysis, 118–119 Counterexamples, 47 Counterexplanations, 47 Counterfactuals, 141–142, 218 CPOs (causal process observations), 131 Croco, S., 250 Cronbach’s Alpha, 193 Cross tabulation, 100–101 Curriculum vitae, 56n1 Data, information compared, 136–137 Data analysis, quantitative composite variables, 193–200, 194–196t, 196p, 200p endogeneity, 201–203 missing data, 183–185 nonlinear models, 204–205 omitted variable bias, 164n6, 203–204 preparation for, 185–192, 188–189f preservation of original data, 199– 200, 199n21 recoding variables, 198–200, 200p simultaneity, 201 transferring data to stats programs, 181–183 Data analysis section overview, 56

Index 291

qualitative design, 217–221, 220n11 quantitative design, 210–217 Data archives, 174 Data collection, qualitative counterfactuals, 141–142 human subjects research, 148–151, 148n13 leverage from, 139–142 measurement, 137–139 quantitative compared, 141 research from sources, 144–147 resources for, 152–153 sources of, 140–141 writing about, 155–156 Data collection, quantitative overview, 157 concept-indicator mismatches, 178–179 forms for, 178, 178n19 need identification, 157–162, 158–159p novel data, 175–176, 175nn15–16 qualitative compared, 141 ready-to-use data, 171–175 variables, 162–167, 164–165p Data collection forms, 178, 178n19 Data management, 153–155, 175–177, 176n17. See also Coding of data Data points, 136n2 Data preservation, 199–200, 199n21 Dataset observations (DSOs), 131 Datasets, 136, 174–175 Data sources, 12–15 Davis, B., 150 Decision makers in theories, 42–43 Deductive theorizing, 26, 26t, 27n2 Degrees of freedom, 110–112, 110n1 Déjà Vu All Over Again, 70 Democratic peace, 10n11, 38, 87 Demographics, 37 Dependent (outcome) variables, 44–45 Design. See Research design Designing Social Inquiry (King, Keohane & Verba), 5n5, 22, 28, 97n12 Deterministic hypothesis types, 89–92, 90–91t, 92–93p Dichotomous variables, 84, 89, 204 Dictionaries, 146–147 Difference of means test, 216

Direct relationship, 39 Directional hypotheses, 82–84, 84–85p, 86 Direct quotations, 218–219, 249 Discussants, 245, 251–252 Document analysis, 144 Document identification numbers (DOI), 176n18 Domains, 24–25, 157 Double-blind, 59 Drafts, 73, 73nn21–22 Dress codes, 246, 253 Dropbox, 176n17, 200n22 DSOs (dataset observations), 131 Dummy dependent variables, 102–103 Editing literature review section, 77–80 methods for, 226–234 peer review compared, 236–237, 239 redrafting compared, 73 Effects of causes versus causes of effects, 97 Elections 1992 elections, 42n17 2000 elections, 31–35, 31n7, 32f, 34t, 36t, 42n17 2004 elections, 31–35, 31n7, 32f, 34t, 35, 36t Electoral College, 31n7, 32 Electronic bibliography management tools, 65–68, 66t Electronic collections, 152–153 Elite interviewing, 148–151 Ellipses, 249 Empirical papers overview, 55–56, 207 abstracts, 207–208 analysis section, 210–217 conclusion section, 56, 210 data collection section, 155–156 introduction section, 55, 209–210 length of, 6, 6n6 methodology section, 132–134, 133nn21–22 research design, 155–156 theory section, 51–53, 55–56

292

Empirical Research and Writing

Empirical questions, 4, 4n3 Empirical research, xiii–xiv, 4–5 EndNote, 66t, 67 Endogeneity, 201–203 Environment and economic development, 38–39, 39n15 Episodic records, 140 Epstein, P., 219 Equal probability of selection method (EPSEM), 158 Error bars, 259, 260f Etiquette, prepublished works, 62, 62n10 EuroBarometer survey, 14 EverNote, 154–155 Evidence data analysis section, 217–219 data and, 136–137 hypotheses and, 46–48 peer review about, 241–242 for posters, 254 for presentations, 247–248 what constitutes, 5 Excel, 182, 259 Excisions, 249 Executive summaries, 239 Exhaustive and mutually exclusive values, 100n17 Expectations/predictions, 23 Experiments, 26–27 Eyeball Switch, 70 Factual/procedural questions, 4 Falsifiability, 3 Falsifiers, 47, 218 Feedback. See Peer review process; Poster sessions; Presentations Fertility, 3, 11 Figures and tables, 211, 257 Finding aids, 144–145 First person language, 208 Fixed effects (FE), 203–204 Fonts, 229 Formal modeling, 104n21 Foundational works, 56 Funding, 261 Fuzzy Set Analysis (FSA), 106–107 Gamma statistic, 101 Garbage can models, 164, 192

Gender, 49, 49n20, 168n7 Generalizability, 96t, 113 Geography, limits of, 14 Gerring, J., 99 Goodness of fit, 192n18, 214, 214n7 Google, 61, 173 Google Scholar, 61, 120, 173 Gore, A., 42n17 Graduate studies, 265–267 Grand Theories, 7n7 Graphic organizers, 31–35, 32f, 34t, 36t, 53, 70–71 Gross domestic product (GDP), 14, 167 Guilt by Association, 70 Hard tests, 114, 130 Hempel, C., 2n1 Hendess, S. C., 232 Hensel, P., 174 Histograms, 185–186, 189 Human Development Index (HDI), 167 Human Subjects Research Committees, 148n13 Human subjects research, 148–151, 148n13, 172n9 Hypotheses constants and variables and, 47–48 defined, 5 deterministic types, 89–92, 90–91t, 92–93p leverage over, 139–142 parts of, 44–45 for posters, 254 for presentations, 247 probabilistic types, 82–89, 84–85p research design and, 98–99 theories and, 39, 44–51 Hypotheses of no relationship, 87 Hypothetical questions, 4 Impact, use of term, 89 Independent (cause) variables, 44–45 Index cards, 71–72, 154 Indicators, 44, 48–51 Indices (books), 144–145 Indices/scales, 193 Indirect relationship, 39 Inductive theorizing, 25–26, 26t Inferential statistics, 158

Index 293

Information, data compared, 136–137 Informed consent, 150 Inglehart, R., 39n15 Insignificant coefficients, 214 Institutional Review Boards, 148n13. See also Human subjects research Instrumental variables, 202 Intellectual curiosity, 266–267 Interaction terms, 88–89, 194–195, 194–196t, 196p Inter-coder reliability, 147n10, 171 Interlibrary loans (ILL), 59, 61 Intermediate-N designs, 106–108 International Organization (IO), 10n12, 59n5, 60 International politics, domains and, 25 International Studies Review (ISR), 10, 10n12, 60 Inter-University Consortium for Political and Social Research (ICPSR), 174 Interval-ratio level of measurement, 137 Interval-ratio variables, 83, 100n15, 101–102, 185–186 Interviews, elite, 148–151 Introduction section, 55, 209–210 Intuitive regression, 126 Inverse relationship, 39 Iowa, 33–34, 34n10

Letterhead, 149n15 Levels of measurement, 83, 137 Levene’s test, 216 Likert scales, 83, 83n4 Linear probability models, 102, 102n20 Literary reviews, 58 Literature reviews finding literatures, 58–62 goals of, 57 misconceptions about, 57–58, 57n2 number of sources, 67–68 organization of, 63–68, 64t, 66t as research question source, 10–11, 10n12 thinking about, 56–57 Literature review section overview, 55 beads on a string, 57 certainty versus uncertainty, 77 forest versus trees, 79 number of sources in, 67–68 organization of, 69–72 for posters, 254 for presentations, 247 revision, 77–80 time for, 68 writing, 72–77, 76p Logit models, 102–103, 103t, 204–205 Log transformations, 187–190, 188–189f Longhand versus typing, 225

JabRef, 66t Jamaat-e-Islami, 126–127 Journal of Public Policy Processes, 60n8 Journals, 246n17, 263–265 JSTOR (Journal Storage Project), 59–60, 60n7

Main diagonals, 91, 91n8, 92t Maine, 33n9 Marginal effects, 215–216 Master’s degrees (MA), 265–266, 265nn18–19 Math skills, 81n1, 266–267 Matrices, 63–64, 63n11, 64t Maximum likelihood estimates, 214n7 McCain, J., 33n9 Measurement defined, 137 peer review about, 241, 242 for posters, 254 for presentations, 247 qualitative design, 137–139 quantitative design, 168–171 Media sources, 140 Methodology, 97n11

Keohane, R., 5n5, 22, 28, 97n12 King, G., 5n5, 22, 28, 97n12 Kirschner, S., 177 KKV. See Designing Social Inquiry (King, Keohane & Verba) Knowledge cumulation, 3, 11 Lagged variables, 201 Latinobarometro survey, 14 Leftovers files, 18–19, 52n27, 68n15, 73n21

294

Empirical Research and Writing

Methodology section for papers, 132–134, 133nn21–22 Methodology section for presentations, 247 Microsoft Excel, 182, 259 Microsoft OneNote, 154 Microsoft Word, 66t, 67, 211 Midwest Political Science Association (MPSA), 262–263 Mill’s Methods of Agreement and Disagreement, 124–125, 124n11. See also Case control method Mind maps, 71 Missing data, 183–185 Momentum, 225–226 Monographs, 152 Most similar/different systems. See Case control method Moving walls, 60, 60n7 Multicolinearity, 191n15 Multivariate hypotheses, 163n4

Observations, 111, 136, 158 OCLC ArchiveGrid, 152–153 Off diagonals, 91n8 Omitted variable bias (OVB), 164n6, 203–204 OneNote, 154 Ontology, 24n1 Operationalization, 49, 168 Ordered dependent variables, 101 Ordered logit models, 103 Ordered probit models, 103 Ordinal level of measurement, 137 Ordinal variables, 83, 83n4, 101–102, 103t, 185n8 Ordinary least squares (OLS), 101–102, 162, 214n7. See also Regression Out of bounds, 102n20 Out-of-sample tests, 123–124 Oxford Handbook of Political Science, 10, 60n8

N, 106n24, 211, 214, 216 Nader, R., 42n17 National Conference for Undergraduate Research (NCUR), 262, 262n15 Nebraska, 33n9 Necessary and sufficient conditions, 91–92, 92t Necessary conditions, 89–90, 90t Negative evidence, 117–118, 218 Nerd Words, 231 New Hampshire, 33–34 New Mexico, 33–34, 34n10 1992 elections, 42n17 Nominal level of measurement, 137 Nominal variables, 84, 100–101, 103t, 185n8 Nonlinearity, 185–190, 188–189f Nonlinear models, 204–205 Normative questions, 3–4 Norris, P., 39n15 Notebooks. See Research notebooks Null hypotheses, 87 Number of cases (N), 106n24, 211, 214, 216

Painter’s tape, 72n19 Panel presentations, 245–252 Panel studies, 203–204 Papers. See Empirical papers Paragraphs, 227 Paraphrasing, 219 Parsimony, 3 Partisanship, 49–50, 49f Pascal, B., 80 Paschal, R., 70 Passive voice, 232 Patterned diversity, 98 Patterns, 1–2 PD (Prisoners’ Dilemma), 122 Peer-reviewed journals, 58–60, 58n4, 263–265 Peer review process editing compared, 236–237, 239 effectiveness of, 234–236, 235n11 for journals, 59 procedures for, 237–243, 238p purposes of, 236–237 writing up review, 243–244, 244n17 Perot, H. Ross, 42n17 Personal statements, 267 PhD programs, 266–267, 266nn20–21 Philosophy of Natural Science (Hempel), 2n1

Obama, Barack, 33n9 Oberzinger, H., 70 Observable implications, 44, 46–47

Index 295

Pittsburgh Steelers fans, 37n12 Plagiarism, 153 Policy questions, 60 Policy Studies, 60n8 Poli Sci Data site, 173, 174 Political world, predictability of, 1–2 Polychotomous indicators, 84 Pop culture references, 231, 231n8 Population census, 116, 120, 128 Populations, 113, 158 Positivist empirical tradition, 24n1 Possibility principle, 118 Poster sessions, 252–257, 262–263 Postmaterialist arguments, 38–39, 39n15 PowerPoint, 247–248, 250–251, 254– 255, 257–259, 258n14 Power transition theory (PTT), 194–195, 194–195t Powner, L., 10 Predictability, of political world, 1–2 Predictions/expectations, 23 Predictiveness, 3 Prepublished works, 62, 62n10 Pre-research, 55. See also Literature reviews Presentations, 245–252, 260–263 Pretesting practices, 146–147 Previews, 209 Primary sources, 140 Print collections, 152 Printing, 65 Prisoners’ Dilemma (PD), 122 Private data, 140 Probabilistic hypothesis types, 82–89, 84–85p Probit models, 102–103, 103t, 204–205 Procedural questions, 4 Process tracing, 104, 130–132, 141–142, 217–218 Professional conferences, 262–263 Project MUSE, 60 Proper nouns, 8, 8n10, 60 Proto-hypotheses, 39 Prove, avoiding use of, 77 Proximate causes, 98 PTT (power transition theory), 194–195, 194–195t Publication, 246n17, 263–265 Public data, 140 Publishers, 59, 59n6

Puns, 231, 231n8 Purposive sampling, 115–116 P values, 212, 215 Qualifiers, 219 Qualitative Comparative Analysis (QCA), 106–107 Qualitative Data Archive, 153 Qualitative design overview, 109–110 analysis forms, 104–108 analytic narratives, 122–124 background research, 115–116 case control method, 124–129 case selection, 112–118 content analysis, 118–122, 118n6 data analysis section, 217–221, 220n11 number of cases needed, 110–112 peer review about, 241–242 process tracing, 130–132 quantitative compared, 95–97, 96t, 109, 136 second and third generation of, 97n12 strengths and weaknesses of, 97–99, 97n12 structured focused comparison method, 129–130 writing about, 132–134, 133nn21–22 Quantitative design analysis forms, 100–103, 103t analysis section, 210–217 peer review about, 242 qualitative compared, 95–97, 96t, 109, 136 strengths and weaknesses of, 97–99, 97n12 Questionnaires, 129, 178 Questions. See Research questions Quotations, 150, 153–154, 218–219, 249 R², 192n18, 214, 214n7 Railroad schedules, 248n3 Random sampling, 115, 121, 121n7 Ratio-level variables, 83n3 Reading aloud, 229 Reading and Understanding Political Science (Powner), 10, 251n6 Ready-to-use data, 171–175 Recoding variables, 198–200, 200p Redrafting, editing compared, 73

296

Empirical Research and Writing

Reference lists, 65–68, 66t References, 248, 255 Refworks, 66t Regression analysis section, 212–216 interval-ratio variables and, 101–102, 185–186 intuitive regression, 126 linearity and, 186 Relative hypotheses, 87 Reliability, 138–139, 170–171 Replicability, 3, 5, 5n5, 11 Replication datasets, 12, 172–174 Reporter questions, 7 Representative samples, 161–162 Research design overview, 81–82 analysis selection, 95–99 hypotheses and, 98–99 measurement and, 139 peer review about, 241–242 for posters, 254 for presentations, 247 See also Qualitative design; Quantitative design Research design section, 56, 132–134, 133nn21–22, 155–156 Research notebooks overview, 18 audit of writing process, 224 coding of data, 176, 176n18, 178 data management and, 153 recoding variables, 199 variables, 184 Research programs, defined, 10n11 Research questions attribution of, 11 brainstorming for, 17p crafting, 6–9 geographic limits on, 14 importance of, 9 narrowing, 13 phrasing, 12–15 samples of, 15–16p sources of, 9–12 types of, 3–4, 4nn3–4 Research topics, 5–6 Reserve slides, 250–251 Respondent driven sampling, 149

Resumes, 56n1 Reviews of the literature, 9–10, 60 Revise and resubmit (R&R) recommendations, 243 Revision, 226–234. See also specific sections of papers Rhetorical questions, 233 Road Maps, 70 Robustness checks, 167, 215 Running records, 140 Salmond, R., 257–258 Samples, 158 Sample size, regression and, 102 Sampling background research and, 115–116 random sampling, 115, 121, 121n7 representative samples, 161–162 research design and, 99 selection bias and, 120 snowball samples, 149 stratified random sampling, 121n7 See also Case selection Scales/indices, 193 Scatter plots, 185n8, 186 Scholarly literature, defined, 56 Science, defined, 3 Scooping, 173 Scope conditions, 24–25, 43, 53 Scrivener, 154 Secondary sources, 140 Selection bias overview, 14 analysis and, 162 peer review about, 242 qualitative design, 116–117, 121 sampling and, 120 Selection effects, 117, 161–162 Selection on the dependent variable, 125n12 Self-editing, 226–234 Sentences, 227 Sex, 49, 49n20 Simultaneity, 201 Simultaneity bias, 201 Sinatra Inference, 114, 114n3 Skocpol, T., 90 Slide shows, 247–248, 250–251, 254–255 Smith, D. T., 257–258

Index 297

Smoking gun tests, 131 Snowball samples, 149 Soaking and poking, 122n9 Social laws, 2n1 Social Networks and Archival Context project, 153 Social revolutions, 90 Social Science Research Network (SSRN), 61–62 Social Sciences Citation Index (SSCI), 61 Software, 154–155, 181–182. See also Data analysis, quantitative Sole consideration norm, 264 Solicited reviews, 60–61 Spearman’s rho, 101 Special collections, 152 Specification, 215, 242 Sponsors, 261, 263 SPSS, 182 Spurious relationships, 87 Square brackets, 226 Standard indicators, 168–169 Stata. See Data analysis, quantitative States and Social Revolutions (Skocpol), 90 Stat Transfer, 182 Stratified random sampling, 121n7 Structured focused comparison (SFC) method, 105–106, 106n23, 129–130, 218 Student conferences, 260–262, 262n15 Student journals, 246n17, 263–265 Style and tone, 231–234 Sufficient conditions, 90–91, 91t Surface errors, 228 Survey sampling, 161 Swing states, 31–35, 32f, 34n10, 34t Swiss Cheese, 70

causal mechanisms and, 36–39, 39–40p characteristics of, 3 concepts and, 48–51 defined, 22–23 development of, 28–35, 29p, 32f, 34t, 36t hypotheses and, 44–51 importance of, 22 indicators and, 48–51 parts of, 23–25 peer review about, 241 for posters, 254 for presentations, 247 scope conditions and, 43 sources of, 21–22, 25–28, 26t validity and, 48–51 writing about, 51–53 Theorizing, 7, 25–27, 26t, 27n2 Theory families, 68n15 Theory section, 51–53, 55–56 Thesis, defined, 5 Thesis shifts, 227 Titles, 211, 211n2, 247, 254 Tone and style, 231–234 Topics, 5–6 Tracing, 119 Transparency, 96t, 107, 211 Triangulation, 135, 151 T-tests, 216 Two-by-two matrices, 33–34, 34t, 89–90, 89n6, 90t Two-stage least squares (2SLS), 202n24 2000 elections, 31–35, 31n7, 32f, 34t, 36t, 42n17 2004 elections, 31–35, 31n7, 32f, 34t, 35, 36t Typing versus longhand, 225

Table notes, 211–212 Tables and figures, 63–64, 63n11, 64t, 211, 257 Tables of contents, 144–145 Tertiary sources, 140–141 Text mining, 146–147, 155–156 Theoretical questions, 4 Theories assumptions and, 42–43

Uncertainty estimates, 141 Underlying causes, 98 Units of analysis, 158–159p, 158n1 Unobserved predictions, 141 U.S. Census, 13 Validity, 48–51, 138–139, 168–170 Variable labels, 184, 185t Variable names, 182, 182n2, 184, 185t

298

Empirical Research and Writing

Variables composite variables, 192–200, 194–196t, 196p, 200p confounding variables, 203 constants and, 47–48, 113 control variables, 112–113, 162–164, 164–165p data analysis, quantitative, 184, 185t defined, 136 dependent variables, 44–45, 102–103 dichotomous variables, 84, 89, 204 independent variables, 44–45 instrumental variables, 202 interval-ratio variables, 83, 100n15, 101–102, 185–186 labels for, 184, 185t lagged variables, 201 names of, 182, 182n2, 184, 185t nominal variables, 84, 100–101, 103t, 185–186 ordinal variables, 83, 83n4, 101–102, 109t, 185n8 ratio-level variables, 83n3 recoding of, 198–200, 200p Venn diagrams, 31–32, 32f, 71 Verba, S., 5n5, 22, 28, 97n12 Visual design, 256 Visuals, 53, 211, 216. See also Tables and figures Voice, 232

Voter laws, 36 Voting, assumptions about, 42 War crimes, 85 Web of Science, 61 Welch’s procedure, 216 Welzel, C., 39n15 Wikipedia, 115n4, 116, 116n5 Within-case designs, 104, 111 Word choice issues, 228, 228n7 Word counting, 118–119 Wordiness, 232–234 Word software, 66t, 67, 211 Working papers, 56n1, 62, 62n10 World Politics (WP), 10n12, 60 World Values Survey (WVS), 39n15 Writer’s handbooks, 228 Writing centers, 237n14 Writing process avoidance of, 224 awareness of, 223–226 discipline about, 225–226 drafts, 73, 73nn21–22 self-editing, 226–234 sequence and format, 224–225 time for, 18, 18n19 See also Empirical papers; Peer review process; specific sections of papers Zotero, 66t, 67

CQ Press, an imprint of SAGE, is the leading publisher of books, periodicals, and electronic products on American government and international affairs. CQ Press consistently ranks among the top commercial publishers in terms of quality, as evidenced by the numerous awards its products have won over the years. CQ Press owes its existence to Nelson Poynter, former publisher of the St. Petersburg Times, and his wife Henrietta, with whom he founded Congressional Quarterly in 1945. Poynter established CQ with the mission of promoting democracy through education and in 1975 founded the Modern Media Institute, renamed The Poynter Institute for Media Studies after his death. The Poynter Institute (www.poynter.org) is a nonprofit organization dedicated to training journalists and media leaders. In 2008, CQ Press was acquired by SAGE, a leading international publisher of journals, books, and electronic media for academic, educational, and professional markets. Since 1965, SAGE has helped inform and educate a global community of scholars, practitioners, researchers, and students spanning a wide range of subject areas, including business, humanities, social sciences, and science, technology, and medicine. A privately owned corporation, SAGE has offices in Los Angeles, London, New Delhi, and Singapore, in addition to the Washington DC office of CQ Press.

The essential online tool for researchers from the world’s leading methods publisher

More content and new features added this year!

Find exactly what you are looking for, from basic explanations to advanced discussion

Discover Methods Lists— methods readings suggested by other users

“I have never really seen anything like this product before, and I think it is really valuable.” John Creswell, University of Nebraska–Lincoln

Watch video interviews with leading methodologists

Explore the Methods Map to discover links between methods

Search a custom-designed taxonomy with more than 1,400 qualitative, quantitative, and mixed methods terms

Uncover more than 120,000 pages of book, journal, and reference content to support your learning

Find out more at www.sageresearchmethods.com