The Sage handbook of methods in social psychology 9781782689447, 1782689443

Table of contents :
PART I Introduction and Overview. 1. The Research Process: Of Big Pictures, Little Details, and the Social Psychological Road in between / Carol Sansone and Carolyn C. Morf and A.T. Panter. PART II Fundamental Issues in Social Psychological Research. 2. The Methodological Assumptions of Social Psychology: The Mutual Dependence of Substantive Theory and Method Choice / Thomas D. Cook and Carla Groom --
3. Ethical Issues in Social Psychology Research / Allan J. Kimmel --
4. Developing a Program of Research / Susan T. Fiske. PART III Design and Analysis. Implications of a Heterogeneous Population: Deciding for Whom to Test the Research Question(s), Why, and How --
5. Culturally Sensitive Research Questions and Methods in Social Psychology / Joan G. Miller --
6. Individual Differences in Social Psychology: Understanding Situations to Understand People, Understanding People to Understand Situations / Yuichi Shoda --
Operationalizing the Constructs: Deciding What to Measure, Why, and How --
7. Constructing and Evaluating Quantitative Measures for Social Psychological Research: Conceptual Challenges and Methodological Solutions / Duane T. Wegener and Leandre R. Fabrigar --
8. Measures and Meanings: The Use of Qualitative Data in Social and Personality Psychology / Laura A. King --
9. Implicit Methods in Social Psychology / John F. Kihlstrom --
10. Mediated and Moderated Effects in Social Psychological Research: Measurement, Design, and Analysis Issues / Rick H. Hoyle and Jorgianne Civey Robinson --
Research Designs: Deciding the Specific Approach for Testing the Research Question(s), Why, and How --
11. Experimental Design and Causality in Social Psychology Research / S. Alexander Haslam and Craig Mcgarty --
12. Quasi-Experimental and Correlational Designs: Methods for the Real World When Random Assignment isn't Feasible / Melvin M. Mark and Charles S. Reichardt --
13. Within-Subject and Longitudinal Experiments: Design and Analysis Issues / Stephen G. West and Jeremy C. Biesanz Oi-Man Kwok --
14. Measuring Individuals in a Social Environment: Conceptualizing Dyadic and Group Interaction / Richard Gonzalez and Dale Griffin --
15. Quantitative Research Synthesis: Examining Study Outcomes over Settings, Samples, and Time / Wendy Wood and P. Niels Christensen. PART IV Emerging Interdisciplinary Approaches: The Integration of Social Psychology and other Disciplines. 16. Methodological and Ethical Issues in Conducting Social Psychology Research via the Internet / Michael H. Birnbaum --
17. Social Neuroscience: Bridging Social and Biological Systems / John T. Cacioppo, Tyler S. Lorig, Howard C. Nusbaum AND Gary G. Berntson --
18. Supplementing the Snapshots with Video Footage: Taking a Developmental Approach to Understanding Social Psychological Phenomena / Eva M. Pomerantz, Diane N. Ruble and Niall Bolger. PART V The Application of Social Psychology and its Methods to other Domains. 19. Program Evaluation, Action Research, and Social Psychology: A Powerful Blend for Addressing Applied Problems / Geoffrey Maruyama --
20. Methodological Challenges and Scientific Rewards for Social Psychologists Conducting Health Behavior Research / Peter Salovey and Wayne T. Steward --
21. Research Methods of Micro Organizational Behavior / Leigh Thompson, Mary Kern and Denise Lewin Loyd --
22. Conducting Social Psychological Research in Educational Settings: "Lessons We Learned in School" / Judith M. Harackiewicz and Kenneth E. Barron.

Citation preview

The SAGE

Handbook of Methods in Social Psychology

the memory of Kurt Lewin, whose contributions continue to instruct and inspire generations of social psychologists.

The SAGE

Handbook of Methods in Social Psychology

Edited by Carol Sansone University of Utah

Carolyn C. Morf and A T . Panter University of N o r t h Carolina, Chapel Hill

SAGE Publications International Educational and Professional Thousand Oaks • London • New Delhi

Publisher

Copyright © 2 0 0 4 by Sage Publications, Inc. All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher. For

information: Sage Publications, Inc. 2 4 5 5 Teller Road Thousand Oaks, California 9 1 3 2 0 E-mail: [email protected] Sage Publications Ltd. 6 Bonhill Street London EC2A 4PU United Kingdom Sage Publications India Pvt. Ltd. B-42, Panchsheel Enclave New Delhi 110 0 1 7 India

Printed in the United States of America Library of Congress Cataloging-in-Publication Data The Sage handbook of methods in social psychology / Carol Sansone, Carolyn C. Morf, A. T. Panter, editors, p. cm. Includes bibliographical references and index. ISBN 0 - 7 6 1 9 - 2 5 3 5 - X (Cloth) — ISBN 0-7619-2536-8 (Paper) 1. Social psychology—Methodology. I. Sansone, Carol. II. Morf, Carolyn C. III. Panter, A. T. HM1019.S24 2004 302'.01—dc21 2003004673 This book is printed on acid-free paper. 03

04

05

06

10

Acquisitions Editor: Editorial Assistant: Copy Editor: Production Editor: Typesetter: Proofreader: Indexer: Cover Designer:

9

8

7

6

5

4

3

2

Jim Brace-Thompson Karen Ehrmann A. J . Sobczak Diane S. Foster C & M Digitals (P) Ltd, Penny Sippel Juniee Oneida Michelle Kenny

1

Brief Contents

Preface

Part I: INTRODUCTION AND OVERVIEW 1. T h e Research Process: O f Big Pictures, Little Details,

and the Social Psychological Road in Between

xxvii

1



3

CAROL SANSONE, CAROLYN C. MORF, AND A. T . PANTER



Part II: FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH

2. T h e Methodological Assumptions of Social

Psychology: T h e Mutual Dependence of Substantive

Theory and Method Choice

17

19

THOMAS D . COOK AND CARLA GROOM

3 . Ethical Issues in Social Psychology Research

45

ALLAN J . KIMMEL

4 . Developing a Program of Research

71

SUSAN T . FISKE

Part III: DESIGN AND ANALYSIS

91

Section A. Implications of a Heterogeneous

Population: Deciding for W h o m to Test the

Research Question(s), Why, and H o w

91

5.

Culturally Sensitive Research Questions and

Methods in Social Psychology JOAN G . MILLER

93



6.

Individual Differences in Social Psychology: Understanding Situations to Understand People, Understanding People to Understand Situations

117

YUICHI SHODA

Section B . Operationalizing the Constructs: Deciding W h a t to Measure, Why, and H o w 7. Constructing and Evaluating Quantitative Measures for Social Psychological Research: Conceptual Challenges and Methodological Solutions

143

145

DUANE T . WEGENER AND LEANDRE R . FABRIGAR

8. Measures and Meanings: T h e Use of Qualitative Data in Social and Personality Psychology

173

LAURA A. KING

9. Implicit Methods in Social Psychology

195

JOHN F . KIHLSTROM

1 0 . Mediated and Moderated Effects in Social Psychological Research: Measurement, Design, and Analysis Issues

213

RICK H . HOYLE AND JORGIANNE CIVEY ROBINSON

Section C. Research Designs: Deciding the Specific Approach for Testing the Research Question(s), Why, and H o w

235

1 1 . Experimental Design and Causality in Social Psychology Research

237

S. ALEXANDER HASLAM AND CRAIG MCGARTY

12. Quasi-Experimental and Correlational Designs: Methods for the Real World When Random Assignment Isn't Feasible

265

MELVIN M . MARK AND CHARLES S. REICHARDT

1 3 . Within-Subject and Longitudinal Experiments: Design and Analysis Issues

287

STEPHEN G . WEST, JEREMY C. BIESANZ, AND OI-MAN KWOK

1 4 . Measuring Individuals in a Social Environment: Conceptualizing Dyadic and Group Interaction RICHARD GONZALEZ AND DALE GRIFFIN

313

1 5 . Quantitative Research Synthesis: Examining Study Outcomes Over Settings, Samples, and Time

335

WENDY WOOD AND P. NIELS CHRISTENSEN

Part IV: EMERGING INTERDISCIPLINARY APPROACHES: THE INTEGRATION OF SOCIAL PSYCHOLOGY AND OTHER DISCIPLINES 1 6 . Methodological and Ethical Issues in Conducting Social Psychology Research via the Internet

357

359

MICHAEL H. BIRNBAUM

17. Social Neuroscience: Bridging Social and Biological Systems

383

JOHN T . CACIOPPO, TYLER S. LORIG, HOWARD C. NUSBAUM, AND GARY G . BERNTSON

1 8 . Supplementing the Snapshots With Video Footage: Taking a Developmental Approach to Understanding Social Psychological Phenomena

405

EVA M . POMERANTZ, DIANE N . RUBLE, AND NIALL BOLGER

Part V: THE APPLICATION OF SOCIAL PSYCHOLOGY AND ITS METHODS TO OTHER DOMAINS 1 9 . Program Evaluation, Action Research, and Social Psychology: A Powerful Blend for Addressing Applied Problems

427

429

GEOFFREY MARUYAMA

2 0 . Methodological Challenges and Scientific Rewards for Social Psychologists Conducting Health Behavior Research

443

PETER SALOVEY AND WAYNE T . STEWARD

2 1 . Research Methods of Micro Organizational Behavior

457

LEIGH THOMPSON, MARY KERN, AND DENISE LEWIN LOYD

2 2 . Conducting Social Psychological Research in Educational Settings: "Lessons W e Learned in School" JUDITH M . HARACKIEWICZ AND KENNETH E . BARRON

471

Name Index

485

Subject Index

501

About the Editors

519

About the Contributors

521

Detailed Contents

Preface Acknowledgments Reference

Part I: INTRODUCTION AND OVERVIEW 1. T h e Research Process: O f Big Pictures, Little Details,

and the Social Psychological R o a d in Between

xxvii xxix xxix

1 3

CAROL SANSONE, CAROLYN C. MORF, AND A. T. PANTER

The Research Process The Starting Point: The Phenomena The Research Question To Whom Does the Question Apply? Operationalizations and Design Can We Answer the Question? Organization of This Handbook Organizing Principles Specific Organization Part II: Fundamental Issues in Social Psychological Research Part III: Design and Analysis Part IV: Emerging Interdisciplinary Approaches Part V: The Application of Social Psychology and Its Methods to Other Domains Conclusion References

Part II: FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH

2 . T h e Methodological Assumptions of Social

Psychology: T h e Mutual Dependence of Substantive

Theory and Method Choice THOMAS D. COOK AND CARLA GROOM

6 6 7 7 8 9 11 11 11 11 12 13 14 14 15

17

19



Introduction The Hypothetico-Deductive Method The Types of Theory That Social Psychologists Construct The Theory-Hypothesis Link Form of Data Collection The Dual Hegemony of ANOVA and the Laboratory Experiment Specific Theoretical Concerns and Their Methodological Implications The Social Cognitive Revolution The Relative Neglect of Theories of Interpersonal Dynamics The Neglect of High-Impact Manipulations and the Kinds of Theory They Promote The Average Person as the Locus of Explanation The Assumption of Irrelevant Domains and Hence the Generation of Theories With Minimal Grounded Content Conclusion References 3. Ethical Issues in Social Psychology Research

19 22 22 24 27 29 32 32 34 34 35

37 39 40 45

ALLAN J . KIMMEL

Chapter Overview The Evolution of Ethical Debate and Regulation in Social Psychology Governmental Regulations for Behavioral Research in the United States Professional Ethical Standards Ethical Dilemmas in Social Psychological Research Defining "Ethics," "Morality," and "Ethical Dilemma" Ethical Issues in the Conduct of Laboratory, Field, and Applied Research Laboratory Research Issues Field Research Issues Privacy Informed Consent Social Psychology Research and the Internet Applied Research Issues Ethical Safeguards and Institutional Review Debriefing and Other Safeguards

46 46 47 48 51 51 53 53 57 58 58 59 59 61 61

Institutional Review Impact and Effectiveness of the Review Process Conclusion: Ethical Challenges and Opportunities References 4 . Developing a Program of Research

63 64 65 65 71

SUSAN T. FISKE

Start by Knowing That Many Perspectives Are Not Yet Represented Compelling, Coherent Hypotheses: What's the Big Picture? Intellectual Sources Personal Sources Group Sources Worldview Sources General Principles, Regardless of Source Convincing Research: Read This Book Readable Write-ups Readers Will Read Outlets: Visible and Invisible Programmatic Approach: Follow Your Bliss Collaboration: Beside Every Good Researcher Stands a Team Teaching: A Piece of the Research Enterprise Funding: Aha! Plus . . . Service: Giving It Away Conclusion: From Madness to the Methods References

72 73 73 74 74 74 75 76 78 79 80 82 83 84 85 87 88

Part III: DESIGN AND ANALYSIS

91

Section A. Implications of a Heterogeneous Population: Deciding for W h o m to Test the Research Question(s), Why, and H o w

91

5. Culturally Sensitive Research Questions and Methods in Social Psychology

93

JOAN G. MILLER

Downplaying of Cultural Issues in Social Psychology Key Reasons for Downplaying of Culture Culture-Free Approach to Situations Physical Science Ideals of Explanation Apparent Universality and Explanatory Breadth of Psychological Theories Disappointment With Recent Cultural Traditions of Research

94 95 95 96 96 97

Conceptual Issues in Giving More Attention to Culture Views of Culture Integrating Cultural Considerations With Situational and Person Factors Methodological Strategies for Enhancing Cultural Sensitivity Cultural Understanding Sampling Noncomparative "Prototypic" Sampling Strategies Noncomparative Cultural Sampling Strategies Comparative Cultural Sampling Strategies Representativeness and Equivalence in Sampling Culture as Process Culturally Appropriate Measures Conclusions Notes References 6. Individual Differences in Social Psychology:

Understanding Situations to Understand People,

Understanding People to Understand Situations

97 98 99 99 99 101 101 102 103 103 104 106 108 109 109

117

YUICHI SHODA

Why Stable Individual Differences Need to Be Taken Into Account Lewin's Equation, B = f(P, E) P = An Individual's Dynamic Social Information Processing System: An Example Studying Person x Situation Interactions What Individual Differences? Individual Differences That Interact With Situations Processing Dynamics Type and Diagnostic Situations Types of Person Variables That Affect Processing Dynamics Interactions May Involve Highly Content-Specific Person and Situation Characteristics Going Beyond the Bandwidth-Fidelity Trade-off Behavioral Signatures of Person Types Guide an Inductive Approach to Discovering Individual Difference Constructs Methodological Challenges for Intensive Within-Subject Analyses Finding, Evaluating, and Using Measures of Individual Differences

118 119 119 119 121 122 122 124 124 125

125 127 128



What Makes a "Good" Measure: The Intertwined Nature of Reliability and Validity Bootstrapping Upward in the Evolution of Constructs, Theories, and Measures An Example of Construct Validation Research: The Multitrait-Multimethod Matrix Construct Validation of Individual Differences Measures via Experiments A Valid Measure Has Been Found! What Should We Do With It? Implications for Data Analysis and Experimental Design Continuous or Categorical? It Can Matter To Block or Not to Block on Individual Difference Measures? Recasting the Problem: Going Beyond Individual Differences as a Poor Person's Substitute for an Experiment Understanding the Effects of Situations for Each Person First Trading Instant Generalizability for Ultimate Generalizability: The Implications of a Person- and Type-Centered, More Inductive, Approach Concluding Thoughts: Understanding Situations to Understand People, Understanding People to Understand Situations Notes References Section B . Operationalizing the Constructs:

Deciding W h a t to Measure, W h y , and H o w 7. Constructing and Evaluating Quantitative

Measures for Social Psychological Research:

Conceptual Challenges and Methodological Solutions

128 129 130 130

131 131 131

133 133

134

135 136 137 143

145

DUANE T. WEGENER AND LEANDRE R. FABRIGAR

Defining Quantitative Measures Stages in Constructing Quantitative Measures Specifying Measurement Goals and Theoretical Assumptions Specifying One's Goals for the Measure Specifying Theoretical Assumptions Item Generation

146 146 147 147 147 148



Creating Items Item Content and Wording Number of Items Response Scale Format Response Option Order and Item Order Traditional Scaling Procedures for Item Generation Thurstone Equal-Appearing Intervals Likert Summated Ratings Semantic Differentials Item Evaluation and Selection Judge Ratings Between-Group Differentiation Item Descriptive Statistics Item-Total Correlations Factor Analysis Item Response Theory Evaluating Measure Quality Reliability Internal Consistency Stability (Test-Retest) Validity "Associative" and "Dissociative" Forms of Validity Evidence Associative Forms of Validity Dissociative Forms of Validity The M T M M Approach Beyond Self-Report Measures Notes References 8. Measures and Meanings: T h e Use of Qualitative

Data in Social and Personality Psychology

148 148 149 149 150 151 151 151 151 152 152 152 153 153 153 155 156 157 157 158 160 160 160 162 163 164 166 167 173

LAURA A. KING

Qualitative Data in Social Psychology: An Empirical Example Advantages of Asking Open-Ended Questions Qualitative Data May Answer Many Questions at Once Qualitative Data Allow Us to Measure What Isn't Said or Can't Be Said Qualitative Data Give Us the Flavor of the Whole Qualitative Data Are (Relatively) Timeless Methods of Qualitative Research Participant Selection and Recruitment

174 175 175 175 176 177 178 178

Deciding What Questions to Ask and How to Ask Them Coding the Data Extant Coding Schemes Creating New Coding Schemes Using Naive Coders The Training Phase The Coding Phase Naive Coders and the Bottom-Up Approach Reliability and Validity Reliability Validity Additional Challenges of Using Qualitative Data in Social Psychology Methodological Problems and Confounds Losing the Trees for the Forest Special Ethical Considerations New Approaches to Quantifying Qualitative Data Conclusion References 9. Implicit Methods in Social Psychology

179 179 179 180 183 183 184 184 185 185 188 189 189 189 190 190 191 192 195

JOHN F. KIHLSTROM

The Psychodynamic Heritage Projective Tests The Subtle and the Obvious The Priming Solution The Importance of Matching Tasks Priming as a Measure of Implicit Attitudes Critique of Priming The Implicit Association Test The IAT as a Psychometric Device Critique of the IAT The Unobtrusive, the Automatic, the Implicit—and the Psychologist's Fallacy Notes References 1 0 . Mediated and Moderated Effects in Social

Psychological Research: Measurement, Design,

and Analysis Issues

196 196 197 198 199 201 202 203 204 205 207 209 209

213

RICK H. HOYLE AND JORGIANNE CIVEY ROBINSON

Measurement Issues Practical Benefits of Theory

215 215

Formally Designating the Status of Variables in a Model An Example Optimal Measurement Design Issues Asserting Causal Priority Timing and Tests of Mediation Experimental Designs Analysis Issues Analysis Issues Specific to Mediation Analysis Issues Specific to Moderation Stumbling Blocks Ambiguous Theory One-Shot Data Limited Sample Size Limited Number of Indicators Conclusion References

216 218 219 222 222 223 224 225 227 228 231 231 231 231 232 232 232

Section C. Research Designs: Deciding the

Specific Approach for Testing the Research

Question(s), Why, and H o w

235

1 1 . Experimental Design and Causality in

Social Psychology Research

237

S. ALEXANDER HASLAM AND CRAIG MCGARTY

Introduction Designing Controlled Experiments: The Aims and Structure of This Chapter Why and When Should We Do Experiments? The Logic of Experiments Replication Identifying Plausible Confounds Multiple Factors When Not to Conduct Experiments Experimental Components Independent Variables (TVs) Randomization Theoretical Relevance Manipulation Checks Between- and Within-Subjects Designs Dependent Variables (DVs) Scale Construction

237 238 238 239 240 240 241 241 242 242 242 243 244 245 245 245

The Relevance-Sensitivity Trade-Off The Experimental Sample Representativeness The Importance of Theory Using Specialized Samples Experimental Assembly Threats to Internal Validity Dealing With Confounds Uncertainty Management Ruling Out Alternative Hypotheses: Dealing With Specific Threats to Internal Validity Maturation Effects History Effects Experiment Effects Sample Effects Experimental Control Threats to External Validity Reactivity Artificiality Inferring Causal Relationships From Experimental Research Lessons From the Social Psychology of Causal Inference Detecting Covariation Beyond Covariation Mediational Analysis The Logic of Mediation Problems of Interpretation Conclusion Note References 1 2 . Quasi-Experimental and Correlational Designs:

Methods for the Real World When Random

Assignment Isn't Feasible

246 247 248 249 249 251 251 251 251 252 252 252 253 253 254 254 255 256 256 256 257 257 258 258 258 259 260 260

265

MELVIN M. MARK AND CHARLES S. REICHARDT

Quasi-Experimental Designs: An Overview A Counterfactual Conception of Causality From the Concept of Causality to Kinds of Comparisons Comparisons Across Time The One-Group, Pretest-Posttest Design

266 266 267 267 268

Interrupted Time-Series Designs Interrupted Time-Series Designs With a Control Group Analysis of Interrupted Time-Series Designs Comparisons Across Groups The Regression-Discontinuity Design Nonequivalent Group Designs Analysis of the Pretest-Posttest Nonequivalent Group Design The Pretest-Posttest Nonequivalent Group Design With Separate Pretest and Posttest Samples Complex Nonequivalent Group Designs Summary Correlational Designs Threats to the Validity of Correlational Designs The Analysis of Data From Correlational Designs Beyond Individual Studies: Research Programs, Lines of Research, and Research Syntheses Conclusions Design Matters But Design Is Not Everything The Pattern of Observed Effects Also Matters Recognizing and Reporting Threats to Validity Matters Too Finally, Social Psychologists Should Not Be One-Trick Ponies References 1 3 . Within-Subject and Longitudinal Experiments:

Design and Analysis Issues

269 271 272 272 272 275 276 277 277 279 279 280 281 281 283 283 283 284 284 284 284 287

STEPHEN G. WEST, JEREMY C. BIESANZ, AND OI-MAN KWOK

Perspectives on Causal Inference Three Design Elements Basic Within-Subject Design Unit Homogeneity Randomization Within-Subject Experiments Approaches to Controlling for Order of Treatment Conditions Random Ordering Counterbalancing Latin Square Designs

288 289 289 289 290 291 291 291 292 292

Randomized Matched Designs (Predictor Sort Designs) Moderator Effects Mediation Summary and Conclusions Longitudinal Experiments Threats to Causal Inference Attrition SUTVA Some Longitudinal Experimental Designs Pre-Post Randomized Experiment Solomon Four-Group Design Multiwave Longitudinal Experiment Analysis Approaches: Traditional and Modern Univariate Analysis of Variance Three Possible Improvements to the Univariate Analysis of Variance Approach Correction of p Values Multivariate Approach Contrast Approach Growth Models Extensions Mediation Summary and Conclusions Final Conclusion Notes References 1 4 . Measuring Individuals in a Social Environment:

Conceptualizing Dyadic and Group Interaction

293 294 295 295 296 297 297 298 298 298 299 299 300 300 301 301 301 302 303 306 306 307 308 309 310 313

RICHARD GONZALEZ AND DALE GRIFFIN

Non-Independence and Interdependence The Intraclass Correlation Graphical Representation of the Intraclass Correlation Individual and Group Effects: One Is Not Enough The Use of Dyad Means as Indicators of Shared Variance Influence and Interaction: A Model of Interdependence Hierarchical Linear Modeling: Same Old Story or a New Perspective? Morals References

315 317 319 323 326 327 329 331 333



1 5 . Quantitative Research Synthesis: Examining

Study Outcomes Over Settings, Samples, and Time

335



WENDY WOOD AND P. NIELS CHRISTENSEN

Uses for Quantitative Research Synthesis Evaluating Existing Theories Testing Novel Hypotheses Procedures in Conducting a Research Synthesis Determining if You Have Enough Studies Defining the Problem, Variables, and Sample Locating Relevant Studies Forming the Meta-Analytic Database Coding Study Features Selecting Computer Programs to Calculate and Analyze Effect Sizes Calculating Effect Sizes Problems (and Solutions) When Calculating Effect Sizes Independence of Observations Complex Primary Study Designs Correcting for Effect Size Bias Strategies for Nonreported Results Analyzing Meta-Analytic Data Step 1: Choosing a Model Step 2: Estimating Means and Variability Step 3: Investigating Possible Moderators Step 4: Reporting Findings Drawing Conclusions From Meta-Analyses Interpreting Effect-Size Statistics The Impact of Synthesis Findings and the Future of Research Synthesis Notes References

Part IV: EMERGING INTERDISCIPLINARY APPROACHES: THE INTEGRATION OF SOCIAL PSYCHOLOGY AND OTHER DISCIPLINES 1 6 . Methodological and Ethical Issues in Conducting

Social Psychology Research via the Internet

336 336 337 338 338 339 340 341 341 342 342 344 344 345 346 346 346 347 348 349 350 350 350 351 352 353

357

359

MICHAEL H. BIRNBAUM

Minimum Requirements for Online Experimenting Potential Advantages of Research via the W W W

360 361



Examples of Recruiting and Testing Participants via the Internet Recruitment Method and Sample Characteristics Demographics Web Participants Do Not Represent a Population Effect of Diversity on Power and Generality Experimental Control, Measurement, and Observation Two Procedures for Holding an Exam Precision of Manipulations and Measurements The Need for Pilot Work in the Lab The Need for Testing of H T M L and Programming Testing in Both Lab and Web Dropouts and Between-Subjects Designs Experimenter Bias Multiple Submissions Ethical Issues in Web and Lab Risks of Psychological Experiments Ease of Dropping Out From Online Research Ethical Issues Peculiar to the W W W Deception on the W W W Privacy and Confidentiality Good Manners on the Web Concluding Comments References 17. Social Neuroscience: Bridging Social

and Biological Systems

362 363 363 364 365 366 366 367 368 369 369 370 371 372 374 374 375 376 376 377 378 378 379 383

JOHN T. CACIOPPO, TYLER S. LORIG, HOWARD C. NUSBAUM, AND GARY G . BERNTSON

Social Neuroscience and Links to Biological Systems Inferring the Psychological Significance of Physiological Signals The Psychological and Physiological Domains Functional Magnetic Resonance Imaging (fMRI) The BOLD Response Task Demands Tissue and Psychology Electrophysiological Measures of Brain Activity Summary Notes References

383 386 388 389 390 393 394 395 398 399 400

1 8 . Supplementing the Snapshots With Video Footage:

Taking a Developmental Approach to Understanding

Social Psychological Phenomena

405

EVA M. POMERANTZ, DIANE N . RUBLE, AND NIALL BOLGER

What Does It Mean to Use the Video Camera? Why Use the Video Camera? How to Use the Video Camera Developmental Designs The Cross-Sectional Design The Longitudinal Design The Cross-Sequential Design Operationalizing Developmental Phase Micro-Analytic and Macro-Analytic Strategies Identifying the Processes Underlying Developmental Change Other Considerations in Conducting Developmental Research Recruitment Procedural Equivalence Methodological Benefits of Conducting Developmental Research But You Can't Just Use Only the Video Camera Conclusion References

Part V: THE APPLICATION OF SOCIAL PSYCHOLOGY AND ITS

METHODS TO OTHER DOMAINS

1 9 . Program Evaluation, Action Research,

and Social Psychology: A Powerful Blend

for Addressing Applied Problems

406 406 408 408 408 410 411 412 413 414 415 415 416 417 418 420 421

427

429

GEOFFREY MARUYAMA

What Is Program Evaluation? Framework for the Chapter The Program Evaluation Field Action Research and Program Evaluation Action Research Applied to Issues of Educational Opportunity "Traditional" Evaluations of School Structures Policy-Relevant Evaluation: Modeling Impacts of School Accountability

430 431 434 435 437 438 439

Longitudinal Evaluation Designs and Collaborative Work: Understanding Relations of Poverty With Achievement Methodological Tools and Their Uses Limitations of Action Research Models for Program Evaluation Closing Note References 2 0 . Methodological Challenges and Scientific

Rewards for Social Psychologists Conducting

Health Behavior Research

439 440 441 441 442

443

PETER SALOVEY AND WAYNE T. STEWARD

A Brief Introduction to Research in the Health, Emotion, and Behavior (HEB) Laboratory Challenge #1: Serving Two Masters Challenge #2: Testing Your Question Challenge #3: Ethical Issues Relationship With the Community Professional Development Graduate School Postdoctoral Training Employment Integrating Social Psychology and Health Behavior References 2 1 . Research Methods of Micro Organizational Behavior

444 445 449 451 452 453 453 453 453 454 455 457

LEIGH THOMPSON, MARY KERN, AND DENISE LEWIN LOYD

Relationship of Micro OB to Organizational Research Comparison of Micro OB to Social Psychology The Key Factor Distinguishing Micro OB From Social Psychology Key Independent Variable Key Dependent Variable Omnipresence Research Project Development in Micro OB Step 1: Problem in Real World Stimulates Unresolved Question Step 2: Researcher Reformulates the Real World Problem Into a Testable Research Question Step 3: Researcher Consults Theory to Derive Hypotheses Step 4: Researcher Devises Study to Test Hypotheses

458 458 459 459 459 459 459 460 460 461 461



Step 5: Data Analysis and Results Step 6: Conclusions: Theoretical and Prescriptive Step 7: Application Common Methodologies in Micro OB Research Setting Design Classroom Setting Field Setting Laboratory Setting Conclusion References 2 2 . Conducting Social Psychological Research in

Educational Settings: "Lessons W e Learned in School"

462 462 462 463 463 463 464 465 466 467 468

471

JUDITH M. HARACKIEWICZ AND KENNETH E . BARRON

Moving Out of the Lab and Into the Classroom: Choosing the Setting Moving Into the Classroom: Choosing a Design Moving Into the Classroom: Choosing Measures Moving Into the Classroom: Implications and Methodological Trade-Offs Moving Back to the Lab Back to the Classroom: Validity Issues Revisited Lessons We Learned in School Lesson #1: The Importance of Using Multiple Research Methodologies in Multiple Settings Lesson #2: Dealing With the Dilemma of the Social Psychologist Lesson #3: Increasing the Credibility and Valuation of Research References

472 474 474 475 477 478 480 480 481 481 482

N a m e Index

485

Subject Index

501

About the Editors

519

About the Contributors

521



Preface

T

he genius o f social psychology as a field has been its ability to investigate the seemingly unstudiable, c o m p l e x behaviors that characterize humans as social creatures. T h e field has a rich history o f methodological innovation

with strong contributions t o basic and applied research. However, it is sometimes difficult for both n e w and seasoned researchers to keep up with innovations that allow a greater diversity in the kinds and levels o f research questions that can be addressed. As a result, the nature o f the questions asked by many researchers m a y be unnecessarily constrained. Conversely, a rush to embrace newer approaches can lead t o a less-than-thorough consideration o f fundamental issues that transcend any

particular approach. W e believe that the decision t o use a particular methodological approach is optimally made when grounded in careful considerations o f the "big picture" o f a program o f research. T h u s , methodological decisions are tied inextricably to w h a t the researcher, ultimately, wants t o k n o w . O u r major purpose in editing this h a n d b o o k was to create an integrated collection o f conceptually guided chapters that address the c o m m o n and unique methodological decisions that researchers must m a k e when using both traditional and cutting-edge research paradigms. Based on our " t o p - d o w n " perspective, chapters in this volume emphasize the conceptual basis o f the methodology, with an explicit focus on the meaning o f data when obtained via a particular methodology. O u r thinking has been heavily influenced by the writings o f Kurt Lewin, t o whose memory we dedicate this b o o k . Lewin believed firmly that theory and method are completely intertwined and that we should use our questions t o c o m e up with creative methodologies t o address them. T o Lewin, "research is the art o f taking the next step" (Lewin, 1 9 4 9 / 1 9 9 9 , p. 2 5 ) . W e believe we have captured this art as well as the science with the present collection o f chapters. W e implemented the top-down perspective in t w o ways. First, the overall organization o f the h a n d b o o k parallels what we see as the "big picture" o f the overall research process. Beginning chapters address issues related t o selecting and identifying research questions and populations, middle chapters address issues related to design and analysis issues, and later chapters address issues related t o expanding the original social psychological questions t o other disciplines within and outside psychology. Statistical analysis is considered a process in service o f research

SAGE HANDBOOK OF M E T H O D S IN SOCIAL P S Y C H O L O G Y design, and it is included to the extent that it helps to illuminate the distinct meaning o f data obtained through a particular methodological approach or design. Thus, the focus is on the conceptual meaning o f the data and analysis, rather than on microlevel " h o w t o " guidance through analytical issues. Second, we have attempted to maintain the top-down perspective within each chapter. All contributors were asked t o follow a general template in which they first describe a concrete and relevant social psychological research problem (or problems) and then discuss relevant methodological issues in the c o n t e x t o f that problem. Contributors t o this volume were selected because they have developed expertise on particular methodological approaches or issues in social psychology—and, more important, they did so in response t o attempting t o discover the best way to understand the psychological phenomena that interested them. T h u s , these researchers fit the "Lewin m o d e l " in that they have let the research questions guide their methods, rather than the reverse. These expert researchers discuss traditional and state-of-the-art methodological advances by first outlining concrete research phenomena and related questions o f interest and then showing h o w these questions may be best answered through design and analysis decisions. Adopting the top-down perspective led t o several features o f this h a n d b o o k that set it apart from other methods b o o k s . In addition to traditional methodological areas relevant for social psychologists, the b o o k includes innovative chapters such as those on ethics, culture and diversity, and individual differences. M o r e o v e r , the h a n d b o o k captures social psychology's increasing emphasis on research that crosses disciplines both within and outside psychology (e.g., social neuroscience, social development, and social psychology and the Internet). Also included is a section on some applications o f social psychology and its methods to other domains (e.g., program evaluation, health, education, and organizations). It was impossible to include all possible domains o f application, but we chose domains that we thought would have the broadest interest and that would have c o m m o n issues as well as unique challenges. In reading these chapters, it is evident that there are many similarities across areas o f application; thus, our hope is that these diverse samplings will also allow for translations to other areas. W e intend the audience for this h a n d b o o k to be active researchers interested in using social psychological approaches to address their research questions. T h i s audience includes graduate students and advanced undergraduates w h o are being introduced to the methods o f social psychology. It futher includes more advanced behavioral scientists in academic and research settings w h o are interested in learning about modern perspectives on classic approaches as well as newer methodological approaches in social psychology. O u r hope is that readers will c o m e away with an appreciation for the complexity of the field's phenomena along with a sense o f excitement about the fun and value of the research methods that can be used to unravel these phenomena. As editors o f this volume, we have learned a lot from the authors and their chapters, and we hope that readers do the same!

Preface ACKNOWLEDGMENTS W e wish to thank our colleagues Irwin Altaian, W a l t e r Mischel, M o n i s h a Pasupathi, and Bert U c h i n o for their sage advice and thoughtful feedback at critical points throughout this process. W e are also grateful to Angela N e w m a n at the University of Utah for her critical help in bringing order out o f chaos as we tried to keep track of all the various versions o f files and other paperwork associated with developing and finalizing the b o o k . J i m B r a c e - T h o m p s o n , senior editor at Sage Publications, provided both the initial enthusiasm and constant support for this project, and he helped t o m a k e our vision o f the b o o k a reality. W e thank him for that. W e also thank A. J . Sobczak for his extraordinarily thoughtful and critical w o r k as copy editor and Diane Foster for all her efforts as production editor. Finally, we would each like t o note some personal acknowledgments. C a r o l Sansone would like to thank her family, friends, and students for their patience and support during the several years she devoted to completion o f the b o o k . She would also like t o thank her younger brother, D o n , for allowing her t o publicly confess her past misdeed in Chapter 1. Carolyn C. M o r f would like to acknowledge with gratitude the help and support o f her family and friends during the preparation o f this volume. Finally, A. T . Panter would like to express her sincere thanks t o her major supports during this project: George H u b a , N e c h a m a and Y a a k o v H u b a , Sarajane Brittis, and her family, especially Danielle, M i c h a e l a , and Gideon Panter. Carol Sansone Carolyn C. M o r f A. T . Panter

REFERENCE Lewin, K. (1999). Cassirer's philosophy of science and the social sciences. In M . Gold (Ed.), The complete social scientist: A Kurt Lewin reader (pp. 23-36). Washington, DC: American Psychological Association. (Original work published 1949)

\

xxvii

Parti INTRODUCTION AND OVERVIEW

C H A P T E R

1

The Research Process Of Big Pictures, Little Details, and the Social Psychological Road in Between CAROL SANSONE University of Utah CAROLYN C . MORF A . T . PANTER University of North Carolina, Chapel Hill

W

hen 6 years old, one o f us [CS] hit

easy to see why the problem occurs. Social

her younger brother on the head

psychologists study very complex behaviors,

with a rock. H e was bending over

which are simultaneously connected with a

to look into a basement window, and a rock

host of contextual features and o f internal, not

was on the ground next to him. Curious about

directly observable, processes (e.g., perceiving,

what would happen if the rock hit his head,

construing, feeling, goal-striving). Considerable

she was unprepared (given a steady diet o f

creativity and thought have gone into creat-

Looney Tunes cartoons) for the result: It hurt

ing methodological approaches that

allow

him. Though long forgiven and mostly forgot-

the researcher to focus in on subsets o f these

ten (except at holidays), this episode illustrates

complex, interrelated features and processes.

both the motivation behind most science—the

An early example o f this creativity was the

need to k n o w what happens, and why—and

work by Kurt Lewin and his students. At the

the potential negative consequences o f allow-

time that psychodynamic and

behaviorist

ing available methods t o shape our questions,

approaches

motivation

rather than the reverse. Formal training in research methods and

were emphasizing

defined in terms o f instincts or reward outcomes, Lewin and colleagues proposed that is attached to the process

of

design can mask this elemental problem, but

motivation

the problem remains. In social psychology, it is

goal-striving. From both psychodynamic and

4

INTRODUCTION AND OVERVIEW behaviorist perspectives, this proposal would

attitude change

have been neither worthwhile nor feasible to

about a topic after individuals read persuasive

(e.g., measuring

attitudes

examine. In contrast, Lewin and colleagues

messages), Asch's ( 1 9 4 6 ) work on impression

developed

in

formation (e.g., measuring people's impres-

which the researcher creates a situation in the

sions o f a hypothetical individual after they

laboratory that should trigger the unobserv-

are given a list o f traits that

able psychological process (e.g., goal-striving)

describe the individual), Festinger's (e.g.,

and compares it to a situation that should not.

Festinger 8c Carlsmith, 1 9 5 9 ) work on cog-

a methodological approach

ostensibly

Furthermore, these researchers proposed that

nitive dissonance (e.g., the insufficient justifi-

the relative motivational differences created in

cation

these situations, which could not be measured

Schachter & Singer, 1 9 6 2 ) w o r k on emotion

directly, should be reflected in related variables

(e.g., the misattribution paradigm).

that

could

be

measured.

For

example,

paradigm),

When

and

S c h a c h t e r ' s (e.g.,

methodological

paradigms

are

Zeigarnik ( 1 9 2 7 , as cited in Lewin, 1 9 5 1 )

created that seem to capture, at least to some

developed a method for testing the hypothesis

extent, the complex behaviors and processes

that motivation attached to goal-striving is

involved in social psychological phenomena,

reflected at the thinking level; the method

the paradigms usually include a typical setting

involved measuring the relative memory for

(e.g., lab), participant population (e.g., college

completed

students), operationalizations,

approach

and

uncompleted

tasks.

Her

led to the identification o f the

"Zeigarnik effect" (remembering roughly twice

and

analyses.

All

these

procedures,

aspects

of

the

paradigm tend to be repeated in subsequent

as many o f the uncompleted tasks)—an objec-

applications. And it is here that the problem

tively measured, empirical finding that reflects

can

a hidden, dynamic psychological process. This

paradigm can start to guide and constrain the

ability to test hypotheses involving complex,

questions that researchers ask. For example,

occur—that

is,

the

methodological

unobservable processes created a foundation

instead o f continuing to ask questions about

for the laboratory science o f social psychology

the

that followed. In addition, Lewin and col-

researchers may begin to limit their investiga-

leagues investigated these processes in "real

tions to factors that affect memory for novel

world" settings, translating hypotheses

and

tasks completed in a lab. Although knowledge

developing creative ways to test their predic-

gained about the memory process may be use-

tions in organizations facing change, among

ful in its own right, these studies may take us

nature

o f the

goal-striving

process,

housewives dealing with wartime rationing,

away from the original phenomenon o f the

and so on.

goal-striving process (which is reflected in,

T h e work by Lewin and his group represents one o f the earliest examples o f the methodological creativity found in social psychology, but it certainly is not the last. In fact, many o f the early "classics" in social psychology research are known for the creation o f novel methodologies in addition t o the ideas that led to them. M a n y o f these methodologies are still used. Just a few examples o f this

but not limited to, memory for these tasks). Perhaps a more recent example o f this problem may be seen in the rise o f studies that employ cognitive neuroscience without

first

considering

paradigms

whether

these

paradigms are the best way to study the phenomena. (For a discussion o f this problem, see Cacioppo, Lorig, Nusbaum,

and Berntson,

Chapter 1 7 , this volume.)

his

Figure 1.1 illustrates the idealized research

colleagues at Yale (e.g., Hovland, Janis, &

process. T h e figure shows a process o f research

Kelley, 1 9 5 3 ) investigating persuasion

that is relatively linear, stagewise, and iterative.

include

the

work

by

Hovland

and

and

The Research

The Idealized Research Process

Process

What are your next steps? What additional studies will you do?

How close are you to understanding the original phenomena?

What can you and can't you conclude about your research questions?

i

»

What type of methodological approach will you use?

1

To whom (i.e., what populations) do your research questions apply?

What are your research questions about these phenomena? What are the real world What are the real world Tj j Phenomena of interest? phenomena of interest? \ j [ l 5 ^} JTim_e_2] '• 1

What are the real world phenomena of interest? [Time 11

Figure 1.1

The

I

rn

A Schematic D r a w i n g o f t h e I d e a l i z e d Research Process

process starts with our identifying the

questions that arose as consequences of our

phenomena that we want to understand. W e

research findings. At some point, we may con-

then generate specific questions and

hypo-

verge at some broader set o f conclusions, and

theses, as well as decide if these questions or

in some cases we may shift our emphasis on a

hypotheses should

be universal. W e next

given question. Thus, over time, with empirical

operationalize our hypothesized constructs in

studies, samples, and methodologies, we grad-

some more concrete way that invoke important

ually build a knowledge base about the phe-

measurement principles, select an appropriate

nomena—perhaps broad enough so that we

research design, analyze the results, and decide

can apply this knowledge to treat, change, or

to what extent our initial hypotheses were sup-

alleviate important social problems.

ported. Depending on the evidence we gather

If the process is so clear and is agreed

and our conclusions about the patterns in our

upon, then why do we believe that allowing

data, we hope that we gain some increased

one's methods to drive the research question

understanding of our phenomena o f interest—

is an elemental problem? In the following sec-

and probably have even more questions than

tion, we describe h o w this idealized research

when we began. W e then cycle

through

process can be affected by the very social psy-

the process again with revised questions or

chological processes that social psychologists

5

6

INTRODUCTION AND OVERVIEW study. W e will also briefly outline what we see

someone else (a member o f the majority group)

as the consequences of these effects, conse-

being given more positive feedback for the

quences that can contribute to this elemental

same work product because the person is being

problem. W e then describe this

held to a lower standard. In one sense, these

handbook,

which we have organized according to these

two perspectives of the phenomena are the

ideas about the research process.

same: People believe that the supervisor is using different standards to evaluate performance as a function of the worker's group

THE RESEARCH PROCESS

membership. T h e different takes on the phenomena, however, will lead to very different

The Starting Point:

The Phenomena

directions in the subsequent research process. If we assume that the phenomena have exis-

Often, the phenomena we focus on c o m e

tence outside our perceptions (see Gergen,

us: the

2 0 0 1 , for a different perspective), then each

experiences we have, the curious patterns that

researcher, no matter what methodological

from observing the world around

we see in people's behaviors, the social prob-

approach is used or how many attempts are

lems we would like to be able to alleviate.

made, can only approximate the phenomena,

Whatever the specific phenomenon, what we

just as a single measurement can never fully

actually focus on for our research efforts

define a latent construct. Furthermore, phe-

is necessarily only one small part, filtered

nomena are not static; they (and our perspec-

through our own construal processes. As we

tives on them) evolve and change as illustrated

identify and define what aspects of the phe-

in Figure 1.1 by the gradual shift in the "phe-

nomena are o f particular interest to us, then,

nomena o f interest" over time.

we have begun to make decisions that will limit what we discover about the phenomena. This reality argues in favor o f having a

Discussing the potential effects only o f the individual

researchers' construal

processes

would

be misleading, however. Decisions

diverse group of scientists identifying the phe-

about

what

nomena, because different experiences, back-

important

phenomena

to

study,

are

and

considered

how

they

are

grounds, and contexts will lead to identifying

defined, are often shaped by the social con-

different phenomena, or in focusing on differ-

text that the "field" represents. T h e field can

ent aspects o f the phenomena, or in defining

(and should) positively affect the research

those aspects from a different perspective (Sue,

process. Reading the past literature provides

1 9 9 9 ) . For example, suppose a researcher is

researchers

with

interested in the phenomena of how people

phenomena,

perhaps supporting but

react when a supervisor provides

challenging

the

different

feedback to two subordinates when, objec-

other

perspectives

individual

on also

researcher's

perspective (and any biases he or she might

tively, their work products seem to be the

have). M o r e o v e r , the

same. A researcher from a majority perspective

information and suggestions about various

literature

provides

(e.g., being white or male) may define the phe-

ways to examine the phenomena empirically,

nomena in terms of watching someone else (a

allowing the individual researcher to build

member of a minority group) get more positive

on what others have done, rather than hav-

feedback for the same work because the other

ing to "re-invent the wheel."

person is being given special treatment. In con-

In addition to the positive effects that the

trast, someone from the minority perspective

larger research community can have on the

(e.g., being African American or female) may

research process, however, there may also be

define the phenomena in terms o f watching

some negative effects. For example, in our

The Research

|

Process

work as editors, reviewers, and advisers, the

and "research hypotheses" often are used

broader significance o f the topic being studied

interchangeably in discussing the research pro-

is a key dimension for our judgments o f pro-

cess, they can be distinct—that is, a hypothe-

posals and manuscripts. It is natural to think

sis proposes

that topics in which we are already interested

question. As such, hypotheses include the

an

answer

to the

research

and invested are important to research (e.g.,

question (sometimes implicitly), but the ques-

Renninger, 2 0 0 0 ) . Thus, this human tendency

tion does not necessarily have to include

can create a gateway through which pheno-

hypotheses—as the rock episode at the begin-

mena

ning o f the chapter illustrates. Moreover, the

already being researched (with

established methodological paradigm)

an are

same research question may lead a researcher

evaluated as more important by the "field."

to plan multiple studies that together address

One consequence of this is that when phe-

the

nomena are identified and defined from per-

more specific to a given study. Because it is

spectives that are not a major part o f the status

broader and more inclusive, therefore, we use

quo, they may be less likely to make it through

"research question" in our discussion.

the gateway. For example, a meta-analysis o f editorial decisions at Personality

and

question,

whereas

hypotheses

are

Sometimes the question is only a small step

Social

from the original observation. At other times,

found that articles sub-

however, the research question may be a

mitted by women as lead authors were less

number o f steps removed. In the supervisor

likely to be accepted for publication (Petty,

feedback

Fleming, &

Fabrigar, 1 9 9 9 ) ; one possible

researcher may ask whether the effect o f expe-

explanation for this finding is that during the

riencing differential feedback on a worker's

Psychology

Bulletin

example discussed previously, a

period studied, women were relatively more

motivation depends on the worker's beliefs

likely to choose phenomena to research that

about why it happened.

were not considered as central to the field.

researcher has taken a complex phenomenon

In this case, the

A second consequence o f peer review is

that may be tied to a particular context and

that by maintaining the status quo in terms

time (see Altman, 1 9 8 8 ) and has abstracted

of the research topics that are considered

out one small dimension about which to ask

important, it also tends to maintain estab-

questions. Just as in identifying and defining

lished methodological paradigms. O f course,

the phenomena, then, the researcher makes

there is nothing inherently wrong with using

choices o f what to ask about the phenomena,

established methodological paradigms: T h e y

and these choices are guided by his or her per-

often become established precisely because

spective as well as that provided by the field.

they have been very useful in testing research

These research questions guide as well as limit

questions. Established paradigms become a

subsequent methodological choices and what

problem only when they start to constrain

we can eventually learn.

the kinds o f questions asked.

To Whom Does the Question Apply?

The Research Question Once the phenomena o f interest are identi-

Once we have decided on what question or

fied, the next step in the process is to articu-

questions we want to ask, we should then

late more specific research questions. T h a t is,

decide to whom

what about the phenomena, more specifi-

apply.

cally, are you trying to understand or predict?

have been very good at identifying situational

Although the terms

parameters

"research

questions"

the question or questions

Traditionally, social psychologists for their questions, and

they

7

INTRODUCTION AND OVERVIEW

8

often see this identification as progress in

constructs and the design t o use are often

understanding the phenomena.

Identifying

made concurrently because these decisions

potential population parameters, however, is

typically constrain each other. F o r example,

often an overlooked step in the research pro-

h o w closely does the researcher in our super-

cess. Does the research question ask about a

visor feedback example want to capture the

psychological process or outcome that we

original phenomena? D o e s he or she want to

assume is universal to all humans? T o all liv-

investigate the question in a real organiza-

ing creatures? O r is it particularly relevant to

tional setting where a supervisor gives feed-

some cultures, or to some groups within cul-

back t o subordinates? O r create an analog

tures (e.g., majority or minority, male or

to the original situation, assuming that the

female, individuals high or low in narcissism)?

relevant psychological processes generalize

In fact, when we first start to investigate a

beyond that

setting? These decisions are

particular research question, we probably do

likely to be affected by whether (and to what

not k n o w how widely the question should

degree) the researcher's goal is to draw causal

apply. Even at this point, however, we believe

inferences a b o u t the phenomena.

it is essential to ask the question because the " I don't k n o w " response is important

In social psychology, this goal typically

to

means using an experimental design in which

acknowledge when making subsequent deci-

people are randomly assigned to conditions—

sions about operationalizations and designs,

and this often means (but does not have to

as well as when interpreting findings. By mak-

mean) creating an analog for the situation.

ing sure to ask the question before conducting

These decisions should be guided by consi-

a study, we are in a better position to discover

deration o f what approach will allow the

important information about the boundaries

researcher to most clearly address the research

of a particular methodological approach as

question. In fact, however, the emphasis on

well as insights into the phenomena itself. For

experimental lab studies often precludes a

example, by asking this question, we may dis-

careful

cover that older women are less likely than

approach provides the best match for the

college-age women to conform to a unani-

question. For example, if the research question

mous majority (Pasupathi, 1 9 9 9 ) , or that lack

involves what happens as the result o f real

of choice is less likely to negatively affect

interpersonal interactions, the research design

children's motivation when they c o m e from

should

Asian American backgrounds

research question asks about what happens

(Iyengar 8c

consideration

of

whether

this

include these interactions. I f the

Lepper, 1 9 9 9 ) . These "limitations" to the

over time, the design needs to include or

assumption o f universality may in fact lead to

reflect some kind o f longitudinal component.

a

better understanding o f the

underlying

psychological process.

Because designs incorporating such elements as interactions and passage o f time are difficult and expensive to implement, as well as

Operationalizations and Design

more complicated t o analyze, researchers often settle for questions that can be answered

Only after these earlier steps have been

by more typical methods—and perhaps miss

considered can the researcher make final

the aspect o f their research questions that is

decisions about which measures will reflect

most interesting, important, or critical.

the most appropriate operationalizations o f

These early steps in the research process

their constructs, and relatedly, about the best

can

methodological setting in which to collect the

starting to generate their research questions

data. Decisions about how to operationalize

or research hypotheses after

easily be transposed, with researchers deciding on an

The Research existing methodological paradigm.

When

using an existing paradigm, the researcher

W i t h o u t a firm

Process

sense o f the

9

research

question (or questions), it also becomes diffi-

necessarily constrains w h a t questions can be

cult t o select the best method o f analysis.

asked. These constraints are a problem t o the

Unfortunately, it can be easy t o select analy-

extent that the researcher ends up drifting

ses for reasons other than that they are

away from the phenomena he or she wants

the best fit to the research question. F o r

t o understand.

example, researchers m a y use

traditional,

suggested

easily available approaches because they are

that intrinsic motivation is based on feelings

the approaches with which they (and most

o f competence that result from effectively

reviewers) are most familiar. Conversely,

For

example, many theorists

controlling one's environment (e.g., W h i t e ,

researchers may choose state-of-the-art anal-

1959). T o test this idea, pioneer researchers in

yses just because they are the latest trend.

the area (e.g., Deci, 1975) created a paradigm

The

optimal match o f design t o analytic

to study the effects o f competence feedback.

method emerges from a careful consideration

In this paradigm, college students are asked to

o f the best w a y t o answer the basic research

play a skill game or puzzle, then receive c o m -

question in a direct and comprehensible way

petence feedback that differs in terms o f

(Wilkinson

valence or the manner in which it is conveyed.

Statistical Inference, 1999).

8c the A P A T a s k F o r c e

on

M u c h o f the subsequent research used this

T h e mismatch between research question

paradigm as the starting point. As one result,

and data analysis is one reason that T u k e y

the research questions drifted from asking

( 1 9 6 1 ; reprinted in 1986) advised that the

about the nature o f intrinsic motivation itself

standard

to questions about the parameters o f the effects

doctoral students be readjusted and reevalu-

o f competence feedback.

ated. H e suggested that a m o r e effective

dissertation research process for

Sansone (1986) proposed that the role o f

approach when graduate students are devel-

competence may have been overemphasized

oping ideas for dissertation projects is for

in our understanding o f intrinsic motivation

them t o begin by thoroughly analyzing a pre-

because this typical paradigm used activities in

viously collected data set relevant to the phe-

which doing well was the goal o f engagement,

n o m e n o n o f interest. O n l y then would they

and it compared receiving competence feed-

be encouraged to generate specific research

back with no feedback. Over several studies

questions, presumably with a much clearer

using tasks and feedback that varied in their

understanding o f what kinds o f analysis are

emphasis on personal competence (i.e., that

most appropriate to particular questions. In

differed

that

from

the established

paradigm),

Sansone and colleagues found that feelings o f

way, one's initial research

question

would be fully informed by data analysis

competence enhanced intrinsic motivation

(especially o f the exploratory type), and the

primarily when competence goals were salient

typical data analysis stage o f a dissertation

at the outset o f the task (Sansone, 1 9 8 6 , 1 9 8 9 ;

project would be less prone to being mini-

Sansone, Sachau, & Weir, 1989). By using a

mized and rushed, as often happens when the

different methodological paradigm, therefore,

overriding focus is on completing the project.

this research led to some different conclusions about the nature

o f intrinsic motivation,

which ultimately led to different being asked

and

different

questions

models

Can We Answer the Question?

being

A critical stage in the research process is

constructed (e.g., Sansone & Harackiewicz,

interpreting what we have found. Can we

2 0 0 0 ; Sansone 8c Smith, 2000).

answer our original research question? T o what

10

I N T R O D U C T I O N AND OVERVIEW extent do the data support our more specific

clear by Lewin's ( 1 9 5 1 ) emphasis on "gradual

hypotheses (assuming that we have them)? It is

approximation" and by Cronbach and Meehl's

only if we have clearly articulated the research

( 1 9 5 5 ) seminal paper on construct validation, a

question that we can know how well our ques-

psychological

tion has been answered. Assuming that we have

understood in depth through bridging levels

phenomenon

becomes

best

used operationalizations, designs, and analyses

of analysis and through the use of multiple

appropriate to our question, any answer is

methodologies. At this point in the research

valuable. In fact, in discovering that a hypothe-

process, therefore, a researcher can choose mul-

sis was not supported, we often find a different

tiple directions in which to go. H e or she may

(and sometimes more interesting) answer (or

follow up some finding using the same method-

question), one that helps understand the origi-

ological paradigm (e.g., to identify potential

nal phenomena beyond our initial filter. Thus,

moderators or mediating processes) but can

having clearly articulated research questions

also make connections to other settings, popu-

does not mean that we overlook serendipitous

lations, disciplines, and researchers. As noted

findings or effects that we did not foresee. In

by Cronbach and Meehl ( 1 9 5 5 ) , the deeper the

fact, we believe the reverse. Unless the unex-

construct analysis of the phenomenon goes, the

pected finding is dramatic (or draws blood, as

more extensive and interconnected to related

in our opening example), it is only when we

work the validation becomes.

know what we are looking for that we recognize that we have found

something

For

example,

Morf

and

Rhodewalt

else.

( 2 0 0 1 a , 2 0 0 1 b ) described the paradoxical

Otherwise, our confirmation biases can sweep

lives o f narcissists, w h o , in their continual

findings into our "model" and miss informa-

efforts to construct and maintain a grandiose

tion suggesting something different.

self, engage in behaviors that

continually

Ultimately, the question is this: W h a t have

undermine and erode these efforts. T h e inter-

we learned about the original phenomena as

nal logic and coherence o f these paradoxical

the result o f this research? D o e s what we

behaviors became clear only in the course o f

have learned lead us to redefine the pheno-

conducting a program o f research involving

mena? Focus on different

different

aspects o f the

measures

and

By employing

methodological

phenomena? E x p a n d to w h o m it applies? It

approaches.

is at this point that the cycle begins again, as

matic approach, M o r f and R h o d e w a l t were

this

program-

we further differentiate the phenomena o f

able to identify h o w narcissists' thoughts,

interest, perhaps moving t o different, related

feelings, and motivations interrelate within

phenomena, or perhaps changing w h a t we

their self-system t o create the prototypic, self-

see as the phenomena. W e believe it is criti-

defeating

cally important

this personality syndrome.

that researchers recognize

that at this point in the process, they have

behavioral patterns

Although

observed

in

the idealized process involves

learned only as much as their methodological

cumulative and programmatic research that

decisions allowed them to. W e tend to recog-

casts the nomological net wider and wider,

nize this important fact only when we are

making connections to other disciplines within

using innovative methodologies (e.g., func-

and

tional magnetic resonance imaging [ f M R I ] ,

knowledge of methodological paradigms and

outside

psychology typically involves

Implicit Associations Test [IAT])—it is then

analysis strategies that are not part of our usual

that we tend t o become aware that the nature

training. Making these necessary connections

of our research questions is shaped by the

therefore may necessitate

possibilities o f a particular method.

collaboration. M o r e o v e r , as reviewers o f

multidisciplinary

W h a t we conclude about our question will

others' work, we are usually ill equipped to

determine the next steps that we take. As made

evaluate the quality and level of contribution

The Research made

by

this

research.

As

one

result,

researchers tend to stick to their familiar

interdisciplinary

Process

11

(e.g., social neuroscience,

social development, social psychology and the

paradigms, particularly when they are con-

Internet) and applied (e.g., program evalua-

cerned about getting (or keeping) a j o b .

tion, health, education, organizations) focus.

Nevertheless, researchers should be encour-

Second, we have attempted t o maintain

aged to spread their wings and venture beyond

the top-down perspective within each chap-

their home territory. T h e potential benefits we

ter. T h u s , all contributors were asked to fol-

reap from applying different epistemological

low a general template in which they first

and methodological lenses to our phenomena

describe a concrete and relevant social psy-

of interest can be great—as can the fruits o f

chological research problem (or problems)

multidisciplinary collaborations, no

and then discuss relevant

matter

methodological

how difficult they are initially to establish.

issues in the

Science at its best is not a solitary enterprise; it

Chapters emphasize the conceptual basis o f

is cumulative, as each investigation adds its

the methodology, with an explicit focus on

context of that

problem.

unique contribution to the puzzle of under-

the meaning o f data when obtained via a

standing a phenomenon (or interrelated set

particular methodology. Statistical analysis is

of phenomena).

considered a process in service o f research design, and it is discussed to the extent that it helps to illuminate the distinct meaning o f

ORGANIZATION OF THIS HANDBOOK

data obtained through a particular methodological approach or design. Contributors to this volume were selected

Organizing Principles

because they have developed expertise on particular

methodological approaches

that the decision to use a particular method-

issues

social

As we hope is clear at this point, we believe

in

psychology—and,

or

more

ological approach is optimally made when

important, they did so in response to their

grounded in careful considerations o f the

attempts to discover the best way t o under-

"big picture" o f a program of research. Thus,

stand

methodological decisions are tied inextricably

interested them. Thus, these researchers have

to what the researcher, ultimately, wants to

let the research questions guide the methods,

know. W e have used this "top-down" per-

rather

spective to develop and organize this hand-

researchers discuss traditional and state-of-

book

the-art

in

two

ways.

First,

the

overall

the psychological p h e n o m e n a

than

the

reverse.

These

that

expert

methodological advances by

first

organization o f the handbook parallels the

outlining concrete research phenomena and

picture o f the research process that we have

related questions o f interest and then showing

discussed. As a result, the beginning chapters

h o w these questions may be best answered

address issues related to identifying and defin-

through design and analysis decisions.

ing phenomena, research questions, and populations. Middle chapters address issues related to design and data analysis, and later chapters

Specific Organization

address issues related to expanding the original social

psychological

questions

to

other

disciplines within and outside psychology.

Part II: Fundamental in Social

Psychological

Issues Research

Because of this top-down approach, the book

In the next part, chapters highlight the set

includes innovative chapters such as those

of decisions that must be made no matter the

on ethics, culture and diversity, and individual

particular type o f methods chosen. One set o f

differences as well as chapters that have an

decisions involves the initial

assumptions

12

INTRODUCTION AND OVERVIEW upon which all subsequent methodological

the implications o f explicitly acknowledging

issues rest, including the role o f theory in

and incorporating the heterogeneity of a popu-

guiding the research questions, whether one

lation through research design and measure-

questions

ment o f key constructs. Miller's chapter on

involves objective or constructive processes,

believes that

"testing" research

cultural sensitivity addresses these issues in the

the level o f analysis at which researchers

context o f potential variability among different

choose to examine their research questions,

populations

and so on. Thus, in the first chapter in this sec-

populations, and it provides important insights

and among subgroups

within

tion, C o o k and G r o o m address some of the

about why diversity o f participant populations

issues associated with these kinds o f decisions

is often

and suggest that there is a gap between h o w

research. Shoda's Chapter 6, on personality

social psychologists should

and individual

make these deci-

ignored

in

social psychological

differences, addresses these

sions and how they typically do so, posing

issues in the context o f potential individual

some challenges for researchers to consider.

variability within groups. This chapter also dis-

Chapter 3 , by Kimmel, addresses the impor-

cusses different ways to conceive of and mea-

tant ethical issues surrounding the decisions

sure

about particular research questions and asso-

differences and various aspects o f situations.

ciated methodological approaches in social

the

interactions

between

individual

Section B—"Operationalizing the

Con-

psychology. For example, how do and should

structs: Deciding W h a t to Measure, W h y , and

ethical considerations shape the nature o f the

How"—addresses

research questions it is possible to ask? H o w

operationalizing constructs, both as predic-

do these ethical considerations change as the

tors (e.g., manipulated or naturally occurring)

the decisions related

to

methodological approach changes, and what

and as outcomes. These chapters address the

are some o f the important ethical considera-

strengths and the weaknesses associated with

tions for the future? In Chapter 4 , Fiske con-

different kinds o f measures to help readers

siders how to make these decisions in the

choose optimal measures for testing a given

context o f generating a long-term program o f

research question or questions for a given

research, rather than focusing on a single

population. These chapters cover some o f the

study. She also discusses the balance between

more traditional ways to operationalize con-

generating a long-term program, publication

structs as well as address some newer distinc-

and funding issues, and the importance of fol-

tions that have emerged as important in the

lowing one's own interests or passions.

field. For example, Wegener and Fabrigar's Chapter 7, on quantitative measures, focuses on traditional psychometric approaches to

Part III: Design

and

Analysis

capturing social psychological phenomena in

Part III focuses on the set of design and

terms o f the amount o f the construct in ques-

analysis decisions researchers must make when

tion. Thus, this chapter includes discussions

they adopt given methodological approaches.

of self-report (e.g., scales) as well as nonself-

Section A o f this part—"Implications o f a

report (e.g., behavioral, archival) quantitative

Heterogeneous

for

measures and addresses key lessons from the

W h o m to Test the Research Question(s), W h y ,

survey methodological literature about scale

and How"—addresses issues surrounding the

construction and design. In contrast, the sec-

study populations and the implications of

ond chapter on qualitative measures (King,

assuming an "average response" when the

Chapter 8) focuses on attempts to capture

Population:

Deciding

population is not homogeneous. Chapters in

social psychological p h e n o m e n a

this section also consider the reverse: that is,

more

open-ended

data

through

sources.

King

The Research

Process

addresses issues related t o using self-report

traditional experimental manipulations. The

and nonself-report qualitative measures (e.g.,

chapter includes discussion of issues relevant to

behavioral acts, narratives, interviews), as

that goal.

well

as issues relevant

to

transforming

In

contrast

t o these m o r e

traditional

these measures through coding. Kihlstrom's

approaches, the next three chapters discuss

Chapter 9 addresses the distinction between

newer approaches that often involve some

implicit and explicit measures o f an underly-

combination and extension o f traditional

ing construct, and what that distinction might

approaches. T h u s , Chapter

mean for understanding the construct. It also

Biesanz,

and

Kwok,

1 3 , by West,

addresses

designs,

discusses this distinction in light of current

assumptions, and related analytic strategies

popular implicit measures (e.g., I A T and reac-

for experiments whose focus is on examining

tion times). In the final chapter in this sub-

people

section, Hoyle and Robinson (Chapter 1 0 )

Chapter 1 4 , by Gonzalez and Griffin, describes

address the distinctions related to under-

conceptual issues and data-analytic options

within

contexts

and

over

time.

standing moderation and mediational pro-

related to designs that involve people inter-

cesses when describing the link between a

acting, primarily with one other person. The

variable and an outcome, as well as dis-

final chapter in Part III covers the empirical

cussing key design decisions that affect inter-

aggregation of results across studies

pretation o f these processes.

meta-analysis

(Wood

and

using

Christensen,

Section C o f Part III—"Research Designs:

Chapter 1 5 ) . In this case, the focus is on exam-

Deciding the Specific Approach for Testing

ining studies across people, time, and contexts.

the Research Question(s), W h y , and H o w " — addresses decisions concerning the

specific

research design. This subsection includes con-

Part IV:

Emerging

sideration of the more traditional approaches

Interdisciplinary

Approaches

but also includes emerging, state-of-the-art

The chapters in Part I V address how the

perspectives. F o r example, Chapter 1 1 , by

sets o f decisions described previously play out

Haslam and M c G a r t y , addresses traditional

in some o f the newer, cutting-edge topics and

experimental designs as a way to test social

interdisciplinary approaches in social psychol-

psychological research questions. T h e strength

ogy. These chapters highlight recent attempts

of these kinds o f designs is the ability to iden-

to expand beyond traditional social psychol-

tify causation. Haslam and M c G a r t y include

ogy

discussion of issues related to that goal (e.g.,

emerging technologies and

demand characteristics, random assignment,

approaches from other areas both inside and

experimental control) but also include discus-

outside psychology. Although they do not pre-

sion o f the necessary trade-offs made between

sent an exhaustive list, these chapters highlight

internal

external validity. Similarly,

the benefits o f cross-fertilization when the

Chapter 1 2 , by M a r k and Reichardt, also

strengths o f different disciplines are combined

includes a discussion o f trade-offs between

and integrated. F o r example, Birnbaum's

internal and external validity, but it does so in

Chapter 1 6 addresses research that combines

and

questions and methods to incorporate methodological

the context of quasi-experimental and correla-

social psychology with the technological

tional designs as means o f testing social

world o f the Internet. Birnbaum addresses

psychological questions. T h e strength o f these

h o w traditional social psychology questions

kinds o f designs lies in the ability to identify

may be addressed

predictive

of

online, and he explicitly compares studies

variables that may not lend themselves to

conducted via the Internet with traditional

relationships

among

sets

through data collected

13

14

|

INTRODUCTION AND OVERVIEW

lab-based studies in terms o f likely populations, procedures,

and

ethical

programs and often involves a delicate balance

considerations.

between the goals of the organization conduct-

Chapter 1 7 , by Cacioppo, Lorig, Nusbaum,

ing the program and the social psychologist

and Berntson, focuses on the biological bases

researcher (Maruyama, Chapter 19). Typically,

of and connections to social psychological

evaluation o f these programs requires combin-

phenomena. T h e authors consider some o f the

ing knowledge from multiple areas within and

newer neuroimaging and other neuroscience

across disciplines, and these combinations are

techniques and how these may be used to

rarely tested in laboratories.

inform

social psychological questions

or

The other three chapters in Part V focus on

issues. This chapter also includes a considera-

particular domains o f applications, rather than

tion of how our understanding o f biological

on a given methodological approach such as

systems may be enhanced by including social

program evaluation. These chapters provide an

psychological phenomena.

1 8 , by

overview of the application of social psychol-

Pomerantz, Ruble, and Bolger, addresses the

ogy questions and methods to important and

Chapter

inclusion o f developmental questions when

highly relevant domains: clinical and health

examining social psychological phenomena,

psychology (Salovey & Steward, Chapter 2 0 ) ,

using the metaphor o f the video camera. T h e

organizations

authors address the implications o f the conti-

Chapter 2 1 ) , and education (Harackiewicz &

(Thompson, Kern, &

Loyd,

nuity and discontinuity o f social psychological

Barron, Chapter 2 2 ) . T a k e n together, these

phenomena across the life span. They also

chapters highlight the commonalities across

address how including social psychological

applications while also addressing the chal-

approaches to developmental questions forces

lenges and contributions unique to each area o f

a shift in thinking about development

application.

as

an intraindividual process to thinking about development

as embedded

in the

social

dynamics that make up our everyday life.

Part V: The

W e are inherently social beings, and as such

Application

of Social Psychology Methods

CONCLUSION

to Other

the goal o f understanding h o w a person can

and Its

affect

Domains

and

be affected

by others

captures our attention, our thoughts,

often our

In the final part of the book, chapters

feelings, and our concern. T h a t goal is the

address the applications o f social psychology

subject matter o f great (and little) b o o k s ,

methods and knowledge to "real world" set-

movies, television shows, computer games,

tings and domains outside social psychology.

and advice columns. Clearly, social psychol-

These relatively shorter chapters include discus-

ogists study things that we want to k n o w

sion of issues relevant to the typical popula-

about each other and ourselves, and do so in

tions, measures, designs, and analyses used in

creative ways.

each domain. These chapters also briefly

In that context, we hope that this hand-

discuss the added requirement that is often

b o o k will allow the field t o continue its

an important part o f applying social psychol-

strong tradition but also to grow to include

ogy approaches to the "real world": that is,

newer

the need to create and maintain relationships

approaches. Even more important, we hope

with outside agencies and organizations. For

that this b o o k helps social psychologists con-

example, program evaluation allows researchers

sider different kinds o f research questions

to study the effectiveness of treatments

from the ones they may typically ask by tying

or

methodological developments

and

The Research

Process

\

the discussion o f methodological details to

aware o f the critical methodological decisions

the "big picture" o f a research program. In

that must be made in pursuit o f those ques-

addition, we believe that the chapters in this

tions. W e have learned a lot ourselves from

h a n d b o o k provide an excellent starting point

the many wonderful chapters in this volume—

for researchers w h o want to pursue these

and we k n o w younger brothers everywhere

newer questions by helping them t o become

will sleep m o r e soundly!

REFERENCES Altman, I. (1988). Process, transactional/contextual, and outcome research: An alternative to the traditional distinction between basic and applied research. Social Behavior, 3, 2 5 9 - 2 8 0 . Asch, S. E. (1946). Forming impressions of personality. Journal of Abnormal and Social Psychology, 41, 2 5 8 - 2 9 0 . Cronbach, L. J . , & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 2 8 1 - 3 0 2 . Deci, E. L. (1975). Intrinsic motivation. New York: Plenum. Festinger, L., 8c Carlsmith, J . M . (1959). Cognitive consequences of forced compliance. Journal of Abnormal and Social Psychology, 58, 2 0 3 - 2 1 0 . Gergen, K. J . (2001). Psychological science in a postmodern context. American Psychologist, 56, 803-813. Hovland, C. I., Janis, I. L., & Kelley, H. H. (1953). Communication and persuasion. New Haven, CT: Yale University Press. Iyengar, S. S., 8c Lepper, M. R. (1999). Rethinking the value of choice: A cultural of Personality and Social perspective on intrinsic motivation. Journal Psychology, 76(3), 349-366. Lewin, K. (1951). Field theory in social science: Selected theoretical papers (D. Cartwright, Ed.). New York: Harper 8c Row. Morf, C. C , & Rhodewalt, F. (2001a). Unraveling the paradoxes of narcissism: A dynamic self-regulatory processing model. Psychological Inquiry, 12, 177-196. Morf, C. C , 8c Rhodewalt, F. (2001b). Expanding the dynamic self-regulatory processing model of narcissism: Research directions for the future. Psychological Inquiry, 12, 2 4 3 - 2 5 1 . Pasupathi, M . (1999). Age differences in response to conformity pressure for emotional and nonemotional material. Psychology and Aging, 14, 170-174. Petty, R. E., Fleming, M . A., 8c Fabrigar, L. R. (1999). The review process at PSPB: Correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin, 25, 188-203. Renninger, K. A. (2000). Individual interest and its implications for understanding intrinsic motivation. In C. Sansone 8c J . M . Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 3 7 5 - 4 0 7 ) . San Diego: Academic Press. Sansone, C. (1986). A question of competence: The effects of competence and task feedback on intrinsic interest. Journal of Personality and Social Psychology, 51, 918-931. Sansone, C. (1989). Competence feedback, task feedback, and intrinsic interest: An of Experimental Social examination of process and context. Journal Psychology, 25, 3 4 3 - 3 6 1 . Sansone, C , 8c Harackiewicz, J . M. (Eds.). (2000). Intrinsic and extrinsic motiva­ motivation and performance. San Diego: tion: The search for optimal Academic Press.

INTRODUCTION AND OVERVIEW Sansone, C , Sachau, D. A., & Weir, C. (1989). Effects of instruction on intrinsic interest: The importance of context. Journal of Personality and Social Psychology, 57, 819-829. Sansone, C , 8t Smith, J . (2000). Interest and self-regulation: The relation between having to and wanting to. In C. Sansone & J . M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 3 4 1 - 3 7 2 ) . San Diego: Academic Press. Schachter, S., & Singer, J . E. (1962). Cognitive, social and physiological determinants of emotional state. Psychological Review, 69, 379-399. Sue, S. (1999). Science, ethnicity, and bias: Where have we gone wrong? American Psychologist, 54, 1070-1077. Tukey, J . W. (1961). Statistical and quantitative methodology. In D. P. Ray (Ed.), Trends in social science (pp. 84-136). New York: Philosophic Library. Tukey, J . W. (1986). Statistical and quantitative methodology. In L. V. Jones (Ed.), The collected works of John W. Tukey: Vol. III. Philosophy and principles of data analysis, 1949-1964 (pp. 143-186). Belmont, CA: Wadsworth. White, R. W. (1959). Motivation reconsidered: The concept of competence. Psychological Review, 66, 2 9 7 - 3 3 3 . Wilkinson, L., & the APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.

Part II FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH

C H A P T E R

2

The Methodological Assumptions of Social Psychology The Mutual Dependence of Substantive Theory and Method Choice THOMAS D . COOK Northwestern

University

CARLA GROOM University of Texas,

Austin

Assume . . . 1. To take up or adopt. . . . 2. To undertake. . . . 3 . T o arrogate to oneself. . . . 4. To take for granted; suppose to be a fact. . . . What a debater postulates he openly states and takes for granted without proof; what he assumes he may take for granted without mention. —Webster's

Comprehensive

INTRODUCTION

Dictionary:

International

Edition

" W h a t is social psychology?" This is because it does not prioritize among

If you open any edition of the Handbook Social

Psychology

(1984)

of

or any recent textbook on

its constitutive

elements, even though different

components

have been emphasized at different times in the

social psychology, you will almost certainly

history o f social psychology. M o r e fundamen-

find G. W . Allport's 1 9 5 4 definition of the field

tally, a full account o f any scientific commu-

as "an attempt to understand and explain how

nity's belief system will likely reveal some

the thought, feeling, and behavior of individu-

purposes that are latent and thus not amenable

als are influenced by the actual, imagined, or

to explicit definition (Polanyi, 1 9 5 8 ) .

implied presence o f others" (p. 5 ) . As wide-

A more complete understanding o f social

spread and useful as Allport's description is, it

psychology is achieved through

provides only part o f an answer to the question

inspection o f the research practices o f those

inductive

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH who

call themselves social psychologists.

community o f social psychologists. T o take full

the effective

account o f this heterogeneity would entail dis-

boundaries o f all fields are delimited by the

cussing all the forms of research practice found

tacit agreements that a specific community o f

in social psychology today—from

scholars makes about the norms regulating

applied research, from lab to field settings,

how research practice should be carried out.

from quantitative to qualitative work, and

Some o f the preferences are explicit, but others

from single studies to research syntheses. It

are implicit. This chapter discusses the most

would also have to consider generational dif-

salient explicit postulates about h o w to do

ferences in how social psychology is pursued.

According to Kuhn

(1970),

basic to

social psychology while also trying to force

Space limitations preclude doing any of this

out some of the field's more latent assump-

with integrity.

tions about method.

An alternative would be to analyze only

Identifying w h o social psychologists are is

elite and/or cutting-edge methodological prac-

tricky. There are several social psychology

tice—the strategy adopted by many writers o f

communities, and they only partially overlap.

chapters in this volume. Although this would

T h e most distinctive consists o f researchers

be interesting and important, it would not cap-

belonging to professional societies such as the

ture what most contemporary social psycholo-

Society for Experimental Social Psychologists,

gists actually do in most of their work. They

the

Society

for

Personality

and

Social

Psychology, the Society for the Psychological Study o f Social Issues, and Division 8 o f the

are journeymen and -women, not pioneers in the construction or use of novel methods. The approach we eventually adopted is to

American Psychological Association. These

identify and explore modal

individuals

North

practice, hoping that more readers will recog-

American psychology departments, and their

nize themselves and their work in such an

work is the main focus o f this chapter.

account. W e acknowledge that the account we

are mostly located in

Their operating

assumptions,

however,

methodological

offer depends on an ideal type characterization

are not identical with the assumptions o f

of methodological practice, so there are legiti-

researchers w h o call themselves social psy-

mate exceptions to every generalization we

chologists and w h o w o r k in American soci-

make.

ology departments

or business schools or

There is a second reason to focus on modal

even those w h o w o r k in psychology depart-

practice. T h e distribution

ments in Europe. Although

cal practices within social psychology seems

it would

be

of methodologi-

instructive to explore the differences among

quite leptokurtic—that is, the proportion o f

these various communities, we do not have

researchers using what we later describe as

the space to do so here.

modal practices seems to be very high relative

Restricting ourselves to the practice o f social

to the proportion working in the tails of the

psychologists working in American psychol-

distribution, where practice is more idiosyn-

ogy departments leads to an unfortunate solip-

cratic. T o illustrate this, consider the number

sism. Both authors o f this chapter are social

and heterogeneity o f method chapters in the

psychologists working in the United States and

various Handbooks

so are immersed in the field's assumptions. W e

There were 9 such chapters in the 1 9 5 4 edition

of Social

Psychology.

cannot hope to take the methodological tem-

(Lindzey, 1 9 5 4 ) , covering many different kinds

perature o f the subdiscipline as well as future

of method. There were 1 0 in Lindzey and

historians o f science will.

Aronson ( 1 9 6 9 ) , just as varied, but there were

T o add to this complication, scientific practice is heterogeneous even within the target

only 6 in the 1 9 8 5 edition and 4 in Gilbert, Fiske, and

Lindzey ( 1 9 9 8 ) — o n e each

on

The Methodological

Assumptions

of Social Psychology

\

experimentation, surveys, measurement, and

A further difficulty arises in seeking to

data analysis. Missing by that fourth edition

define methodology. Is it simply method, a

were chapters on observational methods, con-

group o f techniques for data collection and

tent analysis, program evaluation, and many

analysis? O r does it enter into a chicken-and-

other topics. M a n y social psychologists aspire

egg relationship with

to methodological eclecticism, but most o f the

making their analytic separation unfruitful?

research programs that are achieved employ a

substantive

theory,

Individual methods were designed to answer specific types of questions that implicitly place

restricted set o f methods. O u r description o f the methods constitut-

a higher priority on some forms o f theory than

the

others. Thus, scientific surveys were created to

method chapters in recent editions o f the

generate multi-attribute descriptions o f well-

ing modal practice relies in part on Handbook

of Social Psychology.

But we rely

defined populations, thus privileging theories racial, gender, class, or even national lines.

the

Social

Experiments were designed to describe the

and

effects o f manipulable causal agents, originally

well

in agricultural research, where the pragmatic

Psychology

of

Personality

and

(JPSP) and the Personality

Social Psychology

Bulletin (PSPB)—as

population

along

reports from the most widely read journals— Journal

compare

groups

even more on our own analysis o f research

that

as on content analyses o f these reports where

goal was t o identify whether

available (e.g., Rozin, 2 0 0 1 ) .

worked better than something else—a purpose

something

But given the editorial policies and the

for which substantive theory need not play any

system of rewards in science today, journals

role. Although experiments always privilege

cannot provide perfect data on the method

causal questions o f an "if-then" kind, they are

practices actually used in a particular field.

theory-relevant only to the extent that they can

Journal writers have to impress gatekeeping

help distinguish between competing theories or

editors and reviewers. In part, they seek to do

otherwise illuminate a single theory.

this by illustrating their knowledge of and

Path analysis was designed to ascertain the

adherence to the community's currently domi-

degree of correspondence between a substan-

nant methodological norms and theoretical

tive theory o f a phenomenon and data col-

forms. From these incentives follow sins o f

lected about that same

omission and commission that inadvertently

the theory usually being specified in the form

distort accounts o f the original intentions o f a

o f multiple, causally ordered determinants o f

research project and of the procedures actually

the outcome. Path analysis seeks to identify

used in it.

the complexly ordered causes o f a given effect,

Well-constructed random surveys that ask social psychologists about their practice would

phenomenon—with

in contrast to experiments, which seek to describe some o f the effects o f a given cause.

provide a useful supplement to analyses of the

Other purposes have accreted to each o f

methods detailed in individual journal articles,

the general methods above, yet the original

but to our knowledge, these surveys have not

purposes still remain dominant and linked to

been done recently. W e are forced, therefore,

a unique form o f substantive theory. T h e

to supplement the journal analysis with our

kinds o f research questions posed are partial

own firsthand experience of social psycholo-

products of the tools at hand. T h e implication

gists' behavior (inevitably including our own),

of this is that if a given tool is not in the reper-

with

toire o f a researcher or her circle o f profes-

other social psychologists' informal with

sional acquaintances, then the type o f question

published critiques o f the field—even those

corresponding to that tool is not likely to be

with somewhat o f a whistle-blowing flavor.

asked. And if asked, the question certainly will

accounts of their own behavior, and

21

22

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH be answered poorly. The assumptions behind

deals with group dynamics, emphasizing how

the dominant forms o f theory in a field thus

small social systems evolve, respond to their

are inextricably tied t o the

assumptions

internal and external environments, and then

behind its most widely used methods. N e w

change their borders and operations. Finally,

researchers are routinely taught that theoreti-

also borrowed from psychology, writ large,

cal questions precede and determine method

were theories o f social cognition. Variously,

choices, but in actual research practice the

these emphasize individuals as rational know-

reverse causal flow is also to be found—

ers, as cognitive misers, as cognitive tacticians,

method capabilities determine the kinds o f

or as quasi-automata programmed to respond

questions that get tested and hence the kinds

to social primes.

of theory that are generated.

Social psychology places strong normative emphasis on testing hypotheses deduced from substantive

Wilson, &

ories come from within theory classes such as

METHOD

those described above. Thus, Higgins's selfdiscrepancy theory ( 1 9 8 9 ) is part o f a broader

The Types of Theory That Social Psychologists Construct "theory"

(Aronson,

Brewer, 1 9 9 8 ) . T h e most commonly tested the-

THE HYPOTHETICO-DEDUCTIVE

If

theory

were

easy

to

class o f social comparison theories, postulating that people's goals motivate their actions, not

explicate,

philosophers would not study meta-theory.

directly,

but

via

observed

discrepancies

between these goals and one's present state.

But because they do, and because social psy-

Sherman's encoding flexibility model of stereo-

chology is intimately involved with substan-

type function (e.g., Sherman, Lee, Bessenoff, &

tive theory and its testing, we must explore

Frost, 1 9 9 8 ) suggests ways in which stereo-

what kind o f theory social psychologists test.

types regulate the processing of information

This is not easy.

that is or is not consistent with the stereo-

"Grand" theories o f the reach of, say,

type—all in the service o f maximizing infor-

Weber or Durkheim in sociology or o f the neo-

mation

classical model in economics have had a short

categorized as coming from within the broader

life in psychology. This was true o f psychoan-

class of cognitive theories and even

alytic theory, systems theory, and behaviorism,

within the subclass o f theories emphasizing

and it is likely to be the fate of a pure cognitive

human limits to information processing.

science that excludes biology and culture.

gain.

Sherman's

theory

is

best from

At first glance, evolutionary psychology

T o be relevant to social psychology, any

seems to be a much broader type o f theory

grand theory has to be adapted to social refer-

than the theory o f Higgins or Sherman. It pro-

ents. Doing so has prompted the development

ceeds by identifying key social problems that

of the various theory types with which social

our ancestors must have faced before deducing

psychologists have worked over the years.

from these problems the kinds o f psychological

Cognitive consistency theories are one such

mechanisms that might have evolved to solve

type, with their emphasis on tension states

them. For example, did women's need to

that demand resolution, as with dissonance or

choose mates who could provide sufficient

balance theory. Another type consists o f social

resources to rear a child lead to hard-wired sex

on

differences in mate preference that can then be

social modeling and other ways o f acquiring

learning theories, with their emphasis

experimentally tested for in 21st-century men

social knowledge, as with research on attitude

and women? O n grounds like these, evolution-

formation and change. Another type o f theory

ary psychology has been offered as a unifying

The Methodological

Assumptions

of Social

Psychology

"meta-theory" for all of social psychology

prevalent in psychology than are theories of the

(Buss & Kenrick, 1 9 9 8 ) . Y e t to do this, evolu-

effects o f this same cause. As a result, little

tionary psychology would have to subsume or

attention is given to providing a full explana-

otherwise explain theories within the other the-

tory model o f any single aspect o f social behav-

ory types such as cognitive consistency, social

ior, thought, or emotion. Does X affect Y ? is

learning, and social cognition, not to speak o f

the question; not W h a t are all the causes of Y ?

all the exemplars that fall within each o f them.

Some critics (e.g., R o z i n , 2 0 0 1 ) have

But it does not and cannot do that. In practice,

argued that social psychology suffers not from

therefore, evolutionary psychology functions

testing too narrow a form o f theory, but from

just like any other lower-order theory in social

too much theory altogether. Preoccupation

psychology—as a source o f hypotheses about

with theory can block exploratory research,

the validity of parts o f that theory.

and exploratory research is certainly not

O n e striking feature o f theories in social

modal in social psychology. Such preoccupa-

psychology is h o w many o f them postulate a

tion can also encourage perseveration and

linear flow o f influence, irrespective o f h o w

confirmation

many variables are thought

theory is no longer productive (Greenwald,

to

intervene

bias long after

a

particular 1986),

between a causal agent and its distal effect.

Pratkanis, Leippe, & Baumgardner,

(The mode, by the way, seems to be one,

particularly when investigators persist with

though some path analytic models

failing ideas to which their names are publicly

have

more.) W i t h its multiple connections, feed-

linked. W h e n theories are o f limited general-

back loops, and other dynamic properties,

ity, preoccupation with them can retard the

systems

theory

involves

quite

different

development o f broader theories that might

assumptions about the course o f influence.

reconcile disparate

But social psychologists rarely use systems

within

theory as the explicit meta-theory organizing

(Moscovici, 1 9 7 2 ) . Thus, the preoccupation

relationships a m o n g constructs.

with testing theory o f a particular type may

It is not clear why the linear flowchart is at

some

findings

new

and

theories

overarching

theory

not be cost-free.

the heart of most substantive models in social

These issues aside, normative practice in

psychology, but it may have to do with social

social psychology is clear. Social psychologists

psychologists' distinct preference for the exper-

use and apotheosize substantive theory. They

iment. The experiment was designed to assess

construct it at some level that is certainly not

h o w one deliberately manipulated

variable

"grand" and probably is not "middle level" if

marginally affects some measured outcome.

the latter is understood as theory at the level

Systems' notions are more multivariate. They

of, say, social learning, cognitive consistency,

depend on longer time lines and reciprocal

or group dynamics. M o s t o f the specific theo-

causal relations, and they do not necessarily

ries that social psychologists examine come

privilege a single causal agent. This raises a

from within such broad theory classes. Thus,

question: Does the method social psychologists

theories o f rational actors, cognitive misers,

use most often inadvertently constrain the

cognitive tacticians, or semi-automata react-

form o f theory that gets tested?

ing to primes are all examples from within the

Does it also affect a second salient difference

class o f cognitive theories. Tests of social cog-

between theory in social psychology and the

nition as a class are rare. T o our knowledge,

other social sciences—the relative paucity of

no one has yet conditioned varying numbers

theories that seek to detail the multiple sources

or constellations o f social attributes to a neu-

of influence on a given outcome? Theories of

tral stimulus in order to examine how they

the determinants o f the cause are much less

affect

subsequent

information

processing.

23

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Instead, the theories typically tested are those

C o o k , & Campbell, 2 0 0 2 ) ; that is, it specifies

of modest generality, at what

what should happen to Y if X is deliberately

we

might

roughly call a "lower middle" level.

varied. This preference entails that Y is validly measured and X validly manipulated. O f these

The Theory-Hypothesis

two tasks, the manipulation o f X usually is

Link

more problematic. T o illustrate why this is

Popper's ( 1 9 3 4 / 1 9 5 9 ) formulation o f the

the case, consider priming subjects to elabo-

requires

rate on a persuasive message. It is inevitable

deducing a clear hypothesis from some theory

that priming will be imperfectly linked to

(whatever its reach), and then validly testing

elaborated

this hypothesis. T h e preferred hypotheses are

members o f the experimental group may not

hypothetico-deductive

method

information

processing.

Some

those that probe the theory's core postulates;

do any elaborating at all, while others may do

and the preferred tests o f such hypotheses

little elaborating or not the kind that is under

depend on setting up a situation so that no

test. Moreover, some o f the control subjects

other theory predicts the same pattern o f data

may spontaneously elaborate even if not the

as the hypothesis under test. T h e explicit task

same message content on which members of

therefore is to differentiate the target theory

the treatment group elaborate.

from other theories by virtue o f the closeness

In ordinary language, naming a cause ( X )

of the fit between the data obtained and the

requires using abstract language to describe

data uniquely predicted by the target theory

those components within a treatment manipu-

(Cook & Campbell, 1 9 7 9 ) .

lation that, on theoretical grounds, are thought

Doing all this requires highly explicit the-

to bring about an effect. However, in experi-

ory with prioritized postulates and, hence,

mental practice (and logic), the causal agent is

prioritized hypotheses. Basing hypothesis for-

always a contrast and not what happens in a

mulation on abstract language, insight, or

single group. Usually, it is the contrast between

intuition alone is not enough; nor is it enough

the theoretically specified components of a

to claim that a hypothesis tests some part o f a

treatment and whatever happens to controls.

theory. At issue is testing the central assump-

Given that the components o f the treatment

tions that identify a given theory and make it

and control interventions can overlap, false

substantively unique.

negative conclusions about cause can ensue if

M o s t journal studies report tests o f theory-

researchers insist on describing the active

derived hypotheses, but these are rarely the

causal agent with recourse only to what hap-

most identity-conferring

from

pens in the treatment group. T h e situation is

that theory. In part, this is because most the-

even more complicated if the control group

hypotheses

ories are not that explicit. In part also, it is

includes components that are irrelevant to the

because time and experience sometimes are

theory under test but that are correlated with

needed to decide which theoretical postulates

both the outcome and knowledge o f being

are key. As a result, the modal journal study

included in a study. Then, both false positive

does not seek to validate (or invalidate) an

and false negative causal conclusions can result

entire theory. Instead, it probes whether the

from describing the causal agent exclusively in

obtained results are consonant with some

treatment group terms.

part o f a theory, or it elucidates some o f the

To

avoid these causal pitfalls

requires

boundary conditions under which the theory

(a) carefully explicated substantive theory about

does and does not hold.

the cause and h o w it is related to the presumed

T h e modal hypothesis that gets tested is

effect, (b) planned treatment contrasts that are

causal in the activity theory sense (Shadish,

as large as possible, (c) valid (and hence

The Methodological

Assumptions

of Social

Psychology

also reliable) measures to assess whether the

are initially identical except for the presence

experimental manipulation is varying what it

o f the treatment under test.

is supposed to vary, and (d) collecting data on

After a correlation has been demonstrated

these measures across all the study groups,

between a demographic variable and some

including no-treatment controls. T h e era o f

possible mediator o f an "effect," the preferred

"black b o x " experimentation is over. At a

practice in social psychology is not to derive

minimum, researchers need to realize that

the complex statistical interaction predictions

all causes are comparative and to demonstrate

mentioned above. Rather, it is to create a the-

that only small shortfalls occur between the

ory o f the mediator and then to turn this medi-

comparative cause as it is conceptualized,

ator into an independent variable. Thus, if data

implemented, and measured.

indicate that younger teachers are more effec-

The preference for an activity theory o f

tive than older ones, this might lead to the

cause also entails an implicitly negative valua-

hypothesis that the younger teachers have

tion o f certain other kinds o f theory, especially

higher expectancies for their students and to

theory a b o u t

the corollary hypothesis

demographic

variables

that

cannot be manipulated directly. Race, gender,

that

it is these

expectancies that drive greater learning. Both

class, and age are examples o f this. Sometimes,

of these hypotheses can then be tested directly

though, theory is specific enough to detail the

in order to develop a more complete substan-

individual or social processes that are thought

tive theory o f teacher expectancies.

to mediate a demographic variable's suspected

O n e rationale for turning presumed medi-

effects. In this case, knowledge o f these mech-

ating variables into tested independent vari-

anisms can be used to collect data that test

ables is that most mediators are manipulable

hypotheses about the conditions under which

whereas

a demographic variable varies in the size or

Another rationale is that theories o f media-

direction of its relationship to a given outcome.

tors are more general because the same medi-

Essentially, this involves testing a statistical

ator often can be activated many ways, both

interaction hypothesis that specifies h o w the

in the laboratory and in the real world. Going

demographic variable and mediator mutually

back to expectancies for student

influence the outcome. Formulating

mance,

more

demographic

many

older

variables are

teachers

not.

perfor-

have

high

complex contingency hypotheses is a strategy

expectancies

that is also heavily used in evolutionary psy-

younger teachers have higher expectancies on

chology, another field in which researchers

the average. In addition, teachers are not the

cannot manipulate variables easily.

for

their

students

even if

only ones with education-relevant expectan-

T h e key to the method's success is famil-

cies—most parents also have them. Thus, as a

iar, however. Hypotheses still need t o be

causal agent, educational expectancies have a

made that are so novel in their implications

potentially broader reach than teacher age.

that no other theory makes these same pre-

A third rationale is that theories featuring

dictions. It is not enough merely to demon-

individual-level psychological mediators gen-

strate a correspondence between hypothesis

erally are more interesting to psychologists

and data: N o alternative interpretations must

than

remain viable. W i t h demographic variables

variables such as age, race, and gender. There

are theories featuring

demographic

(and evolutionary theory), the hope is to

are many reasons for this, but one is surely

achieve such uniqueness o f prediction

that the mediators typically examined are

specifying a c o m p l e x theory-derived

by data

m o r e causally proximal to outcomes that

pattern rather than through a simple c o m -

psychologists value than are

parison o f values derived from groups that

variables.

demographic

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH The preference for simple hypotheses that

manipulating brain blood flows, though it is

transform theoretical mediators into manipu-

difficult to imagine experimentally re-creating

lated causes goes beyond the

demographic

context. T h e main explanatory

constructs

an exact pattern o f earlier observed changes in multisite cranial blood flow. But given social

in psychology are nearly all specified at the

psychology's

individual level and are hypothetical rather

explanatory variables specified at the individ-

broad

use

of

hypothetical

than ostensive—that is, they cannot be directly

ual level, it is obvious that such manipulation

pointed to and so are rarely amenable to direct

is not easy. A different, effective, and entirely

manipulation or measurement. Indeed, exper-

statistical way to get around the selection

iments are designed in the expectation that

problem is, in some circumstances, to use ran-

the observed manipulation (let us call it X )

dom assignment as an instrumental variable in

will impact on the outcome (Y) because the

order to examine the effect o f a measured Z

explanatory variable (Z) followed from X and

variable (Angrist, Imbrens, & Rubin, 1 9 9 6 ) .

affected Y . Sometimes, this chain o f reasoning

This last strategy requires a measure o f Z ,

is advanced without any measurement o f Z or

though, and most researchers cannot wait for

of proxies for it. Then, interpretation depends

the next study in order to turn their suspected

heavily on the face validity of X as well as on

Z into an X . So in actual research practice,

the theoretical uniqueness o f the X - t o - Y pre-

p r o x y measures

diction and on h o w closely the data fit that

mediators often are used to complement the

prediction.

treatment manipulation. But these measures

Sometimes, though, a measure o f Z is pos-

o f theoretically specified

inevitably entail validity and selection prob-

sible, albeit usually in proxy form rather than

lems, leading to a serious conundrum: H o w

as a truly direct measure. Thus, in an experi-

can experiments be used to test multivariate

ment on whether elaborated message process-

explanatory theory about Z variables when

ing mediates attitude change, students can be

experiments were designed to test causal

asked to write down their reactions to a per-

connections between two (but rarely more)

suasive message during exposure to it. These

observed (and not even hypothetical) X and Y

thoughts

content-analyzed.

variables? Theory-based explanation requires

Alternatively, brain scans might be used to see

prior knowledge o f the determinants o f the

which local areas o f the brain experience

cause (for selection purposes)

blood flow changes when the message is

specification

being processed. A test o f causal mediation

between the cause and effect. Y e t experiments

then entails seeing if the relationship between

were not designed for either o f these purposes.

can

then

be

X and Y decreases when Z is statistically con-

o f the

Transforming

processes

and correct mediating

causal explanations

into

trolled (Hoyle & Robinson, Chapter 1 0 , this

independent

volume; Judd & Kenny, 1 9 8 1 ) .

difficult the more elaborate the causal explana-

But two problems stand out here: Is Z

variables becomes increasingly

tory theory is. Consider dual process models

validly measured? And, does the link from Z

within the social cognition domain. They pos-

to Y entail a threat to internal validity from

tulate two forms of processing: (a) controlled

selection because individuals were not ran-

or "systematic" processes that are resource

domly assigned to their Z scores? T h e pre-

demanding,

ferred way around this last problem is to turn

subject to awareness; and (b) automatic or

intentional,

controllable,

and

the Z into an X in the next study, directly

"heuristic" processes that generally are effi-

manipulating elaborated processing by prim-

cient, unintentional, uncontrollable, and out of

ing individuals to process with and without

awareness (see Bargh, 1 9 9 4 ) . As noted, prim-

elaboration or even (within ethical limits)

ing methods can be used to induce each of

The Methodological these, albeit within limits; and M R I scans can

Assumptions

of Social

Psychology

Form of Data Collection

discriminate between the two. But what to do with theories that postulate how these two

From Popper's falsificationist perspective,

processes operate together—whether in paral-

social psychologists should collect data about

lel, as with Eagly and Chaiken's Heuristic

hypotheses that can prove a theory wrong. The

Systematic Model o f persuasion ( 1 9 9 8 ) , or as

social psychology o f social psychological prac-

alternatives, as with Petty and

Cacioppo's

tice suggests, however, that researchers are

Elaboration Likelihood Model (1986)? H o w

more likely to want to show that their theory

can

is right. From this arises the concern with pos-

one practically test a hypothesis

that

requires simultaneously manipulating the two

sible biases emanating from the researcher's

presumptively unique processes? Human inge-

own wishes, hopes, expectations, and dreams.

nuity being what it is, the task may not be

Related to this are setting biases, especially in

impossible, but it is extremely difficult.

the laboratory, where respondents might try to

Meta-scientists such as Kuhn ( 1 9 7 0 ) remind

second guess situational norms in order to con-

us that all theories inevitably are underspeci-

form with them and help the researcher

fied, so that no hypothesis is capable o f defini-

through how they react to them.

tive refutation. W h e n disconfirming data are

T o disconfirm hypotheses, data have to be

generated, the validity o f the deduction and/or

theory-free; otherwise they cannot function as

of the data can be questioned. It is also possi-

neutral adjudicators between truth claims. But

ble to claim that the theory is true, but under

epistemologists remind us that all data are

circumstances different from those tested to

fallible, and therefore no single hypothesis

date. This is why certainty-seeking philoso-

test can carry the adjudicatory weight Popper

phers disparage the substantive theories scien-

assigns it. However, studies that self-con-

tists work with, seeing them as etched more in

sciously incorporate the discipline's recognized

putty than in the stone required for definitive

mechanisms of bias control can help better

refutation.

approximate the generation o f theory-neutral

A more charitable epistemological position

data. So can independent research programs

is that all practical theories are works in

that vary the source and direction of bias. Facts

progress ( C o o k , 1 9 8 5 ) . Thus, when a theory

may not be logically possible, but many obser-

first seems t o be disconfirmed, this should

vations are stubbornly replicated, whatever the

lead t o its revision rather than t o claims it has

theory used to generate them. They have great

been refuted, and so on, as each succeeding

facticity and provide the bedrock from which

theoretical revision is itself tested and perhaps

novel theories are developed. A theory that

again empirically disconfirmed,

fails to subsume the relevant facts o f the past is

generating

yet another version o f the original theory. As the conditions under which the theory is demonstrably

invalid c o n t i n u e

to

not going to get much respect. Even so, individual researchers are capable

grow,

of less self-criticism about the quality of data

enthusiasm for the theory is likely to wane in

and results than is provided by the social

the relevant community o f scholars. Although

system o f science that editors and reviewers

the theory might be true under some as yet

represent. It is they who begin the process o f

still untested circumstances, these may be so

determining what level of merit should be

limited that scholars no longer see any point

assigned to an article, serving as precursors to

to persisting with the theory. It is therefore

a process o f public discussion o f merit that

rejected on pragmatic grounds, not because

develops after the article is published. Data

logic nor data has shown it to be false under

never provide perfect tests o f the empirical

all conditions.

claims a hypothesis makes, though some forms

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH of data are stronger than others. T h e ultimate

and not multi-experimental, design. T h e latter

warrant for claims about data quality reside in

is not fully worked out yet, even in a theory of

a complex mix o f social consensus, logic, and

method sense, let alone practically.

the results of past empirical research on biases.

So, what are the goals o f a multistudy

In social psychology, a supreme value is

strategy? O n e is surely to achieve some form

accorded to data that confirm or disconfirm

of replication; and another is to extend the

an a priori hypothesis. D a t a that do not

theory under test by examining more o f its

make this kind o f contribution are judged to

implications. Theories o f multi-experimental

be less worthy o f journal space. T h i s prefer-

design will require subtle distinctions among

ence for hypothesis-testing has some unin-

types o f replication (exact, partial, concep-

tended consequences.

tual, etc.) as well as realistic analyses o f the

First, it has encouraged some researchers whose data

do not fit the hypothesis

to

conditions under which each is most useful. It will also require discussions about what con-

reframe their hypothesis so that they can write

stitutes a sufficiently close replication to allay

up the results as though they had been pre-

fears o f a Type I error. Also needed will be

dicted. Kerr ( 1 9 9 8 ) calls this " H A R K i n g :

elaboration o f what constitutes a theoretical

Hypothesizing After the Results are K n o w n . "

elaboration that is sufficiently distinctive to

T h e reality is that many studies

produce

advance the research program conceptually,

results at odds with the original prediction. T o

as well as discussion o f the relative value o f

abandon the work at this point entails a con-

independent

siderable waste o f time and effort and can

tions. Finally, statistical wisdom on combin-

endanger a career. H A R K i n g has evolved as

ing results will be needed, especially when

an uneasy compromise between the goals and

experiments are few and not independent.

structure o f the hypothetico-deductive method and the needs o f individual

and

non-independent

replica-

Meta-analysis is the best developed form of

scientists for

multi-experimental design and analysis, and its

w h o m publication is a major goal. Beginning

use is c o m m o n in social psychology (see W o o d

social psychologists soon learn o f this mis-

& Christensen, Chapter 1 5 , this volume). Its

match between formally and informally sanc-

virtues include (a) more statistical power than

tioned epistemology and practice. Indeed, in

single studies typically achieve; (b) the ability to

his guide to professional issues in social psy-

probe potential biases associated with mea-

chology, Bern ( 1 9 8 7 ) explicitly advocates ret-

sured design features; (c) the empirical assess-

rospective hypothesis creation. In contrast,

ment

Kerr laments the capitalizing on chance that

measured setting, person, and time variables,

inevitably results.

of how

robust

results

are

across

as well as across coded ways of operationaliz-

Another consequence o f fallible hypothesis

ing the cause and effect; and (d) the chance to

tests is the growing editorial demand for single

examine more o f the causal explanatory theo-

articles that present the results from multiple

ries about processes mediating between a cause

experiments linked into a small research pro-

and effect. For all these reasons, reviews o f

gram. Wegner ( 1 9 9 2 ) has pointed out the con-

empirical findings constitute the dominant unit

sequences of this practice for effective alpha

of progress in science, though path-breaking

rates. Across two experiments, each with an

single studies get all the glamour. Reviews are

alpha rate o f . 0 5 , the true rate is . 0 0 2 5 . Does

not part o f modal practice in social psychol-

the field really want a level o f stringency that

ogy, however, except as part o f the perfunctory

culls out even more ideas than those already

narrative introduction

to an article. Single

abandoned by the current .05 rate? Moreover,

studies, or small programs o f research bundled

social psychologists are trained in experimental,

into a single article, constitute modal practice.

The Methodological

Assumptions

of Social

Psychology

Although such bundling is an advance over a

THE DUAL H E G E M O N Y OF

single report o f a single study, it is still far from

ANOVA AND T H E LABORATORY

being a literature review or meta-analysis.

EXPERIMENT

T o judge by mainstream journals, modal practice in social psychology also places little

The overwhelming majority o f social psychol-

value on exploratory research, whether as the

ogy studies are conducted in a laboratory

original study goal or as an overt attempt to

setting, use undergraduate participants, last

discover why the data failed to confirm with

no longer than an hour, and involve random

prior expectations (Kenny, 1 9 8 5 ) . Exploration

assignment to two or more experimental con-

should be more prevalent in social psychology

ditions. Rozin ( 2 0 0 1 ) examined articles in the

and done so as not to capitalize on chance

Attitudes

and

Social

C o g n i t i o n and

the

(Tukey, 1 9 7 7 ) . Exploration brings to the sur-

Interpersonal Relations and Group Processes

face novel issues that are worth further study,

sections o f JPSP,

and it is a central function o f ethnography.

found that 7 3 % o f the 4 4 articles used North

But

ethnography

m a k e s other claims,

volume 6 6 ( 1 9 9 4 ) ,

and

American undergraduate samples, 9 5 % used

including some that directly link it to the

analysis o f variance ( A N O V A ) , and just 7 %

hypothetico-deductive

used observation or interview methods.

method

that social

psychologists prefer and that, at first sight,

Fortunately, data from randomized labora-

seems so antithetical to ethnographic prac-

tory experiments analyzed by A N O V A consti-

tice. Y e t , as explicated by Becker

(1958),

tute one o f the strongest frameworks for a

ethnography involves formulating a hypothe-

certain kind o f causal inference-making in

sis from observations, thinking through the

the social sciences today. Heavy reliance on the

implications o f this hypothesis, and

experiment makes the description o f causal

then

going out into the field to collect data to

relationships o f an if-then kind (Cook

examine these implications. W h e n the data

Campbell, 1 9 7 9 ) the dominant framework for

and theoretical implications fit imperfectly,

constructing research problems and organizing

the earlier hypothesis about implications is

thinking in social psychology today (Kenny,

&

revised and new implications are then devel-

1 9 8 5 ) . But causal description is not necessarily

oped and tested. If the next round o f data still

causal explanation, and science values expla-

fails to fit the new implications, then the

nation over description. T o make the transi-

hypothesis is revised again, new implications

tion from description to explanation requires

are developed, and new data are collected.

selecting for study only those causal hypothe-

This iterative hypothetico-deductive process

ses that have been deduced as central postu-

continues until closure is reached.

lates from an explicit and overarching theory

Few social psychologists k n o w about this

that seeks to explain a given phenomenon.

correspondence between epistemology and

Hence, in the method section o f their journal

practice in ethnography and social psychol-

articles, researchers give considerable attention

ogy.

would

both to describing and to justifying their inde-

largely

pendent and dependent variables in terms of

But we suspect few o f them

appreciate

the

correspondence,

because the data ethnographers collect are

some broader theory as well as to illustrating

mostly qualitative. Y e t the data are clearly

just how the selected if-then causal proposition

empirical, and they are products o f the same

will be tested.

disciplined process o f theory development, hypothesis deduction,

collection that drives so much psychological practice.

O t h e r types o f question are relatively

data

neglected, whether about the prediction o f an

o f social

outcome, about the form o f structural rela-

and empirical

tionships among the multiple causes o f an

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH outcome, or about the exploration of new ideas

measures or the same number o f them. It is

emanating from outside established theoreti-

progress indeed to move beyond statistical sig-

cal b o x e s .

nificance testing as a flawed way to compute

Cause o f an

if-then

kind

is

paramount, and A N O V A is responsive to this

the size o f an effect. But standardized effect

primacy. So is the laboratory setting, which is

sizes, although superior, are far from perfect

designed to keep competing alternative inter-

for carrying out the comparative mission to

pretations out o f the explanatory system so

which they speak. Fields such as economics

that the hypothesized cause-effect relationship

choose not to use them, preferring instead to

can play out without perturbation from irrele-

frame their questions around effects that are

vant outside forces. T h e A N O V A emphasis

measured in easily understood interval scale privileges

the

metrics so that results can be reported out as

search for simple main effects caused by few

unstandardized regression coefficients. Dealing

causal variables. O n e manipulation (relative

with variables that are more hypothetical, psy-

to no-treatment controls) seems to be as

chology does not have the same luxury.

c o m m o n as t w o manipulations o f explicit theoretical interest. Higher-order experiments are rare, as are

factorial

T h e dominance o f A N O V A also means that it is rare to find social psychological the-

parametric

ories explicitly constructed around interac-

experiments that vary many levels o f the

tions between different treatments or between

independent variable in order t o identify the

a given treatment and either person or setting

functional form o f the cause-effect relation-

variables. Rozin's review o f JPSP

revealed

ship. S o , the central question is whether there

that fewer than 1 0 % o f the articles reported

is a causal relationship from A to B and not

participants' social class or race or religion or

identifying the form o f this relationship. It is still rare to find effect sizes being

the year or season in which data were collected. Researchers did report participant sex

reported in order to characterize the size of a

in 7 2 % o f the articles; however, they did not

causal relationship, though meta-analysis and

necessarily report the relationship between

the 2 0 0 1 APA publications manual are chang-

sex and the observed outcome. It is as though

ing this, as is the requirement by some journals

respondents are considered interchangeable

to report them. Even so, greater use of effect

or simply sources o f error variance, not

sizes probably will engender further method-

sources o f information about the possible

ological discussion. One likely topic is the

boundaries o f a causal connection.

desirability o f using dependent variables with

The reason for this neglect is that most

grounded metrics that are easily communicable

social psychologists seem willing to believe

and do not need to be standardized (such as

that most cause-effect relationships are gen-

time or money). Another is the comparability

eral across persons and settings. If one believes

of effect sizes when certain choices exist for

this assumption to be true, then there is no

computing them. One is when they can be

reason to search for interactions between inde-

computed with or without covariates. Using

pendent variables and person or setting char-

covariates increases effect sizes, but not all

acteristics. T o do so would entail the hassle o f

studies have them or the same set of them.

constructing a different kind o f theory, build-

Another is when effect sizes can be computed

ing a different kind o f design, and then col-

as a difference between means or between

lecting and

slopes. With repeated measures, the effect size

without any guarantee that one will learn any-

analyzing data differently, all

of a final point difference might be much less

thing important about restrictions to the gen-

than the effect size for the contrast between

erality o f a causal claim. Campbell ( 1 9 6 9 )

the slopes. But not all studies have repeated

asserted that physical scientists also assume

The Methodological

Assumptions

that any independently replicated finding is

setting, or time had

of Social Psychology been used.

\

Causal

universal until subsequent research shows oth-

relationships constitute the key theoretical

erwise. Given that, and because the value o f a

issues in the field, not preexisting group mean

substantive theory is widely held to be contin-

differences. T h e latter is the province o f per-

gent on its reach, why should social psycholo-

sonality theory and research.

gists assume that

not

Even if a causal relationship should turn out

universal? T o do so denigrates the very theory

their theories are

to be contingent, some types o f contingency

they are about to test.

have more serious theoretical implications than

In this connection, consider the contact

others. Moderator variables that reverse the

hypothesis. As a simple main effect, it asserts

sign o f a causal connection clearly restrict

that increasing the amount of contact between

causal generalization. So, if this were the pre-

individuals from different groups will reduce

ferred criterion for inferring that effects are

any intergroup conflict between them. This

non-universal, then the technical stringency of

idea was o f great interest in social psychology

meeting the sign-reversal criterion makes it rea-

5 0 years ago, but research on it almost petered

sonable to surmise that causal effects in social

out once it emerged that such contact worked

psychology generally are universal and that

only under a large number o f quite disparate

overgeneralization is not much o f a problem.

circumstances that still cannot be integrated

However, the case is murkier with a less strin-

into a novel, parsimonious theory (Hewstone,

gent generalization criterion, such as differ-

1 9 9 6 ) . Researchers abandoned the contact

ences in slopes without a necessary difference

hypothesis as overly contingent.

in their sign. Using this criterion, we would be

The validity o f the ontological assumption that most causal relationships are universal is of great moment. O n it depends the utility o f

more likely to discover that the world is more complicated than simple main effects. It would therefore be useful to assess the fre-

more

quency o f stable statistical interactions in social

complicated designs that entail different kinds

psychology that involve person, setting, and

of theory and more extensive or more repre-

time features, both with and without sign

sentative sampling designs, measurement, and

reversals. However, this enterprise would be

simple A N O V A designs rather than

analysis. Social psychologists are conscious o f

fraught with perils. It is notoriously difficult to

relying on college samples, stripped down lab-

conduct statistically powerful tests o f interac-

oratory settings, few and short-lasting inter-

tions and to uncover stable ones that replicate.

ventions, and simple data-analytic procedures

T o achieve this requires (a) strong theory spec-

like A N O V A . In this regard, Sears ( 1 9 8 6 ) out-

ifying the exact form o f the interaction, (b) reli-

lined some special properties of student popu-

ably measured samples of persons and settings

lations that can be inferred

other

that manifestly include the tails o f the distribu-

psychological work. Among other things, he

tions, (c) samples that are large, (d) causal vari-

from

noted that their late-adolescent status proba-

ables that are not truncated, (e) dependent

bly makes them especially susceptible to social

variables without floor or ceiling effects, and

influence. However, most psychologists (e.g.,

(f) dependent

variables whose theoretical

M o o k , 1 9 8 3 ) do not worry that individual

meanings would change if their interval scales

differences o f person, setting, or time might

were arithmetically transformed.

affect levels o f compliance. Their concern is

these requirements is difficult in all social

only whether such differences condition h o w

research but clearly more so in laboratory than

the manipulation and compliance are related,

survey work.

thus creating a different list o f causal factors than would emerge if a different population,

Although

Meeting

the incidence o f interactions

cannot be tested sensitively, indirect evidence

31

32

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH suggests that stable interactions are quite rare

study results like those o f Milgram ( 1 9 7 4 ) ,

and that the world might be organized accord-

Latane and Darley ( 1 9 7 0 ) , and Zimbardo,

ing to many broad main effects. Few of the

and those laboratory results that are counter-

meta-analyses conducted to date o f which we

intuitive or otherwise provocative. Even so,

are aware have resulted in many stable inter-

laboratory results do not seem to command as

actions between a treatment class and either

much respect in the human sciences as do

person, setting, or time variables. Moreover,

results from surveys, field experiments, and

Cronbach and Snow (1977) tried to develop a

longitudinal studies, a verdict that would be

list o f replicated interactions involving some

unfair if laboratory results do indeed routinely

kind of teaching intervention and student or

generalize to other settings and kinds o f

teacher characteristics. They met with what we

respondents. But the criticism is nonetheless

judge to be little success. In addition, many

made and, because o f its laboratory

social psychologists are proud that ordinary

undergraduate emphasis, social psychology

Americans seem to recognize themselves in

does less well in the public relations domain

reports about laboratory studies that get into

than many other social sciences.

and

the media, suggesting that these reports speak to Everyman's experience, or at least to the experience of the North American Everyman. (However, a selection factor would operate

SPECIFIC T H E O R E T I C A L

here if journalists choose to feature a result

CONCERNS AND THEIR

because it seems to them to be general.) In any

METHODOLOGICAL

event, the empirical evidence on the universality

IMPLICATIONS

assumption is far from complete and difficult to evaluate. Such as it is, that evidence does not seem to seriously invalidate the assumption.

The Social Cognitive Revolution If A N O V A and the laboratory experiment

T w o things are clear, though. Just as the

are the knife and fork of social psychology,

logic o f designing multi-experiment studies is

then cognitive explanatory variables are cur-

different in subtle and important ways from

rently its bread and butter. Indeed, the central-

the logic o f designing single studies, so the

ity o f information

logic and practice o f testing interaction pre-

representations led Markus and Zajonc ( 1 9 8 5 )

processing and

mental

dictions is subtly different from the logic o f

to remark that social psychology has essen-

testing main effects. Few social psychologists

tially become the study o f the social mind.

seem to realize this, perhaps because they

They suggested that social cognition is no

need not do so as long as they continue t o

longer concerned with an " O - S - O - R " (i.e.,

assume that the majority o f interesting causal

stimulus-organism-response) formulation that

connections are universal.

privileges how attributes o f the organism medi-

T h e belief in causal universality that allows

ate causal relationships. Instead, they prefer an

social psychologists to rely on college sopho-

" O - S - O - R " formulation

mores, laboratory settings, and short-lasting

causally central the organism's subjective con-

because it makes

interventions is not shared by other social sci-

strual o f the world, not the world itself.

entists or policy actors. Social psychology is

In the 1 9 7 0 s and 1 9 8 0 s , social psychologists

routinely criticized for an overreliance on the

unearthed or resurrected many biases that tes-

laboratory that seems to undermine the face

tify to the mismatch between external reality

validity o f results. This is not to deny that

and human perception of it, including unrealis-

some social psychological findings are com-

tic optimism (Irwin, 1 9 4 4 ) and the actor-

mented on beyond the field, particularly field

observer attributional bias (Jones & Nisbett,

The Methodological

Assumptions

of Social

Psychology

1 9 7 2 ) . Once the key influence on a person's

Milne, & Bodenhausen, 1 9 9 4 ) , or through

thought, feeling, and behavior was recognized

presenting the face o f a member o f a stereo-

not to be reality but to be the perception

typed group (e.g., Fazio, J a c k s o n , Dunton, &

thereof, then the mental processes that created

Williams, 1 9 9 5 ) . All are valid operational def-

and reacted to these perceptions became the

initions o f the same latent construct, o f the

proper focus o f social psychology. The objec-

same psychologically active mechanism.

tive nature o f the world assumed a lesser status.

Other examples o f the relative lack of con-

By the time this turn was being made in

cern with operational specifics can be found

social psychology, cognitive psychologists had

within social cognition. Researchers have

already been exploring perception, reasoning,

demonstrated that the accessibility of a repre-

categorization, mental representation,

and

sentation (e.g., a stereotype) shapes information

other ways in which the mind imposes struc-

processing in basically the same way regardless

ture on the world. Their concepts and meth-

of whether the accessibility results from a

ods were now keenly explored by social

chronic traitlike tendency to have that repre-

psychologists,

and

sentation accessible or from a temporary prim-

applied them to the social domain. Social cog-

ing o f that representation in the laboratory

nition was born. Sixteen years later, Tesser

(Bargh, Bond, Lombardi, & T o t a , 1 9 8 6 ) . Belief

and Bau ( 2 0 0 2 ) set out to review the current

that there are many ways to manipulate the

themes in the field and concluded that social

same underlying process has the

cognition was still dominant. Indeed, they

advantage

who

borrowed

them

fortunate

o f enhancing construct validity,

characterized the 1 9 9 0 s as "the decade o f

especially when it can be empirically shown

social cognition" (p. 8 1 ) . T h e perceiver's

that several different manipulations all activate

"black b o x " had become the central concern

the same measured mechanism and have the

of social psychology, in sharp contrast to the

same relationship with the outcome. W h e n this

stimulus-response approach.

pattern o f results emerges, it is clear that the

The

methodological implications o f this

particular form o f the manipulation

is an

shift included a need to develop ways to tap

irrelevancy and that causal potency may well

hidden cognitive processes (e.g., via reaction

lie with the common mechanism each manipu-

times, error rates, and other alternatives to

lation elicits.

introspection). Also implied was less emphasis

Pursuing a strategy that assigns secondary

on the precise characteristics o f the indepen-

status to observable independent

dent variable because the primary aim o f

attributes

experimental stimuli was n o w to activate a

instance, awareness o f the prime, however

variable

requires considerable care. F o r

relevant mental representation. Whether it

induced, does matter, and it can produce con-

was specifically conceived o f as a stereotype or

trast

trait category, the mental representation was

attempts to correct away from its influence

effects in judgment

as the

person

usually held to be an abstraction (Sherman,

(Schwarz & Bless, 1 9 9 2 ) . So, the theoretically

1 9 9 6 ) . Because an abstraction is, by defini-

mundane and merely vehicular components o f

tion, a synthesis across multiple, partially dif-

independent variables should not be decided

ferent stimuli, the precise characteristics o f

on lightly.

independent variables are no longer crucial. It

Moreover, the assumption is made that

hardly matters whether a stereotype is acti-

external environments do not affect individuals

vated through unscrambling a stereotype-

other than through individuals' perception of

related sentence (e.g., Banaji, Hardin, &c

them. Is this universally true o f neighborhood,

R o t h m a n , 1 9 9 3 ) , through subliminal priming

school, family, peer, media, and work influ-

of

ences (Cook, Herman, Phillips, & Setterston,

a stereotyped

category (e.g., M a c r a e ,

34

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH 2 0 0 2 ) , let alone true of the influence o f more

intergroup

distal social events such as city and national

escalated, intervened with cooperative tasks. It

politics? It is true that perceptions o f neighbor-

would not be possible to collect such long-term

hoods are more strongly correlated

with

adolescent outcomes than are objective neigh-

conflict, and,

when

tensions

data in a conventional laboratory setting using paid undergraduates.

borhood attributes (Jessor & Jessor, 1 9 7 3 ) ,

However, confederates can be used in the

but this does not mean that the objective char-

laboratory to manipulate controlled interper-

acteristics play no independent role (Cook,

sonal events, as they have since the days of

Herman, et al., 2 0 0 2 ; Furstenberg, C o o k ,

Asch's ( 1 9 5 2 ) conformity studies. Even so,

Eccles, Elder, & Sameroff, 1 9 9 9 ) .

many interpersonal events are rather resistant t o experimental control, and even when they can be brought into the lab, it can be difficult

The Relative Neglect of Theories of Interpersonal Dynamics

to analyze the data they provide

within

A N O V A . This is because people influence each

Given the dominance of social cognition

other in complex ways through time, entailing

and laboratory settings, modal research prac-

observations that rarely are independent. As

tice pays little attention to theories o f interper-

Kenny ( 1 9 8 5 ) noted, this violates the indepen-

sonal dynamics. Pennebaker ( 2 0 0 2 ) noted that

dent errors assumption of A N O V A and forces

there is no APA division devoted to language

researchers to use more complicated hierarchi-

even though language "is the basis o f most

cally ordered statistics that few social psychol-

human communication

ogists now know (Bryk &c Raudenbush, 1 9 9 2 ;

and

is the

filter

through which we understand and learn about

Gonzalez & Griffin, Chapter 1 4 , this volume).

ourselves and others" (p. 9 ) . The Interpersonal

So, for quite different reasons, interpersonal

and Group Processes section of JPSP often has

interdependence is a tepid topic in social psy-

the least journal space devoted to it, and the

chology today, left to an intrepid but perhaps

articles published there tend to focus on phe-

increasing few. W h e n preferences are clustered

nomena like social comparison that involve

around a dominant theory class (such as social

social cognition about a specific (but not nec-

cognition), this limits not just the form o f pre-

essarily present) person or group o f people.

ferred theory but also method choices. There is

Identity negotiation researchers w h o

focus

little immediate need to venture into the novel

directly on the dynamic exchange between a

methodological territories that different kinds

perceiver and target, for example in the case o f

of theoretical framing require.

self-fulfilling prophecies (e.g., Swann, 1 9 8 7 ) , are the exception rather than the rule. Admittedly, it is difficult to capture interpersonal dynamics in a short-term lab study. In the early years o f social psychology, when theories about interpersonal relations were more

The Neglect of High-Impact Manipulations and the Kinds of Theory They Promote In social psychology's history, the words

prevalent, field studies were conducted that

" l a b " and "experiment" did not automatically

allowed participants to interact relatively freely

go together as they do today. Involvement in

over a significant period of time, during which

both non-experiments and field experiments

the researchers observed and

administered

was more c o m m o n . W h e n researchers are

interventions. For example, Sherif, Harvey,

willing to proceed on the basis o f a direct rela-

White, Hood, and Sherif ( 1 9 6 1 ) arranged a

tionship between objective situations

and

boys

overt behavior, field experiments are enabled.

randomly to teams, observed the resulting

Thus, to take one from many examples,

summer camp where they assigned

The Methodological Milgram, Bickman, and Berkowitz ( 1 9 6 9 ) induced varying numbers o f confederates t o

Assumptions

of Social

Psychology

The Average Person as the Locus of Explanation

stare up at the sixth floor o f an arbitrarily chosen N e w Y o r k building. As predicted, the percentage o f passersby joining in the "gawking" was a diminishing

function

o f the

number o f confederate "gawkers."

Both

Asch ( 1 9 5 2 )

and

Lewin

(1951)

regarded the interaction between person and situation as the critical subject matter of social psychology. Neither one nor the other claimed

It is not the purpose of experiments to re-cre-

special status, but their intersection did. W h e n

ate reality. Rather, it is to decompose and

one considers modern practice in social psy-

recombine what is in the world, often heighten-

chology, it is apparent that the field does not

ing it but always clarifying it—in Bacon's words,

share this interactionist emphasis. Instead, it

"Twisting Nature by the Tail." High-impact

places most emphasis on the individual, more

treatments push the boundaries of human expe-

accurately the average individual studied, leav-

rience, as in Milgram's work and in the labora-

ing less room for supra-individual phenomena

tory simulation of prison life by Haney, Banks,

as explanatory constructs. T h e field seems to

and Zimbardo (1973) that had to be terminated

have reasserted F. H. Allport's ( 1 9 2 4 ) reduc-

prematurely because of the sadistic tendencies

tionist view that collective phenomena

that developed among the "guards"

understood properly with reference only to the

toward

are

their "prisoners." But since then, few other lab-

minds o f individuals—primarily as these indi-

oratory experiments can be offered as examples

viduals perceive the social world and process

of high impact. Motivationally, most operate in

information about it.

a lower key, as short-term analogs to deeper and longer lasting experiences.

Individual differences are rarely part o f the modern story, though they would be required

Ironically, it was probably ethically ques-

for a full interactionist perspective (see Shoda,

tionable attempts to construct high-impact

Chapter 6, this volume, for a detailed discus-

manipulations in the laboratory that helped

sion o f different

crown the low-impact laboratory study as

including individual differences). There are

possible approaches

to

sovereign and that hastened the move toward

both moral and ideological reasons why indi-

explicitly analog, nondeceptive

approaches

vidual differences have lost their appeal. T h e

(e.g., Prisoner's Dilemma games to study coop-

assumption o f universal responsivity to causal

eration, lists of traits to study impression for-

agents is compatible with the egalitarian, mer-

mation). Jones ( 1 9 8 5 ) has even opined that

itocratic ideology o f the West. Moreover,

Milgram's obedience study and Festinger's dis-

social psychologists have tried to avoid the

sonance experiments bear partial responsibility

mistakes o f individual difference researchers,

for "the range o f available procedures [being]

whose early work (say, on group differences in

restricted by ethical concerns enforceable by

intelligence) seemed t o support questionable

human subjects committees" (p. 9 8 ) . W h a t we

political actions (Kamin, 1 9 7 4 ) . Also psychol-

want to note here is that the timing of these eth-

ogists may be sensitive to their own proclivity

ical concerns (Baumrind, 1 9 6 4 ) coincided with

as laypersons to overlook situational explana-

the rise of an information-processing paradigm

tions of behavior in favor o f dispositional

that rarely requires high-impact manipulations

ones,

(see also Kimmel, Chapter 3, this volume).

(Nisbett & Ross, 1 9 8 0 ) . Fears like these prob-

Thus, there were both theoretical and ethical

ably have helped reinforce the field's move

the

fundamental

attribution

error

reasons why social psychologists moved away

toward assigning a major causal role to indi-

from the study o f "hot" motivated behavior

vidual psychological constructions o f situa-

and toward "colder" social cognition.

tions rather than to individual dispositions

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH or

interplay

constructs are evident in notions such as

Unfortunately perceptions of social situa-

person actually takes on a different self when

objective situations or to the

depersonalization,

between the two.

according to which

the

tions are not always anchored in a broad range

acting as a group member (e.g., Turner

of comparative experience and so may be quite

Onorato, 1 9 9 9 ) . T h e contrast here is with

unstable. They certainly run the risk o f over-

American cousins to these ideas, such as the

looking stable structural forces that might have

interdependent self-construal of M a r k u s and

serious long-term repercussions. Eagly's social

Kitayama ( 1 9 9 1 ) that again places most con-

role theory of sex differences in psychological

ceptual focus on the individual.

&

functioning (Eagly, W o o d , & Diekman, 2 0 0 0 )

Moving away from an individual focus has

is sensitive to this last dilemma, being an

methodological repercussions. For instance,

attempt to sketch how social structure creates

data analyses would have to take account o f

power and resource inequalities that have per-

h o w social systems are hierarchically ordered,

petuated stable but learned sex differences in

with individuals nested within groups

psychological functioning. Her perspective on

groups within larger aggregates like schools or

objective and sustained social contexts is much

neighborhoods

more like what sociologists assume in their

move analysis toward

studies of how family, school, and neighbor-

modeling ( H L M ) rather than A N O V A (see

and

or universities. This would hierarchical linear

hood forces sustain cross-generational stability

Gonzalez & Griffin, Chapter 1 4 , this volume;

in social class standing. For them, as for Eagly,

West, Biesanz, & K w o k , Chapter 1 3 , this vol-

perceived contexts count; but actual contexts

ume).

also do.

also has methodological repercussions. M o s t

Even cross-cultural psychologists locate

How

individuals

are conceptualized

social psychologists prefer a variable-centered

culture in the individual mind, with B o n d

conceptualization

(2002)

cross-cultural

concepts such as race, age, and gender or sin-

researchers for falling prey to the ecological

gle personality attributes such as neuroticism

fallacy o f ascribing to the person properties

or

that are only demonstrably true o f the group

developmental psychologists are experiment-

recently

criticizing

o f individuals,

invoking

degree o f loneliness. In contrast, some

(as with the construct o f individualism). A

ing with person-centered approaches to indi-

similar reluctance is evident when it comes to

vidual

differences

(e.g.,

Magnusson

&

reciprocal causation between the person and

Bergmann, 1 9 9 0 ) that require tricky multi-

environment. For instance, prejudice tends t o

variate techniques such as cluster analysis to

be studied either in terms o f perceiver bias

construct multivariate profiles o f types o f indi-

and representations (e.g., Devine, 1 9 8 9 ) or in

viduals. Knowledge o f A N O V A will not suf-

terms o f the consequences o f stigma for the

fice for anyone studying either interpersonal

target (e.g., Crocker & M a j o r , 1 9 8 9 ; Steele &

relationships or individual

Aronson, 1 9 9 5 ) . But Shelton ( 2 0 0 0 ) recently

ceived as multivariate profiles.

differences con-

asked why psychologists are not examining

Most social psychologists like to believe they

the actual interactions between, say, African

are studying the "average human" or at least

Americans and European Americans in order

"the average North American," but they actually

to explain prejudice. Like culture, prejudice

study the average person in whatever sample

seems to be "psychologized," restricted to

they have at hand. This sample is rarely ran-

being a property o f individuals.

domly selected from a clearly designated and the-

Attempts by European social psychologists to

explore

supra-individual

explanatory

oretically justified population,

a

preferred

strategy for generalization in science. Purposive

The Methodological

Assumptions

of Social

Psychology

samples are the order of the day. These can

O f course, individual differences are not

sometimes promote generalization, but only

completely ignored; nor are statistical inter-

when accompanied by ancillary information and

actions with treatments.

analyses (Cook, 1 9 9 3 ) . Furthermore, even with

they occur, the focus is mostly on those

However,

when

purposive samples, detailed description of the

individual

sample is very important for generalization, but

moderate psychological mechanisms assumed

difference

factors t h a t

might

social psychologists rarely provide such descrip-

to be universal. F o r example, Oyserman,

tion in detail. True, there are some researchers

Kemmelmeier,

with interests in more circumscribed popula-

cross-cultural

tions, for instance the average neurotic, the aver-

"identify cultural contingencies that moder-

and

Coon

psychology

(2002) as

depict

seeking

to

age shy, or the average African American person.

ate general processes o f human cognition,

At a rninimum, these researchers need to show

affect, and behavior" (p. 1 1 0 ) . These "cul-

that the persons they sample in their studies

tural

belong in the target class, even if they were not

framed in terms o f psychological processes

randomly selected from that class and even if the

such as values and traditions rather than in

correlates of class membership (and hence poten-

terms o f sociological or political processes

contingencies"

are

nearly

always

tial confounds for interpretation) have not been

such as class inequality or the distribution o f

described as fully as they might have been.

resources. W h e n national

Sample description is rudimentary in social psy-

demonstrated, most social psychologists are

differences

are

chology because practitioners generally believe

then inclined to move t o a m o r e abstract level

that individual differences do not matter much

o f theory that subsumes the differences by

and that theories about specific populations are

invoking some novel underlying c o m m o n a l -

less important than theories about people in gen-

ity. J a h o d a ( 1 9 8 6 ) has argued against this,

eral. So why go to all the trouble?

however, contending that it is better to stay

Given this belief system, social psycholo-

with the national particulars and to construct

gists do not develop theories about individual

different psychologies for different countries.

differences that might interact with an inter-

In his view, it is better to gain a full and data-

vention. N o r , in general, do they test many

based psychological understanding that is

interaction possibilities in post hoc fashion. O f

applicable in at least some national contexts

course, with the small samples and truncated

than it is to create some universalist explana-

distributions they typically use (because o f

tion framed

w h o is recruited into experiments), the statis-

terms.

in exclusively individualistic

tical power o f the interaction tests they do conduct is limited. This points to a serious inferential difficulty that would ensue if interaction predictions were more c o m m o n in the field, as well as to the need for social psychologists to be more knowledgeable about the design and sampling modifications that serious interaction testing requires. In any event,

The Assumption of Irrelevant Domains and Hence the Generation of Theories With Minimal Grounded Content For the modal social psychologist, the pro-

believing that social psychology deals with the

cesses acting upon mental representations are

reactions o f the average person helps promote

construed to be essentially similar, not only

the field's status as a cumulative science that

across all human beings but also across all

does not have to worry much about

domains of content. Evolutionary psychologists

population dependence o f results.

the

(e.g., T o o b y & Cosmides, 1 9 9 2 ) do not share

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH this view, believing that the human mind is an

Third, G. W . Allport's ( 1 9 5 4 ) description

evolved set of tools specialized for discrete

of social psychology invokes social contexts

domains of life that were important to survival

that are real, implied, or imagined, without

(e.g., face perception, mate preferences). They

differentiating

postulate that some psychological processes will

domain generality, the need to differentiate

therefore be domain-specific, an

is reduced.

assumption

otherwise rarely shared in social psychology. The

opposite

assumption—of

cross-

between

the

In this regard,

three.

Given

consider

the

diverse ways that social rejection has been operationalized.

Manipulations

include

domain generalizability—is more widespread

(a) using live confederates who exclude the

and has several important implications. First,

participant from a game o f catch (Nezlek,

it effectively erases the line between social

Kowalski, Leary, Blevins, & Holgate, 1 9 9 7 ) ,

and nonsocial cognition (Hastie & Carlston,

(b) using "internet chat r o o m s " that remove

1 9 8 0 ) . This justifies the construction of social

face-to-face contact but not the perceived real-

cognition as an enterprise concerned more

ity o f the interaction (Gardner, Pickett, &

with a standard cognitive analysis applied to

Brewer, 2 0 0 0 ) , and (c) asking participants to

social phenomena than with the generation o f

imagine

unique analyses designed to identify h o w cog-

Kimball, & Rehak, 1 9 7 9 ) . Using Immersive

nitive performance differs because a social ver-

Virtual Environment Technology, Blascovich

being

rejected

(e.g.,

Craighead,

sus nonsocial object is being appraised. This

et al. (in press) studied the conditions under

last emphasis is clear in early social psycho-

which participants would show social influ-

logical work on the symbolic properties o f

ence effects such as maintaining interpersonal

objects,

such as Bruner and

Goodman's

distance or conforming to the level o f bets

( 1 9 4 7 ) demonstration that the value associ-

placed by others in a virtual casino. Results

ated with (heavily socially imbued) coins influ-

showed differences depending on whether the

enced people's judgments about the size o f

respondents

circles—a social cognitive cause o f a seemingly

with figures or with "avatars" that either pos-

nonsocial perception.

sessed photographic realism (e.g., were repre-

Second, social psychologists w h o assume

thought

they were

interacting

sented by a computerized person rather than a

cross-domain generalization can avoid study-

beach ball), or corresponded to a real person

ing such problematic domains as divorce,

rather than a simple computer program.

bereavement, parenting, cults, and so on, so

These results all suggest that the assump-

long as they can plausibly argue that the

tion o f stimulus-independent processes is not

same social, cognitive, and emotional pro-

always

cesses account for most o f the variance found

cannot be used willy-nilly in lieu o f more

between these domains. But h o w can this

realistic ones. Similarly, real and perceived

assumption

neighborhoods are not always substitutable.

be tested

(better,

"probed")

except through a laborious process o f measurement,

experimentation,

and

synthesis

appropriate,

that

artificial

stimuli

Social psychologists seem to believe that nature is structured with important general

across these various domains? Until empiri-

truths (laws?) at its center, and with less

cal research on domain specificity is done,

important

social psychology will seem rather content-

periphery. T h e r e is n o obvious reason why

free to most outside observers, not imbued

the social psychological world

with deep insider knowledge o f the behav-

organized around general laws rather than

iorally grounded

around a vast collection o f w-way interac-

phenomena to which its

theories and results are thought t o apply.

specific

qualifications

at

should

its be

tions that would be detected if research were

The Methodological

Assumptions

of Social

Psychology

more sensitively designed so as to detect

for collecting relevant data. Sometimes, a single

interactions rather than main effects.

method will suffice, like use o f the sample

Even so, epistemologists value parsimony

survey when a population description is called

more than complexity, and the prestigious

for. M o r e often, though, multiple methods

physical sciences employ a law-oriented view

will be needed, given the limitations o f many

of knowledge growth. T h e y have generally

individual methods. Even then, though, there

independent

is no justification for using multiple methods

replication justifies the assumption that results

willy-nilly. An explicit justification is required

prospered

believing that

one

are general until subsequent research proves

for the several methods chosen because they

otherwise. Given these realities, we cannot

have to differ in their imperfections and in the

envisage social psychologists mounting a seri-

direction o f any biases presumed to attend

ous attack to describe a more complex onto-

these imperfections ( C o o k , 1 9 8 5 ) .

logical structure than they n o w assume.

The

main problem for a field is when

c o m m i t m e n t t o a particular style o f research inadvertently precludes theoretical growth

CONCLUSION

by restricting the kinds o f questions asked.

Methodological practice in social psychology

psychology as it is practiced in Sociology

has changed over time and will continue to

departments, the social psychology we have

do so. T h e rigid impression o f research meth-

described here runs that risk. Seen meta-the-

ods one finds in most introductory textbooks

o r e t i c a l l y , it rarely

C o m p a r e d both t o its own past and to social

deals

with

systems

is an illusion. F r o m the w a y the questions are

notions, including multivariate and recipro-

formulated t o the w a y data are collected,

cal causation. It rarely deals with the social

analyzed, and interpreted, empirical w o r k

and individual determinants o f the social

involves hundreds o f choices at every stage.

factors it varies. It rarely probes the person,

Some o f these choices are not conscious

setting, and time factors that might condi-

because the alternatives are not known, or

tion a causal relationship, instead assuming

are rendered invisible or inappropriate by the

a routine generalization o f findings that indi-

collective decision o f the social psychology

vidual studies can rarely test, given the small

community to do things a particular way.

and homogeneous samples typically used. In

But the alternatives are still there.

addition, field research is rare relative t o lab-

Although this chapter is a plea for greater

oratory research; research with populations

diversity o f method choice in social psychol-

other than university students is also rare;

ogy, we acknowledge that taking the position

and exploratory research is much rarer than

of the modal social psychologist has limited

hypothesis testing. Cognitive theories are

our account o f the variation that already

still dominant, though interpersonal, e m o -

exists in the field. Fortunately, the chapters

tional, biological, and cultural concepts are

following this one put some o f this variation

gaining in currency, albeit often linked to

back into play, as they seek t o introduce

cognition, as in cognitive neuroscience. T h e

social psychologists t o newer techniques and

best case for methodological pluralism in the

hence to extend their options.

field is not that better answers can be pro-

However, we see no virtue to methodolog-

vided t o traditional questions; it is that a

ical pluralism per se. T h e key need is for

wider range o f questions can be asked that

researchers to get their questions or issues

will encourage a wider

straight and then to use the best available tools

development.

range o f theory

F U N D A M E N T A L ISSUES I N S O C I A L P S Y C H O L O G I C A L R E S E A R C H

REFERENCES A l l p o r t , F. H . ( 1 9 2 4 ) . Social psychology.

Boston: H o u g h t o n M i f f l i n .

A l l p o r t , G . W . ( 1 9 5 4 ) . T h e h i s t o r i c a l b a c k g r o u n d o f m o d e r n social p s y c h o l o g y . I n

G. L i n d z e y (Ed.), The handbook

of social

( V o l . 1 , p p . 3-56).

psychology

Cambridge, M A : Addison-Wesley. A m e r i c a n P s y c h o l o g i c a l A s s o c i a t i o n . ( 2 0 0 1 ) . Publication

Psychological

manual

of the

American

( 5 t h ed.). W a s h i n g t o n , D C : A u t h o r .

Association

A n g r i s t , J . D . , I m b r e n s , G . W . , & R u b i n , D . B. ( 1 9 9 6 ) . I d e n t i f i c a t i o n o f causal effects

u s i n g i n s t r u m e n t a l v a r i a b l e s , journal

of

the

American

Statistical

91, 4 4 4 - 4 5 5 .

Association,

A r o n s o n , E., W i l s o n , T . D . , & B r e w e r , M . B. ( 1 9 9 8 ) . E x p e r i m e n t a t i o n i n social psyc h o l o g y . I n D . T . G i l b e r t , S. T . Fiske, & G . L i n d z e y (Eds.), The handbook social

of

(4th ed., V o l . 2 , pp. 99-142). N e w Y o r k : M c G r a w - H i l l .

psychology

A s c h , S. E. ( 1 9 5 2 ) . Social psychology. Banaji, M . R., H a r d i n , C , &

N e w Y o r k : Prentice-Hall.

R o t h m a n , A . J. (1993). Implicit stereotyping

i n p e r s o n j u d g m e n t . Journal

of Personality

& Social

65(2),

Psychology,

272-281. Bargh, J. A . (1994). T h e f o u r horsemen o f automaticity: Awareness, intention, efficiency, a n d c o n t r o l i n social c o g n i t i o n . I n R. S. W y e r & T . K . K r u l l (Eds.),

Handbook

of social

( V o l . 1 , p p . 1-40). H i l l s d a l e , N J : Lawrence

cognition

Erlbaum. B a r g h , J . A . , B o n d , R. N . , L o m b a r d i , W . J . , & T o t a , M . E. ( 1 9 8 6 ) . T h e a d d i t i v e n a t u r e o f c h r o n i c a n d t e m p o r a r y sources o f c o n s t r u c t accessibility. Journal

Personality Baumrind,

D.

(1964).

Some

of

50(5), 869-878.

and Social Psychology,

t h o u g h t s o n ethics o f research:

M i l g r a m ' s " B e h a v i o r a l Study o f O b e d i e n c e . " American

After

reading

Psychologist,

19(6),

421-423. Becker, H . S. ( 1 9 5 8 ) . P r o b l e m s o f inference a n d p r o o f i n p a r t i c i p a n t o b s e r v a t i o n .

American

Sociological

23, 6 5 2 - 6 6 0 .

Review,

Bern, D . J . ( 1 9 8 7 ) . W r i t i n g t h e e m p i r i c a l j o u r n a l a r t i c l e . I n M . P. Z a n n a & J . M .

D a r l e y (Eds.), The compleat social

scientist

academic:

A practical

guide for the

beginning

(pp. 171-201). N e w Y o r k : R a n d o m House.

Blascovich, J . , L o o m i s , J . , Beall, A . C , S w i n t h , K . R., H o y t , C . L . , & B a i l e n s o n , J . N . (in press). I m m e r s i v e v i r t u a l e n v i r o n m e n t t e c h n o l o g y as a m e t h o d o l o g i c a l t o o l

f o r social psychology. Psychological

Inquiry.

B o n d , M . H . ( 2 0 0 2 ) . R e c l a i m i n g t h e i n d i v i d u a l f r o m H o f s t e d e ' s ecological a n a l y s i s — A 2 0 - y e a r odyssey: C o m m e n t o n O y s e r m a n et al. ( 2 0 0 2 ) .

Bulletin,

128(1),

Psychological

73-77.

B r u n e r , J . S., & G o o d m a n , C . C . ( 1 9 4 7 ) . V a l u e a n d need as o r g a n i z i n g f a c t o r s i n

perception. Journal

of Abnormal

42, 3 3 - 4 4 .

& Social Psychology,

B r y k , A . S., & R a u d e n b u s h , S. W . ( 1 9 9 2 ) . Hierarchical

linear models.

Newbury

P a r k , C A : Sage. Buss, D . M . , & K e n r i c k , D . T . ( 1 9 9 8 ) . E v o l u t i o n a r y social p s y c h o l o g y . I n D . T . G i l b e r t , S. T . Fiske, 8c G . L i n d z e y (Eds.), The handbook

of social

psychology

(4th ed., V o l . 2 , pp. 982-1026). N e w Y o r k : M c G r a w - H i l l . C a m p b e l l , D . T . ( 1 9 6 9 ) . Prospective: A r t i f a c t a n d c o n t r o l . I n R. R o s e n t h a l & R. L . R o s n o w (Eds.), Artifact

in behavioral

research

(pp. 351-382). N e w York:

A c a d e m i c Press. C o o k , T . D . ( 1 9 8 5 ) . P o s t - p o s i t i v i s t c r i t i c a l m u l t i p l i s m . I n R. L . S h o t l a n d & M . M .

M a r k (Eds.), Social Sage.

science

and social

policy

( p p . 2 1 - 6 2 ) . Beverly H i l l s , C A :

The Methodological

Assumptions

of Social

Psychology

Cook, T. D. (1993). A quasi-sampling theory of the generalization of causal relationships. In L. Sechrest 8c A. G. Scott (Eds.), New directions for program evaluation: Vol. 57. Understanding causes and generalizing about them (pp. 39-82). San Francisco: Jossey-Bass. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and anal­ ysis issues. Boston: Houghton Mifflin. Cook, T. D., Herman, M., Phillips, M., 8c Setterston, R. J . , J r . (2002). Some ways in which neighborhoods, nuclear families, friendship groups and schools jointly affect changes in early adolescent development. Child Development, 73(4), 1283-1309. Craighead, W. E., Kimball, W. H., & Rehak, P. J . (1979). Mood changes, physiological responses, and self-statements during social rejection imagery. Journal of Consulting & Clinical Psychology, 47(2), 3 8 5 - 3 9 6 . Crocker, J . , 8c Major, B. (1989). Social stigma and self-esteem: The self-protective properties of stigma. Psychological Review, 96(4), 608-630. Cronbach, L. J . , 8c Snow, R. E. (1977) Aptitudes and instructional methods: A handbook for research on interactions. New York: Irvington. Devine, P. G. (1989). Stereotypes and prejudice: Their automatic and controlled components. Journal of Personality and Social Psychology, 56(1), 5-18. Eagly, A. H., & Chaiken, S. (1998). Attitude structure and function. In D. T. Gilbert, S. T. Fiske, 8c G. Lindzey (Eds.), The handbook of social psychology (4th ed., Vol. 1, pp. 2 6 9 - 3 2 2 ) . New York: McGraw-Hill. Eagly, A. H., Wood, W., 8c Diekman, A. B. (2000). Social role theory of sex differences and similarities: A current appraisal. In T.T.H.M. Eckes (Ed.), The devel­ opmental social psychology of gender (pp. 1 2 3 - 1 7 4 ) . Mahwah, NJ: Lawrence Erlbaum. Fazio, R. H., Jackson, J . R., Dunton, B. C , 8c Williams, C. J . (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69(6), 1013-1027. Furstenberg, F. F., Jr., Cook, T. D., Eccles, J . , Elder, G. H., 8c Sameroff, A. (1999). Managing to make it: Urban families in high-risk neighborhoods. Chicago: University of Chicago Press. Gardner, W. L., Pickett, C. L., 8c Brewer, M . B. (2000). Social exclusion and selective memory: How the need to belong influences memory for social events. Personality and Social Psychology Bulletin, 26(4), 4 8 6 - 4 9 6 . Gilbert, D. T., Fiske, S. T., 8c Lindzey, G. (Eds.). (1998). The handbook of social psychology (4th ed.). New York: McGraw-Hill. Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., 8c Baumgardner, M. H. (1986). Under what conditions does theory obstruct research progress? Psychological Review, 93(2), 2 1 6 - 2 2 9 . Haney, C , Banks, C , 8c Zimbardo, P. (1973). Interpersonal dynamics in a simulated prison. International Journal of Criminology & Penology, 1(1), 69-97'. Hastie, R., 8c Carlston, D. E. (1980). Theoretical issues in person memory. In R. Hastie, T. M. Ostrom, E. B. Ebbesen, R. S. Wyer, Jr., D. L. Hamilton, 8c D. E. Carlston (Eds.), Person memory: The cognitive basis of social perception (pp. 1-53). Hillsdale, NJ: Erlbaum. Hewstone, M . (1996). Contact and categorization: Social psychological interventions to change intergroup relations. In C. N. Macrae, C. Stangor, 8c M. Hewstone (Eds.), Stereotypes and stereotyping (pp. 3 2 3 - 3 6 8 ) . New York: Guilford. Higgins, E. T . (1989). Self-discrepancy theory: What patterns of self-beliefs cause people to suffer? In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 2 2 , pp. 93-136). San Diego: Academic Press.

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Irwin, F. W. (1944). The realism of expectations. Psychological Review, 51, 120-126. Jahoda, G. (1986). Nature, culture and social psychology. European Journal of Social Psychology, 16(1), 17-30. Jessor, R., 8c Jessor, S. L. (1973). The perceived environment in behavioral science. American Behavioral Scientist, 16(6), 801-827. Jones, E. E. (1985). Major developments in social psychology during the past five of social psychology decades. In G. Lindzey 8c E. Aronson (Eds.), Handbook (3rd ed., Vol. 1, pp. 4 7 - 1 0 7 ) . New York: Random House. Jones, E. E., 8c Nisbett, R. E. (1972). The actor and the observer: Divergent perceptions of the causes of behavior. In E. E. Jones, D. E. Kanouse, H. H. Kelley, R. E. Nisbett, S. Valins, & B. Weiner (Eds.), Attribution: Perceiving the causes of behavior (pp. 79-94). Morristown, NJ: General Learning Press. Judd, C. M., 8c Kenny, D. A. (1981). Process analysis: Estimating mediation in treatment evaluations. Evaluation Review, 5, 602-619. Kamin, L. J . (1974). The science and politics of I.Q. Potomac, M D : Lawrence Erlbaum. Kenny, D. A. (1985). Quantitative methods for social psychology. In G. Lindzey 8c E. Aronson (Eds.), The handbook of social psychology (3rd ed., Vol. 1, pp. 4 8 7 - 5 0 8 ) . New York: Random House. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality & Social Psychology Review, 2(3), 196-217'. Kuhn, T. S. (1970). The structure of scientific revolutions: Chicago: University of Chicago Press. Latane, B., 8c Darley, J . M . (1970). The unresponsive bystander: Why doesn't he help? New York: Appleton-Crofts. Lewin, K. (1951). Field theory in social science: Selected theoretical papers (D. Cartwright, Ed.). New York: Harpers. Lindzey, G. (Ed.). (1954). The handbook of social psychology (1st ed.). Cambridge, MA: Addison-Wesley. of social psychology Lindzey, G., 8c Aronson, E. (Eds.). (1969). The handbook (2nd ed.). Reading, MA: Addison-Wesley. Lindzey, G., 8c Aronson, E. (Eds.). (1985). The handbook of social psychology (3rd ed.). New York: Random House. Macrae, C. N., Milne, A. B., 8c Bodenhausen, G. V. (1994). Stereotypes as energysaving devices: A peek inside the cognitive toolbox. Journal of Personality and Social Psychology, 66(1), 37-47. Magnusson, D., 8c Bergman, L. R. (1990). A pattern approach to the study of pathways from childhood to adulthood. In L. N. Robins 8c M . Rutter (Eds.), (pp. 101-115). Straight and devious pathways from childhood to adulthood Cambridge, UK: Cambridge University Press. Markus, H. R., 8c Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 2 2 4 - 2 5 3 . Markus, H., 8c Zajonc, R. B. (1985). The cognitive perspective in social psychology. In G. Lindzey 8c E. Aronson (Eds.), The handbook of social psychology (3rd ed., Vol. 1, pp. 137-230). New York: Random House. Milgram, S. (1974). Obedience to authority. New York: Harper 8c Row. Milgram, S., Bickman, L., 8c Berkowitz, L. (1969). Note on the drawing power of 13(2), crowds of different size. Journal of Personality and Social Psychology, 79-82. Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379-387.

The Methodological

Assumptions

of Social

Psychology

Moscovici, S. (1972). Society and theory in social psychology. In J . Israel 8c H. Tajfel (Eds.), The context of social psychology: A critical assessment (pp. 17-81). London: Academic Press. Nezlek, J . B., Kowalski, R. M., Leary, M. R., Blevins, T., & Holgate, S. (1997). Personality moderators of reactions to interpersonal rejection: Depression and trait self-esteem. Personality and Social Psychology Bulletin, 23(12), 1235-1244. Nisbett, R. E., 8c Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice-Hall. Oyserman, D., Kemmelmeier, M., 8c Coon, H. M. (2002). Cultural psychology, a new look: Reply to Bond (2002), Fiske (2002), Kitayama (2002), and Miller (2002). Psychological Bulletin, 128(1), 110-117. Pennebaker, J . W. (2002). What our words can say about us: Toward a broader language psychology. Psychological Science Agenda, 15, 8-9. and persuasion: Central and Petty, R. E., 8c Cacioppo, J . T. (1986). Communication peripheral routes to attitude change. New York: Springer-Verlag. Polanyi, M. (1958). Personal knowledge. London: Routledge 8c Kegan Paul. Popper, K. (1959). The logic of scientific discovery. London: Hutchinson. (Original work published 1934) Rozin, P. (2001). Social psychology and science: Some lessons from Solomon Asch. Personality & Social Psychology Review, 5(1), 2-14. Schwarz, N., 8c Bless, H. (1992). Constructing reality and its alternatives: An inclusion/exclusion model of assimilation and contrast effects in social judgment. In L. L. Martin 8c A. Tesser (Eds.), The construction of social judgments (pp. 217-245). Hillsdale, NJ: Lawrence Erlbaum Associates. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51(3), 515-530. Shadish, W. R., Cook, T. D., 8c Campbell, D. T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin. Shelton, J . N. (2000). A reconceptualization of how we study issues of racial prejudice. Personality & Social Psychology Review, 4(4), 374-390. Sherif, M., Harvey, O. J . , White, B., Hood, W., 8c Sherif, C. (1961). Intergroup con­ flict and cooperation: The Robber's Cave experiment. Norman: Institute of Group Relations, University of Oklahoma. Sherman, J . W. (1996). Development and mental representation of stereotypes. Journal of Personality and Social Psychology, 70, 1126-1141. Sherman, J . W., Lee, A. Y . , Bessenoff, G. R., 8c Frost, L. A. (1998). Stereotype efficiency reconsidered: Encoding flexibility under cognitive load. Journal of Personality and Social Psychology, 75(3), 589-606. Steele, C. M., 8c Aronson, J . (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 69(5), 797-811. Swann, W. B. (1987). Identity negotiation: Where two roads meet. Journal of Personality and Social Psychology, 53(6), 1038-1051. Tesser, A., 8c Bau, J . J . (2002). Social psychology: Who we are and what we do. Personality & Social Psychology Review, 6(1), 72-85. Tooby, J . , 8c Cosmides, L. (1992). The psychological foundations of culture. In J . Barkow, L. Cosmides, 8c J . Tooby (Eds.), The adapted mind: Evolutionary psychology and the generation of culture (pp. 19-136). New York: Oxford University Press.

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Tukey, J . W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley. Turner, J . C , & Onorato, R. S. (1999). Social identity, personality, and the selfconcept: A self-categorizing perspective. In T. R. Tyler R. M. Kramer, & O. P. John (Eds.), The psychology of the social self: Applied social research (pp. 11-46). Mahwah, NJ: Lawrence Erlbaum. Wegner, D. M . (1992). The premature demise of the solo experiment. Personality and Social Psychology Bulletin, 18(4), 5 0 4 - 5 0 8 .

C H A P T E R

3

Ethical Issues in Social Psychology Research ALLAN J . KIMMEL ESCP-EAP,

I

European

School of

Management

t is n o w c o m m o n l y understood within

research practices have entered into the collective

social psychology that decisions about

psyche of investigators within the discipline.

particular research questions and associ-

Less than 5 0 years ago, research ethics rarely, if

ated methodological issues are inextricably

ever, represented a formal component of the

bound to a wide range o f ethical considera-

training and practice of social psychologists. As

tions. This understanding, however, has not

Kelman ( 1 9 9 6 ) has noted, prior to the 1 9 6 0 s

always been as evident as it might have been,

the idea had not yet taken hold that systematic

but rather grew out of an extended process of

attention to researchers' moral obligations to

self-reflection and debate. Focused attention on

research participants and to society as a whole

ethical issues within the discipline also devel-

represented an integral element o f the research

oped amid fears that future research necessarily

process. The fact that today these matters are

would be restricted as a result o f the growing

considered as a matter of course suggests that

influence of external regulation by governmen-

social psychologists have taken great strides in

tal and other regulatory bodies. Although ethi-

terms o f acknowledging and responding to their

cal considerations played a minor role, if any

ethical and moral responsibilities. Nonetheless,

role at all, in the research process during social

it is understandable that the ethical dimension

psychology's formative period, today they have

can be seen as a source o f added complications

a formidable influence on most of the decisions

in the actual conduct of a research investiga-

relative to the planning and conduct of investi-

tion. Confronted by an increasingly daunting

gations, from the recruitment o f participants to

array o f ethical guidelines, governmental regu-

the subsequent application o f research findings.

lations, and institutional review, investigators

It is impressive to recognize the relative

often are compelled to weigh methodological

rapidity by which the moral imperatives of

and ethical requirements in order to choose

AUTHOR'S NOTE: The author thanks the volume editors for their constructive comments during the preparation of this chapter.

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH whether and how to pursue particular research

immediate data-collection setting also have

questions. The practical difficulties imposed by

been addressed. For example, research find-

attempts to cope with these two sets o f demands

ings might be exploited for various political or

often are linked to the recognition that the most

personal ends to the detriment o f certain soci-

methodologically sound study is not necessarily

etal groups; ethical misconduct could reduce

the most ethical one, and vice versa.

public trust in psychologists or negatively

In retrospect, much o f the impetus for ethical progress in social psychology was sparked

influence perceptions o f the scientific process; and

mass-mediated

accounts o f unethical

by a significant increase in the implementation

social psychological research could tarnish the

of ethically questionable research practices.

discipline's public image and ultimately jeop-

For example, by the m i d - 1 9 7 0 s , the practice

ardize community support and funding for the

of deceiving research participants in social

research enterprise. Indeed, it is essential that

psychology studies had

judgments pertaining t o an

become c o m m o n -

place. According to various estimates, the

investigation's

moral dimension take into account the possi-

percentage o f studies using deception in the

ble effects o f the research on all those impli-

Journal

cated by it, whether directly or indirectly.

of Personality and Social

Psychology

(JPSP) rose from around 2 0 % in 1 9 6 0 to nearly 7 0 % in 1 9 7 5 (e.g., Adair, Dushenko, & Lindsay, 1 9 8 5 ; M c N a m a r a & W o o d s , 1 9 7 7 ) . W h e n psychological journal editors

were

asked to nominate studies they considered to be empirical landmarks in social psychology, the five most frequently nominated involved elaborate laboratory deceptions o f the studies' purpose

and

procedures

(Diamond

&

M o r t o n , 1 9 7 8 ) . It is possible that most of, if not all, these studies would not have been approved (at least in their original form) by contemporary ethical review boards, reflecting the necessity to bear in mind the impact o f the prevailing ethical climate on judgments pertaining to research issues.

CHAPTER OVERVIEW This chapter focuses on ethical issues in social psychology pertaining

to decisions a b o u t

particular research questions and associated methodological approaches. T h e discussion begins with a consideration o f the evolution o f ethical debate and regulation in the discipline. Next, an overview is provided o f the ethical issues that emerge in the conduct of experimental, field, and applied research, including a focus on problems related to the use o f deception. At various points, I refer to my own ongoing research on rumors in order to illus-

Although deception arguably has proven t o

trate some o f these issues. T h e last section o f

be the most pervasive ethical issue in social

the chapter addresses issues related to ethical

psychology research, other practices also have

safeguards and the impact o f ethical review.

aroused considerable concern, including the invasion o f participants' right to privacy, failure to protect the anonymity o f respondents

THE EVOLUTION OF ETHICAL

or the confidentiality o f data,

DEBATE AND REGULATION IN

unobtrusive

observations o f unsuspecting participants, and

SOCIAL P S Y C H O L O G Y

the use o f psychological or physically risky manipulations

in experimental studies. In

Developments toward ethical regulation often

addition t o the possibility that research prac-

tend to follow in the wake o f disclosures o f

tices may pose a risk o f harm directly to indi-

ethical misconduct and

vidual participants, issues pertaining to the

studies involving the mistreatment o f research

implications o f research that go beyond the

participants (Kuschel, 1 9 9 8 ) . For example, in

the reporting

of

Ethical Issues in Social Psychology sociology, trade"

Humphrey's

study

(1970)

Research

\

"tearoom

social psychology is Milgram's (1963) obedience

o f h o m o s e x u a l behavior in

studies, a series o f laboratory investigations

public restrooms aroused considerable contro-

conducted from

versy and prompted sociologists to reevaluate

research volunteers w h o were led to believe

their

code.

they were administering dangerous electric

Humphrey did not inform the participants

shocks to an innocent victim. Although osten-

discipline's formative

ethical

1 9 6 0 to

1 9 6 4 involving

w h o congregated in the restrooms that he was

sibly presented as a study about learning, the

observing their conduct for research purposes;

experiment's intent was to observe the extent

later, he altered his appearance and presented

to which the participants would obey the

himself as a health service worker in order to

orders o f a malevolent authority (the experi-

interview the men at their homes for the pur-

menter). W h a t aroused great concern in the

pose o f obtaining information about homo-

scientific community, in addition to the fact

sexual lifestyles. In political science, a study

that 6 5 % o f all participants in the standard

that aroused extensive controversy was Project

experiments

administered

the

strongest

Camelot, a 1 9 6 4 research project conducted

" s h o c k s " in response to the authority's com-

under the auspices o f the U.S. government that

mands, was that participants apparently expe-

was intended to study counterinsurgency in

rienced

Latin America (Horowitz, 1 9 6 7 ) . The project

physical distress as a function o f the experi-

was quickly condemned on ethical grounds as

mental guise. Despite the lack o f direct evi-

a blatant attempt by the project sponsors to

dence that anyone suffered lasting harm as a

identify means for suppressing popular revolts

result o f the research, the obedience experi-

in foreign countries.

ments remain at the forefront o f most discus-

intense

psychological upset

and

These and similar studies added fuel to the

sions o f research ethics in social psychology

fire o f public and professional discourse on

and related disciplines. Although the target o f

matters related t o scientific ethics and the

scathing ethical and methodological attacks

treatment o f research participants. This also

for m a n y years, M i l g r a m ' s research

has

was the case in psychology, most notably as a

received renewed attention by those w h o have

result o f experimentation involving deception.

praised its insights into obedient behavior

In a series o f field experiments conducted

(Miller, Collins, & Brief, 1 9 9 5 ) .

during the early 1 9 6 0 s , young military recruits

Social psychology experiments rarely elicit

unknowingly were subjected to bogus life-

the sorts o f intense reactions in participants

threatening emergencies (such as the apparent

as in the studies described here; nonetheless,

imminent crash landing o f their aircraft) in

the latter were instrumental

order to study their reactions to psychological

an impetus for ethical scrutiny within the

stress (Berkun, Bialek, Kern, & Yagi, 1 9 6 2 ) .

discipline and the public domain.

in

providing

In another controversial study, alcoholic volunteers were led to believe they were participating in an experiment to test a possible treatment for alcoholism but instead were injected with a drug that caused a terrifying,

Governmental Regulations for Behavioral Research in the United States

albeit temporary, respiratory paralysis, lead-

Federal safeguards concerning the rights

ing many o f the participants to believe that

and welfare o f human research participants

they were dying (Campbell, Sanderson,

have been in place as part o f U.S. Public Health

&

Laverty, 1 9 6 4 ) . N o doubt the most widely known

Service (PHS) policy since 1 9 6 6 , although the and

initial focus was limited to clinical research in

controversial example o f deceptive research in

medical fields. In 1 9 6 9 , PHS policy was

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH extended to cover all research involving human

argued that deceptive procedures pose certain

participants, including biomedical and behav-

psychological risks to participants, including

ioral investigations. T h e concept o f consent

loss o f self-esteem and dignity and a possible

was emphasized in these initial governmental

loss o f trust in legitimate authority. In view of

regulations as well as the necessity for commit-

the methodological drawbacks of some decep-

tee review of research that put participants at

tion studies, she further questioned whether

risk. In July, 1 9 7 4 , following Senate hearings

such studies held much potential for serious

on some prominent cases o f research abuses,

benefit. According to other prominent critics,

Congress

deception had become a commonplace proce-

signed

into

law

the

National

Research Act ( 1 9 7 4 ) , which led to the creation

dure that increasingly was being employed

of the National Commission for the Protection

even in research situations where it was unnec-

of

essary (Kelman, 1 9 6 7 ) .

Human

Subjects

of

Biomedical

and

Behavioral Research.

Largely in response to the changing nature

Following an extended period o f review,

of research in the discipline, the American

the National Commission ( 1 9 7 9 ) issued its

Psychological Association (APA) has taken

final recommendations in 1 9 7 8 along with the two-volume Belmont

Report,

which proposed

norms for ethical conduct in various areas,

steps to codify and periodically modify ethical principles for human participant

studies.

These principles have served as a model for

including competence o f the researcher, iden-

other professional associations around the

tification o f risks and benefits, appropriate

world

selection o f participants, voluntary informed

derived largely through an empirical approach

consent, and compensation for injury. Formal

based on a survey o f critical incidents pertain-

regulations were published by the Department

ing to ethical dilemmas experienced by the

and are unique in that they were

of Health and H u m a n Services ( D H H S ) in

association's members. The current version o f

the January 2 6 , 1 9 8 1 , issue o f the

Federal

the code (APA, 2 0 0 2 ) , is the result of a 5 0 -

M o s t noteworthy among the regula-

year history o f development and revision and

Register.

proper

is presumed to reflect the values of APA

review by institutional review boards (IRBs)

members as well as the moral growth of the

for the approval o f DHHS-funded research

discipline o f psychology (Jones, 2 0 0 1 ) .

tions were requirements mandating

projects, including the exemption of broad

T h e APA research principles became sub-

categories o f research that posed little or no

stantially more stringent in the 1 9 7 3 version of

risk o f harm. The current D H H S regulations

the code (American Psychological Association,

governing the implementation o f the National

Ad H o c Committee on Ethical Standards in

Research Act, which have been amended over

Psychological Research, 1 9 7 3 ) , as well as in

the years, are available in the Code of Federal

subsequent versions, in light o f the controversy

Regulations issued by the Office for Protection

surrounding the dramatic increase in deception

From Research Risks ( O P R R ) ( 1 9 9 1 ) .

experiments and

criticisms regarding

the

broadly stated and qualified nature of some of the principles. M a n y apparently perceive the

Professional Ethical Standards

current principles to be more exacting than

By the m i d - 1 9 6 0 s , a growing number o f

they were intended to be, as they were meant

critics had begun to question the prolifera-

to be applied within a cost-benefit framework

tion o f deceptive and potentially

harmful

that permits the researcher to consider whether

in psychological research.

the benefits o f the research outweigh possible

For example, Baumrind ( 1 9 6 4 ) , largely in

harm to participants. This utilitarian approach

reaction t o M i l g r a m ' s obedience research,

has continued to fuel further debates over

manipulations

Ethical Issues in Social Psychology interpretation

o f the principles and

their

implementation.

Exceptions

to

informed

Research consent

\ are

allowed under federal regulations and profes-

The current principles emphasize voluntary

sional standards when certain conditions are

participation and informed consent as funda-

met, such as the research involves no more

mental prerequisites for human

than minimal risk to participants; the study

participant

research. The principle o f voluntary informed

could not practically be carried out

with

consent was most notably introduced in the

informed consent; the research consists o f

Code ( 1 9 4 9 ) , a general set o f stan-

anonymous questionnaires, naturalistic obser-

dards formulated to prevent atrocities like

vations, or archival research for which disclo-

those perpetrated by Nazi researchers during

sure o f responses would not result in negative

World W a r II and the forerunner t o all subse-

consequences for participants; and so forth

Nuremberg

quent guidelines governing experimentation with human

(Schuler, 1 9 8 2 ) .

participants

(see APA, 2 0 0 2 ; O P R R , 1 9 9 1 ) .

Obtaining

partially or fully informed consent in practice

Following increasing disclosures o f objection-

can be difficult, particularly when the cogni-

able research in the social and behavioral

tive capacity o f participants is limited or

sciences, it became apparent that the Nuremberg

impaired. Such is the case when participants

Code (along with the subsequent Helsinki dec-

are selected from vulnerable groups, such as

larations for guiding the conduct o f health and

children, the elderly, the mentally disabled and

medical researchers) was insufficient to ensure

handicapped, and underprivileged persons.

the safety and well-being o f research participants (Kuschel, 1 9 9 8 ) .

from vulnerable groups consist o f whether

Consistent with U.S. federal guidelines and

they understand what they are told about the

Psychological

nature and purpose o f a study, are able to

the APA standards require informed

weigh the risks that a study may entail for

some scientific journals (e.g., Science),

The primary ethical concerns for persons

consent for nearly all human research. W h e n

them, and can consent on their own, without

obtaining informed consent from prospective

the agreement o f a legal guardian. There also

research participants, social psychologists are

is the possibility that they may have a lower

expected t o communicate the following types

tolerance for potentially risky or deceptive

of information (APA, 2 0 0 2 ) : (a) the purpose

manipulations. Professional ethics codes are

of the research, expected duration, and pro-

consistent in specifying that research involving

cedures; (b) the right to decline to participate

children and participants with

and to withdraw from the research once par-

that will limit their understanding

ticipation has begun; (c) the foreseeable con-

special safeguarding procedures. A c o m m o n

impairments requires

withdrawing;

approach to dealing with the issue o f consent

(d) reasonably foreseeable factors that may be

is to use parental or proxy consent as a substi-

sequences

o f declining

or

expected to influence willingness t o partici-

tute for, or in addition to, obtaining consent

pate, including potential risks, discomfort, or

from the vulnerable participant (e.g., British

adverse effects; (e) any prospective research

Psychological Society, 1 9 9 5 ) .

benefits;

(f) limitations on

confidentiality;

A related issue that emerges in considerations

(g) incentives for participation; and (h) w h o m

of informed consent pertains to the potential

to contact for questions about the research

coercion (and related forms o f exploitation) of

and participants' rights. F o r guidance as to

individuals for research participation. This issue

the design, administration, and evaluation o f

has been most widely discussed in terms of the

valid consent forms and

use o f students for research purposes. University

Fischman

(2000),

Kimmel ( 1 9 9 6 ) .

procedures,

Grundner

(1986),

see and

subject pools typically comprise students who are encouraged (and sometimes required) to

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH participate in campus research in order to obtain

response rates (American Association for

some sort of incentive. Although the incentive

Public Opinion Research [ A A P O R ] ,

may be monetary in nature, it is more likely

Bearden, Madden, & Uscategui, 1 9 9 8 ) , has

2002;

to take the form of course credit or fulfillment

increased in psychology over the past decade as

of a prerequisite to obtaining a course grade

psychologists have made a more concerted

(Lindsay & Holden, 1 9 8 7 ) . Despite college

effort to include nonstudent adult participants

recruits representing a convenient, inexpensive

in their samples (Kimmel, 2 0 0 1 ) .

study population, several ethical concerns have

With regard to deception, the APA code

been voiced about the use of this population

dictates that deception should be used only if

(e.g., Korn, 1 9 8 8 ) . Specifically, there are fears

a study's results are likely to be sufficiently

that alternative means of satisfying course

important, an alternative nondeceptive proce-

requirements may not be offered or else are

dure is not feasible, the research is not likely to

excessively time-consuming or noxious; that

cause physical pain or severe emotional distress,

students may receive little if any educational

and the deception is to be explained to partici-

debriefing; and that readily accessible complaint

pants as early as possible. In short, the critical

procedures are not available. Several guidelines

determinant o f the acceptability of deception

have been proposed for the ethical use of

is that it is "justified by the study's significant

student subject pools (see, for example, APA,

prospective scientific, educational, or applied

1 9 8 2 ) , and in recent years students have

value" and that the only way the study feasibly

been offered an increasing array of alternatives

could be carried out is by introducing deception

to the research requirement (McCord, 1 9 9 1 ;

as an integral aspect of the research procedure

Raupp & Cohen, 1 9 9 2 ) .

(APA, 2 0 0 2 , p. 1 1 ) . The criterion o f value has

Keeping in mind individuals' freedom to

proven contentious for critics who point to the

decline to participate, several strategies for

subjective nature inherent in die researcher's

recruiting research participants have been pro-

determination of a study's prospective benefits.

posed, including financial inducements, lotter-

However, in recent years, the federal require-

ies, gifts, or the simple promise to provide a

ment that scientists obtain institutional approval

summary of the study's results. Perhaps the

prior to conducting research has evolved as an

most serious ethical concern related to recruit-

important component of the APA code.

ment involves the point at which offers of

In addition to governmental regulations and

inducements to participate become coercive,

the APA code, social psychologists can obtain

thereby threatening one's right not to partici-

guidance for research conduct by consulting

pate or one's freedom to withdraw from an

standards that have been promulgated by other

ongoing study. As of yet, there are no clear

professional groups. For example, codes of

ethical standards for obtaining guidance in

ethics for survey research have been developed

such cases beyond the necessity for researchers

by

to maintain a sensitivity to the possibility that

Chamber of Commerce/European Society for

organizations such as the International

they are exerting excessive pressure on targeted

Opinion and Marketing Research (ICC/ESO-

participants

to enhance recruitment.

For

M A R ) ( 2 0 0 1 ) , the Council of American Survey

example, the APA code recommends

that

Research Organizations (CASRO) ( 1 9 9 5 ) , the

"reasonable efforts" be made to avoid offers

American Association for Public Opinion

of excessive or inappropriate inducements to

Research (AAPOR) ( 2 0 0 2 ) , and the Council

potential participants that "are likely to coerce

for Marketing and Opinion Research ( C M O R )

participation" (APA, 2 0 0 2 , p. 1 1 ) . T h e use o f

( 1 9 9 9 ) . As well as clarifying the rights of

monetary incentives, which has been recom-

respondents, clients, or sponsors and

mended as an effective practice for maximizing

professional responsibilities o f researchers,

the

Ethical Issues in Social Psychology

Research

\

these codes offer extensive ethical principles in

ethical decision making. Consequently, social

the conduct of public opinion research and in

psychologists should be particularly sensitive to

the use of such research for policy and decision

foreign ethical guidelines during the conduct o f

making in the public and private sectors. The

cross-cultural research, as different standards o f

codes vary in specificity with regard to the

acceptable and unacceptable conduct are likely

range o f major ethical issues involving research

to prevail in different country contexts.

participants, including deceptive practices, invasion o f privacy, and lack o f consideration/concern for respondents (N. C. Smith &

ETHICAL DILEMMAS IN SOCIAL

Klein, 1 9 9 9 ) . Bearing in mind that many social

PSYCHOLOGICAL RESEARCH

psychologists are employed in applied settings, it is important to note that corporate codes o f

Investigations into widely researched social

conduct have been implemented by a number

psychology subject areas such as attitudes,

of companies in North America and Europe

aggression, prejudice, intimate relationships,

(Berenbeim, 1 9 9 2 ; Chonko, 1 9 9 5 ) .

group dynamics, impression formation, and

Finally, professional codes of ethical conduct

social identity commonly touch upon a variety

have been developed by a number of psycho-

of interests and values that are prerequisite and

logical associations around the world, includ-

central to ethical decision making. Because of

Psychological Society

the sensitive nature of much social psychologi-

( 1 9 8 6 ) , the Canadian Psychological Associ-

cal research, the question is not whether ethical

ation ( 1 9 9 1 ) , the British Psychological Society

dilemmas will be encountered, but rather when

ing

the

Australian

( 1 9 9 5 ) , the German Association o f Profes-

and under which circumstances they are likely

sional

to pose the most difficult problems for the

Psychologists

(1986),

the

French

Psychological Society ( 1 9 7 6 ) , the Netherlands

researcher (and other parties) involved, how

Institute o f Psychologists ( 1 9 8 8 ) , the General

such dilemmas can be anticipated before they

Assembly for the Scandinavian Psychological

emerge, and what steps can be taken by the

Associations (European Federation o f Psycho-

researcher to either avoid or satisfactorily

logy Associations, 1 9 9 8 ) , the Psychological

resolve the dilemmas. At the core o f most o f the

Association of Slovenia ( 1 9 8 2 ) , the Spanish

ethical conflicts encountered within social psy-

Psicologo (Colegio Oficial de Psicologos,

chology are the sometimes opposing interests o f

1 9 8 7 ) , and the Swiss Federation o f Psychol-

science and the protection of others. This point

ogists (Federation Suisse des Psychologues,

was emphasized in an insightful early critique

1 9 9 1 ) . Each of these codes emphasizes and

of

promotes an overriding high regard for the

Vinacke ( 1 9 5 4 ) , w h o raised important ques-

deceptive

psychology

experiments

by

well-being and dignity o f research participants,

tions about the "proper balance between the

as reflected in the attention given to such top-

interests of science and the thoughtful treat-

ics as informed consent, protection from harm,

ment of the persons who, innocently, supply the

and privacy issues (including confidentiality)

data" (p. 1 5 5 ) .

(Kimrnel, 1 9 9 6 ) . Nonetheless, the codes differ in terms o f the degree o f specificity with which principles are formulated and in the basic ethical positions taken, with stronger emphasis

Defining "Ethics," "Morality," and "Ethical Dilemma"

placed either on research benefits or risks to

A first step in being able to anticipate an

research participants. Such differences are

ethical dilemma is knowing what this and

reflective o f cultural variations in value sys-

related terms represent in the context o f

tems, moral judgments, and approaches to

research decisions. T h e word

" e t h i c s " is

51

52

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH derived from the Greek ethos,

meaning a

valid measure of behavior) against the ethical

person's character or disposition, whereas

imperative of informed consent. T h e decision

"morality" is derived from the Latin

to opt for deception in an attempt to maxi-

moralis,

meaning custom, manners, or character. As it

mize the validity of a study runs counter to

has evolved in c o m m o n usage, the

term

the obligation to be forthcoming with research

has come to pertain to questions

participants. In one form or another, argu-

about whether specific acts are consistent

ments against deception claim that because it

morality

with accepted notions o f right or wrong. T h e

involves lying and deceit, its use in research is

is more likely to be used to

morally reprehensible and may have poten-

connote rules o f behavior or conformity to a

tially harmful effects on each o f the parties

code or set o f principles (Frankena, 1 9 7 3 ;

implicated by it (e.g., Ortmann & Hertwig,

Reynolds, 1 9 7 9 ) .

1 9 9 7 ) . Adair et al. ( 1 9 8 5 ) clearly summarized

term ethical

For example, we have seen that the APA research principles condone the use of deceit

some o f the key concerns about its use in psychology by suggesting that deception

under certain specified circumstances, meaning that an investigator who deceives research participants could be considered as acting within the bounds of ethical propriety.

However,

we still may feel that the researcher's behavior was not right in a moral sense by applying the deontological principle that deceiving others can never be justified. As is apparent, deception in this case might be viewed as proper according to one set of principles (those by

violates the individual's right to voluntarily choose to participate, abuses the basic interpersonal relationship between experimenter and subject, contributes to deception as a societal value and practice, is a questionable base for development of the discipline, is contrary to our professional roles as teachers or scientists, and will ultimately lead to a loss of trust in the profession and science of psychology, (p. 61)

which professional psychologists are guided), but not according to another more general set

W h e n considering the potential dilemmas

of principles derived from a particular moral

involving the use o f deceptive research prac-

reasoning approach.

tices, it is important to recognize that the

Choices for conduct in each research situ-

effects o f deception may be positive (i.e., ben-

ation are related to the decision maker's

eficial to recipients) or negative (i.e., harmful

values, and these values must be weighed

to recipients), short or long term, and imme-

carefully when important decisions are to

diate or delayed. For example, a research

is apparent in

participant may be initially unaffected by the

made. An ethical

dilemma

research situations in which two or more

awareness o f having been duped into believing

desirable values present themselves in a seem-

that a fictitious organization had sponsored

ingly mutually exclusive way, with the values

the study but may experience a short-term loss

suggesting different courses o f action that

of self-esteem when later reading a magazine

cannot be maximized simultaneously. M a n y

article about h o w easily people are deceived

ethical issues that arise in social psychology

by researchers. In this case, the deception

research result from conflicting sets o f values

effects are negative, delayed, and short term.

involving the goals, processes, or outcomes o f

Although deception is most readily thought o f

an investigation.

as a practice that is employed during the data

Deception often lies at the core o f ethical

collection stage, in fact it may be used at each

dilemmas for social psychology researchers,

stage o f the research process (see Table 3 . 1 ) .

w h o must weigh the scientific requirements o f

Specific issues linked to the use o f deception

validity (i.e., in obtaining an objective and

are considered in detail below.

Ethical Issues in Social Psychology Table 3.1 Subject

Research

Use of Deception at Various Stages of the Research Process Recruitment

Research

Identity of researcher and/ or sponsor

Procedure

PostresearchlApplication

Misrepresentation of purpose

Purpose of research

False information about procedures, measures, etc.

Participation incentives

Withholding information

Involving people in research without their knowledge

Concealed observation

Violation of promise of anonymity Breach of confidentiality Misrepresenting implications of research results False feedback during debriefing session

SOURCE: Adapted from Kimmel and Smith (2001).

about research procedures and instructions,

ETHICAL ISSUES IN T H E CONDUCT OF LABORATORY, FIELD, AND APPLIED RESEARCH

false feedback given to the participant, and misleading information about the timing or setting o f the investigation (e.g., when the

W e next turn our attention to some o f the

study actually begins and ends or related stud-

ethical considerations that emerge in the use

ies presented as unrelated).

of specific methodological approaches com-

The

primary

justification

for

using

monly employed in the investigation o f social

deception in laboratory settings is that if

behavior. T h e degree o f control over a study's

researchers conformed to the letter o f the law

circumstances and the characteristics o f one's

regarding

role obligations are two o f the more important

deceive participants at all, then many investi-

factors likely to engender a variety of unique

gations either would be impossible to conduct

ethical dilemmas in laboratory, field,

and

informed

consent and

did

not

or would result in biased findings. Consistent

applied settings. I have recognized some o f

with this point, it has been shown

these varying dilemmas at first hand in the

informing participants o f the true purpose

context

o f my research

on

that

marketplace

and procedures o f a study can exacerbate the

rumors, as described in the following sections.

problem o f research artifacts, distorting participant responses and severely jeopardizing the tenability o f inferred causal relationships

Laboratory Research Issues

(see Broder, 1 9 9 8 ) . O n e can readily imagine

Issues related to deception have dominated

h o w a completely informed consent obtained

ethical concerns linked to social psychological

during investigations into social psychologi-

research conducted in laboratory settings. T h e

cal phenomena such as altruism and prejudice

controlled conditions o f the laboratory offer

could cause participants to behave differently

the experimenter

in order to present a more socially acceptable

a ready

opportunity

to

actively mislead participants about the true

(as opposed to a natural or typical) image to

nature o f the situation and purpose o f the

the researcher. In this way, informed consent

investigation. Other active deceptions

in labo-

essentially

operates

as

an

independent

the use o f

variable—studies conducted with or without

research "confederates" w h o act out predeter-

it may come up with vastly different results

mined roles, untrue statements about

the

(Resnick & Schwartz, 1 9 7 3 ) . Thus, for some

information

intended investigations, the decision to be

ratory

settings have included

researcher's identity, incorrect

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH made is not whether to use deception, but

risks were far outweighed by the importance of

whether the research is necessary. T h e decision

the subject matter and potential gain in knowl-

not to do a study because it would require

edge about helping behavior during emergen-

deception is perhaps as morally problematic

cies. Deceiving participants about the

true

as the decision to do a study involving decep-

nature of the project may have been the only

tion when one considers the potential loss o f

feasible way to collect data to test the causal

knowledge

hypotheses under study.

involved

(Haywood,

1976;

Rosenthal & R o s n o w , 1 9 8 4 ) .

By contrast, I have chosen to eschew the

In addition to eliciting more spontaneous behavior from participants

than

laboratory setting in my own research on the

otherwise

factors that give rise to marketplace rumors

might be the case, deception can increase the

and the efficacy of strategies to offset their

researcher's degree of methodological control

effects. Although the rumor process has effec-

over the experimental situation. These advan-

tively been investigated in some ingenious lab-

tages were evident in some of the classic stud-

oratory experiments (e.g., Kamins, Folkes, &c

ies of helping behavior carried out by Latane

Perner, 1 9 9 7 ; Tybout, Calder, & Sternthal,

and Darley ( 1 9 7 0 ) in order to

1 9 8 1 ) , I chose to study the circumstances

determine

whether the number o f bystanders

present

surrounding naturally occurring rumors using

the

non-experimental field approaches (such as in-

likelihood of intervention by any one bystander.

depth consumer interviews and mail surveys

Clearly,

involving brand managers). In so doing, I was

during an emergency would the

researchers

influence

could

not

have

expected an emergency to occur repeatedly in

willing to sacrifice a large degree o f method-

a natural setting under precisely the same cir-

ological control that the laboratory setting

cumstances with a different number o f onlook-

would have provided. Moreover, I did not

ers present during each occurrence. Instead, it

want to exacerbate the rumor problem by cre-

was more feasible for them to conduct their

ating false information in the laboratory and

studies in the laboratory, where the number of

running the risk that (even debriefed) partici-

bystanders present could be systematically

pants might spread it to others once their

manipulated in a series o f carefully contrived

involvement in the study had ended.

"emergencies" (such as an apparent fire in an adjoining room or an epileptic seizure experi-

Criticisms o f deception have been directed t o the very core o f the

methodological

enced by a research confederate). By creating

assumptions upon which the use of the proce-

such fictional environments in the laboratory,

dure depends: (a) that the level of naivete

investigators can manipulate and control the

among research participants is high, (b) that

variables of interest with much greater facility

the procedure does not produce cues that sug-

than if deception is not used.

gest to participants that deception is taking

One might legitimately question whether

place, and (c) that participant suspiciousness

the extreme deceptions used in the helping

of deception does not alter the experimental

experiments can be justified. In addition to

effect. D o u b t has been raised that deception

moral concerns about misleading research par-

adequately serves its methodological purpose

ticipants, it can be argued that the research

in many research situations, that o f preventing

paradigm exposed participants to psychologi-

participants from discovering the study's true

cal risks, including guilt and a threat to their

purpose or certain aspects o f the procedure

self-esteem for not helping, stress during the

that could influence their natural response to

emergency itself, and embarrassment at being

experimental variables. For example, some

duped by the researchers. However, one might

evidence suggests that Milgram's obedience

defend the procedures by contending that the

experiments

lacked

experimental

realism

Ethical Issues in Social Psychology

Research

j

because several participants may have seen

t o justify when ethical principles are applied

through

and are more likely t o encounter problems

the deception

(Orne 8c Holland,

1968; Patten 1977). One difficulty in assessing

when

the degree to which a deception study pos-

However,

sesses experimental realism has to do with

unlikely to cause harm to participants, they

the "pact o f ignorance" that may develop

still can be morally problematic (N. C . Smith,

between

Klein, 8c Kimmel, 2 0 0 2 ) . Some o f the possi-

the

researcher

and

participants

subjected

to

although

committee mild

review.

deceptions

are

(Orne, 1959). Participants may conceal that

ble consequences o f deception for each o f the

they

parties involved in the research process are

saw

through

a

deception

scenario

because it would compromise the value o f

summarized in Table 3.2.

their participation, and researchers may make

Another set o f related potential problems

little effort to uncover information that would

particularly salient in laboratory research

invalidate a participant's data and thereby

pertains to the levels o f suspiciousness that

delay completion o f the study. With regard to the potential consequences

participants bring to the research setting, especially for individuals w h o have been deceived

of deception, the degree o f severity o f any neg-

in previous studies, and the effects o f suspi-

ative effects must enter into decisions about

ciousness and deception on subsequent exper-

whether or not to proceed with a study as

imental performance. Some persons come to

planned. "Severe deceptions" are those that

the laboratory setting unaware that deception

create false beliefs about central, important

may take place or else have only a vague

issues related to participants' self-concept or

knowledge that it is a possibility. In such

personal behavior, as when an experimental

cases, suspicions may be aroused by demand

manipulation leads participants to believe they

characteristics (i.e., various

lack self-confidence. " M i l d deceptions" are

hints and cues), such as certain comments in

those that create false beliefs about relatively

the instructions which suggest that something

unimportant issues peripheral to participants'

is going on that is different from the experi-

task-orienting

self-concept, such as misleading them about

menter's description. Others may

the research sponsor or study purpose (Toy,

harbor suspicions as a result o f information

already

Olson, 8c Wright, 1989). Severe deceptions

obtained elsewhere (such as in the recruitment

can be expected to create negative effect both

appeal or campus scuttlebutt).

during and after actual participation in the

Additionally, there is some evidence that

research (e.g., upset or anxiety linked to a

research participants w h o have been debriefed

reduced self-image), whereas mild deceptions

may communicate the true purpose and other

are unlikely to create negative beliefs and

details o f studies to future

effect until the debriefing session at the end o f

tendency referred to as "leakage" (Diener,

the study (e.g., disappointment that the study

M a t t h e w s , 8c Smith, 1972). Expectations

participants, a

was not really supported by an environmental

about a study could have a counterproductive

protection group).

effect on the results, motivating participants to

The fact that social psychologists are more

behave in ways that do not reflect their natural

likely to employ severe deceptions that are

behaviors or compelling them to behave in

relevant to the fundamental beliefs and values

uncooperative ways in order to undermine the

of research participants than are investigators

research.

in related fields, such as consumer research,

There has not been much research on the

to some extent explains why deception has

extent of suspiciousness or leakage in the

been such a central issue in social psychology.

research setting, and researchers do not rou-

Studies involving severe deceptions are harder

tinely probe levels of participant suspiciousness.

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Table 3.2

Potential Costs and Benefits of Deception Studies

Recipient

Benefits

Costs

Participant

Increased understanding of science and the research process Feeling of having contributed to science Self-insight (from personal content revealed by deceptive probes)

Researcher

Capacity to elicit spontaneous behavior from Legal sanctions participants (e.g., if confidentiality breached) Increased degree of methodological control Undermines integrity and Enhanced reputation from successful endeavors commitment to the truth Tarnished image

Profession

Facilitates attempts to determine validity of theories, previous research, and assessment instruments

Society

Scientific advancement and progress Increased understanding of behavior Insight into applications toward the betterment of humanity

Inflicted insight Embarrassment Image of science lowered Mistrust of others

Exhausts pool of naive participants Jeopardizes community and industry support for the research enterprise Undermines trust in expert authorities and science Increased suspiciousness (e.g., self-consciousness in public)

SOURCE: Adapted from Kimmel and Smith (2001).

Estimates

of

the

overall

percentage

of

problem o f suspiciousness is its potential for

participants identified as suspicious in social

influencing research performance. Research

psychology laboratory studies have ranged

results on the effects of participant distrust are

from only 1 . 8 % to 3 % (Adair e t a l . , 1 9 8 5 ;

somewhat inconsistent (e.g., Epstein, Suedfeld,

Kimmel, 2 0 0 1 ) . These results may be some-

&

what suspect given that participants cannot be

Jackson, 1 9 6 7 ) and have led some behavioral

counted on to be totally forthcoming

Silverstein, 1 9 7 3 ; Strieker, Messick, &

in

scientists to conclude that in general there are

revealing their suspicions or knowledge about

no major differences between the data of

research procedures and hypotheses. Along

suspicious and reportedly naive participants

these lines, Taylor and Shepperd ( 1 9 9 6 ) have

(Kimmel, 1 9 9 6 ; Schuler, 1 9 8 2 ) . The effects o f

offered

the following suggestions: (a) the

suspicion on behavior in an experiment is

simple admonishment t o participants not to

likely to be mediated by several factors, includ-

discuss the details o f an experiment among

ing the participant's perceptions of the situa-

themselves is inadequate, (b) the experimenter

tion, motivations for acting on the suspicions,

should not leave participants

and the possibility that suspicion can operate

unsupervised

during a deceptive experiment in situations

differentially across different conditions of a

where information about the study could be

study (Rosnow & Aiken, 1 9 7 3 ) .

discussed, and (c) investigators should more

A final set o f considerations pertaining to

carefully evaluate the procedures they use to

the use o f deception has to do with societal

assess perceptions o f the study and levels o f

attitudes regarding its use, participant reac-

participant suspiciousness.

tions to having

Perhaps the issue that has raised the greatest concern among

researchers regarding

the

been deceived and, in a

broader sense, the impact o f deception on perceptions o f the discipline and science in

Ethical

Issues in Social Psychology

Research

\

general. Surveys intended to assess reactions

Miller (Chapter 5 ) in this b o o k , including

to deception have shown that individuals in

archival and cross-cultural research. Field research offers investigators a ready

the general population do not have serious objections

to

its

use

for

psychological

alternative to ameliorate some o f the ethical

research purposes (Collins, Kuhn, & King,

issues that arise in the laboratory context.

1 9 7 9 ; Sullivan & Deiker, 1 9 7 3 ) and that

M a n y social psychological research problems

attitudes toward behavioral science research

can

have not been negatively affected by the

necessitating the deception or manipulation of

continued use o f deception by psychologists

participants. For example, I was able to gather

(Sharpe, Adair, & Roese, 1 9 9 2 ) .

be studied in everyday reality without

Further,

a wealth o f information about the ways com-

studies have revealed that individuals w h o

panies treat rumors about their consumer

have participated in deception experiments

offerings by having brand managers respond

versus nondeception experiments

to a mail questionnaire in the very settings

reported

that they did not mind being deceived and

where

viewed the deception research as

encountered (Kimmel & Audrain, 2 0 0 2 ) .

having

greater educational benefit (e.g., Christensen,

the rumors

were

naturally

being

Unlike the approach used in my rumor

1 9 8 8 ; C. P. Smith, 1 9 8 1 ) . Overall, it appears

research, in which managers could choose

that researchers and regulators tend to be

whether or not to respond to a questionnaire,

more severe critics o f deception than are

in some field investigations participants are

current or potential research participants.

unaware that they are being observed or

Evidence suggests that deception rates in

otherwise taking part in a research study. For

social psychology have declined in recent

example, early investigators obtained samples

years as researchers respond to ethical guide-

of ongoing conversation among unsuspecting

lines, review mechanisms, and social pres-

people in such natural settings as railroad

sure. Concurrently, social psychologists have

stations, college campuses, restaurants, and

become more inclined to utilize alternative

streets in residential and commercial areas

circumvent some o f the

(e.g., Landis & Burtt, 1 9 2 4 ; M o o r e , 1 9 2 2 ) .

ethical issues, such as nondeceptive field

One obvious difference between naturalistic

research (Kimmel, 2 0 0 1 ; N i c k s , K o r n ,

field investigations and laboratory research is

procedures

that

&

that it is evident to participants in the latter

Mainieri, 1 9 9 7 ; Vitelli, 1 9 8 8 ) .

that they are participating in a research study; in fact, the very act o f entering a psychology

Field Research Issues

laboratory implies a certain degree o f tacit

A substantial amount o f behavioral science

consent to undergo scientific procedures and

research is conducted "in the field"—that is,

experiences that may involve something more

outside the artificial setting o f the laboratory.

than what is readily apparent.

Although laboratory research continues to be

Although many ethical issues emerge in non-

the preferred choice for a majority o f social

laboratory settings, two major areas of concern

psychologists,

the

stand out. The first has to do with the privacy

potential value o f field research, and there is

rights o f participants. Because the inherent

evidence that its use in social psychology has

nature of much field research is to elude the

gradually

many

now

recognize

increased over recent

decades

(Adair et al., 1 9 8 5 ; Kimmel, 2 0 0 1 ) .

awareness of those w h o are observed, the

This

possibility that participants' privacy will be

includes a rise in use o f some o f the develop-

invaded must be fully considered. The second

ing research approaches that serve as the

area o f concern involves the informed consent

focus o f the chapters by King (Chapter 8) and

of participants. As in laboratory

studies,

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH informed consent can be problematic in the

their own homes). A second dimension is the

field because much research simply could not

publicness o f the person—public personalities

be carried out with the full awareness o f

regularly are subject to observations

research participants.

reporting that would be considered invasions

and

of privacy by less public individuals. Third, the degree of anonymity provided must be consid-

Privacy

ered. Privacy clearly is maintained when the

T h e circumstances characterizing participa-

linkage between the individual and the infor-

tion in field research are such that the individ-

mation obtained for research has been com-

ual's determination

to

pletely severed; conversely, the risk of privacy

participate frequently is violated. This occurs

invasion is high when information can be

when people are not informed o f their role as

linked to identifiable persons. The aforemen-

research participants and are unaware that a

tioned conversation studies can be justified on

study is in progress. A social contract has not

ethical grounds by the public and anonymous

o f whether

or not

researcher—

nature o f the observations, as well as by the

informed consent cannot be given, and indi-

fact that the risks posed (in this case, having

viduals cannot choose to refuse to participate

information o f a private nature revealed to

or leave once the study is in progress. In many

others) were no greater than those encountered

cases, it will not be possible to debrief partici-

in daily experience.

been

established

with

the

pants or inform them o f the results once the

A final factor in privacy considerations

study has been completed. (These points also

is the nature o f the information disclosed

are relevant to the use o f archival data for

during a study. Certain information (e.g.,

research purposes.) Under such circumstances,

income level, alcohol and drug use, birth

if researchers obtain or reveal (wittingly or

control practices) can be expected to raise

unwittingly)

attitudes,

privacy issues. Ethical judgments must take

motivations, or behavior that a participant

information

about

into account the possibility that disclosed

would prefer not to have revealed, the latter's

information,

basic right to privacy will have been breached

associated with individual participants, may

(see Allen, 1 9 9 7 , for an example from field

be perceived as an invasion o f privacy.

particularly when

it can

be

research on special cultures). At least four dimensions underlie the placement of different research situations on an

Informed

Consent

invasion o f privacy continuum, a continuum

It is difficult to consider the ethical issues

that ranges from situations in which privacy

pertaining to privacy fully without taking into

could not be said to be violated (e.g., observa-

account the principle of informed consent. Even

tions of the public behavior of public figures)

the most private settings can be studied without

to situations in which privacy could be said

raising ethical concerns if the people within

to be violated (e.g., nonpublic figures who pre-

those settings freely consent to observation

sumably are unaware of the possibility they are

once sufficiently informed. However,

being observed) (Webb, Campbell, Schwartz,

field research involves situations in which

Sechrest, & Grove, 1 9 8 1 ) . First, there is the

informed consent either is not feasible or is

element o f publicness in the location o f behav-

detrimental to the interests of the research. O n

much

ior under study. People can lay less claim to pri-

the other hand, informed consent will be irrele-

vacy protections for behavior that takes place in

vant for many research activities that involve

public settings (shopping malls, airports, etc.) as

observations of ongoing public behavior or the

opposed to behavior in private settings (such as

analysis of public records and archives.

Ethical Issues in Social Psychology

Research

J

Field investigations are less likely to involve

adequacy o f informed consent and debriefing

the direct presentation o f mistruths to partici-

procedures, and the potential loss of participant

pants than

anonymity or confidentiality.

are laboratory studies. Unlike

active deceptions, where false information is provided to the participant, passive

deceptions

involve truths that are left unspoken

(i.e.,

The absence of an investigator to a certain extent reduces the potential for coercion and thus represents an ethical benefit o f Internet-

withholding key information about the study).

based research. However, it also means that

W h e n I obtained the consent o f consumers for

the researcher likely will be unable to respond

an interview study on rumors related to elec-

to participant concerns or adverse reactions

tricity that I was conducting for the French

once the study is under way. Moreover, should

national electric company (EDF), I chose to

the participant choose to withdraw early from

inform participants that the study pertained to

a study, this will undermine the possibility that

their usage behavior and concerns related

an adequate debriefing can be carried out.

t o electricity (which, in part, it did), but not

Thus, special care should be taken to ensure

that we specifically were interested in rumors.

that the informed consent process is thorough

This was to see if participants would sponta-

and includes clear instructions that will enable

neously mention rumors on their own prior to

debriefing for those persons w h o leave the

our asking direct rumor-related questions. In

study early (see Nosek et al., 2 0 0 2 , for more

our judgment, the use o f passive deception

specific recommendations). Researchers also

during the recruitment stage was justified

should take all steps necessary to protect

because it did not pose any identifiable risks

against the possibilities that data may be inter-

and we presumed that the decision to partici-

cepted by a third party or accessed once stored

pate would not have been different had par-

in files on an Internet-connected server (see

ticipants known that the study focused on

Sharf, 1 9 9 9 ) .

rumors. It would have been more difficult to justify withholding

information

about

the

study sponsor, as many o f our French participants may have been upset at having learned after the fact that the research was sponsored by the national utility company.

Applied Research Issues Applied research is oriented toward the acquisition o f information that will prove relevant to some practical problem, defined as such from the perspective o f the researcher, society, or a specific group (such as a govern-

Social

Psychology

Research

and the

ment agency, community association, or busiInternet

ness organization) (see Part V o f this volume).

As an increasing number o f social psychol-

T h e goals o f applied research are oriented

ogists begin to exploit the potential o f the

toward modifying or improving the present

Internet and other emerging technologies for

situation, as would be the case for a study

research purposes, it is likely that new ethical

intended to develop an effective advertising

dilemmas will emerge and some familiar ones

campaign for the use o f condoms in an

will be recast in a different light (see Birnbaum,

attempt to control the spread o f AIDS. M a n y

Chapter 1 6 , this volume). In their analysis of

social psychologists are employed as applied

issues pertaining to Internet-based research,

researchers in industrial

Nosek, Banaji, and Greenwald ( 2 0 0 2 ) identi-

research firms, advertising agencies, treat-

fied three key differences between Internet and

ment facilities, and government agencies to

standard

evaluate the effectiveness o f current policies

laboratory research: the physical

absence o f a researcher, the

questionable

settings,

and programs, or to design new ones.

market

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Among the more serious ethical issues in

T o illustrate, we can consider h o w ethical

the context o f applied research are those

dilemmas might have arisen during the con-

involving the misuse o f new scientific knowl-

duct o f my rumor study for E D F . O n e o f the

edge or the improper

of

company's overriding interests in electricity-

widely accepted procedures and principles.

related rumors was to identify the kinds o f

implementation

T h e inappropriate utilization o f research find-

beliefs and fears prevalent among the French

ings outside clearly stated limiting conditions

consuming public relative to electromagnetic

can have serious and quences, and

far-reaching conse-

such utilization raises some

fields

(EMFs)

emanating

from

outdoor

power plants. W h e n company representatives

social

explained to me prior to data collection that

researchers consult with and report their data

their goal was to better inform consumers and

to organizations, human service and commu-

reduce anxieties linked to distorted

nity agencies, legal and educational officials,

about E M F s , I recognized that the company's

important

ethical

questions

when

beliefs

and the like. Granted, these considerations

expectations were consistent with my own.

also are relevant within the realm o f more the-

T h e project enabled me to gain further insight

oretical (or so-called "basic") research, despite

into the social-psychological dynamics under-

the often-expressed position that non-applied

lying the emergence and spread o f rumors in

research is value free and morally neutral (see

the context o f an ongoing problem situation,

Kimmel, 1 9 8 8 ) . Ethical dilemmas pertaining

while the company searched for more effective

to the use o f research findings are more likely

means o f communicating with its customers.

to emerge, however, when the research is

W e shared mutual interests in ultimately being

conducted in collaboration with others whose

able to reduce public misconceptions and the

goals, interests, and values may be at odds

spread o f fear-inducing rumors.

with those of the researcher or, more gener-

By contrast, imagine how opposing interests

ally, those o f the scientific discipline (Mirvis &

or role conflicts could have led to ethical dilem-

Seashore, 1 9 8 2 ) .

mas in this case. If I had suspicions that the

Researchers must fulfill the obligation to treat research participants fairly; in addition,

company intended to use the results o f the research to scare homeowners

away

from

they should attempt to fulfill the expectations

areas where the company planned to erect new

of the client or research user. T h e investiga-

power stations or to foment the spread of

tor also has a responsibility to protect the

rumors so as to increase the purchase of elec-

well-being o f the public when the results are

trical safely devices, I would not have been able

put into action. Ethical conflicts may arise as

to continue my involvement with the research

the researcher recognizes that certain duties

project. T h e company's goals would have con-

and responsibilities toward one group are

flicted with my own moral values and with the

inconsistent with those toward some other

ethical standards o f my profession, which clar-

group or with one's own values. For example,

ify that research findings should not be applied

it may be that the only way t o obtain reliable

to the detriment o f individuals or groups.

data that will satisfy certain obligations to a

Ethical dilemmas involving conflicting role

client is by deceiving respondents about the

expectations should be anticipated prior to

true nature o f a study. It thus becomes one's

carrying out an investigation, especially when

responsibility as a researcher to clarify and

entering a relationship with a client whose

openly communicate from the outset one's

priorities may subsequently change. As an

own role in the situation and t o establish

alternative to forgoing involvement with a

limits in terms o f assisting the organization in

troublesome study altogether, a determination

meeting its anticipated goals.

should be made as to whether a more ethical

Ethical

Issues in Social Psychology

Research

\

research approach for obtaining the desired

APA, 2 0 0 2 ) . Ethical decision making involves

information or outcomes is available. With

balancing a set o f considerations as to how

regard to the potential misuse of scientific

best to contribute to science and human wel-

knowledge, it is important to consider such a

fare. Important aspects of this decision-making

possibility as one weighs potential costs and

process are the recognition that most research

anticipated benefits prior to conducting the

questions in science can be pursued in more

research. T o be sure, scientists should not be

than

considered responsible for a misreading or mis-

researcher is one who selects the methodologi-

interpretation of their work as long as special

cal approach that is most likely to satisfy

one

manner

and

that

the

ethical

care has been taken, when publicizing the

research goals while minimizing potentially

research, to state boundary conditions perti-

negative consequences.

nent to the usefulness o f the research in applied contexts.

Debriefing and Other Safeguards Ethical guidelines typically necessitate that

ETHICAL SAFEGUARDS AND

all deceived participants

INSTITUTIONAL REVIEW

within a reasonable period following their

be fully

debriefed

involvement in a study. This requirement is Whereas formal review and external monitor-

often cited as an important safeguard against

ing originally were limited mostly to large,

some of the potential risks inherent in the use

funded

of deception. T h e debriefing session can serve

research projects, nowadays

most

human participant studies are subjected to

a variety o f functions, foremost o f which are to

some kind o f external review process. In fact,

provide researchers with a means o f assessing

the proliferation and increased role o f ethics

whether participants were adversely affected

committees, professional standards, legalities,

by the research procedures and to serve as an

and other external restrictions have already

opportunity to eliminate any harm or lasting

subjected psychologists to a higher level o f

false impressions about the study. It is during

professional ethical accountability than is

the debriefing period that psychologists should

found in many other professions (including

refer a participant to an appropriately trained

law, politics, and marketing), where

provider if something problematic (such as

both

passive and active forms o f deception are

severe depression) has been revealed about

commonplace (Rosnow, 1 9 9 7 ) . Over time,

that person during the study. Moreover, in

participation in research has become much

special cases in which confidentiality must be

safer than many o f the everyday activities in

breached (e.g., studies in which it is learned

which people engage (Diener, 2 0 0 1 ) .

that certain individuals are suicidal or intend to

Although there are a number o f safeguards

harm others), participants can be reminded

in place for preventing most of the serious

about any limitations to confidentiality that

breaches o f ethics that might occur in social

were agreed upon during the consent proce-

psychology research, the first line o f defense for

dure, if feasible (see Behnke & Kinscherff,

the protection o f the various interests involved

2 0 0 2 , for additional recommendations).

in and affected by the research process consists

The effectiveness o f debriefing in success-

of researchers themselves. This point has been

fully correcting a participant's misconceptions

emphasized in most ethics codes, which point

resulting from deception is questionable, par-

out that

ticularly in cases where the debriefing proce-

researchers have a

professional

responsibility to evaluate carefully and thor-

dure involves only a cursory attempt by the

oughly the ethics o f their investigations (e.g.,

researcher to inform participants that they

61

62

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH were deceived. Effective debriefing may require

Lepper, & Hubbard, 1 9 7 5 ) . It is for these

both "dehoaxing" (i.e., convincing deceived

reasons that some researchers have recom-

participants that the information they had

mended

been given was in fact fraudulent and relieving

focusing on the psychological processes that

a process approach to

debriefing,

any anxiety resulting from that information)

underlie the effects o f deception and debrief-

and "desensitizing" (i.e., helping deceived par-

ing and structuring postexperimental proce-

ticipants to deal with new information about

dures accordingly (see Aronson & Carlsmith,

themselves acquired as a consequence o f their

1 9 6 8 ; Mills, 1 9 7 6 ; and T o y , Wright, e t a l . ,

behavior during the study). It is possible that

2 0 0 1 , for specific suggestions for designing

the realization that one has been deceived

thorough process-oriented debriefings).

could result in a loss o f self-esteem and embar-

Given these points, it is clear that an effec-

rassment, in addition to creating a negative

tive debriefing interview should be treated

attitude toward the researcher or science (e.g.,

seriously as an

Baumrind, 1 9 8 5 ) . In this light, it is important

research process. T h e researcher should bear

to recognize that

process,

in mind its functions as an educational tool as

although designed to resolve ethical problems

well as a method for identifying and amelio-

and

provide

the debriefing

essential element o f the

a methodological check

research methods, paradoxically can

on

rating

any

adverse

effects.

Initially,

the

have

researcher should explain the procedures and

unintended adverse effects on research partic-

reasons for them in language that is under-

ipants (Toy, Wright, & Olson, 2 0 0 1 ) . In some

standable to participants (this may require

cases, for example, it may be appropriate to

pilot testing), including a discussion o f the

withhold

importance o f the study and its relevance to

certain information

during

the

debriefing (e.g., about individual differences)

understanding social behavior. W h e n decep-

when it is judged that awareness could cause

tion is revealed, the researcher should sensi-

more harm than good to participants.

tively explain that the procedure was selected

Unless debriefing is carried out with "care,

as a last resort, apologize for having used it,

effort, and vigilance" (Holmes, 1 9 7 6 , p. 8 6 7 ) ,

and fully explain h o w the deception was car-

there is the possibility that persons already

ried out, perhaps by displaying and explaining

deceived once may question the validity o f the

specific research materials. During the entire

information provided during the debriefing.

process, one needs to carefully monitor and

This is one reason that deceptive debriefings

appropriately

respond

reactions

to the

and

participant's

are especially ill-advised. T h e so-called "perse-

affective

verance process," whereby perceptions and

encouraging honest feedback about the study.

comments

while

beliefs created during a study continue long

As previously discussed with respect to

after they have been discredited, also may cast

research using the Internet, debriefing tends to

doubt on the effectiveness o f debriefings in

be more difficult to carry out in nonlaboratory

undoing the effects o f deceptive manipula-

settings, especially in cases where participants

tions. It has been shown that self-relevant and

initially are unaware that they have been stud-

nonself-relevant perceptions (e.g., created by

ied for research purposes, are no longer acces-

deceptive feedback following

experimental

sible to the researcher, or are unwilling to pay

tasks) may become cognitively detached from

attention t o the debriefing. In certain situa-

the evidence that created them; as a result,

tions, such as the naturalistic studies o f

even after the basis for the perceptions is dis-

conversations described above,

confirmed (via a debriefing), individuals may

participants once the observations have been

tend to cling to the original beliefs (Ross,

made could do more harm than good. There

debriefing

Ethical

Issues in Social Psychology

Research

\

is not much a researcher can do if a participant

system o f external review for overseeing the

indicates displeasure at having been secretly

ethicality o f research. T h e review process con-

observed, given that the observation already

sists o f a set of mechanisms, including I R B

has been carried out. Moreover, the debriefing

evaluations o f research proposals near the

could serve to raise levels o f discomfort or

beginning o f a study and continuing through

paranoia in other public settings and could

journal editor scrutiny of the procedures in

have a negative impact on the image o f scientists

research reports submitted for publication.

in general.

The

Another somewhat more uncertain remedy for some o f the potential adverse effects o f deception is forewarning,

whereby researchers

ethical review o f proposals now is

required at nearly all American and Canadian research institutions

before researchers are

given a green light to proceed with planned

take steps to brief participants about the study

investigations. (External review o f human

at the outset, informing them that certain

participant

information may have to be withheld until the

North America; see Kimmel, 1 9 9 6 ) . Certain

research varies greatly

outside

end o f the investigation and that they are free

additional hurdles may have to be cleared

to withdraw at any time. T h e researcher then

prior to reaching the review board stage (e.g.,

can carry out the study only with those indi-

departmental approval in academic and orga-

viduals who are willing to continue. As a form

nizational settings; parental consent; school,

of limited consent, forewarning may be seen

hospital, or prison

as ethically preferable to not obtaining con-

further review may occur at various intervals

sent at all because individuals essentially agree

once data collection has begun.

board

approval),

and

to be deceived and the researcher will not

T o ensure the protection and welfare o f

have directly misled them (Diener & Crandall,

participants, review boards typically attempt

1 9 7 8 ) . However, some participants may be

to ascertain that the anticipated benefits o f an

sensitized by the forewarning to engage in

investigation are greater than any risks posed

problem-solving behavior aimed at identifying

and that informed consent procedures are ade-

the nature o f the deception (Geller, 1 9 8 2 ) . In addition to forewarning, other alterna-

quate. Whereas at one time this formal review process emphasized the protection o f partici-

tive procedures to deception in the laboratory

pants from "extraordinary risks," the identifi-

setting have been proposed, such as role play-

cation o f even everyday risk n o w is obligatory

ing and simulations (see Greenberg, 1 9 6 7 , and

for all proposals. During such a review, the

Geller, 1 9 8 2 , for a discussion o f these proce-

investigator is required to present

dures). In general, these procedures appear to

information about all aspects o f a proposed

have only limited potential and are unlikely to

study, including specifics about the character-

supersede the use o f deception in the foresee-

istics o f participants,

able future.

research materials, the nature o f any decep-

detailed

the procedure

and

tions, confidentiality, risks, and method o f

Institutional Review Beyond the decision-making responsibilities

debriefing. Over the years, membership requirements and review criteria for IRBs have undergone

of the individual researcher, whose objectivity

several

may come into question as a result o f a vested

Hansen, 2 0 0 1 ) . The initial U.S. federal regula-

interest in conducting a study guided more by

tions requiring I R B review stipulated that an

changes

(Gray

&c C o o k e , 1 9 8 0 ;

methodological and theoretical concerns than

I R B was to consist o f at least five members with

ethical ones, there now exists an extensive

varying backgrounds and fields of expertise, at

63

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH least one member who was not affiliated with

whether the scientific aspects of investigations

the institution, and representation from each

ought to be taken into account by IRBs, espe-

gender (Department of Health, Education, and

cially by members w h o lack scientific expertise

Welfare, 1 9 7 5 ) . The composition of review

(e.g., C o l o m b o , 1 9 9 5 ; Diener, 2 0 0 1 ) .

boards n o w has broadened so that a majority

Evidence regarding

the effectiveness o f

of the participants may not be researchers,

ethical

but rather members o f the clergy, lawyers,

research participants

medical professionals, and the like w h o have

impact o f such committees on research is

review

committees from

in

protecting

risk and

the

minimal familiarity with the methodological

somewhat mixed. Mueller and Furedy ( 2 0 0 1 )

intricacies o f the research process and per-

have pointed out some o f the

difficulties

haps little appreciation o f the potential merits

inherent in attempts to assess review board

of scientific research. T h e efficiency and fair-

performance, including the probability that

ness o f an I R B no doubt can be maximized

unscrupulous

when it consists o f a diversity o f members

review process entirely; the built-in inade-

whose expertise is commensurate with the

quacies o f considering number o f incidents

researchers will bypass

the

types of research that the committee typically

(e.g., participant complaints) over time as an

reviews. Diversity is essential in light o f evi-

indicator o f effectiveness; and the misconcep-

dence suggesting that differences in individual

tion that "a problem found" with a research

background characteristics (such as gender,

proposal equates with "an incident avoided."

and

Further, a growing body o f evidence suggests

moral philosophy) lead to predictable biases

age, professional experience, culture,

that there may be important deficiencies in

in

the performance o f some I R B s (Ceci, Peters,

ethical

judgments

(Kimmel,

1991;

Schlenker & Forsyth, 1 9 7 7 ) .

& Plotkin, 1 9 8 5 ; M o r d o c k , 1 9 9 5 ; Prentice & Antonson, 1 9 8 7 ; Shea, 2 0 0 0 ) ,

particularly

in terms o f inconsistencies in the application

Impact and Effectiveness of the Review Process T h e expanded influence o f external review

of

decision

standards

and

subsequent

recommendations. Several

suggestions

for

improving

the

has brought with it a growing concern that

review process have been offered. For example,

review boards are overstepping their intended

to sensitize I R B members to the costs and ben-

role in an overzealous effort to force behav-

efits o f doing and not doing research, a case-

ioral and social research into a biomedical

b o o k of actual research protocols that have

mold, thereby making it increasingly difficult

received extensive review and analysis by both

for many researchers to proceed with their

investigators and participants can be provided

studies. Considerations that are not specifi-

(Rosnow, Rotheram-Borus, Ceci, Blanck, &

cally related to the rights and welfare o f

Koocher, 1 9 9 3 ) . For unique research cases, an

research participants, such as the study design

advisory board could be created within a disci-

and methodology, now are routinely included

pline's professional association, and that board

in evaluations of research proposals. Because a

would be charged with analyzing and review-

poorly designed study can have serious ramifi-

ing an I R B decision when

cations and costs, one might argue that the

emerge. Another approach to improving the

technical elements of a proposed investigation

review process is to take steps to minimize

in fact should be included as a dimension o f

problems o f communication

ethical review (Rosenthal, 1 9 9 4 ) . Nonetheless,

members and investigators and to encourage

there remains widespread disagreement over

regular communication between IRBs (see

disagreements

between I R B

Ethical Issues in Social Psychology Hansen [ 2 0 0 1 ] and Tanke and Tanke [ 1 9 8 2 ] for further discussion). Among other recommendations for temper-

Research

CONCLUSION: ETHICAL CHALLENGES AND OPPORTUNITIES

ing review panel evaluations o f research, especially research posing minimal risk (e.g.,

It is understandable that many social psychol-

Ilgen & Bell, 2 0 0 1 ) , Diener ( 2 0 0 1 ) has pro-

ogists react with trepidation when confronted

posed that (a) exemptions should be granted

with the maze o f apparently

cumbersome

with greater frequency for research without

rules and regulations that have evolved over

true risk o f harm; (b) prototypes o f certain

the years to respond to research issues. As

types o f research protocols can be given

described by R o s n o w ( 1 9 9 7 ) , "even experi-

approval, with subsequent research

fitting

enced researchers often find themselves caught

these prototypes granted expedited review;

between the Scylla o f methodological and the-

(c) review boards should recognize their obli-

oretical requirements and the Charybdis o f

gation to foster potentially beneficial research,

ethical

in addition to protecting research participants;

(p. 3 4 5 ) . Nonetheless, these developments are

and (d) compensation assistance should be

typical o f ethical progress and are essential for

provided to researchers to offset the costs

providing a c o m m o n set o f values by which

dictates

and

moral

sensitivities"

accrued as a result o f lengthy and unreasonable

scientific discovery can proceed. Moreover,

delays and demands (e.g., help in completing

whenever ethical sensitivities have been raised

forms;

research

as a result o f an increased attention to moral

approved). (See Azar [ 2 0 0 2 ] for an overview o f

issues within society or because o f unfortunate

several ongoing projects oriented toward the

research abuses, many positive changes within

advice

for

getting

the

rewriting o f current I R B regulations, education

the scientific disciplines have followed, includ-

of researchers and review board members, and

ing innovative procedures that conform to

standardization o f the system.)

both scientific and ethical standards.

REFERENCES Adair, J . G., Dushenko, T. W., 8c Lindsay, R. C. L. (1985). Ethical regulations and their impact on research practice. American Psychologist, 40, 59-72. Allen, C. (1997). Spies like us: When sociologists deceive their subjects. Lingua Franca, 7, 30-39. American Association for Public Opinion Research. (2002). Code of professional ethics and practices. Retrieved February 15, 2 0 0 3 , from www.aapor.org American Psychological Association. (1982). Ethical principles in the conduct of research with human participants (Rev. ed.). Washington, DC: Author. American Psychological Association. (2002). Ethical principles of psychologists and code of conduct 2 0 0 2 . Retrieved February 17, 2 0 0 3 , from www.apa.org/ethics/ code2002.html American Psychological Association, Ad Hoc Committee on Ethical Standards in Psychological Research. (1973). Ethical principles in the conduct of research with human participants. Washington, DC: Author. Aronson, E., 8c Carlsmith, J . (1968). Experimentation in social psychology. In G. Lindzey 8c E. Aronson (Eds.), The handbook of social psychology (Vol. 2, pp. 1-79). Cambridge, MA: Addison-Wesley.







65

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Australian Psychological Society. (1986). Code of professional conduct. Parkville, Victoria, Australia: Author. Azar, B. (2002). Ethics at the cost of research? Monitor on Psychology, 33, 38-40. Baumrind, D. (1964). Some thoughts on ethics of research: After reading Milgram's "Behavioral study of obedience." American Psychologist, 19, 4 2 1 - 4 2 3 . Baumrind, D. (1985). Research using intentional deception: Ethical issues revisited. American Psychologist, 40, 165-174. Bearden, W. O., Madden, C. S., & Uscategui, K. (1998). The pool is drying up. Marketing Research, 10, 26-33. Behnke, S. H., & Kinscherff, R. (2002). Must a psychologist report past child abuse? Monitor on Psychology, 33, 56-57. Berenbeim, R. E. (1992). Corporate ethics practices. New York: The Conference Board. Berkun, M., Bialek, H. M., Kern, P. R., & Yagi, K. (1962). Experimental studies of psychological stress in man. Psychological Monographs: General and Applied, 76, 1-39. British Psychological Society. (1995). Code of conduct, ethical principles and guide­ lines. Leicester, UK: Author. Broder, A. (1998). Deception can be acceptable. American Psychologist, 53, 85-86. Campbell, D., Sanderson, R. E., & Laverty, S. G. (1964). Characteristics of a conditioned response in human subjects during extinction trials following a of Abnormal and Social single traumatic conditioning trial. Journal Psychology, 68, 627-639. Canadian Psychological Association. (1991). Canadian code of ethics for psycholo­ gists (Rev. ed.). Old Chelsea, Quebec: Author. Ceci, S. J . , Peters, D., & Plotkin, J . (1985). Human subjects review, personal values, Psychologist, 40, and the regulation of social science research. American 994-1002. Chonko, L. B. (1995). Ethical decision-making in marketing. Thousand Oaks, CA: Sage. Christensen, L. (1988). Deception in psychological research: When is its use justified? Personality and Social Psychology Bulletin, 14, 6 6 4 - 6 7 5 . Colegio Oficial de Psicologos. ( 1 9 9 3 ) . Codigo deontologico del psicologo [Deontological code of the psychologist]. Madrid: Author. Collins, F. L., J r . , Kuhn, I. F., J r . , & King, G. D. (1979). Variables affecting subjects' ethical ratings of proposed experiments. Psychological Reports, 44, 155-164. Colombo, J . (1995). Cost, utility, and judgments of institutional review boards. Psychological Science, 6, 318-319. Council for Marketing and Opinion Research. (1999). Respondent bill of rights. Retrieved February 15, 2 0 0 3 , from www.cmor.org Council of American Survey Research Organizations. (1995). Code of standards for survey research. Port Jefferson, N Y : Author. Department of Health and Human Services. (1981, January 2 6 ) . Final regulations amending basic HHS policy for the protection of human research subjects (45 CFR, Pt. 4 6 ) . Federal Register, 46(16), 8 3 6 6 - 8 3 9 1 . Department of Health, Education, and Welfare. (1975, March 13). Protection of human subjects: Technical amendments (45 CFR, Pt. 4 6 ) . Federal Register, 40, 11854-11858. Diamond, S. S., & Morton, D. R. (1978). Empirical landmarks in social psychology. Personality and Social Psychology Bulletin, 4, 2 1 7 - 2 2 1 . Diener, E. (2001). Over-concern with research ethics. Dialogue, 16, 2.

Ethical Issues in Social Psychology Diener, E., &c Crandall, R. (1978). Ethics in social and behavioral research. Chicago: The University of Chicago Press. Diener, E., Matthews, R., & Smith, R. (1972). Leakage of experimental information to potential future subjects by debriefed subjects. Journal of Experimental Research in Personality, 6, 2 6 4 - 2 6 7 . Epstein, Y . M., Suedfeld, P., & Silverstein, S. J . (1973). The experimental contract: Subjects' expectations of and reactions to some behaviors of experimenters. American Psychologist, 28, 2 1 2 - 2 2 1 . European Federation of Psychology Associations. (1998). Ethical principles for Scandinavian psychologists. Brussels: Author. Lausanne, Federation Suisse des Psychologues. (1991). Code deontologique. Switzerland: Author. Fischman, M . W. (2000). Informed consent. In B. D. Sales & S. Folkman (Eds.), Ethics in research with human participants (pp. 35-48). Washington, DC: American Psychological Association. Frankena, W. K. (1973). Ethics (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall. French Psychological Society. (1976). Code de deontologie. Paris: Author. Geller, D. M . (1982). Alternatives to deception: Why, what, and how? In J . E. Sieber (Ed.), The ethics of social research: Surveys and experiments (pp. 39-55). New York: Springer-Verlag. code of German Association of Professional Psychologists. (1986). Professional ethics for psychologists. Bonn: Author. Gray, B., & Cooke, R. A. (1980). The impact of institutional review boards on research. Hastings Center Report, 10, 3 6 - 4 1 . Greenberg, M . (1967). Role playing: An alternative to deception? Journal of Personality and Social Psychology, 7, 152-157. Grundner, T. M. (1986). Informed consent: A tutorial. Owings Mills, M D : National Health Publishing. Hansen, C. (2001). Regulatory changes affecting IRBs and researchers. APS Observer, 14(7), 13-14, 2 5 . Haywood, H. C. (1976). The ethics of doing research . . . and of not doing it. American Journal of Mental Deficiency, 81, 311-317. Holmes, D. S. (1976). Debriefing after psychological experiments. American Psychologist, 31, 858-867. Horowitz, I. L. (1967). The rise and fall of Project Camelot. Cambridge: M I T Press. Humphrey, L. (1970). Tearoom trade. Chicago, IL: Aldine. Ilgen, D. R., &c Bell, B. S. (2001). Informed consent and dual purpose research. American Psychologist, 56, 1177. International Chamber of Commerce/European Society for Opinion and Marketing Research. (2001). ICC/ESOMAR international code of marketing and social research practice. Retrieved February 14, 2003 from www.esomar.nl Jones, S. E. (2001). Ethics code draft published for comment. Monitor on Psychology, 32(2), 76. Kamins, M . A., Folkes, V. S., & Perner, L. (1997). Consumer responses to rumors: Good news, bad news. Journal of Consumer Psychology, 6, 165-187. Kelman, H. C. (1967). Human use of human subjects: The problem of deception in social psychological experiments. Psychological Bulletin, 67, 1-11. Kelman, H. C. (1996). Foreword. In A. J . Kimmel, Ethical issues in behavioral research: A survey (pp. xiii-xv). Cambridge, MA: Blackwell. Kimmel, A. J . (1988). Ethics and values in applied social research. Newbury Park, CA: Sage.

Research

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Kimmel, A. J . (1991). Predictable biases in the ethical decision making of American psychologists. American Psychologist, 46, 7 8 6 - 7 8 8 . Kimmel, A. J . (1996). Ethical issues in behavioral research: A survey. Cambridge, MA: Blackwell. Kimmel, A. J . (2001). Ethical trends in marketing and psychological research. Ethics & Behavior, 11, 131-149. Kimmel, A. J . , & Audrain, A.-F. (2002, August). Rumor control strategies within French consumer goods firms. Paper presented at the 110th annual convention of the American Psychological Association, Chicago. Kimmel, A. J . , 8c Smith, N. C. (2001). Deception in marketing research: Ethical, methodological, and disciplinary implications. Psychology & Marketing, 18, 663-689. Korn, J . H. (1988). Students' roles, rights, and responsibilities as research participants. Teaching of Psychology, IS, 74-78. Kuschel, R. (1998). The necessity for code of ethics in research. Psychiatry Today: Journal of the Yugoslav Psychiatric Association, 30, 2 4 7 - 2 7 4 . Landis, M . H., 8c Burtt, H. E. (1924). A study of conversations. Journal of Comparative Psychology, 4, 81-89. Latane, B., 8c Darley, J . M . (1970). The unresponsive bystander: Why doesn't he help? New York: Appleton-Century-Crofts. Lindsay, R.C.L., 8c Holden, R. R. (1987). The introductory psychology subject pool in Canada. Canadian Psychology, 28, 45-52. McCord, D. M . (1991). Ethics-sensitive management of the university human subject pool. American Psychologist, 46, 1 5 1 . McNamara, J . R., 8c Woods, K. M . (1977). Ethical considerations in psychological research: A comparative review. Behavior Therapy, 8, 7 0 3 - 7 0 8 . Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 3 7 1 - 3 7 8 . Miller, A. G., Collins, B. E., 8c Brief, D. E. (1995). Perspectives on obedience to authority: The legacy of the Milgram experiments. Journal of Social Issues, 51, 1-19. Mills, J . (1976). A procedure for explaining experiments involving deception. Personality and Social Psychology Bulletin, 2, 3-13. Mirvis, P. H., 8c Seashore, S. E. (1982). Creating ethical relationships in organizational research. In J . E. Sieber (Ed.), The ethics of social research: Surveys and experiments (pp. 79-104). New York: Springer-Verlag. Moore, H. T. (1922). Further data concerning sex differences. Journal of Abnormal and Social Psychology, 17, 2 1 0 - 2 1 4 . Mordock, J . B. (1995). Institutional review boards in applied settings: Their role in judgments of quality and consumer protection. Psychological Science, 6, 320-321. Mueller, J . H., 8c Furedy, J . J . (2001). Reviewing for risk: What's the evidence that it works? APS Observer, 14(7), 1, 2 6 - 2 8 . National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979). The Belmont report: Ethical principles and guide­ of human subjects of research. Washington, DC: lines for the protection Government Printing Office. National Research Act, Public Law 93-348, Title II—Protection of Human Subjects of Biomedical and Behavioral Research (Part A). (1974). Netherlands Institute of Psychologists. (1988). Professional code for psychologists. Amsterdam: Author. Nicks, S. D., Korn, J . H., 8c Mainieri, T. (1997). The rise and fall of deception in social psychology and personality research, 1921 to 1 9 9 4 . Ethics & Behavior, 7, 69-77.

Ethical Issues in Social Psychology Nosek, B. A., Banaji, M . R., 8c Greenwald, A. G. (2002). E-research: Ethics, security, design, and control in psychological research on the Internet. Journal of Social Issues, 58, 161-176. Nuremberg Code. (1949). In Trials of war criminals before the Nuremberg military tribunals under Control Council Law No. 10,2 (pp. 181-182). Washington, DC: Government Printing Office. Office for Protection From Research Risks, Protection of Human Subjects. ( 1 9 9 1 , June 18). Protection of human subjects: Title 4 5 , Code of Federal Regulations, Part 4 6 (GPO 1 9 9 2 O-307-551). OPRR Reports, pp. 4-17. Orne, M . T. (1959). The nature of hypnosis: Artifact and essence. Journal of Abnormal and Social Psychology, 58, 2 7 7 - 2 9 9 . Orne, M . T., 8c Holland, C. H. (1968). On the ecological validity of laboratory deceptions. International Journal of Psychiatry, 6, 2 8 2 - 2 9 3 . Ortmann, A., & Hertwig, R. ( 1 9 9 7 ) . Is deception acceptable? American Psychologist, 52, 746-747. Patten, S. C. (1977). Milgram's shocking experiments. Philosophy, 52, 4 2 5 - 4 4 0 . Prentice, E. D., 8c Antonson, D. L. (1987). A protocol review guide to reduce IRB inconsistency. IRB: A Review of Human Subjects Research, 9, 9 - 1 1 . Psychological Association of Slovenia. (1982). Code of ethics for psychologists. Ljubljana, Slovenia: Author. Raupp, C. D., 8c Cohen, D. C. (1992). "A thousand points of light" illuminate the psychology curriculum: Volunteering as a learning experience. Teaching of Psychology, 19, 25-30. Resnick, J . H., 8c Schwartz, T. (1973). Ethical standards as an independent variable in psychological research. American Psychologist, 28, 134-139. Reynolds, P. D. (1979). Ethical dilemmas and social science research. San Francisco: Jossey-Bass. Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting psychological research. Psychological Science, 5, 127-134. Rosenthal, R., 8c Rosnow, R. L. (1984). Applying Hamlet's question to the ethical conduct of research: A conceptual addendum. American Psychologist, 39, 561-563. Rosnow, R. L. (1997). Hedgehogs, foxes, and the evolving social contract in psychological science: Ethical challenges and methodological opportunities. Psychological Methods, 2, 345-356. Rosnow, R. L., 8c Aiken, L. S. (1973). Mediation of artifacts in behavioral research. Journal of Experimental Social Psychology, 9, 1 8 9 - 2 0 1 . Rosnow, R. L., Rotheram-Borus, M . J . , Ceci, S. J . , Blanck, P. D., 8c Koocher, G. P. (1993). The institutional review board as a mirror of scientific and ethical standards. American Psychologist, 48, 821-826. Ross, L., Lepper, M . R., 8c Hubbard, M . (1975). Perseverance in self-perception and social perception: Biased attributional processes in the debriefing paradigm. Journal of Personality and Social Psychology, 32, 880-892. Schlenker, B. R., 8c Forsyth, D. R. (1977). On the ethics of psychological research. Journal of Experimental Social Psychology, 13, 3 6 9 - 3 9 6 . Schuler, H. (1982). Ethical problems in psychological research. New York: Academic Press. Sharf, B. F. (1999). Beyond Netiquette: The ethics of doing naturalistic discourse Critical research on the Internet. In S. Jones (Ed.), Doing Internet research: issues and methods for examining the Net (pp. 2 4 3 - 2 5 6 ) . Thousand Oaks, CA: Sage. Sharpe, D., Adair, J . G., 8c Roese, N. J . (1992). Twenty years of deception research: A decline in subjects' trust? Personality and Social Psychology Bulletin, 18, 585-590.

Research

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Shea, C. (2000). Don't talk to the humans: The crackdown on social science research. Lingua Franca, 10, 2 6 - 3 4 . Smith, C. P. (1981). How (un)acceptable is research involving deception? IRB: A Review of Human Subjects Research, 3, 1-4. Smith, N. C , & Klein, J . G. (1999). Ethics in marketing research and the use of remedial measures to mitigate the deception of respondents. Unpublished manuscript, McDonough School of Business, Georgetown University. Smith, N. C , Klein, J . G., & Kimmel, A. J . (2002). The ethics of deception in consumer research. Unpublished manuscript, London Business School. Strieker, L. J . , Messick, S., & Jackson, D. N. (1967). Suspicion of deception: Implications for conformity research. Journal of Personality and Social Psychology, 5, 379-389. Sullivan, D. S., & Deiker, T. A. (1973). Subject-experimenter perceptions of ethical issues in human research. American Psychologist, 28, 5 8 7 - 5 9 1 . Tanke, E. D., & Tanke, T. J . (1982). Regulation and education: The role of the institutional review board in social science research. In J . E. Sieber (Ed.), The ethics of social research: Fieldwork, regulation, and publication (pp. 131-149). New York: Springer-Verlag. Taylor, K. M., & Shepperd, J . A. (1996). Probing suspicion among participants in deception research. American Psychologist, 51, 886-887. Toy, D., Olson, J . , Sc Wright, L. (1989). Effects of debriefing in marketing research involving "mild" deceptions. Psychology & Marketing, 6, 69-85. Toy, D., Wright, L., & Olson, J . (2001). A conceptual framework for analyzing deception and debriefing effects in marketing research. Psychology & Marketing, 18, 663-689. Tybout, A. M., Calder, B. J . , & Sternthal, B. (1981). Using information processing theory to design marketing strategies. Journal of Marketing Research, 18, 73-79. Vinacke, W. E. (1954). Deceiving experimental subjects. American Psychologist, 9, 1 5 5 . Vitelli, R. (1988). The crisis issue assessed: An empirical analysis. Basic and Applied Social Psychology, 9, 301-309. Webb, E. J . , Campbell, D. T., Schwartz, R. D., Sechrest, L., & Grove, J . B. (1981). Nonreactive measures in the social sciences (2nd ed.). Boston: Houghton Mifflin.

CHAPTER

4

Developing a Program of Research

SUSAN T . FISKE Princeton

University

A

prospective graduate student yesterday

appropriate, much the way one calibrates a

asked me h o w I ended up doing the

certain amount o f snow as appropriate for a

work I do. This question

appears

proper winter. I don't claim that the neigh-

with alarming frequency—a sign, one fears, o f

borhood's integration was flawless, but it was

becoming a Fixture in the Field. Fixtures are

deep and abiding, and the adults I knew

fixed, and science is moving, so in the spirit o f

seemed proud o f it. M o v i n g to Boston for col-

motion, let's consider the process o f develop-

lege, I was struck by an absence that t o o k me

ing, growing, maintaining, and refreshing a

a while to place. Although there was the right

program o f research. M y own entry into the

amount o f fluffy white stuff, the people were

field was motivated by both nature and nur-

far t o o white. T h e lack o f ethnic variety in

ture (my father a psychologist, my mother a

the Boston I encountered—the result o f heavy

community volunteer), both o f which suited

de facto segregation—seemed odd to me.

me to puzzle over experiences with people in

Probably primed by my mother's interest in

social contexts. T h e influences o f the social

communities, I couldn't

context, o f course, are what social psychology

people would want to live that way. Probably

is all about, so it makes sense that a social psy-

primed by my father's orientation t o research,

figure

out

why

chologist would believe that the nature o f a

I realized there must be empirical answers.

person's social contexts shapes that person's

Puzzle number one gave rise to my research

research program. Whatever your own social

on stereotyping, about which more later.

contexts and resulting research puzzles, I will

Some time later, I observed up close some

suggest in this chapter that the processes o f

organizations with few women in high places,

developing a research career are knowable,

though plenty in low places. Some women who

manageable, and even fun.

were extremely competent, by all reasonable

Consider four cases o f social contexts that

standards, simply were not promoted. None

produced creative puzzles, for at least one

were obvious cases o f gender discrimination

budding social scientist. As a kid in the 1 9 5 0 s

because some women—a very few—seemed to

on the South Side o f Chicago, I calibrated a

be getting ahead. But the proportions and the

certain level o f ethnic diversity as natural and

standards were off. O n e case in point was Ann

72

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Hopkins, a would-be partner in a top accounting

Finally, puzzle number four resulted from a

firm, who brought in millions o f business

continuing sense of wonder about people as

dollars but was faulted for lacking interper-

unbelievably complex. H o w in the world do

sonal skills. Because the advice from her sup-

we manage to make sense o f each other? As an

porters included the exhortation to be more

adolescent, it's a normal developmental task to

feminine, gender clearly played some role,

think about your peers' opinions. Like my

according to the Supreme Court, where her

peers, I worried especially about how people

case ended up. This setting (in which I testified

made sense of me and what they thought of

as the plaintiffs expert witness) was not an iso-

me. Later, I became more concerned about

lated instance o f the phenomenon. In various

how I and others made sense o f other people,

organizations, women who behaved like door-

and whether we were being fair. In particular,

mats apparently had a better chance o f being

I worried about whether people were putting

promoted than women who did not, so some

too much emphasis on often

combination of gender and personality was at

aspects of each other: appearance, ethnicity,

play. Clearly, personality matters a lot to men's

gender, and the like. Puzzle number four led to

success as well, but it was mattering more for

an abiding interest in social cognition.

some unarguably competent women. Probably primed to notice because of my

suffragist

inappropriate

This chapter takes us from scattered realworld phenomena o f personal and social inter-

grandmother and great-grandmother, I could-

est to a long-term program o f research. Topics

n't figure out why organizations would shoot

will include how to generate the following: a

themselves in the foot by depriving themselves

personal perspective, compelling hypotheses,

of such considerable talent. Puzzle number two

convincing research, readable write-ups, appro-

gave rise to my research on ambivalent sexism.

priate outlets, programmatic approaches, and a

Puzzle number three came from a more

willingness to be wrong. T h e chapter will also

directly personal experience. At the time, I

address critically important sideshows to per-

couldn't

forming

figure

out why,

as an

assistant

research: collaboration,

teaching,

department

funding, and service. All o f this intends to sug-

head's every gesture and expression. It was a

gest that the processes o f developing a long-

hierarchical place, and the department head

term research program are knowable, and one

held a lot o f power over my fate. Outcomes

can manage one's progress toward a research

professor, I was glued to my

and

program by becoming and remaining aware of

research funds all depended heavily on this

choices one makes at each step—and still solve

person's opinion o f me. I wasn't the only one

some personal puzzles along the way.

such as salary, promotion,

teaching,

w h o analyzed his every twitch. In a heavily vertical organization, attention seemed to be directed upward. Spurred by this observation,

START B Y KNOWING THAT

reflecting back on graduate school, I recalled

M A N Y PERSPECTIVES ARE

a similar phenomenon among all the graduate

N O T Y E T REPRESENTED

students, whereby we all overanalyzed our advisers' every reaction. As a professor rising

A social psychologist straddles the domain

in the field, I began to notice students w h o

between social sciences, which have clear

overreacted to my slightest irrelevant grimace

political and social implications, and most o f

or tired sigh. Later, in a more democratic,

psychology, which deals with parts o f the per-

horizontal department, I noticed much less o f

son. Social psychology, because it deals with

that vigilance. This kind o f puzzle motivated

the whole person in a social context, has

my research on outcome dependency.

implications for our own perspectives not

Developing

a Program

of

Research

only as scientists and intellectuals, but also as

might argue for a hierarchy here, but to the

insightful people with group identities, poli-

extent that one can combine these levels of

tics, and even moral beliefs. Empirically exam-

insight, my experience and perception are that

ining a variety o f perspectives through social

one does better acknowledging

psychological research deepens and tests our

sources. In my view, one does not have to

all these

own perspectives. Because each o f us brings a

choose among theoretical sophistication, social

unique combination o f perspectives, each o f

problems, and everyday appeal. What's impor-

us has a particular starting point for a research

tant is keeping in mind ideas that are interesting

program. Social psychology needs a variety o f

from a big-picture perspective. (For another

perspectives to be a healthy science precisely

perspective on big-picture sources of ideas,

because it deals with people in social context.

read McGuire's [ 1 9 7 3 ] " T h e Y i n and Yang of

Competing perspectives, with solid empirical

Progress in Social Psychology: Seven K o a n . " )

evidence, will better approximate the truth. As a college senior, I recall auditing a course on gender differences and thinking that the dif-

Intellectual Sources

ferences all seemed to put women at a disad-

T h e most c o m m o n intellectual fount o f

vantage. For example, it certainly sounded

ideas is a theoretical discontent. Existing per-

better to be field independent than field depen-

spectives simply prove inadequate. Perhaps

dent (guess which gender and which race are

they

field dependent). Then I wondered: W h a t if

Perhaps they possess internal contradictions.

one called the variable "field sensitivity"

Perhaps they omit important aspects o f the

instead? Likewise, internal control or primary

phenomena.

control sounded better than succumbing to

thing, so they are untestable.

do

not

explain the

Perhaps

existing

data.

they explain every-

external control or secondary control (again,

Just as one may move forward by rejecting

with gender, ethnicity, and cultural differences

current theory, one may progress by reviving

going in one direction). But what if one called

old theories viewed through modern lenses.

it "social harmony control" instead? Clearly, I

For example, the Z a j o n c ( 1 9 9 4 ) theory o f

realized, people's values inform the research

emotions as regulating cerebral blood flow

questions they bother to ask, the methods they

came in part from an older theory proposing

use, the interpretations they make, and cer-

similar ideas, but without modern techniques

tainly what they name their variables. It

to test it. Historical revival also underlies some

seemed important, then, to have researchers

of my own w o r k on the continuum model of

with a variety o f perspectives compete in the

impressions (S. T . Fiske, Lin, 8c Neuberg,

contest of ideas. I also realized that one cannot

1 9 9 9 ; S. T . Fiske & Neuberg, 1 9 9 0 ) , which

have a credible voice without the correspond-

partly came from contrasting Solomon Asch's

ing methodological expertise. So one has to

( 1 9 4 6 ) two processes o f impression forma-

take one's puzzles

into

more

systematic

conceptualizations and operationalizations.

tion. M o r e generally, William James ( 1 8 9 0 ) and Fritz Heider ( 1 9 5 8 ) are rich sources o f ideas to be newly framed and tested afresh. A general meta-theoretical perspective can

COMPELLING,

provide ideas as well. For example, a pragmatic

COHERENT HYPOTHESES:

approach (James's "thinking is for doing"; S. T .

WHAT'S T H E BIG PICTURE?

Fiske, 1 9 9 3 b ) can provide hypotheses about

The sources o f one's research ideas can be intel-

testing people's reactions against the practical

lectual, personal, group, or worldview. Some

functions served. Similarly, an evolutionary

why people think, feel, or do what they do, by

73

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH standpoint, a cultural contrast, a consistency

for systematic analyses by theoreticians o f

theory perspective, or a

interpersonal perception processes.

self-enhancement

assumption each can shape hypotheses. Synthesis across areas also provides a rich source o f ideas. O n e can work across similar

Group Sources

psychology—for

In parallel, one's perspective on the exist-

example, competing models o f impression for-

ing viewpoints m a y be informed by one's

mation or attitude change, which may turn out

social group identity. M y budding insights

to be two equally valid modes that operate

into the ways that values inform the research

under different circumstances (Chaiken and

process were fueled no doubt by the fact that

literatures

within

social

Trope, 1 9 9 9 , collected 3 5 theories that com-

most o f the researchers whose theories I stud-

bine two modes in this way). O r one can read

ied were men. W h e n I graduated from col-

or talk to other social scientists—such as polit-

lege, the editorial board o f the Journal

of

included 1

ical scientists, sociologists, or economists—

Personality and Social Psychology

w h o operate at a more macro level but m a y

w o m a n and 2 0 men, the reviewers for each

make assumptions about psychological pro-

issue averaged about 1 w o m a n and 2 0 men,

cesses that seem psychologically undeveloped

the senior authors averaged about 1 w o m a n

or implausible but salvageable for new theory

per issue, and the editors included

nonetheless. Some o f the most innovative work

Social psychology as a field began t o include

none.

occurs at the boundaries between disciplines

women more rapidly than many areas o f psy-

and subdisciplines.

chology and more than many areas o f science

O n e also may develop theory by critiquing the theory o f specific other viewpoints. M o r e often than not, though, pure critique does not go far. Building hypotheses merely as a reaction to someone else's hypotheses leaves limited ground t o explore. O n e ends up by picking at the toenails o f giants. Better to

(see Berscheid, 1 9 9 2 , for a compelling discussion). M y point here is simply this well-worn one: If certain socially significant groups do not participate, then the field loses the variety of perspectives needed for a healthy set o f dialogues. An underrepresented point o f view can counter unconscious biases in prior work

stand on their shoulders.

or in the dominant approach.

Personal Sources

Worldview Sources People's explicit value systems—religious,

Intuition, hunches, and personal experience also can inform

one's

hypotheses.

ethical, political—can create a conviction that

People often will develop a sense that their

a

own experiences, or those o f people they

Whatever value-driven basis one may have

fundamental

truth

is

being

missed.

widespread

(the good Samaritan study o f Darley and

phenomenon that has been overlooked. Case

Batson, 1 9 7 3 , comes to mind), the research

studies can c o m e from real people or even

itself must be logically reasoned and method-

from literature, songs, and movies. Psychol-

ologically rigorous in order to survive the

ogical insight is many people's hobby, though

scientific review process, which looks with

k n o w well, represent a more

only well-trained methodologists k n o w how

justifiable suspicion on research with a value-

to formulate a logical series of testable pro-

based agenda. Nonetheless, if one can con-

positions from it. In Heider's ( 1 9 5 8 ) view,

ceptually articulate and operationally define

commonsense psychology was a foundation

one's predictions, a value-inspired

agenda

Developing remains a valid perspective for

informing

a Program

of

Research

See whether your idea has general appeal. T h e "cocktail party test" is whether you can

empirical tests.

explain your idea to a nonspecialist in a way that is clear, brief, and interesting. If the lis-

General Principles, Regardless of Source

tener immediately develops an urge for hors d'oeuvres, you may not have focused enough

W h e t h e r the source is intellectual, per-

on the m o s t compelling aspect o f your

sonal, group, worldview, or—most likely—a

hypotheses. Y o u r idea should

combination, several principles contribute to

enough that a neophyte can remember the

be simple

uncovering an idea. First, one must mind the

main point the morning after. One journal

gap, in any o f the preceding sources. While

editor told me that he would read manuscripts

reading, hearing, or teaching the research

in the evening and see whether he could

literature, listen to the still, small voice o f dis-

remember the main idea during his next-

comfort, disturbance, or disruption. It takes

morning jog. If so, the author had made a last-

a subtle inner ear to hear that voice, but cul-

ing impression. As J a c o b Cohen ( 1 9 9 0 ) put it,

tivate it. W h e n you do hear the voice, ponder

in promoting simplicity, "less is more."

what's wrong or missing, m a k e a note, and

W a l l o w in a reliable effect, to generate compelling hypotheses. Especially if you

then w o r k with it later. Some people keep a folder o f ideas. I f you

yourself have uncovered a reliable, original

do, you'll be surprised at h o w often the same

effect, pursue its moderators. See what other

or similar ideas occur t o you. This is your

theoretically interesting independent variables

perspective. F o l l o w it. Patterns in w h a t tends

shape it. Boundary conditions

t o annoy you will turn into a program o f

define any phenomenon. If you can make an

ultimately

effect come and go at will, then you begin

research. In sorting through your ideas, pick perspec-

truly to understand it. Besides moderators,

tives that are underrepresented. A fresh idea

pursue mediators. W h a t underlying psycho-

creates an excited following. Few ideas, how-

logical mechanisms explain your effect? W h a t

ever, start a procession o f follow-up studies. If

process comes between your main indepen-

you can't lead the parade, at least anticipate

dent and dependent variables? I f your effect

the parade route and get there early to point

matters, then its moderators and mediators

the way. W a t c h smart, interesting people and

will matter t o o . Consider stereotype threat

see what they are beginning to consider, before

(Steele, Spencer, &c Aronson, 2 0 0 2 ) . T h e first

most people have noticed. Think about the

generation o f research demonstrated

that

implications o f their work and the probable

black people and w o m e n could underperform

directions in which it will move the field. D o n ' t

in contexts that made salient the relevant neg-

imitate those you observe, but apply their gen-

ative stereotypes about their groups, even

eral direction to your own interests.

when their performance would otherwise be

Beware, however, o f runaway

bandwag-

equivalent. T h e second wave o f research

ons. O n c e a trend has swelled beyond a cer-

addressed generalizability (defining modera-

tain size, you w o n ' t have that much to add to

tors and boundaries), showing that a variety

it. Avoid crowds: Y o u w o n ' t stand out, and

of groups are vulnerable to stereotype threat

your pocket is likely t o get picked. I f you

in domains where they have negative reputa-

work

too

tions: men on emotional sensitivity, lower

crowded, it is hard t o say anything new, and

classes on academic performance, whites in

it is all t o o easy t o be scooped.

sports, whites in academics relative to Asians,

in an

area

after

it b e c o m e s

75

76

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH and more. T h e second wave o f research also

If you indicate that you value colleagues'

tackled potential mediators such as anxiety,

intellect, they are more likely to value yours,

distraction, vigilance, and

assuming the dialogue is equal. If you approach

effort. Second-

generation research can create those all-impor-

a senior colleague with a draft of a research

tant publications at certain career stages, but I

design or a grant proposal, they cannot plausi-

wouldn't make a career of tweaking someone

bly claim later that it was all their idea. But it's

else's finding. When you tweak your own orig-

fine if they later say they had an influence on

inal finding, it's called programmatic research

some aspect o f it. Consider their suggestions

on a series of compelling hypotheses.

without being defensive, and use the useful

Whatever you w o r k on, follow your pas-

ones. Find a comfortable balance on the con-

sion. Enjoy it. Study what intrigues you. W h y

tinuum from being overly isolated to being col-

else put up will all the grief associated with

legial to being overly dependent. Chances are,

research?

you'll err on the side o f being too independent, so don't be embarrassed about asking whether someone might have time to give you feedback.

CONVINCING RESEARCH: READ THIS B O O K

Better too early than too late. Having to talk about your research also puts it into perspective. Y o u have to explain

M a n y problems in research design are most

first why you are conducting this particular

easily solved by conversation and feedback—

research, and that will keep your eye on the

early and often. All o f us think we should go it

big picture. I have noticed a pattern among

alone without help. Graduate students and

my students, especially undergraduates, over

junior faculty often think help-seeking will

the years. W h e n w e first discuss

(a) bother the busy important senior faculty or

research together, or when they describe their

(b) reflect badly on them when they are evalu-

research to someone else, they almost always

ated. Au contraire. W h e n you are evaluated,

start with the method and forget to mention

doing

someone (preferably several someones) will

the hypothesis. This is like deciding whether

have to be your advocate(s), or else the cynical

to drive, bike, walk, or swim—before you

critics (of which academia has many) will win

k n o w where you want to go. There may be

and you will lose. T o be an advocate, the per-

times when the ride is the point, but typically

son has to k n o w your research intimately. If

not in science. Different destinations require

the person has talked with you about your for-

different

mative decisions, the person will be a far more

hypotheses suggest different methods. If you

modes

of

transport.

Different

credible advocate than if he or she reads your

want to go for a bike ride, with n o destination

work one night before the meeting. This does

in mind, that's fine, but don't conduct an

not require you to collaborate and coauthor

experiment just because you thought o f a

with more senior people—a little o f this is O K ,

clever procedure. Probably, if you have a

but too much leads to obvious problems in

method in mind, a hypothesis may be lurking

attributing credit. All you need to do is seek

in a mental corner somewhere. Some search-

advice sometimes. People love to give advice;

ing

they love to feel invested in your work and

hypotheses, which then need to be specified

probably

will

uncover

the

implicit

your career. Benjamin Franklin once said

and developed. But do wait to choose a

something to the effect o f "ask a favor, gain a

method until after the concepts are clear

friend; do a favor, lose a friend." Even older

(novices tend t o seize on a method too early).

faculty want to feel appreciated and valued.

Moving from concept to operation, from

All social psychologists know about reciprocity:

hypothesis to method, disciplines the mind.

Developing

a Program

of Research

\

For example, to create a working definition o f

t o choose one's battles. Sometimes, inventing

aggression for an experiment, the researcher

a new procedure is worthwhile, if that is the

must decide what kinds o f aggression count:

focus of one's contribution, but sometimes

Indirect as well as direct? Passive as well as

effort and energy must go elsewhere. M a k e a

active? Nonverbal as well as verbal? H o w does

conscious choice.

aggression differ from assertiveness? After hav-

In general, many impactful social psychol-

ing decided on a particular kind (e.g., physical

ogy experiments display drama in the depen-

violence), what levels are appropriate for the

dent variable and subtlety in the independent

hypothesis, the participants, and the setting?

variable. Small changes in the situation (such

W h a t is ethical and feasible? Will it be possible

as an experimenter merely saying "the exper-

to study people blasting another person with

iment requires that you continue") cause dra-

noise? Shocking another

matic changes in the participants' behavior

person?

Hitting

another person? Any area poses challenges o f

(e.g.,

operationalization: Being forced to specify, for

Milgram, 1 9 6 5 ) . This advice, credited

instance, what specific activities are interesting

Stanley Schachter (L. Ross 8c Nisbett, 1 9 9 1 ) ,

shocking

someone

else t o

death; to

or boring, what is discrimination versus preju-

fits other famous studies' seemingly trivial

dice, what is the affective versus cognitive

independent variables ( $ 1 versus $ 2 0 pay-

aspect o f an attitude, all helps one to think

ment, many or few bystanders, having retirees

more clearly about the concepts involved.

water their own houseplants or not) and dra-

Besides the conceptual discipline involved

matic dependent variables (liking a patently

in operationalization, practical discipline

boring task, rescuing an accident victim, mor-

accompanies the working definitions. M o s t

tality rates).

of the concerns are obvious: T h e procedure

Whatever your variables, multiple methods

must be feasible and plausible, in all the ways

matter. Whatever method you choose, it will

you will k n o w for your specific

setting.

have drawbacks. Only by converging opera-

Probably the most c o m m o n but not obvious

tions can researchers k n o w whether the effect

issues involve trying to do t o o much at once.

is true and not simply an artifact. T h e same

M o r e focused questions invite more careful

effect across methods is more compelling than

operationalizations, so they are more likely

the same method applied to many different

to yield results. O n e can't do everything well.

phenomena. For example, the bogus pipeline

Moreover, the first time you run a study, it

(E. E . Jones &c Sigall, 1 9 7 1 ) is one method for

may not work, so you will have t o fine-tune

getting

at people's true racial

attitudes,

the method, which is more difficult with too

beyond the socially desirable response. If other

many

play.

methods (subliminal priming, response times,

variables

simultaneously

in

Simple and elegant is more effective than

nonverbal indicators) also show that people

complex and baroque. You'll just have to give

have racial prejudices that they fail to admit,

up testing some o f the side issues right away.

then the converging result, across methods, is

Because of all the uncertainties in blazing

compelling. In personality psychology, this

new trails, it can be helpful to build on estab-

idea—first expressed by Campbell and D . W .

lished paradigms that have worked in the past

Fiske ( 1 9 5 9 ) — h o l d s that a researcher more

for you, your collaborators, or others in the

reliably assesses personality traits by using

field. The method you know has certain estab-

multiple measures across multiple traits. In the

lished strengths and weaknesses. Inventing a

multitrait-multimethod matrix, one looks for

new method will almost certainly (a) take

traits that emerge reliably across methods, and

more time, (b) invite more criticism, and

one can detect methods that have their own

(c) not work out exactly as planned. O n e has

effects, regardless of particular traits. For

78

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH example, one might find that questionnaire

that people read like they take vitamins. Some

methods

element

people will do it dutifully, but they won't ask

(acquiescence bias), whereas response time

for seconds and they won't successfully rec-

all c o n t a i n a c o m m o n

measures all contain another (age bias). F o r

ommend it to others. Presenting your work

both

effectively is more than mere showiness; it

personality

and

social

psychology,

methodological pluralism allows researchers

allows your ideas to communicate clearly

to triangulate on the same phenomenon from

what they can contribute.

the perspective o f different

methods

with

different strengths and weaknesses.

Some great advice about writing psychology articles comes from Bern ( 2 0 0 3 )

and

Having collected the data, attack the analy-

Sternberg ( 1 9 9 3 ) . O n e particular highlight

ses with enthusiasm. Let your data do the talk-

includes the ideal hourglass shape o f an

ing. At the first stage, the researcher becomes a

article, starting with the broadest context;

detective, in Abelson's ( 1 9 9 5 ) terms, analyzing

becoming progressively more specific through

the data in every conceivably useful way. N o

the introduction; leading into the most con-

analysis is forbidden. Try anything, and see

crete, narrow specifics o f method and results;

squeezed,

then again broadening outward to the end o f

pulled, opened, shaken, and inverted the data,

the discussion, which takes the reader back to

you will see that some patterns keep appearing

the widest context.

what

happens.

Having

poked,

and some seem elusive. T h e data are trying to

Considering the article's overall flow, one

tell you what the robust results really are. Trust

wants to create tension and suspense: Will the

the ones that don't disappear when the analytic

hypotheses hold up? Will the

assumptions change slightly.

interpretation win out? Where will it all end?

alternative

Having played the detective, the researcher

Suspense can be arranged by making a plausi-

moves to the next role, as a lawyer advocat-

ble case for the competing alternative to your

ing a particular interpretation o f the evidence.

favored hypothesis. Otherwise, with hindsight

T h e lawyer must play according to certain

bias (Fischhoff, 1 9 7 5 ) , your hypothesis and

procedural rules, agreed-upon methods, sta-

results will seem all too obvious to the reader,

tistical techniques, and ethical obligations.

w h o will fail to be impressed.

Nevertheless, having a certain perspective to

The single most important principle o f

argue, one tries to make the best scientific

writing anything is to make an argument for

case, within the rules. T o keep you honest, the

something. T h e argument in this chapter was

judge and jury are editors and reviewers, as

stated at the outset: T h e processes of develop-

well as other readers.

ing a long-term research program are knowable, and one can manage one's progress toward a research program by becoming and

READABLE WRITE-UPS

remaining aware o f choices one makes at each

READERS WILL READ

step—and enjoy it. In any kind o f writing,

Readers are busy, distracted people: W h y

make, but play by the rules. People forget this

always make an argument and have a point to should they bother with your article? T o para-

basic premise with surprising predictability.

phrase Dahl ( 1 9 6 1 ) , science is a sideshow in

W h a t do you want to say? W h a t is the take-

the great circus o f life. People read your arti-

home message? W h e n my students (both grad-

cles because they thereby acquire nuggets o f

uate and undergraduate) read research articles

knowledge, insight, entertainment, ideas, and

for class, I always insist that they tell me what

clues. T o assume that people will read your

the author(s) were trying to do. It is surpris-

work because it is good for them is to assume

ingly hard to find the hypothesis in many

Developing

a Program

of Research

\

research articles and harder still to find the

your work after a break. Both lend an outside

argument in a review article. Having collected

perspective t o the too-familiar prose. Better

all that data and read all those articles, what

still, find an honest friend and heed the per-

did you learn and what do you want to tell us?

son's advice. However annoying, the reader is

A well-written article o f any sort has a thesis.

always right (or at least diagnostic). T h a t is, if

It does not read like a string o f note cards

one reader has a problem, others will too. It

joined as a string o f paragraphs. Instead, use

is better to hear the bad news from a friend,

specific aspects o f specific studies to support

w h o may save you a round o f rejection and

each point.

revision.

T o make a strong empirical article, which is a data-based argument, go with the strongest studies. M o s t researchers are

tempted—

O U T L E T S : VISIBLE A N D INVISIBLE

having gone to enormous trouble to run each study, having tended it through conception,

Assess the market value o f your research. It

realization, and analysis—to include every last

almost all hinges on the quality o f your data,

one, with every last measure that might be

n o matter h o w elegant your theory. Y o u may

relevant. Unfortunately, several weak studies

have a hypothesis that deserves to be true,

do not equal one strong study. Readers do not

but reviewers are trained to attack at the sign

sum the quality o f evidence over studies; they

of any weakness, and editors go with the

average it, so weak data dilute strong data.

most negative review (Fogg & Fiske, 1 9 9 3 ) .

Try not to become so attached to each study

Be ruthless with the quality o f your evidence;

that you cannot evaluate it with a cold, hard

don't waste your time and reviewers' time

eye, and k n o w when it is time to leave it aside.

with outlets that simply w o n ' t w o r k . If you

M o r e generally, admit your limits. K n o w

are t o o close to your product to judge it

the strengths o f your data (that part is easy),

yourself, ask some colleagues. It is better to

but also anticipate the reviewers' criticisms,

hear bad news from allies than critics.

however unreasonable. T h e n , explain their

Develop a thick skin: The criticisms are not

possible criticism ("A critic might argue . . . " )

directed at you personally, no matter how

and show why it is wrong or at least not

nasty they may seem. It's not you; it's this par-

deadly. Readers are more impressed when

ticular version o f this particular set o f studies.

authors do not oversell their work. M a n y

O f all the people I've known well enough to

reviewers are impressed when they can say,

trade rejection stories, practically none receives

"Every time I thought o f a flaw or potential

an immediate acceptance for any study, no

problem, the authors addressed it on the

matter how brilliant. (This says nothing about

next page."

my choice of friends!) M o s t o f us make person-

Finally, do pay attention to the basic rules

specific attributions for our rejections: This

of good writing. Learning to write is a lifelong

editor or suspected reviewer hates me, I am a

project. Y o u can get a leg up from Strunk and

foreigner, I am unknown, I am too famous, my

White's slim classic, The Elements

Style

ideas are too mainstream, my ideas are too

( 2 0 0 0 , currently priced at about $ 6 . 9 5 ) . T h e

new. T h e more parsimonious explanation is

of

most c o m m o n problems in writing psychol-

the 9 0 % rejection rate for the best journals.

ogy are passive voice and needless words.

Rejection is part of the ritual, so get used to it.

Learn to excise the fat, leaving only the lean,

M o s t o f us cope by going through stages of

muscular, healthy prose. If this proves diffi-

grief: first shock, then anger at the stupid

cult, try editing on paper as well as on the

*#@& %

computer screen, and try coming back to

I should sell shoes), then

A

reviewers, then sadness

(maybe

(after a decent

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH interval) the gradual realization that maybe the

and people often acquire them unconsciously

reviewers made a few reasonable points. As

from others, or perhaps because those ideas

one considers how to address their criticisms,

are the next plausible step in the progress of

one begins to feel relieved that they didn't

science. W h a t you do own is your work on the idea, developing it and operationalizing it.

accept the article with all its glaring flaws. If your article is rejected, heed your own

N o two people work in precisely the same

clarion call for more research: Consider doing

way, so you should have some new angle to

it. Alternatively, you may decide that you are

contribute. Besides, any one study is only one

willing to settle for a lesser outlet. There is

small part o f a research program. Keep your

almost always a trade-off between time and

eye on the big picture.

quality; it takes longer to get into the best journals. O n e professor publishes only in the best journals, jettisoning studies and doing new

PROGRAMMATIC APPROACH:

ones until the reviewers cave in. This may be

F O L L O W Y O U R BLISS

hard on graduate students and junior faculty, however, w h o may not get publications in

T h e research program itself should be your

time to find or keep a j o b . Another equally

passion, so do not allow a few rejections to

successful professor used to have the envelope

set you back t o o far. Follow whatever inter-

ready for the next journal even before receiv-

ests you the most; this enterprise is t o o much

ing feedback from

the

w o r k to tolerate anything less. Caring about

grounds that reviews are arbitrary and non-

what you do will carry you over the bumps

overlapping, so revision to suit one bunch o f

in the road. If you don't k n o w for sure what

reviewers is futile. I take a slightly different

you want to pursue, you can discover your

tack, having

the first one, on

learned

that

editors

recruit

o w n preferences as revealed by the patterns

reviewers with differing expertise (D. W . Fiske

of your chosen research problems. It w o n ' t

& Fogg, 1 9 9 0 ) , so o f course they don't agree;

take you long to discover the pattern, and

they are evaluating the article on

then you can build on it.

dimensions. S o , one can learn

different from

the

Above all, as mentioned earlier, treasure a

reviews, and can collect new data or not, but

reliable effect when you find it. Domesticate it

the ultimate decision rests on the timing/qual-

by trying it in different theoretically interest-

ity trade-off, which may depend on the stage

ing contexts and variations. T e a c h it new

of your career (how important is it to get stuff

tricks by making it c o m e and go at will. If you

out

o f your

can make an effect appear and disappear, as

research topic (which also may determine how

noted, then you really understand it. In effect,

important it is to get stuff out quickly).

knowing the moderator variables explains

quickly) and

the trendiness

If you are scooped, don't panic. Chances

much about the effect. Also, learn

what's

are, the other person did the research differ-

inside it: W h a t are the mediating variables

ently than you did. Being scooped is annoying

that link the primary cause to the primary

at best and deeply wounding at worst (though

effect?

be

Having played awhile with your treasured

relieved not to have to complete a study when

effect, don't fully housebreak it. Let it outside,

someone else does it first). Especially at the

for others to take for a walk. If you clean up

start o f one's career, when one has not yet

all the mess from your effect, no one else will

I k n o w one professor w h o claims to

had many ideas, each idea is even more pre-

be interested. Let other people do something

cious. Nevertheless, people do not really own

with it—unless, o f course, you are utterly

ideas; they own the work. Ideas are in the air,

driven to know every last detail of its nature.

Developing Some researchers believe in letting others

What

about

a Program

fillers?

of

Many

Research evaluators

housebreak their discoveries, whereas other

discount chapters in edited volumes. It can be

researchers are more possessive. In general,

good to do one or two as a graduate student, if

what B o b Abelson calls the "neats" like to

your adviser asks you to collaborate, because

clean up every detail, whereas the "scruffies"

the writing and literature review experience

like to propose an idea and let others clean up.

can be useful. But if you are going to do all that

Researchers w h o belong to the neats and

work, why not do it for a review journal? If

the scruffies differ also in their willingness t o

you are going to do all that work, why be sec-

be wrong. Scruffies generally believe it is

ond author? Within the bounds o f maintaining

better t o be wrong than boring, t o flame out

a good relationship with your adviser, discuss

in a burst o f fireworks, pick up, and start

the issues o f costs and benefits for you to do a

over after making a great but

chapter together. For junior faculty, there are

misguided

show. Neats generally believe it is better to

likely to be fewer invitations, either from edi-

be careful and cautious, building an argu-

tors or from more senior collaborators, but

ment brick by brick. B o t h perspectives have

that is probably just as well. For graduate

some merit, and each o f us has t o calibrate

students, chapters may be evidence o f some

our own willingness to be wrong. Research

form o f low-level activity, but for junior

requires taking some risks, trying on new

faculty, that isn't much help. L o o k up the cita-

ideas to see h o w they fit, and being willing to

tion rates for someone's chapters, compared to

discard them if they don't, but high-risk,

articles. The chapters are low, low, low.

high-gain research can leave you with nothing at the end o f the day. Balance is key.

So why do chapters at all? First, for fun. If you have something you want to say, and you

Besides individual differences in the willing-

don't want to have to deal with reviewers

ness to be wrong or to take risks, one's situation

(only the editors w h o invited you, probably

matters, of course. It is easier to take risks after

for the perspective they k n o w you have), a

achieving tenure; that's the whole point of life-

chapter is a good outlet. Second, for prestige.

long job security. It is easier to take risks with

Some very few volumes, o f course, carry a lot

one's left hand if one is already maintaining a

of prestige, and you will k n o w which ones

more reliable research program with one's right

and presumably

hand. It is easier to take risks if you have some

Third, for a particular audience w h o may not

accept those

invitations.

solid publications already. Staking the initial

otherwise see your w o r k . Fourth, for a litera-

stages o f your career on one high-risk, high-

ture review you have written for another pur-

gain project probably is not a good idea.

pose. O n e o f my more cited chapters was

Evaluators do look for both quantity and

prepared originally as a grant proposal. It is

quality. T h a t is the hard truth. M o s t depart-

the rare edited b o o k , however, that has much

ments want t o k n o w what your own partic-

impact, and likewise the chapters therein.

ular phenomenon is. I was once told to find

Chapter invitations are gratifying, as a

a "Fiske effect." But most departments also

sign to you and your evaluators that you've

have

sheer

had an impact on the field in a particular

number o f publications per year (I've heard

area. T h e key is weighing the opportunity to

an

implicit e x p e c t a t i o n

for

the number two, in one o f the top few jour-

be associated with a particular collection o f

nals, but it varies a lot). T h e trick is to try t o

editors and authors, t o be able t o speculate

keep projects in the pipeline—some things at

and go beyond the data, to m a k e a contro-

the planning stage, others piloting, others

versial argument, and t o support the editors'

running, other analyzing, others (one hopes)

enterprise, on one hand, with the oppor-

writing, and others under review.

tunity costs, on the other hand, such as

81

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH sacrificing

time that

could

be spent

on

articles and grants.

assumptions that they do not test; political psychology and behavioral economics come

W h e n in doubt, concentrate on research

out of such collaborations. Some o f the latest

articles. I f you do chapters, do them as a sec-

work in cognitive neuroscience has discovered

ond priority, not a first priority. Save your

that it needs social psychologists to help inter-

best hours in the day for research writing.

pret the pesky intrusiveness of emotion and

on

other social variables. Social psychologists

weekends or during the evenings, when they

Some people write chapters at h o m e

need neuroscientists, physicists, and statisti-

have nothing more pressing to do. But if you

cians to do social neuroscience. Health psy-

put your name on it, you still have t o do a

chologists need collaborators with

medical

good j o b . It should not be t o o far afield from

expertise. Psychologists and

your research program,

much to share. M a n y funding agencies are par-

so you

do

have

lawyers

have

something intelligent to say, without under-

ticularly excited by interdisciplinary collabora-

taking a whole new line o f inquiry, and then

tions. Cross-boundary collaborations glue our

other people can see why you

bothered.

field together at a time when it is threatening to

Otherwise and overall, all else being equal,

fly apart into tiny specialized pieces. Building

stick with the journal articles.

bridges

is useful

in

sticking

psychology

together. Cross-boundary collaboration also carries COLLABORATION: BESIDE

some risks, as I've recently suggested (S. T .

EVERY GOOD RESEARCHER

Fiske, 2 0 0 2 ) : T h e more micro

STANDS A T E A M

sciences l o o k down on the more ("softer")

sciences; b o t h

("harder") macro

may resent

the

In each arena—coming up with ideas, forming

intrusion or defection; neither side owes alle-

hypotheses, designing research, analyzing it,

giance t o you, so resources, alliances, and

writing it, dealing with rejection, and doing

identity may be at risk; and lack o f expertise

programmatic research—good collaborators

is a real issue. O n the other hand, collabora-

are priceless. In finding collaborators, as in

tion cures many o f these problems, and some

finding

of the most creative w o r k emerges from this

romantic

partners,

be

open

to

serendipity but be choosy. Each of us has

kind o f project.

research interests and talents that form a tem-

Once you have agreed to collaborate, inter-

plate; this template can mesh with a variety o f

dependence can work marvels if each of you

other templates, but not all. T h e point is that

contributes in areas where the other one is

no one adviser or collaborator is the be-all and

learning. Managing the collaboration requires

end-all. Y o u can work happily with various

deliberate attention. Sometimes it is important

people. Y o u r template is at the ready; various

to have authorship discussions up front, to

prospects can fit. So choose what works, but

avoid later misunderstandings. Having the idea

don't agonize over the perfect match.

is not enough to merit first authorship. As with

Interdisciplinary collaborations in particu-

strangers in the field, so too with collaborators.

lar matter right now, given both the complex-

N o one can really own the free-floating idea;

ity of some paradigms and the potential for

only the work establishes ownership. Some

creativity at the boundaries of disciplines (see

researchers refrain from discussing their ideas

Cacioppo, Lorig, Nusbaum,

and Berntson,

with anyone except a collaborator, to avoid

Chapter 17, this volume, and Part V of this

being scooped. But even with a collaborator,

handbook). As noted, some o f the more macro

the ideas are likely to develop in ways that are

social sciences make strong psychological

difficult to track, so explicit discussions are key.

Developing

a Program

of Research

\

In discussing the w o r k that does establish

the students had not called whom they said

ownership, people typically award authorship

they had called; they simply spoke to the first

to the person w h o did more work. But

person available. This was a failure o f teaching

beware the self-serving bias here: Y o u r col-

and motivating on my part, as well as irre-

l a b o r a t o r s ) probably did more than

you

sponsibility on the R A s ' part. H a d they under-

think. Add together each person's estimate o f

stood the importance o f random sampling, one

his or her own contribution, and the total

hopes they would not have cut corners.

will be more than 1 0 0 % ( M . Ross & Sicoly, 1 9 7 9 ) . Even if you agree that one person did more, you each are likely to underestimate the

T E A C H I N G : A PIECE O F T H E

other(s). Respect for the self-serving bias sug-

RESEARCH ENTERPRISE

gests giving the other person benefit o f the doubt. Include research assistants, staff, and

From

younger collaborators as coauthors, when-

indeed is a form o f collaboration. But o f

ever possible and appropriate. T h a t is, people

course it is more, because one person clearly

the research perspective,

teaching

deserve authorship for scholarly input, but

has more knowledge and authority, not to

they do not deserve authorship just for run-

mention power to evaluate, than the other.

ning participants for pay or providing techni-

Some departments recognize the teaching that

cal assistance. It is better to be direct, honest,

goes on in one-to-one supervision and lab

and blunt than to be perceived as exploitative,

meetings. M o s t departments do not, perhaps

unethical, or unfair. Although I usually err on

because they view research and teaching as

the side of inclusiveness, each new coauthor

orthogonal.

dilutes the perceived impact o f the

prior

But

are they?

One

could

argue

that

authors. In general, communicate, communi-

research and teaching should be negatively

cate. (See Fine and Kurdek, 1 9 9 3 , for some

correlated, based on scarcity o f time and

other

energy, differing personality

reflections

on

faculty-student

co-

requirements,

authorship, and Z a n n a , 2 0 0 3 , on mentoring

and divergent rewards. O n e could also argue

graduate students.)

that the relationship should be positive, based

Responsibility for other people's training

on conventional wisdom

and the

shared

suggests letting go of some control, letting the

requirement for intelligence. One could even

less experienced people try their hand at the

argue that the relationship should be zero,

next step for which they might be ready. Even

because they are different enterprises and the

if you could do the work more efficiently your-

relevant personality dimensions are unrelated.

self, you are responsible for training those w h o

Meta-analysis shows that the overall effect is

work with you. T h e care and feeding o f junior

ever so slightly positive, if anything, but it

collaborators entails not exploiting them or

depends heavily on the evaluative dimension

taking them for granted, but making

(Feldman, 1 9 8 7 ; Hattie &

the

Marsh,

1996).

research experience at whatever level a learn-

Active researchers rate high on knowledge,

ing experience in the conduct o f science. Both

commitment, enthusiasm, and organization.

good teaching and research quality control

Research conveys little or no advantage in

require

close supervision.

Horror

stories

facilitating

interaction

or

managing

the

abound. I once told undergraduate research

course. Contrary to popular

assistants to follow a particular telephone sam-

teaching and research quality are correlated

assumption,

pling procedure to obtain a stratified random

somewhat more at liberal arts colleges than at

sample. W h e n I called the participants later,

universities, perhaps because the variance on

for an unexpected follow-up, I discovered that

research productivity is greater. The small

83

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH

84

positive relationship holds especially in the

undergraduate, it never occurred to me that

social sciences, compared t o natural sciences

faculty would want to w o r k with me because

and humanities. Across moderators of the

I k n e w I didn't k n o w anything. O n e o f my

effect size, the relationship is always slightly

r o o m m a t e s pointed out that I would be free

positive, though

(Oddly

labor, which motivated me to go volunteer to

enough, time spent on teaching does not

w o r k on a faculty member's research. M a n y

correlate with teaching quality.) T h e relevant

a research career has started that way. As a

typically small.

point here is that time spent on research does

faculty

not undermine quality o f teaching, but it does

choosy about your individual students. If you

predict articles published. Log those research

c a n , find people w h o will w o r k o n projects

member

or graduate student, be

hours, even when you are busy teaching. Set

of mutual interest. Point out that they will

aside time for course prep, meetings with

get more enthusiastic and expert help if it's a

students, and research. Protect each o f those

topic you're pursuing yourself. (Specific tips

spots on your schedule, to keep control o f

on h o w to teach lie outside the scope o f the

your time.

current volume, but see Bernstein, 2 0 0 3 . )

Given that our jobs entail both teaching and research, and that the two are at best loosely coupled, how can we help teaching improve

the

quality

of

our

F U N D I N G : AHA! PLUS . . .

research?

Teaching upper-level courses provides a built-

Funding your students—especially graduate

in

students—lies in both your and their best

incentive to

keep

up,

whether

or

undergraduates.

interest, because if they are funded, they are

Everyone knows that teaching a seminar in

less distracted from research by having to

your specially is a plum assignment, but even

earn

lower-level survey courses c a n provide oppor-

Fundable research ideas occur in all the ways

students are graduates

your

money

in o t h e r

less useful

ways.

tunities to scan the literature for readable arti-

indicated for any piece o f research. T h e dif-

cles and

for

ference is that you have to plan it all in

lecture. Having to become expert on the topic

advance. In fact, even if you d o n ' t get funded

pithy,

up-to-date e x a m p l e s

and explain it to students can provide insights

right

into unanswered

research program is not a bad idea. In the

research questions. Also,

a w a y , having

t o plan

a

plausible

some courses have a research component.

stress and overload o f the semester, when a

O n e professor has students with

different

new student is ready to w o r k with you, you

accents collect local housing discrimination

can pull one o f your proposed studies off the

data by making phone inquiries about rental

shelf (or at least out o f the proposal) and

listings and coding the agency's responsive-

develop it together. At least you will k n o w

ness (i.e., whether

that it fits well with your research program.

they even call

back).

Another professor has students collect ques-

Fundable

research, in my experience,

tionnaire data from family members and ana-

requires an interesting idea and proof o f feasi-

lyzes the data for class. Thinking about your

bility. T h e interesting idea maybe highly inno-

own research goals can enrich your teaching,

vative, or it may be the next logical step o f an

and vice versa.

old idea. Whichever it is, the good idea is eval-

In survey courses, keep an eye out for the

uated the way any o f our research ideas are

best, most enthusiastic students. Let them

evaluated for publication; all the same criteria

k n o w that they can do research with faculty,

apply. Perhaps with funding more than other

as an independent

research assis-

enterprises, the fine line between a great new

tantship, or senior thesis. W h e n I was an

idea and credibility established by a track

study,

Developing record is particularly delicate. Some proposals

a Program

of Research

\

sometimes fail to get their own research done

reviewers

because they do more than their share: being

claim is nothing new. Others are rejected for

everyone else's favorite statistical consultant,

get rejected for promising what

being so new that they are completely untried.

advising all the minority students, advising all

The best combination is a fresh new idea, con-

the women, advising all the athletes, or run-

vincing pilot data, and systematic develop-

ning the best meetings. Nevertheless, building

ment o f the approach over a series o f studies.

your program builds your home away from

Methods and statistics have to be credible, so

home where you spend all your days (and

one must prove one's expertise by compul-

some o f your evenings). If you want it to be a

sively specifying all the tiresome details in

place you enjoy, you have to contribute what-

advance. (For more thoughts on grant writing,

ever you do best.

see Steinberg, 2 0 0 3 ; Sternberg, 2 0 0 3 . )

In your department, also do your share,

All the advice about rejection by journals

but no more, unless you are building an

applies here as well. T h e difference is that you

administrative vita. Notice what other people

often get to try again (at least a couple o f

at your level do, and strive for equity. I f you

times) with the same funding agency and

do less, people will resent you. If you do

therefore the same reviewers. In the words o f

more, people may be grateful, but you w o n ' t

the / Ching, perseverance furthers. At a mini-

get your own research done, and they won't,

mum, the reviewers build some cognitive dis-

ultimately, promote you out o f gratitude.

sonance if they initially said the basic idea

In your university, o f course, you can also

was worthy, and then you keep doing every

build an administrative vita, if that is your

revision they ask. T h e y can still reject your

career trajectory, but m a k e it a deliberate

proposal, but at least it requires more mental

choice, not an accident. Active researchers

gymnastics than if you are less responsive to

can do well by doing good: Offer your partic-

their feedback. It is important to remember

ular forms o f expertise above all. (Let the less

that the reviewers are people like us, only they

productive people offer the general adminis-

are doing their bit for the field by plowing

trative labor that runs the university.) Y o u r

through

particular expertise can serve for its intrinsic

other

people's grant

proposals.

W h i c h brings us t o . . .

usefulness. F o r example, social psychologists k n o w a lot about affiliation, which is useful for student retention; about identity, which is useful for student affairs and housing; about

SERVICE: GIVING I T A W A Y

diversity, which is useful on a multicultural Some wise colleagues pointed out that service

campus; about persuasion, which is useful in

is the least useful tool for acquiring tenure

marketing the university; and so on. In addi-

(Roediger, 2 0 0 3 ; Taylor, 2 0 0 3 ) . Nevertheless,

tion, social psychologists are trained to ana-

tenure, promotion, and collegiality require

lyze a social situation for the variables that

that each person carries some of the shared

matter, t o measure them, and t o interpret

load. Generosity presumably is owed from the

data. F e w other specialties prove as useful in

inside out, in concentric circles.

academic management. T h e other benefits to

In your own program, service helps to

offering your expertise to the

university,

col-

besides being the right thing to do, are that it

leagues' lives. It is to everyone's advantage to

earns you respect among your colleagues and

be part o f a lively, active program, which

gives you ideas for research.

build team spirit and facilitate your

requires some effort on behalf o f the group. Be generous, but don't be a chump. People

Service to our national organizations keeps psychology

healthy.

Grant

and

journal

85

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH reviewing is the most obvious, c o m m o n form

psychological science from

of service. D o your share: Y o u will learn a lot

crazies, w h o wield considerable power and

the

talk-show

about current work in the field by people

influence. T h e y must not be the only voices

most relevant to your own areas o f expertise.

telling Americans about psychology. Ditto

Y o u ' l l learn a lot about what it takes to get

granting interviews to the media.

published or funded. You'll keep the enter-

Service to the larger community is a matter

prise going. Y o u ' d be surprised at h o w con-

of conscience. M a n y o f us have active social

troversial peer review can be, outside modern

consciences. Our field is biased to select ideal-

scientific circles. It's a privilege for us to mon-

ists because we believe the interesting variance

itor ourselves, to uphold scientific standards,

lies in the situation. T h e situation is more

and to teach each other. O n e professor called

mutable

than variance that

allegedly lies

it on-the-job

wholly within the individual

(e.g., narrow

training.

When

you

write

reviews, write about the manuscript or the

interpretations o f genetics, personality, prior

proposal, not the researcher as a person. Be

development). If you believe that the situation

direct but sensitive to the other person's feel-

influences people, then the answer to social

ings. D o n ' t use reviews to show how smart

problems lies in public policy that changes the

you are, at the other person's expense. D o use

situation. This is a liberal bias. (If you believe

reviews as a teachable moment. For grant

that the interesting variance lies within individ-

reviews, especially, be sure to mention the

uals, then public policy that changes the situa-

strengths o f the proposal. Social psychologists

tion is fruitless; it would be better to minimize

have a reputation for killing each other off,

interventions because people are responsible

leaving no one alive to be funded. Offer some

for their own outcomes. This is a conservative

praise, in case this imperfect proposal is nev-

bias.) Social psychologists tend to favor social

ertheless one o f the better ones and the pro-

change; the largest constituency of the Society

gram officer wants to fund it. Whether you

of the Psychological Study o f Social Issues is

do journal and proposal reviewing at h o m e or

social psychologists. T h e practical use of good

as part o f a review panel that requires travel

theories (Lewin, 1 9 4 3 ) dates back to the

may depend in part on your family situation

origins o f our field, for good reason.

or career stage. Weigh all the factors, but do your share somehow, sometime.

Ethnic minorities and women tend to feel some obligation to give back to their own

T h e mid-range organizations (Society for

communities and to help vulnerable in-group

Personality and Social Psychology, Society o f

members navigate the academic system. (For

Experimental Social Psychology) help all o f us

more on being an academic from one o f

by running journals and conferences. Typical

these groups, see J . M . Jones [ 2 0 0 3 ] and Park

stints on these committees are short, perhaps 3

[2003].) People from underrepresented groups

years, and you're off. If asked, you should do

also get asked to serve on committees precisely

it once in your career. T h e largest psychol-

so that their group is represented. This combi-

ogical organizations cal

(American Psychologi-

Association, American

nation o f factors can increase demands for ser-

Psychological

vice. In addition are all the informal networks

Society) help all of us by lobbying Congress

that increase advising demands. Especially if

for research funding. N o one else can do it as

you are yourself from an

effectively as they can, and two organizations

group, keep an eye on people at comparable

are better than one. Consider your dues to be

rank. If you are doing a lot more, go to your

underrepresented

effort,

chair and discuss the issue. If the person won't

which is expensive but vital. Consider any

help you cut back, do it yourself by saying no

national service to be your bit in protecting

to additional commitments. If you sink under

your contribution

to the lobbying

Developing

a Program

of

Research

an untenable load o f service, you are doing no

control resources. People higher up are less

one any long-run favors.

outcome-dependent on those lower down, so

In

department

and

university

service,

they are free to attend less carefully. Hence,

remember that service is not portable. Research

they are vulnerable

is portable from job to j o b , and research repu-

(S. T . Fiske, 1 9 9 3 a , 2 0 0 0 ; Goodwin, Gubin,

tations are the ticket to the next job. Teaching

Fiske, & Yzerbyt, 2 0 0 0 ) . M e n and women have

is somewhat portable—the class you prep one

particular kinds of power relations, as well as

to

stereotype

others

place may serve in another place. Service, how-

interdependence, which results in sexism having

ever, is not especially portable, again unless

more than one dimension, sexist male benevo-

you want to build an administrative career.

lence directed toward cooperative female subor-

Although service may earn you local gratitude

dinates, and sexist male hostility directed

and help you to network in the field, you can't

toward competitive female peers and superiors

live on gratitude, so keep a balance.

superiors (both measured by the Ambivalent Sexism Inventory or ASI) (Glick & Fiske, 1 9 9 6 , 2 0 0 1 ) . Social structure shapes reactions to a variety of out-groups, depending on perceived

CONCLUSION: F R O M

status and competition, with predictable effects

MADNESS T O T H E M E T H O D S

on perceived traits and emotional prejudices a

(S. T . Fiske, Cuddy, Glick, & X u , 2 0 0 2 ) . I give

research program is a bit mad. I have argued

these examples in an effort both to illustrate my

here that the processes o f developing

a

own social issues perspective and to illustrate

research career are knowable, manageable,

how different lines o f work end up connecting.

Offering

generic advice about building

and even fun. In doing this, I have drawn on

How

do all these personal-intellectual-

my own experience, observations made, and

social research puzzles become programmatic?

advice received. I have definitely drawn from

Some o f the links among these lines o f research

one particular perspective, and others will

are deliberate, such as Peter Glick and me

disagree. (For another perspective on begin-

thinking hard about the nature o f male-female

ning a p r o g r a m

o f research, see Z a c k s

interdependence and power relations, partly as

[ 2 0 0 3 ] ; for several wise perspectives on being

a result o f the prior work on outcome depen-

an academic, see the collection o f chapters in

dency. Some of the links are serendipitous,

Darley, Z a n n a , and Roediger [ 2 0 0 3 ] . )

such as being convinced that

out-groups

Research careers are highly idiosyncratic,

include more texture than simple antipathy,

and none is a universal example. As noted at

and consequently thinking about envious prej-

the outset, my own research interests admit-

udice against groups all over the world who

tedly derive from a social issues perspective:

immigrate as entrepreneurs (Jews in Europe,

experiences with neighborhood diversity, job-

Indians in East Africa, Chinese in Indonesia,

related sexism, organizational dynamics, and

Koreans in Los Angeles), as well as paternalis-

sheer wonder. M y theoretical approaches have

tic prejudice toward traditional women and

tended to emphasize the importance of social

people with disabilities; only later did we make

structure in understanding these phenomena.

the link to subtypes o f women and the fit to the

Whether people stereotype or individuate each

ASI. Some o f the links are accident, pure and

other depends on situation-driven interaction

simple: M y adviser in graduate school studied

goals (S. T . Fiske, Lin, etal., 1 9 9 9 ; S. T . Fiske

attention in social situations (Taylor & Fiske,

& Neuberg, 1 9 9 0 ) . Attention focuses up the

1 9 7 5 ) , so I became interested in that as a

power hierarchy in organizations

dependent variable. Probably an undergradu-

people's

goals

depend

on

because

people

who

ate degree in interdisciplinary social relations

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH set the stage for these intellectual interests.

doing what you love to do figures prominently

People not only are shaped by their environ-

on the agenda. One of the joys and challenges

ments but also choose them. And this is only

of being an academic is that you do set your

one person's story. It is O K to have multiple

own research agenda, largely planning your

passions, to enter new areas, and to learn

own use o f time and energy. Y o u can control

throughout one's career as one moves through

your time and your research life, to some

different environments and areas o f concern.

extent, so managing your research career can

Seemingly separate lines o f work are likely to

be deliberate to some extent. At a minimum,

intersect over the course o f one's career.

being aware o f some o f the processes in devel-

Whatever trajectory you create in building, maintaining, or refreshing a research program,

oping a research program makes you more aware o f the choices.

REFERENCES Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum. Asch, S. E. (1946). Forming impressions of personality. Journal of Abnormal and Social Psychology, 41, 2 5 8 - 2 9 0 . Bern, D. J . (2003). Writing the empirical journal article. In J . M . Darley, M . P. Zanna, &C H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Bernstein, D. (2003). Tips for effective teaching. In J . M . Darley, M . P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Berscheid, E. (1992). A glance back at a quarter century of social psychology. Journal of Personality and Social Psychology, 63, 5 2 5 - 5 3 3 . Campbell, D. T., &c Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Chaiken, S., & Trope, Y . (Eds.). (1999). Dual process theories in social psychology. New York: Guilford. Cohen, J , (1990). Things I have learned (so far). American Psychologist, 45, 13041312. Dahl, R. A. (1961). Who governs? Democracy and power in an American city. New Haven, CT: Yale University Press. Darley, J . M . , &c Batson, C. D. (1973). "From Jerusalem to Jericho": A study of situational and dispositional variables in helping behavior. Journal of Personality and Social Psychology, 27, 100-108. Darley, J . M . , Zanna, M . P., & Roediger, H. L., III. (Eds.). (2003). The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Feldman, K. A. (1987). Research productivity and scholarly accomplishment of college teachers as related to their instructional effectiveness: A review and exploration. Research in Higher Education, 26, 2 2 7 - 2 9 8 . Fine, M . A., Sc Kurdek, L. A. (1993). Reflections on determining authorship credit American and authorship order on faculty-student collaborations. Psychologist, 48, 1141-1147. Fischhoff, B. (1975). Hindsight is not equal to foresight: The effect of outcome of Experimental knowledge on judgment under uncertainty. Journal 1, 2 8 8 - 2 9 9 . Psychology: Human Perception and Performance,

Developing

a Program

of

Fiske, D. W., 8c Fogg, L. F. (1990). But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45, 5 9 1 - 5 9 8 . Fiske, S. T. (1993a). Controlling other people: The impact of power on stereotyping. American Psychologist, 48, 6 2 1 - 6 2 8 . Fiske, S. T. (1993b). Social cognition and social perception. In M . R. Rosenzweig & L. W. Porter (Eds.), Annual review of psychology (Vol. 4 4 , pp. 155-194). Palo Alto, CA: Annual Reviews. Fiske, S. T. (2000). Interdependence reduces prejudice and stereotyping. In S. Oskamp (Ed.), Reducing prejudice and discrimination (pp. 115-135). Mahwah, NJ: Erlbaum. Fiske, S. T. (2002). A case for lumping—neatly: Building bridges within and outside psychological science. American Psychological Society Observer, 15(7), 5, 37. Fiske, S. T., Cuddy, A. J . , Glick, P., 8c X u , J . (2002). A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology, 82, 878-902. Fiske, S. T., Lin, M . H., 8c Neuberg, S. L. (1999). The Continuum Model: Ten years later. In S. Chaiken 8c Y . Trope (Eds.), Dual process theories in social psy­ chology (pp. 2 3 1 - 2 5 4 ) . New York: Guilford. Fiske, S. T., 8c Neuberg, S. L. (1990). A continuum model of impression formation, from category-based to individuating processes: Influence of information and motivation on attention and interpretation. In M . P. Zanna (Ed.), Advances in experimental social psychology (Vol. 2 3 , pp. 1-74). New York: Academic Press. Fogg, L., 8c Fiske, D. W. (1993). Foretelling the judgments of reviewers and editors. American Psychologist, 48, 2 9 3 - 2 9 4 . Glick, P., 8c Fiske, S. T. (1996). The Ambivalent Sexism Inventory: Differentiating hostile and benevolent sexism. Journal of Personality and Social Psychology, 70, 4 9 1 - 5 1 2 . Glick, P. 8c Fiske, S. T. (2001). Ambivalent sexism. In M. P. Zanna (Ed.), Advances in experimental social psychology (Vol. 3 3 , pp. 1 1 5 - 1 8 8 ) . New York: Academic Press. Goodwin, S. A., Gubin, A., Fiske, S. T., 8c Yzerbyt, V. (2000). Power can bias impression formation: Stereotyping subordinates by default and by design. Group Processes and Intergroup Relations, 3, 2 2 7 - 2 5 6 . Hattie, J . , 8c Marsh, H. W. (1996). The relationship between research and teaching: A meta-analysis. Review of Educational Research, 66, 5 0 7 - 5 4 2 . Heider, F. (1958). The psychology of interpersonal relations. New York: Wiley. James, W. (1890). The principles of psychology. Cambridge, MA: Harvard University Press. Jones, E. E., 8c Sigall, H. (1971). The bogus pipeline: A new paradigm for measuring affect and attitude. Psychological Bulletin, 76, 349-364. Jones, J . M. (2003). The dialectics of race: Academic perils and promises. In J . M . Darley, M . P. Zanna, 8c H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Lewin, K. (1943). Psychology and the process of group living. Journal of Social Psychology, SPSSI Bulletin, 17, 1 1 3 - 1 3 1 . McGuire, W. J . (1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26, 4 4 6 - 4 5 6 . Milgram, S. (1965). Some conditions of obedience and disobedience to authority. Human Relations, 18, 57-76. Park, D. (2003). Women in academia. In J . M . Darley, M. P. Zanna, 8c H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association.

Research

FUNDAMENTAL ISSUES IN SOCIAL PSYCHOLOGICAL RESEARCH Roediger, H. L., III. (2003). Managing your career: The long view. In J . M . Darley, M. P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Ross, L., &C Nisbett, R. E. (1991). The person and the situation: Perspectives of social psychology. New York: McGraw-Hill. Ross, M., & Sicoly, F. (1979). Egocentric biases in availability and attribution. Journal of Personality and Social Psychology, 37, 3 2 2 - 3 3 6 . Steele, C. M., Spencer, S. J . , & Aronson, J . (2002). Contending with group image: The psychology of stereotype and social identity threat. In M. P. Zanna (Ed.), Advances in experimental social psychology (Vol. 3 4 , pp. 379-440). San Diego: Academic Press. Steinberg, J . (2003). Obtaining a research grant: The view from the granting agency. In J . M. Darley, M. P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Sternberg, R. J . (1993). The psychologist's companion: A guide to scientific writing for students and researchers (3rd ed.). New York: Cambridge University Press. Sternberg, R. J . (2003). Obtaining a research grant: The applicant's view. In J . M. Darley, M . P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Strunk, W., &£ White, E. B. (2000). Elements of style (4th ed.). Boston: Allyn & Bacon. Taylor, S. E., &c Fiske, S. T . (1975). Point of view and perceptions of causality. Journal of Personality and Social Psychology, 32, 4 3 9 - 4 4 5 . Taylor, S. E. (2003). The academic marathon: Controlling one's career. In J . M. Darley, M. P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Zacks, J . (2003). Setting up your lab and beginning a program of research. In J . M. Darley, M. P. Zanna, &c H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association. Zajonc, R. B. (1994). Emotional expression and temperature modulation. In S.H.M. van Goozen & N. E. Van de Poll (Eds.),. Emotions: Essays on emotion theory (pp. 3-27). Hillsdale, NJ: Lawrence Erlbaum Associates. Zanna, M. P. (2003). Mentoring: Managing the faculty-graduate student relationship. In J . M . Darley, M. P. Zanna, & H. L. Roediger III (Eds.), The compleat academic: A career guide (2nd ed.). Washington, DC: American Psychological Association.

Part III DESIGN AND ANALYSIS Section A Implications of a Heterogeneous Population: Deciding for Whom to Test the Research Question(s), Why, and How

CHAPTER

5

Culturally Sensitive Research Questions and Methods in Social Psychology JOAN G . MILLER New School

University

S

ocial psychology is distinguished by its

constituted

attention to the power of the situation

resulting in a need to take culture into account

and to the dynamics o f social groups. It

also is highly sensitive to the active role of the

by

sociocultural

processes,

in all research designs, even in work conducted with single populations.

sense o f experience.

There are many answers to the question of

However, even with this sensitivity to context

why cultural considerations must be consid-

and to processes o f individual construal and

ered in social psychological research. It is per-

meaning making, the field gives little weight to

haps most commonly recognized that we need

observer in making

culture in its theories and methods. T h e present

to attend to culture for methodological

chapter offers methodological strategies for

purposes.

control

It is critical to take into account cul-

enhancing the cultural sensitivity o f social psy-

turally related differences in individuals' back-

chology, strategies that are critical in increasing

ground, knowledge, experiences, or outlooks

the field's theoretical power and explanatory

that may differentially affect their understand-

breadth, as well as its applied relevance. While

ings o f methodological procedures and lead to

involving design decisions, entailing such issues

such procedures not having equivalent mean-

as sampling, choice o f procedure, and interpre-

ing for different subgroups. Thus, for example,

tation of findings, the strategies also involve

populations that are unfamiliar with certain

key

research stimuli may perform poorly on some

conceptual issues, with

strategies for

enhancing the cultural sensitivity o f research

of the standard items included on intelligence

methods in social psychology depending on

tests (Laboratory o f Comparative

Human

understanding the theoretical role o f culture in

Cognition, 1 9 8 3 ) . Likewise, even such mun-

informing the field's core conceptual notions.

dane methodological strategies as tapping

It must

background information at the start of a ques-

be recognized that psychological

experience always occurs in and is, in part,

tionnaire can have detrimental

effects

on

94

DESIGN AND ANALYSIS performance for certain subgroups, as research

Bromley, 1 9 7 3 ) , a trend not only believed to

on stereotype threat has documented (Steele &

be universal but also assumed to result from developmental changes in young children's

Aronson, 1 9 9 5 ) . A second motive for attending to cultural

cognitive facilities in abstraction and in the

This type

range of their experiences. M y research docu-

of effort is guided by concerns with assessing

mented that Hindu Indians do not display the

the assumed universality o f existing psycho-

age

issues is for theory-testing purposes.

increase

in

dispositional

inferences

logical theories through sampling culturally

observed among U.S. respondents. Rather,

diverse populations, as well as with identifying

they show an age increase in their emphasis on

mediating or moderating variables that affect

contextual factors—an age effect notably not

the manifestation o f particular psychological

observed among U.S. children. This work was

effects. An example o f this type o f approach

important in offering a new explanation of the

may be seen in comparative research that has

processes underlying developmental change in

tested the universality o f Baumrind's highly

social attribution. It became clear that previ-

influential model o f parenting, a framework

ous cognitive and experiential interpretations

that was developed initially based on data

of age changes were incomplete and that it

from middle-class samples (Baumrind, 1 9 9 6 ) .

was critical to recognize that enculturation

This research has uncovered the important

processes contribute to such age changes. It

phenomenon

parenting

also became clear that the direction o f devel-

practices that had been found to have negative

opmental change in social attribution is cul-

that

authoritarian

effects in middle-class environments tend to

turally variable rather

have positive effects in the context o f danger-

previously assumed.

ous and

impoverished

neighborhoods,

than universal,

as

in

In sum, taking cultural considerations into

which they are associated with the provision

account in social psychological research is

of higher levels o f support and supervision

needed not only for the methodological reasons

(Baldwin, Baldwin, & Cole, 1 9 9 0 ) .

of ensuring the validity o f assessment tech-

Notably, culturally based research is also

niques but also for the theoretical reasons of

construction

testing the universality of psychological theories

with this aim central to the newly

and of formulating new conceptual models.

reemerging perspective of cultural psychology

Extending beyond merely an understanding o f

(e.g., Fiske, Kitayama, M a r k u s , & Nisbett,

diversity in psychological functioning,

1 9 9 8 ; Markus, Kitayama, & Heiman, 1 9 9 6 ;

attention can provide new process understand-

Miller, 1 9 9 7 , 1 9 9 9 ; Shweder, 1 9 9 0 ) . This type

ings o f the psychological functioning of widely

of approach is concerned not merely with

studied Western populations.

increasingly guided by theory goals,

such

uncovering diversity in modes o f psychological functioning but also with identifying the previously unrecognized cultural dependence

DOWNPLAYING OF CULTURAL

of existing psychological theories. It was this

ISSUES I N SOCIAL P S Y C H O L O G Y

type o f agenda, for example, that motivated my early cross-cultural developmental investi-

Although recent years have seen a renewed

gation contrasting the everyday social expla-

interest in cultural issues in social psychology,

nations o f samples of Euro-American and

such considerations nonetheless remain in a

Hindu Indian adults and children

peripheral

(Miller,

position in the field. Whereas

1 9 8 4 ) . Previous developmental research had

increasing efforts are being made to sample

documented an age increase in dispositional

culturally diverse subgroups, most contempo-

inference (Damon & Hart, 1 9 8 2 ; Livesley &c

rary social psychological research centers on

Culturally Sensitive Research

Questions

and

Methods

the predominantly middle-class Euro-American

1963) or in the prison simulation study of

college populations that historically have con-

Zimbardo and his colleagues (Haney, Banks

stituted the prototypic population for social

8c Zimbardo, 1973). In another example, this

psychological inquiry. Within the major text-

type o f insight also informs contemporary

books and substantive handbooks in the field,

research on priming and on the mere exposure

basic theory tends to be presented in universal

effect, work that is documenting the power o f

terms. Thus, in some illustrative examples,

situations to influence behavior in ways that

recent major handbooks of social psychology

are outside individuals' conscious awareness

include only a single chapter devoted to cultural

(e.g., Bargh, 1996; Bornstein, Kale, 8c Cornell,

psychology, with the indexes revealing rela-

1990). As approached within this dominant

tively few references to culture in the other

perspective, the situation is treated as present-

chapters in the volumes (e.g., Gilbert, Fiske, 8c

ing a veridical structure that can be known

Lindzey,

1998; Higgins & Kruglanski, 1996).

through inductive or deductive information

T o give increasing weight t o sociocultural

processing. N o consideration is given to cul-

considerations in social psychology, it is criti-

ture as necessarily implicated in the definition

cal t o understand the reasons why culture

of the situation or to cultural presuppositions

tends to be downplayed in the field. It is these

as constituting prerequisites o f what is consid-

addressed

ered objective knowledge. It is assumed that

through gaining a greater understanding o f

variability in judgment arises from differences

the nature o f cultural processes and their role

in the information available to individuals or

in psychological p h e n o m e n a

from differences in their informative process-

types o f concerns that can be

as well

as

through the adoption o f m o r e culturally sen-

ing, resulting in certain judgments being more

sitive methodological strategies.

or less cognitively adequate or veridical than others (Nisbett

8c Ross, 1980).

This realist view o f situations gives rise to

Key Reasons for Downplaying of Culture

explanatory frameworks focused on factors in the situation and in the person. Within such

T h e reasons for the downplaying o f cul-

frameworks, culture is viewed merely as a dis-

ture in social psychology are both conceptual

tal causal factor with impacts on psychologi-

long-standing

cal effects through its influences on proximal

assumptions in the field about the nature o f

situational or person factors, rather than as a

and empirical. T h e y reflect

social psychological explanation as well as

factor

disappointment with the findings from vari-

explanatory force. Thus, for example, in cer-

ous traditions o f culturally based social psy-

tain early models in cross-cultural psychology,

chological research.

such as the eco-cultural model developed by Berry

that

itself

contributes

additional

(1976), the situation is treated as pre-

senting varied resources and constraints that Culture-Free

Approach

One o f the landmark

to

Situations

are seen as making varied forms o f psychologi-

contributions o f

cal response adaptive, such as field depen-

social psychology is that it has highlighted the

dence being linked to agricultural modes o f

power o f situations in affecting behavior. It is

subsistence and field independence

this insight that underlies some o f the early

linked to hunting and gathering modes (Berry,

being

groundbreaking programs o f research docu-

1976; Witkin 8c Berry, 1975). This type o f

menting ways in which situational influences

treatment of the situation, it should be empha-

can lead to antisocial behavior, such as in the

sized, is important in taking into account that

Milgram conformity experiments (Milgram,

individuals from different backgrounds may

DESIGN AND ANALYSIS be exposed to different ecological experiences.

vision as its dominant research

However, it treats culture merely as a consid-

social psychology has a tendency to consider

eration that

is already accommodated

in

the social psychological focus on situational factors.

cultural

considerations

paradigm,

as mere

content

effects and thus as factors that ideally should be held constant in order to focus on isolating

Equally, culture may be treated as an individual difference factor, a stance that is seen,

more fundamental underlying psychological mechanisms (Malpass, 1 9 8 8 ) .

for example, in the enthusiasm shown for assessing culture through individual difference approaches, such as scale measures o f individ-

Apparent

ualism/collectivism (e.g., see the recent review

and Explanatory

Breadth

by Oyserman, Coon, & Kemmelmeier, 2 0 0 2 ) .

of Psychological

Theories

From such a perspective, cultural group mem-

The

bership is viewed as giving rise to individual differences in attitudes, understandings,

Universality

limited interest shown

in

cultural

research within social psychology also reflects

and

the sense within the discipline that social psy-

available information. Thus, it is viewed as a

chological findings, in fact, have been docu-

consideration that

into

mented in most cases to be cross-culturally

account in social psychological explanation,

robust and to have considerable explanatory

already

is taken

through the field's present attention to indi-

scope. It is thus concluded that no significant

vidual differences or person factors.

cross-cultural variation exists in basic psychological phenomena (Brown, 1 9 9 1 ) . The conclusion o f apparent universality in

Physical Ideals of

cross-cultural research is linked with method-

Science

ological strategies o f administering existing

Explanation

T h e tendency to downplay cultural considerations in social psychology also stems from the field's embrace o f an

idealized

physical-science model o f explanation. As Higgins and

Kruglanski

(1996)

recently

explained, this type o f stance involves a view of psychological science as the search for deep structural explanatory mechanisms:

research instruments in diverse cultural settings, after making only minor changes in their content to ensure familiarity, and narrowing the scope of the phenomena being investigated in ways that exclude possibly significant cultural variation. An example o f the first type o f approach may be seen in the extensive body o f cross-cultural research that tested the universality o f Kohlberg's theory o f

A discovery of lawful principles governing a realm of phenomena is a fundamental objective of scientific research. . . . A useful scientific analysis needs to probe beneath the surface. In other words, it needs to get away from the "phenotypic" manifestations and strive to unearth the "genotypes" that may lurk beneath, (p. vii)

moral development, through

administering

standardized Kohlbergian research protocols in more than 4 5 different societies (Snarey, 1 9 8 5 ) . Although the results revealed that the distribution o f the highest levels o f moral development were highly skewed and

the

highest levels tended to be found primarily in Western urbanized cultures, Kohlberg and his colleagues interpreted the results as confirm-

From this perspective, psychological pro-

ing the universality o f his stage model, because

cesses are viewed as resembling the laws o f

all responses could be seen as either higher or

physical science in being timeless, ahistorical,

lower stages o f Kohlbergian moral

and culturally universal. In adopting

development

this

stage

(Kohlberg, 1 9 8 4 ; C. Levine,

Culturally Sensitive Research

Questions

and

Methods

an

disillusionment with cultural research that

example o f the strategy o f adopting method-

was stimulated by M a r k u s and Kitayama's

Kohlberg, &

Hewer,

1 9 8 5 ) . In turn,

arguably exclude

( 1 9 9 1 ) groundbreaking article on culture and

potentially significant sources o f variation

the self, with its introduction o f the distinc-

may be seen in research on the coding o f emo-

tion between independent and interdependent

tional facial expressions. T h e widely accepted

cultural self-construals. O n e o f the

conclusion o f fundamental similarity in basic

widely cited articles ever in social psychology,

emotion concepts that has emerged from the

this work has given rise to extensive research

extensive cross-cultural research conducted on

that has been inspired by this latter construct,

this topic (e.g., Ekman, 1 9 9 2 ; Izard, 1 9 9 2 )

with the focus on examining the extent to

ological procedures that

most

stems, at least in part, from the use o f proce-

which variation in psychological functioning

dures that tend to gloss over potentially signifi-

can be predicted by scale measures o f this

cant sources o f variation in emotion concepts,

construct (Singelis, 1 9 9 4 ; Triandis, 1 9 9 5 ) .

such as differences in h o w emotion concepts

However,

as recent criticisms o f this

are expressed in everyday language usage, and

rapidly growing literature make clear, the

that downplay the significance of lexicalized

results observed utilizing scale measures o f

emotion terms whose translation into English-

interdependent/independent

language concepts is inexact (see critique in

have been disappointing (Hong, Morris, Chiu,

Russell, 1 9 9 4 ) .

self-construals

& Benet-Martinez, 2 0 0 0 ; M a t s u m o t o , 1 9 9 9 ;

Indirect evidence for the universality of psy-

Oyserman et al., 2 0 0 2 ) . M u c h o f the work has

chological theories also comes from the high

been associated with a stereotypical stance

levels o f intercorrelation observed between

that

psychological constructs. T o illustrate, support

between and within cultures and that gives

glosses over

important

distinctions

for

the universality o f the theory o f self-

insufficient attention to the impact o f context

determination developed by Deci, Ryan, and

on behavior. T h e same type o f sophisticated

their associates (Deci & Ryan, 1 9 8 5 , 1 9 9 0 ) is

understanding o f situational influences that is

based not only on research indicating that scales

evident in mainstream social psychological

of autonomy support show the same empirical

research is not evident in this type o f social

relationships in a country such as Bulgaria as

psychological work, which much o f is focused

they do in U.S. samples (Deci, Ryan, Gagne,

on cultural questions. Notably, work in this

etal., 2 0 0 1 ) but also through studies demon-

tradition is also yielding findings that, in some

strating

cases, appear to contradict directly the claims

that

self-determination

constructs

predict psychological functioning in related

of

the

interdependent/independent

domains. In this regard, for example, it is

construal paradigm,

demonstrated that self-determined motivation is

reported by Oyserman et al. ( 2 0 0 2 ) , based on

related empirically to such variables as adaptive

their extensive meta-review, that "relationship

parenting, higher self-esteem, and higher stages

and family orientation are not empirically

of Kohlbergian moral development (e.g., Deci,

closely linked to collectivism" (p. 4 3 ) .

such as the

self-

findings

Ryan, Gagne, etal., 2 0 0 1 ; Grolnick, Deci, & Ryan, 1 9 9 7 ; Grolnick & Ryan, 1 9 8 9 ) .

Disappointment Cultural

With

Traditions

of

Recent Research

CONCEPTUAL ISSUES IN GIVING MORE ATTENTION T O CULTURE The

remainder o f this chapter focuses on

Finally, the downplaying o f the signifi-

specific methodological research strategies

cance o f cultural research also reflects certain

that are important t o adopt in enhancing the

98

DESIGN AND ANALYSIS cultural sensitivity o f social psychological

Symbolic approaches

treat

culture

as

research. Before turning directly to these

shared meanings that are embodied in arti-

methodological strategies, however,

facts and practices and that form a medium

atten-

tion first focuses briefly on some o f the

for human development (e.g., Cole, 1 9 9 5 ;

conceptual issues that must

such

D'Andrade, 1 9 8 4 ; R . A. LeVine, 1 9 8 4 ; Shore,

methodological efforts and that respond to

1 9 9 6 ) . It is recognized that cultural meanings

some o f the reasons for the field's downplay-

and practices not only represent experience

inform

ing o f culture noted above. These considera-

but also are constitutive o f experience, in

tions bear on the nature o f culture and its

serving to create socially constituted realities

influences on psychological processes.

(Bartlett, 1 9 3 2 ) . F o r example, not only do social categories and institutions depend on cultural

Views of Culture From an ecological perspective, culture is understood

definitions

(e.g., "bride,"

"mar-

riage"), but even psychological concepts are

as adaptations

to the

varying

recognized to be, in part, culturally based. Thus, as seen in the example o f the Japanese

requirements of contrasting physical and social

concept o f amae

structural environments (e.g., Bronfenbrenner,

even psychological phenomena, such as emo-

1 9 7 9 ; Whiting & Whiting, 1 9 7 5 ) . Ecological

tions, depend in part on cultural distinctions

approaches to culture are o f value in highlight-

embodied

ing the varied resources and constraints that

discourse, and

individuals from different sociocultural com-

(Shweder, 1 9 8 4 ; Wierzbicka, 2 0 0 2 ) .

]

(Dot, 1 9 9 2 ; Russell, 1 9 9 1 ) ,

in natural

language categories,

everyday

social practices

munities experience and that influence their

Challenging the identification of cultural pro-

behavior. For example, ecological frameworks

cesses exclusively with the situational factors

have informed most contemporary psychologi-

taken into account in social psychological expla-

cal studies with U.S. minority populations,

nation, a symbolic approach to culture high-

and this work is calling attention to ways in

lights the need to recognize that

which

differential

meanings do not bear a one-to-one relationship

resources and their experiences of bias and dis-

to objective aspects of the situation. Culture

crimination

then cannot be understood merely by consider-

individuals' affect

access to important

intellectual,

social, and health outcomes (e.g., M c L o y d &c

cultural

ation of the objective affordances and con-

Flanagan, 1 9 9 0 ; Neighbors & Jackson, 1 9 9 6 ) .

straints o f particular contexts but

It may be noted, however, that whereas eco-

requires taking into account cultural beliefs, val-

logical approaches to culture extend the domi-

ues, and practices that are not purely function-

nant social psychological models in their

ally based. T o give an example, research has

instead

recognition that the adaptive context for psy-

shown that Japanese teachers consider the ideal

chological development is culturally variable,

teacher/student ratio in preschools to be consid-

rather than universal, these approaches retain a

erably higher than do their U.S. counterparts

view o f the context as an objective environ-

(Tobin, W u , & Davidson, 1 9 8 9 ) . The decisive

ment. In this respect, then, while essential, such

consideration notably is not the consideration o f

approaches do not challenge the traditional

higher cost in teacher salaries but the value of

social psychological explanatory focus

on

socializing children to be competent members

features o f the person and of the objective

of social groups. As one Japanese teacher

situation. For this reason, it is critical to

explained, "Children need to have the experi-

complement ecological approaches to culture

ence of being in a large group in order to learn

with

to relate to lots of kinds of children in lots of

approaches

grounded.

that

are

symbolically

kinds o f situations" (Tobin et al., 1 9 8 9 , p. 3 7 ) .

Culturally Sensitive Research

Integrating Cultural Considerations With Situational and Person Factors Finally, it must be recognized that cultural considerations complete but do not replace the focus on situational and person factors in

Questions

diverse populations.

and

Methods

Given the reality o f

psychological experience always occurring in specific cultural contexts, sensitivity to cultural issues is needed in all social psychological investigations.

social psychological explanation. This implies that hypotheses involving cultural influences need to be formulated in ways that take into account both contextual variation and individual differences. Equally, it must be recognized that in many cases the impact o f individual difference and o f contextual factors may themselves be culturally variable. For example, research has shown that whereas U.S. respondents utilize more abstract self-references in a task context that is abstract as compared with concrete, Japanese respondents

display

the

opposite effect of context (Cousins, 1 9 8 9 ) . In sum, the key to enhancing the cultural sensitivity o f social psychology is understanding culture and its role in psychological functioning. Attention must be paid to culture as an ecological context that presents certain objective affordances and constraints, as well as to culture as a symbolic environment that entails certain meanings and practices that are not entirely functionally based. It must be recognized that a consideration o f culture does not replace an attention to person and situational factors but contributes an additional dimension to social psychological explanation.

Cultural

Understanding

As a field, social psychology bases many o f its research hypotheses, in part, on informal observations

made

by researchers

about

behavioral effects that they have observed or personally experienced. In this regard, it is not uncommon for social psychologists to draw on informal personal anecdotes as a preliminary way of communicating to readers the nature o f a particular effect. In fact, it has even been argued that much o f the success o f social psychology, in terms o f the generative nature of its ideas and its applied relevance, reflects this interplay between lay understandings and formal scientific inquiry. As Moscovici once commented: The real advance made by American social psychology was . . . in the fact that it took for its theme of research and for the content of its theories the issues of its own society. Its merit was as much in its techniques as in translating the problems of American society into sociopsychological terms and in making them an object of scientific inquiry. (1972, p. 19) A concern that may be raised about this

METHODOLOGICAL STRATEGIES FOR ENHANCING CULTURAL SENSITIVITY

type o f stance, however, entails its cultural boundedness. T h e assumptions that make the research questions and hypotheses o f social

Building on the conceptual issues discussed

psychology compelling for North American

above, this section identifies methodological

psychologists, because they speak to issues that

strategies that are valuable to adopt in efforts

are familiar and

to enhance the cultural sensitivity o f social

tribute to making them less significant for

psychology. T h e strategies discussed include

researchers from other cultural groups who

considerations that are important not only in

may not share these same cultural experiences

comparative

also in

and outlooks. As conveyed in the following

research that does not focus explicitly on cul-

firsthand account by a Chinese psychologist,

tural questions and/or on tapping culturally

individuals from other cultural backgrounds

research designs but

socially meaningful,

con-

99

100

\

DESIGN AND ANALYSIS

may find that their own assumptions

and

concerns are not adequately taken into account: I found the reasons why doing Westernized psychological research with Chinese subjects was no longer satisfying or rewarding to me. When an American psychologist, for example, was engaged in research, he or she could spontaneously let his or her American cultural and philosophical orientations and ways of thinking be freely and effectively reflected in choosing a research question, defining a concept, constructing a theory and designing a method. On the other hand, when a Chinese psychologist in Taiwan was conducting research, his or her strong training by overlearning the knowledge and methodology of American psychology tended to prevent his or her Chinese values, ideas, concepts and ways of thinking from being adequately reflected in the successive stages of the research process. (Yang, 1997, p. 65)

(McLoyd, 1990); in psychology of women, women generally refers to White women (Reid, 1988). When we mean other than White, it is specified. (Reid, 1994, p. 525) It must be recognized that there is no single human population that can serve as a normative baseline for understanding

human

development

2001a;

(see

also

Miller,

Shweder & Sullivan, 1 9 9 3 ) . In working to gain an understanding o f cultural

sensibilities

that

differ

from

the

researcher's own background, it is important to seek cultural knowledge that, as far as is feasible, is nuanced and specific to the particular group under consideration. This implies that researchers should avoid turning to the widely utilized scale measures o f individualism/collectivism to provide this type of insight, because o f the limited cultural sensitivity o f such measures (Miller, 2 0 0 2 ) . Fortunately, whereas some commitment is required on the

As Y a n g suggests, there is a sense in which

part o f the researcher to make the necessary

culturally specific themes influence all phases

effort to acquire a greater understanding o f

of the research process, often unintentionally

other cultural viewpoints, many strategies are

excluding certain other cultural sensibilities.

available for achieving this goal.

T h e present considerations highlight the

One

valuable

importance, as part o f the initial phase o f any

knowledge

program

about

strategy

for

other cultures

obtaining involves

of

drawing from relevant research literature in

researchers working to enhance their under-

related fields, such as anthropology and socio-

standing both o f their own cultural back-

linguistics, work that in many instances may

grounds and o f those o f their

be ethnographic in nature. In the case o f my

of

psychological

research,

participant

populations and o f challenging the tendency

own research in India, for example, I was able

within psychology to privilege the perspec-

to develop insight into Hindu Indian culture

tives o f middle-class Euro-Americans. As

through

Reid ( 1 9 9 4 ) observed:

and philosophical literature on Hindu Indian

reading

available anthropological

beliefs, practices, values, and everyday family Culture has not so much been ignored in mainstream research as it has been assumed to be homogeneous, that is, based on a standard set of values and expectations primarily held by White and middle-class populations. The research literature across the subdisciplinary areas in psychology demonstrates clearly this assumption of cultural homogeneity. For example, in developmental psychology, children means White children

life. Notably, one can see the same kind o f stance as having informed the perspective adopted by M a r k u s and Kitayama ( 1 9 9 1 ) in their seminal article on culture and the self. Thus, although they proposed a global distinction linked to individualism/collectivism, the references cited in the article are grounded primarily in interdisciplinary research focused specifically on Japan.

Culturally Sensitive Research C o l l a b o r a t i n g with

a member

Questions

and

Methods

o f the

as expressed within a group context. Focus

comparison cultural community under con-

groups offer the advantage o f being highly

sideration represents another valuable strat-

flexible and can be employed effectively both

egy for gaining cultural knowledge, one that

to explore general cultural concerns and to tap

may be particularly useful in cases in which

respondents' open-ended reactions to issues

there is little or no available research litera-

identified as o f theoretical interest in a partic-

ture on a particular community. Ideally, such

ular research program.

collaborations should

include

researchers

w h o have both insider and outsider knowledge o f the cultures under

consideration

Sampling

(Greenfield, 1 9 9 7 a ) . Collaborations o f this

Attention needs to be given t o the cultural

type have been extremely generative in recent

implications o f different types o f sampling

cultural research in social psychology, as illus-

strategies. In this regard, effort should be

trated by the growing numbers o f studies

made to go beyond the present tendency for

being conducted involving researchers drawn

most social psychological research t o be con-

from the United States and from various East

ducted on convenience samples o f college

Asian cultural groups (e.g., J i &

Nisbett,

2 0 0 0 ; Peng & Nisbett, 1 9 9 9 ) .

students. In fact, the need to go beyond convenience samples has been emphasized in the

Greater cultural understanding also may be

National Institutes o f Health's recent man-

obtained through building into research pro-

date to address minority inclusion (or scien-

jects, as a prelude to formal data collection,

tifically justify exclusion) explicitly as part o f

activities and procedures that focus on gaining

all currently submitted grant proposals.

insight into the outlooks and practices o f particular cultural populations. This can entail spending time in such communities conduct-

Noncomparative

ing informal observations. F o r example, in the

Sampling

case o f my first series o f studies in India, I lived

"Prototypic"

Strategies

T h e prototypic sampling strategy in social

for several months in Mysore, India, prior to

psychology is noncomparative, with

initiating any formal data collection, as a

research experimentally manipulating situa-

means o f gaining insight into the

tional effects or assessing individual differ-

culture

such

through observing and participating in every-

ences, while tapping a population (generally

day life. In cases in which it is not feasible to

college students) that is treated as though it is

undertake informal preliminary observations

homogeneous and can provide grounds for

of this type, focus group techniques provide a

making universal claims. In efforts to increase

highly valuable approach that may be utilized

the cultural sensitivity o f this type o f sampling

to gain cultural insight (Hughes & D u M o n t ,

practice, it is essential not only for researchers

1 9 9 3 ; Knodel, 1 9 9 3 ) . A form o f organized

to acknowledge potential limitations on the

small-group discussions, focus groups consti-

generality o f their findings from this type of

tute small groups that investigators assemble

design but also to give greater conceptual

and engage in processes o f informal group dis-

attention to the nature o f these limitations.

cussion, as a means o f tapping participants'

T h u s , qualifications on the generality o f

personal experiences and reactions to particu-

results should not be issued in a perfunctory

lar topics (Powell & Single, 1 9 9 6 , p. 4 9 9 ) . T h e

way. Rather, it is important for researchers to

goal o f focus groups is to make possible the

address in what specific

gathering of qualitative information regarding

be anticipated t o be culturally bound

the attitudes, beliefs, and feelings o f participants,

alternatively, the question o f for what

respects a claim may or,

specific

102

DESIGN AND ANALYSIS reasons it is likely to prove universal. In short,

motivated either by a concern with obtaining

serious attention needs to be given to the cul-

normative data or by the methodological

tural meaning o f research findings, even when

requirements o f particular research method-

employing sampling designs that are noncom-

ologies, such as ethnographic or case study

parative in nature and not explicitly focused

approaches.

on cultural questions.

Sampling o f single cultural populations is

Equally, greater effort must be paid to

increasingly being adopted in research as a

unplanned sources o f cultural heterogeneity

means o f working to expand the normative

that exist within particular research samples

baseline for psychological theory, with such

and that are commonly overlooked in the

efforts encouraged by major U.S. funding

default

stance o f treating populations

organizations, such as the National Science

though

they are culturally homogeneous.

Foundation (NSF) and National Institutes o f

Thus, whenever there are sufficiently large

Health (NIH), in their issuing o f specific calls

as

numbers o f participants in different cultural

for research with underrepresented minority

subgroups to make this feasible, effort should

populations. It is recognized that psychologi-

be made to conduct separate analyses o f

cal theory can effectively be made more cul-

effects within subgroups to observe empiri-

turally inclusive only when its descriptive base

cally whether similar results obtain in all cases.

is broadened to include information about

It is recommended that subgroups be analyzed

psychological functioning in diverse cultural

at levels that are linked with cultural traditions

samples. This type o f sampling approach, it

and that attend as well to issues o f socioeco-

may be noted, also is occurring through the

nomic status. It is important that analyses o f

increasing internationalization o f social psy-

this type be undertaken in ways that are sensi-

chology, with new journals, such as the Asian

tive to areas o f overlap and

Journal

intermixing

of Social

supporting

Psychology,

between subgroups. As theorists have empha-

work on exclusively Asian samples, even as

sized (Hermans & Kempen, 1 9 9 8 ; Phinney,

the journal also publishes comparative studies.

1 9 9 9 ) , cultures assume hybrid forms as a

Sampling o f single cultural

populations

result of the many interconnections and trans-

represents the strategy o f choice in ethno-

formations occurring between populations,

graphic or case study research, in which the

and thus it is problematic to conceptualize cul-

focus is on a single cultural setting, if not on

tures as discrete geographically defined enti-

a single population from that setting. T o illus-

ties. Nonetheless, taking group membership

trate, ethnographic w o r k conducted

into account provides a vehicle for giving

inner-city African American families is pro-

with

"voice" to the outlooks o f different communi-

viding

ties, perspectives that may be obscured in

accounts

stances that deny the possibility o f making

stresses experienced within such commun-

any distinctions between groups on cultural

ities

grounds (Jahoda, 1 9 8 6 ; Miller, 1 9 9 7 ) .

coping observed (e.g., Burton, Allison, &

2

highly

and

informative

o f the of

the

multiple

and

complex

patterns

Obeidallah, 1 9 9 5 ; Jarrett, 1 9 9 5 ) , Noncomparative

Strategies

Sampling

of noncomparative

of

whereas

recent ethnographic w o r k among urban street

Cultural

Sampling

in-depth

environmental

gangs is affording access to study populations and settings that generally remain untapped cultural

by questionnaire or survey approaches (e.g.,

populations also may be utilized effectively

Heath, 1996).

in research that is focused explicitly on cul-

graphic case study techniques are adopted

tural questions. These projects generally are

commonly in w o r k by sociocultural theorists

In another example, ethno-

Culturally Sensitive Research

Questions

and

Methods

(e.g., Cole, 1 9 9 6 ) in their examination o f h o w

1 9 9 1 ; Miller, 1 9 9 4 ) but also from the focus

use o f cultural tools or modes o f cultural

on maintaining good interpersonal relations that is more central in J a p a n (Shimizu, 2 0 0 1 ) .

social organization affect cognition.

Notably, these examples do not imply that distinct psychological theories need to be forComparative Sampling

Cultural

mulated for every cultural or

Strategies

subcultural

group (see arguments for generality in Miller, designs

2 0 0 1 b , 2 0 0 2 ) ; however, they caution against

are employed c o m m o n l y in research that

the tendency, which is reflected in the con-

tests the universality o f particular psychologi-

temporary widespread reliance on measures

Comparative cultural

sampling

cal effects or that examines cultural variation

of independent/interdependent self-construals,

in basic psychological constructs and theo-

to adopt comparative designs that gloss over

ries. In such work, it is important for sam-

this type o f significant variation.

pling decisions t o be culturally nuanced.

As emphasized in recent anthropological

In utilizing comparative studies to examine

work on culture (e.g., Shore, 1 9 9 6 ; Strauss 8 t

cultural influences on social psychological

Quinn, 1 9 9 7 ) , it also is important to give

phenomena, greater consideration must be

more attention to within-culture variation in

given to the distinctive nature o f cultural ori-

perspectives related to factors such as socio-

entations (e.g., Dien, 1 9 9 9 ; Harkness, Super,

economic status and even place. This implies

& van Tijen, 2 0 0 0 ) . Equally, greater atten-

adopting more fluid outlooks on

tion needs to be paid t o the overlap and het-

boundaries and avoiding the c o m m o n ten-

cultural

To

dency in psychology to identify cultures with

illustrate, cultural research is pointing to fun-

nation states or even larger units, as when

erogeneity

of

cultural

perspectives.

damental variation in psychological processes

speaking

that is subtler in form than is captured in the

American" cultures. Illustrating the informa-

of

"East

Asian"

or

"North

individualism/collectivist dichotomy. T h u s ,

tive nature o f such a stance, research has

for example, the concern with affection and

uncovered variation in individualism across

respect that R o b i n H a r w o o d , Nydia Irizarry,

different regions o f the United States (Plaut,

and I (Harwood, Miller, fit Irizarry, 1 9 9 5 )

M a r k u s , 8c Lachman, 2 0 0 2 ; Vandello 8c

have found to be central to the outlooks on

Cohen, 1 9 9 9 ) as well as documented qualita-

attachment

tive variation in forms o f individualism linked

emphasized

by Puerto

Rican

mothers differs not only from the focus on balancing

autonomy

emphasized assumed

among

within

and

to socioeconomic status (Kusserow, 1 9 9 9 ) .

connectedness

Euro-Americans

attachment

and

theory (e.g.,

Ainsworth, 1 9 7 8 ) but also from the concern

Representativeness Equivalence

in

and Sampling

with amae identified within Japanese popula-

Although it is important to address con-

tions (Rothbaum, Weisz, Pott, M i y a k e , 8c

cerns about the anticipated cultural general-

Morelli, 2 0 0 0 ; Yamaguchi, 2 0 0 1 ) . Equally,

ity o f results, it also must be recognized that

the voluntaristic outlook on

interpersonal

representative sampling is not an essential

morality that is assumed in Carol Gilligan's

feature o f culturally based research designs

morality o f caring model (Gilligan, 1 9 8 2 ) dif-

and, with the exception o f large-scale sur-

fers not only from the interpersonal moral

veys, it is rarely achieved in social psychol-

outlooks based on dharma

i

that tend to be

researchers w h o are sampling U.S. college

Buddhist populations (Huebner 8 t Garrod,

students need to tap a representative sample

among

Hindu

Indian

ogy. J u s t as there is n o expectation that

and

emphasized

104

DESIGN A N D ANALYSIS

of college students from across the nation,

valuable in two-group research designs, given

much less the world, there should be n o

the many uncontrolled sources o f variation

expectation that researchers w h o may be

that

m a y influence any effect

(Cook

&

and

Campbell, 1 9 7 9 ) . Use o f this type o f compara-

Japanese college students need t o tap popu-

tive sampling is illustrated in my early cross-

lations

all

cultural research on social attribution (Miller,

Americans, much less o f all Japanese. T h i s

1 9 8 4 ) . In that investigation, the central cross-

implies that reviewers should not utilize rep-

cultural comparisons involved middle-class

resentative sampling as a criterion in evaluat-

Hindu Indian and middle-class Euro-American

comparing

the

that

responses

are

of

U.S.

representative

of

ing culturally based psychological research

samples. However, to evaluate potential alter-

because such a standard would lead to all

native interpretations of the results, additional

such w o r k being appraised negatively, with

sampling was undertaken both of a lower-class

the exception o f large-scale survey designs.

Hindu Indian sample and of a Westernized

In lieu o f the criterion o f samples being rep-

middle-class Christian Anglo-Indian sample.

resentative, however, concern needs to be

The finding that no effects o f socioeconomic

given to achieving equivalence in the popula-

differences

tions tapped in comparative studies and in

Hindu subgroups provided evidence to suggest

individuals' responses to research stimuli.

that differences in wealth could not explain the

Given the skewing o f samples that can result,

attributional variation observed in the main

were observed within the

two

matching samples on preexisting background

U.S./India cross-cultural comparison. T h e find-

characteristics should be avoided or utilized

ing that Anglo-Indians displayed a pattern of

only to a minimal extent. Rather, it is prefer-

social

able, to the extent feasible, to identify naturally

between that observed among the middle-class

occurring samples that are as comparable as

Hindu Indian and middle-class U.S. samples

possible, in terms o f background characteris-

lent support to the claim that a tendency to

attribution

that

was

intermediate

tics salient in the particular study (Cole &c

emphasize personality factors in social attribu-

Means, 1 9 8 6 ) . T o control for possible con-

tion is related to Westernization.

founding preexisting group differences, use also may be made of such statistical control techniques as covariate analysis or the partial-

Culture as Process

ing out o f variance. T o illustrate, in one study

Within contemporary social psychology,

in which we assessed U.S. and Indian respon-

widespread use is made o f the scale measure

dents'

of independent/interdependent self-construals

moral

appraisals

of

hypothetical

research vignettes, we observed that the two

developed by Singelis ( 1 9 9 4 ) as well as o f

groups differed in their perceptions o f the com-

other measures o f individualism/collectivism

monness of the vignettes portrayed (Bersoff &c

developed by researchers in the tradition o f

Miller, 1 9 9 3 ) . T o control for this a priori dif-

cross-cultural psychology (see, e.g., Triandis,

ference, we utilized a regression procedure to

1 9 9 5 ) . Interest also is shown in priming as

partial out the variance predicated by partici-

a way o f simulating cultural effects under

pants' commonness ratings from their moral

experimentally controlled conditions

reasoning responses (Bersoff & Miller, 1 9 9 3 ) .

H o n g et al., 2 0 0 0 ; Oyserman et al., 2 0 0 2 ) .

(e.g.,

The inclusion of control samples in research

However, serious limitations exist in both o f

designs is a valuable strategy that may be

these strategies, leaving a need to adopt more

employed in efforts to rule out alternative

dynamic methodological approaches.

interpretations o f particular effects related to

As critics have noted (e.g., Miller, 2 0 0 2 ;

sampling—a technique that is particularly

Strauss, 2 0 0 0 ) , scale measures of individualism/

Culturally Sensitive Research

Questions

and

Methods

105

collectivism and o f independent/interdependent

understanding other cultural meanings t o

self-construals subsume cultural variation into

which the response is linked. Dispositional

two fundamental types, a stance that glosses

and situational inferences are generated in all

over variation in outlooks that exists between

cultural groups, with their display affected

and within different cultural communities.

by contextual factors. T h u s , when individu-

Furthermore, individual items on these scales

als m a k e a dispositional or situational infer-

tend to portray collectivist cultures in some-

ence in a priming task, this may be merely

what pejorative terms and to lack adequate

because the prime is serving as a contextual

construct validity. Such characteristics may be

manipulation and not because it represents a

seen in the inclusion o f items that portray the

manipulation o f cultural outlook per se.

self as subordinate to the group in collectivist

In

lieu o f utilizing scale measures

to

outlooks. As recent research has shown, how-

assess individualism/collectivism or priming

ever, the self may be experienced as satisfied

approaches to tap cultural processes, it is

4

5

and fulfilled, rather than as subordinated, in the

recommended

fulfillment of the types o f role expectations

process-oriented

that

adopt

researchers

approaches

to

culture

emphasized in various collectivist communities

(Greenfield, 1 9 9 7 a ) . This includes tapping

(e.g., Iyengar & Lepper, 1 9 9 9 ; Miller, in press-

m o r e directly the psychological processes

fa; Miller & Bersoff, 1 9 9 4 ) . Measures of indi-

that are implicated in particular culturally

vidualism/collectivism also are problematic in

variable psychological responses as well as

treating psychological processes as bearing a

assessing the everyday cultural routines and

one-to-one relationship to cultural outlooks, a

practices that support such responses.

stance that fails to recognize the extent to which

Methodological approaches that tap the

behavior is normatively based rather than

psychological processes underlying particular

reflective of individual attitudes or personality

effects include such strategies as assessing

(Shweder, 1 9 7 9 ; T a k a n o & Osaka, 1 9 9 9 ) .

online processing as well as

Given these many weaknesses, it is not surpris-

culturally variable patterns

ing that many results obtained utilizing individ-

relationships

ualism/collectivism scales are o f questionable

processing involves evaluating

validity (Matsumoto, 1 9 9 9 ; Oyserman etal.,

immediately as it is encountered and contrasts

2 0 0 2 ; T a k a n o & Osaka, 1 9 9 9 ) . For example,

with cognitive processing based on long-term

(Kitayama,

of

identifying functional

2002).

Online

information

whereas findings within the United States based

memory T h e use o f online processing to

on

explore cultural influences is illustrated in a

individualism/collectivism scales

show

Latinos as no higher in collectivism than Euro-

recent comparative study on the correspon-

Americans (Coon &

Kemmelmeier, 2 0 0 1 ) ,

dence bias, an attributional tendency in which

such a finding does not accord with the conclu-

an individual's dispositions are seen as corre-

sions stemming from research that does not rely

sponding to his or her behavior even when

on individualism/collectivism measures (e.g.,

the behavior is socially constrained (Miyamoto

Delgado-Gaitan, 1 9 9 4 ; Harwood et al., 1 9 9 5 ) .

& Kitayama, 2 0 0 2 ) . This investigation not

It also is problematic to utilize priming

only demonstrated that Japanese respondents

approaches to simulate cultural processes

are less vulnerable to this bias than are U.S.

and to measure individualism/collectivism.

respondents

As discussed elsewhere (Miller, 2 0 0 2 ) , it is

that this difference is linked to contrasting

but also importantly

showed

n o t possible to interpret a particular behav-

types o f online attitudinal inferences. Thus, it

ioral response, such as a dispositional infer-

was demonstrated that, in contrast to the U.S.

ence, that might be primed as reflective o f an

respondents, the Japanese respondents were

individualistic or collectivist outlook without

more situationally focused in their online

106

DESIGN AND ANALYSIS inferences. In turn, the approach o f identifying

cultural understanding, this sensitivity is a

culturally variable patterns

functional

matter o f ensuring both that measures are

relationships is illustrated in cross-cultural

equivalent in meaning for different popula-

research highlighting the contrasting cultural

tions and that they are culturally informative.

of

it

T h e first issue represents a long-standing con-

has been demonstrated that whereas social

cern in cross-cultural psychology and bears

reticence tends to be linked to negative out-

fundamentally on issues o f reducing bias in

comes in family and school contexts within

comparative research designs (for extended

North

discussion o f these issues, see, e.g., Greenfield,

meanings

accorded to

shyness. T h u s ,

American cultural settings

(Kagan,

1 9 9 4 ) , it is linked to positive family and

1 9 9 7 a , 1 9 9 7 b ; van de Vijver, 2 0 0 1 ; van de

school outcomes within China (Chen, Rubin,

Vijver &c Leung, 1 9 9 7 ) . In turn, the second issue, which to date has received more limited

& Li, 1 9 9 5 ) . Greater effort also needs to be paid to assessing Greenfield,

cultural

practices

(see,

e.g.,

1 9 9 7 a ; M a r k u s , Mullally,

&

Kitayama, 1 9 9 7 ; Phinney & Landin, 1 9 9 8 ; Shweder,

Goodnow,

Hatano,

attention, bears on ensuring that the constructs tapped in psychological

measuring

instruments are sufficiently culturally inclusive to accommodate diverse outlooks.

LeVine,

In terms of ensuring the equivalence of

Markus, & Miller, 1 9 9 8 ) . T h e value of this

measuring instruments in different cultural or

type o f approach

is illustrated

in recent

subcultural

populations,

it is critical not

research by Evans ( 2 0 0 1 ) which showed that

merely to adopt such conventional strategies

differences

as the use o f back translation but also to take

in the receptivity to creationist

beliefs among fundamentalist

vs. nonfunda-

into account the contrasting expectations,

mentalist U.S. Christian families could be

social knowledge, values, and modes o f com-

explained, in part, by the families' everyday

munication

social practices such as having books

on

different sociocultural backgrounds. T o illus-

dinosaurs in their homes and attending church

trate, certain populations may be unfamiliar

regularly. Likewise, in a different example, it

with the convention that psychological tests

maintained

by individuals

of

has been by focusing on differences in everyday

are not designed to measure socially useful

social practices in schools and homes, such as

information and thus may respond to an I Q -

time spent on academic tasks and styles of

type measure with an answer that is pragmat-

teaching, that Stevenson and his colleagues

ically useful but that is scored as incorrect

have been able to identify the cultural pro-

according to the norms

cesses that underlie the dramatic differences in

Greenfield, 1 9 9 7 b ) . F o r example, village pop-

mathematics achievement that distinguish U.S.

ulations have been observed to respond spon-

from

taneously in object-sorting cognitive tasks by

Chinese and Japanese schoolchildren

(Stevenson &

Lee, 1 9 9 0 ; Stigler, Lee,

&

of the test (e.g.,

grouping items into functionally meaningful

Stevenson, 1 9 8 7 ) . (For work utilizing situation

pairings (e.g., grouping a knife and potato

sampling techniques to assess cultural prac-

together because the knife is used to cut the

tices, see, e.g., Kitayama, Markus, Matsumoto,

potato) rather than into the taxonomic group-

& Norasakkunkit, 1 9 9 7 . )

ings expected by the researchers (e.g., grouping all implement items together, all food

Culturally Appropriate

Measures

Finally, it is critical that the procedures

items together) (Cole, Gay, Glick, & Sharp, 1 9 7 1 ) . Interestingly, this type o f difference can lead, in certain cases, to various populations

in social psychological

experiencing difficulty in responding to multiple-

research be culturally sensitive. Presupposing

choice questions. Thus, in research among the

that

are adopted

Culturally Sensitive Research

Questions

and

107

Methods

Zinacantecan M a y a , Greenfield and Childs

specific assumptions and need to be broadened

( 1 9 7 7 ) observed that respondents with limited

conceptually to a c c o m m o d a t e the

diverse

schooling treated the multiple options pro-

outlooks o f contrasting cultural and subcul-

vided in multiple-choice questionnaires as pat-

tural populations. Until this is done, the field

terns to be put together to create a larger

will continue to yield results that, while iden-

meaning, rather than

tifying apparent

as discrete

options

universals, are based

on

whose only function is to test understanding.

methods that lack sufficient cultural sensitivity

T h e social context o f the test situation also

to succeed in tapping the cultural variability

may affect the level o f comfort that individu-

that exists. It is this property o f present psy-

als experience in testing situations and their

chological research methodology, in fact, that

readiness to respond. Thus, for example,

leads psychological research to form some-

Mexican-immigrant parents within the United

what o f a closed system, in which it becomes

States spontaneously use questioning less fre-

difficult to produce findings that challenge the

quently as a conversational strategy at home

explanatory scope o f existing theoretical mod-

than do Euro-American parents, a cultural dif-

els and in which results on diverse psychologi-

ference that is reflected in the former being

cal measures tend to be highly intercorrelated

more reluctant to answer questions in stan-

(Miller, in press-a). Thus, for example, it was

dard interviewing situations (Delgado-Gaitan,

only when researchers developed new concep-

1 9 9 4 ; Greenfield, 1 9 9 7 b ) .

tual models for understanding morality, such

Notably, in working to ensure the cross-

as in Gilligan's ( 1 9 8 2 ) morality o f caring

group appropriateness o f measures, equiva-

framework and in various cultural approaches

lence needs to be achieved at the level o f

(e.g., Miller, 1 9 9 4 ; Snarey, 1 9 8 5 ) , as well as

meaning, a feature that may require utilizing

provided methodologies that were sensitive

somewhat different objective procedures in

enough to tap this variation, that the conclu-

(1997b)

sion o f the universality o f the Kohlbergian

different

groups.

As Greenfield

observed, "the use o f parallel

procedures

across cultures . . . works best when cultures are not t o o different. . . the use o f

model of moral development was challenged effectively.

qualita­

T h e present considerations highlight the

across cultures

need for researchers to be m o r e aware o f the

works best when the cultures are very differ-

extent to which the response options pro-

ent" (p. 3 0 8 ) . T o illustrate use o f this type o f

vided on standard questionnaires or coding

strategy, in my early cross-cultural attribution

schemes may lack sufficient cultural sensitiv-

research (Miller, 1 9 8 4 ) , my decision to have

ity t o succeed in tapping the outlooks o f

tively

different

procedures

individuals explain events from their own

diverse

experiences, rather than to respond to identi-

example, in the scales utilized in research on

cal experimentally constructed event situa-

self-determination theory (e.g., Deci & Ryan,

cultural

populations.

Thus,

for

tions that I supplied to them, was motivated

1 9 8 7 ) , the " e x t e r n a l " motivational orienta-

by a sense that greater equivalence in meaning

tion is conceptualized as a stance involving

could be obtained in this way, since the

the fear o f external sanctions, as reflected in

behaviors being explained would have greater

items such as "Because I will get in trouble if

ecological validity for all cultural and age

I don't do well," whereas the "identified"

groups.

and "intrinsic" motivational stances are con-

In turn, to ensure the cultural inclusiveness o f research methods, it is critically important

ceptualized as involving autonomous

indi-

vidual interest, as reflected in items such as

to recognize that many assessment instru-

"Because I w a n t t o understand the subject"

ments currently in use embody

and "Because it's important to me t o do

culturally

DESIGN AND ANALYSIS

108

my h o m e w o r k . " These types o f response alternatives, however, do not capture

CONCLUSIONS

the

view o f social e x p e c t a t i o n s

In conclusion, bringing culture more centrally

emphasized in a culture such as Hindu India,

into the methods o f social psychology is inte-

endogenous

in which the motive to uphold duty relates t o

grally related to bringing culture more cen-

spiritual fulfillment, not fear o f sanctions or

trally into the constructs and theories o f the

mere social conformity (Miller, in press-b).

field. As has been argued here, the relative

T o give another example, the emphasis on

invisibility o f culture in social psychology, and

training

among

in psychology more generally, stems in part

Chinese Americans, as C h a o ( 1 9 9 4 ) points

from the limited attention that we give it in

(chiao

observed

shun)

out, includes an emphasis on positive affect

our theories, as well as from our adoption o f

in conjunction with highly directive parental

methods that are insufficiently sensitive to the

behavior. It then is not accommodated in the

impact o f cultural processes on psychological

theoretical framework

o f parenting

devel-

phenomena. As M a t s u m o t o ( 2 0 0 1 ) recently

oped by Baumrind, which presents a scheme

commented,

for

"all psychologists are cross-

parental

cultural in some way; the only difference is in

behavior into alternatives that link parenting

whether they are aware o f the cultures being

either to an affectively harsh stance ("author-

studied,

itarian" parenting) or to stances that are

explicit or implicit in their w o r k " (p. ix).

conceptualizing and

coding

much less directive (i.e., either

"authorita-

tive" or "permissive" parenting). o f the

this comparison

is

The effort to make social psychology more culturally inclusive must build on the com-

T o address the issue o f the insufficient culturally inclusive nature

and whether

constructs

plexity and sophistication o f the discipline, with the onus

on cultural

researchers to

tapped in many existing psychological mea-

develop approaches to culture that, in their

suring instruments, the constructs embodied

attention to the nuances o f cultural outlooks

in our methods need to be expanded. Thus, to

and to the contextual dependence and often

give an example, cultural researchers have

implicit nature o f psychological phenomena,

argued for including the construct o f relation-

embody the rich insights o f contemporary

ship harmony and not only the construct o f

social psychology. Equally critical, however, is

self-esteem in tapping the predictors o f life

the need to overcome the complacency of

satisfaction (Kwan, Bond, & Singelis, 1 9 9 7 ) .

social psychology, which has resulted in rele-

It is also valuable t o utilize assessment instru-

gating culture to a peripheral role as a mere

ments that are less constraining o f response

descriptive enterprise with little implication

options and more accommodating t o diverse

for basic theory. As has been shown, the con-

cultural viewpoints. T h u s , in my own pro-

ceptual

grams of research, for example, I have tended

methodological approaches that constitute the

to rely heavily on methodological approaches

mainstream perspective o f the discipline have,

stances, sampling

practices,

and

that are less directive than standardized ques-

in many cases, obscured significant cultural

tionnaires, such as tapping responses to the

variation, yielding findings o f universality that

projective measure o f hypothetical vignette

may be more apparent than real.

situations (e.g., Miller & Bersoff, 1 9 9 8 ) and

Notably, taking cultural

considerations

utilizing open-ended questioning to explore

into account more centrally in social psychol-

individuals'

ogy promises to yield a richer understanding

reasoning

(Miller

&

Bersoff,

1 9 9 5 ) (see also King, Chapter 8, this volume;

of basic psychological processes and o f the

Peng, Nisbett, & W o n g , 1 9 9 7 ) .

diversity o f outlooks that characterize human

Culturally Sensitive Research psychological functioning.

Such an

Questions

and Methods

\

effort,

produce a discipline that is not only more truly

which needs to be integrated with efforts to

universal but also more theoretically sophisti-

identify brain and other biological founda-

cated in its process accounts o f psychological

tions for psychological behavior, stands to

phenomena and in its applied implications.

NOTES 1. Experienced in the context of close relationships that entail both attachment and dependence, the Japanese concept of amae involves feelings of being able "to depend and presume upon another's love or bask in another's indulgence" (Doi, 1992, p. 8). Individuals experience amae in close relationships in being able to presume that their inappropriate behavior will be accepted by their counterpart (Yamaguchi, 2 0 0 1 ) . 2. Within social psychology, stances that deny the possibility of distinguishing between cultural traditions have been adopted by theorists associated with such postmodern perspectives as social constructionism and discursive psychology (e.g., Edwards, 1 9 9 5 ; Gergen, 1 9 9 2 , 1994; Shorter, 1993). As Gergen commented: We are not speaking . . . of the blending of all, the emergence of monoculture, but rapid and continuous transformations in cultural forms, as they are subject to multiple influences. . . . If there is a continuous blending, appropriation, dissolution, and the like, how are we to draw distinctions among cultural processes? (Gergen, as interviewed in Gulerce, 1995, pp. 149-150) 3. The concept of dharma denotes both moral duty and inherent character and is based on perceived spiritually based laws of nature (Marriott, 1990). 4. This type of assumption can be seen, for example, in the following items that appear on the widely used Singelis (1994) measure of independent vs. interdependent self construals: "I will sacrifice my self-interest for the group that I am in" and "I will stay in a group if they need me, even when I'm not happy with the group." 5. The present recommendation applies only to the use of priming for purposes of simulating cultural effects. There are many other important purposes for which it is appropriate to use priming in culturally based research that assesses cognitive processing.

REFERENCES Ainsworth, M . D. (1978). Patterns of attachment: A psychological study of the strange situation. Hillsdale, NJ: Lawrence Erlbaum. Baldwin, A. L., Baldwin, C., & Cole, R. E. (1990). Stress-resistant families and stress-resistant children. In J . Rolf, A. Masten, D. Cicchetti, K. Neuchtherlin, & S. Weintraub (Eds.), Risk and protective factors in the development of psychopathology (pp. 2 5 7 - 2 8 0 ) . Cambridge, UK: Cambridge University Press. Bargh, J . A. (1996). Automaticity in social psychology. In E. T. Higgins & A. W. Kruglanski (Eds.), Social psychology: Handbook of basic principles (pp. 169-183). New York: Guilford.

109

110

[

DESIGN AND ANALYSIS Bartlett, F. C. (1932). Remembering: An experimental and social study. Cambridge, UK: Cambridge University Press. Relations: Baumrind, D. (1996). The discipline controversy revisited. Family Journal of Applied Family and Child Studies, 45(4), 4 0 5 - 4 1 4 . Berry, J . W. (1976). Human ecology and cognitive style. New York: Sage-Halsted. Bersoff, D. M., & Miller, J . G. (1993). Culture, context, and the development of Psychology, 29(4), 664-676. moral accountability judgments. Developmental Bornstein, R. F., Kale, A. R., 8c Cornell, K. R. (1990). Boredom as a limiting conand Social dition on the mere exposure effect. Journal of Personality Psychology, 42, 239-247. Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press. Brown, D. E. (1991). Human universals. Philadelphia: Temple University Press. Burton, L. M., Allison, K., & Obeidallah, D. (1995). Social context and adolescence: Perspectives on development among inner-city African-American teens. In L. Crockett 8c A. C. Crouter (Eds.), Pathways through adolescence: Individual development in relation to social context (pp. 118-138). Hillsdale, NJ: Lawrence Erlbaum. Chao, R. K. (1994). Beyond parental control and authoritarian parenting style: Understanding Chinese parenting through the cultural notion of training. Child Development, 65(4), 1111-1119. Chen, X . , Rubin, K. H., 8c Li, B. (1995). Social and school adjustment of shy and 7(2), 337-349. aggressive children in China. Development and Psychopathology, Cole, M . (1995). Culture and cognitive development: From cross-cultural research to creating systems of cultural mediation. Culture and Psychology, I, 25-54. Cole, M. (1996). Cultural psychology: A once and future discipline. Cambridge, MA: Harvard University Press. Cole, M., Gay, J . , Glick, J . , & Sharp, D. W. (1971). The cultural context of learn­ ing and thinking. New York: Basic Books. Cole, M., 8c Means, B. (1986). Comparative studies of how people think: An intro­ duction. Cambridge, MA: Harvard University Press. Cook, T. D., 8c Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Coon, H. M., 8c Kemmelmeier, M . (2001). Cultural orientations in the United States: (Re)Examining differences among ethnic groups. Journal of CrossCultural Psychology, 32(3), 348-364. Cousins, S. D. (1989). Culture and self-perception in Japan and the United States. Journal of Personality and Social Psychology, 56, 1 2 4 - 1 3 1 . Damon, W., 8c Hart, D. (1982). The development of self-understanding from infancy through adolescence. Child Development, 53, 3 2 3 - 3 3 8 . D'Andrade, R. G. (1984). Cultural meaning systems. In R. A. Shweder 8c R. A. LeVine (Eds.), Culture theory: Essays on mind, self, and emotion (pp. 88-119). New York: Cambridge University Press. Deci, E. L., 8c Ryan, R. M . (1985). Intrinsic motivation and self-determination in human behavior. New York: Plenum. Deci, E. L., 8c Ryan, R. M. (1987). The support of autonomy and the control of behavior. Journal of Personality and Social Psychology, 53(6), 1024-1037. Deci, E. I., &C Ryan, R. M. (1990). A motivational approach to self: Integration in personality. In R. A. Dienstbier (Ed.), Nebraska Symposium on Motivation: Vol. 38: Perspectives on motivation (pp. 237-287). Lincoln: University of Nebraska Press. Deci, E. L., Ryan, R. M., Gagne, M., Leone, D. R., Usunov, J . , &c Kornazheva, B. P. (2001). Need satisfaction, motivation, and well-being in the work organizaPsychology tions of a former Eastern Bloc country. Personality and Social Bulletin, 27(8), 930-942.

Culturally Sensitive

Research

Questions

and

Delgado-Gaitan, C. (1994). Socializing young children in Mexican-American families: An intergenerational perspective. In P. M . Greenfield 8c R. R. Cocking (Eds.), Cross-cultural roots of minority child development (pp. 5 5 - 8 6 ) . Hillsdale, NJ: Lawrence Erlbaum. Dien, D. S. (1999). Chinese authority-directed orientation and Japanese peer-group of General orientation: Questioning the notion of collectivism. Review Psychology, 3(4), 3 7 2 - 3 8 5 . Doi, T. (1992). On the concept of amae. Infant Mental Health Journal, 13, 7 - 1 1 . Edwards, D. (1995). A commentary on discursive and cultural psychology. Culture and Psychology, 1, 55-66. Ekman, P. (1992). Are there basic emotions? Psychological Review, 99, 550-553. Evans, E. M . (2001). Cognitive and contextual factors in the emergence of diverse belief systems: Creation vs. evolution. Cognitive Psychology, 42(3), 2 1 7 - 2 6 6 . Fiske, A. P., Kitayama, S., Markus, H. R., 8c Nisbett, R. E. (1998). The cultural of social psychology (4th ed., matrix of social psychology. In The handbook Vol. 2, pp. 9 1 5 - 9 8 1 ) . Boston: McGraw-Hill. Psychologist, Gergen, K. J . (1992). Psychology in the postmodern era. The General 28, 10-15. Gergen, K. J . (1994). Realities and relationships: Soundings in social construction. Cambridge, MA: Harvard University Press. Gilbert, D. T., Fiske, S. T., 8c Lindzey, G. (Eds.). (1998). The handbook of social psychology (4th ed.). New York: McGraw-Hill. Gilligan, C. (1982). In a different voice: Psychological theory and women's devel­ opment. Cambridge, MA: Harvard University Press. Greenfield, P. M . (1997a). Culture as process: Empirical methods for cultural psychology. In J . W. Berry, Y . H. Poortinga, 8c J . Pandey (Eds.), Handbook of cross-cultural psychology: Vol. 1. Theory and method (2nd ed., pp. 3 0 1 - 3 4 6 ) . Boston: Allyn 8c Bacon. Greenfield, P. M . (1997b). You can't take it with you: Why ability assessments don't cross cultures. American Psychologist, 5 2 ( 1 0 ) , 1115-1124. Greenfield, P. M., 8c Childs, C. P. (1977). Understanding sibling concepts: A developmental study of kin terms in Zincantan. In P. Dasen (Ed.), Piagetian psy­ chology: Cross-cultural contributions (pp. 3 3 5 - 3 3 8 ) . New York: Gardner. Grolnick, W. S., Deci, E. L., 8c Ryan, R. M . (1997). Internalization within the family: The self-determination theory perspective. In J . E. Grusec 8c L. Kuczynski (Eds.), Handbook of parenting and the transmission of values (pp. 135-161). New York: Wiley. Grolnick, W. S., 8c Ryan, R. M. (1989). Parent styles associated with children's selfPsychology, regulation and competence in school, journal of Educational 81(2), 143-154. Gulerce, A. (1995). Culture and self in postmodern psychology: Dialogue in trouble? [Interview with K.J. Gergen). Culture and Psychology, 1, 147-159. Haney, C , Banks, C , 8c Zimbardo, P. (1973). Interpersonal dynamics in a simulated prison. International journal of Criminology and Penology, 1, 69-97. Harkness, S., Super, C. M., 8c van Tijen, N. (2000). Individualism and the "Western mind" reconsidered: American and Dutch parents' ethnotheories of the child. of the child (New Directions for Child In Variability in the social construction and Adolescent Development No. 87) (pp. 23-39). San Francisco: Jossey-Bass. Harwood, R. L., Miller, J . G., 8c Irizarry, N. L. (1995). Culture and attachment: Perceptions of the child in context. New York: Guilford. Heath, S. B. (1996). Ruling places: Adaptation in development by inner-city youth. In R. Jessor, A. Colby, 8c R. A. Shweder (Eds.), Ethnography and human devel­ opment: Context and meaning in social inquiry (pp. 225-251). Chicago: University of Chicago Press.

Methods

112

DESIGN AND ANALYSIS Hermans, H., & Kempen, H. (1998). Moving cultures: The perilous problem of Psychologist, 53, cultural dichotomies in a globalizing society. American 1111-1120. Higgins, E. T., &c Kruglanski, A. W. (1996). Social psychology: Handbook of basic principles. New York: Guilford. Hong, Y . - Y . , Morris, M . W., Chiu, C.-Y., & Benet-Martinez, V. (2000). Multicultural minds: A dynamic constructivist approach to culture and cognition. American Psychologist, 55(7), 709-720. Huebner, A., & Garrod, A. (1991). Moral reasoning in a Karmic world. Human Development, 34, 341-352. Hughes, D., & DuMont, K. (1993). Using focus groups to facilitate culturally Journal of Community Psychology, 21(6), anchored research. American 775-806. Iyengar, S. S., & Lepper, M . R. (1999). Rethinking the value of choice: A cultural of Personality and Social perspective on intrinsic motivation. Journal Psychology, 76(3), 349-366. Izard, C. E. (1992). Basic emotions, relations among emotions, and emotioncognition relations. Psychological Review, 99, 5 6 1 - 5 6 5 . Jahoda, G. (1986). Nature, culture and social psychology. European Journal of Social Psychology, 16, 17-30. Jarrett, R. L. (1995). Growing up poor: The family experiences of socially mobile youth in low-income African-American neighborhoods. Journal of Adolescent Research, 10, 111-135. Ji, L.-J., Peng, K., & Nisbett, R. E. (2000). Culture, control, and perception of relaPsychology, tionships in the environment. Journal of Personality and Social 78(5), 9 4 3 - 9 5 5 . Kagan, J . (1994). Galen's prophecy: Temperament in human nature. New York: Basic Books. Kitayama, S. (2002). Culture and basic psychological processes—Toward a system Bulletin, view of culture: Comment on Oyserman et al. (2002). Psychological 128(1), 89-96. Kitayama, S., Markus, H. R., Matsumoto, H., & Norasakkunkit, V. (1997). Individual and collective processes in the construction of the self: Self-enhancement in the United States and self-criticism in Japan. Journal of Personality and Social Psychology, 72(6), 1245-1267. Knodel, J . (1993). The design and analysis of focus group studies: A practical approach. In D. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 35-50). Newbury Park, CA: Sage. Kohlberg, L. (1984). The psychology of moral development: The nature and valid­ ity of moral stages. San Francisco: Harper & Row. Kusserow, A. S. (1999). De-homogenizing American individualism: Socializing hard and soft individualism in Manhattan and Queens. Ethos, 27(2), 2 1 0 - 2 3 4 . Kwan, V.S.Y., Bond, M . H., & Singelis, T. M . (1997). Pancultural explanations for life satisfaction: Adding relationship harmony to self esteem. Journal of Personality and Social Psychology, 73(5), 1 0 3 8 - 1 0 5 1 . Laboratory of Comparative Human Cognition. (1983). Culture and cognitive develof child psychology: History, theory opment. In W. Kessen (Ed.), Handbook and method (pp. 2 9 5 - 3 5 6 ) . New York: Wiley. Levine, C , Kohlberg, L., & Hewer, A. (1985). The current formulation of Kohlberg's 28(2), 94-100. theory and a response to critics. Human Development, LeVine, R. A. (1984). Properties of culture: An ethnographic view. In R. A. Shweder & R. A. LeVine (Eds.), Culture theory: Essays on mind, self, and emotion (pp. 67-87). New York: Cambridge University Press.

Culturally Sensitive

Research

Questions

and

Methods

Livesley, W. J . , 8c Bromley, D. B. (1973). Person perception in childhood and adolescence. London: Wiley. Malpass, R. S. (1988). Why not cross-cultural psychology?: A characterization of some mainstream views. In M. H. Bond (Ed.), The cross-cultural challenge to social psychology (Cross-Cultural Research and Methodology Series) (pp. 2 9 - 3 5 ) . Beverly Hills, CA: Sage. Markus, H. R., 8c Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 2 2 4 - 2 5 3 . Markus, H. R., Kitayama, S., 8c Heiman, R. J . (1996). Culture and "basic" psychological principles. In E. T. Higgins 8c A. W. Kruglanski (Eds.), Social psychology: Handbook of basic principles (pp. 857-913). New York: Guilford. Markus, H. R., Mullally, P. R., 8c Kitayama, S. (1997). Selfways: Diversity in modes of cultural participation. In The conceptual self in context: Culture, experience, self-understanding (pp. 13-61). New York: Cambridge University Press. Marriott, M. (1990). India through Hindu categories. New Delhi: Sage. Matsumoto, D. (1999). Culture and self: An empirical assessment of Markus and Kitayama's theory of independent and interdependent self-construals. Asian Journal of Social Psychology, 2, 2 8 9 - 3 1 0 . Matsumoto, D. (Ed.). (2001). The handbook of culture and psychology. New York: Oxford University Press. McLoyd, V. C , 8c Flanagan, C. A. (Eds.). (1990). Economic stress: Effects on family life and child development. San Francisco: Jossey-Bass. Milgram, S. (1963). The behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 4 6 7 - 4 7 2 . Miller, J . G. (1984). Culture and the development of everyday social explanation. Journal of Personality and Social Psychology, 46(5), 9 6 1 - 9 7 8 . Miller, J . G. (1994). Cultural diversity in the morality of caring: Individually oriented versus duty-based interpersonal moral codes. Cross-Cultural Research: The Journal of Comparative Social Science, 28(1), 3-39. Miller, J . G. (1997). Theoretical issues in cultural psychology. In J . W. Berry, Y . H. Poortinga, 8c J . Pandey (Eds.), Handbook of cross-cultural psychology: Vol. 1. Theory and method (2nd ed., pp. 85-128). Boston: Allyn 8c Bacon. Miller, J . G. (1999). Cultural psychology: Implications for basic psychological theory. Psychological Science, 10(2), 8 5 - 9 1 . Miller, J . (2001a). The cultural grounding of social psychological theory. In A. Tesser 8c N. Schwarz (Eds.), Blackwell handbook of social psychology:

Vol. 1. Intrapersonalprocesses (pp. 2 2 - 4 3 ) . Maiden, MA: Blackwell. Miller, J . G. (2001b). Culture and moral development. In D. Matsumoto (Ed.), The

handbook of culture and psychology (pp. 151-169). New York: Oxford University Press. Miller, J . G. (2002). Bringing culture to basic psychological theory: Beyond individualism and collectivism: Comment on Oyserman et al. (2002). Psychological

Bulletin, 128(1), 97-109. Miller, J . G. (in press-a). The cultural deep structure of psychological theories of social development. In R. Sternberg 8c E. Grigorenko (Eds.), Culture and com­

petence. Washington, DC: American Psychological Association. Miller, J . G. (in press-b). Culture and agency: Implications for psychological theories of motivation and social development. In V. Murphy-Berman 8c J . Berman (Eds.), Nebraska Symposium on Motivation: Vol. 49. Cross-cultural differences

in perspectives on the self. Lincoln: University of Nebraska Press. Miller, J . G., 8c Bersoff, D. M . (1994). Cultural influences on the moral status of reciprocity and the discounting of endogenous motivation. Personality and

Social Psychology Bulletin, 20(5), 5 9 2 - 6 0 2 .

113

114

DESIGN AND ANALYSIS Miller, J . G., & Bersoff, D. M . (1995). Development in the context of everyday family relationships: Culture, interpersonal morality, and adaptation. In M. Killen &c D. Hart (Eds.), Morality in everyday life: Developmental perspec­ tives (pp. 2 5 9 - 2 8 2 ) . New York: Cambridge University Press. Miller, J . G., & Bersoff, D. M. (1998). The role of liking in perceptions of the moral Social responsibility to help: A cultural perspective. Journal of Experimental Psychology, 34(5), 4 4 3 - 4 6 9 . Miyamoto, Y . , & Kitayama, S. (2002). Cultural variation in correspondence bias: The critical role of attitude diagnosticity of socially constrained behavior. Journal of Personality and Social Psychology, 83(5), 1239-1248. Moscovici, S. (1972). Society and theory in social psychology. In J . Israel & H. Tajfel (Eds.), The context of social psychology: A critical assessment (pp. 17-69). New York: Academic Press. Neighbors, H. W., 8c Jackson, J . S. (Eds.). (1996). Mental health in Black America. Thousand Oaks, CA: Sage. Nisbett, R., 8c Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice-Hall. Oyserman, D., Coon, H., & Kemmelmeier, M . (2002). Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analyses. Psychological Bulletin, 128(1), 3-72. Peng, K., & Nisbett, R. E. (1999). Culture, dialectics, and reasoning about contradiction. American Psychologist, 54(9), 741-754. Peng, K., Nisbett, R. E., & Wong, N. Y . C. (1997). Validity problems comparing Methods, 2(4), values across cultures and possible solutions. Psychological 329-344. Phinney, J . S. (1999). An intercultural approach in psychology: Cultural contact and Psychology Bulletin, 33(2), 2 4 - 3 1 . identity. Cross-Cultural Phinney, J . S., & Landin, J . (1998). Research paradigms for studying ethnic minority families within and across groups. In V. C. McLoyd (Ed.), Studying minor­ ity adolescents: Conceptual, methodological, and theoretical issues (pp. 89-109). Mahwah, NJ: Lawrence Erlbaum. Plaut, V. C , Markus, H. R., & Lachman, M. E. (2002). Place matters: Consensual features and regional variation in American well-being and self. Journal of Personality and Social Psychology, S 3 ( l ) , 160-184. Powell, R. A., 8c Single, H. M . (1996). Focus groups. International Journal of Quality in Health Care, 8(5), 4 9 9 - 5 0 4 . Reid, P. T. (1994). The real problem in the study of culture. American Psychologist, 49(6), 5 2 4 - 5 2 5 . Rothbaum, F., Weisz, J . , Pott, M., Miyake, K., & Morelli, G. (2000). Attachment Psychologist, and culture: Security in the United States and Japan. American 55(10), 1 0 9 3 - 1 1 0 4 . Russell, J . A. (1991). Culture and the categorization of emotions. Psychological Bulletin, 110, 4 2 6 - 4 5 0 . Russell, J . A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102-141. Shimizu, H. (2001). Japanese adolescent boys' senses of empathy (omoiyari) and Carol Gilligan's perspectives on the morality of care: A phenomenological approach. Culture and Psychology, 7(4), 4 5 3 - 4 7 5 . Shore, B. (1996). Culture in mind: Cognition, culture and the problem of meaning. New York: Oxford University Press. Shorter, J . (1993). Cultural politics of everyday life: Social constructionism, rhetoric and knowing of the third kind. Buckingham, UK: Open University Press.

Culturally Sensitive

Research

Questions

and

Shweder, R. A. (1979). Rethinking culture and personality theory Part I: A critical examination of two classical postulates. Ethos, 7(3), 2 5 5 - 2 7 8 . Shweder, R. A. (1984). Anthropology's romantic rebellion against the enlightenment, or there's more to thinking than reason and evidence. In R. A. Shweder 8c R. A. LeVine (Eds.), Culture theory: Essays on mind, self, and emotion (pp. 27-66). Cambridge, UK: Cambridge University Press. Shweder, R. A. (1990). Cultural psychology—What is it? In J . W. Stigler, R. A. Essays on comparative Shweder, 8c G. Herdt (Eds.), Cultural psychology: human development (pp. 27-66). New York: Cambridge University Press. Shweder, R. A., Goodnow, J . , Hatano, G., LeVine, R. A., Markus, H., 8c Miller, P. (1998). The cultural psychology of development: One mind, many mentalities. of child psychology (Vol. 1, pp. 865-937). In W. Damon (Ed.), Handbook New York: John Wiley 8c Sons. Shweder, R. A., 8c Sullivan, M. A. (1993). Cultural psychology: Who needs it? Annual Review of Psychology, 44, 497-527. Singelis, T. M . (1994). The measurement of independent and interdependent selfconstruals. Personality and Social Psychology Bulletin, 20(5), 5 8 0 - 5 9 1 . Snarey, J . R. (1985). Cross-cultural universality of social-moral development: A critical review of Kohlbergian research. Psychological Bulletin, 97(2), 2 0 2 - 2 3 2 . Steele, C. M., 8c Aronson, J . (1995). Stereotype threat and the intellectual test perPsychology, formance of African Americans. Journal of Personality and Social 69(5), 7 9 7 - 8 1 1 . Stevenson, H. W., 8c Lee, S.-Y. (1990). Contexts of achievement: A study of of the Society for American, Chinese, and Japanese children. In Monographs Research in Child Development (Serial No. 2 2 1 , vol. 5 5 , Nos. 1-2). Chicago: University of Chicago Press. Stigler, J . W., Lee, S.-Y., Sc Stevenson, H. W. (1987). Mathematics classrooms in 58(5), 1 2 7 2 - 1 2 8 5 . Japan, Taiwan, and the United States. Child Development, Strauss, C. (2000). The culture concept and the individualism-collectivism debate: Dominant and alternative attributions for class in the United States. In Culture, thought, and development (pp. 85-114). Mahwah, NJ: Lawrence Erlbaum Associates. Strauss, C , 8c Quinn, N. (1997). A cognitive theory of cultural meaning. New York: Cambridge University Press. Takano, Y., 8c Osaka, E. (1999). An unsupported common view: Comparing Japan and the U.S. on individualism/collectivism. Asian Journal of Social Psychology, 2, 3 1 1 - 3 4 1 . Tobin, J . J . , Wu, D. Y . H., 8c Davidson, D. H. (1989). Preschool in three cultures: Japan, China, and the United States. New Haven, CT: Yale University Press. and collectivism. Boulder, CO: Westview. Triandis, H. C. (1995). Individualism van de Vijver, F. (2001). The evolution of cross-cultural research methods. In D. Matsumoto (Ed.), The handbook of culture and psychology (pp. 79-97). New York: Oxford University Press. van de Vijver, F. J . R., 8c Leung, K. (1997). Methods and data analysis of comparHandbook ative research. In J . W. Berry, Y . H. Poortinga, 8c J . Pandey (Eds.), of cross-cultural psychology (pp. 2 5 7 - 3 0 0 ) . Boston: Allyn 8c Bacon. Vandello, J . A., 8c Cohen, D. (1999). Patterns of individualism and collectivism across the United States. Journal of Personality and Social Psychology, 77(2), 279-292. Whiting, B. B., 8c Whiting, J . W. (1975). Children of six cultures: A psycho-cultural analysis. Cambridge, MA: Harvard University Press. Wierzbicka, A. (2002). Right and wrong: From philosophy to everyday discourse. Discourse Studies, 4(2), 2 2 5 - 2 5 2 .

Methods

115

116

DESIGN AND ANALYSIS Witkin, H. A., &c Berry, J . W. (1975). Psychological differentiation in cross-cultural perspective. Journal of Cross-Cultural Psychology, 6, 4-87. Yamaguchi, S. (2001). Culture and control orientations. In D. Matsumoto (Ed.), The handbook of culture and psychology (pp. 2 2 3 - 2 4 3 ) . New York: Oxford University Press. Yang, K.-S. (1997). Indigenizing westernized Chinese psychology. In M . H. Bond (Ed.), Working at the interface of culture: Eighteen lives in social science (pp. 62-76). London: Routledge.

6

C H A P T E R

Individual Differences in Social Psychology Understanding Situations to Understand People, Understanding People to Understand Situations YUICHI SHODA University of Washington

[GJeneral

laws and individual

mutually

dependent

study of the

differences

on each other

are merely two aspects

of one problem;

and the study of the one cannot

proceed

they are

without

the

other. —Lewin ( 1 9 4 6 , p. 7 9 4 )

P

eople think, feel, and do different things

figure out

in different situations, and the changes

underlie the observed variations. T h i s hand-

what

processes and

mechanisms

and variations from situation t o situation

b o o k is full o f examples o f thoughtful and inge-

are not all random. Understanding the nature

nious ways to pursue that goal. This chapter

o f such variations, o f course, is one o f the main

will focus on providing a broad

missions o f social psychology. T h e goal is to

for

framework

the role o f individual differences in the

AUTHOR'S NOTE: Preparation of this chapter was supported in part by Grant M H 3 9 3 4 9 from the National Institute of Mental Health. I am grateful to the editors of this volume for their encouragement to embark on this chapter initially and for their extremely careful and thorough reading of multiple drafts throughout its development as well as their constructive suggestions. Kathy Cook, Scott LeeTiernan, Jason Plaks, Vivian Zayas, and Naomi Zavislak provided many opportunities for productive brainstorming sessions. In addition, Kathy Cook provided many of the historical materials cited in this chapter. I am deeply grateful for their support. Correspondence concerning this article should be addressed to Yuichi Shoda, Department of Psychology, University of Washington, Box 3 5 1 5 2 5 , Seattle, WA 9 8 1 9 5 - 1 5 2 5 . Electronic mail may be sent to [email protected].

117

118

DESIGN AND ANALYSIS investigation o f the effects of situations and

and naturally existing individual differences.

individuals' social information processing sys-

W h y might that be? This chapter approaches

tems that mediate them. One might ask, W h y

that question from the point o f view o f social

focus on stable individual differences? Surely

psychologists, whose mission is to understand

some people may be more likely on average to

the effects o f situations and the psychological

display a certain behavior than others, but what

mechanisms that underlie those effects.

does that have to do with the effects of situa-

Consider experiments in which people are

tions? W h y does one need to take individual

exposed to conditions that differ on a situa-

differences into account? This chapter begins

tional factor o f interest, producing

by addressing this "why" question (part I), fol-

ences in their thoughts, feelings, or behaviors

lowed by a discussion of "what" questions,

on average. If we do experiment after exper-

namely the nature of individual difference vari-

iment in which all sorts o f situational factors

differ-

ables that interact with situations (part II); an

are manipulated, singly and jointly, we will

overview of the issues relating to " h o w " ques-

arrive at an understanding o f the mechanism

tions, such as identifying individual difference

that underlies social information processing,

constructs

situations

resulting in essence in a giant regression

(part III); and operationalizing them in actual

equation in which the additive and interac-

experiments in the form o f measures (part IV).

tive effects o f all variables whose effects have

that

interact

with

been studied are represented. O r will we? This chapter argues that there is a potential problem in this strategy. T h e problem is not

W H Y STABLE INDIVIDUAL DIFFERENCES NEED T O BE

simply that it is difficult to manipulate some

TAKEN INTO ACCOUNT

variables of importance. T o be sure, it is unfortunate if the field relies solely on situa-

Almost half a century ago, Lee Cronbach, in

tional manipulation

his APA presidential address, observed that

excluding factors whose variations are largely

and as a result risks

independent

"presented by Nature" rather than created

research traditions in psychology, which he

in laboratories. But some variables can be

there had

been t w o largely

correlational

manipulated; so, can't their effects, at least, be

(Cronbach, 1 9 5 7 ) . In the experimental tradi-

established confidently? T h a t may be the case

tion, one varies the aspects o f situations

if the effects o f those variables don't depend

called the experimental and

hypothesized to influence the behavior o f

on other variables that are not manipulated.

interest, while holding constant all other fac-

But what if the effects of the situational

tors. T h e focus is on the variation created by

manipulation critically depend on some unob-

the experimenter and isolating the effects of

served variables? T h e n effects o f situations can

it. In contrast, the correlational approach

differ from one person to another. Might aver-

focuses on the already existing variations

aging the effects across people at least repre-

(Cronbach, 1 9 5 7 ,

sent the effects o f the situation on the average

p. 6 7 1 ) , embedded in a complex web o f inter-

person? Unfortunately, the "average" person

related variables, only a small fraction o f

may exist only in statistical abstraction (see

which are observable. Cronbach argued then,

C o o k and G r o o m , Chapter 2 , this volume).

"presented

by N a t u r e "

1

as well as 2 0 years later in his APA distin-

In summary, (a) many important psycho-

guished scientist award address (Cronbach,

logical variables cannot easily be manipulated,

1 9 7 5 ) , that neither alone is likely t o be suffi-

(b) even those that can be manipulated may

cient, and that one must focus on the

interac­

interact with those that cannot, and (c) behav-

tions between manipulated situational factors

iors may reflect the emergent properties of

Individual

Differences

in Social

Psychology

systems. Consequently, if

in the cognitions and affects identified from a

one does not take individual differences into

literature review as relevant for the decision

account, a sole focus on the "effects" o f the

and actual performance o f a breast self-exam-

person-situation

manipulated variables on people "in general"

ination (BSE) and the network o f relations

may result in at best a partial, and at worst, a

that guides their activation (Miller, Shoda, &

misleading, understanding o f the process

Hurley, 1 9 9 6 ) . Examples o f cognitive and affective "units" relevant to B S E are illus-

underlying the behavior o f interest.

trated schematically inside the large ovals in Figures 6.1 and 6 . 2 . A solid arrow connecting

Lewin's Equation, B = f(P, E)

one cognitive-affective unit to another indi-

Although Cronbach's plea t o combine the

cates that the activation of the first increases

experimental and correlational traditions was

the activation o f the second. Broken arrows

more from the point o f view o f research

show that the activation o f the first reduces

paradigms, the need to consider both the sit-

the activation o f the second. T h e thoughts,

uations and persons was also central to Kurt

the affects, and the connections that charac-

Lewin's theory in which the behavior is

terize each type are shown in boldface type.

conceptualized as a function o f the field, the

For a w o m a n whose network resembles

whole gestalt consisting o f persons (or the

Figure 6 . 1 , health-risk information is highly

" o b j e c t " in the following quotation) and situ-

likely to activate the thought, " I may develop

ations involved: " O n l y by the concrete whole

breast cancer," as indicated by the thick solid

which comprises the object and the situation

arrow (Figure 6 . 1 , Arrow 1 ) . In contrast, for a

are the vectors which determine the dynamics

w o m a n whose network resembles Figure 6.2,

of the event defined" (Lewin, 1 9 3 1 , p. 1 6 5 ) .

with a thin Arrow 1, the objective risk information is less likely to activate this thought.

P = An Individual's

Dynamic

Social Information

Processing

System: An

W i t h regard

t o observable

behavioral

responses, depending on whether a woman's cognitive-affective processing network resem-

Example

bles Figure 6.1 or Figure 6 . 2 , the risk informa-

H o w can persons be conceptualized in

tion can have potentially opposite outcomes.

order to help understand the effects o f situa-

For

tions, and how they may vary meaningfully

represents the information influences the level

example, h o w an individual

mentally

from one person to another? T o account for

of arousal and anxiety (Arrow 2 8 ) . Thus,

such person x situation interactions, it is help-

whether she focuses more on the information

ful to conceptualize personality as a dynamical

itself (e.g., " N o w I need to look for changes in

system (Shoda, LeeTiernan, et al., 2 0 0 2 ) . In

this area," Arrow 2 0 ) or on a more emotion

such a system, stability is expected in the

arousing aspect (e.g., " I f I find a lump, that

underlying

the

means hundreds o f thousands o f cancer cells

thoughts, feelings, and behaviors which them-

are already there," Arrow 2 1 ) has very differ-

selves can change from one moment to the

ent impacts on the continued practice of BSE.

structure

that

generates

next in response to situations. The stability o f the underlying structure will be reflected not in the constancy o f thoughts, feelings, and behav-

Studying

Person

x

iors, but rather in the way they change, in the

Situation

way the thoughts and affects come and go.

W i t h this example in mind, consider the

T o illustrate, Figures 6.1 and 6.2 depict differences between two types o f individuals

Interactions

question we asked at the outset o f this chapter: H o w

would

one investigate the

119

120

DESIGN AND ANALYSIS

Figure 6.1

An Illustrative CAPS Network That Undermines Intention and Performance of BSE

SOURCE: Based on, and adapted from, Figure 5 of Miller, Shoda, and Hurley (1996), p. 83. NOTE: Situational features activate specific subsets of the mediating units, which in turn activate other mediating units. The network of connections is considered stable and characterizes the individual. Arrows indicate activation relationships, such that when one unit is activated, other units that receive solid arrows from it will receive activation proportional to the weight associated with each arrow. The weight may be positive (solid arrows) or negative (dashed arrows).

mechanisms that underlie people's responses

mediate the effects o f manipulated variables,

to the situational input? Consider these possi-

following Baron and Kenny's ( 1 9 8 6 ) guide-

bilities in the BSE example. If all women more

lines, also as discussed by Hoyle and Robinson

or less fit the processing structure shown in

(Chapter

1 0 , this volume). But what if

Figure 6 . 1 , one may perform a series o f experi-

Figure 6.1 described only a small proportion o f

ments in which each situation feature and cog-

women, while other women fit Figure 6.2, and

nitive-affective unit is manipulated, and their

yet another group o f women were character-

effects on other cognitive-affective units are

ized by cognitive-affective dynamics that nei-

assessed. F o r those mediating

that

ther Figure 6.1 nor Figure 6.2 describes? Does

are difficult or unethical to manipulate, if

heightened awareness increase or decrease the

they are measurable, one may apply statistical

intention to perform BSE? This is analogous

approaches

to the question that Gordon Allport ( 1 9 3 7 ,

units

and assess evidence that

they

Individual

Figure 6.2

Differences

in Social

Psychology

An Illustrative CAPS Network That Enhances Intention and Performance of BSE

SOURCE: Based on, and adapted from, Figure 6 of Miller, Shoda, and Hurley (1996), p. 84.

p. 1 0 2 ) posed: W h a t is the effect o f cooking

critical aspect is its structure, or the set o f

heat on foods—does it harden or soften them?

relationships among its components, consti-

T h e question is meaningless unless one speci-

tuting a distinctive system

fies the type o f food. It results in one answer if

processing for each individual.

o f information

the food is a raw egg, but another if the food is a stick of butter. T o summarize, in order to understand the psychological

mechanisms

that

underlie

W H A T INDIVIDUAL DIFFERENCES?

behaviors, one must go beyond examining a statisti-

H o w does one find the particular aspects of

cal abstraction. T o b o r r o w Lewin's phrase,

an individual's processing dynamics that are

one needs to examine "the concrete

relevant for the behavior one wishes to under-

the responses o f people on average,

whole

which comprises the object and the situa-

stand? T h e rest o f this chapter seeks to pro-

t i o n " [emphasis added], o f which Figure 6.1

vide at least the beginning o f an answer to this

may be one and Figure 6.2 m a y be another.

question: This question can be approached

compo-

both in a top-down, deductive fashion, as

nents, many o f which may not be manipula-

It includes potentially numerous

well as in a bottom-up, inductive fashion.

ble or even observable. M o s t important, a

In a deductive approach, one starts with an

DESIGN AND ANALYSIS individual difference construct chosen on the

psychologists can no longer afford to ignore

basis o f researchers' informal observations

individual differences, and personality psy-

and intuition, or on previous studies or a

chologists can no longer ignore situations. Just

theory. T h e inductive approach, on the other

as studying the effects o f individual differences

hand, focuses on discovering the construct to

in situations

begin with. Its goal is t o systematize the pro-

studying the effects o f situations on people

cess of discovery and identification o f the

general

in general

can be misleading, in

can be misleading.

construct. W h e t h e r one follows the deductive or the inductive path, just what are the important ways in w h i c h people

differ

from

one

Individual Differences That Interact With Situations

(1936)

W h a t individual difference constructs inter-

found, the English language has some 1 8 , 0 0 0

act with situational factors? W h a t kinds o f

words describing characteristics o f a person,

personal characteristics serve as the " P " in

and people potentially can differ on all o f

Person x

them. Are there

" P " x " S " interactions? Unfortunately,

another? As Allport and

Odbert

any guidelines t o

help

narrow the search?

Situation interactions, that

is, the

question has rarely been addressed directly in

O f course, the search for the individual

standard compilations o f measures. In this sec-

difference construct must be guided by its

tion, some general characteristics of individual

relevance to the particular behavior one is

difference variables that interact with situa-

interested in understanding. But for the pur-

tions are identified, based on reviews of the

poses o f understanding the effects of situations

literature as well as an informal e-mail survey

and the mechanisms underlying them, one

sent to members o f the Society of Personality

general guideline applies: T h e types of individ-

and Social Psychology asking them to nomi-

ual differences one needs to understand for

nate

that purpose are those that interact

have been shown to interact with situations."

with situ-

"individual

difference constructs

that 2

ation characteristics or situationally induced internal states. Stated differently, one can relatively safely ignore individual difference vari-

Processing

ables that simply increase or decrease the level

Type and Diagnostic

Dynamics Situations

of behavioral response while not affecting the

A major type o f individual difference vari-

impact of situational variables. A "division of

ables that interact with situations is those that

labor" of sorts might be achieved, in which

characterize individuals by their particular pro-

the study o f such individual differences is con-

cessing dynamics. The nature of the dynamics

sidered the subject matter o f personality psy-

in turn identifies "diagnostic situations" that

chologists while social psychologists focus on

engage the dynamics, situations in which that

the effects of situations. In fact, that is often

group o f individuals is expected to respond

h o w the fields o f personality psychology and

distinctively. As an example of individual dif-

social psychology, or "correlational"

ference constructs that identify processing

and

"experimental" approaches, are divided, with

dynamics, consider uncertainty

the consequence that one field's main variance

(Sorrentino &

is considered by the other as "that outer dark-

guishes people w h o are relatively comfortable

ness known as 'error variance'" (Cronbach,

dealing with uncertainty and strive to resolve it

1 9 5 7 , p. 6 7 4 ) . However, to the extent that

from those who are more uncomfortable with

orientation

Roney, 2 0 0 0 ) , which distin-

situational variables interact with individual

uncertainty and are likely to avoid situations

difference

that increase a subjective sense of uncertainty.

variables, as discussed, social

Individual

Differences

in Social Psychology

\

Uncertainty orientation has been shown to be

(Mendoza-Denton, Purdie, Downey, 8c Davis,

important in answering diverse questions such

in press), diagnostic situations are ones in

as the following: Would instructional tech-

which rejection on the basis o f one's race is

niques that involve a cooperative situation

perceived as possible or likely. M a n y of the

(e.g., the Jigsaw classroom) be better than a

individual difference constructs that interact

traditional expository technique? and Would

with situations function in a similar way. They

experiencing lack o f control in a situation

describe people w h o are characterized by a

make a person seek new information? It turns

particular type o f processing dynamics, which

out that the answers to these questions depend

in turn specifies the diagnostic situations that

on an individual's uncertainty orientation. For

trigger the characteristic dynamics.

those w h o are high in uncertainly orientation,

A notable subset o f this type o f interaction

the answer is yes. But for those w h o are low in

involves situations that are generally more

uncertainty orientation, the answer was the

demanding,

opposite:

example, those w h o are high versus low on

Traditional

than

better

instructions

cooperative

ones

work (Huber,

taxing,

and

stressful.

neuroticism differ most in their

For

negative

Sorrentino, Davidson, & Epplier, 1992); and

affect in situations high in stress (Bolger 8c

experiencing uncontrollability makes it more

Zuckerman,

likely that these people, especially if depressed,

2 0 0 2 ) . People scoring high on the measure o f

will

Behavior Inhibition System (BIS) were most

avoid

new

information

(Walker &

Sorrentino, 2000).

1 9 9 5 ; M r o c z e k 8c Almeida,

distinctive in their responses t o negative daily

Although the particular situation features that interact with uncertainty orientation dif-

events

(Carver 8c W h i t e , 1 9 9 4 ; G a b l e ,

Reis, & Elliot, 2 0 0 0 ) . J . C r o c k e r , T h o m p s o n ,

fer widely, they have one theme in common,

M c G r a w , and Ingerman (1987) found that

namely the activation o f a subjective feeling o f

threats t o self-concept served as a diagnostic

uncertainty. Situations that contain this fea-

situation that differentiated individuals high

ture trigger the processing dynamics that are

in self-esteem from those low in self-esteem

characteristic o f those w h o are low, and those

in their tendency t o derogate outgroups.

w h o are high, in uncertainty

orientation.

T h e r e are m a n y examples o f processing

Another type of diagnostic situation involves public versus private

settings. M o r f

and

dynamics that interact with situations. T o

Rhodewalt (1993) found that narcissists' ten-

name a few, for individuals with achievement-

dency to derogate those w h o

oriented processing dynamics, competitive

them was more distinctive in public compared

situations with externally assigned performance

to private settings. Schiitz and DePaulo (1996)

goals enhance the engagement and enjoyment,

found that in private settings, people with low

but for those w h o are low in achievement

self-esteem did not differ from those high in

motivation, noncompetitive situations mastery

optimize

enjoyment

their

engagement

outperformed

with

self-esteem in how they evaluated others' art-

and

work, but in public settings people with low

(Barron & Harackiewicz, 2001).

self-esteem were more positive than people with

Opportunities for bolstering their grandiose

high self-esteem. Chen, Shechter, and Chaiken

self-concepts define situations in which narcis-

(1996) found that people who are high versus

sists exert much effort toward

providing

low in self-monitoring did not differ in their

evidence o f their superiority, even if doing so

expressed opinions in private, but they differed

may offend or hurt others (e.g., M o r f 8c

in public, such that when they expected to have

Rhodewalt,

Morf,

a discussion with their partner on a topic, high

are characterized

self-monitors adjusted their expressed attitudes

2 0 0 1 ; Rhodewalt &

1998). For people w h o by

racial

rejection

sensitivity

dynamics

to be more in line with their partner's.

DESIGN AND ANALYSIS

124

Types of Person

These individuals

Variables

That Affect Processing

m a y not differ in

the

chronic activation o f the thought " I may find

Dynamics

a lump someday." B u t they may differ in its Are there any commonalities among the

association to another thought, "I'll get early,

individual difference variables that interact

effective treatment, and I'll be cured" com-

with situations? Those that alter the nature o f

pared to its association to the thought "I'll be

the effect o f situations are likely to influence

subjected to painful

the way in which individuals process social

ment." In short, individuals' social informa-

but ineffective treat-

information. O f course, how social informa-

tion processing systems may differ (a) in the

tion is processed is one o f the basic questions

availability

addressed by social psychology, so it would

domain-relevant cognitions and affects and

not be surprising that the same type o f vari-

and

chronic

accessibility o f

(b) in the network

o f associations among

ables that have proven useful for characteriz-

them

Shoda,

ing the effects o f situations may also be useful

LeeTiernan, et al., 2 0 0 2 ) . M a n y o f the vari-

in understanding individual differences. T h a t ,

ables that have been shown to interact with

of course, is what Lewin proposed more than

situations refer to either one or more of the

half a century ago and what is practiced in

component cognitions and affective reactions

(Mischel &

1 9 9 5 ; Shoda,

contemporary research when, for example,

(e.g., self-efficacy, perceived social support,

operationalizing

self-esteem) as well as an entire configuration

individual

differences

in

terms o f the chronic activation o f knowledge

or system (e.g., neuroticism, shyness).

structures (e.g., Higgins, 1 9 9 9 ) . These processoriented constructs were proposed

as

the

basis for characterizing individuals by Mischel

Interactions

( 1 9 7 3 ) , w h o called them "cognitive social

Highly

person variables." A closer analysis (Shoda,

and Situation

1 9 9 9 ) found that the primary effects o f these variables often are not to increase or decrease any given behavior in general, but rather their behavioral manifestation is in the form o f if...

then . . . profiles that characterize the

relationship between situations and behaviors,

May

Involve

Content-Specific

Personal

Characteristics

The personal and situational characteristics that interact may be highly contentspecific. One example of this is seen in studies of the self-evaluation maintenance model (e.g., Beach et al., 1 9 9 8 ) , which examines people's emotional responses upon observing

that

as mediated by the individuals in their charac-

their partner did particularly well in a task. It

teristic ways.

turns out that whether the participants felt

In addition to the activation o f concepts, in

happier, basking in the reflected glory, or

recent years research in social cognition has

instead felt threatened by their partner's supe-

turned to the associations

among concepts,

rior performance depends on the specifics o f

particularly those that operate automatically.

the task. In particular, when the partner's

Such implicit associations have been the tar-

superior performance was in a domain (e.g.,

get o f both theorizing and empirical assess-

math) highly central to the participants' self-

ment, as detailed by Kihlstrom (Chapter 9 ,

concept, the effect was an increased sense o f

this volume). Individuals may differ not only

threat. O n the other hand, when the partner's

in the chronic accessibility o f these mediating

superior performance was in a domain (e.g.,

cognitions and affective reactions but also in

music) less central to the participants, it was

h o w specific thoughts are associated with

more likely they felt happier as a result. N o t e

each other. Consider the t w o hypothetical

that the relevant constructs are not the part-

"minds" depicted in Figures 6.1 and 6 . 2 .

ner's level o f performance in general or the

Individual

Differences

in Social

Psychology

participants' global self-evaluation. Rather, it

below the surface bedrock lies, what

was the particular domain (e.g., math, music)

bedrock consists of, etc.). T h e specific findings

o f performance and the particular aspects o f

about the bedrock under a particular hill

self-concept (centrality o f math or music) that

might not merit publication in a top geology

that

constituted the active ingredients, so to speak,

journal; however, the principles that were

of the situations and persons that produced

used and new principles that emerge from

the crossover interaction.

studies o f many specific hills, or chemical compounds, do merit publication. Similarly,

Going Beyond the Bandwidth-Fidelity

in a given study o f the self-evaluation mainte-

Trade-off

nance process, for example, there may be a large number o f individuals representing a

O n e o f the lasting dilemmas a social psy-

specific configuration o f self-concepts. But it is

chologist faces is the trade-off between the

possible to discover general principles that

accuracy o f predictions (fidelity) and the gen-

characterize the relationship

between

the

erality o f findings (bandwidth). Ideally, one

specific self-concept o f a participant and the

would like t o be able t o understand the psy-

domain

chological processes o f a given individual so

mance. This is worth reporting, as it has gen-

of her partner's

excellent perfor-

well that it becomes possible to accurately

erality. In addition,

predict that person's behavior in a given sit-

principle does not say anything about whether

although

the general

uation. O n the other hand, unless one is

a specific individual will feel threatened by the

studying significant historical figures (e.g.,

superior musical performance o f his partner, it

Runyan, 1 9 8 3 ) , knowing the specifics o f a

spells out a recipe o f sorts for what one needs

given person and situation does not con-

t o find out from a person (i.e., the domain of

tribute much t o the field.

performance central to self) and a situation

How

can this dilemma

be solved? In

(i.e., the domain o f the partner's superior per-

addressing this question, it may be useful to

formance) to predict their interaction with a

consider the historical tendency for psychol-

high degree o f accuracy. T h e challenge is to

ogy to model itself after physics, striving to

identify the relationships among the specifics

describe individuals and situations by a small

that warrant generalization, and that in turn

number o f quantitative variables, and estab-

will indicate what one needs to assess in a per-

lishing general laws that specify the relation-

son and in a situation to predict their interac-

ships among them. I f we insist on applying the

tion at a specific, non-abstract, level.

same framework to psychology, the range o f socially and personally significant behaviors that can be understood sufficiently well may be quite limited. It is often the specifics that matter, as in the example above concerning the effect o f partners' superior performance. Instead, other sciences might provide a better

Behavioral Signatures of Personal Types Guide an Inductive Approach to Discovering Individual Difference Constructs Focusing on individual

difference con-

model, and they may also provide a solution

structs that specify particular types o f pro-

to the accuracy vs. generality dilemma. T a k e ,

cessing dynamics—and in particular those

for example, organic chemistry or geology.

that speak to individuals' beliefs, goals, and

There are thousands o f different geological

values, as well as emotions and

locations, and there are even greater numbers

strivings, for example—can help narrow the

of organic compounds one might be interested

search a little. But that still leaves the question

in learning more about (e.g., just how far

of what beliefs, what goals, and what kinds o f

personal

125

126

DESIGN AND ANALYSIS processing dynamics are most relevant in

occasions. In this framework, Cattell discussed six distinct types o f correlations one can com-

understanding behaviors o f interest. The basic principle discussed so far, that

pute: R , Q , P, O , S, and T . T h e correlations

one needs to seek individual difference con-

reported in the overwhelming majority o f

structs that interact with situations, suggests a

studies in social and personality represent the

route. Individuals' behavioral

R-type, which indicates the similarity or dis-

signatures,

or

the set o f if to (situation) then (behavior) pat-

similarity in the rank ordering o f individuals

terns (Shoda, Mischel, & Wright, 1 9 9 4 ) reflect

with regard to their behavior observed in one

the individuals' characteristic ways o f process-

situation (e.g., one test) with the rank ordering

ing social information. Behavioral signatures

observed in another situation (another test). In

reflect the features o f situations an individual

contrast, the strategy needed for the inductive

notices, the social categories that become acti-

approach discussed above represents the Q-

vated to encode situations, and the beliefs, val-

type analysis, which focuses on the pattern of

ues, goals, and affective reactions that in turn

variations observed within each person and

become activated (Shoda, 1 9 9 9 ) . Thus, if one

seeks to index the similarity or dissimilarity

finds groups o f individuals who share similar

among individuals with regard to their pattern

behavioral signatures, there is a good chance

of situation-to-situation variation.

that the resultant groups represent distinct

Although the Q-type analysis is rare in

types of processing dynamics. Computational

social and personality psychology, in memory

tools that make such typologies possible have

and perception research, it is not uncommon

become available, as has research showing that

for a participant to return day after day for

groups o f individuals with similar behavioral

literally hundreds of trials in which stimuli are

signatures in fact shared similar cognitive

presented under different conditions. A publi-

and affective responses to situations (e.g.,

cation

Vansteelandt & V a n Mechelen, 1 9 9 8 ) .

nomenon may be based on only a dozen

reporting

a

new

perceptual

phe-

N o t e that this strategy requires one to

participants. T h e small number o f participants

shift to a more inductive approach to e x a m -

used in these studies might invoke a chorus o f

ine within-individual variations before group

reactions among social psychologists, "with

trends are addressed. Doing so requires hav-

such a small N, how can one be sure the results

ing an experimental design in which partici-

are reliable?" But for these types of experi-

pants are exposed to multiple situations. (For

ments, the relevant N is not the number o f

further discussion o f this issue, and in partic-

participants but the number of trials. In the

ular on the question o f h o w one goes from

extreme, one participant may be enough to

the

establish a psychological phenomenon (e.g.,

individual's

response

to

the

group

response, see W e s t , Biesanz, and

Kwok,

Chapter 1 3 , this volume.)

Ebbinghaus, 1 8 8 5 ) . T h e purpose o f multiple participants in this design is to test if the effects

Such a strategy is not new in other fields o f

seen in one individual can generalize to other

psychology. As early as 1 9 4 6 , Cattell envi-

individuals. In so doing, this framework does

sioned a research strategy in which one con-

not assume that the effects of the experimental

ceptualized behaviors as a function of the

conditions are the same for all individuals,

person, situation, and occasion, discussed in

although in many cases that does turn out to

detail by Cronbach ( 1 9 8 4 ) . A full design

be true in the kinds o f studies that employ this

would fill each cell o f a three-dimensional data

type o f experimental design. Ironically, this

array, or "Cattell's data b o x , " in which each

design is rarely employed in social psychology

"slice" represents a person, and the columns

even though that is when the effects are most

and rows o f each slice represent situations and

likely not the same for all individuals.

Individual In light o f these observations, when it is possible to operationalize the

Differences

in Social Psychology

research, if one adopts

an

\

"establish the

experiments

effects o f situational factors in each person

using a within-subjects design, an inductive,

first" approach, then one can find clusters o f

bottom-up strategy becomes available for iden-

individuals w h o are similar in their pattern o f

tifying individual differences construct(s) rele-

responses to the situational factors, that is,

vant for a behavior o f interest. T o illustrate,

those with similar behavioral signatures, and

consider as an analogy a drug trial study.

systematically look for factors that differenti-

Imagine that

an

experimental drug

and

ate among these groups o f people.

placebo are given to each participant at various times, and the degree o f symptom alleviation (and side effects) are observed. After extensive trials to ensure a reliable assessment o f

Methodological Challenges for Intensive Within-Subject Analyses

response to the drug and placebo conditions, it

It is o f course not an accident that experi-

is possible to determine the effectiveness o f the

ments in social psychology have not widely

drug vs. the placebo for each individual sepa-

adopted the kinds o f intensive within-subject

rately. Suppose the following results were

analysis described above. T h e nature o f many

obtained when each person's data were ana-

classical experimental manipulations in social

lyzed separately. For 5 0 % o f the people, there

psychology precludes repeated

was a statistically significant symptom reduc-

T h e elaborate staging needed to create exper-

exposures.

tion due to the drug (where the N for a statis-

imental conditions often makes it too time-

tical test is the number o f repeated trials each

consuming to expose participants repeatedly

individual encountered and p indexes the sig-

t o each o f the conditions.

nificance o f the effects o f the drug for a given

exposing subjects to exactly the same situa-

For another 5 0 % o f the people,

tion multiple times is often not a viable

there was a statistically significant negative

option. However, there are a number o f ways

effect of the drug, either a worsening of the

to minimize the factors that sometimes result

symptoms or serious side effects emerging. If

in drawbacks for within-subject designs, and

the study was done as a between-subjects

to take advantage o f the intrinsic benefit o f

study, the result would have been precisely no

them (e.g., Greenwald, 1 9 7 6 ) . Technological

individual).

Furthermore,

overall effect. And unless the particular per-

advances have made it possible t o present

sonal factor that interacts with the drug is iden-

realistic stimuli (e.g., virtual reality technol-

tified a priori, it would not be possible to test

ogy) in an engaging, interactive way, in which

for interactions with that variable.

multiple virtual situations may be created and

But if the study is designed to allow establishing for each person, then the fact that

programmed

to interact with

participants.

Research suggests that reactions observed in

5 0 % o f the patients reliably respond posi-

such "simulated" situations converge with

tively to the drug while another 5 0 % reliably

those obtained in real situations

respond negatively to it is a significant finding

Robinson

&

Clore, 2 0 0 1 ) .

(M. D.

Furthermore,

that merits reporting. M o r e important, publi-

many o f the research agenda in the field have

cation o f such a finding can then lead to stud-

shifted to make use o f repeatedly presented

ies t o

identify

the

critical factors

that

simple stimuli, such as the brief presentation

differentiate individuals for w h o m the drug is

of primes and targets on computers. These

effective and those for w h o m it is detrimental.

advances and changes make examination and

T h a t , in turn, can prove highly informative in

establishment, beyond statistical doubt, o f the

discovering the mechanisms o f the

drug

action. Similarly, in social psychological

experimental effect for each participant increasing possibility.

an

127

128

DESIGN AND ANALYSIS In addition, in the past, statistical procedures

phenomena at an individual level, rather than

have

in the aggregate, in turn leading to discovering

between-group

important differences among individuals in the

experiments in mind, in part reflecting, and at

effects of situational factors (i.e., person x

the

situation interactions).

available

for

social

psychologists

largely been designed with same

time

influencing,

the

training,

assumptions, and common practices in the field. However,

more powerful

statistical

frameworks, algorithms, and most important, easily accessible computer packages that implement them, are becoming increasingly accessible (e.g., see Gonzalez & Griffin, Chapter 1 4 , this volume). For example, the multilevel analysis framework

(e.g., M o s k o w i t z

&

Hershberger, 2 0 0 2 ) allows fitting linear models within each individual, whereas the parameter estimates for each individual (e.g., beta weights indicating the effect o f a situational variable on a given behavior for that person)

can in turn be

predicted from other variables such as individual differences. Computer implementations o f

FINDING, EVALUATING, AND USING MEASURES OF INDIVIDUAL DIFFERENCES Given the importance o f taking

individual

differences into account in social psychology research particularly and the availability o f strategies for identifying potential o f individual

differences in an

aspects

inductive,

bottom-up process, h o w does one actually operationalize individual differences and use them in testing models o f social behavior?

these statistical procedures are available, such as the Hierarchical Linear M o d e l ( H L M ) (Raudenbush & Bryk, 2 0 0 2 ; see www.ssicentral.com/hlm/hlm.htm), SAS Proc M i x e d (SAS Institute), and the N L M E (nonlinear mixed effects) package (Pinheiro & Bates, 2 0 0 0 ; see

What Makes a "Good" Measure: The Intertwined Nature of Reliability and Validity Fortunately, throughout the history o f

http://cm.bell-labs.com/cm/ms/departments/

social and personality psychology in the last

sia/project/nlme) available for S-Plus and for

century, a large number o f measures have

the freely available and rapidly maturing sta-

been developed. They do not, of course,

tistical software called R (www.r-project.org).

exhaust all the important

Highly related to the multilevel framework is

people differ, and no doubt new measures will

ways in which

growth-curve modeling and the Mplus soft-

be devised, as models o f individuals

ware that

Muthen,

dynamic social information processing sys-

2 0 0 2 ; L. Muthen & B . Muthen, 1 9 9 8 - 2 0 0 1 ;

tems evolve. Nonetheless, a bewildering array

implements it (B. O .

as

www.statmodel.com). These statistical models

of measures already exists. For example, the

make it more natural to design and analyze

volume

experiments in which individuals' responses to

Wrightsman

a variety of experimental stimuli are analyzed

measures o f response bias, subjective well-

by J . P. R o b i n s o n , Shaver, ( 1 9 9 1 ) has chapters

and

covering

and conceptualized at an individual level first.

being, self-esteem, social anxiety and shyness,

Even without adopting these new comprehen-

depression

and

loneliness, alienation

sive statistical frameworks, a simple statistic

anomie, interpersonal

such as the percentage o f participants

toward

for

human nature,

trust and

and

attitudes

locus o f control,

whom the situational factors had the expected

authoritarianism, sex roles (masculinity, femi-

effect can be reported,

ninity, and androgyny), and values. Well over

and

its statistical

significance can be tested. Doing so naturally

a dozen particular measures are described in

draws one's attention

each chapter. H o w do we choose?

to looking at

the

3

Individual First, a good measure must reflect the target construct. Despite the

Differences

in Social

Psychology

the difference in the constructs researchers

voluminous

intended to measure using the same instru-

w o r k on this topic, the ultimate goals are

ment. Relatedly, it is important "to remember

4

simple: to minimize noise in the measurement

that

and to be sure that the measure reflects what

Reliability is a property o f the scores on a test

is intended to be measured and not something

for a particular population o f examinees"

a test is not reliable or

unreliable.

else. If the measures were a radio, the recep-

(Wilkinson

tion should be clear, and it should be tuned t o

Statistical Inference, 1 9 9 9 , p. 5 9 6 ) .

&

the A P A T a s k F o r c e on

the right station. But o f course the plot thickens as one actually starts tuning the research radio. Reliability can be shown in a variety o f ways that do not always produce the same conclusion. Different measures o f the same

Bootstrapping Upward in the Evolution of Constructs, Theories, and Measures

thing may be highly related, for example, dif-

So, a first step is to be clear about the con-

ferent halves o f the multi-item questionnaire

struct the measure is intended to measure. But

(split-half reliability), indicating that

in most cases, the quality being "measured" is

the

items that make up a scale reflect something

not something that can be seen or felt, let alone

in c o m m o n (internal consistency). But the

directly measured, but rather a hypothetical

same measure filled out to describe the same

construct. An attitude, a belief, a self-concept—

person by different raters (interrater reliabil-

these are all qualities that are hypothesized

ity), or on different

to characterize an individual. W h e n the target

occasions (test-retest

reliability), may result in the conclusion that

of measurement is a hypothetical quality that

the measure is not reliable. T o untangle this,

cannot be observed directly, h o w does one

one needs to address a fundamental concep-

even ask the question o f whether a measure

tual issue: W h a t is noise, and what is signal?

reflects it accurately? H o w does one address

5

Often the issues o f reliability and validity

the validity o f a measure? T h a t is precisely

are considered in a sequential fashion. First

the question addressed by the APA Committee

establish reliability, to m a k e sure that the

on Psychological Tests half a century ago

measure is reliably measuring

(Cronbach & Meehl, 1 9 5 5 ) , which introduced

something,

then worry about validity, to make sure that

the notion o f construct

validity.

Constructs are

is in fact what one intended to

defined by the theory or model in which they

measure. But what " n o i s e " is depends on the

are embedded. They are an integral part o f a

nature o f what is intended t o be measured

model o f phenomena one wishes to under-

(see Fiske, 2 0 0 2 ) . F o r example, observer

stand, and they evolve as the model evolves

the something

ratings o f h o w often a w o m a n spontaneously

from a crude approximation to a more accu-

smiles while talking to a particular person

rate representation o f the processes that give

may be a fairly reliable measure o f h o w she is

rise to observed phenomena.

feeling in that particular situation, but it may

C r o n b a c h and M e e h l ( 1 9 5 5 , p. 2 8 6 ) illus-

be an unreliable measure o f her friendliness or

trated the construct validation process by the

her attitude toward the person. T o o often,

evolution o f the construct and measure o f

reliability is considered a property o f a partic-

temperature. Neither perceived warmth nor

ular measure, independent o f the construct it

expansion o f mercury is intrinsically more

is intended to measure. M a n y an apparent

valid than the other, but the theories in

contradiction between studies that

physics relate the latter to a construct that is

report

high reliability o f a measure and those that

hypothesized to underlie a wider range o f

report the measure as unreliable stems from

phenomena, which were in fact predicted

129

130

|

DESIGN AND ANALYSIS

more accurately by mercury

thermometer

constructs, examined systematically in light of

readings than observers' judgments o f the

the commonalities in the method of measure-

warmth o f an object. Thus, what is more

ment

"valid" is not the measure alone but rather

approach, called the

the whole system consisting o f the theory

matrix ( M T M M ) , the "theory" being tested is

(Campbell &

Fiske, 1 9 5 9 ) . In this multitrait-multimethod

relating multiple constructs (e.g., tempera-

that measures o f the same construct should

ture, pressure) and the measures o f the con-

correlate with each other even if they employ

structs. Cronbach and M e e h l noted that the

different methods (i.e., convergent validity),

"whole process o f conceptual

and

enrichment

that measures o f different

constructs

begins with what in retrospect we see as an

should not correlate with each other even if

extremely fallible 'criterion'—the human tem-

they employ the same method (i.e., discrimi-

perature sense. T h a t original criterion has

nant validity). With the development of struc-

n o w been relegated to a peripheral position.

tural equation modeling and, in particular,

W e have lifted ourselves by our bootstraps,

confirmatory factor analysis, methods

but

quantitative analyses o f M T M M

in

a

legitimate

and

fruitful

way"

(Cronbach & Meehl, 1 9 5 5 , p. 2 8 6 ) .

for

are now

available (e.g., Kenny & Kashy, 1 9 9 2 ) and are

T h e theory and measures evolve when predictions based on them fail. W h e n they fail, one asks: Is it because the measure is

described in accessible form

(e.g., Kenny,

1 9 9 5 ; M a r s h & Grayson, 1 9 9 5 ) . T h e M T M M is only one specific, and per-

faulty, or because the theory, in which the

haps

construct is embedded, is faulty? W h a t needs

approaches to construct validity. T h e theo-

to be fixed? I f alternative measures o f the

ries in which the constructs are embedded

same construct can accurately predict the

can be more substantive in nature than sim-

phenomena

expected by the theory,

the

simplest,

example

of

general

then

ply expecting measures o f the same construct

perhaps the theory is fine and the particular

t o be m o r e strongly correlated than measures

measure is at fault. O n the other hand, if

o f different constructs. W i t h confirmatory

phenomena predicted by other theories could

factor analysis and structural equation mod-

be predicted accurately using the measure in

eling in general, it should be possible to test

question, but not those predicted by the first

the fit o f a theoretical model with an empiri-

theory, then it's the theory that is faulty, par-

cally obtained pattern o f correlations. T h e

ticularly if that theory continues t o fail t o

general principle remains the same: If the fit

receive support when using other measures.

o f the theory and the data is good, then keep both the theory and the measures used to

An Example Validation

of

Construct

Research:

The

Mu Ititrait-Mu Itimethod

Ma trix

T h e construct validation process discussed

operationalize the theory. I f not, revise either the theory or the measure, or both, taking into account the success o f the theory using other measures and/or the success o f the measures when used to test other theories.

above is particularly important for psychological constructs and theories, and a general discussion of construct validation research can be

Construct

found, for example, in Messick ( 1 9 8 9 , 1 9 9 5 ) .

Individual

One major aspect of such programs that has

Measures

Validation

of

Differences via

Experiments





received much attention is the pattern o f inter-

M o s t important for social psychologists,

correlations among alternative measures o f the

there is no reason for the theories being tested

same construct, as well as those o f different

to be confined to patterns o f intercorrelations

Individual

Differences

in Social

Psychology

among different types of individual differences.

of interactions between individual difference

O n the contrary, a more decisive test o f the

variables

measure comes from testing theories that

approaches described in these publications

m a k e predictions about the effects o f situa-

allow one to avoid the need for artificially

and

situational

factors,

the

tions. If a measure in fact measures a particu-

forming discrete groups (e.g., by dichotomiz-

lar attitude, then

change as a

ing via median-split), which not only reduces

function o f situational manipulations known

statistical power but also can result in spuri-

to induce attitude change. M o r e relevant for

ously significant results (e.g., M a c C a l l u m ,

the present chapter, if a measure identifies

Zhang, Preacher, 8c Rucker, 2 0 0 2 ; M a x w e l l

individuals with different processing dynam-

8c Delaney, 1993).

it should

ics, then individual differences are expected to

At the same time, as is the case for most

interact with situational factors in ways pre-

statistical models used to compute statistical

dicted by the theory o f the particular dynam-

significance, the accuracy and appropriateness

ics. In short, a particularly relevant question

of conclusions based on these regression-based

for those evaluating the construct validity o f a

frameworks also depend on the basic assump-

measure of individual differences for under-

tions made by the model. This is particularly

standing social psychological phenomena is

relevant as one takes advantage of the power

this: Has it been shown t o interact with theo-

of

retically relevant situational factors?

quantitative

data-analytic

frameworks

relationships

examining

among

interval

measurement scales, as compared to the more traditional analysis o f variance

approaches

based on categorical variables that are either

A VALID M E A S U R E HAS BEEN FOUND! W H A T SHOULD WE DO W I T H IT? IMPLICATIONS F O R D A T A ANALYSIS AND EXPERIMENTAL DESIGN

Continuous or Categorical? It Can Matter

nominal or at most ordinal. It is all the more important to examine relatively modelindependent Specifically,

"raw" it

is

summaries useful

to

regression-based analyses with

of

data.

complement exploratory

analyses (e.g., Behrens, 1 9 9 7 , Tukey, 1 9 6 9 , 1977, 1980), to observe data without the lens of parametric statistical models. Models that

Let us suppose that a measure has been

do not rely on assumed linear relationships are

found with a great track record for produc-

also becoming available, such as the regression

ing theory-predicted patterns o f interactions.

analyses, as well as to become aware o f devel-

W h a t , then, would be the next step? W h a t

opments in nonlinear regression (e.g., Huet,

considerations apply as one plans to actually

1996), nonparametric regression (e.g., F o x ,

use the measure?

2000), and other nonparametric and robust

Numerous guides have been written on this topic, both for social psychology research in

methods (e.g., Hettmansperger 8c M c K e a n , 1998; Sprent, 2001).

particular (e.g., Judd, 2000) and in textbooks that provide an integrative and unified framework for analyzing continuous and categorical variables, within- and

between-subject

To Block or Not to Block on Individual Difference Measures?

designs, and fixed and random effects (e.g.,

Although a variety o f publications discuss

Aiken & West, 1 9 9 1 ; Judd & McClelland,

data-analytic methods, a small sample of which

1 9 8 9 ; also see J . Cohen, Aiken, West, 8c

are mentioned above, there is yet another issue

Cohen, 2002). M o s t relevant for the analysis

that merits attention in considering the role o f

132

DESIGN AND ANALYSIS individual differences in social psychological

manipulation

research. T h a t is, how should one take indi-

across individuals. If so, findings about the

vidual differences into account in designing

a

effects

do

not vary

systematically

o f the manipulation obtained for one

study (compared to what to do once data are

individual should apply to another individual

collected)? Similar to many

as well. But what if the cognitive-affective

data-analytic individual

dynamics o f 8 0 % o f the study participants

differences, this also stems from the fact that,

resemble Figure 6 . 2 , and what if risk salience

unlike experimentally manipulated variables

facilitates self-checking behavior only among

issues

involving

measures

of

operationalized by random assignment into

those with a low expectation o f developing

conditions, aspects o f observed as well as

cancer, while it interferes with self-checking

unobserved individual differences are corre-

among those with higher expectations? In

lated with each other. This simple and obvi-

that case, if we do the study with a random

ous fact has an important implication for the

sample o f students in this population, we are

design

likely to conclude that risk salience on the

o f experiments

investigating

the

effects o f situations as well, as discussed in some detail below.

whole facilitates self-checking. T h e conclusion from this low-risk popula-

Suppose one wishes to determine if making

tion, o f course, could be misleading when

salient one's risk for gender-specific types o f

generalizing the findings to a population o f

cancer would

older adults w h o

either increase or

decrease

c o m e for their

annual

health-protective behaviors (e.g., frequency o f

checkups at a clinic. Blocking on the cancer

breast self-examination for women or testicu-

expectation to ensure adequate

lar self-examination for men). T o study this,

tion o f the entire range o f risk expectation,

representa-

students enrolled in an introductory psychol-

therefore, would increase the likelihood o f

ogy course are randomly assigned to a "risk

detecting the interaction. O f course, after

salience"

are

such blocking, the sample is no longer repre-

reminded o f their risks, or to a "reassurance"

sentative o f the Psychology 1 0 1 population.

condition that de-emphasizes the risk while

However, the information about the interac-

encouraging self-examination. Suppose also

tion would make it less risky to extrapolate to

that students' expectation that they might

populations o f older adults with a higher

develop breast or prostate cancer had been

expectation of risk.

condition

in

which

they

assessed in a mass testing session at the begin-

Is blocking, then, an ultimate answer? N o t

ning of the term and was found to be quite

necessarily. Consider the possibility

low, reflecting the fact that they were typical

health risk expectations may be correlated

young adults. Should one simply randomly

with other variables. For example, suppose,

assign all students who sign up for this study

thanks to the successful breast cancer risk

that

to one of the two conditions? Or, because we

consciousness-raising campaigns, that breast

know that those who expect to develop cancer

cancer risks are more salient to women than

are underrepresented in this sample, should

prostate cancer risks are to men. Then block-

we try to recruit an equal number o f partici-

ing on the expectation o f gender-specific

pants high vs. low in this expectation? T h a t is,

cancer risk would result in more women in the

should we block on the expectation variable,

high risk expectation block. The observed

to employ a randomized block design?

interaction with the risk salience manipulation

T h e question is relatively inconsequential

may therefore be, in reality, an interaction

if the effects o f subject characteristics and the

with gender. T h a t is, risk expectation may be

situational manipulation (salience o f risk) do

confounded with gender. One may, o f course,

not interact, such that the effects o f the

further block on gender as well, sampling an

Individual

Differences

in Social Psychology

\

equal number o f women and men within high

one group is sampled from the high end o f a

risk expectation and low risk expectation

distribution and the other is from the low

blocks respectively. But there are potential pit-

end. (For excellent discussions o f this and

falls in doing so as well. For example, male

other methodological issues, see C o o k and

college students w h o are high in their health

Campbell [ 1 9 7 9 , pp. 1 7 5 - 1 8 2 ] . )

risk expectation may have some unusual characteristics not found in other men, such as being more pessimistic, or even hypochondriacal, while female college students with a comparable level o f health risk expectation may be no more pessimistic than average for female students. O n e could statistically examine and remove the effects o f these subject variables in a multiple regression framework, but that would require appreciable and

meaningful

variations in risk expectations in the sample o f participants studied and being able to safely assume that the effects o f these variables are linear and can be extrapolated beyond the range observed in the particular sample. These are big assumptions.

Recasting the Problem: Going Beyond Individual Differences as a Poor Person's Substitute for an Experiment H o w might one solve this dilemma? As for many tough problems, the "solution" may require recasting the problem. W h a t is the nature o f the problem, and what are the assumptions that lead us t o it? T h e ultimate source o f the " p r o b l e m " is this: Individual difference characteristics are correlated with each other. In addition to the problem o f confounding—not being sure exactly which of the correlated variables are really responsi-

In the end, for every measured individual

ble for the phenomenon—there is the issue

difference characteristic, there can be many

that it is often not possible t o find people w h o

m o r e unobserved

are

represent certain combinations o f variables,

correlated with it, and that interact with

making it again more difficult to isolate the

situations significantly. Blocking on all these

effects o f each variable. O n e

variables is not practical, because to the

assumption, or a conceptual framework, that

extent that two variables are correlated posi-

led us to this quandary is that our goal is to

characteristics that

fundamental

tively, it would be difficult to obtain samples

study the effects o f individual variables,

that are high on one variable and low on the

we are approaching individual differences as

other, and if one does, that sample may have

a way t o operationalize the variables.

some particular

characteristics associated

with being rare. F o r example, would the values,

and

But it is

a poor person's operationalization at best, as there is no random assignment.

expectations, and professional experi-

ence o f female professors o f engineering be understood simply by virtue o f their being female and being professors o f engineering? A

Understanding

the

of Situations

for Each

Effects Person

First

regression-based estimation o f the "effects"

W h a t if, instead, we view each individual's

of being female and o f being a professor o f

social processing system, each with a distinc-

engineering observed in a typical random

tive configuration o f bundles o f variables "pre-

sample, which is likely to contain only a few,

sented by Nature" (Cronbach, 1 9 5 7 ) , as a

if any, actual female professors o f engineer-

natural unit, and study its functioning? T o the

ing, could be a tenuous exercise in extrapola-

extent that individuals' genetics, life experi-

tion. Furthermore, measurement error in the

ences, and culture shapes their relatively

scores used for blocking results in a differen-

unique social information processing systems,

tial regression to the mean, especially when

it may not be entirely unreasonable to liken the

133

134

DESIGN AND ANALYSIS role in social psychology of individual differences

2 0 0 2 ) . T h e "confounding" o f variables stops

among people to the role in biology of differ-

being a "problem" if we seek to understand

ences among species. Biologists don't view

the functioning o f the type o f system charac-

species as an operationalization o f variables—

terized by the specific configuration o f the

size, shape, number o f limbs, presence of feath-

variables, "as presented by Nature."

ers, and so on—and examine the "effects" of each of these characteristics in a regression equation. Rather, they take each species one

Trading

Instant

at a time and study the functions o f their

for Ultimate

structure—large-scale structures as well as the

The Implications

molecular structure, as encoded by the unique

Person-

D N A sequence that defines the species. In their

More Inductive,

work, it is clear that it does not even make

This way o f approaching science has some

sense to speak of the function o f a gene inde-

practical implications for social psychology.

pendent o f the context o f the rest of the

First, it should not be necessary to fret too

species's genome and the cellular chemical

much that the results of any given experiment

environment. T h a t is, the behaviors o f an

may be limited to the population o f individu-

organism, as well as the effect o f a particular

als studied, not because it's not true, but

and

Generalizability Generalizability: of a Type-Centered, Approach

gene, are a function of the Gestalt, the whole

because o f a frank acknowledgment that that

configuration o f the genome and the micro

is the case in every study one ever conducts.

(cellular) and macro (ecological) environment.

Integral for the proposed release from "gener-

If one likens species to individual persons

alizability fretting," however, is a shift in how

with their unique configuration of beliefs,

one conceptualizes empirical research. Instead

expectations, values, goals, and competencies,

of trying to achieve generality in any given

the fact that it is difficult to find people with

study, an alternative goal is a commitment to

every possible combination o f characteristics

making every empirical study a part o f a

should not be disconcerting, just as not being

cumulative scientific endeavor. This is based

able to find a species with feathers and gills

on faith, for lack o f a better word, in the ulti-

does not pose a problem for biologists. People

mate success by the scientific community in

need not be seen as an operationalization o f

identifying regularities and generalizable pat-

variables one would rather manipulate via

terns and principles, as studies conducted with

random assignment. Instead, it may be more

different populations o f individuals accumu-

fruitful to see each individual as a distinctive

late. But "faith" alone is not sufficient, o f

social information system

that dynamically

course. It requires that any single study not

interacts with the situations and generates

claim to have found a "general l a w " o f social

thoughts,

psychology that is independent o f the charac-

feelings, and

behaviors. In this

framework, one route for pursuing generaliz-

teristics o f the individuals serving as partici-

able knowledge, rather than knowledge spe-

pants. In turn, such a claim should not be

cific to a single individual, may be to identify

required as a criterion for publication. Instead,

types o f individuals w h o are similar in their

the burden on researchers is to find ways to

social information processing system. T h a t is,

make available as much information as possi-

it may be more fruitful to frame our mission

ble about the individuals w h o served as par-

in a less variable-centered, and more person-

ticipants, for future use in meta-analysis. At

(e.g., Anderson &

least, one can do better than describing study

Sedikides, 1 9 9 1 ; Magnusson, 1 9 9 8 ; Robins,

participants as "students enrolled in an intro-

J o h n , &c Caspi, 1 9 9 8 ; Shoda & LeeTiernan,

ductory psychology class at a large Western

and type-centered, way

Individual

Differences

in Social

135

Psychology

university," which presupposes that all that

important for the individual, engaging the

matters is the subject o f the class and the size

natural distinctions among situations that the

and general geographical location o f the uni-

person's

versity. Without any increase in word count,

makes. If seeing George, Dick, or D o n in

one could name the university, potentially

charge o f a situation makes a person more

conveying a lot more information.

nervous than when others are in charge, then

Second, the use o f a within-subject design, whenever it is possible, would

potentially

information

processing

system

situations involving George, Dick, and D o n share

some important

features

that

the

allow for at least a rough assessment o f the

observer's social information processing sys-

extent to which there are individual differ-

tem picks up, and that in turn activate certain

ences that can moderate the effects o f the situ-

thoughts and feelings. But what is it about the

ations. If, for example, the effects o f situations

situations involving George, Dick, and D o n

vary reliably across participants, then, even if

that are different from others? T o achieve a

it is not known what individual difference

better understanding of the individual's pro-

characteristic is relevant, one at least knows

cessing system, and to go beyond observations

there is a need to be particularly cautious in

made

localizing the findings to the population stud-

responses to novel situations (would a situa-

ied and to report as much potentially relevant

tion involving Colin at the helm o f an organi-

in any given situation and

predict

information about the population as is avail-

zation make the observer nervous too? O r

able. Such a finding is also an indication that

what if Madeline is in charge?), it is necessary

it would be fruitful to launch a concerted

to understand what it was about the situations

effort to identify the important differences in

that made each person respond in his or her

the social information

characteristic manner. T h a t is, situations need

processing systems

underlying the behavior being studied, which

to be understood at the level o f their

in turn may lead to identifying c o m m o n types

logical

o f processing systems. Advancement in the

level (Shoda, Mischel, et al., 1 9 9 4 ) .

features,

rather than at the

psycho­ nominal

statistical models, such as the multilevel anal-

One concrete example o f this approach is a

yses, and practical computer packages that

recent systematic attempt at generating a typol-

implement them, such as H L M , Mplus, and

ogy o f both situations and persons (Kelley

NLME

(discussed earlier in this chapter),

et al., 2 0 0 2 ) . T h e effort primarily focused on

would make it possible to characterize the

"situations o f interdependence," in which the

effects o f situations for each individual, and

behavior o f one partner in a dyad influences

then consider variations among individuals in

the rewards, such as pleasure and gratification,

the nature and magnitude o f such effects.

as well as the costs, such as physical or mental effort, pain, embarrassment, or anxiety, for the other partner. For example, a situation may be

CONCLUDING THOUGHTS: UNDERSTANDING SITUATIONS TO UNDERSTAND PEOPLE, UNDERSTANDING PEOPLE T O UNDERSTAND SITUATIONS

characterized as one in which the basic features of the Prisoner's Dilemma situation apply, so that if reciprocation is assured, cooperative behavior results in the best outcome for both, but one stands to lose much if the partner does not reciprocate with cooperative behavior. In

Studies o f situations and persons go hand in

turn, for each type o f situation characteristic, a

hand. Predictable patterns o f an individual's

type (or types) o f persons could be identified

behavior variation across situations suggests

w h o share distinctive ways o f

that the distinctions among the situations are

to them.

responding

136

DESIGN AND ANALYSIS Patterns of interdependence are, of course,

possibility o f rejection on the basis of race is a

only one aspect o f situations. For a given behav-

major distinguishing feature of situations for

ior o f interest, and for a given domain o f life,

another person, then it seems likely that the

there are likely to be a relatively small number

individual may be characterized by the process-

of key psychological features of situations that

ing dynamics o f racial rejection sensitivity

are particularly important. Identifying them can

(Mendoza-Denton et al., in press). The analysis

in turn lead to identifying types o f persons who

of the situations thus facilitates the identifica-

share distinct dynamics, as discussed earlier.

tion o f distinctive personal characteristics. And

For example, if for an individual a major dis-

the analysis o f the processing dynamics that

tinction among social situations is the presence

characterize an individual in turn leads one to

or absence of opportunities for demonstrating

situation features that characterize diagnostic

his or her superiority, it seems likely that the

situations. Thus, in order to truly understand

individual may be characterized by narcissistic

people, one needs to understand situations, and

processing dynamics (e.g., M o r f & Rhodewalt,

to really understand situations, one needs to

2 0 0 1 ) . If the presence or absence o f the

understand people.

NOTES 1. Furthermore, although for brevity and clarity the present chapter focuses on the immediate effects of situations, the role of individual differences of course is not limited to people's immediate responses to situations. Naturally occurring situations may differ in the kinds of people they attract, and people's reactions to situations in turn affect the future situations they encounter. Thus it may be necessary to understand the system consisting of the particular combination of the person and the environment, rather than each of them separately. That is, the behaviors of individuals may be the emergent properties of the system consisting of the individual and the environment, rather than simply a sum of the effects of situations, or persons, that can be observed in isolation. For further discussion of examples of such person-situation interactions, see Shoda, LeeTiernan, and Mischel (2002) as well as Zayas, Shoda, and Ayduk (2002). 2. The results of the survey are summarized in a table available at the following URL: http://depts.washington.edu/pxs/. It lists (a) The "P"—the individual difference, (b) The "S"—the situational dimension/feature with which the "P" construct interacted, and (c) The " B " in B = f (P x S)—the dependent variable on which the "P" and " S " interacted. It is intended as a continuously evolving table; readers who would like to contribute additional constructs are encouraged to send e-mail to P x [email protected], a dedicated e-mail account checked periodically to update the table. 3. In addition, a publication of the American Psychological Association, The Directory of Unpublished Experimental Mental Measures (Goldman, Mitchel, & Egelson, 1997), lists 2,078 measures that appeared in 37 journals in psychology, education, and sociology from 1991 to 1995. Literally thousands of measures are listed in such publications as Tests in Print V (Murphy, Impara, & Plake, 1999), Tests: A Comprehensive Reference for Assessments in Psychology, Education, and Business (Maddox, 1997), and The ETS Test Collection Catalog (Educational Testing Service, 1993). 4. For excellent discussions of this topic, see the Robinson, Shaver, and Wrightsman (1991) chapter titled "Criteria for Scale Selection and Evaluation," the John and Benet-Martinez (2000) chapter titled "Measurement: Reliability,

Individual

Differences

in Social

Psychology

Construct Validation, and Scale Construction," and textbooks such as Aiken (1997), Anastasi and Urbina (1997), R. J . Cohen and Swerdlik (1999), Gregory (1996), and Kaplan and Saccuzzo (1997). The generalizability theory (Cronbach, Gleser, Nanda, 8c Rajaratnam, 1972) provides a unifying framework in which the many "types" of reliability can be thought of as a facet and is covered in Marcoulides (1999). 5. Discussions of various types of reliability and the ways of estimating them are readily available in other more in-depth treatments such as L. Crocker and Algina (1986), Feldt and Brennan (1989), and Nunnally and Bernstein (1994).

REFERENCES Aiken, L. R. (1997). Psychological testing and assessment (9th ed.). Boston: Allyn 8c Bacon. Aiken, L. R., 8c West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage. Allport, G. W. (1937). Personality: A psychological interpretation. New York: Holt, Rinehart 8c Winston. Allport, G. W., 8c Odbert, H. S. (1936). Trait-names: A psycho-lexical study. Psychological Monographs: General and Applied, 47, 1-171. Anastasi, A., 8c Urbina, S. (1997). Psychological testing (7th ed.). Englewood Cliffs, NJ: Prentice Hall. Anderson, C. A., 8c Sedikides, C. (1991). Thinking about people: Contributions of a typological alternative to associationistic and dimensional models of person perception. Journal of Personality and Social Psychology, 60, 2 0 3 - 2 1 7 . Baron, R. M., 8c Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182. Barron, K. E., 8c Harackiewicz, J . M. (2001). Achievement goals and optimal motiand Social vation: Testing multiple goal models. Journal of Personality Psychology, 80, 706-722. Beach, S.R.H., Tesser, A., Fincham, F. D., Jones, D. J . , Johnson, D., 8c Whitaker, D. J . (1998). Pleasure and pain in doing well, together: An investigation of performance-related affect in close relationships. Journal of Personality and Social Psychology, 74, 9 2 3 - 9 3 8 . Behrens, J . T. (1997). Principles and procedures of exploratory data analysis. Psychological Methods, 2, 131-160. Bolger, N., 8c Zuckerman, A. (1995). A framework for studying personality in the stress process. Journal of Personality and Social Psychology, 69, 890-902. Campbell, D. T., 8c Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Carver, C. S., 8c White, T. L. (1994). Behavioral inhibition, behavioral activation, and affective responses to impending reward and punishment: The BIS/BAS Scales. Journal of Personality and Social Psychology, 67, 3 1 9 - 3 3 3 . Cattell, R. B. (1946). Personality structure and measurement. I. The operational determination of trait unities. British Journal of Psychology, 36, 88-103. Chen, S., Shechter, D., 8c Chaiken, S. (1996). Getting at the truth or getting along: Accuracy-versus impression-motivated heuristic and systematic processing. Journal of Personality and Social Psychology, 71, 2 6 2 - 2 7 5 . Cohen, J . , Aiken, L. S., West, S. G., 8c Cohen, P. ( 2 0 0 2 ) . Applied multiple regression-correlation analysis for the behavioral sciences. Mahwah, N J : Lawrence Erlbaum.

137

DESIGN AND ANALYSIS Cohen, R. J . , 8c Swerdlik, M. E. (1999). Psychological testing and assessment: An introduction to tests and measurement (4th ed.). Mountain View, CA: Mayfield. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Crocker, J . , Thompson, L. L., McGraw, K. M., & Ingerman, C. (1987). Downward comparison, prejudice, and evaluations of others: Effects of self-esteem and threat. Journal of Personality and Social Psychology, 52, 907-916. Crocker, L. M., & Algina, J . (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston. Cronbach, L. J . (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671-684. Cronbach, L. J . (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116-127. Cronbach, L. J . (1984). A research worker's treasure chest. Multivariate Behavioral Research, 19, 2 2 3 - 2 4 0 . Cronbach, L. J . , Gleser, G. C , Nanda, H., & Rajaratnam, N. (1972). The depend­ Theory of generalizability for scores and ability of behavioral measurements: profiles. New York: Wiley. Cronbach, L. J . , & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 2 8 1 - 3 0 2 . Ebbinghaus, H. von. (1885). Vber das Gedachtnis. Untersuchungen zur experi­ Leipzig: Duncker 8c Humblot. mentellen Psychologic Educational Testing Service. (1993). The ETS test collection catalog. Phoenix, AZ: Oryx Press. Educational Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), (3rd ed., pp. 105-145). Washington, DC: American Council on measurement Education/Oryx. Fiske, D. W. (2002). Validity for what? In H. I. Braun & D. N. Jackson (Eds.), The role of constructs in psychological and educational measurement (pp. 169-178). Mahwah, NJ: Lawrence Erlbaum. Fox, J . (2000). Nonparametric simple regression: Smoothing scatterplots. Thousand Oaks, CA: Sage. Gable, S. L., Reis, H. T., & Elliot, A. J . (2000). Behavioral activation and inhibition in everyday life. Journal of Personality and Social Psychology, 78, 1135-1149. Goldman, B. A., Mitchel, D. F., & Egelson, P. E. (1997). The directory of unpub­ lished experimental mental measures (Vol. 7). Washington, DC: American Psychological Association. Greenwald, A. G. (1976). Within-subjects designs: To use or not to use? Psychological Bulletin, 83, 3 1 4 - 3 2 0 . Gregory, R. J . (1996). Psychological testing: History, principles, and applications (2nd ed.). Boston: Allyn 8c Bacon. Hettmansperger, T. P., 8c McKean, J . W. (1998). Robust nonparametric statistical methods. New York: Wiley. Higgins, E. T. (1999). Persons or situations: Unique explanatory principles or variability in general principles? In D. Cervone 8c Y . Shoda (Eds.), The coherence of personality: Social-cognitive bases of consistency, variability, and organiza­ tion (pp. 61-93). New York: Guilford. Huber, G. L., 8c Sorrentino, R. M., Davidson, M . A., 8c Epplier, R. (1992). Uncertainty orientation and cooperative learning: Individual differences within and across cultures. Learning and Individual Differences, 4, 1-24. Huet, S. (1996). Statistical tools for nonlinear regression : A practical guide with S-PLVS examples. New York: Springer. John, O. P., 8c Benet-Martinez, V. (2000). Measurement: Reliability, construct validation, and scale construction. In H. T. Reis 8c C. M. Judd (Eds.), Handbook

Individual

Differences

in Social

Psychology

of research methods in social and personality psychology (pp. 339-369). New York: Cambridge University Press. Judd, C. M . (2000). Everyday data analysis in social psychology: Comparisons of linear models. In H. T. Reis 8c C. M. Judd (Eds.), Handbook of research meth­ ods in social and personality psychology (pp. 3 7 0 - 3 9 3 ) . New York: Cambridge University Press. Judd, C. M., 8c McClelland, G. H. (1989). Data analysis: A model-comparison approach. San Diego: Harcourt Brace Jovanovich. testing: Principles, appli­ Kaplan, R. M., & Saccuzzo, D. P. (1997). Psychological cations, and issues (4th ed.). Wadsworth, CA: Brooks/Cole. Kelley, H. H., Holmes, J . G., Kerr, N. L., Reis, H. T., Rusbult, C. E., &c van Lange, P. A. M . (2002). An atlas of interpersonal situations. New York: Cambridge University Press. Kenny, D. A. (1995). The multitrait-multimethod matrix: Design, analysis, and conceptual issues. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, meth­ ods, and theory: A festschrift honoring Donald W. Fiske (pp. 111-124). Hillsdale, NJ: Lawrence Erlbaum. Kenny, D. A., 8c Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172. Lewin, K. (1931). The conflict between Aristotelian and Galileian modes of thought in contemporary psychology. Journal of General Psychology, 5, 141-177. Lewin, K. (1946). Behavior and development as a function of the total situation. In L. Carmichael (Ed.), Manual of child psychology (pp. 7 9 1 - 8 4 4 ) . Oxford, UK: Wiley. MacCallum, R. C , Zhang, S., Preacher, K. J . , & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40. Maddox, T. (1997). Tests: A comprehensive reference for assessments in psychol­ ogy, education, and business. Austin, T X : Pro-Ed. Magnusson, D. (1998). The logic and implications of a person-oriented approach. In R. B. Cairns, L. R. Bergman, 8c J . Kagan. (Eds.), Methods and models for studying the individual: Essays in honor of Marian Radke-Yarrow (pp. 33-64). Thousand Oaks, CA: Sage. Marcoulides, G. A. (1999). Generalizability theory: Picking up where the Rasch IRT model leaves off? In S. E. Embretson 8c S. L. Hershberger (Eds.), The new rules of measurement: What every psychologist and educator should know (pp. 1 2 9 - 1 5 2 ) . Mahwah, N J : Lawrence Erlbaum. Marsh, H. W., & Grayson, D. (1995). Latent variable models of multitrait-multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 177-198). Thousand Oaks, CA: Sage. Maxwell, S. E., 8c Delaney, H. D. (1993). Bivariate median splits and spurious statistical significance. Psychological Bulletin, 113, 181-190. Mendoza-Denton, R., Purdie, V., Downey, G., 8c Davis, A. (in press). Sensitivity to race-based rejection: Implications for African-American students' transition to college. Journal of Personality and Social Psychology. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan. Messick, S. (1995). Validity of psychological assessment. American Psychologist, SO, 741-749. Miller, S. M., Shoda, Y . , &C Hurley, K. (1996). Applying cognitive social theory to health protective behavior: Breast self-examination in cancer screening. Psychological Bulletin, 119, 70-94. Mischel, W. (1973). Toward a cognitive social learning reconceptualization of personality. Psychological Review, 80, 2 5 2 - 2 8 3 .

139

140

DESIGN AND ANALYSIS Mischel, W., 8c Shoda, Y . (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102, 2 4 6 - 2 6 8 . Morf, C. C., & Rhodewalt, F. (1993). Narcissism and self-evaluation maintenance: Bulletin, Explorations in object relations. Personality and Social Psychology 19, 6 6 8 - 6 7 6 . Morf, C. C., & Rhodewalt, F. (2001). Unraveling the paradoxes of narcissism: A dynamic self-regulatory processing model. Psychological Inquiry, 12, 177-196. Moskowitz, D. S., &C Hershberger, S. L. (2002). Modeling intraindividual variabil­ ity with repeated measures data: Methods and applications. Mahwah, NJ: Lawrence Erlbaum. Mroczek, D. K., & Almeida, D. M. (2002). Age differences in the effect of stress and personality on negative affect. Unpublished manuscript, Fordham University. Murphy, L. L., Impara, J . C., &C Plake, B. S. (1999). Tests in print V: An index to tests, test reviews, and the literature on specific tests. Lincoln, NE: Buros Institute of Mental Measurements. Muthen, B. O. ( 2 0 0 2 ) . Beyond S E M : General latent variable modeling. Behaviormetrika, 29, 81-117. Muthen, L., &c Muthen, B. (1998-2001). Mplus user's guide. Los Angeles: Muthen &C Muthen. theory (3rd ed.). Nunnally, J . C., &c Bernstein, I. H. (1994). Psychometric New York: McGraw-Hill. models in S and S-PLUS. Pinheiro, J . C., & Bates, D. M . (2000). Mixed-effects New York: Springer-Verlag. Applications Raudenbush, S. W., &C Bryk, A. S. (2002). Hierarchical linear models: and data analysis methods (2nd ed.). Newbury Park, CA: Sage. Rhodewalt, F., &c Morf, C. C. (1998). On self-aggrandizement and anger: A temporal analysis of narcissism and affective reactions to success and failure. Journal of Personality and Social Psychology, 74, 6 7 2 - 6 8 5 . Robins, R. W., John, O. P., & Caspi, A. (1998). The typological approach to studying personality. In R. B. Cairns, L. R. Bergman, & J . Kagan. (Eds.), Methods and models for studying the individual: Essays in honor of Marian RadkeYarrow (pp. 1 3 5 - 1 6 0 ) . Thousand Oaks, CA: Sage. Robinson, J . P., Shaver, P. R., & Wrightsman, L. S. (1991). Criteria for scale selection and evaluation. In J . P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), and social psychological attitudes (pp. 1-16). San Measures of personality Diego: Academic Press. Robinson, J . P., Shaver, P. R., & Wrightsman, L. S. (Eds.). (1991). Measures of per­ sonality and social psychological attitudes. San Diego: Academic Press. Robinson, M . D., & Clore, G. L. (2001). Simulation, scenarios, and emotional appraisal: Testing the convergence of real and imagined reactions to emotional stimuli. Personality and Social Psychology Bulletin, 27, 1520-1532. Runyan, W. M . (1983). Why did Van Gogh cut off his ear? The problem of alternative explanations in psychobiography. Journal of Personality and Social Psychology, 40, 1070-1077. Schiitz, A., &c DePaulo, B. M . (1996). Self-esteem and evaluative reactions: Letting people speak for themselves. Journal of Research in Personality, 30, 137-156. Shoda, Y . (1999). Behavioral signatures of personality: Perception and generation of coherence. In D. Cervone 8c Y . Shoda (Eds.), The coherence of personality: bases of personality consistency, variability, and organization Social-cognitive (pp. 155-181). New York: Guilford. Shoda, Y., &c LeeTiernan, S. (2002). What remains invariant?: Finding order within a person's thoughts, feelings, and behaviors across situations. In D. Cervone &

Individual

Differences

in Social

Psychology

W. Mischel (Eds.), Advances in personality science (Vol. 1, pp. 2 4 1 - 2 7 0 ) . New York: Guilford. Shoda, Y . , LeeTiernan, S., & Mischel, W. (2002). Personality as a dynamical system: Emergence of stability and constancy from intra- and inter-personal interactions. Personality and Social Psychology Review, 6, 3 1 6 - 3 2 5 . Shoda, Y . , Mischel, W., & Wright, J . C. (1994). Intra-individual stability in the organization and patterning of behavior: Incorporating psychological situations into the idiographic analysis of personality. Journal of Personality and Social Psychology, 67, 674-687. Sorrentino, R. M., 8c Roney, C. J . (2000). The uncertain mind: Individual differ­ ences in facing the unknown. London: Psychology Press. Sprent, P. (2001). Applied nonparametric statistical methods. Boca Raton, FL: Chapman &c Hall. Tukey, J . W. (1969). Analyzing data: Sanctification or detective work? American Psychologist, 24, 8 3 - 9 1 . Tukey, J . W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley. Tukey, J . W. (1980). We need both exploratory and confirmatory. The American Statistician, 34, 2 3 - 2 5 . Vansteelandt, K., 8t Van Mechelen, I. (1998). Individual differences in situationbehavior profiles: A triple typology model. Journal of Personality and Social Psychology, 75, 7 5 1 - 7 6 5 . Walker, A. M., & Sorrentino, R. M. (2000). Control motivation and uncertainty: Information processing or avoidance in moderate depressives and nondepressives. Personality and Social Psychology Bulletin, 26, 4 3 6 - 4 5 1 . Wilkinson, L., &c the APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 5 9 4 - 6 0 4 . Zayas, V., Shoda, Y . , &c Ayduk, O. (2002). Personality in context: An interpersonal systems perspective. Journal of Personality, 70, 8 5 1 - 8 9 8 .

Part III DESIGN AND ANALYSIS Section B Operationalizing the Constructs: Deciding What to Measure, Why, and How

CHAPTER

7

Constructing and Evaluating Quantitative Measures for Social Psychological Research Conceptual Challenges and Methodological Solutions DUANE T . WEGENER Purdue

University

LEANDRE R . FABRIGAR Queen's

F

University

rom social psychology's earliest days,

Consider the following example. Attitude

researchers have based their theories

researchers have long been interested in under-

and

the

standing why persuasive appeals sometimes

study o f latent psychological constructs (e.g.,

fail. O n e reason may be that messages often

Thurstone, 1 9 2 8 ) . Concepts such as attitudes,

fail to target the underlying basis of an attitude.

attributions, stereotypes, and

Specifically, researchers have proposed

empirical investigations on

interpersonal

that

attraction are all hypothetical constructs that

attitudes (i.e., relatively enduring and global

cannot be directly observed but are nonethe-

evaluations) can be based on two distinct types

less presumed to play an important role in

of information: affect and cognition. T h e affec-

social behavior. O n e o f the great challenges to

tive basis refers to emotions and m o o d states

and successes o f social psychology has been

that a person associates with the attitude

the

object. For example, a person might form a

development

of

reliable

and

valid

approaches to measuring such unobservable

positive attitude toward a Porsche based in

constructs. W i t h o u t

the

part on the emotions the car elicits (e.g., excite-

empirical examination o f social psychological

ment). T h e cognitive basis refers to beliefs

theories would not be possible.

about attributes o f the attitude object. For

such approaches,

146

DESIGN AND ANALYSIS instance, a person might develop a positive

DEFINING QUANTITATIVE

attitude toward a Porsche based on beliefs

MEASURES

about the car's many positive features (e.g., superior handling and performance). It has

Quantitative measures typically represent in

been proposed that some attitudes may be

numerical form the standing of people or

based on affect whereas others may be based

objects

on

some

construct

of

interest.

on cognition (e.g., Katz & Stotland, 1 9 5 9 ) and

Quantitative measures are intended to pro-

that attitudes will be differentially susceptible

duce scores that at least approximate

to persuasive appeals

versus

interval level o f measurement (i.e., units o f

mismatch the basis o f the attitude (e.g.,

measurement represent equal intervals). Thus,

that

match

quantitative measures can be contrasted with

Edwards, 1 9 9 0 ; Fabrigar & Petty, 1 9 9 9 ) . Any empirical investigation of such matching effects requires measures for a variety o f unobservable constructs. First, one must have some way o f measuring attitudes. Second, a researcher must be able to determine if attitudes are based predominantly on affect or on cognition. This requires the development o f measures of affective and cognitive bases. Such measures could be used either to confirm the success o f an experimental manipulation or as a means of categorizing naturally

occurring

an

attitudes.

Finally, this research question requires confirmation that manipulations o f the affective and cognitive nature o f the persuasive appeals were successful. Such confirmation requires measures to assess the nature of the evaluative responses produced by the messages.

measures intended to reflect categorical distinctions among people or objects (i.e., nominal scale measures) or simple rank ordering o f people or objects on a construct o f interest (i.e., ordinal scale measures). There are two potential advantages to having measures with interval level properties

(see Gaito, 1 9 8 0 ;

Townsend & Ashby, 1 9 8 4 ) . Such measures provide information not only about the relative standing o f people on a construct (as in nominal or ordinal data) but also about the magnitude of the difference between people. Also, whether a measure has interval level properties

c a n have implications for

the

appropriateness o f particular statistical procedures. F o r example, A N O V A , multiple regression, and factor analysis are more likely to be appropriate when measures at least approxi-

In this chapter, we review a variety o f

mate the interval level.

issues related to the construction and evaluation o f quantitative measures o f latent psychological constructs. W e begin by defining what quantitative measures are and contrasting them with other types o f measures. W e

STAGES IN C O N S T R U C T I N G QUANTITATIVE MEASURES

then summarize the major steps and key

Constructing quantitative measures can be

issues in constructing quantitative measures

thought o f as occurring in three major stages.

(primarily within the c o n t e x t o f direct self-

First, the researcher must specify the goals o f

report measures o f attitudes and related con-

the measure and formulate the theoretical

structs). N e x t , we review procedures

for

assumptions

that

guide

its

construction.

evaluating the quality o f a quantitative mea-

Second, a pool o f potential items must be gen-

sure. Finally, we turn our attention to the

erated. Finally, the performance o f the indi-

development and assessment o f indirect or

vidual items must be evaluated and items for

nonself-report

the

the final scale selected. In the sections that

psychometric

follow, we outline key challenges that occur at

measures, emphasizing

applicability o f traditional

procedures t o these alternative measures.

each stage and discuss strategies for dealing

Constructing

and Evaluating

Quantitative

Measures

147

with these challenges. Throughout, we discuss

the measure intended for use in laboratory

these issues primarily within the context o f

settings, mass testing sessions, or telephone

self-report measures. W e do so because such

surveys? F o r example, a measure designed for

measures are the most widely used type o f

telephone surveys might have to be relatively

measure

in social psychology.

However,

brief because survey respondents often must

nearly all the challenges we discuss as well as

interrupt ongoing activities to complete the

many o f the strategies for dealing with them

measure and might need to consist o f items

are applicable to the construction o f quantita-

with relatively few response options because

tive measures that do not involve self-reports.

respondents must hold these response options in memory.

Specifying Measurement Goals and Theoretical Assumptions Specifying

Specifying Theoretical

One's

Goals for the

All too often, social psychologists fail to

Measure

recognize the interdependence o f theory and

Successful measurement begins with specifying the intended

goals for the

Assumptions

measurement

(Crites et al., 1 9 9 4 ; Ostrom,

measure

1 9 8 9 ) . Any time a researcher uses a particular

(e.g., see Aiken, 1 9 9 7 ; Friedenberg, 1 9 9 5 ) .

measure, that researcher is implicitly accepting

Obviously, the researcher must determine the

certain assumptions regarding the construct o f

domain the measure is intended to assess. For

interest. Consider a measure o f attitude basis

instance, in our example involving attitude

that asks people to directly report how much

bases, the domain o f interest includes the con-

their attitudes are based on emotions versus

structs o f attitudes, affective bases, and cogni-

beliefs. Such a measure requires the researcher

tive bases. However, outlining the relevant

to accept the assumption that people can suc-

constructs is not sufficient. One should also

cessfully introspect about the extent to which

consider the specificity o f the measures that

their attitudes are based on affect versus cog-

will be used. For instance, does the researcher

nition. Another approach might be to ask

wish to create general measures o f attitude,

people to report their emotions, beliefs, and

affect, and cognition that can be used across

attitudes. T h e researcher could then use statis-

a range of attitude objects (Crites, Fabrigar, &

tical procedures to assess the extent to which

Petty, 1 9 9 4 ; Eagly, Mladinic, & O t t o , 1 9 9 4 ) ,

the two bases are associated with the overall

or does the researcher wish to design mea-

attitude. This measurement strategy does not

sures for a single attitude object or a specific

require the assumption that people can suc-

class o f attitude objects (Abelson, Kinder,

cessfully introspect but would

Peters, &

1984)?

researcher to accept that people can provide

Another issue is the population for which the

valid self-reports o f the content o f the bases o f

measure will be used. This has implications

their attitudes.

Fiske,

1 9 8 2 ; Breckler,

for later decisions such as the wording o f

require

the

Researchers should explicitly acknowledge

instructions and o f items as well as the over-

the interdependence

all length o f the measure. F o r example, a

theory and thus clearly delineate their funda-

measure designed for children might require

mental assumptions before constructing the

simpler wording, fewer items, and simpler

actual measure (e.g., see Crites et al., 1 9 9 4 ) .

o f measurement

and

response options than a measure designed for

Specification o f assumptions should address a

adults. Finally, the context in which the mea-

variety o f issues. Obviously, it should include

sure will be used should also be considered. Is

precise definitions o f constructs. F o r example,

148

DESIGN AND ANALYSIS J a c k s o n ( 1 9 7 1 ) suggested that researchers

or negative. Stating assumptions also can

should

descriptions o f

guide decisions regarding measurement fea-

people w h o score high and low on the con-

tures such as question format. For example, if

struct o f interest. Assumptions regarding the

one assumes positive and negative emotions

structural properties o f the key constructs

can coexist, one might choose to present emo-

might also be important. For example, one

tional states in a unipolar scale format rather

might consider attitude, affect, and cognition

than presenting opposing affective states in a

write

behavioral

to be strictly bipolar or to be conceptually

bipolar scale format. A final benefit o f speci-

independent, with potential for coexisting

fying assumptions is that it communicates

positive and negative responses within each

information to users of the measure that can

construct. O n e would also want to address

assist in proper interpretation. O n e problem in

whether the measures are intended to apply

social psychology has been that

to many objects or whether new measures

researchers often have set out to study the

would be necessary for each new object.

same construct but have used measures that

Finally, one should

specify

different

assumptions

implicitly suggest very different conceptualiza-

regarding h o w the constructs will manifest

tions o f the construct (see Crites e t a l . , 1 9 9 4 ,

themselves. F o r instance, an attitude measure

for discussion o f this in the context of affective

based on overt behaviors rests on the assump-

and cognitive attitude bases).

tion that attitudes and overt behaviors should be strongly associated. Self-report measures

Item Generation

of attitude, affect, and cognition require the assumption

that

these constructs

should

manifest themselves in differential responses to specific semantic stimuli.

Creating

Items

Item Content

and Wording.

Perhaps the

most fundamental issue in creating items is

Articulating theoretical assumptions prior

determining the content and wording o f

to constructing measures can translate into a

items. Although writing items requires sub-

number

jective judgment and creativity, there are

o f practical advantages. First, it

encourages careful evaluation o f assumptions.

strategies that can assist in the process. First,

Implicit assumptions are seldom questioned,

if conceptual assumptions

whereas explicit assumptions can be critically

measure have been specified, items can be

considered for potential flaws. Theoretical

written

assumptions also can provide a set o f stan-

respect t o the stated assumptions. Consider

underlying

the

to have good face validity

with

dards capable o f guiding item generation and

our definition o f the affective basis o f atti-

evaluation. Precise definitions help to clarify

tudes as comprising specific evaluative emo-

whether the items generated are appropriate

tional states associated with an object. O n e

in their content and comprehensive in their

might begin the process o f constructing items

we

by generating a list o f words c o m m o n l y used

defined the affective basis o f attitudes as com-

t o convey particular positive or negative

prising specific emotional states associated

emotional states. T h i s list could be further

with an object that are evaluative in nature.

expanded by using a thesaurus.

coverage. F o r example, imagine that

This definition could help determine which

Another complementary strategy is t o

content would be appropriate for items. For

examine related theories and measures. For

instance, it suggests that terms should reflect

example, theories o f emotion (e.g., Izard,

1

specific emotions rather than undifferentiated

1 9 7 7 ; Russell, 1 9 8 0 ) could be a source for gen-

positive or negative affect. It also suggests that

erating items for the measure of the affective

the emotion terms should be clearly positive

basis. Previously constructed measures also

Constructing

and Evaluating

Quantitative

Measures

\

can be used. Obviously, in drawing on past

regarding whether appropriate coverage has

theory and measures, researchers should not

been achieved. Also, because even a carefully

blindly accept what has been proposed or used

implemented item generation process is likely

in the past. Instead, items drawn from these

t o produce some p o o r items, "content satu-

sources should be assessed in light of the stated

ration" o f the construct is advisable. T h a t is,

assumptions for the current measure. For

the researcher should have multiple items

instance, consider the Circumplex Model o f

designed t o represent each aspect o f the con-

Affect (e.g., Russell, 1 9 8 0 ) . This theory states

struct. F o r instance, an initial pool o f affec-

that affective states comprise differing amounts

tive items would need t o be sufficiently large

o f two underlying dimensions: evaluation and

and diverse such that one could argue that

arousal. Thus, in the context of this theory, it

each basic category o f emotion was repre-

is possible to have some affective states with

sented by multiple items.

little evaluative content. Such emotions do not fit within the stated theoretical assumptions o f

Response

Scale

Format.

Respondents

our example. Hence, in using this theory as a

usually rate items on some underlying bipolar

source for items, one would want to select only

or unipolar continuum. One key design issue is

the subset o f affective states that fit the theo-

how many scale points should be provided.

retical assumptions.

This decision requires balancing two compet-

Generating items also requires considera-

ing concerns (see Krosnick & Fabrigar, 1 9 9 7 ,

tion of how the items will be worded (e.g., see

in press). A response scale with many scale

Janada, 1 9 9 8 ; Krosnick & Fabrigar, in press;

points can potentially capture subtle distinc-

Schuman

tions in ratings. However, there may be a m a x -

&c Presser, 1 9 8 1 ) .

Researchers

should minimize ambiguity. Obscure words

imum level o f refinement at which respondents

that are unlikely to be recognized by most

can make their ratings. Scales exceeding this

respondents should be avoided, as should

level may provide no additional information or

complex wording structures (e.g., items with

even lead to poorer measurement as a result o f

double negatives or double-barreled items).

increased random error in respondents' rat-

Researchers should also avoid wording that

ings. Because the optimum number o f scale

introduces subtle biases in responses. For

points is likely to vary with respondent charac-

instance, item wording should not convey a

teristics and the constructs being assessed,

preference for one response option

there probably is no single number of scale

over

points that will always be ideal. Nonetheless,

another.

studies examining the impact o f scale point Number

Multiple-item measures

of Items.

number on item reliability and validity have

are more likely to provide satisfactory psy-

suggested that the optimum number o f scale

chometric properties

points is usually five to seven (see Krosnick &

than are single-item

measures. This expectation is based on the

Fabrigar, 1 9 9 7 , in press).

2

fact that even well-designed items have some

Y e t another decision is h o w to label the

ambiguities and biases, but these sources o f

scale. At a minimum, a rating scale's end-

error tend to cancel out when items are

points must be defined with verbal labels.

aggregated. Furthermore, many constructs

However, intermediate intervals can be speci-

are broad in scope, making it difficult t o rep-

fied in a number o f ways. Some scales use

resent them adequately with a single item.

boxes; others use numerical values. Some

T h u s , researchers should generate a large

scales use verbal labels for all scale points

pool o f initial items. Specification o f theoret-

(often

ical

numbers). Interestingly, such design features

assumptions

can

provide

guidance

in

conjunction

with

boxes

or

149

150

DESIGN AND ANALYSIS can influence responses. For example, the

(Krosnick 8c Fabrigar, in press;

numerical values used in the response scale

et al., 1996).

can convey subtle differences regarding the

options are used and the distribution o f

nature

assessed.

responses for these questions is o f specific

his colleagues (Schwarz &

interest, it is advisable to rotate the order o f

o f the

Schwarz and

construct

being

Sudman

Thus, when categorical response

B r a d b u r n , 8c

options. Fewer studies have examined the

Schwarz, 1996) conducted a series o f studies

impact o f response option order for rating

Hippler,

1 9 9 1 ; Sudman,

on the impact o f numerical values o f response

scale questions. Given the ordinal nature o f

scales. They found that respondents inter-

response options for these types o f items, only

preted the 11-point rating scale as unipolar

two orders are generally sensible. T o date,

when the numerical values were 0 to 1 0 , but

research suggests that there is a tendency for

as bipolar when the values were - 5 to + 5 .

primacy effects to occur with rating scales

Studies also have examined the impact o f

(Krosnick 8c Fabrigar, in press). Thus, if

versus

researchers have an interest in interpreting the

labels only for endpoints. M o s t studies have

distribution o f responses across the options, it

verbal labels for each scale point found

that

reliability

and

validity

are

increased when fully labeled scales are used (Krosnick &

Fabrigar,

1 9 9 7 , in

is advisable to counterbalance the order o f the rating scale options.

press).

Researchers must also specify the order in

However, use o f verbal labels for all scale

which they will present the items. Such deci-

points is likely t o be feasible only when a

sions can influence the distribution o f item

modest number o f scale points is used (i.e.,

responses as well as the correlation between

seven points or less) and is likely to improve

items. Predicting the nature o f these effects is

item quality only when appropriate labels are

difficult because a variety o f cognitive pro-

chosen.

labels,

cesses can play a role in item order effects (see

researchers should choose labels that have

Krosnick 8c Fabrigar, in press; Schuman 8c

Thus,

when

selecting

precise meanings, reflect the full range o f the

Presser,

continuum o f judgment, and represent rela-

Nonetheless, there are some practices with

tively equal intervals along the continuum.

respect to item ordering that generally will be

See Krosnick and Fabrigar (in press) for a

appropriate. W h e n using several multiple-

description o f studies aimed at determining

item measures, it is best to present the items

the scale values o f particular verbal labels.

associated with each measure as a clearly

1 9 8 1 ; Sudman

etal.,

1996).

defined block o f items rather than interminOrder.

gling items from different measures. A block

T h e order of response options can influence

presentation is less likely to confuse respon-

Response

Option

Order

and Item

being selected

dents and can even more effectively commu-

(Krosnick 8c Fabrigar, in press; Schuman 8c

nicate the intent o f the measure, thereby

Presser, 1 9 8 1 ; Sudman et al., 1996). M o s t o f

increasing reliability (Knowles, 1988). In

this work has been done with questions using

addition, when one measure assesses a more

categorical response options

general construct than another, it is best to

the likelihood o f options

rather

than

rating scales. These studies have indicated that

present the general measure prior to the spe-

either primacy effects (i.e., selecting earlier

cific measure. Finally, when measures are o f

options rather than later options) or recency

comparable

generality,

counterbalancing

effects (i.e., selecting later options rather than

order is advisable when feasible. Thus, in the

earlier options) can occur depending

context o f our example, it would be sensible

on

whether questions are presented orally or

to present the items from the attitude, affec-

visually and on the plausibility o f the options

tive basis, and cognitive basis measures as

Constructing

and Evaluating

Quantitative

Measures

three sets of items. T h e attitude measure

an " X " next to any statements with which

should be presented first, with the affective

they disagree. T h e final score is computed by

and cognitive measures counterbalanced.

averaging the scale values o f the statements endorsed by the respondent. Although developed for assessing attitudes, the procedure

Traditional

Scaling

Procedures

for Item

can be used for a wide range o f constructs. Generation

For example, Breckler ( 1 9 8 4 ) used it to assess

Thus far, we have discussed numerous pro-

affective and cognitive bases o f attitudes. H e

cedures for creating items. These procedures

generated statements reflecting varying levels

can be used in a number o f combinations to

of positive and negative emotional reactions

construct items that best suit the research

to the attitude object (i.e., snakes) that he then

objectives. However, it is worth noting that a

had judges rate according to the level o f

number o f formal scaling procedures for gen-

positivity/negativity these statements reflected.

erating and presenting items have been devel-

Similarly, he developed statements reflecting

oped. These procedures were developed to

beliefs about varying levels o f positive and

measure attitudes,

but the logic o f each

approach can be adapted to measure a variety

negative attributes o f the object that judges also rated in terms o f positivity/negativity.

of other constructs. In the sections that follow, we discuss item generation for three o f the

Likert

Summated

Ratings.

T h e Likert

most widely used o f these scaling procedures.

M e t h o d of Summated Ratings (Likert, 1 9 3 2 ;

Later, we will return to these approaches

Mueller, 1 9 8 6 ) also begins by defining the

when discussing methods o f item selection.

attitude object o f interest. T h e researcher then generates approximately 3 0 statements reflect-

The

ing positive or negative evaluations o f the

Thurstone Equal-Appearing Interval (EAI)

object. Unlike the EAI method, half of the

Thurstone method

Equal-Appearing

Intervals.

(see Mueller, 1 9 8 6 ; Thurstone &

statements should

be clearly positive

and

Chave, 1 9 2 9 ) begins the item generation pro-

the other half clearly negative. Statements are

cess by specifying the attitude object to be

then presented (in random order) with 5-point

evaluated. A pool is generated that includes

response scales ranging from strongly

approximately 5 0 statements reflecting vary-

strongly

disagree.

agree to

Respondents indicate levels

ing levels o f positivity and negativity toward

of agreement with each statement. T h e score is

the object. T h e goal o f this stage is to create

computed

by assigning a value o f 5 for

statements that represent all levels o f the eval-

"strongly agree" responses to positive state-

uative continuum.

subse-

ments, a value o f 4 for "agree" responses to

quently rated by judges (usually 1 0 or more)

positive statements, and so on. Negative items

on an 11-point scale indicating the extent to

are reverse-scored. T h e sum o f item responses

which each statement reflects a positive or

reflects the attitude. As discussed later, items

Statements are

negative evaluation o f the object. T h e average

are chosen for the final attitude measure based

scale values across judges are calculated for

on the correlation o f individual items with the

each statement. T h e final EAI measure con-

total score.

sists o f two statements with average scale values at each o f the 1 1 levels o f the evaluative continuum.

These statements

(in random

order) are presented in a checklist format in

Semantic

Differentials.

ential approach

T h e semantic differ-

(Mueller, 1 9 8 6 ; Osgood,

Suci, &c Tannenbaum, 1 9 5 7 ) usually consists

which respondents place a check next to any

of 4 to 1 0 bipolar rating scales. These scales

statements with which they agree and place

typically are 7-point scales with scale points

DESIGN AND ANALYSIS represented by boxes (numerical values are

versus 3.4) because they allow a researcher to

sometimes used) that the respondent checks to

select items at equal intervals. Optimal items

indicate his or her rating. T h e endpoints are

also have relatively little variance in judges'

labeled with adjectives that reflect opposite

ratings because low variance suggests high

meanings and are highly evaluative in nature

consensus among judges on the location o f

(e.g., good/bad, positive/negative). Respon-

that item on the evaluative continuum. Thus,

dents rate the attitude object on each of these

selection o f items for a Thurstone EAI scale

bipolar scales. T h e attitude is typically com-

typically involves choosing two items to repre-

puted by assigning a value o f 7 for maximally

sent each o f the 11 equally spaced intervals.

positive ratings, a value of 6 to the next most

These two items for each point are the two

positive ratings, and so on. T h e overall score is

items that come closest to that point in their

the sum or average o f ratings. This approach

mean scale values and have the smallest

could be used to assess any construct for which

variances in ratings.

it is possible to find word pairs that reflect the

Although judges' ratings usually have been

opposite ends o f the continuum o f interest.

used to select items for Thurstone EAI atti-

Breckler ( 1 9 8 4 ) and Crites e t a l . ( 1 9 9 4 ) con-

tude scales, there is nothing precluding use

structed semantic differential scales to assess

of this approach to select items assessing

the affective bases o f attitudes (e.g., happy/sad)

other constructs (e.g., see Breckler, 1 9 8 4 ) .

and the cognitive bases o f attitudes (e.g.,

O n e could have judges rate any set o f state-

safe/unsafe).

ments on some underlying

dimension o f

judgment and use these ratings for item selec-

Item Evaluation and Selection

tion. Furthermore, if items needed to possess several properties, one could obtain ratings

Even when constructed with care, it is

on multiple dimensions. For instance, for

unlikely that all items generated will be satis-

scales assessing affective and cognitive bases

factory. Thus, it is necessary to evaluate the

of attitudes, one might have judges rate where

quality o f items and discard those that are

the statements fall on the evaluative contin-

inadequate. Various approaches have been

uum. T h e same or different judges could also

proposed for accomplishing this goal. These

rate the extent t o which the

procedures are not mutually exclusive, and

reflected emotional reactions to or beliefs

statements

thus item selection is most effective using

about the attitude object. T h e final scales

some combination o f procedures.

would consist o f items that reflect equal inter-

3

vals along the evaluative continuum and that were seen as either clearly emotional in nature Judge's

Ratings

(for the affective scale) or clearly object

Judges' ratings can be examined to see h o w well items reflect the desired Perhaps the best-known

attributes (for the cognitive scale).

properties.

example o f this

approach is the Thurstone EAI procedure. As

Between-Group

Differentiation

noted earlier, judges rate each item on the

Another strategy for selecting items is to

extent to which that item implies a negative or

examine if items can detect between-group

positive evaluation o f the attitude object.

differences (e.g., see Aiken, 1 9 9 7 ; Janada,

Descriptive statistics are then computed for

1 9 9 8 ) . This strategy involves identifying two

each item. Optimal items are those items that

groups of people for which there is a strong

have a mean rating very close to one o f the

basis to assume differences on the construct

11-scale points (e.g., a mean value of 3.1

of interest. Items are administered to both

Constructing

and Evaluating

Quantitative

Measures

\

groups, and statistical tests for mean item

expected

differences across the groups are conducted.

approach is based on the logic that such items

Items that fail to show significant differences

do not differentiate between high and low

direction

are

discarded.

This

discarded.

scorers on the overall measure. Thus, these

Obviously, the utility o f this approach rests on

items contribute little to the overall score and

the soundness o f the groups selected. Group

are likely to assess a different construct from

differentiation

the

in the expected direction

are

approaches have been used

majority

of

items

in

the

measure

widely in the development o f psychological

(Friedenberg, 1 9 9 5 ) . T h e item-total correla-

measures. For example, within social/person-

tion is integral to the Likert summated ratings

ality psychology, the Need for Cognition Scale

method

o f attitude measurement

(Likert,

was partially developed using this approach

1 9 3 2 ; Mueller, 1 9 8 6 ) and is discussed in most

(Cacioppo & Petty, 1 9 8 2 ) . Need for cognition

texts on psychological measurement (e.g., see

is the dispositional tendency t o engage in and

Friedenberg, 1 9 9 5 ; J a n a d a , 1 9 9 8 ) .

enjoy effortful cognitive activity. O n e method by which items were selected was to examine differences in items between university faculty

Factor

Analysis

members (a group presumed to be high on this

O n e o f the most widely used and infor-

trait) and assembly-line workers (a group

mative selection procedures is exploratory

assumed to be low on this trait).

factor analysis (EFA). This approach was the basis for the semantic differential method o f

Item Descriptive

attitude measurement (Osgood et al., 1 9 5 7 ) ,

Statistics

and it has been used in constructing numer-

Descriptive statistics also can be used to identify problematic items (e.g., Allen 8c Y e n ,

ous other measures (Floyd 8c

Widaman,

1 9 9 5 ) . E F A is a set o f statistical procedures

1 9 7 9 ) . Some methodologists have suggested

designed to uncover the number and nature

that items with mean response levels near the

of latent factors underlying a set o f items.

endpoints o f the scale should be discarded

Implementing E F A involves a multistep pro-

because nearly everyone responds similarly t o

cess in which researchers have a choice o f a

these items (e.g., agrees or disagrees). These

variety o f different procedures t o accomplish

among

each step. F o r this reason, conducting E F A

people. Likewise, items with little variance

can be confusing, and it is n o t unusual for

items are unlikely to differentiate

are unlikely t o differentiate among respon-

errors

dents. O f note, this concerns variance in rat-

M a c C a l l u m , 8c Strahan, 1 9 9 9 ) . A discussion

to

be m a d e

(Fabrigar, Wegener,

ings o f agreement. This is a separate issue

o f the key issues involved in implementing

from the identification o f items low in vari-

EFA is beyond the scope o f this chapter.

ance on judges' ratings o f item favorability in

However, reviews o f these issues are available

construction o f the Thurstone scales.

(Fabrigar, Wegener, et al., 1 9 9 9 ; Finch 8c West,

1 9 9 7 ; Floyd 8c W i d a m a n , 1 9 9 5 ;

Wegener 8c Fabrigar, 2 0 0 0 ) . Item-Total

Correlations

An E F A provides several types o f informa-

Researchers often assess items by comput-

tion useful for constructing measures. First,

ing the correlation between each item and the

EFA indicates h o w many factors (latent con-

total score or the corrected total score (i.e., the

structs) are needed to account for the correla-

total score on the measure except for the item

tions a m o n g

being correlated with it). Those items that fail

dimensionality o f a set o f items can be tested.

to show sizable item-total correlations in the

For example, imagine a pool o f items assessing

4

items. T h u s , the

predicted

153

DESIGN AND ANALYSIS

154

the affective and cognitive bases o f attitudes. It

contain

would be expected that these items should tap

error and/or are strongly influenced by con-

two distinct dimensions. E F A could be used to

structs specific to only that item (which pre-

substantial

random

measurement

test if two factors underlie this pool o f items.

sumably go beyond the construct that the set

If this did not occur, this would suggest either

of items was designed to assess).

that the items had been poorly constructed or that the researcher's assumptions

regarding

Although

E F A has long been used

to

develop measures, confirmatory factor analysis

these constructs were in error. Second, E F A

(CFA) has become increasingly popular. C F A

provides information regarding the magnitude

is based on the same underlying mathematical

and direction o f relations o f the factors with

model as E F A but involves a somewhat differ-

each item (i.e., factor loadings). Thus, for any

ent set o f procedures. E F A presumes that a

given factor, a researcher can determine which

researcher does not have precise prior knowl-

items that factor influences as well as which

edge of the underlying structure of items. Thus,

items it does not. T h e factor loadings can then

the number of factors and the relations o f fac-

provide insight into the nature o f each factor.

tors to items is determined empirically rather

For instance, in our example, a researcher

than specified a priori. In contrast, C F A

would expect to find two factors, one primar-

assumes relatively precise knowledge

ily influencing the affect items and the other

requires a researcher to specify a priori how

and

primarily influencing the cognition items. If

many factors exist as well as which items these

not, this would suggest poorly constructed

factors will and will not influence.

items or erroneous theoretical assumptions. Factor loadings also allow a researcher to

In recent years, a number of researchers have advocated the use o f C F A rather than

identify problematic items. Affect items ideally

EFA

should show substantial factor loadings on the

O n e possible reason is that some methodolo-

(e.g., J o h n & Benet-Martinez, 2 0 0 0 ) .

affect factor (in the theoretically expected

gists have criticized the seemingly arbitrary

direction) but show low loadings on the cog-

nature o f E F A . However, many o f these criti-

nition factor. An affect item that fails to load

cisms stem from improper implementation o f

on the affect factor is not assessing its intended

EFA rather than inherent flaws in the method

construct. In contrast, an affect item that sub-

(Fabrigar, Wegener, et al., 1 9 9 9 ; Wegener &

stantially loads on both the affect factor and

Fabrigar, 2 0 0 0 ) . Indeed, when selecting items,

the cognition factor is assessing a construct in

it is often more appropriate to use E F A than

addition to what it was designed to assess.

CFA because the items being examined are

An E F A generates two other useful pieces

often newly constructed items whose proper-

of information. W h e n oblique rotations are

ties are not well established. Also, assump-

used, a matrix o f correlations among the

tions regarding the structure o f the items

factors is provided. This indicates the extent

generally

to which factors are distinct from one another

assumptions regarding underlying

and can assist in interpreting the nature o f the

are especially tentative when large pools of

have

not

been

tested.

Strong structure

factors. For example, this information would

items are examined because o f the increased

indicate the extent to which the affective and

possibility that

cognitive bases o f attitudes are distinct from

emerge. Thus, when items are initially evalu-

unexpected

factors

could

one another. E F A also produces communality

ated, there is often insufficient basis to confi-

estimates for items. These estimates indicate

dently specify one or a small subset of a priori

the

item

models as required in C F A . For example, if

explained by the factors. Items with low c o m -

items are being developed for a unidimen-

munalities are problematic because such items

sional measure, there is often little theory to

proportion

o f variance in the

Constructing guide development

o f multifactor

and Evaluating

Quantitative

Measures

models

conduct a C F A in a subsequent study to

(including the number and nature o f alterna-

provide more precise tests of the measure's

most

underlying structure. Thus, the development

affected by them). Y e t , E F A could identify

of a measure often begins with E F A and

when multiple factors influence the items,

moves to C F A at later stages in the research

thereby identifying complexities and problems

program

within the item set.

Gorsuch, 1983).

tive factors and

the specific items

(Fabrigar, Wegener, et al., 1 9 9 9 ;

Some researchers might also prefer C F A because o f the presumed precision o f being

Item

able to test the goodness o f fit for alternative models and of being able to statistically compare the fit o f nested models (as one typically

Response

Theory

O n e increasingly popular

approach

to

item evaluation is item response theory ( I R T )

would have when differing numbers o f fac-

(Embretson 8c Reise, 2 0 0 0 ; M c K i n l e y 8c

tors are hypothesized to influence the same

Mills, 1 9 8 9 ; Steinberg 8c Thissen, 1995).

items). Because E F A and C F A are based on

I R T is a class o f procedures examining the

the same statistical model, it is possible to

relationship o f people standing

on

some

conduct the same sorts o f model tests and sta-

latent construct to responses on a set o f

tistical comparisons o f models that differ in

items. T h e heart o f I R T is the "item informa-

the number o f factors (see Fabrigar, Wegener,

tion curve." This is a graphical depiction o f

et al., 1999). In fact, some indices o f model fit

the relation between a person's standing on a

that are popular for C F A were originally

construct and the probability o f selecting a

developed for use with E F A (e.g., T u c k e r &

particular response on an item. A wide range

Lewis, 1973).

of

procedures

exists for I R T . Although

Another strength of E F A is that it provides

detailed discussion is beyond the scope o f

information that C F A is not well suited to pro-

this chapter, it is useful to distinguish among

vide. M o s t notably, E F A can confirm not only

certain basic features.

that an item loads on the factor it is intended

T h e best known I R T models were devel-

to load on but also that it does not load on fac-

oped for dichotomous response items where

tors it should not load on (Gorsuch, 1983).

the items are presumed to reflect a single latent

Examining possible cross-factor loadings in

construct. O n e feature

C F A is much more cumbersome. Because

among these models is the number o f param-

cross-factor loadings are unexpected, they are

eters in the model. One-parameter models rep-

often not specified a priori in a C F A . Thus, a

resent differences among items solely in terms

that

distinguishes

researcher may fail to detect items that are

of a "threshold" parameter. This parameter

problematic because of cross-factor loadings.

indicates at what level on the construct there is

Alternatively, if each item is specified to load

a .50 probability o f endorsing the item. T w o -

on multiple factors in a C F A , the researcher

parameter models also include an "item dis-

may encounter model identification problems

crimination"

or produce a solution that is difficult to inter-

parameter represents differences in the extent

pret because it has not been rotated for simple

to which items are related to the underlying

structure (as done in E F A ) .

construct. Three-parameter models add

5

W e do not wish to imply that C F A is not useful. Instead, our position is that E F A is

parameter.

"lower-asymptote"

This

additional

a

parameter. T h e lower-

asymptote parameter reflects the probability

often more appropriate when initially selecting

of endorsing an item at the lowest level o f the

items. Once items have undergone some evalu-

construct. In the context o f objective tests, this

ation and refinement, it is often sensible to

parameter indicates h o w likely it is that a

156

|

DESIGN AND ANALYSIS

person will correctly guess the answer in the

constructs, or relating to the construct o f

absence o f any knowledge. In social psychol-

interest in a manner different from that o f

ogy, the two-parameter model is likely to be

other items. A second type o f information

the most conceptually sensible (Embretson &

that can guide item selection is the estimates

2000; Finch & West, 1997; Steinberg 8c Thissen, 1995). Although unidimensional

of item parameters, which can be used to

Reise,

generate item information curves for each

I R T models for dichotomous response items

item. As noted earlier, these curves (in a two-

have been most c o m m o n , models have been

parameter

developed to deal with a broader range o f sit-

regarding the threshold o f endorsement for a

8c Reise, 2000). F o r

particular response and h o w well the item

example, one- and two-parameter unidimen-

differentiates among people on the construct.

uations

(Embretson

model)

convey

information

sional models for items using rating scale

Selection o f items using information curves

formats have been developed.

depends on the goals o f the measure.

Two-parameter rating scale models seem

Imagine that the goal is to develop an affec-

best suited for the items social psychologists

tive basis measure capable o f effectively dis-

usually examine. Like dichotomous response

criminating among people at all levels o f the

item models, these models produce item infor-

construct. In the case o f rating scale response

mation curves. However, rather than produc-

formats, it is possible for items to differentiate

ing a single item curve, these models produce

among people along the full range of the

a set o f curves. Each curve reflects the proba-

affect. Thus, items with high discrimination

bility o f selecting one of the response options

values and for which each o f the response

as a function o f the level o f the construct. For

options contributes useful information should

example, in a 5-point strongly

agree/strongly

disagree scale, there would be a curve for the

be selected (see Steinberg

8c Thissen, 1995). In

other cases, the measure may not need to

probability o f selecting the "strongly agree"

discriminate at all levels o f the construct. For

response, a curve for the probability o f select-

instance, if the construct is normally dis-

ing the "agree" response, and so on. These

tributed in the population, the majority of

curves indicate how well the item discrimi-

respondents will tend to fall within the mod-

nates among people along the range o f the

erate range o f the continuum. In such a situa-

construct and the extent to which each scale

tion, it may not be necessary to select items

point contributes information regarding the

that can differentiate at the extremes. Instead,

construct (see Embretson

8c Reise, 2000; Steinberg 8c Thissen, 1995). T h i s allows the

one might place an emphasis on items capable

researcher to assess whether respondents are

continuum. In contrast, if a researcher wishes

making full use o f the response scale or

to differentiate people at one o f the extremes,

whether

emphasis should be placed on items that dis-

the researcher can simplify

the

of differentiating people in the middle of the

response scale (e.g., reduce from seven t o three

criminate at that end o f the continuum. Thus,

options) without loss o f information.

another benefit o f I R T is its ability to con-

W h e n using I R T to select items, t w o types of

information

(Embretson

can

guide

decisions

struct measures designed to assess a specific portion o f the dimension o f interest.

8c Reise, 2000). First, it is possi-

ble to compute item-fit indices. O n e should discard items demonstrating p o o r fit based

EVALUATING MEASURE QUALITY

on the logic that p o o r item fit is often a function o f items n o t reflecting the construct o f

Once items have been selected, a researcher

interest,

must evaluate the quality o f the measure. M a n y

being influenced

by

unintended

Constructing

and Evaluating

Quantitative

Measures

of the considerations involved in selecting items

T h e concept of r a n d o m error is central to

also relate directly to evaluation. F o r example,

a classical test theory view o f measures (in

dimensionality often determines the utility o f

which a response is conceived as "true score

the measure (e.g., if the purpose is to measure

plus e r r o r " ) (e.g., L o r d & N o v i c k , 1 9 6 8 ) .

a unidimensional construct). As discussed in

These random errors should not, in principle,

the following sections, the general issues o f

be reproduced across multiple responses (i.e.,

validity and generalizability cross the tradi-

either multiple items or multiple responses to

tional boundaries between measure construc-

the same item). This would be one reason to

tion and evaluation. In m a n y ways, evaluation

construct measures consisting o f multiple

(traditionally described as reliability and valid-

rather than single items. Even a measure con-

ity) comes down to the all important match

sisting o f multiple items ultimately would

between the theoretical construct and the spe-

have some amount o f error, however. F r o m

cific items intended to tap the construct. Some

this point o f view, reliability is defined as the

questions about the items relate to the consis-

proportion o f variance that can be attributed

tency of the items with one another. This is

to true scores rather than random error. If all

partly assessed by factor-analytic or other

the variance in responses is due t o the true

inductive methods o f item selection (e.g., item-

score (i.e., to differences in levels o f the con-

total correlations). But the items must also

struct o f interest), then reliability would be 1.

relate closely t o the intended construct, and

If all the variance is due t o random error,

that assessment involves questions o f validity

reliability would be zero. T h e correlation

rather than reliability. W e begin this section

between the observed response and the true

by discussing traditional issues o f reliability

score would be the square root o f the relia-

along with an assessment o f the most c o m m o n

bility o f the measure.

index of reliability (i.e., Cronbach's alpha)

Evidence for reliability traditionally has

(Cronbach, 1 9 5 1 ) . W e continue by describing

taken many forms, including test-retest (sta-

a broad structure for thinking about (and

bility), internal consistency (split-half), and

organizing) the various types o f evidence one

equivalence reliability. O f course, these dif-

might garner in the process o f validating a

ferent types o f reliability address

proposed measure.

sources for error in measures. Test-retest

6

different

(stability) reliability addresses variation in individuals' responses (generally to the same

Reliability

items) across occasions. Internal consistency

As noted when discussing item selection, it

addresses errors that occur because o f differ-

is likely that nearly all psychological measures

ent specific items used to represent the con-

are influenced, in part, by error. Techniques

struct

o f interest.

Similarly, equivalence

such as factor analysis explicitly incorporate

addresses errors associated with the sampling

random

of items used in t w o alternative forms o f the

and

systematic errors

into

the

measurement model. T h a t is, responses to

same measure.

7

items can be influenced by random events (such as unexpected noise in the environment that

temporarily distracts the

respondent

Internal

Consistency

from the item) or by factors that influence

Reports o f internal consistency of measures

only that item (such as reactions to a w o r d

are ubiquitous in social psychology. Perhaps

that appears only in that item, even if the

the presentation o f information about internal

reaction t o that w o r d is n o t the intended

consistency is so frequent because internal

construct to be measured).

consistency is easily assessed using only the

157

158

DESIGN AND ANALYSIS responses o f interest in the study itself.

the items) (e.g., see Schmitt, 1 9 9 6 ) . Because

Although there are a variety o f indices of inter-

alpha does not address the dimensionality o f

nal consistency (see Rosenthal &

the

Rosnow,

1 9 9 1 ) , the most frequently used index is

measure,

other

methods

(such

as

exploratory factor analysis or confirmatory

Cronbach's alpha (Cronbach, 1 9 5 1 ) . Alpha

factor analysis) are necessary. As has been

can be used any time the same people respond

noted for some time, Cronbach's alpha should

to multiple items intended to assess the same

not be used if a measure is found to be multi-

construct. It does not require two or more test-

dimensional (Cronbach, 1 9 5 1 ; Schmitt, 1 9 9 6 ) ;

ing occasions (as in assessments o f test-retest

instead, one should create unidimensional sub-

reliability), nor does it require construction o f

scales and then use alpha separately for each

alternative forms o f the measure (as in tests o f

subscale (John & Benet-Martinez, 2 0 0 0 ) .

equivalence). T h e coefficient alpha is often the

It should also be noted that extremely high

only evidence given that the items in the

internal consistency can sometimes signal

measure sufficiently "hang together." Related

potential problems with a measure. Imagine

to this, a large alpha is sometimes interpreted

that a person uses multiple items to measure

as evidence that the items all index a c o m m o n

cognitive bases o f attitudes, but every item

construct.

uses the same attribute o f the object. The only far

difference across items in this example is the

beyond the information actually provided by

use o f different but equivalent names for the

Unfortunately,

such

inferences go

coefficient alpha. Alpha is influenced by both

attitude object.

the interrelatedness o f the items and

the

would have extremely high interitem correla-

number of items in the measure (see J o h n &

tions (and a high Cronbach alpha), but they

Benet-Martinez, 2 0 0 0 ) . This means that one

would also represent a much more narrow

could have equally high alpha coefficients

representation of the construct than if the

from one measure with many items but low

items assessed different attributes.

interitem

correlations

and

from

Such items

undoubtedly

another

measure with few items but higher interitem correlations. As formalized in the Spearman-

Stability

(Test-Retest)

Brown prophecy formula (see Lord & Novick,

Test-retest reliability generally is exam-

1 9 6 8 ; Rosenthal & Rosnow, 1 9 9 1 ) , alpha

ined by computing the correlation between

always increases as additional items are added

two different administrations o f the same

(assuming

similar interitem correlations),

measure. T h e magnitude o f the correlation is

though this is less the case as the number of

presumed to reflect the reliability o f the mea-

items increases and as the overall interitem cor-

sure. However, it is important to recognize

relations increase. Because coefficient

alpha

that test-retest correlations reflect not only

represents the mean o f the reliabilities com-

the reliability o f the measure but also the

puted from all possible split halves, this also

stability o f the construct it assesses, the simi-

means that alpha does not measure the homo-

larity o f the contexts in which the measure is

geneity o f the interitem correlations; nor does

administered, and the degree to which there

it reflect the unidimensionality o f the scale.

were intervening influences on the construct.

That is, the mean reliability could increase by

T h u s , there are a number o f limitations with

introducing pockets o f highly intercorrelated

use o f simple correlations between

items. In effect, alpha could increase by mak-

administrations o f the same measure or par-

ing the measure multidimensional rather than

allel measures (see also Bollen, 1 9 8 9 ) .

two

unidimensional (if added items from a second

One might argue that the Pearson correla-

dimension raise the average intercorrelation of

tion represents a reasonable index o f reliability

Constructing

and Evaluating

Quantitative

159

Measures

per se only if there was either perfect stability

parties tend to be quite stable over periods of

of the theoretical construct over time or if the

years, whereas attitudes

shift in the construct were identical for all indi-

policy stands are much more variable (e.g.,

toward

particular

viduals responding to the measure (so the rela-

Converse, 1964). Recent research also has

tive standing

construct

shown that attitudes based on effortful cogni-

should, with a perfectly reliable measure, be

tive processing are more likely to persist over

o f people on the

the same at each point in time). However, if

time than are equally e x t r e m e

anything

between

formed or changed with lower levels of pro-

measurements such that some people increase

cessing (e.g., see Petty, Haugtvedt, 8c Smith,

on the construct whereas others decrease, then

1995). Therefore, when examining stability as

influences the construct

attitudes

the correlation coefficient would treat as unre-

a form o f reliability, it is less than meaningful

liability (due to measurement error in the

to provide or seek global recommendations

classic test theory approach) variance that is

for levels o f "acceptable" test-retest reliability.

actually the result of changes in the construct

O n e would clearly expect different levels for

of interest. Whenever one attempts to assess

different constructs or even with different test-

test-retest reliability, there is a balance between

ing procedures for the same construct.

allowing enough time to elapse that consis-

Whereas changes in the construct can lead

tency of response is not due to memory of pre-

to correlations that underestimate reliability,

vious responses (and attendant consistency

the test-retest correlation could also overesti-

pressures) and yet not so much time that there

mate reliability if the "errors" at each occa-

have been substantial changes in the construct

sion are positively correlated. T h a t is, if the

of interest (e.g., see Remmers, 1963).

same omitted factors influence both measures,

In interpreting test-retest correlations, one

the simple correlation could

overestimate

must take into account the extent to which

the reliability o f the measure o f the intended

one would theoretically expect the construct

construct.

to be stable versus malleable. For example, if

threaten the validity o f inferences about the

one is assessing overall attitudes or the affec-

measure when

tive or cognitive bases o f attitudes

about

solely in terms o f the intended construct (see

products that are commonly encountered in

later sections on different forms o f validity

television ads, it should not be surprising if

evidence).

there are substantial changes in the attitudes

Of

course,

this

the measure

would is

also

interpreted

Because correlations across two time peri-

or in the bases over time (if the delay between

ods can conflate instability o f the measure

assessments is sufficiently long). One might

with unreliability, it might often be useful to

actually consider variables as falling along a

employ methods that can separate instability

continuum o f theorized stability, with vari-

of the construct from unreliability of the mea-

ables such as cognitive abilities or personality

sure. O n e example procedure combines latent

traits expected to be quite stable over time,

growth curve analysis with latent state-trait

with variables such as attitudes or beliefs

models to derive separate indices o f stability

expected to be relatively stable but capable o f

and reliability (Tisak 8c Tisak, 2 0 0 0 ; for an

change, and with variables such as concept

application

accessibility or focus o f attention expected to

Cunningham,

in

social

psychology,

Preacher, 8c Banaji,

see

2001).

be quite malleable, even from moment to

Similar to the earlier discussion, Tisak and

moment. O f course, even within a single type

Tisak assume that most psychological con-

of construct, there might be great variability in

structs

how stable or malleable the variable should

aspects and "trait" (stable) aspects. Therefore,

be. For example, attitudes toward political

one might construct indices o f reliability that

include

both

"state"

(malleable)

160

I

DESIGN AND ANALYSIS

are appropriate for different types o f variables

bear on what the measure does and does not

(e.g., including the malleable component as

assess. In the following sections, we organize

"true score" if the construct is thought to

these types o f data using a scheme that seems

have "state" aspects; see also Steyer and

somewhat

Schmitt [ 1 9 9 0 ] ) .

sometimes overlapping types o f validity that

more straightforward

than

the

have been discussed previously as components of construct validity. For example, Messick

Validity

( 1 9 8 9 ) specified six types o f validity that are

Traditional views o f validity specify a

represented in the concept o f construct valid-

number o f types o f validity, including content

ity (see also J o h n & Benet-Martinez, 2 0 0 0 ;

validity (i.e., items fully representing the con-

Loevinger, 1 9 5 7 ) . W h e n all is said and done,

struct o f interest), face validity (i.e., appear-

however, one might usefully organize the

ance o f the items as assessing the intended

various types o f validity into two

construct), and criterion (or external) validity

types o f validity evidence: "associative" and

(i.e., ability o f the measure t o predict related

"dissociative" forms o f evidence.

simple

judgments or behaviors or to correlate with conceptually related constructs). Even from rather early on, however, researchers began

"Associative"

to talk about the various sources o f validity as

Forms

and

of Validity

"Dissociative" Evidence

all relating to the general (and encompassing)

Associative forms o f validity evidence sup-

notion o f construct validity (see Cronbach &

port the utility o f the measure by showing that

Meehl, 1 9 5 5 ; Loevinger, 1 9 5 7 ) . As J o h n and

the measure is associated with factors and out-

Benet-Martinez ( 2 0 0 0 ) put it, "what seemed

comes that would be predicted by relevant

like different types o f validity are just differ-

theory. These associations make the case for

ent sources o f evidence that address particular

"what the measured construct is." In contrast,

questions o f construct validity" (p. 3 5 1 ; see

"dissociative" forms o f validity evidence dis-

also Messick, 1 9 8 9 ) . A construct validity

tinguish the construct from theoretically dis-

approach to measures refocuses one's atten-

tinct constructs, thereby making the case for

tion on systematic errors in measures that

"what the measured construct is N O T . " T h e

might occur when a measure taps unintended

"associative" and

constructs, in addition to any random error

are similar, but not identical, to the Campbell

"dissociative" categories

of the type emphasized in discussions o f

and Fiske ( 1 9 5 9 ) categories o f "convergent"

reliability. Therefore, similar to issues o f

and "discriminant" validity. T h e following

confounding in experimental design, issues

sections describe these relations in some detail.

of "purity" o f the measure are paramount in measurement

validation

(see

Wegener,

Downing, Krosnick, & Petty, 1 9 9 5 ) .

Associative Forms of Validity. M a n y types of associations can support that a measure

O f course, questions o f whether the mea-

assesses the construct it is intended to assess. In

sure sufficiently taps (and only taps) the con-

general, however, these associations can be

struct o f interest already have been discussed

divided into three types: correlations with

as being important considerations in the gen-

alternative measures o f the same construct,

eration and selection o f items. Therefore, one

concurrent

often addresses many aspects o f construct

structs, and correlations with antecedents and/

correlations with related con-

validity in the process o f constructing mea-

or consequences o f the construct. Correlations

sures. O n c e one has developed a measure, a

with alternative measures o f the construct have

number o f types of data might be brought to

long been referred to as reflecting "convergent

Constructing validity" (e.g., Campbell & Fiske,

1959). Some

and Evaluating

Quantitative

Measures

construct of interest. For example, Weary and

(1994) conceived causal uncertainty

alternative measures are quite similar to each

Edwards

other (as when parallel forms of the same ques-

as following, in part, from chronic perceptions

tionnaire are used to test equivalence reliabil-

of loss o f control. This notion was consistent

ity), whereas other alternatives are

quite

with the obtained significant positive correla-

dissimilar. For example, one could imagine

tion between CUS and external locus of

measuring the affective bases o f

control (Rotter,

attitudes

1966). O f course, evidence o f

through two different sets of self-reports (see

this type becomes even stronger when the

Crites et al., 1994) or through both self-reports

antecedent is measured some time in advance

and physiological measures of facial muscle

of the occasion at which the dependent mea-

groups (see C a c i o p p o , M a r t z k e , Petty, &

sure is completed. Although the

Tassinary,

1988). Although Campbell and Fiske (1959) recommended "maximal distinc-

measure is treated as the criterion in this case, the longitudinal nature o f the evidence would

proposed

tiveness" o f methods used in multitrait-multi-

have much in c o m m o n with traditional "pre-

method ( M T M M ) tests o f convergent validity,

dictive validity" studies in which the construct

use o f markedly different types o f measures

o f interest is used to predict future judgments

might often result in people describing the evi-

or behavior (i.e., proposed consequences o f

dence as falling into "criterion validity" or

the construct).

"external validity" categories. O f course, use

Attitudes

of measures that diverge in methodology push

have

been

treated

as

both

antecedents and consequences, often including

generalizability farther than the use o f similar

some separation in measurement between the

2000).

predictor and the criterion. For example, atti-

methods (see J o h n & Benet-Martinez,

T h e basic form o f evidence is the same,

tudes have often

however—that o f associations between the

antecedents o f behavior in studies of attitude-

been conceptualized

as

measure and some alternative form o f measure

behavior consistency. In such studies, attitudes

of the same construct.

often are measured at one point in time, and

Researchers also c o m m o n l y m a k e the case

then behaviors are observed or reported either

for construct validity by examining associa-

relatively soon thereafter

tions between the construct o f interest and

Z a n n a , 1981) or after a substantial delay (e.g.,

theoretically related constructs. F o r example,

Davidson

(e.g., Fazio 8c

8c Jaccard, 1979). As one might

(1994) supported the

expect, the ability o f earlier measures o f atti-

"causal uncertainty" construct (measured by

tudes to guide later behaviors depends, in

W e a r y and Edwards

the Causal Uncertainty Scale, o r C U S ) by

large part, on the ability o f the attitude to

correlating the CUS with the related concepts

remain stable between the measurements o f

o f intolerance for ambiguity (Budner, 1962)

attitude and behavior (e.g., Doll 8c Ajzen,

and

preference

for

order

( W e b s t e r 8c

1992). There are also contexts in which

1994). Sometimes people use the

behaviors can be antecedents rather than con-

term "convergent validity" for such correla-

sequences o f attitudes. F o r example, when a

(1959)

person lies t o someone about the pleasantness

limited the use o f this term t o the associa-

of an activity, this can then influence atti-

tions among alternative measures o f the same

tudes toward the activity (e.g., Festinger 8c

Kruglanski,

tions, although Campbell and Fiske

(rather than related) constructs. A closely related type o f validity evidence

Carlsmith,

1959).

antecedents

and

Relations

consequences

between not

only

would be obtained when the theoretically

address forms o f "concurrent" and "predic-

related construct can also be conceived as

tive" validity but also address what Messick

either an antecedent or a consequence o f the

(1989) referred to as "substantive validity"

DESIGN AND ANALYSIS (i.e., evidence reflective o f theoretically related

construct's domain. T o continue with

processes). For example, if one finds

that

example o f affective versus cognitive bases o f

our

people with attitudes primarily based on affect

attitudes, an understanding of affective bases

are later more persuaded by affective rather

would not only suggest that such measures

than cognitive messages (e.g., Fabrigar 8c

should correlate with related concepts (such as

Petty, 1 9 9 9 ) , this would support the hypothe-

manipulations

sized "persuasion matching" processes (e.g.,

antecedent—or

Edwards, 1 9 9 0 ; Katz 8c Stotland, 1 9 5 9 ) .

framed

o f affective experience—an persuasion

information—a

by

affectively

consequence). This

W h e n validity evidence takes the form o f a

same theoretical approach also would suggest

naturally occurring correlation, one must be

that affectively based attitudes should not be as

particularly aware o f the possibility o f "third

highly related to previous cognitive experience

variable" effects. T h a t is, a third variable

with the attitude object or with persuasion by

might create both the antecedent and the

cognitively

consequence. Stronger inferences about the

M o r e generally, a measure o f a construct

antecedent or consequence status o f a con-

should not only relate to conceptually similar

struct can be obtained when

oriented

persuasive

messages.

experimental

variables; it also should relate less or not at all

manipulations are used. F o r example, evi-

with variables that are conceptually unrelated

dence o f affective attitude bases serving as

to the construct of interest. Campbell and Fiske

antecedents

by

( 1 9 5 9 ) addressed this relative "lack of rela-

"affective" messages is strongest when the

tion" under the rubric o f "divergent validity."

bases are manipulated (see Fabrigar 8c Petty,

In their classic M T M M approach, one would

to

facilitated

persuasion

1 9 9 9 ) . Similarly, one can use manipulations

include not only alternative measures for each

to provide evidence that a measure assesses its

construct of interest but also constructs that

intended construct, while markedly decreas-

varied in their theoretical relations, in order to

ing concerns about possible third variables.

allow for evidence of differences (i.e., diver-

This has been done with measures o f affective

gence) across the constructs.

versus cognitive bases o f attitudes. O n e could

It would make little sense for dissociative

easily imagine that such measures might tap

forms o f validity evidence to include relations

factors such as the amount o f prior experi-

between alternative forms o f the same mea-

ence with the attitude object, instead of, or in

sure

addition to, tapping the affective or cognitive

M T M M terminology), but the other forms o f

nature o f the prior experience. O n e way to

dissociative validity directly parallel those for

increase confidence that the measures

are

the associative category. Whereas associative

assessing "pure" affect or cognition is to cre-

validity evidence comes from relations with

(used for convergent validity in the

ate new attitudes based on equal amounts o f

like constructs, dissociative validity comes

experience toward a novel attitude object, but

from relative lack o f relation with unlike con-

experience that differs regarding its affective

structs. F o r example, in the W e a r y

or cognitive nature (e.g., Crites et al., 1 9 9 4 ,

Edwards ( 1 9 9 4 ) work on causal uncertainty,

and

Study 2 ) . If the measures are altered by this

dissociative (divergent) validity was shown by

experimental manipulation in the expected

failing to find any relation between causal

way, this provides support for their validity.

uncertainty and social desirability or general intelligence. Similarly, the exploratory factor

of Validity. Just as it is

analyses reported in the Crites e t a l . ( 1 9 9 4 )

important to support the nature of a construct

article on affective and cognitive bases o f atti-

by examining associations with related con-

tudes (in which affective and cognitive items

structs, one must not ignore the limits to the

often load on separate factors) imply (and the

Dissociative

Forms

Constructing

and Evaluating

Quantitative

Measures

original correlation matrices support) that the

is relatively straightforward, the assessment o f

affective items correlate more highly with

the conditions laid out by Campbell and Fiske

other affective items than they do with the

is inherently subjective. In part, this is because

cognitive items. In addition, the cognitive

there is no clear metric for h o w much any two

items correlate more highly with the other

correlations should differ in order to "satisfy"

cognitive items than with the affective items.

the conditions; also, there are often many cor-

Dissociative forms o f validity evidence also

relations involved in a given assessment. An

appear in examinations o f antecedents and

additional limitation o f the Campbell and

consequences. For example, exposure to affec-

Fiske approach is that there is no accounting

tive experience with the attitude object (a pre-

for measurement error in the bivariate corre-

sumed antecedent o f affective attitudes) results

lations forming the M T M M matrix.

in attitudes that are more strongly reflective of

In an effort to provide more parsimonious

affect than of cognition. In contrast, exposure

summaries of M T M M matrices while incor-

to cognitive information about the attitude

porating measurement error in the model, a

object results in attitudes more reflective o f

variety o f analysis procedures

cognition than affect. Regarding consequences

developed that use some form o f confirma-

o f affective and cognitive bases, Fabrigar

tory factor analysis (see M a r s h & Grayson,

have

been

and Petty ( 1 9 9 9 ) found that affective attitude

1 9 9 5 ; Visser, Fabrigar, Wegener, & Browne,

bases resulted in responsiveness to affective

2 0 0 3 ; Widaman, 1 9 8 5 ) . T h e classic confir-

but not cognitive persuasive messages. In con-

matory

factor analysis model ( J o r e s k o g ,

trast, cognitive attitude bases resulted in rela-

1 9 7 4 ) treats each measure as a function o f the

tively more responsiveness to cognitive rather

relevant construct, the relevant method, and

than affective messages.

error. Unfortunately, this model suffers from problems with both estimation and interpre-

The

MTMM

Approach.

Since Campbell

tation, because the model includes a large

and Fiske's ( 1 9 5 9 ) seminal paper, researchers

number o f parameters compared with the

have used the M T M M design (crossing two or

number o f measures used t o estimate the

more traits—constructs—with two or more

model, and the free parameters include some

methods) as a way to address the generaliz-

potential logical inconsistencies (see Visser

ability o f a construct across different methods

et al., 2 0 0 3 ) . Some revised versions o f the

(i.e., convergent validity) and the relative dis-

C F A model attempt to solve the problem of

tinctiveness o f any one construct when com-

t o o many parameters by fixing certain paths

pared with the other constructs (i.e., divergent

in the model (e.g., Kenny & Kashy, 1 9 9 2 ;

validity). One would recognize convergent

Millsap, 1 9 9 2 ) , but at the expense o f being

validity as one o f a number o f forms of asso-

able to address certain questions o f interest in

ciative validity evidence. Divergent validity is

MTMM

clearly o f the dissociative variety.

Although most o f the M T M M models have

designs (see Visser et al., 2 0 0 3 ) .

In the original Campbell and Fiske ( 1 9 5 9 )

been conceptualized as additive models (i.e.,

approach, inferences about convergent and

with traits and methods independently influ-

discriminant validity were based on relatively

encing responses), some methodologists have

informal inspection of measures o f the same

argued that traits and methods often interact.

construct using different methods as com-

For

pared with measures of different traits using

( 1 9 6 7 ) found

the same methods or measures o f different

highly correlated, sharing the same method o f

traits using different methods. Although the

measurement resulted in a substantial infla-

inspection o f individual bivariate correlations

tion in intertrait correlations. This did not

example,

Campbell

and

O'Connell

that when two traits were

DESIGN AND ANALYSIS

164

occur to the same extent, however, when the

the attitudes domain in particular and in

two traits were less correlated. Though less

social psychology more generally. Undoub-

frequently used in the social psychological lit-

tedly, this is partly because o f the ease with

erature, multiplicative models exist that allow

which such responses can be obtained. In

for these variations in method effects across

addition, for many if not most constructs

traits and vice versa. In fact, free download-

assessed in social psychology, there is little

able software exists to run one such model,

concern about respondents'

the composite direct product model (CDP)

motivation to provide their perceptions o f the

ability and/or

(Browne, 1 9 8 4 ) . This software is compara-

judgment target and/or construct. It is worth

tively easy to program, and it provides infor-

noting that this in no way conflicts with the

mation that relates quite directly t o questions

oft-cited limitations on people's reports o f

of

psychological process

convergent and

divergent

validity

in

(e.g., Nisbett & Wilson,

M T M M studies (see Visser et al., 2 0 0 3 ) . T h e

1 9 7 7 ) . In most areas o f social psychology,

CDP model also requires fewer free parame-

people are not asked to report on process, but

ters than the traditional C F A model, thereby

rather on content. For example, in classic

running into far fewer estimation problems.

studies o f information processing in attitude

Finally, the C D P puts fewer restrictions on

change, message recipients are simply asked to

the types o f questions that can be asked when

report the content o f their attitudes (e.g., to

compared with the revisions o f the traditional

what extent does the advocated policy seem

CFA

model that place constraints on the

good or bad). T h e message recipients are not

model in an attempt t o decrease the number

asked to report the extent to which they pro-

of free parameters. Limits to the C D P approach

cessed the available information "centrally" certainly

versus

"peripherally"

(Petty &

Cacioppo,

exist. F o r example, as with some o f the addi-

1 9 8 6 ) or "systematically" versus "heuristi-

tive approaches, the C D P output provides

cally" (Chaiken, Liberman, &c Eagly, 1 9 8 9 ) .

little information regarding the performance

O f course, there are some situations in

of individual observed variables. Although

which one might question the ability or moti-

useful summaries across methods or traits

vation o f people to report the true content o f

are provided, the output does not provide

their attitudes. Such concerns have been the

direct information about whether methods

motivating factors behind

converged best for certain traits or vice versa.

"implicit" measures o f attitudes, and these

"indirect"

and

It is also not entirely clear when the additive

types o f measures often have been employed

versus multiplicative assumptions underlying

in

the various C F A approaches are most appro-

researchers have inferred the positivity or

studies

of

prejudice.

For

example,

priate or consequential. In our estimation,

negativity o f attitudes toward social groups

however, the C D P is currently the

by measuring h o w close the participant sits to

most

generally useful o f the C F A approaches t o

a

M T M M data (see Visser et al., 2 0 0 3 ) .

J a c k s o n , Dunton, & Williams, 1 9 9 5 ; M a c r a e ,

person

from

that

group

(e.g., Fazio,

Bodenhausen, Milne, & Jetten, 1 9 9 4 ) . Also, a variety o f measures have been based on speed

BEYOND SELF-REPORT MEASURES

of responding. F o r example, participants have been asked to evaluate unrelated

words

Throughout much o f the current chapter, we

following race primes. T o the extent that a

have dealt with measures that fall squarely

person's view o f the target

into the category o f "direct"

group is negative, responses are facilitated to

self-report

measures. Such measures are quite frequent in

(stigmatized)

negative targets that follow group

primes

Constructing

and Evaluating

Quantitative

Measures

\

(e.g., Fazio, J a c k s o n , et al., 1 9 9 5 ; also see

by Wegener and Petty ( 1 9 9 8 ) , however, one's

Greenwald, M c G h e e , & Schwartz, 1 9 9 8 ) . O f

choice o f type o f measure would depend a

course, other indirect measures have been

great deal on the type o f "implicit cognition"

developed as well. F o r example, H a m m o n d

one wishes to investigate. Some conceptions o f

( 1 9 4 8 ) developed the information error (error

"implicit cognition" question the ability o f

choice)

respondents to report the content o f their

technique

in w h i c h

respondents

answer seemingly factual questions,

with

views. O n e possible situation where this might

none o f the provided answers being correct.

be suspected is if a person high in internal

T h e direction o f errors is taken as an indica-

motivation

tion o f the person's attitude. Also, physiolog-

reports low levels o f prejudice on explicit

to

avoid

prejudice

privately

ical measures have a rich history as indirect

(direct) measures but still shows prejudice on

measures. Some recent and useful approaches

implicit (indirect) measures (see Nosek, 2 0 0 2 ) .

include use o f Event-Related Potentials (i.e.,

W h e n inability to report content is suspected,

electrical activity in the brain when a target

one would likely choose one or more implicit

object differs in valence from a set o f preced-

(indirect) measures designed to tap into the

ing items) (Cacioppo, Crites, Gardner,

&

suspected prejudiced associations in memory.

Lorig,

In other instances, however, one could still

Nusbaum, & Berntson, Chapter 1 7 , this vol-

often use explicit (direct) measures to study

Berntson,

1 9 9 4 ; see

Cacioppo,

ume) and facial electromyography (Cacioppo

"implicit cognition." This is because many

& Petty, 1 9 7 9 ; see Petty & Cacioppo, 1 9 9 6 ,

studies o f "implicit cognition" address the

for discussion o f other traditional indirect

inability o f people to realize what has

measures). Though less used in the attitudes

enced

area, social psychologists also use "nonself-

views is readily reportable. F o r example,

influ­

their views, even if the content o f their

report" data o f the archival (e.g., see Kerr,

Greenwald and Banaji ( 1 9 9 5 ) described halo

Aronoff, & Messe, 2 0 0 0 ) , observational (e.g.,

effects as implicit when people fail to realize

see Bakeman, 2 0 0 0 ) , and qualitative varieties

that the evaluation o f a novel attribute (e.g.,

(e.g., see King, Chapter 8, this volume).

character) is influenced by a known attribute

By way o f wrapping up this chapter, we

(e.g., physical attractiveness). In such a case,

would like to emphasize that the construction

one could often identify "implicit halo effects"

and validation o f indirect or implicit measures

even if one were

can and should follow the same basic steps

attributes

t o measure

both

the

using explicit (direct) measures,

outlined for direct measures. T h a t is, the con-

because the "implicit" aspect is whether or

struction o f such measures would begin with

n o t people

the specification o f the goals o f the measure

attribute on the other, not in whether people

and the theoretical assumptions about the

can report the content o f the attributes. In

qualities of the construct. As noted earlier, the

fact, it is interesting to note that most o f the

conceptualization o f direct self-report measures

examples o f implicit cognition provided by

realize the

influence

o f one

includes both ability and willingness on the

Greenwald and Banaji

part o f respondents to report the content

before many o f the most recent "implicit"

( 1 9 9 5 ) — a t a time

of their views of a target. In contrast, the call

measures o f content were fully developed—

for indirect measures (especially in areas such

actually utilized direct self-report measures o f

as stereotyping and prejudice) generally has

content. If one is truly interested in implicit

been motivated by concerns about both ability

content (rather than implicit influence), how-

and motivation to report certain attitudes,

ever, direct measures o f content would not

especially toward controversial social groups

suffice (see Wegener &c Petty, 1 9 9 8 ; see also

(e.g., Greenwald & Banaji, 1 9 9 5 ) . As noted

Kihlstrom, Chapter 9, this volume).

165

166

DESIGN AND ANALYSIS Construction o f indirect measures has all

reason, for example, that an error choice

too often stopped with specification o f the

technique ( H a m m o n d , 1 9 4 8 ) would perform

goals and theory. T h a t is, the generation and

much better if one were to start with a larger

evaluation o f specific items have not played

pool o f items that are then put through tradi-

the same kind o f central role in development

tional types o f item analysis. Given the theo-

of indirect measures that they have for tradi-

retical conceptions o f targets or settings in

tional self reports. Y e t , there is little reason

which indirect measures are needed (e.g., in

for this to be the case. F o r example, as aptly

studies o f stereotyping or prejudice), one

noted

would also want to seek validity evidence that

by

Himmelfarb

(1993),

behavioral indices have often

although been con-

goes beyond between-group

differentiation

structed from aggregated sets o f behaviors

(as shown by H a m m o n d , 1 9 4 8 ) . T h a t is, one

(e.g., Tittle &

these indices

would want to explicitly address the ability o f

"would ideally be subjected to the item anal-

the error choice items to overcome social

Hill, 1 9 6 7 ) ,

ysis procedures associated with the

tradi-

desirability concerns. Addressing associative

tional attitude scaling techniques" (p. 6 4 )

and dissociative forms o f validity, one could

(such as Thurstone's use o f judge ratings,

also assess relations between the information

Likert's use o f item-total correlations, or

error measure and existing implicit versus

Osgood's use of factor analyses). Indeed,

explicit scales (perhaps using a full M T M M

reviews o f many o f the older indirect attitude

design) (see Visser et al., 2 0 0 3 ) .

measures concluded that these measures left

One o f the great successes o f social psy-

much to be desired. F o r example, Kidder and

chology has been the development o f valid

our

and reliable measures o f a wide variety o f psy-

imperfect measures, these are apparently not

Campbell

(1970)

noted that

" o f all

chologically meaningful constructs. As these

the least impure" (p. 3 3 6 ) . Y e t , it is not clear

measures continue to expand in exciting and

that many o f even the older indirect measures

interesting ways, traditional methods o f mea-

have gone through the typical item selection

sure development and evaluation form a firm

and evaluation procedures that have been

foundation on which the new measures can

typical for the direct measures. It stands to

be built and tested.

NOTES 1. Sometimes researchers adapt items, complete measures, or methods of administration for their current purposes. When making such changes, one must recognize that these alterations can influence the psychometric properties of the items. Therefore, one should conduct the same types of evaluations of items with these modified measures as with newly constructed items. 2. A related issue that researchers must consider is whether to specify a response scale that includes a midpoint. Unfortunately, studies examining the impact of including scale midpoints on item reliability and validity have produced conflicting findings (see Krosnick & Fabrigar, 1997, in press). Thus, current empirical research does not provide a basis for a simple recommendation regarding the use of scale midpoints. However, a variety of conceptual issues should be considered in making such a decision (see Krosnick &c Fabrigar, 1997, in press). 3. Although various item selection procedures are somewhat distinct, the performance of an item across different selection procedures is not unrelated. An

Constructing

and Evaluating

Quantitative

Measures

item p e r f o r m i n g well o n one procedure often w i l l tend t o p e r f o r m w e l l o n other p r o c e d u r e s . F o r e x a m p l e , t h e p a t t e r n o f results f o r a n i t e m - t o t a l c o r r e l a t i o n analysis w i l l o f t e n suggest s i m i l a r choices w i t h respect t o i t e m selection as a f a c t o r analysis ( G o r s u c h , 1 9 8 3 ) , a l t h o u g h a f a c t o r analysis does p r o v i d e a d d i t i o n a l information. 4 . T h r o u g h o u t o u r discussion o f E F A , w e assume t h a t a researcher has c o n d u c t e d a n E F A based o n t h e c o m m o n f a c t o r m o d e l w i t h a n o b l i q u e r o t a t i o n . W e d o so because w e believe this a p p r o a c h is t h e m o s t a p p r o p r i a t e t y p e o f E F A f o r t h e vast m a j o r i t y o f research questions investigated b y social p s y c h o l o g i s t s ( F a b r i g a r , W e g e n e r , et a l . , 1 9 9 9 ; W e g e n e r & F a b r i g a r , 2 0 0 0 ) . W h e n o t h e r types o f E F A are used ( o r o t h e r types o f analyses, such as p r i n c i p a l c o m p o n e n t s analysis), t h e i n f o r m a t i o n p r o v i d e d is s o m e w h a t d i f f e r e n t f r o m w h a t w e describe here (see G o r s u c h , 1 9 8 3 ) . 5. I f a researcher w e r e t o specify a C F A m o d e l w i t h t h e m a x i m u m n u m b e r o f items w i t h m u l t i p l e f a c t o r l o a d i n g s i n w h i c h t h e m o d e l is still i d e n t i f i e d , this w o u l d be m a t h e m a t i c a l l y i d e n t i c a l t o a n E F A m o d e l w i t h t h e same n u m b e r o f f a c t o r s (see F a b r i g a r , W e g e n e r , et a l . , 1 9 9 9 ) . I n effect, t h e researcher w o u l d be c o n d u c t i n g a n E F A w i t h o u t r o t a t i o n f o r s i m p l e s t r u c t u r e . Such a n a p p r o a c h w o u l d have n o advantages over E F A a n d w o u l d be m u c h m o r e c u m b e r s o m e t o implement and interpret. 6. Equivalence r e l i a b i l i t y generally is d e f i n e d as t h e e x t e n t t o w h i c h t w o p a r a l l e l measures o f t h e same c o n s t r u c t are c o r r e l a t e d w i t h o n e a n o t h e r . I n m a n y respects, equivalence r e l i a b i l i t y is a n a l t e r n a t i v e t y p e o f i n t e r n a l consistency i n w h i c h o n e c o m p a r e s t w o d i s t i n c t sets o f items r a t h e r t h a n a l l possible s p l i t halves o f a single set o f items (as i n C r o n b a c h a l p h a ) . A l t e r n a t i v e l y , equivalence r e l i a b i l i t y c a n be c o n c e p t u a l i z e d as a f o r m o f c o n v e r g e n t v a l i d i t y . I n t r a d i t i o n a l t r e a t m e n t s o f c o n v e r g e n t v a l i d i t y ( C a m p b e l l & Fiske, 1 9 5 9 ) , emphasis is p l a c e d o n assessing the c o r r e l a t i o n b e t w e e n t w o m a x i m a l l y d i f f e r e n t m e t h o d s o f m e a s u r e m e n t . I n c o n t r a s t , m o s t tests o f equivalence e x a m i n e t h e c o r r e l a t i o n b e t w e e n t w o measures t h a t are q u i t e s i m i l a r i n m e t h o d o l o g y . 7. O n e also encounters issues o f i n t r a r a t e r r e l i a b i l i t y (e.g., consistency o f r a t i n g s m a d e o n t w o occasions b y t h e same rater) a n d i n t e r r a t e r r e l i a b i l i t y (i.e., consistency across raters) w h e n raters code o p e n - e n d e d responses b y research p a r t i c i p a n t s (see B a k e m a n [ 2 0 0 0 ] a n d K i n g , C h a p t e r 8, t h i s v o l u m e , f o r d e t a i l e d discussions o f i n t e r r a t e r agreement a n d r e l i a b i l i t y ) .

REFERENCES A b e l s o n , R. P., K i n d e r , D . R . , Peters, M . D . , & Fiske, S. T . ( 1 9 8 2 ) . A f f e c t i v e a n d semantic c o m p o n e n t s i n p o l i t i c a l p e r s o n p e r c e p t i o n . Journal

Social Psychology,

of Personality

and

42, 6 1 9 - 6 3 0 .

A i k e n , L. R. ( 1 9 9 7 ) . Psychological

testing and assessment

( 9 t h ed.). B o s t o n : A l l y n &

Bacon.

A l l e n , M . J . , & Y e n , W . M . ( 1 9 7 9 ) . Introduction

to measurement

theory.

Belmont,

CA: Wadsworth. B a k e m a n , R. ( 2 0 0 0 ) . B e h a v i o r a l o b s e r v a t i o n a n d c o d i n g . I n H . T . Reis &C C . M .

J u d d (Eds.), Handbook

of research methods

in social and personality

psychol­

ogy ( p p . 1 3 8 - 1 5 9 ) . N e w Y o r k : C a m b r i d g e U n i v e r s i t y Press.

B o l l e n , K . A . ( 1 9 8 9 ) . Structural equations

with latent variables.

N e w York: Wiley.

Breckler, S. J . ( 1 9 8 4 ) . E m p i r i c a l v a l i d a t i o n o f affect, b e h a v i o r , a n d c o g n i t i o n as dis-

t i n c t c o m p o n e n t s o f a t t i t u d e . Journal 1191-1205.

of Personality

and Social Psychology,

47,

\

167

168

DESIGN AND ANALYSIS Browne, M . W. (1984). The decomposition of multitrait-multimethod matrices. British Journal of Mathematical and Statistical Psychology, 37, 1-21. Budner, S. (1962). Intolerance of ambiguity as a personality variable. Journal of Personality, 30, 2 9 - 5 0 . Cacioppo, J . T., Crites, S. L., Jr., Gardner, W. L., 8c Berntson, G. G. (1994). Bioelectrical echoes from evaluative categorizations: I. A late positive brain potential that varies as a function of trait negativity and extremity. Journal of Personality and Social Psychology, 67, 115-125. Cacioppo, J . T., Martzke, J . S., Petty, R. E., 8c Tassinary, L. G. (1988). Specific forms of facial E M G response index emotions during an interview: From Darwin to the continuous flow hypothesis of affect-laden information processing. Journal of Personality and Social Psychology, 54, 5 9 2 - 6 0 4 . Cacioppo, J . T., 8c Petty, R. E. (1979). Attitudes and cognitive response: An electrophysiological approach. Journal of Personality and Social Psychology, 37, 2181-2199. Cacioppo. J . T., 8c Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116-131. Campbell, D. T., 8c Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Campbell, D. T., 8c O'Connell, E. J . (1967). Method factors in multitrait-multiBehavioral method matrices: Multiplicative rather than additive? Multivariate Research, 2, 4 0 9 - 4 2 6 . Chaiken, S., Liberman, A., & Eagly, A. H. (1989). Heuristic and systematic processing within and beyond the persuasion context. In J . S. Uleman 8c J . A. Bargh (Eds.), Unintended thought (pp. 2 1 2 - 2 5 2 ) . New York: Guilford. Converse, P. E. (1964). The nature of belief systems in the mass public. In D. E. Apter (Ed.), Ideology and discontent (pp. 2 0 6 - 2 6 1 ) . New York: Free Press. Crites, S. L., Jr., Fabrigar, L. R., 8c Petty, R. E. (1994). Measuring the affective and cognitive properties of attitudes: Conceptual and methodological issues. Personality and Social Psychology Bulletin, 20, 6 1 9 - 6 3 4 . Cronbach, L. J . (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 2 9 7 - 3 3 4 . Cronbach, L. J . , 8c Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 2 8 1 - 3 0 2 . Cunningham, W. A., Preacher, K. J . , 8c Banaji, M. R. (2001). Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science, 12, 163-170. Davidson, A. R., 8c Jaccard, J . (1979). Variables that moderate the attitude-behavior relation: Results of a longitudinal survey. Journal of Personality and Social Psychology, 37, 1 3 6 4 - 1 3 7 6 . Doll, J . , 8c Ajzen, I. (1992). Accessibility and stability of predictors in the theory of planned behavior. Journal of Personality and Social Psychology, 63, 754-765. Eagly, A. H., Mladinic, A., 8c Otto, S. (1994). Cognitive and affective bases of attitudes toward social groups and social policies. Journal of Experimental Social Psychology, 30, 113-137. Edwards, K. (1990). The interplay of affect and cognition in attitude formation and change. Journal of Personality and Social Psychology, 59, 2 0 2 - 2 1 6 . Embretson, S. E., 8c Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum. Fabrigar, L. R., 8c Petty, R. E. (1999). The role of the affective and cognitive bases of attitudes in susceptibility to affectively and cognitively based persuasion. Personality and Social Psychology Bulletin, 25, 3 6 3 - 3 8 1 .

Constructing

and Evaluating

Quantitative

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C , 8c Strahan, E. J . (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 2 7 2 - 2 9 9 . Fazio, R. H., Jackson, J . R., Dunton, B. C , 8c Williams, C. J . (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69, 1013-1027. Fazio, R. H., & Zanna, M . P. (1981). Direct experience and attitude-behavior consistency. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 14, pp. 161-202). New York: Academic Press. Festinger, L., 8c Carlsmith, J . M . (1959). Cognitive consequences of forced compliance. Journal of Abnormal and Social Psychology, 58, 2 0 3 - 2 1 0 . Finch, J . F., 8c West, S. G. (1997). The investigation of personality structure: Statistical models. Journal of Research in Personality, 31, 4 3 9 - 4 8 5 . Floyd, F. J . , 8c Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286-299. Friedenberg, L. (1995). Psychological testing: Design, analysis, and use. Boston: Allyn 6c Bacon. Gaito, J . (1980). Measurement scales and statistics: Resurgence of an old misconception. Psychological Bulletin, 87, 564-567. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: Attitudes, selfesteem, and stereotypes. Psychological Review, 102, 4-27. Greenwald, A. G., McGhee, D. E., 8c Schwartz, J . L. K. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74, 1 4 6 4 - 1 4 8 0 . Hammond, K. R. (1948). Measuring attitudes by error-choice: An indirect method. Journal of Abnormal and Social Psychology, 43, 38-48. Himmelfarb, S. (1993). The measurement of attitudes. In A. H. Eagly &C S. Chaiken, The psychology of attitudes (pp. 23-88). Fort Worth, T X : Harcourt Brace Jovanovich. Izard, C. E. (1977). Human emotions. New York: Plenum. Jackson, D. N. (1971). The dynamics of structured personality tests. Psychological Review, 78, 2 2 9 - 2 4 8 . Janada, L. H. (1998). Psychological testing: Theory and applications. Boston: Allyn 8c Bacon. John, O. P., &C Benet-Martinez, V. (2000). Measurement: Reliability, construct validation, and scale construction. In H. T. Reis 8c C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 3 3 9 - 3 6 9 ) . New York: Cambridge University Press. Joreskog, K. G. (1974). Analyzing psychological data by structural analysis of covariance matrices. In R. C. Atkinson, D. H. Krantz, R. D. Luce, 8c P. Suppes (Eds.), Contemporary developments in mathematical psychology (Vol. 2, pp. 1-56). San Francisco: W. H. Freeman. Katz, D., 6c Stotland, E. (1959). A preliminary statement to a theory of attitude structure and change. In S. Koch (Ed.), Psychology: A study of a science: Vol. 3. Formulations of the person and the social context (pp. 4 2 3 - 4 7 5 ) . New York: McGraw-Hill. Kenny, D. A., 8c Kashy, D. A. (1992). Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172. Kerr, N. L., Aronoff, J . , 8c Messe, L. A. (2000). Methods of small group research. of research methods in In H. T. Reis 8c C. M . Judd (Eds.), Handbook

Measures

170

DESIGN AND ANALYSIS social and personality psychology (pp. 160-189). New York: Cambridge University Press. Kidder, L. H., 8c Campbell, D. T. (1970). The indirect testing of social attitudes. In G. F. Summers (Ed.), Attitude measurement (pp. 3 3 3 - 3 8 5 ) . Chicago: Rand McNally. Knowles, E. S. (1988). Item context effects on personality scales: Measuring and Social Psychology, 55, changes the measure. Journal of Personality 312-320. Krosnick, J . A., & Fabrigar, L. R. (1997). Designing rating scales for effective measurement in surveys. In L. Lyberg, P. Biemer, M . Collins, E. De Leeuw, C. Dippo, N. Schwarz, & D. Trewin (Eds.), Survey measurement and process quality (pp. 141-164). New York: Wiley. Krosnick, J . A., & Fabrigar, L. R. (in press). Designing questionnaires to measure attitudes. New York: Oxford University Press. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 44-53. Loevinger, J . (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 635-694. Lord, F., & Novick, M . R. (1968). Statistical theories of mental tests. New York: Addison-Wesley. Macrae, C. N., Bodenhausen, G. V., Milne, A. B., 8c Jetten, J . (1994). Out of mind but back in sight: Stereotypes on the rebound. Journal of Personality and Social Psychology, 67, 808-817. Marsh, H. W., 8c Grayson, D. (1995). Latent variable models of multitrait-multiConcepts, method data. In R. H. Hoyle (Ed.), Structural equation modeling: issues, and applications (pp. 177-198). Thousand Oaks, CA: Sage. McKinley, R. L., & Mills, C. N. (1989). Item response theory: Advances in achievement and attitude measurement. In B. Thompson (Ed.), Advances in social science methodology (Vol. 1, pp. 71-135). Greenwich, CT: JAI. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: Macmillan. Millsap, R. E. (1992). Sufficient conditions for rotational uniqueness in the additive M T M M model. British Journal of Mathematical and Statistical Psychology, 45, 125-138. Mueller, D. J . (1986). Measuring social attitudes: A social handbook for researchers and practitioners. New York: Teachers College Press. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal report on mental processes. Psychological Review, 84, 2 3 1 - 2 5 9 . Nosek, B. (2002). Moderators of the relationship between implicit and explicit atti­ tudes. Unpublished doctoral dissertation, Yale University. Osgood, C. E., Suci, G. J . , 8c Tannenbaum, P. H. (1957). The measurement of meaning. Urbana: University of Illinois Press. Ostrom, T. M . (1989). Interdependence of attitude theory and measurement. In A. R. Pratkanis, S. J . Breckler, & A. G. Greenwald (Eds.), Attitude structure and function (pp. 11-36). Hillsdale, NJ: Erlbaum. and persuasion: Central and Petty, R. E., 8c Cacioppo, J . T. (1986). Communication peripheral routes to persuasion. New York: Springer-Verlag. Petty, R. E., 8c Cacioppo, J . T. (1996). Attitudes and persuasion: Classic and con­ Boulder, CO: Westview. temporary approaches. Petty, R. E., Haugtvedt, C. P., 8c Smith, S. M. (1995). Elaboration as a determinant of attitude strength. In R. E. Petty &c J . A. Krosnick (Eds.), Attitude strength: Antecedents and consequences (pp. 93-130). Mahwah, NJ: Lawrence Erlbaum.

Constructing

and Evaluating

Quantitative

Remmers, H. H. (1963). Rating methods in research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching (pp. 3 2 9 - 3 7 8 ) . Chicago: Rand McNally. Rosenthal, R., & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York: McGraw-Hill. Rotter, J . B. (1966). Generalized expectancies for internal versus external control of reinforcement. Psychological Monographs, 80(1, Whole No. 6 0 9 ) . Russell, J . A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1 1 6 1 - 1 1 7 8 . Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 3 5 0 - 3 5 3 . Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. San Diego: Academic Press. Schwarz, N., & Hippler, H. J . (1991). Response alternatives: The impact of their choice and ordering. In P. Biemer, R. Groves, N. Mathiowetz, & S. Sudman (Eds.), Measurement error in surveys (pp. 41-56). Chichester, UK: Wiley. Steinberg, L., & Thissen, D. (1995). Item response theory in personality research. In P. E. Shrout & S. T. Fiske (Eds.), Personality research, methods, and the­ ory: A festschrift honoring Donald W. Fiske (pp. 1 6 1 - 1 8 1 ) . Hillsdale, NJ: Lawrence Erlbaum. Steyer, R., & Schmitt, M. J . (1990). The effects of aggregation across and within occasions on consistency, specificity, and reliability. Methodika, 4, 5 8 - 9 4 . Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass. Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 5 2 9 - 5 5 4 . Thurstone, L. L., &c Chave, E. J . (1929). The measurement of attitude. Chicago: University of Chicago Press. Tisak, J . , & Tisak, M . S. (2000). Permanency and ephemerality of psychological measures with application to organizational commitment. Psychological Methods, 5, 175-198. Tittle, C. R., & Hill, R. J . (1967). Attitude measurement and prediction of behavior: An evaluation of conditions and measurement techniques. Sociometry, 30, 199-213. Townsend, J . T., &c Ashby, G. (1984). Measurement scale and statistics: The misconception misconceived. Psychological Bulletin, 96, 3 9 4 - 4 0 1 . Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10. Visser, P. S., Fabrigar, L. R., Wegener, D. T., & Browne, M. W. (2003). Analyzing social psychological multitrait-multimethod data. Unpublished manuscript, University of Chicago. Weary, G., & Edwards, J . A. (1994). Individual differences in causal uncertainty. Journal of Personality and Social Psychology, 67, 3 0 8 - 3 1 8 . Webster, D. M., & Kruglanski, A. W. (1994). Individual differences in need for cognitive closure. Journal of Personality and Social Psychology, 67, 1049-1062. Wegener, D. T., Downing, J . , Krosnick, J . A., &: Petty, R. E. (1995). Measures and manipulations of strength-related properties of attitudes: Current practice and future directions. In R. E. Petty &c J . A. Krosnick (Eds.), Attitude strength: Antecedents and consequences (pp. 4 5 5 - 4 8 7 ) . Mahwah, NJ: Lawrence Erlbaum.

Measures

DESIGN AND ANALYSIS Wegener, D. T., & Fabrigar, L. R. (2000). Analysis and design for nonexperimental data: Addressing causal and noncausal hypotheses. In H. T. Reis 8c C. M . Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 412-450). New York: Cambridge University Press. Wegener, D. T., 8c Petty, R. E. (1998). The naive scientist revisited: Naive theories and social judgment. Social Cognition, 16, 1-7. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitraitmultimethod data. Applied Psychological Measurement, 9, 1-26.

C H A P T E R

8

Measures and Meanings The Use of Qualitative Data in Social and Personality Psychology LAURA A . KING University of Missouri,

Q

Columbia

ualitative data typically are defined

prospects, which are the most important for

as unstructured sources o f informa-

study. A useful distinction t o be made is that

tion that do not lend themselves

between

readily to quantification. Such data

approaches. In the top-down approach, the

may emerge from interviews, written answers

researcher comes to the data interested in

to questions, videotaped conversation, and

examining a theoretically derived construct or

other sources. Relative to more

structured

hypothesis. O f course, it is hoped that any

data, gathering these data typically involves a

person conducting any empirical study has a

larger amount o f time and effort on the part of

theory that drives that particular investiga-

participants and investigators alike. As a result,

tion. However, in all research, qualitative or

"top-down"

and

"bottom-up"

there is certainly a temptation to forgo the use

otherwise, the data sometimes present inter-

of such data. There are times, however, when

esting new dilemmas. A bottom-up approach

qualitative data

appropriate

to qualitative data involves coming to the

are the only

means to answer the scientist's questions.

data themselves to see what's there. Although

Although open-ended measures may be very

this sort o f procedure may appear alarmingly

attractive to those w h o are fascinated by the

post h o c , it is worthwhile to note that no

immediacy of human experience that such

researcher has unlimited foresight and that it

measures are more likely to convey, the onus o f

is sometimes worthwhile to examine surprises

responsibility is on the researcher to demon-

(as is often the case in more straightforward

strate that all the work is

quantitative investigations, to be sure).

qualitative

measures

warranted—that

provide

something

beyond what might have been accomplished with more straightforward measures.

Some areas o f social science (and

the

humanities) regard its "unquantiftable" nature as essential to qualitative data. Sociologists

In approaching qualitative data, one must

and anthropologists may routinely come to

decide, from among a very rich array o f

their data and engage in a more intuitive or

174

DESIGN AND ANALYSIS intellectual "data processing" in order to come

scientific

to conclusions. Although such analyses remain

1999),

abstracts

subject to scientific rigor (Altheide & Johnson,

(Bonanno et al., 2 0 0 2 ; Dovidio, Kavakami, &

facial

(Pennebaker &

pictures,

and

King,

videotapes

1 9 9 4 ) , concern for issues of reliability, sample

Gaertner, 2 0 0 2 ; Keltner & Bonanno, 1 9 9 7 ) to

size, generalizability, replicability, establishing

examine important independent and depen-

causality, and control groups are not as press-

dent variables. However and importantly, in

ing as they typically are in social psychology

social psychological inquiries, qualitative data

(Blaikie, 2 0 0 0 ; Vidich & Lyman, 1 9 9 4 ) . These

typically are conceived in terms o f quantifiable

differences in emphases are

research goals, and they are quickly trans-

given

the

heightened

understandable

interest

of

these

formed into quantitative data for analyses.

researchers in more contextualized phenomena. Research in these areas is more likely to be characterized as reflecting a level o f comfort with, and confidence in, more

bottom-up

approaches to analyses. Though top-down

QUALITATIVE D A T A IN SOCIAL PSYCHOLOGY: AN EMPIRICAL EXAMPLE

approaches are also used, even in these contexts, questions that are "second nature" to

A study by Stirman and Pennebaker ( 2 0 0 1 )

social psychologists

provides

(e.g., replicability o f

results) are rarely major concerns (Denzin,

an

illustration

o f the

contrast

between typical uses of qualitative data in

1 9 8 3 ; Guba & Lincoln, 1 9 8 9 ; Phillips, 2 0 0 0 ) .

other fields and the approach used by social

In contrast, purely qualitative inquiries are

psychologists. These researchers were inter-

rare in personality and social psychology.

ested in examining the apparent tendency o f

Perhaps the use o f psychobiography in per-

poets to commit suicide. Previous research in

sonality psychology is an exception (see Elms,

this area was more typical o f purely qualita-

1994;

and

tive research (focusing on only on a single

Runyan,

1 9 8 2 ) . In personality

social psychology, qualitative data typically

poet's work, examining a few poems from the

reside alongside more structured data (e.g.,

individual's oeuvre, and using a single rater—

questionnaires or laboratory manipulations).

usually the author of the research). N o con-

Although qualitative data may represent the

trol groups were used (e.g., Hoyle, 1 9 6 8 ;

centerpiece o f a program of research, such

Lester,

data rarely are treated in purely qualitative

( 2 0 0 1 ) adopted a more typical social psycho-

1994).

Stirman

and

Pennebaker

ways. Nevertheless, it is worth noting h o w

logical approach to this question, using qual-

much qualitative data have become incorpo-

itative data

rated into research in personality and social

content analyzed poems from three different

psychology. M a n y procedures

have

life periods (early, mid, and late career) for

become essential parts o f our methodological

each o f a sample o f 1 8 poets. A control group

toolbox involve the problem o f transforming

of nine nonsuicidal poets was matched with

unstructured free responses into quantifiable

nine suicidal poets for age, nationality, and

units. Researchers in social psychology have

m o o d disorder (which is also, apparently,

turned to personal narratives (Baumeister,

fairly

Wotman,

Pennebaker hypothesized that suicidal poets

&

Stillwell,

that

1 9 9 3 ; Georgeson,

in a quantitative

common

in

poets).

way. They

Stirman

and

diary entries,

would distinguish themselves from nonsuici-

idiographic goals (e.g., Emmons 8c King,

dal poets by various aspects o f the language

Harris, Milich, & Young, 1999),

1 9 8 8 ) , dyadic interactions (Berry &c Miller,

used in their poems. Specifically, they pre-

2 0 0 1 ) , historical documents (Winter, 1 9 9 2 ) ,

dicted that suicidal poets would show tenden-

poems

cies toward

(Stirman

&

Pennebaker,

2001),

self-absorption,

preoccupation

Measures

and

Meanings

with death, and social detachment in their

authentic) smiling and laughter and found,

poems, compared to their nonsuicidal coun-

surprisingly, that instances o f such smiling

terparts. Using a computerized word count

even during a conversation about bereave-

system, these researchers found support for

ment predicted better adjustment at a later

their hypotheses. First, suicidal poets were

time. Qualitative data allow us to examine

more likely than nonsuicidal poets to use first

questions not only o f content but also o f

person references throughout their careers.

intensity, style, mannerism, and so on. W e

These poets also showed a tendency toward

can examine not only what was said but how

using more death-related words. Even more

it was said. Qualitative data simply provide a

striking, suicidal poets tended to increase in

richer, more varied pool o f information.

social detachment, as manifested in a decrease in the use o f words such as " u s , " " w e , " and " o u r " as they approached the end o f their lives. This investigation shows attention to the typical concerns o f social psychologists in any study, but qualitative data clearly occupy center stage.

Qualitative Data Allow Us to Measure What Isn't Said or Can't Be Said Considering variables such as dissociation, denial, and defensiveness, we can certainly posit that one of the defining features o f these variables is that people don't know they are doing

ADVANTAGES OF ASKING

them. They are non-conscious. Qualitative

OPEN-ENDED QUESTIONS

measures have been especially useful in tapping variables that participants cannot or will not

Although they present some obvious chal-

report on with accuracy. The problem o f non-

lenges, there are some clear advantages to

conscious psychological processes led to the

asking one's open-ended research questions.

development of projective techniques for use in measuring unconscious motives (Morgan &

Qualitative Data May Answer Many Questions at Once

Murray, 1 9 3 5 ) . O f course, the use of projective techniques has been the subject o f a great deal of debate over the years. Sidestepping that

Open-ended responses, like autobiograph-

debate entirely, it remains clear that, at times,

ical stories or interview responses, allow us to

even in laboratory research, variables that the

examine a variety o f psychological processes

investigator would like to measure are not

that may be involved in a particular experi-

always easily tapped using

ence,

automatically.

manipulations or questionnaires. Even if they

Processes such as dissociation, denial, and

are aware o f their motives or values, partici-

simultaneously

and

straightforward

defensiveness can be seen to occur alongside

pants may be too embarrassed, ashamed, or

other coping processes and may be particu-

concerned with the impression they are making

larly difficult to measure using more direct

to respond honestly to direct questions.

means. For example, in examining widows

McClelland ( 1 9 8 0 ; McClelland, Koestner,

talking about the death o f a spouse, Keltner

&C Weinberger, 1 9 8 9 ) drew the distinction

and B o n a n n o ( 1 9 9 7 ) looked not only at what

between respondent and operant behaviors.

the women said but also at facial expressions

Respondent behaviors are those that are per-

that occurred spontaneously and simultane-

formed self-consciously and that may involve

ously with those verbalizations. In examining

a person's awareness that he or she is acting

these data, Keltner and B o n a n n o coded for

consistently with his or her own values. An

the occurrence o f Duchenne (i.e., genuine or

example o f a respondent behavior would be a

175

176

DESIGN AND ANALYSIS response to a questionnaire item (e.g., " H o w

been measured

positively do you generally feel about h o m o -

qualitative data. Without qualitative data, the

using anything other than

sexuals?"). Operant behaviors are performed

conclusions o f the investigation would have

unself-consciously and spontaneously. W h e n

been quite different—that people's explicitly

we are interested in operant behaviors, we

stated attitudes relate to their friendliness dur-

may need to resort to qualitative data. An

ing interactions with others. T h e inclusion o f

example o f this sort o f behavior is a person's

qualitative data allowed for a more accurate

spontaneous stylistic behavior in an interac-

depiction

tion with a member o f an out-group.

explicit attitudes are conveyed in behavior,

An example is provided by research by

o f the ways that

implicit

and

sometimes subtle behavior, toward another.

Dovidio and colleagues on the relation o f racial prejudice measures to actual behavior by European Americans in interactions with African

Americans. Dovidio et al.

(2002)

were interested in the implications o f racial

Qualitative Data Give Us the Flavor of the Whole In addition

to

allowing access to

the

attitudes for behaviors during actual interac-

unconscious and the unspoken,

tion with members o f an out-group. In this

data allow for a sense o f the coherence o f

study,

human experience. Asking only highly struc-

the

racial attitudes

of

European

qualitative

American participants were measured in two

tured questions constrains our capacity as

ways. First, participants completed explicit

researchers to fully tap into the human expe-

questionnaire measures o f prejudice. Second,

rience o f the variables o f interest. Although

they completed

we c a n certainly gain

an implicit attitudes

test,

enormously

from

using a reaction time measure that tested the

purely

ease with which participants associated posi-

same time those aspects o f an individual that

tive and negative terms with black and white

qualitative data are so useful for gleaning—

faces. N e x t , the participants interacted with

issues o f style, spontaneity, intensity, and the

quantitative

investigations, at

the

white and black confederates while being

embeddedness o f a phenomenon in the psy-

videotaped. Participants also rated the level o f

chological life o f the person—can be lost. It

friendliness that they believed they conveyed

has often been said that humans make mean-

during the interaction. Measures o f behavior

ing by telling stories. Collecting accounts o f

during the interactions were gleaned via con-

personal experience is a way o f collecting

tent analysis o f the participants' verbal pro-

units o f meaning.

ductions during the interaction (i.e., what

M y own research interests are in the area of

they actually said) as well as their nonverbal

personality, motivation, and

behaviors

ing. I became particularly interested in how

from

(e.g., seating position,

confederate,

distance

etc.). Interestingly,

meaning-mak-

the

individuals experience changes in themselves

more respondent or explicit measure o f prej-

during and after important life transitions or

udice related systematically to the partici-

traumatic

pants' self-rated friendliness as well as to the

characteristics measured

positivity o f participants' explicit verbal pro-

haven't been shown to reflect much change in

ductions during the interaction. However, the

personality over time, so looking for change

more implicit attitudinal measure related sys-

or development through such questionnaires

tematically to participants' nonverbal behav-

didn't seem like a promising approach. Using

ior as well as to confederates' ratings o f

open-ended questions—about the stories of

participant friendliness. Importantly, aspects

people's life experiences—clearly emerged as the

of this dependent measure could not have

ideal, if sometimes challenging, methodology.

life events. Clearly, personality via

questionnaires

Measures

and

Meanings

Throughout this chapter, I will make use o f

written or transcribed protocols or videotaped

examples from my work, especially from a

interactions simply exist—they simply are.

study o f parents o f children with D o w n ' s

Even if the questions that drove their generation

syndrome (DS) (King, Scollon, Ramsey, &

fade from interest, these productions

Williams, 2 0 0 0 ) . In that study, we were inter-

always be scored for the current research issue,

ested in examining h o w the stories that these

whatever it is. Research by M c A d a m s and col-

parents told about the important life transi-

leagues on the intimacy motive provides an

tion o f discovering they would be rearing a

excellent example, using old stories and recod-

child with D S would relate to aspects o f their

ing them with a new system.

well-being

and

personality

can

development.

M c A d a m s ( 1 9 8 0 ) designed the intimacy

Participants in the study were 8 7 parents

motive scoring system to measure an individ-

w h o provided

sometimes quite compelling

ual's recurrent concern for warm interpersonal

accounts o f this life experience and completed

encounters for their own sake. This coding

measures o f psychological well-being

and

scheme answered a gap in the motivation liter-

personality development. Excerpts from their

ature on the human need for affiliation (which

narratives demonstrate the kind o f power that

tended

can be conveyed in qualitative data.

to

emphasize

more

instrumental

attempts to create and preserve existing interpersonal bonds). This coding scheme was

The first 2 4 hours we were led to believe that our daughter was so bad off that we actually prayed to God to take her from us now versus later. A wave of feelings passed over me: shock, fear, and tremendous sadness and protectiveness toward my son. My heart felt as though it would break . . . Could our family face this sadness? Although all three o f these parents may have responded with high ratings to a questionnaire item asking if they had experienced distress upon learning o f their

children's

diagnoses, even a 7 on a 7-point scale cannot convey the vividness o f these narratives.

developed at least two decades after most o f the previously designed motivational coding systems. M c A d a m s and Bryant ( 1 9 8 7 ) were interested in examining h o w this (at the time) newly developed intimacy motive would relate to important life outcomes. Fortunately, longitudinal data had been collected on a group o f more than 1,000 participants w h o had generated imaginative stories in response to pictures for studies on other motives. M c A d a m s and Bryant again content analyzed these stories specifically for intimacy motive imagery. They found that intimacy motivation was related to positive life outcomes (heightened happiness and need gratification) for women and lack o f strain and lack o f uncertainty for men. This study demonstrates h o w qualitative data can be revisited with the changing interests of researchers. T h e data provide an

Qualitative Data Are (Relatively) Timeless

enduring

resource o f information. Historical documents also can provide rich

Another advantage o f open-ended responses

sources o f data for the social and personality

is that we don't have to decide right away what

psychologist interested in qualitative analyses.

we want to know from these data. Question-

For instance, Winter ( 1 9 9 2 ) examined the

naire data run the risk of becoming seriously

inauguration speeches o f American presidents

outdated. Changes in the meanings of items

for motivational content. H e found that vari-

and traits cause problems in revisiting a data

ous motive configurations were associated

set after many years. Qualitative data such as

with particular

historical events in

one's

177

DESIGN AND ANALYSIS

178

presidency (e.g., high affiliative concern was

important to consider the ramifications o f the

associated with frequency o f scandal; high

context o f the initial data collection.

power was associated with being viewed as a successful leader). Another example o f the multiple uses o f one set o f qualitative data is provided by a special section o f the Journal

of

METHODS OF QUALITATIVE RESEARCH

Personality

edited by Folkman

T h e use o f qualitative data in a study has

( 1 9 9 7 ) . All the articles in this special section

implications at nearly every stage o f a

used the same set o f qualitative data, namely,

research project. T h e special requirements o f

interviews from a sample o f bereaved gay

these data must be considered as decisions

men w h o had lost their longtime partners to

are made with regard to framing research

AIDS. All the participants in this study had

questions, designing materials,

participated in a longitudinal study o f adap-

participants, data coding and analyses, and

and

Social

Psychology

recruiting

tation to bereavement. All had been inter-

finally interpretation and communication o f

viewed,

interview

results. In the next few sections, I will review

different

some o f the issues that should be considered

and

protocols

these

were

transcribed

given

to

four

research teams, each o f w h o m analyzed them from its own

perspective—engaging

in such an investigation.

in a

Rashomon-style exchange over the same data set. All the approaches demonstrated some important relations, all using different coding methods and schemes. Such a convergence would

have been impossible without

Participant Selection and Recruitment Some investigations that use

qualitative

the

methods are certainly applicable to the "cap-

incandescence that is part and parcel o f qual-

tive audience" o f a university participant pool.

itative data.

For instance, diary methods, which require

A caveat is appropriate at this point. T h e stories

used

by

McAdams

and

were imaginative stories told in

Bryant

daily contact of some sort with the study, are more readily performed with individuals who

response

are likely, by virtue of necessity, to be in close

(TAT)

physical proximity to the site of the study. At

(Morgan & Murray, 1 9 3 5 ) pictures. As such,

some institutions, it is possible to collect quali-

to

Thematic

Apperception

Test

they may be somewhat decontextualized nar-

tative data over the course o f a semester as part

ratives, created "on the spot" in response to

of a class, centered around the daily study

somewhat arbitrary stimuli. In addition, the

(Emmons & King, 1 9 8 8 ) . Although using such

interviews used in the bereavement project

samples is enormously convenient, Web-based

had all been collected within a few years o f the

data collection might well allow researchers to

coding that was conducted on those data.

begin to include less studied samples in diary

Although I have stated rather boldly that qual-

and experience sampling studies.

itative data simply "are," it is certainly impor-

Given the added time that typically is

tant to consider the historical and cultural

required for participants in the collection of

contexts o f these data. Just as self-report traits

qualitative data, some special consideration

may change in meaning, so might words used

should be given to participant compensation.

in the natural language of participants. In

In work with nonstudent samples, monetary

addition, responses to particular stems may

compensation often is a necessity. Furthermore,

not be appropriate for use in any investigation.

presenting a study to community adults may

W h e n revisiting a qualitative data archive, it is

require differing recruitment techniques. In our

Measures and Meanings

\

work, we have used newspaper ads but also

a life experience. In such cases, prodding may

have found it useful to visit participant groups

be inappropriate.

of interest wherever they might congregate

Qualitative data also m a y be collected

(e.g., support group or informational meetings

through channels other than the verbal one.

for parents of children with Down's syndrome;

In this case, videotaping may be necessary.

gay advocacy groups, gay bars, and bookstores

Videotaped interactions have proven to be a

for gay men and lesbians; support groups for

rich source o f qualitative data.

divorced women). Often, it is important to

again, it is often necessary t o assign individu-

However,

inform participants that some of the questions

als, couples, or groups a task t o get them

they will be answering are open-ended. These

started, so that the activity o f interest will

questions likely will add to the time to complete

actually find its way o n t o the videotape. I f a

the study because of lengthy interviews or the

variable is thought t o be o f particular rele-

need for written responses. However, it is also

vance only during times o f stress, it may be

worth noting that often it is within the open-

necessary t o create a stressful atmosphere in

ended questions that participants find the most

order to gauge its importance (Campbell,

engaging and interesting aspects o f our work.

Simpson, Kashy, & Rholes, 2 0 0 1 ) .

M a n y times, I have received letters or notes from participants commenting on the value that answering these questions had for them. W e typically try to present our work as focusing on finding out what participants already know—that we are simply trying to acquire an understanding o f what their life experiences have shown them. W e are asking participants to share important aspects o f their lives with us; therefore, establishing a sense o f trust and dispelling (potentially legitimate) concerns over psychologists' tendency to pathologize are crucial.

Deciding What Questions to Ask and How to Ask Them T h e type o f methodology used to collect

Coding the Data As I've already mentioned, in the process o f social psychological inquiry, the movement

from

qualitative

to

quantitative

approaches is extremely quick. T h u s , having an idea o f h o w one plans to treat the qualitative data will help in selecting the questions to ask and the types o f responses that will be most worthwhile. I f responses are to be quantified using a coding scheme, the question that emerges is, which one?

Extant

Coding

Schemes

A variety o f coding schemes have been developed for analyzing narrative and other

the data itself is obviously a key concern.

qualitative

Logistical issues such as time constraints and

scheme for the purposes o f content analysis is

data.

Creating a new

coding

person-hours clearly play a role in these deci-

not terribly different

sions. M o r e important, concern for the match

maligned practice o f proliferating self-report

from

the

rightfully

between h o w one has conceptualized a vari-

measures. It is certainly desirable to consult

able and h o w it might be measured ought to

existing

drive the design o f study. Some variables may

embarking on the creation o f a new system.

be better suited to an interview format in

Using an e x t a n t content-analytic scheme

which an interviewer can prompt participants

allows one to take advantage o f a long history

content-analytic

schemes

before

for more information when necessary. At

of trial and error by previous researchers. In

other times, a researcher might be interested

addition, these schemes often have a body o f

in the participants' own preexisting stories o f

literature behind them offering

persuasive

179

180

DESIGN AND ANALYSIS evidence o f reliability and validity. Finally,

Creating

New Coding

Schemes

many extant schemes have been published along with expertly scored practice materials

Although there clearly is a wealth o f exist-

that are invaluable in learning and eventually

ing coding schemes, there are times when a

training teams o f coders.

theoretically driven research question requires

An impressive compilation o f thematic

the development o f a new coding scheme. For

coding systems was published in a volume

instance, in a study o f the relations o f private

edited by Smith ( 1 9 9 2 ) . This handbook is an

wishes to personality traits (King & Broyles,

excellent resource for researchers interested

1 9 9 7 ) , we collected three wishes from each o f

in learning and teaching a variety o f coding

more than 4 0 0 undergraduates w h o had also

schemes. Chapters include theoretical treat-

completed trait measures of the big five per-

ments, literature reviews, and coding manuals

sonality factors (neuroticism, extraversion,

for implicit motives (e.g., achievement, power,

openness to experience, agreeableness, and

affiliation, affiliative trust-mistrust, intimacy),

conscientiousness) (Costa & M c C r a e , 1 9 8 8 ) .

attributional and cognitive orientations (e.g.,

T h e Five Factor M o d e l ( F F M ) approach

personal causation, explanatory style, integra-

posits that traits have motivational properties

tive complexity), and psychosocial orienta-

so that they ought t o be represented in moti-

tions (e.g., psychological stances toward the

vational tendencies. O u r study o f private

environment, responsibility). Expertly scored

wishes sought to examine whether

practice materials are provided for every cod-

might be evident in the content o f these flights

ing scheme. In my experience, graduate and

of fancy. W e developed a coding scheme

traits

undergraduate students alike can be trained to

specifically for content relevant to the F F M .

score narrative protocols reliably and

effi-

T o develop this system, we gave our coders a

ciently using this handbook. T h e Smith vol-

brief workshop on the F F M . W e gave them

ume, however, is hardly exhaustive. As with

sample items from the short questionnaire

any study, a good rule o f thumb is to check the

measure o f the F F M to examine so that they

literature prior to collecting data in order t o

would have a strong understanding o f each o f

explore preexisting coding strategies.

the five traits. Finally, the raters categorized

Coding systems for other types o f data also are extant. For instance, the Facial Action Coding System (FACS) (Ekman &c Friesen, 1 9 7 8 ) codes for particular muscle movements in the face. This system provides a level analysis o f facial movements especially as they are associated with particular emotional expressions. M o r e global systems also have been developed that allow for a less fine-grained analysis o f facial expressions. Social psychologists have developed methods o f coding for body movements, attractiveness, deception, and

other nonverbal

behaviors.

Although

many systems allow one to learn a system independently,

workshops

that

allow for

training in various methods may also provide an important resource for acquiring the skills to use qualitative data effectively.

each wish as relevant to one or more o f the five traits. As predicted, the content o f wishes related to personality traits. For instance, people high in neuroticism were more likely to wish to stop worrying. Those high in extraversion made more impulsive wishes (e.g.,

"an

unlimited

supply

of

beer").

Agreeable folks were more likely to make altruistic wishes and wishes for peace and harmony (e.g., "for the world to be a safer, friendlier p l a c e " ) . T h e highly conscientious were more likely to make wishes for achievement, and those high in openness to experience made nonconformist, highly intellectual wishes (e.g., "to do away with social conformity," " t o be fluent in six languages"). N o t e that the construction o f this coding scheme was helped immensely by the existence of

Measures

and

Meanings

extremely well-specified definitions o f the

be straightforward and understandable, for

traits in question.

both

In developing an ad hoc coding scheme, a

raters

and

eventual

readers

of

a

manuscript. It is desirable to avoid too much

variety o f issues present themselves. H o w does

overlap in dimension content, to avoid both

one go about constructing a reliable coding

confusion for raters and lack o f discriminant

scheme that is comprehensible to coders, that is

validity in analyses. Removing redundancy in

likely to be represented in the data collected

rating categories is a good idea because it is

with adequate variance, and that taps into

not at all unlikely that coders will start to rate

the construct o f interest? One possibility is

similar dimensions in a simultaneous and (per-

to consult the history of the construction o f

haps) haphazard way (for instance, if raters

such

code for " j o y " and "happiness"). T r y to antic-

schemes in

personality

psychology.

Historically, personality psychologists inter-

ipate clear areas o f potential confusion: For

ested in thematic content analysis have used

instance, is a passage coded for "dialogue" if

criterion groups to create scoring systems for

the person states "we were not talking"?

motives. In this literature, individuals w h o were

Clearly specify when not to code for a partic-

known, a priori, to be high in a motive were

ular dimension. Giving examples o f "close but

asked to tell an imaginative story in response to

not quite" passages will help eventual coders

a picture, and those stories were compared to

make reliable distinctions. It is also best to

stories told by individuals w h o were

not

keep the coding categories to a reasonable

assumed to be high on the motive in question.

number. For example, M c A d a m s ' s intimacy

For instance, for the intimacy motive scoring

scoring

system, M c A d a m s ( 1 9 8 0 ) compared stories

Typically, coding can be thought o f as test

system

comprises

1 0 categories.

told by sorority members w h o had just partici-

construction—a researcher may begin with far

pated in a unity ceremony to those who had

more items on a scale than ultimately will be

not, with the former expected to be higher

included in the final, most reliable version.

interaction.

Categories may be winnowed down through-

Although this kind of inductive strategy for

out the coding process. Obviously, one can

coding schemes has been useful, it is not always

code for a variety o f dimensions and later cre-

in concern for w a r m

human

ate composites, but it is important to consider

possible to identify such a priori groups. Alternatively, as was the case in

our

wishes study, given a variable that is well

coder decay—the tendency for coders t o become exhausted and overwhelmed.

defined, a coding scheme can be developed.

Another consideration in developing a cod-

A detailed theory o f h o w the construct might

ing scheme is what level o f measurement to

communicate itself via narrative or other free

employ. Some coding strategies (e.g., Smith,

response will allow the researcher to easily

1 9 9 2 ) typically code for instances o f an image,

nominate ways that the psychological pro-

word, phrase, or theme. Such coding obviously

cess o f interest might reveal itself in the data.

is done on a categorical basis and

then

Certainly, in developing a coding system, it

summed over the story. It is also possible to

is tempting to code as much as possible.

code specific protocols for "fitting" a particu-

However, it is important to keep in mind what

lar type (e.g., is this a redemption pattern—in

a human rater can and cannot accomplish

which

with efficiency and accuracy. Some advice for

(McAdams,

the creation o f coding schemes can be gained

Bowman, 2 0 0 1 ) . In contrast, it is possible to

events

go

from

Reynolds,

bad Lewis,

to

better?)

Patten, &

by examining some o f the existing schemes

code narratives or other data using Likert-type

that have been developed. First, these systems

scales (for instance, " H o w emotionally positive

are marked by their clarity. Definitions must

is this story?" on a scale from 1 meaning not at

DESIGN AND ANALYSIS all to 7 meaning extremely much). These issues

accommodation might be reflected in

are important to consider for a variety o f rea-

individual's struggle to understand or find

sons. First, it is worthwhile to consider that

meaning in a major life event, it might manifest

interval level data can always be transformed

itself in the stories people told about those life

an

down a notch in levels o f measurement—by

experiences. Thus, based on Block's definition,

converting the ratings to ordinal scales or to

we created a coding scheme that centered on

categories, if it is clear that raters are seeing the

those aspects of accommodation that might be

variables as categorical. Interval ratings may

found in the story of a life experience. Because

also be somewhat easier for raters to perform

this was a first investigation into content ana-

with confidence. At times, naive raters may

lyzing for accommodation, we included a

blanch at the idea that they are making all-or-

broad range o f potentially relevant variables,

nothing judgments. Clearly, the choice of mea-

which were coded on scales from 1 (not at all)

surement technique ought to be tied closely to

to 7 (extremely much). Initially, we started with

a consideration o f the theoretical definition of

nine dimensions, including

"paradigmatic

a construct: Is it all or nothing? Is it conceptu-

shift,"

"exploration,"

alized as existing along a continuum?

"traumatic,"

Finally, consider that a coding scheme,

"active vs. passive," "gradual

vs. sudden change,"

"closure," and "denial." Coding was com-

though developed in response t o a particular

pleted by two raters who rated all the narra-

demand (one data set), may have a life o f its

tives, independently. These raters were blind to

own beyond the initial investigation. It is

all other aspects of the data.

essential to present enough information in

Because they were rather abstract, the

the write-up so that a reader could choose to

accommodation

replicate the study in his or her own lab.

defined in great detail, as can be seen in Table

Given the need for brevity in manuscripts,

8.1. W e completed a factor analysis of the cod-

making coding schemes available on

coding

dimensions

were

the

ing and found that the dimensions tapped into

outstanding

two main issues: closure and accommodation.

option. Tips for training raters on the system

The three dimensions that held together as an

as well as expertly scored practice materials

accommodation factor are shown in Table 8 . 1 .

also are invaluable additions to a W e b site.

W h a t follows is an excerpt from a participant's

W o r l d Wide W e b presents an

Our research on parents of children with DS

story, scoring high in accommodation.

provides an example of the construction of a theoretically derived coding scheme (King, Scollon, et al., 2 0 0 0 ) . In that study, we asked the participants to describe how they found out they would be parenting a child with DS. W e were interested in examining stories of life transition, particularly, in order to examine whether these stories might show signs o f the processes thought

to underlie

personality

development. Block (1982) discussed Piaget's developmental process o f accommodation as mechanisms of personality. Accommodation requires that one rethink one's essential beliefs about the self and

the world—to

create

new structures through which to experience meaning. W e thought that to the extent that

I cried some and experienced waves of "Unknown" embracing me. . . . I knew little about DS—it was an abstraction. Any handicap fell into the category of a childhood memory of seeing "waterheads," as I was told or remember, out on a shopping trip getting into a bus. My daughter was flesh and blood and a good nurser and that was the reality I remember dealing with. I thought very little about her future but I knew I would bow to no predictions. Irrational thoughts came to me at times but did not consume much thinking time: "I must have DS too, it just hasn't been discovered yet." Or "This child must be a consequence for wrong decisions in the past."

Measures Table 8.1

and Meanings

\

Content Categories for Accommodation Coding

Category

Description

1. Paradigmatic shift

This rating concerns the degree to which change entails a paradigmatic shift for the person. The new experience requires a revision of structures—an actual change in response to the environment and a qualitative change in how the person sees the world and him- or herself. The person has been forced to change, centrally and qualitatively, his or her views of the self and the world.

2. Exploration

How much has the person searched and struggled with the change? This may include commenting on his or her own coping processes as well as talking about the process of making sense of the experience. Is the narrator primarily a passive recipient of experience or primarily actively taking part in what is happening?

3. Activity versus passivity

T h e composite a c c o m m o d a t i o n measure predicted personality development concur-

The

Training

Phase.

Raters ought to be

given sufficient time and material to engage in

rently and 2 years later. Applying Block's

intense practice, with feedback, during the

conceptualization t o the stories we collected

training phase. Although taking the time to

represents a top-down approach t o this anal-

train coders may seem like bit o f a burden, this

ysis. T h e winnowing down o f dimensions to

training phase can be a valuable time to edit

the most central ones is also a c o m m o n part

the coding scheme, for instance, to identify

of such investigations. W e came to the narra-

early problems in reliability. Especially if it is

tives with a flexible scheme that was dictated

necessary to create a new coding scheme, it is

investigator's

always recommended that raters be given a

hunches about h o w a particular psychologi-

trial period with the scheme and that they be

cal process might manifest itself in story.

encouraged t o report back their responses to

by theory,

along with

an

the scheme after a brief amount of practice. W e often ask coders to report the numbers they Using Naive

Coders

have assigned to various protocols. It is vital

W h e n using qualitative data in social and

that they realize that disagreement is not a

personality psychology, in general, at least two

problem at this stage. Often, wide differences

independent raters are required for at least

in initial coding help to identify problematic

some of the ratings that are done, in order to

dimensions and can be useful in resolving

allow for reliability estimates. Because of the

ambiguities for all coders. Raters must be given

logistical realities o f academic research, these

an opportunity t o discuss particularly difficult

raters often are graduate or undergraduate

or complex categories with the other raters and

research assistants. As such, they tend to be

the principal investigator. Repeated meetings

truly naive in their interactions with

and training sessions may be necessary. An

the

research materials. Training these individuals

appropriate metaphor is an immersion course

to be proficient raters may be a unique chal-

in a second language. I encourage raters to

lenge. The process ought to be conceived of as

mentally code everything they read, hear, or

occurring in two stages, training and coding.

see for the relevant dimensions. In our study of

183

DESIGN AND ANALYSIS parents o f children with D S , undergraduate

be blind t o the other raters' ratings as well as

raters were given detailed descriptions o f the

to the hypotheses o f the study and other

coding dimensions. In addition, across several

aspects o f the participants that may be under

meetings, coders analyzed a variety of stories

investigation. O n e way t o assure raters that

for these dimensions, including stories o f life

consulting with each other to ensure high reli-

transition collected from other samples, such

ability is not necessary is to emphasize the

as divorced women. In this case, the emphasis

importance o f each rater being consistent

was not on mastering an existing scheme but

within his or her own ratings. T h a t is, raters

on ascertaining that all coders understood the

should be encouraged to develop their own

"gist" of the instructions and could recognize

sense o f what is "high" on a given dimension

potential aspects of accommodation when they

and to stick with that standard

saw it. Again, for these ratings, the emphasis

the coding. Coders should keep their practice

throughout

was on consistency within each rater, rather

materials from the training phase, so that they

than getting a "right answer."

can consult these when they are confused or are

unsure about their coding. In training raters

plenty o f available practice materials. In other

to code for motive imagery, for instance, pre-

F o r many

existing schemes, there

cases, narratives that were used for other

vious training materials that have been "cor-

investigations can be useful practice materi-

rected" by expert scoring can allow them to

als. Finally, it may be necessary to find cre-

check their intuitions with previous materials.

ative alternatives for practice materials. In my

Even with well-trained coders, differences

lab, where the qualitative data typically are

of opinions do erupt in coding. In most cases,

narratives, raters have practiced on works o f

data for analyses are supplied by averaging

fiction, poetry, letters to the editor, and pub-

over the ratings, so these differences are not

lished soap opera summaries. Other practice

crucial. However, if the study involves cate-

materials for coding o f nonverbal behavior

gorizing data, it may be desirable to obtain a

may be T V clips, magazine photos, and films.

"right answer" in the end. Typically, a con-

However one chooses t o train coders, at

sensus can be reached with some discussion,

some point they are ready t o begin actual

particularly with at least one expert coder in

coding. W i t h regard

coding

attendance. However, it is important that the

schemes, this may occur when the investiga-

ratings prior to discussion are retained for

tor has a sense that "everyone gets it." T h i s

the calculation o f reliability estimates.

to

ad

hoc

was more or less h o w we completed training for the D S study. In other cases, there are actual tests that can be administered

(for

Naive Approach.

Coders

and

the

Bottom-Up

Naive coders may relate to the data

instance, for the coding manuals in Smith

in only top-down sorts o f ways. They have

[ 1 9 9 2 ] ) . In this case, after sufficient practice,

been trained on a particular coding scheme

raters can take the test to ensure that their

and are likely, therefore, to attend to just those

ratings are mapping o n t o expert scoring (typ-

dimensions that the scheme presents as rele-

ically t o within 9 6 % o f expert scoring).

vant. Y e t , reading and thinking about the raw data certainly can lend itself to more bottom-

The Coding Phase. Although communica-

up discoveries. H o w can the researcher remain

tion among raters is invaluable during train-

open to such developments while using rela-

ing, once training is complete, it is important

tively naive coders? O n e solution, obviously,

that raters be told not to talk t o each other

is to read all or most o f the protocols oneself.

about specific aspects o f the coding once the

W i t h a large sample, this is not always feasi-

"official" coding has begun. All raters must

ble. Instead, the coders can serve as conduits

Measures

and Meanings

\

of interesting information. In the initial stages

event can play a helpful

of content analyses, our team o f coders is

(Pennebaker &

role in coping

invited to discuss with us any other interesting

was reason t o believe that the use o f fore-

Seagal, 1 9 9 9 ) . Thus, there

things they happen upon—to share particu-

shadowing in stories might relate to positive

larly striking narratives with the group during

psychological functioning. A coding scheme

the coding process. In our lab, coders meet to

for foreshadowing was developed, and all the

discuss coding progress one or two times a

narratives were coded by all raters for the

week. (Specific stories are not discussed, but

presence or absence o f this narrative device.

dimensions are. In this way, raters can discuss

D r a m a t i c examples o f foreshadowing did

general issues without revealing their codes to

indeed emerge in a portion o f these stories.

each other.) I typically ask coders to keep an

For instance, one participant described how,

eye out for what I might be missing and to jot

at their baby shower, her husband opened a

down ideas as they read the narratives, in

child care b o o k at random and started read-

order to continue to get a sense o f what is

ing loud. T h e y both recoiled in horror as they

interesting about the stories.

realized he was reading about D S . Another

An example o f this process is provided,

mother began her story with a visit to an

once again, by our study o f parents o f chil-

amusement park during her pregnancy. It

dren with D S . This study was conducted

happened that there were a number of children

specifically t o explore the possibility that

with D S at the park, and she t o o k this as a

accommodation might be found in stories o f

sign that the child she was carrying would

life transition. However, another interesting

also have D S (King, Scollon, et al., 2 0 0 0 ) .

aspect o f these narratives presented itself as

Interestingly, in accord with our predictions,

we embarked on our coding. In addition to

foreshadowing was associated with height-

coding for accommodation, three additional

ened well-being for these parents.

coders coded the emotional tone o f the beginnings and endings o f the stories. While conducting this coding, these coders noted a

Reliability and Validity

fascinating aspect o f the stories these parents

Because social and personality psycholo-

shared. For most o f the protocols, the story

gists tend t o treat qualitative data in a quan-

began with a doctor's announcement o f the

titative way, these data are expected t o meet

child's diagnosis o f D S , in the hospital, imme-

the same standards o f reliability and validity

diately or shortly after

birth. Sometimes,

however, the story began not with

as m o r e structured measures. In asking about

this

reliability, we want t o k n o w if a measure is

announcement but rather at an earlier point

relatively free o f error o f measurement—are

in time. For instance, one mother wrote about

the ratings consistent across raters? In asking

a dream she had months prior to the child's

about validity, we want to k n o w that there is

birth. Another wrote about having a sinking

evidence that the measure used actually taps

suspicion during her pregnancy that the child

the construct under investigation.

she was carrying would have D S . F r o m a literary perspective, w h a t

we

observed was clearly foreshadowing. In liter-

Reliability

ature, it is clear that foreshadowing can be a

Using qualitative data, the question o f

compelling way to enhance the coherence o f a

reliability typically is phrased in terms o f the

narrative from one moment o f the drama t o

relationships a m o n g ratings done by differ-

the next. It has been suggested that construct-

ent raters. In calculating reliabilities, the

ing a coherent narrative about a traumatic life

implications o f decisions that

have

been

185

186

DESIGN AND ANALYSIS made along the way in the investigation (e.g.,

have counted images and added these up

W h a t scale o f measurement was used to code

within passages). Measuring reliability in this

the data? H o w many raters were used? H o w

case means gauging the consistency across

much o f the data did each rater actually

judges in their patterns o f ratings. Imagine

rate?) c o m e to the fore. M y purpose

in

that three raters read the passage below

reviewing these issues here is t o emphasize

(from the parents o f children with D S ) and

the implications o f these decisions for relia-

coded it for emotional positivity, on a scale

bility and to give practical suggestions for

from 1 (not at all) to 7 (extremely

much):

obtaining acceptable reliability estimates. First, note the difference between coder agreement and coder reliability. Interrater agreement refers to the degree to which coders give the exact same rating to a narrative, videotape,

or

other

item

to

be

coded.

Agreement typically is a concern when ratings have been done using nominal (or categorical) scales. T h e simplest and, perhaps, most easily understood measure o f agreement is the percentage o f agreement. It is important to note, of course, that some agreement among raters may occur by chance, so the percentage o f agreement may, in fact, overestimate

the

It was long enough ago that the word was Mongoloid. I was alone and it was late at night when the doctor told m e . . . . I laugh at this now because I was 33—I called my parents. I think I wanted them to fix things—they had been pretty good at that in the past.. . . Then I realized that I was mourning as if my child had died yet 1 still had a nice fat baby in the nursery. I rang for him to be brought to me expecting him to be a monster instead of the cute thing I saw in the delivery room. I tore all of his clothes off of him and just looked at him. He was beautiful. (King, 2001)

actual level o f agreement by judges that is due to their recognition of the variables o f interest.

If the three raters rated this passage 4 , 5,

A more sophisticated measure o f agreement,

and 7, respectively, there are clearly disagree-

one that is useful when categories are mutually

ments in their assessments o f the absolute level

exclusive, is Cohen's K (Cohen, 1960), which

of the positivity o f the passage. However, if, in

includes an estimate o f the agreement that

the context o f all the ratings these raters did,

would be likely to occur by chance. In cases

we find that this passage actually was rated

where the codes produced by raters are nomi-

relatively highly by all raters, the reliability

nal but the categories are not mutually exclu-

will be high. For instance, if the rather cur-

sive, other alternatives may be phi coefficients

mudgeonly rater w h o gave this passage a 4

among ratings or, again, simple percentage

consistently rated very positive passages as 4 ' s

agreements. In our study o f parents o f chil-

while our very positive rater consistently

dren with D S , we used phi coefficients (essen-

rewarded such passages with 7's, the ratings

tially

will be reliable. Again, reliability estimates are

correlations

between

dichotomous

ratings) to express the strong degree of agree-

not concerned with the mean

ment among our judges for the ratings o f fore-

among raters but are ways o f gauging their

shadowing (average phi = . 9 1 ) .

proportionality or rank order consistency.

In

contrast

to

agreement,

differences

reliability

T h e most obvious and convenient way to

typically refers not to the exact agreement o f

calculate reliability in this case is to simply

ratings but rather to the extent to which

calculate the correlations among the ratings

ratings are proportional across judgments.

and to report, perhaps, the average correla-

Typically, in this case, ratings have been con-

tions

ducted on an interval scale (e.g., scales from

C r o n b a c h ' s alpha for a composite rating. In

1 to 7 ratings) or a ratio scale (e.g., raters

the study o f parents o f children with D S , we

among

multiple

raters,

or

the

Measures

and Meanings

used interrater correlations to justify the

represents the characteristic idiosyncrasies o f

creation o f composites for all the a c c o m m o -

each rater.

|

dation dimensions. It is worth noting that in

First, let's consider the optimal case, in

this study, all the raters rated all the stories,

which all raters have rated all the protocols.

and this decision allowed us the greatest free-

In this situation, we would use the average

dom in calculating the reliability.

among all raters as the actual score given to

It is not always the case that the same

a protocol. Differences among the raters can

raters will rate all materials. Sometimes, data

be assumed to cancel out in the averaging o f

may be collected over the course o f a longer

ratings, so rater variance should not be con-

time period, and so raters are replaced. At

sidered error. T o estimate the reliability o f

other times, the data sets are so large that it

the composite score, we could conduct a

is nearly impossible to have a pair or trio

rater x person two-way A N O V A on the rat-

of raters rate all o f them. Such situations

ings. T h e output would provide the variances

require us t o consider the role o f rater vari-

for raters, persons, and error. T o calculate

ance in the reliability estimate. In this case,

the Spearman-Brown correlation, these mean

the intraclass correlation is preferred because

squares would be inserted into the formula as

it allows the researcher t o decide exactly

follows (Ebel, 1 9 5 1 ; Tinsley & Weiss, 1 9 7 5 ) :

what the place o f interrater variance ought to be vis-a-vis the reliability o f an instrument.

MS

- MS

p e r s o n

T o appreciate this issue, it is helpful to imag-

MS

ine these data in a way that may not c o m e

error

person

intuitively to many researchers—that is, to think o f each rater as a level o f an indepenT o understand the effects o f the levels o f the independent variable (i.e., the raters) on the dependent variable (i.e., the ratings), we must partition the variance in the ratings into attributable

to

rater,

variance

attributable to the object being rated (i.e., the person), and error variance. Thus, we can think o f the ratings as dependent variables in a rater x person analysis o f variance (ANOVA).

Such

an

analysis, in

actual

practice, provides a clear sense o f the role o f rater variance in the ratings. Any output for an A N O V A will include the variance attributable t o the independent

e r r o r

term above represents

only random error. T h e variance partitioned

dent variable in an experiment.

variance

N o t e that M S

variables

(i.e., the mean square for rater) and

that

off for raters is not included in the calculation at all. However, in cases where not all coders have coded the entire data set, or if "doubling up" has occurred on only a subset o f cases, the final ratings given to protocols are not composites o f multiple ratings by the same raters. As a result, between-rater variance is rightfully treated as error. Although using a composite of judges' ratings allows us to use the reliability o f the composite, in this case we really need the reliability o f an average single judge's ratings. Here a simple one-way A N O V A (analyzing ratings by persons) provides the variances for the intraclass reliability estimate:

attributable to the person being rated (i.e., the mean square for person), as well as the leftover (unsystematic) error variance. N o t e that the variance due to person here represents

(MS

person

+ MS

e r r o r

)(K-l)

the degree to which raters were sensitive to the changes in the dimensions o f interest

where K = the number o f judges rating each

across protocols. T h e variance due to raters

person. (If K varies across individuals, the

187

188

DESIGN AND ANALYSIS average number o f judges can be calculated

about the consistency o f a measure, it is

for inclusion in this formula; see Tinsley and

possible to have very high reliability and very

Weiss [ 1 9 7 5 ] . ) N o t e that in this equation, the

low validity. Because reliability refers to sys-

MS

tematic variance, if all raters consistently

e r r o i

includes the between rater variance

m a k e the same mistakes, reliability will be

as error. The bottom line is that if raters complete

high, yet these highly reliable ratings will

ratings on only a portion o f the data, the relia-

consistently miss the variable in question.

bility estimates should treat between-rater dif-

O n c e again, decisions affecting the quality o f

ferences as error and the reliabilities will be

an investigation from its inception clearly

lowered accordingly. Note that for all estimates

impinge on eventual validity: W a s the coding

of reliability, increasing the number o f raters

scheme sufficiently clear? W e r e the raters

will increase the interrater reliability. However,

appropriately trained? W a s the proficiency

the addition of raters has less and less impact on

of the raters well gauged? Although reliabil-

reliability as more raters are added. In terms of

ity clearly is a challenge, validity issues may

what are acceptable reliabilities, generally the

present even greater difficulty, particularly

same rules o f thumb can be used as apply to

for newer coding schemes.

questionnaire data—reliabilities below . 6 0 are likely to raise eyebrows.

Convergent validity can be difficult

to

gauge, particularly if a construct is understood be

to be strongly intertwined with its own mea-

chosen with attention to the scale o f measure-

surement method. For many years, the T A T

In

sum, reliability estimates should

ment used, the number of coders, and the

measures o f motivation were held in low

amount o f overlap by coders. Some methods

regard because they failed to show convergent

for computing reliability are fairly easy (for

validity with questionnaire measures o f the

instance, computing percentage agreements,

same motives (see King, 1 9 9 5 , for a review).

interrater correlations, or alpha reliability

However, theoretical justifications for these

across composites for coders), whereas others

null results provided a new way to think o f

be m o r e challenging logistically (for

these measures (McClelland et al., 1 9 8 9 ) . It is

instance, computing mean square estimates

important to remember that the lack o f rela-

for variance attributed to person x rater). As

tion between self-report and content-analytic

a rule, researchers probably select the reliabil-

measures may be substantively interesting and

can

ity estimate that gives them the best result,

not necessarily indicative o f invalidity of either

but these must always be selected within the

type of measure. For instance, in the study by

confines o f the data themselves. Although

Dovidio and colleagues reviewed earlier, the

many textbooks review the issues o f reliabil-

spontaneous nonverbal behavior shown by

ity, I recommend an article by Tinsley and

white participants in the presence o f a black

Weiss ( 1 9 7 5 ) . It has been extremely useful in

confederate was not related to their verbaliza-

obtaining reliability estimates given varying

tions or to their self-reported attitudes, yet

numbers o f coders, levels o f measurement,

clearly this behavior has importance in its own

and research goals. T h e advice offered is

right. Qualitative measures are used exactly

straightforward and extremely helpful.

because they tap into something that more structured measures cannot get to; thus, it should not be surprising when these measures

Validity

fail to converge.

The validity o f an instrument concerns

Seeking to establish criterion-related valid-

h o w much the measure actually taps w h a t

ity may be a more profitable approach to

we want it t o . Although reliability tells us

establishing the validity of qualitative methods.

Measures and Meanings

\

This possibility is well illustrated in the classic

this problem is a challenge. Clearly, simply

distinction in the Type A Behavior Pattern

including excerpts o f narratives is one way t o

literature. Research comparing the Type A

attempt to preserve the integrity o f the whole.

Structured Interview (which relies on the cod-

However, this solution p r o b a b l y is pro-

ing o f nonverbal behavior) with the Jenkins

foundly unsatisfying for both the reader and

Activity survey (a reliable questionnaire) has

the researcher. In addition, the press for

demonstrated that, reliability differences not

brevity in journal articles may decrease the

withstanding, the more qualitative measure

likelihood o f even this level o f inclusion (or to

does a superior j o b of predicting coronary

the relegation o f these excerpts to rarely read

heart disease (e.g., Matthews, 1 9 8 8 ) .

appendices). W h a t other alternatives exist? O n e possibility is to return to the single case to examine h o w individual narratives reflect

ADDITIONAL CHALLENGES OF USING QUALITATIVE DATA IN SOCIAL PSYCHOLOGY

and challenge group based results (see Allport [ 1 9 6 1 ] ; this advice certainly would also apply to purely quantitative w o r k ) . M y w o r k on the relations between the content o f implicit motives (measured via imaginative stories)

Methodological Problems and Confounds

and personal goals demonstrates h o w the examination o f individual cases can enlighten

There are a variety o f confounds that m a y

even null results. In this case, n o evidence

be unique to the use o f qualitative data. O n e

emerged for the kinds o f straightforward rela-

problem in verbal reports is that o f verbosity.

tions one might expect between implicit

Individuals w h o simply talk a lot more than

motives and goal content (e.g., individuals

others may score higher on measures based

high on need for achievement did not have a

on free responses. Typically, researchers

high number o f achievement-related personal

include the length o f a protocol as a control

goals) (King, 1 9 9 5 ) . An examination o f a few

variable in analyses or m a y convert scores to

single cases, however, demonstrated

"images per 1 , 0 0 0 w o r d s " or some similar

these t w o types o f motivation

measure t o account for differences in w o r d

related within a given person in complex,

that

measures

usage. Other potential confounds in coding

sometimes idiosyncratic ways that could not

nonverbal behavior are the effect o f attrac-

be captured in group analyses. It is important

tiveness, smiling, and clothing on ratings.

that such an examination not be presented as

Typically, researchers strive t o equalize these

an afterthought. Such analyses may inspire

characteristics across rated targets, or code

research on a broader sample. Another possi-

for these potential confounds in order to sta-

bility would be to combine quantitative anal-

tistically control for them later.

yses with more purely qualitative analyses. Such a possibility may require interdisciplinary research that acknowledges and nego-

Losing the Trees for the Forest Perhaps

because

social

psychological

research is more likely t o (rapidly) quantify qualitative data, the risk o f losing the whole in pursuit o f numerical descriptions o f the data is more pressing. Y e t , one o f the very

tiates

the

varying

biases

associated with differing

and

expertise

areas o f social

science. Finally, it might be worthwhile to consider alternative outlets for research that is more purely qualitative, keeping in mind that it is often a question o f where and when, not

strengths o f qualitative data is their capacity

if, a scientifically sound, provocative article

to convey a sense o f the whole. Overcoming

will be published.

189

190

DESIGN AND ANALYSIS

Special Ethical

NEW

Considerations

A final unusual problem in the use o f qualitative data is special consideration for ethical treatment o f the data.

TO

APPROACHES

QUANTIFYING

QUALITATIVE DATA

Narratives,

Advances in technology have led to the develop-

interviews, and videotaped interactions or

ment of a number of innovative computerized

monologues may reveal more than the typi-

systems for analyzing qualitative data, including

cal questionnaire about a particular research

N U D * I S T , nVivo, H y p e r R E S E A R C H ,

participant. Participants w h o are asked to

ATLAS.ti. Each of these systems certainly war-

share stories o f their personal experiences

rants a chapter (or book) of its own, and fortu-

and

must be assured that the data will be pub-

nately, such chapters and volumes have been

lished only in group format. I f a researcher

written (e.g., Fielding & Lee, 1 9 9 1 ; Richards &

would like to use excerpts from various par-

Richards, 1 9 9 4 ) . Critiques also are available

ticipants t o enliven his or her research report,

that compare and contrast the strengths and

he or she ought to ask for permission from

weaknesses of each of these (Huberman

these participants to do so. Typically, if indi-

Miles, 1 9 9 4 ) and compare these systems to

viduals are t o be videotaped, they should give

human ratings (Rosenberg, Schnurr, & Oxman,

&

their permission to be so taped. If telling par-

1990).

ticipants beforehand about the taping might

researchers to manage qualitative data. One

These

systems

provide

ways

for

interfere with natural behavior, the partici-

issue is that these packages don't actually code

pants may be taped surreptitiously but later

for you; they simply allow you to make use of

should be informed o f the taping and offered

their various features for organizing data, track-

the chance to destroy the tape.

ing patterns that your exploration has uncov-

Finally, during coding, it is extremely

ered. As such, these tools may be best suited to

important that coders treat the research pro-

more modestly sized data sets and for use by

tocols with the appropriate level o f respect.

expert coders (see Loxley, 2 0 0 1 ) . Although

In my lab, student raters are asked to sign a

differing in their specific features, these pack-

"contract"

ages essentially help a researcher taking a

that

reviews the

appropriate

treatment o f qualitative data. Raters are not

"bottom-up" approach to build a conceptual

to talk about these data with anyone w h o is

framework for a data set. Coming from a

not in the lab. Raters must behave profes-

top-down approach, they allow a researcher to

sionally while in the lab, in case research par-

organize text features that are theoretically

ticipants c o m e by to drop off questionnaires

relevant in hypothesis testing.

or be interviewed. Furthermore, if a rater recognizes a particular participant from his or

Some packages allow for a simultaneous "macrolevel"

accounting o f the

patterns

her research materials, the rater must stop

emerging in the data. F o r instance, ATLAS.ti

working on those materials immediately and

uses semantic categories to group codes into

turn them over t o the principal investigator

families.

for reassignment to another rater. It is impor-

approach that, in addition to performing code

tant for raters to recognize that participants

and retrieve functions, also provides nodes for

NUD*IST

is

an

index-based

have shared important aspects o f their lives

an index system that functions parallel to the

with us. W i t h o u t their candor and willing-

microlevel coding. This index system allows

ness to share, our w o r k would be impossible.

the research to track and

Their materials must be treated with

dynamic connections between the general con-

highest degree o f respect.

the

also consider

structs emerging in the content analyses.

Measures

and

Meanings

Using these technologies is a truly interactive

to such a molecular level o f analysis. T h e

experience, and the potential for theory build-

human reader may never be replaced entirely

ing appears very strong. M o s t o f the packages

by computerized analyses.

are available in demonstration versions on the World Wide W e b . As the creators of N U D * I S T themselves have

commented, this

T e c h n o l o g y also has

begun

to

show

promising inroads in other areas in which

system

qualitative data have required a great deal o f

"Offers many ways for a researcher never to

time and effort to quantify. For instance,

finish a study" (Richards & Richards, 2 0 0 1 ,

Cohn

p. 4 5 8 ) .

Lien, H u a ,

Perhaps because T h e Linguistic Inquiry and

and colleagues (Cohn, Z l o c h o w e r , 8c K a n a d e , 2 0 0 0 ) have been

developing facial feature matching software

W o r d Count (LIWC) (Pennebaker & Francis,

t o code for emotional facial

1 9 9 6 ) was designed by a social psychologist,

These innovations may allow researchers t o

its reliability has been empirically addressed

avail themselves o f rich qualitative

data

(Pennebaker & King, 1 9 9 9 ) and it has been

sources without the effortful training

and

used on very large data sets successfully (e.g.,

coding

Pennebaker, Mayne, 8c Francis, 1 9 9 7 ) . As

required. O f course, such innovations must

L I W C processes text, every word in a protocol

be shown to have excellent convergent valid-

is matched against a catalog o f words in the

ity with manual coding in order to prove

L I W C dictionaries. These dictionaries include

truly useful for researchers.

that

such

data

expression.

typically

have

a variety of topics o f interest to social psychologists, including positive and

negative

emotion, self-references, social words, deathrelated words, and cognitive words

CONCLUSION

(e.g.,

thinking, causation). T h e output is SPSS ready

Using qualitative

and includes the percentage o f words that the

researcher to tap into a well o f rich informa-

approaches

allows

the

person used that fit into each o f the L I W C dic-

tion. W h e n I first thought about what it

tionaries. L I W C is enormously flexible. T h e

meant to be a psychologist, this is what I

various dictionaries can be combined in a vari-

thought I'd be doing: learning from people

ety o f ways to tap into whatever construct

doing what they do—that is, making meaning

interests the researcher. A weakness o f a word

out o f their experiences. Qualitative

count strategy is that words are necessarily

allow us to examine the natural behavior o f

data

decontextualized, so L I W C misses sarcasm,

human beings in its various forms. Decisions

metaphor, and other aspects o f language that

about what we study and h o w we study it

are less than straightforward.

have far-reaching implications for the even-

T o use these programs, protocols must be

tual usefulness o f such inquiries. Qualitative

transcribed into text files for the computer.

research in social and personality psychology

Although this may be a small obstacle (we

involves a number o f challenges, but in the

sometimes ask participants to type their sto-

end the best o f this w o r k can be seen to

ries onto a computer), it might also make

embody the best o f the hard and the soft in

research on archival texts that have not been

our science. T h e key is to balance the desire to

transcribed laborious. T e x t analysis programs

wring as much from the data as possible with

represent a potential

maintaining the dignity o f the whole, to not

boon to

qualitative

research, but it is important to bear in mind

lose the "big picture" in the search o f that

that not all research questions lend themselves

which easily lends itself to quantification.

192

DESIGN AND ANALYSIS

REFERENCES Allport, G. A. (1961). Pattern and growth in personality. New York: Holt. Altheide, D. L., 8c Johnson, J . M . (1994). Criteria for assessing interpretive validity in qualitative research. In N. K. Denzin 8c Y . S. Lincoln (Eds.), Handbook of qualitative research (pp. 485-500). Thousand Oaks, CA: Sage. Baumeister, R. F., Wotman, S. R., 8c Stillwell, A. M. (1993). Unrequited love: On heartbreak, anger, guilt, scriptlessness, and humiliation. Journal of Personality and Social Psychology, 64, 377-394. Berry, D. S., 8c Miller, K. M . (2001). When boy meets girl: Attractiveness and the five-factor model in opposite sex interactions. Journal of Research in Personality, 35, 62-11. Blaikie, N. (2000). Designing social research: The logic of anticipation. Maiden, MA: Blackwell. Block, J . (1982). Assimilation, accommodation, and the dynamics of personality development. Child Development, 53, 281-295. Bonanno, G. A., Keltner, D., Noll, J . G., Putnam, F. W., Trickett, P. K., Lejeune, J . , 8c Anderson, C. (2002). When the face reveals what words do not: Facial expressions of emotion, smiling, and the willingness to disclose sexual abuse. Journal of Personality and Social Psychology, 83, 94-110. Campbell, L., Simpson, J . A., Kashy, D. A., 8c Rholes, W. S. (2001). Attachment orientations, dependence, and behavior in a stressful situation: An application of the actor-partner interdependence model. Journal of Social & Personal 18, 821-843. Relationships, Cohen, J . (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46. Cohn, J . F., Zlochower, A., Lien, J . , Hua, W., 8c Kanade, T. (2000). Automated face analysis. In C. Rovee-Collier 8c L. Lipsitt (Eds.), Progress in infancy research (Vol. 1, pp. 155-182). Mahwah, NJ: Lawrence Erlbaum. Costa, P. T., 8c McCrae, R. R. (1988). From catalog to classification: Murray's needs and the five-factor model. Journal of Personality and Social Psychology, 55, 258-265. Denzin, N. K. (1983). The art and politics of interpretation. In N. K. Denzin 8c Y . S. Lincoln (Eds.), Handbook of qualitative research (pp. 500-515). Thousand Oaks, CA: Sage. Dovidio, J . F., Kawakami, K., 8c Gaertner, S. L. (2002). Implicit and explicit prejudice and interracial interaction. Journal of Personality and Social Psychology, 82, 62-68. Ebel, R. L. (1951). Estimation of the reliability of ratings. Psychometrika, 16, 407-424. Ekman, P., 8c Friesen, W. V. (1978). Facial Action Coding System manual. Palo Alto, CA: Consulting Psychologists Press. Elms, A. C. (1994). Uncovering lives: The uneasy alliance of biography and psy­ chology. New York: Oxford University Press. Emmons, R. A., 8c King, L. A. (1988). Conflict among personal strivings: Immediate and long-term implications for psychological and physical well-being. Journal of Personality and Social Psychology, 48, 1040-1048. Fielding, N. G., 8c Lee, R. M. (Eds.). (1991). Using computers in qualitative research. London: Sage. Folkman, S. (1997). Introduction to the special section: Use of bereavement narratives to predict well-being in gay men whose partner died of AIDS— Four theoretical perspectives. Journal of Personality and Social Psychology, 72, 851-854.

Measures

and

Georgeson, J . C., Harris, M . J . , Milich, R., & Young, J . (1999). "Just teasing . . .": Personality effects on perceptions and life narratives of childhood teasing. Personality and Social Psychology Bulletin, 25, 1 2 5 4 - 1 2 6 7 . Guba, E. G., 8c Lincoln, S. L. (1989). Fourth generation evaluation. Newbury Park, CA: Sage. Hoyle, J . F. (1968). Sylvia Plath: A poetry of suicidal mania. Literature of Psychology, 18, 187-203. Huberman, M . A., 8c Miles, M . B. (1994). Data management and analysis methods. In N. K. Denzin 8c Y . S. Lincoln (Eds.), Handbook of qualitative research (pp. 4 2 8 - 4 4 5 ) . Thousand Oaks, CA: Sage. Keltner, D., 8c Bonanno, G. (1997). A study of laughter and dissociation: Distinct correlates of laughter and smiling during bereavement. Journal of Personality and Social Psychology, 73, 6 8 7 - 7 0 2 . King, L. A. (1995). Wishes, motives, goals, and personal memories: Relations and correlates of measures of human motivation. Journal of Personality, 63, 985-1007. King, L. A. (2001). The hard road to the good life: The happy, mature person. Journal of Humanistic Psychology, 41, 51-72. King, L. A., 8c Broyles, S. (1997). Wishes, gender, personality, and well-being. Journal of Personality, 65, 50-75. King, L. A., Scollon, C. K., Ramsey, C. M., 8c Williams, T. (2000). Stories of life transition: Happy endings, subjective well-being, and ego development in parents of children with Down Syndrome. Journal of Research in Personality, 34, 5 0 9 - 5 3 6 . Lester, T. (1994). Emotional self-repair and poetry. Omega, 28, 79-84. Loxley, W. (2001). Drowning in words? Using NUD*IST to assist in the analysis of long interview transcripts from young injecting drug users. Addiction Research and Theory, 9, 5 5 7 - 5 7 3 . Matthews, K. (1988). CHD and Type A behavior: Update on and alternative to the Booth-Kewley and Friedman quantitative review. Psychological Bulletin, 104, 373-380. McAdams, D. P. (1980). A thematic coding system for the intimacy motive. Journal of Research in Personality, 14, 4 1 3 - 4 3 2 . McAdams, D. P, 8c Bryant, F. B. (1987). Intimacy motivation and subjective mental health in a nationwide sample. Journal of Personality, 55, 3 9 5 - 4 1 3 . McAdams, D. P., Reynolds, J . , Lewis, M., Patten, A. H., 8c Bowman, P. J . (2001). When bad things turn good and good things turn bad: Sequences of redemption and contamination in life narrative and their relation to psychosocial adaptation in midlife adults and in students. Personality and Social Psychology Bulletin 27, 4 7 4 - 4 8 5 . McClelland, D. C. (1980). Motive dispositions: The merits of operant and respondent measures. In L. Wheeler (Ed.), Review of personality and social psychol­ ogy (Vol. 1, pp. 10-41). Beverly Hills, CA: Sage. McClelland, D. C , Koestner, R., 8c Weinberger, J . (1989). How do self-attributed and implicit motives differ? Psychological Review, 96, 6 9 0 - 7 0 2 . Morgan, C. D., 8c Murray, H. A. (1935). A method for investigating fantasies. Archives of Neurology and Psychiatry, 34, 2 8 9 - 3 0 6 . Pennebaker, J . W., 8c Francis, M. E. (1996). Cognitive, emotional, and language processes in disclosure: Adjustment to college. Cognition and Emotion, 10, 601-626. Pennebaker, J . W., 8c King, L. A. (1999). Linguistic styles: Language use as an individual difference. Journal of Social Psychology, 77, 1 2 9 3 - 1 3 1 2 . Pennebaker, J . W., Mayne, T. J . , 8c Francis, M . E. (1997). Linguistic predictors of adaptive bereavement. Journal of Personality and Social Psychology, 72, 863-871.

Meanings

193

194

DESIGN AND ANALYSIS Pennebaker, J . W., & Seagal, J . D. (1999). Forming a story: The health benefits of narrative. Journal of Clinical Psychology, 55, 1243-1254. Phillips, D. C. (2000). The expanded social scientists' bestiary: A guide to fabled threats and defenses of naturalistic social science. Oxford, UK: Rowman and Littlefield. Richards, T., & Richards, L. (1994). Using computers in qualitative research. In N. K. Denzin & Y . S. Lincoln (Eds.), Handbook of qualitative research (pp. 4 4 5 - 4 6 2 ) . Thousand Oaks, CA: Sage. Rosenberg, S. D., Schnurr, P. P., & Oxman, T. E. (1990). Content analysis: A comparison of manual and computerized systems. Journal of Personality Assessment, 54, 2 9 8 - 3 1 0 . Runyan, W. M. (1982). Life histories and psychobiography: Explorations in theory and method. New York: Oxford University Press. Smith, C. (Ed.). (1992). Motivation and personality: Handbook of thematic content analysis. New York: Cambridge University Press. Stirman, S. W., & Pennebaker, J . W. (2001). Word use in the poetry of suicidal and nonsuicidal poets. Psychosomatic Medicine, 63, 517-522. Tinsley, H. E. A., & Weiss, D. J . (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 3 5 8 - 3 7 6 . Vidich, A., & Lyman, S. (1994). Locating the field. In N. K. Denzin & Y . S. Lincoln (Eds.), Handbook of qualitative research (pp. 19-22). Thousand Oaks, CA: Sage. Winter, D. G. (1992). Content analysis of archival materials, personal documents, and everyday verbal productions. In C. P. Smith (Ed.), Motivation and person­ ality: Handbook of thematic content analysis (pp. 110-126). New York: Cambridge University Press.

CHAPTER

9

Implicit Methods in Social Psychology JOHN F . KIHLSTROM University of California,

I

Berkeley

concerned

Rather, the questionnaire method has been

merely with the impact o f the social situ-

central to social psychology because social

ation on individual behavior, social psy-

psychologists

f

social psychology were

have

embraced

the

twin

chologists would have n o interest in people's

assumptions that people were aware o f the

mental states. T h e y would

attitudes, beliefs, and values that guided their

be concerned

solely with measuring various features o f the

behavior, and that they would be willing to

social environment and various aspects o f

reveal them if asked appropriately.

people's behavior within that environment.

O f course, social psychologists were not

But long before the cognitive revolution in

stupid. They fully understood that some o f the

psychology, social psychologists believed that

mental states in question were highly charged,

social behavior was determined by the per-

and perhaps even embarrassing, and

that

o f the situation in

people might not be willing to talk about them

which that behavior took place. T h e central

with strangers. Consider, for example, the

role o f mental states, in turn, explains why,

lengths to which Alfred Kinsey and his associ-

son's mental

representation

almost from the beginning, social psycholo-

ates went to get people to talk about their sex

gists have relied on self-reports o f attitudes,

lives—as well as the difficulties encountered in

stereotypes and other beliefs, preferences, val-

today's climate o f political correctness in get-

ues, goals, and motives. Self-report question-

ting people to talk frankly about their views

naires and other survey instruments have not

with respect to gender, race, and ethnicity.

been merely a convenient and inexpensive

Social psychologists also understood that the situation

itself

was

prob-

way o f collecting information about people's

investigative

behavior (although they have been that, t o o ) .

lematic. By virtue o f demand characteristics

AUTHOR'S NOTE: Preparation of this chapter was supported by National Institute of Mental Health Grant MH­ 3 5 8 5 6 . I thank Jack Glaser, Tina Pantaleakos, and Carol Sansone for their comments.

DESIGN AND ANALYSIS

196

(Kihlstrom, 2 0 0 2 ;

Orne,

1 9 6 2 ; also see

THE PSYCHODYNAMIC HERITAGE

Haslam & M c G a r t y , Chapter 1 1 , this volexperimenter bias (Rosenthal, 1 9 6 3 ) ,

O f course, the notion o f unconscious determi-

evaluation apprehension (Rosenberg, 1 9 6 5 ) ,

nants o f behavior was not entirely new, given

reactance (Brehm, 1 9 6 6 ) , and a host o f other

that it lies at the roots of Freud's psycho-

ume),

factors, research designs might not possess

analytic theory o f the mind and behavior.

the kind of external validity that would permit

According to Freud, conscious experience,

us to conclude that people actually thought or

thought,

and

action were determined

by

feel.

unconscious sexual and aggressive motives, as

Accordingly, social psychologists have exer-

well as defense mechanisms unconsciously

felt what they seemed to think and

cised a great deal o f ingenuity in getting

deployed against these primitive drives in order

around these problems, from unobtrusive or

to reduce the anxiety caused by their conflict

nonreactive

(see

with the constraints o f the real and social

Wegener & Fabrigar, Chapter 7, this volume)

world. If Freud was right—and many person-

to

behavioral

psychophysiological

measures

(see

ality and social psychologists working in the

Berntson,

first half o f the 2 0 t h century thought he was—

recordings

C a c i o p p o , Lorig, N u s b a u m , &

then a variety o f new techniques was needed to

Chapter 1 7 , this volume). Beginning in the 1 9 8 0 s , a new dimension was added to the problem o f self-reports by

go beyond self-report to tap people's unconscious beliefs, feelings, and desires.

the increasing recognition that people's experiences, thoughts, and actions could be influenced by percepts and memories o f which they were unaware (Kihlstrom, 1 9 8 4 , 1 9 8 7 ) .

Projective Tests In Freud's own work, these unconscious

If unconscious thoughts, feelings, and desires

motives

exist and can influence social behavior while

brought to the light o f conscious awareness, by

were

ostensibly discovered,

and

remaining unconscious, then even the most

means o f the clinical technique o f free associa-

sophisticated questionnaires and surveys will

tion. But very quickly a number o f formal tests

not succeed in tapping the mental states that

were developed for this purpose, beginning

underlie what we do when we interact with

with

other people. This chapter surveys a number

technique o f free associations to serve as a

Jung's

adaptation

o f Freud's

own

of methods recently introduced for assessing

"complex

people's unconscious, or implicit, attitudes,

Standardized versions o f Jung's procedures,

beliefs, and other mental states relevant to

accompanied by rudimentary

social interaction (for alternative coverage o f

quickly developed (Kent & Rosanoff, 1 9 1 0 ;

this material, see Fazio &

Olson,

indicator"

(Jung,

1918/1969). norms, were

2003).

Rapaport, Gill, & Schafer, 1 9 6 8 ; Rapaport,

Although there is some overlap between these

Schafer, & Gill, 1 9 4 4 - 1 9 4 6 ) ; these in turn led

"implicit" methods and those generally called

to the development o f word-association norms

"unobtrusive," there is an important concep-

for purposes o f research with normal individu-

tual distinction. Unobtrusive methods

are

als (e.g., Russell & Jenkins, 1 9 5 4 ) . A number

used to assess attitudes, beliefs, and values o f

of other techniques were soon added (Lindzey,

which people are aware, but that they may be

1 9 5 9 ) , including the Rorschach Inkblot Test,

unwilling to reveal to the investigator. By con-

the Thematic Apperception Test ( T A T ) (see

trast, implicit methods are used to assess atti-

King, Chapter 8, this volume), and the Draw-

tudes, beliefs, and values o f which people are

a-Person Test. Even the Wechsler-Bellevue

unaware. This creates additional methodolog-

Intelligence Scale (WBIS), forerunner to the

ical problems for the investigator.

Wechsler Adult Intelligence Scale and perhaps

Implicit Methods

in Social

Psychology

the prototypical example o f a performance-

proved to have some validity, there is no

based test o f cognitive ability, was co-opted for

evidence that the scores in question actually

the purposes o f projective personality assess-

represented a subject's unconscious

mental

ment, both in the clinic (Rapaport, Gill, et al.,

state.

recent

1968)

literature promoting the T A T as a measure

and by the United States Central

Intelligence Agency (Bern, 1 9 8 3 ; M a r k s , 1 9 7 9 ;

of

Consider, for e x a m p l e , the

implicit,

or

unconscious,

motivation

(McClelland, Koestner, & Weinberger, 1 9 8 9 ) .

M a r k s & Greenfield, 1 9 8 4 a , 1 9 8 4 b ) . All o f this w o r k was predicated on the pro-

T h e low correlation between T A T and ques-

1939a, 1948;

tionnaire measures o f achievement motivation

Rapaport, 1 9 4 2 ) , in which the subject is given

is often interpreted as a reflection o f the inde-

jective

hypothesis

(Frank,

the opportunity "to reveal his way o f organiz-

pendence of unconscious and conscious moti-

ing experience by giving him a field (objects,

vation, but it could simply mean that T A T

materials, experiences) with relatively little

measures lack convergent validity. Similarly,

structure and cultural patterning, so that the

the fact that implicit and explicit measures o f

personality can project upon that plastic field

motivation predict different classes of behavior

his way o f seeing life, his meanings, signifi-

is often taken as evidence o f a pattern of dis-

cances, patterns, and especially his feelings"

criminant validity, but the same pattern o f cor-

(Frank, 1 9 3 9 b , p. 4 0 3 ) . Although the use o f

relations could be interpreted as a result o f

projective techniques does not necessarily

method variance. If the T A T , R o r s c h a c h

mean that the affects, drives, and other mental

(Bornstein, 2 0 0 1 ) , and other projective meth-

states revealed by the test are unconscious,

ods are to acquire the status o f "implicit"

that is the general assumption behind their

methods, more is needed than evidence for

use. As Rapaport, Gill, et al. wrote:

their reliability, validity, and utility. W h a t is needed is convincing evidence that they tap mental states.

The use of projective tests assumes that the examiner is after something in the subject about which the subject does not know or is unable to communicate; otherwise the examiner would ask him about it directly. . . . By means of projective tests we discover tremendous aggressions in persons who appear meek, or great dependent needs in suspicious and manly-appearing [sic] persons who deny having any such inclinations. If taken seriously, these tests therefore refer to unconscious motivation of action and behavior, and necessitate a personality theory that assumes the existence of, and accounts for, these motivations. (Rapaport, Gill, et a l , 1968, pp. 227-228)

unconscious

Despite their continued popularity among

she thinks and does in the real world outside

many clinical psychologists, it is now generally

the testing situation. In such tests, face validity

understood

is very high. Other tests, however, assume no

The Subtle and the Obvious Running parallel with the psychodynamic literature on projective techniques, some hint of the unconscious also can be found in the and signs,

distinctions between samples between subtle and obvious

and

items, in the psy-

chometric literature on objective tests of personality.

Most

personality

and

attitude

questionnaires assume, at least tacitly, that the test items represent samples

o f the respon-

dent's actual behavior—that is, that there is some degree o f isomorphism between the person's performance on the test and what he or

that projective techniques

are

instruments

such isomorphism. From this point o f view,

(Lilienfeld, W o o d , & Garb, 2 0 0 1 ) . Even in the

test items are intrinsically interesting units o f

few instances where projective techniques

behavior that are signs

not satisfactory psychometric

o f some underlying

197

DESIGN AND ANALYSIS disposition. T h e original versions o f the

will not see through them, and thus will be

Minnesota Multiphasic Personality Inventory

tricked into self-disclosure. Similarly, at least

( M M P I ) and

in principle, subtle items might be useful with

California

its " n o r m a l "

offspring,

Psychological Inventory

the

(CPI),

subjects

who

are

appropriately

motiva-

were both constructed under the sign assump-

ted toward

self-disclosure but are simply

tion. As a result, they contain many "subtle"

unaware o f their traits and other mental char-

scale items that lack face validity, even though

acteristics. However, it appears that personal-

they correlate with some empirical criterion

ity tests are disguised in this way at the

(Goldberg & Slovic, 1 9 6 7 ; Seeman, 1 9 5 2 ) .

expense o f validity (Mischel, 1 9 6 8 , 1 9 7 2 ) . O f

Ironically, then, the prototypical "objective"

course, the reply to this weak evidence for the

tests o f personality make the same assumption

empirical validity o f subtle

as do the Rorschach and other projective tech-

items might be the same as for the T A T : that

questionnaire

niques—that test responses are signs o f under-

subtle items, reflecting unconscious tenden-

lying dispositions, not samples of behavior

cies, should not be expected to correlate with obvious items that reflect conscious aware-

(Meehl, 1 9 4 5 ) . The preference for empirically valid signs

ness, and that as such, subtle items might pre-

over face-valid samples reached its apex in

dict different criteria than do obvious items.

Berg's deviation

which held that

However, there is no evidence for the dis-

even preferences for random drawings could

criminant validity o f subtle versus obvious

be used as personality scale items, so long as

items. Nor—and this is the central point—is

people with different dispositions expressed

there any evidence that subtle items tap traits,

different preferences (Berg, 1 9 5 5 ) . In fact,

attitudes, and the like o f which the subject is

hypothesis,

however, most empirically derived personality

unaware. T h e first lesson o f 1 0 0 years o f per-

scales (including those o f the M M P I and CPI)

sonality assessment is this: If you want to

contain a m i x of face-valid items that are obvi-

k n o w what people can tell you, you should

ously related to the substantive domain under

ask them. I f you want to k n o w what people 1

consideration, and "subtle" items that do not

cannot

tell you, unfortunately, subtle ques-

appear to relate to the domain. Although an

tionnaire items, like projective techniques,

early analysis by Seeman found that subtle and

would appear to be risky choices for the

obvious items were equally good predictors

assessment o f unconscious mental states.

of criterion behavior (Seeman, 1 9 5 2 ) , later studies found performed

that obvious M M P I

items

better than subtle items (Duff,

THE PRIMING SOLUTION

1 9 6 5 ; Goldberg &c Slovic, 1 9 6 7 ) . In addition, a study by Hase and Goldberg ( 1 9 6 7 ) showed

If not projective tests, or inventories o f subtle

that questionnaires constructed by "rational"

items, then what? One answer to this question

means, such that each scale item possessed

is provided by research in implicit memory

face validity, performed better than question-

(Schacter, 1 9 8 7 ) . Neurological patients with

naires constructed by "empirical" means

bilateral lesions to the hippocampus and asso-

that resulted in a m i x of obvious and subtle

ciated structures in the medial temporal lobe

items (see also Ashton &

characteristically are unable to remember the

Goldberg, 1 9 7 3 ;

Jackson, 1 9 7 5 ) . Scales consisting entirely o f subtle items

events and experiences that have transpired since

the

onset o f their

brain

damage.

have been advocated in some corners on the

However, it is n o w known that this antero-

ground that subjects w h o approach personal-

grade amnesia affects only conscious recollec-

ity questionnaires with a defensive attitude

tion. W h e n patients are tested with techniques

Implicit Methods

in Social Psychology

\

that do not require conscious recollection,

Kihlstrom, Barnhardt, &

they typically show that some traces o f post-

thinking

and

morbid experience have been encoded, remain

Shames,

&

in storage, and interact with ongoing experi-

Shames, & Dorfman, 1 9 9 6 ) , learning (Kihl-

ence, thought, and action—albeit outside con-

strom, 1 9 9 6 ; Reber, 1 9 6 7 ) , emotion (Berridge

scious awareness. T o take a familiar example,

& W i n k i e l m a n , 2 0 0 3 ; Kihlstrom, Mulvaney,

patients w h o have studied a list of words often

Tobias, &

show various priming

effects, as on tests o f

word-stem and word-fragment

problem Kihlstrom,

Tataryn,

solving

1992),

(Dorfman,

1 9 9 6 ; Kihlstrom,

T o b i s , 2 0 0 0 ) ; and

motivation

(Kihlstrom, Mulvaney, et al., 2 0 0 0 ; McClelland

completion,

et al., 1 9 8 9 ) . Along the same lines, it should be

perceptual identification and lexical decision,

possible to use priming as a measure of the sorts

and free association or category generation—

of implicit, unconscious attitudes, beliefs, and

even though they cannot recall or recognize

values of interest to social psychologists. In a

the items they studied. Because the list items

sense, deriving measures of individual beliefs,

are not accessible to conscious recollection,

attitudes, feelings, values, and motives repre-

priming evidently is an effect o f

sents a revival of Jung's ( 1 9 1 8 / 1 9 6 9 ) use of

unconscious

memory. T h e dissociation between explicit

response latencies on a word-association test as

and implicit memory also can be observed in

a "complex-indicator."

neurologically intact subjects w h o are not particularly amnesic, as when explicit memory is affected by an experimental manipulation, such as level o f processing, that has little or no effect on implicit memory.

The Importance of Matching Tasks In all o f this, it is critical that explicit and implicit expressions o f memory be assessed

These two examples, taken together, give

with

comparable

tasks.

Consider,

for

us the definition o f implicit memory as uncon-

example, an experiment in which explicit

scious memory: In the amnesic patients, prim-

memory is assessed with free recall but

o f conscious

implicit memory is assessed with priming on

ing occurs in the absence

recollection; in the normal subjects, priming

stem completion. Such an experiment might

occurs independently

o f conscious recollec-

well find that more target items are produced

tion. N o t e that the mere fact that a subject

on the priming task than on the free recall

completes the stem mar

task, but this would not be evidence o f a dis-

with

market

is not enough to qualify a

sociation between explicit and implicit mem-

behavior as an implicit or unconscious expres-

ory. T h e reason is that stem completion is a

sion o f memory. T h e word market,

variant on cued recall, in that the stem serves

rather than marble

or at least

some item plausibly related to it, has to have

as a cue for recall o f the whole word, and it is

been on the study list—if not, there is n o sense

well known that cued recall typically is supe-

in talking about memory. But note, t o o , that

rior to free recall. T h e same consideration

the mere fact o f priming is not sufficient to

would apply to a comparison o f free recall

permit discussion o f unconscious

memory.

with priming on perceptual

identification:

Priming occurs in nonamnesic controls, and

Perceptual identification is a variant on recog-

for deeply processed items as well. T o qualify

nition, in that the entire study item is re-

as unconscious, priming has to occur in the

presented on the memory test, and it is well

absence

of, or independently

of, conscious

recollection.

known that recognition is superior to recall. In psychometric terms, then, recognition is

The implicit-explicit distinction in memory

an "easier" test o f memory than cued recall,

can be extended to other domains as well,

which in turn is an "easier" test o f memory

including

than free recall. And in statistical terms,

perception

(Kihlstrom,

1996;

199

200

DESIGN AND ANALYSIS is just neuropsychological

Taylor Manifest Anxiety Scale (MAS) but high

jargon for statistical interaction: Dissociations

levels o f defensiveness on the M a r l o w e -

occur when one variable, such as population

Crowne Social Desirability Scale (SDS). In

(e.g., amnesic vs. nonamnesic) or experimental

other words, they reported low levels o f dis-

manipulation (e.g., level o f processing), inter-

tress, but their high levels of social desirability

"dissociation"

acts with another variable (e.g., explicit vs.

suggested that they might be repressing distress

implicit test) to affect performance. It is well

that they were actually experiencing uncon-

known that spurious interactions can occur as

sciously. O t h e r groups showing

artifacts of task difficulty (e.g., Chapman &

patterns o f M A S and SDS scores, such as non-

different

Chapman, 1 9 7 3 , 2 0 0 1 ) . Accordingly, in any

defensive nonanxious (low M A S , low SDS)

study o f dissociations between explicit and

and nondefensive anxious (high M A S , low

implicit measures, it is important that the

SDS) subjects, served as comparison subjects

tasks be matched as closely as possible on

(for

relevant psychometric characteristics.

Mulvaney, Kihlstrom, Figueredo, & Schwartz,

alternative classification schemes, see

clinch the case for a dissociation

1 9 9 2 ; Weinberger & Schwartz, 1 9 9 0 ) . In the

between explicit and implicit memory, we

experiment, Weinberger et al. asked their sub-

To

must show not just that implicit memory

jects to read phrases with sexual and aggressive

occurs in the absence of, or independently of,

content. Despite their general denial o f distress,

explicit memory. W e must also show that the

"repressors" showed increased response laten-

cues available to the subject remain constant

cies and

across tasks. Thus, the appropriate explicit

response during the task (see also Asendorf &

comparison for stem completion is stem-cued

Scherer, 1 9 8 3 ) . Weinberger et al. concluded

elevated levels o f physiological

recall. In stem-cued recall, the subject is asked

that the "repressors" were repressing after all:

to fill in a stem with an item from a previously

Although they denied being in distress, they

studied word list, thus requiring conscious rec-

clearly were disturbed by what they were

ollection; in stem completion, the subject is

asked to read. Put another way, repressive style

asked to fill in the stem with any appropriate

entails a dissociation between explicit and

word, thus obviating conscious recollection.

implicit expressions of anxiety (Kihlstrom,

Similarly, the appropriate explicit comparison

Mulvaney, et al., 2 0 0 0 ) — a t least in principle.

for perceptual identification is recognition. In

The Weinberger et al. ( 1 9 7 9 ) study is very

recognition, the subject is presented with a

provocative and deserves its status as a minor

copy o f a previously studied word and asked

classic in the experimental study o f psychody-

whether it was on a previously studied word

namics and defense, but it is not definitive evi-

list; in perceptual identification, the subject is

dence o f unconscious emotion. Setting aside

presented with the word and asked to identify

the question o f whether repressors really were

what it is. In the best comparisons, different

repressing anxiety or instead merely denying

items are tested explicitly and implicitly, so

felt distress to the investigators, the study did

that performance on one test does not contam-

not properly test for the dissociation between

inate performance on the other.

explicit and implicit emotion. For such a test

The importance o f test matching is illus-

to be valid, the cues presented to the subject

trated by a classic experiment on "repression"

would have to be the same for both explicit

performed by Weinberger and his colleagues

and implicit conditions. F o r example, subjects

(Weinberger, Schwartz, & Davidson, 1 9 7 9 ) .

might have to rate their emotional response to

In this experiment, the investigators were inter-

each phrase in the explicit condition, and rate

ested in a group o f subjects, labeled

repressors,

the difficulty o f reading the phrase in the

w h o reported low levels of trait anxiety on the

implicit condition. But in the Weinberger

Implicit Methods

in Social Psychology

\

et al. experiment, the "explicit" measure was

experiment, subjects were presented with pairs

reports o f generalized distress on the M A S ,

of letter strings and were asked to judge sim-

whereas the "implicit" measure was behav-

ply whether both were words. O n some trials,

ioral or physiological response to specific

the first word was black or white, and the sec-

sexual and aggressive phrases. There was n o

ond word was associated with the racial

assessment o f explicit emotional response to

stereotypes o f whites or blacks, such as smart

the phrases, so in the final analysis we don't

or lazy. In such a situation, the first word can

k n o w whether the "repressors" were repress-

be considered as a prime for processing the

ing anything at all. T o validate the concept o f

second word. N o t e that the subject's task

repression, we need evidence o f a dissociation

had nothing to do with social judgment.

between explicit and implicit measures o f

Nevertheless, white subjects responded more

emotional response to the same stimulus. T o

quickly when the stimulus paired a positive

validate the concept o f "repressive style" as

word such as smart with the prime white than

an individual-difference, we need evidence

when it was paired with the prime black; there

that this dissociation is greater in individuals

was no difference with negative words such as

identified as repressors, as opposed to nonde-

lazy. Wittenbrink and colleagues, in a similar

fensive subjects w h o truly are high or low in

study, found that white primed lexical deci-

anxiety.

sions concerning positive trait terms, whereas black

lexical decisions concerning

negative characteristics (Wittenbrink, Judd, &

Priming as a Measure of Implicit Attitudes

Park, 1 9 9 7 ) . Similarly, Blair and Banaji had subjects make lexical decisions about words

O f course, the Weinberger et al. study o f repression was performed before the criteria for explicit-implicit dissociations had been formulated, but the lesson holds. M o r e recently, Banaji and Greenwald applied the explicitimplicit distinction to the central social psychological concept o f attitude and other constructs, such as stereotypes and prejudice (Banaji & Greenwald,

primed

1 9 9 4 ; Blair,

2 0 0 1 ; Brauer,

Wasel, &C Niedenthal, 2 0 0 0 ; Greenwald & Banaji, 1 9 9 5 ; Greenwald, Banaji,

Rudman,

et al., 2 0 0 2 ; see also Wilson, Lindsey,

&

Schooler, 2 0 0 0 ) . Although social psychology traditionally has assumed that people

such as doctor and nurse that were primed by male or female first names, such as Jack or Jill (Blair &c Banaji, 1 9 9 6 ) . T h e general finding o f their research was that response latencies were shorter when there was a congruence between the gender o f the name and the gender-role connotations o f the word, as in vs. Jill-doctor.

Jack-doctor

O n the basis o f results such as

these, we might conclude that the lexical decision task reveals people's implicit, or unconscious,

stereotypes

concerning

race

and

gender. Another

priming-based

approach

to

are

implicit attitudes is represented by a series o f

aware o f their attitudes, these authors have

studies by Banaji and her colleagues o f the

suggested that people may possess positive

"false fame" effect documented by J a c o b y

and negative implicit

about them-

(Jacoby, Kelley, Brown, & Jasechko, 1 9 8 9 ;

selves and other people, attitudes that can

J a c o b y , Woloshyn, & Kelley, 1 9 8 9 ) . J a c o b y

attitudes

affect ongoing social behavior outside o f con-

has interpreted this effect in terms o f priming:

scious awareness.

Priming increases familiarity, which is incor-

An early example o f the use o f priming to

rectly interpreted as evidence o f fame. In their

study implicit attitudes is research by Gaertner

experiments, Banaji and Greenwald adapted

and

stereotypes

Jacoby's procedure by dividing the study and

McLaughlin, 1 9 8 3 ) . In this

test lists into equal numbers o f male and

McLaughlin

(Gaertner &

on

racial

DESIGN AND ANALYSIS

202

female names (Banaji & Greenwald, 1 9 9 5 ) .

with respect to the belief or attitude in question.

They found that the false fame effect was

A finding that whites, but not blacks, are more

greater for male than for female names, and a

likely to associate whiteness with smartness,

signal-detection analysis indicated that sub-

and that males, but not females, were more

jects adopted a lower criterion for judging

likely to associate maleness with fame, might

male names as famous than they did for

well support the attribution o f subjects' exper-

female names. Because the average subject

imental behavior to their social

was more likely to associate fame with males

rather than the structure o f the society in

than with females, Greenwald and

attitudes

Banaji

which they live. O f course, it could also hap-

concluded that the paradigm of false fame

pen that blacks and women adopt prevailing

provided evidence for "implicit gender stereo-

social stereotypes concerning race and gender,

types that associate male gender, more than

but

female gender, with achievement" (Greenwald

would at least provide some converging evi-

& Banaji, 1 9 9 5 , p. 1 6 ) .

dence that priming was a measure o f individuals'

Critique of Priming By this point in time, a fairly large number

an in-group

actual

vs. out-group

social attitudes

difference

rather

than

something more generic, such as their abstract knowledge o f stereotypes and prejudice found in their society.

of such studies have been published, too many

M o r e important in the present context,

to be reviewed comprehensively in this chapter

most ostensible studies of implicit social cog-

(for

comprehensive coverage, see Fazio &

nition either fail to make a comparison with

Olson, 2 0 0 3 ) . However, before we interpret

an explicit measure o f the same attitude or

such studies as providing evidence o f uncon-

employ an explicit measure that is inadequate

scious racism and sexism on the part o f sub-

to the task. F r o m the point o f view o f implicit

jects, a few questions need to be addressed.

social cognition, it is not interesting if subjects

First, it is not entirely clear that the perfor-

betray, by their performance in a priming

mance o f subjects in these sorts o f experiments

task, attitudes,

is indicative of their personal attitudes, as

that they are fully aware o f harboring. Priming

opposed to the structure o f the social environ-

effects that are congruent with a subject's con-

stereotypes, and

prejudices

ment. For example, perhaps doctors really are

scious beliefs and attitudes may well be inter-

more likely to be named J a c k and nurses more

esting unobtrusive or nonreactive measures,

likely to be named Jill; and given the hegemony

but more is required to make the inference

of the patriarchy, it may well be that an unfa-

that people's unconscious attitudes, beliefs,

miliar male is more famous (at least in some

and values actually are different from their

quarters) than an unfamiliar female. And

conscious ones. In addition to the indirect

although whites are no smarter than blacks,

assessment o f implicit attitudes, beliefs, and

and blacks are no lazier than whites, it may be

values, there must be a comparative direct

that the subjects' behavior was influenced by

assessment o f their explicit counterparts, and

their knowledge o f this common social stereo-

the correlation between explicit and implicit

type, rather than their personal endorsement of

measures o f the same attitude must be low—

it. This is especially a problem because the

certainly nonsignificant, preferably zero.

experimental task does not require subjects to

M a n y studies simply fail to provide this

make statements about themselves, but only to

sort o f comparison, in which case they stand

make judgments about language. Moreover, many studies o f implicit social

as little m o r e

than

demonstrations

that

attitudes, beliefs, and values can be displayed

cognition fail to test for differences between

in priming effects. W h e n studies do provide

in-groups and out-groups, or other stakeholders

this comparison, they often give contradictory

Implicit Methods results.

For

example,

Gaertner

and

in Social Psychology

skewed, sometimes highly so, and

\

often

McLaughlin ( 1 9 8 3 ) found that implicit racial

characterized by substantial

stereotyping occurred regardless o f subjects'

variability, requiring large numbers o f trials to

scores on a questionnaire measure of racial

achieve satisfactory levels o f reliability.

prejudice, whereas Wittenbrink e t a l . ( 1 9 9 7 ) found

many positive correlations between

within-subject

These considerations underscore the point that the study o f implicit attitudes, beliefs,

explicit and implicit measures. As it happens,

and values described in this chapter reverses

the Gaertner and McLaughlin ( 1 9 8 3 )

the role o f priming in experiments, from that

and

Wittenbrink e t a l . ( 1 9 9 7 ) studies used differ-

of dependent

variable

to that o f

independent

ent questionnaires to assess implicit racial

variable.

prejudice, but perhaps the most

ing not merely as a expression o f attitudes,

important

T h a t is to say, we wish to use prim-

problem is that they used questionnaires at all.

beliefs, and values that are already known

As noted earlier, the most compelling demon-

from people's responses on paper-and-pencil

strations o f the dissociation between explicit

questionnaires, but as an alternative measure

and implicit memory are provided by studies

of these attitudes, beliefs, and values that will

in which the stimuli presented to the subjects

predict people's behavior in a way that their

are the same, but the task demands are differ-

self-reports will not. T h e two roles may w o r k

ent. In memory studies, the explicit task refers

at cross-purposes. In experimental

to a past event, whereas the implicit task does

where the goal is t o determine h o w minds

not. Following this example, future attempts

w o r k in general, the best dependent variables

work,

to use priming tasks to demonstrate a dissoci-

are those that show relatively little between-

ation between explicit and implicit attitudes,

subjects variance. In

beliefs, and values should keep the stimuli pre-

work, where the goal is to predict what dif-

sented to the subject constant, and

vary

ferent people will do in a particular situation,

whether the experimental task refers expressly

the best independent variables are those that

to the subject's mental state.

show relatively wide dispersion across the

individual-differences

A final problem is that although implicit

population. T o the extent that response laten-

attitudes, beliefs, and values should function

cies measure h o w minds w o r k in general,

as individual difference variables, they are not

rather than h o w particular individual minds

always treated as such. In the Gaertner and

work, they may simply not present enough

McLaughlin ( 1 9 8 3 ) study, for example, prim-

variance to make them useful measures o f

ing scores were the dependent variables in an

individual differences.

experiment in which individual differences in racial prejudice served as a blocking variable in an analysis of variance design, and there

THE IMPLICIT

was no attempt to take account o f individual

ASSOCIATION TEST

differences in priming. In the Wittenbrink e t a l . ( 1 9 9 7 ) study, by contrast, both explicit

Although priming studies o f implicit social

and

cognition remain popular, Greenwald and his

implicit measures o f prejudice

construed as individual-difference

were

variables

colleagues recently have introduced another

entered into a multivariate analysis. Historic-

procedure,

ally, o f course, social psychologists have been

(IAT), for the measurement o f implicit atti-

allergic to individual

differences

(Bowers,

tudes,

the Implicit Association Test

beliefs,

and

values

(Greenwald,

1 9 7 3 ; Cronbach, 1 9 5 7 ) , but it may also be

M c G h e e , & Schwartz, 1 9 9 8 ) . Based on the

that priming scores are not well suited to being

general principle o f stimulus-response com-

treated as individual-difference measures. For

patibility (DeHouwer, 2 0 0 1 ) , the I A T requires

example, their distributions

subjects to make a series o f dichotomous

are

naturally

2

203

204

DESIGN AND ANALYSIS judgments about instances o f various concepts,

people's racial or ethnic prejudices, but also

such as black and white American names (e.g.,

that it reveals prejudices o f which the subjects

Alonzo or Adam, Amanda or Aiesha) and pos-

themselves are unaware.

itive and negative words (e.g., caress or abuse, or crash). These responses are made

freedom

by pressing different keys on a keyboard or button b o x . W h e n the two concept sets are combined, Greenwald and

his colleagues

The IAT as a Psychometric Device Since its formal introduction in 1 9 9 8 , the I A T has become extremely popular

as a

found that response latencies are faster when

method

associated concepts share a response key, com-

beliefs, and values in a number o f domains. A

pared to when they do not. Accordingly, in the

search o f the PsycINFO database (keyword:

for measuring

implicit

attitudes,

example cited, the observation o f faster laten-

" I A T " ) identified at least eight studies using

cies when a subject has to make the same

the I A T published in 1 9 9 9 and 2 0 0 0 , and as

response to white names and positive words,

many as 2 0 published in 2 0 0 1 alone. Given

compared to white names and negative words,

the inevitable delays o f the scholarly publica-

reveals an implicit association between white

tion process, this record o f adoption is quite

and positivity. By the same token, observation

remarkable. O f particular interest is a multi-

of faster latencies when a subject has to make

method psychometric study comparing three

the same response to black names and positive

implicit measures o f racial prejudice, includ-

words, compared to black names and negative

ing a priming procedure and two versions Qf

words, reveals an implicit association between

the I A T , in which all tests were completed in

black and positivity.

each o f four testing sessions separated by two

Using this procedure, Greenwald, M c G h e e ,

weeks

(Cunningham,

Preacher, &

Banaji,

e t a l . ( 1 9 9 8 ) showed that subjects implicitly

2 0 0 1 ) . Treating each individual trial as if it

associate flowers with pleasantness and insects

were an item on a test, the two versions o f the

(Experiment 1 ) ; that

I A T yielded acceptable if not outstanding esti-

Korean subjects implicitly associate Korean

mates o f internal consistency, perhaps reflect-

names with pleasantness and Japanese names

ing the inherent instability o f response latency

with

unpleasantness

with unpleasantness, and that Japanese sub-

measures noted earlier—as well as the fact

jects do the opposite (Experiment 2 ) ; and that

that I A T measures o f prejudice are calculated

white

white

as difference scores. R a w test-retest correla-

names with pleasantness and black names

tions for the I A T were relatively low, but after

subjects implicitly associate

with unpleasantness. In the latter two experi-

correction for measurement error these rose

ments, the I A T proved to be more sensitive to

considerably. And although the bivariate cor-

individual differences in ethnic or racial preju-

relations among the implicit measures were

dice (or, if you will, more discriminating) than

quite low, first- and second-order confirma-

explicit measures such as the feeling ther-

tory factor analyses revealed

mometer or the semantic differential (see also

convergence among them. Although

Greenwald & Farnham, 2 0 0 0 ) . In the Korean-

findings still must be confirmed in other

substantial these

Japanese study, the I A T correlated signifi-

domains, taken together they suggest that the

cantly with the feeling thermometer but not

I A T is a promising psychometric instrument

with the semantic differential. In the black-

for the evaluation o f implicit social attitudes,

white

beliefs, and values.

study,

standard

which

also

included

three

questionnaire measures o f racist

However, if utility (or efficiency) of mea-

beliefs, the correlations were uniformly non-

surement is considered as an important prop-

significant. T a k e n together, these

findings

erty o f a psychometric device (Mischel, 1 9 6 8 ) ,

suggest not only that the I A T can measure

it is not clear that the I A T is superior to a

Implicit Methods

in Social Psychology

j

standard priming procedure. In the Cunningham

such as buffering (Greenwald 8c Farnham,

et al. (2001) study, the two versions of the I A T

2 0 0 0 ) . In fact, the implicit measure o f self-

correlated .30 and . 4 8 , respectively, with the

esteem was somewhat more strongly predic-

Modern Racism Scale, while the corresponding

tive of buffering than the explicit measures

correlation for the priming procedure was only

employed.

.26. By any reasonable standard, a priming pro-

Rudman, et al. ( 2 0 0 2 ) have shown that the

cedure is easier to construct, administer, and

I A T gives m o r e satisfactory results

interpret than the IAT. If a low correlation with

explicit measures such as an attitude ther-

explicit measures is a desirable characteristic of

mometer or standard questionnaires, when

an implicit measure (and it must be desirable, if

compared against the predictions o f a variant

the implicit measure is to be truly implicit), then

on Heider's ( 1 9 4 6 ) balance theory.

Finally,

Greenwald,

Banaji, than

this status is achieved far more economically by a standard priming procedure than by the IAT. O f course, for any psychometric procedure

Critique of the IAT

there are always trade-offs between reliability

M o r e studies o f this sort are needed, espe-

and validity of measurement, on one hand, and

cially in light o f the fact that responses on the

utility, on the other. Investigators may be will-

I A T , and perhaps measures o f priming as

ing to sacrifice some utility in the service of

well, are subject to the influence o f a number

increased reliability and validity. Further com-

o f nuisance variables. F o r example, in their

parative studies probably are in order.

original paper, Greenwald, M c G h e e , et al.

O f course, these psychometric analyses are

(1998)

noted that

response latencies are

purely internal. W h a t about the relation with

affected by differences in the familiarity o f the

external factors? T o establish their validity, all

stimuli to which subjects respond, as well as

psychological tests, like the constructs they

by differences in evaluation. M o r e recently,

purport to measure, must relate in signifi-

Brendl,

cant ways to reasonable external criteria

argued that I A T indices o f anti-black atti-

Markman,

and

Messner

(2001)

(Cronbach & Meehl, 1 9 5 5 ; Loevinger, 1 9 5 7 ) .

tudes could be obtained not only from sub-

One very interesting finding is that I A T mea-

jects w h o actually held negative

attitudes

sures o f bias and prejudice toward an out-

toward blacks but also from subjects w h o

group occur even following a minimal group

held neutral

manipulation

&

blacks, so long as they were less favorable

Monteith, 2 0 0 1 ) . In successive experiments,

than their attitudes toward whites. Preferring

the I A T revealed prejudice by white American

white over black names is not necessarily evi-

subjects against names ostensibly associated

dence o f racism: It may be n o different from

(Ashburn-Nardo,

Voils,

or positive attitudes

toward

with Surinam (a real but unfamiliar country),

any other forced " c h o i c e " between two posi-

Marisat (a nonexistent country), and members

tively

of artificial groups (Quans and Xanthies) cre-

Stravinsky over Schoenberg or tiramisu over

ated by random assignment. Such a finding

zabaglione.

strengthens the inference that the I A T really

( 2 0 0 1 ) showed through computational mod-

valued

objects,

such

M o r e important,

as

favoring

Brendl et al.

measures prejudice after all. Another supportive

eling that "evidence" o f prejudice on the I A T

finding is that I A T measures o f anti-black

could emerge not only from differences in

prejudice predict the quality o f white subjects'

familiarity o f the targets being evaluated, as

actual behavioral interactions (e.g., body

Greenwald, M c G h e e , et al. ( 1 9 9 8 ) had also

openness, eye contact, and friendly laughter)

suggested, but also from differences in task

with black targets (McConnell 8c Leibold,

difficulty, which can induce subjects to shift

2 0 0 1 ) . Similarly, I A T measures o f self-esteem

their response criterion between the response-

predicted subjects' response to task failure,

compatible and response-incompatible blocks

205

206

DESIGN AND ANALYSIS of the I A T procedure. Differences in target

analysis o f four data sets collected over the

familiarity can be controlled for, at least in

Internet yielded implicit-explicit correlations

principle; but because response-incompatible

ranging from . 1 7 4 to . 7 7 5 , and

judgments are inherently more difficult than

approximately

response-compatible ones, this problem will

Banaji, 2 0 0 2 , Tables 2 - 6 and p. 2 0 ) .

remain. Brendl et al. remind us that although

averaging

.43 (Greenwald, Nosek, 8c

Like Greenwald, Cunningham et al. ( 2 0 0 1 )

prejudice (conscious or not) may well pro-

noted

duce an effect on the I A T (or, indeed, on any

relationship was weaker than the individual

measure

involving response latencies), an

relations among the implicit-implicit relation-

effect on the I A T or similar measure may not

ships, but the fact remains that the implicit-

that

the

overall

implicit-explicit

indicate prejudice, for the simple reason that

explicit relationship was far from trivial in

such an effect may have multiple causes, such

magnitude. In all these studies, explicit and

as target familiarity or task difficulty, that

implicit attitudes were dissociated in the weak

have nothing to do with prejudice.

sense o f not being highly correlated (no corre-

T h e question o f whether the I A T actually

lation is perfect), but not in the strong sense o f

assesses people's attitudes and beliefs raises

being entirely unrelated. In a sense, the ques-

the critical and thorny question o f whether

tion is whether the glass is half empty or half

the attitudes and beliefs revealed by the I A T

full, and it risks reviving one of the less savory

really are unconscious—that is, whether sub-

aspects o f the trait-situation

jects' responses on the I A T can be predicted

bedeviled the psychology of personality in the

debate

that

by measures o f their corresponding explicit

1 9 7 0 s and early 1 9 8 0 s : W h o s e correlations

attitudes and beliefs. In this regard, the evi-

are bigger? It should be understood, first, that

dence remains mixed. In their original paper,

the explicit measures used in these studies

Greenwald, M c G h e e , et al. ( 1 9 9 8 ) reported

are various forms of questionnaires and self-

that the average correlation between explicit

ratings, and the I A T is first and foremost a

and implicit measures o f the same construct

behavioral measure o f human performance.

(r = . 2 5 ) was lower than the average correla-

Correlations between these two classes o f

tions among explicit measures (r = . 6 0 ) , but

measures are notoriously (and, still, contro-

the implicit-explicit correlations were still

versially) low. T h e explicit-implicit correla-

numerically positive. Similar findings were

tions obtained by Greenwald, M c G h e e , et al.

obtained by Greenwald and Farnham ( 2 0 0 0 )

( 1 9 9 8 ) , for example, were well in line with the

in a comparison o f explicit and implicit self-

typical correlation between

esteem. In the Cunningham

(2001)

measures o f traits and attitudes, on one hand,

study, a first-order confirmatory factor anal-

and actual attitude-relevant behavior on the

ysis revealed significant relations between all

other (Sherman 8c Fazio, 1 9 8 3 ) . In a recent

et al.

three implicit measures of racism and

an

questionnaire

report on the psychometric characteristics of

explicit measure, and in fact the two paths

their instrument,

involving the I A T were stronger than the path

( 2 0 0 2 ) clearly state that "superior I A T mea-

Greenwald, Nosek, e t a l .

involving the priming-based measure. In a

sures should yield higher values for these

second-order analysis, in which the three tests

[implicit-explicit] correlations" (p. 5) and that

were considered to converge on a single latent

"even the smallest positive implicit-explicit

variable, the association between implicit and

correlations appear to demand an interpreta-

explicit prejudice remained strong. Another

tion in terms of construct overlap" (p. 2 0 ) .

study o f racial attitudes found that the I A T

Given that truly implicit measures o f attitude

was correlated significantly with an explicit

and

questionnaire

wncorrelated—with

(McConnell

measure

o f racial

prejudice

8c Leibold, 2 0 0 1 ) . A

recent

belief should

be dissociated—that

is,

their explicit counter-

parts, it seems clear that Greenwald, Banaji,

Implicit Methods

in Social Psychology

\

and their colleagues tend to view the I A T as an

linking cognition to actual social behavior that

unobtrusive

is the core o f social psychology.

measure o f subjects'

attitudes,

beliefs, and values, not as a measure o f truly

Nevertheless, if genuine progress is to be made, investigators need to distinguish among

unconscious mental states. In the final analysis, the issue o f the implicit-

three quite different topics: unobtrusive, non-

explicit relationship can be settled only by

reactive methods

employing measures

attitudes,

beliefs, and values; the automatic generation

beliefs, and values that are comparable to our

of these mental states, whether conscious or

measures o f their implicit

counterparts—

unconscious; and truly implicit, unconscious,

whether the implicit measures are provided by

attitudes, beliefs, and values. T h e relations

the I A T , priming, a psychophysiological mea-

among

sure, or something else. Only when sources of

Presumably, unobtrusive measures are used to

method variance are minimized, if not elimi-

reveal attitudes, beliefs, and values o f which

nated, can we hope to determine the true rela-

the person is consciously aware, but unwilling

tion between explicit and implicit measures o f

to disclose to others. Automatic processes may

o f explicit

o f measuring

these are s o m e w h a t

attitudes,

complicated.

social attitudes, beliefs, and values. It is not at

unconsciously activate attitudes, beliefs, and

all clear that the solution to this problem will

values, but these mental states themselves are

be as straightforward

not necessarily unconscious. Implicit attitudes,

for implicit attitudes,

beliefs, and values as it was for implicit

beliefs, and values may well affect a person's

memories, but the ambiguity o f the current

conscious experience, thought, or action, but

situation calls out for some determined effort

these mental states are, by definition,

to resolve it.

accessible to conscious awareness. If we are

not

interested in the truly unconscious determinants o f social behavior, we cannot be satisTHE UNOBTRUSIVE, THE AUTOMATIC, T H E IMPLICIT—AND THE PSYCHOLOGIST'S FALLACY Based on the model o f implicit memory, an increasing number of investigators are coming to take seriously the proposition that unconscious attitudes, beliefs, and values can influ-

fied

merely

unobtrusive

with

the

measures,

development

of

or even with

the

demonstration that certain attitudes, beliefs, and values are generated automatically. W e must also demonstrate that these mental states are unconscious, in the sense that they can be dissociated from, and are not predicted by, their explicit counterparts.

ence people's social interactions. O f course,

Herein lies the problem: W h a t do we take

such an idea was central to psychoanalytic the-

as evidence for an unconscious mental state?

ory, but the connection to modern cognitive

Long ago, William James noted that

the

psychology frees the idea o f unconscious

unconscious

for

influence from its Freudian death grip. N o w ,

believing what one likes in psychology, and of

when social psychologists talk o f the uncon-

turning what might become a science into

"is the sovereign means

scious, they use the same concepts and meth-

a tumbling-ground

ods as their cognitive colleagues—at least in

1 8 9 0 / 1 9 8 0 , p. 1 6 3 ) . J a m e s also cautioned psy-

for whimsies

(James,

principle. This line of inquiry is still in its

chologists against the psychologist's

infancy, or perhaps the toddler stage (though

which he defined as "the confusion o f his own

fallacy,

we can hope it will avoid the Terrible T w o s

standpoint with that of the mental fact about

and the White Food Stage), and it is extremely

which he is making

promising—not least because it links the inter-

1 8 9 0 / 1 9 8 0 , p. 1 9 6 ) . T h e psychologist's fal-

his report"

(James,

est in unconscious processes clearly present

lacy, in which we assume not only that every

in cognitive psychology with the interest in

event has a psychological explanation, but

207

208

DESIGN AND ANALYSIS also that our psychological explanation is the

they should not correlate with their explicit

correct one, is hard enough to resist with

counterparts. In fact, we want the correla-

mental states, as

tions between explicit and implicit measures

when we infer people's attitudes, beliefs, and

to be as close t o zero as possible, and to be

values from their behavior. But it is particu-

reassured that the low correlations are not

larly vicious with respect to people's

procedural or statistical artifacts.

respect to people's conscious

uncon­

mental states, when they are in no

In such a situation, the validity o f tests o f

position to authoritatively correct our infer-

implicit attitudes, beliefs, and values will rest

scious

ences. Psychoanalysts and some other insight-

on a package o f both convergent and dis-

oriented psychotherapists have been doing

criminant evidence ( C r o n b a c h &

this sort o f thing to their patients for a hun-

1 9 5 5 ; Loevinger, 1 9 5 7 ) . O n the discriminant

Meehl,

dred years (Freud, 1 9 0 5 / 1 9 5 3 ) . In the current

side, it must be demonstrated that implicit

revival o f interest in the psychological uncon-

attitudes, beliefs, and values are essentially

scious, it is important that we not perpetuate

uncorrelated with their explicit counterparts

their errors (Kihlstrom, 1 9 9 7 ) .

(see C o o k & G r o o m , Chapter 2 , this vol-

The study o f implicit memory and its cog-

ume).

Moreover, it is important that

the

nate phenomena addresses this problem in a

explicit measures employed be comparable

number o f ways. First, the implicit expression

with the implicit measures under considera-

of episodic memory, whether in the form o f

tion. It is not enough to compare priming or

priming or some other effect, has to be related

implicit

plausibly t o some independently verifiable

responses

associations

event in the subject's past personal experi-

explicit and implicit tests must be as closely

ence.

or

with

questionnaire

thermometer

settings.

The

between

comparable as possible, differing chiefly in

implicit and explicit memory is documented

whether they require subjects to reflect con-

by comparing subjects' performance on two

sciously on their attitudes, beliefs and values.

closely matched tests, one that refers to, and

O n the convergent side, it must be demon-

S e c o n d , the

dissociation

requires, conscious recollection o f a prior

strated that implicit attitudes, beliefs, and

event

values are associated with construct-relevant

and

another

one

that

does

not.

Something similar needs to happen in the

behaviors or experimental

study o f implicit social cognition if implicit

just

attitudes, beliefs, and values are to be consid-

Depending on theoretical considerations, it

as

their

manipulations,

explicit counterparts

are.

ered truly unconscious. First, we need assur-

may well be that explicit and implicit atti-

ance that the effect

consideration,

tudes, beliefs, and values are affected by dif-

whether a priming effect or an implicit asso-

ferent manipulations, correlate with different

ciation or something else, really is an expres-

variables, and predict different behaviors. O r ,

under

sion o f the person's attitudes and beliefs,

it might be that implicit attitudes, beliefs, and

rather than an artifact o f some stimulus prop-

values are more strongly related t o some

erty such as familiarity or some task properly

external variables than are their explicit

such as difficulty. Ordinarily, such evidence

counterparts. Whatever proves t o be the case,

would be provided by a positive correlation

it is important that these external relations

between the effect and the person's responses

should be construct-relevant, as defined by

t o a questionnaire or some other measure,

the investigator's theory o f the

(e.g., Fazio, Sanbonmatsu, Powell, & Kardes,

under investigation. In this way, we can

construct

1 9 8 6 ) , but that sort o f evidence is not avail-

avoid the psychologist's fallacy and have a

able when it comes to truly implicit attitudes,

genuine science o f the unconscious, not a

beliefs, and values, because by

tumbling ground for whimsies.

definition

Implicit

Methods

in Social Psychology

NOTES 1. The second lesson is that no amount of statistical finesse (Jackson, 1 9 7 1 ) can substitute for a few intelligent people writing items based on clear definitions of the construct to be measured (Ashton & Goldberg, 1973; Hase & Goldberg, 1967; Jackson, 1975). 2. Extensive information on the IAT, including demonstrations, generic software for constructing experiments, and bibliographies of published and unpublished work, is available on the World Wide Web at the following URLs: https://implicit.harvard.edu/implicit/ and http://faculty.washington.edu/agg/iat_ materials.htm. These Web sites also make reference to an "IAT Corp." which presumably was established to promote the IAT as a psychometric instrument for assessment of social beliefs and attitudes.

REFERENCES Asendorf, J . B., & Scherer, K. R. (1983). The discrepant repressor: Differentiation between low anxiety, high anxiety, and repression of anxiety by autonomic fl-verbal patterns of behavior. Journal of Personality and Social Psychology, 45, 1 3 3 4 - 1 3 4 6 . Ashburn-Nardo, L., Voils, C. I., &C Monteith, M . J . (2001). Implicit associations as the seeds of intergroup bias: How easily do they take root. Journal of Personality and Social Psychology, 81(5), 789-799. Ashton, S. G., & Goldberg, L. R. (1973). In response to Jackson's challenge: The comparative validity of personality scales constructed by the external (empirical) strategy and scales developed intuitively by experts, novices, and laymen. Journal of Research in Personality, 7(1), 1-20. Banaji, M . R., & Greenwald, A. G. (1994). Implicit stereotyping and unconscious prejudice. In M . P. Zanna & J . M . Olson (Eds.), The psychology of prejudice: The Ontario Symposium (Vol. 7, pp. 5 5 - 7 6 ) . Hillsdale, NJ: Lawrence Erlbaum. Banaji, M . R., 8c Greenwald, A. G. ( 1 9 9 5 ) . Implicit gender stereotyping in judg68, 1 8 1 - 1 9 8 . ments of fame. Journal of Personality and Social Psychology, Bern, D. J . (1983). Toward a response style theory of persons in situations. In R. A. Dienstbier & M . M . Page (Eds.), Nebraska Symposium on Motivation 1982: Personality—current theory and research (Vol. 3 0 , pp. 2 0 1 - 2 3 1 ) . Lincoln: University of Nebraska Press. Berg, I. A. (1955). Response bias and personality: The deviation hypothesis. Journal of Psychology, 40, 61-72. Berridge, K. C , & Winkielman, P. (2003). What is an unconscious emotion: The case for unconscious "liking." Cognition and Emotion, 17, 1 8 1 - 2 1 1 . Blair, I. V. (2001). Implicit stereotypes and prejudice. In G. Moskowitz (Ed.), On the tenure and future of social cognition Cognitive social psychology: (pp. 3 5 9 - 3 7 4 ) . Mahwah, NJ: Lawrence Erlbaum. Blair, I. V., &C Banaji, M . R. (1996). Automatic and controlled processes in stereotype priming. Journal of Personality and Social Psychology, 70, 1 1 4 2 - 1 1 6 3 . Bornstein, R. F. (2001). Clinical utility of the Rorschach Inkblot Method: Reframing the debate. Journal of Personality Assessment, 77(1), 39-47. Bowers, K. S. (1973). Situationism in psychology—Analysis and a critique. Psychological Review, 80, 3 0 7 - 3 3 6 . Brauer, M., Wasel, W., & Niedenthal, P. M . (2000). Implicit and explicit components of prejudice. Review of General Psychology, 4, 7 9 - 1 0 1 .

\

209

210

DESIGN AND ANALYSIS Brehm, J . W. (1966). A theory of psychological reactance. New York: Academic Press. Brendl, C. M., Markman, A. B., & Messner, C. (2001). How do indirect measures of evaluation work? Evaluating the inference of prejudice in the Implicit Association Test. Journal of Personality and Social Psychology, 81(5), 760-773. Chapman, L. J . , & Chapman, J . P. (1973). Problems in the measurement of cognitive deficits. Psychological Bulletin, 79(6), 3 8 0 - 3 8 5 . Chapman, L. J . , & Chapman, J . P. (2001). Commentary on two articles concerning Psychology, generalized and specific cognitive deficits. Journal of Abnormal 110(1), 31-39. Cronbach, L. J . (1957). The two disciplines of scientific psychology. American Psychologist, 12, 6 7 1 - 6 8 4 . Cronbach, L. J . , & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 2 8 1 - 3 0 2 . Cunningham, W. A., Preacher, K. J . , & Banaji, M. (2001). Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science, 12, 163-170. DeHouwer, J . (2001). A structural and process analysis of the Implicit Association Test. Journal of Experimental Social Psychology, 37, 4 4 3 - 4 5 1 . Dorfman, J . , Shames, V. A., & Kihlstrom, J . F. (1996). Intuition, incubation, and insight: Implicit cognition in problem solving. In G. Underwood (Ed.), Implicit cognition (pp. 2 5 7 - 2 9 6 ) . Oxford, UK: Oxford University Press. Duff, F. L. (1965). Item subtlety in personality inventory scales. Journal of Consulting Psychology, 29(6), 565-570. Fazio, R. H., & Olson, M . A. (2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297-327. Fazio, R. H., Sanbonmatsu, D. M., Powell, M . C , & Kardes, F. R. (1986). On the Psychology, automatic activation of attitudes. Journal of Personality and Social SO, 2 2 9 - 2 3 8 . Frank, L. K. (1939a). Projective methods for the study of personality. Journal of Psychology, 8, 3 8 9 - 4 1 3 . Frank, L. K. (1939b). Projective methods for the study of personality. Transactions of the New York Academy of Sciences, 1, 129-132. Frank, L. K. (1948). Projective methods. Springfield, IL: C. C. Thomas. Freud, S. (1953). Fragment of an analysis of a case of hysteria (Vol. 7). London: Hogarth Press and the Institute of Psycho-analysis. (Original work published 1905) Gaertner, S. L., & McLaughlin, J . P. (1983). Racial stereotypes: Associations and ascriptions of positive and negative characteristics. Social Psychology Quarterly, 46, 23-30. Goldberg, L. R., & Slovic, P. (1967). Importance of test item content: An analysis of a corollary of the deviation hypothesis. Journal of Counseling Psychology, 14(5), 4 6 2 - 4 7 2 . Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: Attitudes, selfesteem, and stereotypes. Psychological Review, 102, 4-27. Greenwald, A. G., Banaji, M. R., Rudman, L. A., Farnham, S. D., Nosek, B. A., & Mellott, D. S. (2002). A unified theory of implicit attitudes, stereotypes, selfesteem, and self-concept. Psychological Review, 109(1), 3-25. Greenwald, A. G., & Farnham, S. D. (2000). Using the Implicit Association Test to and Social measure self-esteem and self-concept. Journal of Personality Psychology, 79(6), 1022-1038. Greenwald, A. G., McGhee, D. E., & Schwartz, J.L.K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480. Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2002). Scoring procedures to improve implicit association test measures. Unpublished manuscript, University of Washington. Retrieved from http://faculty.washington.edu/agg/iat_materials.htm

Implicit

Methods

in Social

Psychology

Hase, H. D., & Goldberg, L. R. (1967). Comparative validity of different strategies of constructing personality inventory scales. Psychological Bulletin, 67(4), 231-248. Heider, F. (1946). Attitudes and cognitive organization. Journal of Psychology, 21, 107-112. Jackson, D. N. (1971). The dynamics of structured personality tests: 1 9 7 1 . Psychological Review, 78(3), 2 2 9 - 2 4 8 . Jackson, D. N. (1975). The relative validity of scales prepared by naive item writers and those based on empirical methods of personality scale construction. Educational and Psychological Measurement, 35, 3 6 1 - 3 7 0 . Jacoby, L. L., Kelley, C , Brown, J . , & Jasechko, J . (1989). Becoming famous overnight: Limits on the ability to avoid unconscious influences of the past. Journal of Personality and Social Psychology, 56, 3 2 6 - 3 3 8 . Jacoby, L. L., Woloshyn, V., 8c Kelley, C. (1989). Becoming famous without being recognized: Unconscious influences of memory produced by dividing attention. Journal of Experimental Psychology: General, 118, 115-125. James, W. (1980). Principles of psychology. Cambridge, MA: Harvard University Press. (Original work published 1890) Jung, C. G. (1969). Studies in word-association. New York: Russell 8c Russell. (Original work published 1918) Kent, G. H., & Rosanoff, A. J . (1910). A study of association in insanity. American Journal of Insanity, 67, 37-96. Kihlstrom, J . F. (1984). Conscious, subconscious, unconscious: A cognitive perspective. In K. S. Bowers 8c D. Meichenbaum (Eds.), The unconscious recon­ sidered (pp. 149-211). New York: Wiley. Kihlstrom, J . F. (1987). The cognitive unconscious. Science, 2 3 7 ( 4 8 2 1 ) , 1445-1452. Kihlstrom, J . F. (1996). Perception without awareness of what is perceived, learning without awareness of what is learned. In M . Velmans (Ed.), The science of con­ sciousness: Psychological, neuropsychological, and clinical reviews (pp. 23-46). London: Routledge. Kihlstrom, J . F. (1997). Suffering from reminiscences: Exhumed memory, implicit memory, and the return of the repressed. In M . A. Conway (Ed.), Recovered mem­ ories and false memories (pp. 100-117). Oxford, UK: Oxford University Press. Kihlstrom, J . F. (2002). Demand characteristics in the laboratory and the clinic: Conversations and collaborations with subjects and patients. Prevention & Treatment [Special issue honoring Martin T. Orne], 5. Retrieved from http://journals.apa.org/prevention/volume5/pre0050036c.html Kihlstrom, J . F., Barnhardt, T. M., & Tataryn, D. J . (1992). Implicit perception. In R. F. Bornstein & T. S. Pittman (Eds.), Perception without awareness: Cognitive, clinical, and social perspectives (pp. 17-54). New York: Guilford. Kihlstrom, J . F., Mulvaney, S., Tobias, B. A., 8c Tobis, I. P. (2000). The emotional unconscious. In E. Eich, J . F. Kihlstrom, G. H. Bower, J . P. Forgas, 8c P. M. Niedenthal (Eds.), Cognition and emotion (pp. 30-86). New York: Oxford University Press. Kihlstrom, J . F., Shames, V. A., 8c Dorfman, J . (1996). Intimations of memory and thought. In L. M . Reder (Ed.), Implicit memory and metacognition (pp. 1-23). Mahwah, NJ: Lawrence Erlbaum. Lilienfeld, S. O., Wood, J . M., 8c Garb, H. N. (2001). The scientific status of projective techniques. Psychological Science in the Public Interest, 1(2), 27-66. Lindzey, G. (1959). On the classification of projective techniques. Psychological Bulletin, 56, 158-168. Loevinger, J . (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 6 3 5 - 6 9 4 . Marks, J . D. (1979). The search for the "Manchurian candidate": The CIA and mind control. New York: Times Books. Marks, J . , 8c Greenfield, P. M . (1984a). The CIA inside the mind: Part 1. Psychology News, 35, 8-11.

DESIGN AND ANALYSIS Marks, J . , 8c Greenfield, P. M. (1984b). How the CIA assesses weaknesses: The Gittinger Personality Assessment System. Psychology News, 36, 7, 10-12, 19. McClelland, D. C , Koestner, R., 8c Weinberger, J . (1989). How do self-attributed and implicit motives differ? Psychological Review, 96, 690-702. McConnell, A. R., 8c Leibold, J . M. (2001). Relations among the Implicit Association Test, discriminatory behavior, and explicit measures of racial attitudes. Journal of Experimental Social Psychology, 37(5), 4 3 5 - 4 4 2 . Meehl, P. E. (1945). The dynamics of "structured" personality tests. Journal of Clinical Psychology, 1, 2 9 6 - 3 0 3 . Mischel, W. (1968). Personality and assessment. New York: Wiley. Mischel, W. (1972). Direct versus indirect personality assessment: Evidence and implications. Journal of Consulting and Clinical Psychology, 38, 319-324. Mulvaney, S., Kihlstrom, J . F., Figueredo, A. J . , 8c Schwartz, G. E. (1992). A continuous measure of repressive style. EGAD Quarterly, 1, 4 0 - 4 9 . Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17, 7 7 6 - 7 8 3 . Rapaport, D. (1942). Principles underlying projective techniques. In M . M . Gill (Ed.), Collected papers of David Rapaport (pp. 91-97). New York: Basic Books. Rapaport, D., Schafer, R., 8c Gill, M . M . (1944-1946). Diagnostic psychological testing. New York: Josiah Macy Jr. Foundation. testing Rapaport, D., Gill, M. M., 8c Schafer, R. (1968). Diagnostic psychological (Rev. ed.) (R. R. Holt, Ed.). New York: International Universities Press. Reber, A. S. (1967). Implicit learning of artificial grammars. Journal of Verbal Learning and Verbal Behavior, 6, 855-863. Rosenberg, M . J . (1965). When dissonance fails: On eliminating evaluation apprehension from attitude measurement. Journal of Personality and Social Psychology, 2(1), 28-42. Rosenthal, R. (1963). On the social psychology of the psychological experiment: The experimenter's hypothesis as unintended determinant of experimental results. American Scientist, 51, 2 7 0 - 2 8 2 . Russell, W. A., 8c Jenkins, J . J . (1954). The complete Minnesota norms for responses to 100 words from the Kent-Rosanoff Word Association Test (Technical Report No. 1 1 , Contract N8 O N R 6 6 2 1 6 , Office of Naval Research). Minneapolis: University of Minnesota. Schacter, D. L. (1987). Implicit memory: History and current status. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 5 0 1 - 5 1 8 . Seeman, W. (1952). "Subtlety" in structured personality tests. Journal of Consulting Psychology, 16, 2 7 8 - 2 8 3 . Sherman, S. J . , 8c Fazio, R. H. (1983). Parallels between attitudes and traits as predictors of behavior. Journal of Personality, 51, 3 0 8 - 3 4 5 . Weinberger, D. A., 8c Schwartz, G. E. (1990). Distress and restraint as superordinate dimensions of adjustment: A typological perspective. Journal of Personality, 58, 381-417. Weinberger, D. A., Schwartz, G. E., 8c Davidson, R. J . (1979). Low-anxious, highanxious, and repressive coping styles: Psychometric patterns and behavioral and physiological responses to stress. Journal of Abnormal Psychology, 88, 369-380. Wilson, T. D., Lindsey, S., 8c Schooler, T. Y . (2000). A model of dual attitudes. Psychological Review, 107(1), 101-126. Wittenbrink, B., Judd, C. M., 8c Park, B. (1997). Evidence for racial prejudice at the implicit level and its relationship with questionnaire measures. Journal of Personality and Social Psychology, 72, 2 6 2 - 2 7 4 .

C H A P T E R

10

Mediated and Moderated Effects in Social Psychological Research Measurement,

Design, and Analysis Issues

RICK H . HOYLE AND JORGIANNE CIVEY ROBINSON University of Kentucky

T

he most rudimentary research questions

represent constructs proposed to explain the

in social psychology c o n c e r n

association between two variables. In social

direct and

unqualified

the

association

between two constructs. Classic examples

are, Does behavior reflect attitudes?

and

psychology, mediators,

sometimes

termed

intervening variables or mechanisms, usually reflect cognitive, affective, or

motivational

Does similarity breed attraction? Although

processes by which an independent variable

such questions represent a fundamental, per-

influences a dependent variable. For instance,

haps essential, starting point for research on

attitudes might influence behavior through an

social behavior, they are but a starting point

elaborate cognitive process that involves selec-

for constructing a detailed and informative

tive attention and biased processing o f behav-

account o f it. In a theory-oriented discipline

ioral cues in the immediate environment (i.e.,

such as social psychology, we want to k n o w

selective attention

h o w attitudes give rise to behavior and why

mediate

similarity engenders

Moreover,

Mediators enrich theoretical accounts o f social

we want to k n o w the situations in which, or

phenomena by virtue o f their focus on process.

attraction.

the people for w h o m , these associations are

the

biased

processing

attitude-behavior

and

relation).

Questions that address the conditions that

strongest and weakest—that is, the condi-

qualify an association concern

tions that qualify the association.

Moderators

Questions o f " h o w " and " w h y " concern mediators.

Mediators

are variables

that

moderators.

are variables that represent con-

structs proposed to magnify, attenuate, cancel, or reverse the association between two vari-

AUTHORS' NOTE: During the writing of this chapter, Rick Hoyle was supported by grants R01-DA12371 and R43­ DA1123 from the National Institute on Drug Abuse and grant R 0 1 - M H 0 1 0 0 3 from the National Institute of Mental Health. Jorgianne Robinson was supported by a Wethington Fellowship from The Graduate School at the University of Kentucky.

213

214

DESIGN AND ANALYSIS ables. Statistical moderation can take many

transmitted through the intervening variable.

forms, but the defining feature o f a moderated

In the three-variable case, the remaining por-

effect is that the association between the inde-

tion o f the effect is transmitted through the

pendent variable and the dependent variable

intervening variable as an indirect

differs in strength or form at different levels

Although the inferential outcome of a test of

of the moderator. For example, attitudinal

mediation often is cast in either-or terms, this

effect.

similarity might be more predictive o f attraction

need not be the case. It is possible that a

for women than for men (i.e., gender moderates

particular mediator accounts for none o f a

the similarity-attraction effect).

Moderators

documented direct effect, all o f the effect, or

define the limits o f theoretical accounts o f

some, but not all, o f the direct effect. Because

social phenomena

through

their focus

on

qualifying conditions. T h e basic logic o f research on mediated and

moderated

effects

any particular intervening variable likely represents only one o f several mechanisms by

is

straightforward,

which an independent variable influences a dependent variable, the latter inferential out-

although there are complications and poten-

come, partial mediation,

tial pitfalls to the implementation o f either. In

come than full mediation,

is a more likely out-

the simplest studies o f mediation or modera-

favoring mediation is warranted.

if an inference

tion, a third variable is introduced into a

The evaluation of a moderated effect, in

research design that previously focused exclu-

conceptual terms, involves an evaluation of the

sively on the effect o f an independent variable

effect (direct or indirect) o f an

on a dependent variable. In the case o f medi-

variable on a dependent variable at different

independent

ation, the third variable usually is reflective o f

levels of a moderator variable. In contrast to

a process (e.g., emotion regulation, delibera-

evaluations of mediated effects, there is no

tion) and believed to be associated with both

assumption o f a previously documented or

the independent and dependent variables. In

demonstrable association between the inde-

the case o f moderation, the third variable usu-

pendent and dependent variables. Indeed, one

ally captures some relatively fixed character-

of the more appealing features of research that

istic o f the individuals or groups being studied

includes possible moderator variables is the

(e.g., gender, group size), feature

prospect o f finding an effect (albeit a qualified

o f the

immediate situation (e.g., number o f people

effect)

present, presence or absence o f a mirror), or

dependent variable when no main effect (i.e.,

of the independent

variable on the

secondary quality o f the independent variable

unqualified

(e.g., attitude

Traditionally in social psychology, moderated

importance, domain

o f ego

association) can

be

inferred.

threat) and need not be associated with either

effects have been referred to as

the independent

effects and evaluated as a matter o f course in

or dependent

variable in

order t o moderate their association.

research involving factorial designs,

At the conceptual level, the evaluation o f a mediated

effect

involves partitioning

interaction

the

from

which data typically are analyzed using analysis o f variance. Increasingly, however, social

effect o f an independent variable on a depen-

psychological studies include at least one inde-

dent variable into two portions, the direct

pendent variable or moderator variable that is

effect and the indirect effect. This evaluation

measured

assumes a documented or demonstrable effect

manipulated. T h e inclusion of such variables is

along a continuum

rather

than

of the independent variable on the dependent

a departure from a pure factorial design, and

variable, and the question is whether

the resultant data are best analyzed using tech-

any

portion o f this effect can be attributed to a

niques that do not evaluate interaction effects

particular intervening variable. T h e

as a matter of course (e.g., multiple regression).

direct

effect is that portion o f the effect that is not

In such cases, researchers must

manually

Mediation

and Moderation

\

construct interaction terms and evaluate them

same time, at one point in time, and the same

in strategically specified predictive equations.

strategy is used to measure all variables, yield-

Inferences regarding moderation are compli-

ing a single score for each one. Although the

cated by the fact that there are many patterns

opportunistic approach would appear to be

by which the effect o f an independent variable

maximally flexible, affording the researcher

on a dependent variable can vary across levels

considerable latitude in h o w to analyze the

of a moderator variable. These range from the

data once they are gathered, the approach is

crossover pattern, in which the independent

severely limited because, with rare excep-

variable has opposite effects on the dependent

tions, the status any variable is assigned in a

variable at the two levels or extremes o f the

statistical hypothesis test is arbitrary.

moderator variable, to interactions in which

Persuasive tests o f mediated and moder-

the effect is discernibly stronger or weaker but

ated effects are possible only in studies that

does not change direction when moving from

conceptualize and measure constructs with

one extreme to the other along the scale o f the

reference t o their predetermined status in a

moderator variable.

theoretical account o f the phenomenon or

In the remainder describe and

o f this chapter,

illustrate, using

a

we

detailed

process o f interest, a reasoned empirical

research.

In

approach

such

to

theoretical

example, basic strategies for designing studies

accounts, hypothetical constructs can be clas-

of mediated and moderated effects in social

sified uniquely as causes, effects, mediators,

psychology. These strategies address the three

or moderators, and the variables that repre-

primary

sent them in empirical research can, in turn,

concerns o f social psychological

research: measurement, design, and analysis.

be classified uniquely as independent, depen-

In the measurement section, we outline a gen-

dent, intervening, and moderator variables,

eral approach to measurement that provides a

respectively. Access t o a rich and detailed

strong foundation for testing hypotheses that

theoretical account is essential t o the devel-

involve mediation or moderation. W i t h regard

opment and testing o f hypotheses regarding

to design, we discuss strategies for gathering

social behavior, particularly hypotheses that

data that allow for inferences essential to

posit mediated and moderated effects.

definitive tests o f mediation and moderation. Finally, we outline statistical approaches to testing for mediated and moderated effects.

Practical Benefits of Theory

W e conclude the chapter with a section on the

Because o f Lewin's ( 1 9 5 1 ) early influence,

various stumbling blocks to a full implemen-

social psychologists have long been committed

tation of the strategies we present.

to building theoretical accounts o f the phenomena and processes they study. The most complete, and

therefore

useful,

accounts

clearly specify the status o f the constructs they

M E A S U R E M E N T ISSUES

comprise. T h e fundamental

distinction is

One approach to empirical research in social

between cause and effect and their empirical

psychology is to develop a list o f variables rel-

counterparts, the independent and dependent

evant to a phenomenon o f interest, find a sin-

variables. At the core of this distinction is the

gle self-report measure o f each, and, on one

concept o f causality ( M a r k &

Reichardt,

occasion, administer the set t o as many par-

Chapter 1 2 , this volume; West, Biesanz, &

ticipants as possible from the population o f

K w o k , Chapter 1 3 , this volume). The funda-

interest. There are numerous drawbacks to

mental criteria for establishing causality

this opportunistic

empirical

that (a) the cause and effect are associated (i.e.,

research. All variables are measured at the

causation implies correlation), (b) the cause

approach

to

are

215

216

DESIGN AND ANALYSIS precedes

(c) the

dependent variable. Moderators typically are

cause-effect association persists after the cause

the

effect

in

time, and

less abstract, often referring to fixed qualities

has been isolated from potential confounding

of individuals or groups or salient features of

variables either through randomized experi-

situations. Although moderators may be spec-

mentation or through statistical control. (See

ified in the initial statement o f a theoretical

Pearl, 2 0 0 0 , for a detailed treatment

account, they also may be added to the

that

addresses both statistical and philosophical

account on the basis o f empirical findings that

concerns and Salmon, 1 9 9 7 , for a purely philo-

emerge from tests o f the theory's basic tenets

sophical treatment.) T h e first criterion is met

using different methods and samples. For this

through empirical means, although attempts to

reason, moderators often signal an evolved

establish an association through empirical

theoretical account that has increased in speci-

means might initially be motivated by a theo-

ficity in order to more precisely account for

retical account that prescribes the association.

manifestations o f the phenomenon.

The temporal relation between constructs is difficult to establish; however, a well-articulated theory makes use of logic and published findings to assert the temporal precedence o f

Formally Designating the Status of Variables in a Model

some constructs over others. Finally, firm

An effective way o f communicating statis-

causal inferences from a documented associa-

tical hypotheses regarding the status o f and

tion between two constructs requires that the

associations among a set o f variables is the

association remain after the putative cause has

path diagram. A path

diagram

represents

been isolated from other constructs. These

variables either as boxes or ellipses, and asso-

potential alternative causes range from features

ciations between variables as either straight,

of the typical operational definition o f the

single-headed

putative cause (e.g., self-report bias, experi-

headed arrows. Boxes indicate variables for

mental artifacts) to constructs that are similar

which there are scores in the data matrix (e.g.,

arrows

or curved,

double-

to or frequently co-occur with it. Thorough

scale scores, observer ratings); ellipses indi-

theoretical accounts prescribe processes that

cate latent variables, which are inferred from

are specific to the posited causal constructs.

the commonality among subsets o f observed

Satisfaction o f these criteria establishes an

variables but for which there are not scores in

important

association

the data matrix (e.g., factors, components).

between two constructs such that, with a rea-

Straight lines indicate directional associations

asymmetry

in

the

sonable degree o f certainly, one can be desig-

(i.e., regression terms), and curved lines indi-

nated the cause and the other the effect.

cated nondirectional associations (i.e., corre-

In addition to causes and effects, detailed

lation terms). Although path diagrams are a

and

staple o f structural equation modeling, when

moderators. Among the constructs elabora-

used for communicating the status o f and

ted in theories, mediators typically are the

associations among a set o f variables, they

most abstract, as they often are mentalistic or

neither convey nor imply a particular analysis

otherwise phenomenological in nature (Kimble,

strategy. In the section o f this chapter on

theoretical accounts specify mediators

1 9 8 9 ) . Mediators occupy a position o f both

analysis issues, we outline multiple statistical

cause and effect in models that include medi-

approaches

ated effects. In a three-variable model, the

depicted in the path diagrams we present.

intervening variable is a proximal effect o f the

to evaluating the associations

Figure 10.1 is a path diagram representation

independent variable (or its interaction with a

of a model that includes the four types of

moderator)

variables we have described, nonarbitrarily

and

a proximal cause

o f the

Mediation

Figure 10.1

and Moderation

217

Path Diagram Illustrating the Status of Four Types of Variables and the Associations Between Them in a Prototypic Model Grounded in Theory

arranged on the basis of reasoned conceptual-

modeled as a latent variable; however, as we

ization and measurement. At the core o f the

demonstrate later in the chapter, the strongest

model is the basic association between an inde-

model is one in which all variables are assessed

pendent and a dependent variable. The inde-

using multiple operations. T h e association

pendent variable is represented at the left o f the

between the independent and dependent vari-

diagram as a b o x , indicating that it is in the

ables is expressed in cl, a regression coeffi-

data set as a single (perhaps composite) score.

cient. Variability in the dependent variable not

The dependent variable is represented at the

explained by other variables in the model is

right of the diagram as an ellipse, indicating

captured by the residual, e2.

that it is not explicitly represented in the data

At the center of the path diagram in Figure

set. T h e dependent variable in this case is a

10.1 is a single b o x representing an observed

latent variable inferred from the commonality

intervening variable. N o t e that the inclusion of

among three variables for which there are

this intervening variable provides an alterna-

scores in the data set, Yl, Y2, and Y3. Notice

tive means by which the independent variable

that two arrows point to each o f these vari-

can influence the dependent variable. T h e com-

ables. This indicates that variability in Yl, Y2,

bination o f paths al and bl represents the indi-

and

two

rect effect o f the independent variable on the

sources—that portion each shares with the

dependent variable through the intervening

Y3

has

been

partitioned

into

other two (i.e., the latent variable) and that

variable. This indirect effect can be contrasted

portion unique to each one, designated

with the direct effect o f the

ul,u2,

independent

and u3, respectively. T h e strength o f the asso-

variable on the dependent variable via path cl.

ciation between Yl, Y2, and Y3 and the latent

A model that includes only the independent,

variable they represent is captured in the factor

dependent,

loadings, 11, 12, and 13. In this hypothetical

nected by paths al,bl,

model, only the dependent

effect" assumption regarding the independent-

variable

was

operationally defined in multiple ways and

dependent

and intervening variables con-

variable

and cl makes a "main association

and

its

DESIGN AND ANALYSIS

218

explanation by the intervening variable. In all

sufficiently large that statistical tests of the

likelihood, the magnitude o f the cl path would

moderated effect are compromised (more will

vary across populations and situations. Indeed,

be said about this in the section o f this chapter

it is possible that the linear effect o f the inde-

devoted to analysis issues). This collinearity

pendent variable on the dependent variable is

problem is remedied rather simply by rescaling

apparent only for certain people or under cer-

scores on the independent and moderator vari-

tain conditions. Moderator variables capture

ables as deviations from their mean score, a

such qualifying conditions, and one position a

strategy known as centering.

moderator variable might occupy in a model is

virtue o f centering is that it facilitates interpre-

An additional

illustrated by the remaining two variables in

tation o f effects by establishing a zero point for

Figure 1 0 . 1 . Note that statistical tests of mod-

variables in the model (Aiken & West, 1 9 9 1 ) .

erator hypotheses require, in addition to the

Both the rl coefficient and the c2 path have

independent variable, two variables, one for

significant implications for statistical tests o f

which data were gathered directly, labeled

moderated effects as well, but we defer our dis-

"Moderator Variable" in the diagram, and one

cussion of these implications until the section

either implicit in the statistical analysis (e.g.,

of the chapter on analysis issues.

interaction effects in analysis o f variance) or

T o summarize, the path diagram is a useful

created by the investigator, labeled " I V x M V "

tool for communicating the status o f variables

in the diagram. Although it is the latter that

in a data set and the form o f the associations

represents the moderated effect o f the indepen-

among those variables. Path diagrams distin-

dent variable, the former must be included for

guish between observed and latent variables

statistical reasons (Cohen, Cohen, West, &

and between directional and nondirectional

Aiken, 2 0 0 3 ; Evans, 1 9 9 1 ) . T h e moderated

associations. Our labeling allows for impor-

effect o f the independent

the

tant distinctions between types o f directional

dependent variable, c3, or indirectly via a3 and

associations. Path coefficients we label " a "

is the effect

variable on

term,

concern the association between independent

IV x M V , above and beyond the main effects

and intervening variables. Those we label b"

of the independent variable (cl, al —> bl) and

concern the association between intervening

moderator variable (c2, al

and dependent variables. Coefficients we label

bl,

of the interaction

—> bl)

on the

u

dependent variable. In this model, the unqual-

" c " concern the association between indepen-

ified and qualified effects of the independent

dent and dependent variables. Path diagrams

variable

on

the

dependent

variable

are

also distinguish between errors o f measure-

expressed both as direct effects and indirect

ment, which we label " « " (for "uniqueness"),

effects through the intervening variable. The only details in Figure 1 0 . 1 we have not covered are the curved arrows that connect the

and errors o f prediction, which we label " e . " These are the

basic building

blocks for

describing and implementing tests of mediated

independent and moderator variables and their

and moderated effects and provide a founda-

interaction term. These represent

possible

tion for identifying and discussing relevant

covariation between each pair o f variables, and

measurement, design, and analysis issues in

the coefficients rl,

the remainder o f the chapter.

r2, and r3, index that

covariation. Although the magnitude of these coefficients is not always evident to researchers testing moderated effects (e.g., when multiple regression analysis is used), knowledge o f this

An Example In one theoretical account o f the associa-

information is important. O f particular con-

tion between attitudes

cern are coefficients rl and r3, which can be

association between an attitude toward an

and

behavior, the

Mediation

Figure 10.2

and Moderation

Path Diagram Depicting Moderated Influence of Attitude on Behavior Mediated Through Orientation Toward and Perceptions of the Attitude Object

NOTE: The 10 nondirectional associations, indicated as r , between the independent variable, moderator variables, t

and interaction terms are not shown.

object and behavior toward the object is

the object (Fazio & Williams, 1 9 8 6 ) . This

moderated by two factors: attitude accessi-

attitude-to-behavior process model is shown as

bility and motivation to deliberate about the

a path diagram in Figure 1 0 . 2 .

object (Fazio, 1 9 9 0 ) . W h e n attitude accessi-

In the path diagram, Attitude

toward

bility is high and motivation to deliberate is

Object, as specified in the theoretical account,

low, the association between attitude

is

and

an

independent

variable.

Attitude

behavior should be strong. Conversely, when

Accessibility and Motivation to Deliberate

the attitude is relatively inaccessible and the

are moderator variables, and their modera-

motivation to deliberate about the object is

tion o f Attitude toward Object is expressed in

relatively high, the association between atti-

the Attitude x Accessibility and Attitude x

tude and behavior toward an object should

Motivation interaction terms.

be weak. In other words, the influence o f

toward Object and Perceptions o f Object are

attitudes on behavior is qualified by accessi-

intervening variables that mediate the unqual-

bility o f the relevant attitude and motivation

ified (i.e., direct) effect o f attitude on behavior

Orientation

t o deliberate about the object o f the attitude.

as well as the effect o f attitude on behavior as

Research inspired by this model has focused

qualified (i.e., moderated) by attitude accessi-

on the mechanism by which highly accessible

bility. Behavior toward

Object is, in this

attitudes influence behavior. One such mecha-

theoretical account, a dependent variable.

nism is the orientation of visual attention toward the object. T h a t is, when an attitude is accessed, it functions to orient visual attention

Optimal

Measurement

toward the object (Roskos-Ewoldsen & Fazio,

Researchers are tempted to take at face

1 9 9 2 ) . Attention directed toward the object

value the empirical associations between vari-

gives rise to perceptions o f the object, which

ables in a model such as the one depicted in

are proximal determinants of behavior toward

Figure 10.2; however, these associations reflect

219

DESIGN AND ANALYSIS

220

more than just the strength and direction o f

attitude

accessibility. It is well

known,

associations in the context of the model. They

however, that any measurement strategy cap-

also reflect the quality o f the operational

tures other sources o f variability as well. These

definitions—their reliability and validity as

sources can range from nuisance constructs

observable manifestations of the constructs

such as fleeting distractions or facility with

prescribed by the theoretical account guiding

language to problematic confounding con-

the research. Although the reliability and valid-

structs such as socioeconomic status or con-

ity of operational definitions is important in

cern for appropriateness. Variability that is

any social psychological study, the importance

common to variables that were measured in

of these concerns is magnified in studies o f

the same way but represent different con-

mediated and moderated effects. As we illus-

structs is referred to as method variance. First

trate in the section on analysis issues, fallible

highlighted by Campbell and Fiske ( 1 9 5 9 ) in

measures of intervening variables can lead to

their demonstration o f the insights provided

an inference o f no mediation when the inter-

by a matrix o f correlations among several

vening variable partially or fully mediates the

traits each measured using several methods

independent-dependent variable association,

(i.e., multitrait-multimethod matrix), method

or only partial mediation when, in fact, the

variance reflects not what a score represents

intervening variable fully accounts for the asso-

but how it was obtained.

ciation. The fallibility o f moderator variables is particularly worrisome because it compounds error in the operational definition o f the independent variable. For these reasons, reliable and valid measures are essential in research on mediated and moderated effects.

The critical concern when key constructs are measured using a single method is that the substantive and methodological sources that give rise to variability in scores generated by the measure are completely confounded. In such cases, extraneous method constructs are

Any variable could be measured in a vari-

an alternative explanation for any observed

ety o f ways. Specific strategies range from the

associations. T h e method explanation is par-

ubiquitous self-report method t o emerging

ticularly compelling when all variables in a

strategies such as physiological monitoring

model are measured using the same method.

(Cacioppo, Tassinary, & Berntson, 2 0 0 0 )

The solution to this problem is rather straight-

and implicit associations (e.g., Greenwald &

forward: Either within each study or across

Farnham, 2 0 0 0 ) . F o r instance, in the attitude-to-behavior

attitude

process model,

accessibility typically is measured

studies within a research program,

opera-

tionally define key constructs using different

as the

strategies. W h e n constructs are operationally

state-

defined using multiple, different measurement

latency in responding to attitudinal

ments or in pressing a computer key indicat-

strategies, it is possible to divorce construct-

ing like or dislike

when presented an image

relevant from construct-irrelevant variance,

descriptor o f the attitude object (e.g.,

thereby clarifying inferences from observed

or

Fazio, Powell, & Williams, 1 9 8 9 ; R o s k o s -

associations between variables.

Ewoldsen & Fazio, 1 9 9 2 ) . Although attitude

As an example, assume that we measured

accessibility is virtually always indexed as

attitude toward an object with multiple mea-

response latencies, the stimulus to which

sures reflecting different measurement strate-

research

gies. M o r e specifically, imagine that

participants

respond

and

the

we

medium by which the stimulus is presented

acquired data on five measures o f attitude:

vary from one study t o the next.

peer report ( Y 2 ) , teacher report ( Y 2 ) , self-

The goal o f this measurement strategy is to capture variability in the theoretical construct,

report feeling thermometer (Y3),

self-report

semantic differential

self-report

(Y4),

and

Mediation

Figure 10.3

and Moderation

221

Path Diagram Illustrating the Partitioning of Variability in Indicators of Attitude Toward an Object Into Common, Unique, and Method Components

Likert-type scale (Y5). W e might model the

correlation, rl, reflects the fact that these two

theoretical construct represented

by these

measures share something in c o m m o n that

measures, attitude toward object, as shown in

they do not share with the remaining measures;

Figure 1 0 . 3 . T h e path diagram in Figure 1 0 . 3

both

represent

a

third-party

perspective

T h a t is, the

regarding the participant's attitude. (Because

focus is within rather than between con-

only two variables are involved, we model this

reflects a measurement

model.

structs. T h e model indicates that variability in

commonality as a simple correlation rather

each o f our measures can be attributed to

than as a factor.) Y3, Y4, and Y 5 are affected

three sources: construct, method, and unique-

by a second latent variable (technically, a sub-

ness. First, look above the boxes representing

factor), which reflects the fact that they share a

our five measures. N o t e that a single latent

c o m m o n influence beyond the influence they

variable, Attitude toward Object, influences

share with Yl and Y2. Factor loadings 16 to 18

all five variables; the strength o f this influence

reflect the influence of this method factor on

would be captured in the factor loadings, 11

these variables and, when contrasted with 13 to

to IS. N o w look below the five boxes. N o t e

15, provide information

that each variable is influenced by its own

influence of construct and method on those

uniqueness

measures.

term.

In m a n y

measurement

models, these two components, commonality

about the relative

W e r e this measurement model substituted

and uniqueness, would be all that is required to

for

model the theoretical construct as represented

Figure 1 0 . 2 , associations involving the atti-

by these measures. For illustrative purposes,

tude construct would not be biased by mea-

the Attitude

toward

Object

box

in

the model depicts two additional features of

surement error in the variables representing

measurement models. First, note that ul and

that construct because variability unique t o

u2 are correlated. Recall that Yl and Y2 are

individual measures or subsets o f the mea-

peer and teacher reports, respectively. T h e

sures (i.e., parcels) would be divorced from

222

DESIGN A N D ANALYSIS

variability c o m m o n to all the measures. This

point still. A third approach is the replicative

is a powerful strategy for contending with

strategy, in which all variables in the model are

measurement error that is particularly advan-

measured at two or more points in time.

tageous for tests of association involving mediators and moderators, for which the ill effects o f measurement error are pronounced.

Asserting Causal Priority Most

theoretical a c c o u n t s go

beyond

simply stating that two constructs are associ-

DESIGN ISSUES

ated to posit that one causes the other. Tests

Although our focus n o w shifts to issues o f

on the design o f the study. T h e opportunistic

o f such propositions m a k e serious demands design in studies o f mediated and moderated

approach described at the beginning o f the

effects, measurement remains a primary con-

chapter is not adequate for tests o f causal pri-

cern, for even if all constructs that are not

ority because, in the absence o f explicit tem-

manipulated are measured using several dif-

poral ordering, the status o f most variables

ferent

strategies and

measurement

error

extracted, the correct inference regarding the associations in a model is not always clear.

generated by this one-shot strategy in a statistical model is arbitrary. The

sequential

strategy

offers

some

T h e efficacy o f a set o f measures for testing

improvement in this regard; however, it is

mediated and moderated effects can vary

limited in an important way: Unless the puta-

dramatically depending on when the mea-

tive cause is manipulated and participants are

sures were administered

randomly assigned to levels o f it, there is no

relative t o each

other. T h e issue o f timing is relevant for two

means

key issues in such tests o f reasoned models:

nondirectional associations between vari-

causal priority and tests o f mediated effects.

ables. This situation is illustrated in Figure

o f differentiating

directional

and

Before elaborating on these issues, we note

1 0 . 4 . T h e simple path diagram in the top por-

three basic strategies for administering a set o f

tion of the figure represents a sequential

measures relevant to questions

model in which attitude toward an object was

involving

mediation or moderation in quasi- and non-

measured at T i m e 1 and behavior toward the

experimental designs. T h e most rudimentary is

object at T i m e 2 . T h e ambiguity in the mean-

strategy

ing o f a significant path, c, stems from the fact

and epitomizes the opportunistic approach t o

that attitude at T i m e 1 is not isolated from

empirical research. In this strategy, all mea-

stable

sures are administered at one sitting. A more

object, and behavior at T i m e 2 is not isolated

sometimes referred to as the one-shot

behavioral tendencies t o w a r d

the

sophisticated approach is the sequential strat­

from stability in attitude toward the object.

egy. In this strategy, the variables are assigned

As such, it is not clear whether c is a true

different statuses by the investigator, and the

representation

timing o f measurement for each variable corre-

attitude and behavior, or whether it, to some

sponds to its status in some model. For

unknown

instance, if an investigator wants to test the

ciation

o f the association between

degree, reflects the stable assobetween

attitude

and

behavior.

object

Moreover, despite the fact that the sequential

causes behavior toward the object and this

measurement strategy has introduced a tem-

hypothesis that attitude toward

an

causal influence is mediated by orientation

poral distinction between attitude and behav-

toward the object, he or she might measure

ior toward the object, it is not possible to infer

attitude at one point in time, orientation at a

directionality. There is no statistical proce-

later point in time, and behavior at a later

dure for sorting out the various explanations

Mediation and Moderation

Time 2

Time 1

0 Behavior

Attitude toward Object

toward Object

Attitude toward Object

toward Object

Attitude

ic3

Behavior

Behavior toward Object

Figure 10.4

toward Object

Path Diagrams Illustrating Two Strategies for Testing the Causal Influence of Attitude on Behavior: The Sequential Strategy (Top) and the Replicative Strategy (Bottom)

for an association, and the ambiguity persists

Timing and Tests of Mediation

even in the context o f more complex models Another issue relevant to when constructs

such as the one shown in Figure 1 0 . 2 . O n e solution to this inferential conun-

are measured concerns the spacing between

drum is illustrated in the path diagram in the

measurement occasions. F o r instance, refer-

b o t t o m portion o f Figure 1 0 . 4 . N o t e that

ring back to Figure 1 0 . 4 , we might ask

attitude and behavior toward the object are

whether the test o f the critical association

measured at both points in time, reflecting

reflected in c3 is more persuasive if T i m e 1

the replicative strategy. By including behav-

and T i m e 2 are separated by 15 minutes,

ior at T i m e 1, it is possible t o model the sta-

3 days, or a month. There are no simple rec-

bility o f behavior toward the object, reflected

ommendations regarding the spacing between

in cl, yielding a model in which the associa-

measurement occasions; however, it is impor-

tion o f interest between attitude at T i m e 1

tant that the spacing be consistent with the

and behavior at T i m e 2 , c3, is based only on

framing o f the measures. F o r instance, if, in a

variability in behavior unique to T i m e 2 . T h e

study conforming t o the design illustrated at

inclusion o f attitude at T i m e 2 permits the

the

modeling

toward

between T i m e 1 and T i m e 2 were 1 week, it

the object, cl, as well as the possibility that

would not be reasonable to ask respondents

the direction o f influence runs from behavior

to report on their behavior during the past 3 0

o f stability in attitude

bottom

o f Figure

1 0 . 4 , the

spacing

to attitude, c4. In the absence o f random

days (a frame often used in research on prob-

assignment to levels o f a manipulated vari-

lem behaviors such as drug use).

able, the replicative measurement o f key con-

A more subtle point concerns the spacing

structs allows for persuasive tests o f causal

between the independent and intervening vari-

hypotheses by ruling out alternatives that

ables and the intervening and dependent vari-

undermine causal inferences using data gen-

ables in a model such as the ones depicted in

erated by one-shot and sequential strategies.

Figures 1 0 . 1 and 1 0 . 2 . T h e issues are easiest to

223

DESIGN AND ANALYSIS

224

describe in a model with a single intervening

unambiguously causally prior to the dependent

variable such as the model shown in Figure

variable, and, assuming random assignment to

1 0 . 1 . Assuming the model corresponds to a

condition, there is no concern that the moder-

theoretical account in which the intervening

ator variable could be caused by the indepen-

variable explains the association between the

dent variable, ruling out the possibility that it

independent variable and the dependent vari-

mediates rather than moderates the effect of

able, the paths of interest are al,bl,

and cl. It

the effect o f the independent variable on the

the independent variable on the dependent variable.

dependent variable is fully mediated by the

Although moderated effects can be evalu-

intervening variable, then paths al and bl will

ated fully in a single experiment, such is not the

be nonzero in magnitude and path cl will,

case with mediated effects. Tests o f mediated

within sampling error, be zero. A key concern

effects require an evaluation o f the effect o f the

is too strong an association between the inde-

independent variable on the intervening vari-

pendent variable and the intervening variable,

able and, if both are manipulated in a single

which would be reflected in a large value for

experiment, there is, by definition, no effect o f

All else being equal, as al increases, bl

the independent variable on the intervening

decreases. And as bl decreases, cl, which we

variable. Nonetheless, the question o f causal

would like to equal zero, increases. Among

priority is critical in tests o f mediation, and the

al.

other things, the magnitudes o f al and bl are

experimental design, when properly applied, is

affected by the spacing between the indepen-

unmatched in its ability to answer this ques-

dent and intervening variables and the inter-

tion. T h e use o f experimental designs to evalu-

vening and dependent variables, respectively.

ate mediated effects fully requires a series o f

Given that our goal in testing for mediation is

experiments. T h e effect o f the independent

to find cl = 0 (more on this later), and that cl

variable on the dependent variable is evaluated

is diminished as bl increases in strength and

in an experiment in which the independent

bl is diminished as al increases in strength, it

variable is manipulated and the dependent

is advantageous to measure an intervening

variable measured. T h e effect o f the interven-

variable closer in time to the dependent vari-

ing variable on the dependent variable is eval-

able than to the independent variable. In

uated

in

an

experiment

in

which

the

short, tests o f mediated effects are maximally

intervening variable is manipulated and the

powerful when bl is larger than al.

dependent variable measured. (Conceivably, these two effects could be studied in a single

Experimental

Designs

experiment in which the independent

and

intervening variables are orthogonally manipu-

Although the dependent variable is mea-

lated.) T h e effect o f the independent variable

sured or observed in all studies o f mediated

on the intervening variable is evaluated in a

and moderated effects, independent, modera-

design in which the independent variable is

tor, and intervening variables can, to consider-

manipulated and the intervening variable is

able benefit, be manipulated and participants

measured. M o r e typically, mediated effects are

randomly assigned to levels o f these variables

evaluated in individual experiments in which

in such studies. In social psychological studies

the independent variable is manipulated and

designed to test moderated effects, it is not

the intervening and dependent variables mea-

uncommon for both the independent variable

sured. Although this approach has the desir-

and the moderator variable to be manipulated.

able property that it allows a strong inference

The payoff o f this approach is twofold. Both

of causality regarding the effect of the indepen-

the independent and moderator variables are

dent variable on the dependent and intervening

Mediation and Moderation variables, it is no better than the one-shot

225

(Note that the intervening variable is treated

a

as a dependent variable at this stage o f the

causal effect o f the intervening variable on the

analysis.) If these effects are statistically sig-

non-experimental

strategy for inferring

nificant, then analysis o f covariance is used to

dependent variable.

evaluate the effect o f the intervening variable (which n o w is treated as a covariate) on the dependent variable, controlling for the inde-

ANALYSIS ISSUES

pendent variable. If the effect o f the covariate There are two major analytic approaches to

is significant in this analysis, then there is sta-

testing mediated effects. C o m m o n to the two

tistical evidence o f a mediated effect. If the

approaches is this basic question: Is some por-

effect o f the independent variable also is sig-

tion of the effect o f an independent variable on

nificant, then the correct inference is partial

a dependent variable attributable to an inter-

mediation. If the effect o f the

vening variable? In the serial approach

variable is nonsignificant, then the findings

to

addressing this question, the data are evaluated

independent

warrant an inference o f full mediation.

against a sequential list o f criteria that must be

If the independent variable is operationally

met in order for this question to be answered

defined in such a way that it is not properly

affirmatively (Baron &c Kenny, 1 9 8 6 ) .

viewed as defining a factor typical o f factorial designs, then all three steps can be evaluated

1. The independent variable is associated with the dependent variable.

more effectively using regression analysis.

2. The independent variable is associated with the intervening variable.

and covariance are special cases, could be

3. The intervening variable is associated with the dependent variable, controlling for the independent variable. At this step, an additional evaluation is made regarding the magnitude of the association between the independent and dependent variables, controlling for the intervening variable. If the effect is, within sampling error, zero, then the correct inference is that the intervening variable fully explains the effect of the independent variable on the dependent variable. If, controlling for the intervening variable, the effect remains significantly different from zero, then the correct inference is partial mediation. There are two primary statistical procedures for implementing the serial approach t o testing mediated effects. If the

independent

(Regression, o f which analysis o f variance used for manipulated

or nominally scaled

independent variables as well.) In the threevariable case, Steps 1 and 2 require simple regression runs in which the dependent and intervening variables, in turn, are regressed on the independent variable. Assuming statistical support at Steps 1 and 2 , Step 3 requires regressing the dependent variable simultaneously on the intervening and

independent

variables and evaluating first the coefficient for the intervening variable, then the coefficient for the independent variable. I f the coefficient

for

the

intervening

variable

is

statistically significant, then there is evidence of a mediated effect. If the coefficient for the independent variable is nonsignificant, then there is evidence o f full mediation. Otherwise, the correct inference is partial mediation. An alternative strategy is the

integrated

in which all relevant effects are

variable is manipulated or measured on a

approach,

nominal scale, thereby defining levels o f a

evaluated within a single model. T h e model

factor, the effect o f the independent variable on

includes both the direct effect o f the indepen-

the dependent and intervening variables typi-

dent

cally is evaluated using analysis o f variance.

and the indirect effect o f the

variable on the dependent

variable

independent

226

DESIGN AND ANALYSIS

Figure 10.5

Path Diagram Depicting a Model in Which the Attitude-Behavior Association Is Mediated by Perceptions of the Object

variable on the dependent variable through an

effects in multiple regression models requires

intervening variable. T h e basic model for the

the creation o f interaction terms after center-

three-variable case is shown in Figure 1 0 . 5 .

ing scores on the independent and moderator

T h e coefficients in such models typically are

variables.

estimated using maximum likelihood proce-

Alternatively, moderated

effects can be

dures (as opposed to ordinary least squares

evaluated using the multigroup

typical of regression) using structural equation

This approach is particularly attractive when

modeling. T h e test o f primary interest in these

the moderator variable is manipulated

models is the indirect, or ab, effect. This test is

nominally scaled and the independent vari-

approach. or

available in most software packages for esti-

able is operationally defined in such a way

mating structural equation models, although

that it could be modeled as a latent variable.

new tests that are not yet available in these

In the multigroup approach, a single predic-

packages promise greater power and precision

tive equation

(MacKinnon, Lockwood, Hoffman, West, &

groups that vary on the moderator variable.

Sheets, 2 0 0 2 ) .

Corresponding

There are two approaches to evaluating moderated effects as well. In the

interaction

is estimated

separately

for

coefficients are statistically

compared and, to the extent that they differ from one group to another, the grouping vari-

the interaction effect is evaluated

able moderates the effect o f that variable on

in the context o f a single model that also

the dependent variable. M o s t statistical soft-

includes the unqualified effects o f the inde-

ware packages do not provide an option to

pendent

approach,

the

compare coefficients when the equations are

dependent variable. As with tests o f media-

estimated for the groups using ordinary least

tion, when the independent variable and/or

squares regression; hence, hand calculation

moderator variable are manipulated or mea-

sometimes is required. W h e n the equations

sured on a nominal scale, the

and

moderator

variables on

evaluation

are estimated as structural equation models,

typically is accomplished with analysis o f

the tenability o f between-groups equality con-

variance. As implemented in most statistical

straints imposed on the relevant coefficients

software packages, the interactions among all

can be evaluated without hand calculation. If

variables in factorial design are evaluated

equality constraints on the coefficients repre-

automatically. Multiple regression is prefer-

senting the effect o f the independent variable

able when either the independent or the mod-

are not tenable across levels o f the moderator

erator variable is operationally defined

to

variable, then there is statistical support for a

define a continuum. Evaluation o f interaction

moderated effect. If the model includes an

Mediation

and Moderation

227

intervening variable and indirect effect, this

from a sample and our measure o f perceptions

strategy can be used to test for

moderated

of the object, the intervening variable, is not

variability in the degree o f media-

error free. W e can show the ill effects o f mea-

tion across the groups representing different

surement error in the intervening variable by

mediation,

levels o f the moderator variable.

substituting hypothetical values in the following equation (Hoyle & Kenny, 1 9 9 9 ) :

Analysis Issues Specific to Mediation In

our

discussion

of

considerations

regarding when to measure, we highlighted the interdependence o f effects involved in

In this equation, r

zz

is an estimate o f the

reliability (e.g., coefficient alpha, test-retest r)

tests o f mediation. T h e relevant effects are

of our measure o f perceptions o f the object.

illustrated in Figure 1 0 . 5 , which displays in

T h e resultant value, c ,

path diagram form a simplified version o f the

would observe given certain values o f a and b

ohs

is the value o f c we

attitude-to-behavior process model that has

(.3 and . 4 , respectively, in this case)

provided context for our discussion. T h e

certain levels o f reliability o f the measure o f

direct effect o f attitude on behavior is repre-

perceptions o f the object. Recall that we k n o w

sented by c, and the alb combination cap-

that c = 0 in the population, a value that indi-

tures

the indirect effect

o f attitude

on

and

cates full mediation o f the attitude-behavior

behavior through perceptions. I f the theoreti-

association by perceptions o f the object. If our

cal account that inspired this model specifies

measure o f perceptions were perfectly reliable,

that perceptions fully mediate the attitude-

indicated by a value of 1.0, we would expect

behavior association, then we would hope to

to observe the population value o f zero for c

find c = 0 in this model. As noted earlier,

because ( 1 - 1 ) (.3)(.4) + 0 = 0. If r = . 9 , then

the c path is influenced by the strength o f b,

we would expect to observe c = . 0 1 2 . If r

the influence o f perceptions on behavior,

.8, a value more typical o f measures in social

and the strength of b is influenced by the

psychological research, we would expect to

ohs

a

a

=

strength of a, the influence o f attitude on per-

observe c = . 0 2 4 . It is not uncommon for the

ceptions. As such, it is critical that we not

reliability o f measures in social psychological

underestimate b lest we overestimate c and infer

research to hover closer to .7. Were this the

partial or no mediation through perceptions.

case, we would expect to observe c — . 0 3 6

W e already noted that it is advantageous to

despite the fact that c = 0 in the population.

position measurement o f the intervening vari-

With a relatively modest sample size, this

able—perceptions, in our example—closer in

coefficient would be significantly greater than

time to the dependent variable than to the

zero, leading to the erroneous inference that

independent variable to avoid the situation in

perceptions do not fully mediate the attitude-

which b is diminished because so much vari-

behavior association. In practice, this error

ability in the intervening variable is accounted

is more pronounced because, in the strongest

for by the independent variable (via path a)

cases o f mediation, the value o f c is trivial

that little is left to explain variability in the

but greater than zero. For instance, if c = . 0 7

outcome. Another means by which b can be

in the population, a value that might be

underestimated and c overestimated involves

viewed as trivial, the estimate o f c when r

measurement error. T o illustrate this point,

.7 will be . 1 1 , an effect size that cannot be

assume that we had population data on error-

ignored.

zz

=

free measures, and we knew that c = 0 , a = . 3 ,

It is clear that error in measures o f inter-

and b = A. N o w assume that we have data

vening variables poses significant problems

DESIGN AND ANALYSIS

228

for tests o f mediated effects. As such, it is crucial that researchers deal with measure-

Analysis Issues Specific to Moderation

ment error in such models. O n e approach is t o select a measure with

well-documented

high reliability, at least . 9 . Such measures are not always available. An alternative is to model the intervening variable as a latent variable,

as

illustrated

for

attitude

Figure 1 0 . 3 , thereby removing

in

uniqueness

from the variable before estimating associations between variables. Depending on the amount

o f information

available on

the

intervening variable, it can be modeled in one of four ways. T h e best approach is t o obtain multiple

measures

representing

different

measurement strategies. T h i s approach has the

additional

benefit

of

allowing

the

removal o f variability attributable to measurement

strategy. Typically, investigators

Measurement error also is detrimental to statistical tests o f moderation. Unlike in tests of mediation, in which the primary concern is measurement error in one variable (the intervening variable), in tests o f moderation the concern involves t w o variables (the independent variable and the moderator variable). In the same way that the moderated effect is represented as a product o f the independent and moderator variables, error in the variable that represents the moderated effect, the interaction term, is a product o f the error in the independent variable and the moderator

variable

(Busemeyer &

Jones,

1 9 8 3 ) . Specifically, the reliability o f the interaction term can be computed as

might have only one measure o f a mediator, but the measure is a multi-item scale. W h e n

2

r r it

this is the case, the individual items can be

+ r

mm

im 2

1+r ,

treated as indicators o f the latent variable, a strategy that provides no information about

where r and r

method variance but nonetheless segregates

pendent and moderator variables, respectively,

H

mm

are reliabilities o f the inde-

commonality across items from uniqueness

and r

associated with individual items or subsets o f

W h e n the independent and moderator vari-

im

is the correlation between the two.

items. Sometimes the number o f items in

ables are uncorrelated, the reliability of the

such scales is t o o large to effectively model a

product term is the product o f the two reliabil-

latent variable o f which each item is an indi-

ities. Thus, if we assume typical values o f relia-

cator (see M a r s h & H a u ,

1 9 9 9 ) . In such

bility, say . 8 , the reliability o f the product term

cases, a compromise solution is to sort items

will vary between . 6 4 , when the independent

into four or more composites, referred t o as

and moderator variables are uncorrelated, and

parcels. Parcels may correspond to subscales,

.7, when they are strongly correlated. This

subsets o f items that emerge from

diminished reliability is especially problematic

factor

analysis, or arbitrary groupings o f items.

because the detection o f moderated

A final strategy for dealing with measure-

under typical research conditions is a challenge

ment error in an intervening variable con-

for

cerns the situation in which there is only a

(McClelland & Judd, 1 9 9 3 ) .

single

indicator.

Assuming

there

reasons other than measurement

effects error

exists

As with tests of mediation, tests o f modera-

enough prior research on the indicator t o

tion are strengthened when measurement error

provide a good estimate o f its reliability, the

is extracted from variables prior to estimating

indicator can be modeled with a fixed error

associations. This extraction for independent

term equal to (1 - r ) s , where r

is the reli-

and moderator variables can be accomplished

is the variance o f

using one o f the strategies described earlier

2

a

ability estimate and s

2

scores on the indicator.

zz

for modeling intervening variables; however,

Mediation and Moderation extracting

measurement

error

from

\

the

issue o f measurement error in the measure o f

interaction term is significantly more compli-

attitude accessibility. T h e first drawback is

cated. O n e approach to this problem would be

not a concern if the moderator variable is

the specification o f a latent variable represent-

categorical and can take on relatively few

ing the interaction term. This specification is

values (e.g., male vs. female, Hispanic vs. African

complicated by the fact that the loadings and

American vs. White). T h e second drawback

uniqueness terms associated with this latent

can be addressed by choosing or developing

variable are nonlinear transformations o f their

measures o f the moderator variable that are

counterparts in the latent variables for the

highly reliable (i.e., r

independent

moderation are significantly compromised by

and

moderator

variables

mm

> . 9 0 ) . In short, tests o f

(Kenny & Judd, 1 9 8 4 ) . Although this nonlin-

measurement error in the independent

earity can be incorporated into the specifica-

moderator variables; hence, effective tests o f

tion o f the latent variable representing the

moderated effects require careful attention to

term, if the number of indicators o f the inde-

measurement error at both the measurement

pendent

and analytic phases.

and

moderator

variables exceeds

three, the specification becomes prohibitively complex.

and

As with tests o f mediated effects (but for different reasons), multicollinearity can pose

O n e solution t o this problem would be to

a problem for tests o f moderated effects. If

limit the number o f indicators o f the indepen-

r a w scores on a single indicator o f the inde-

dent and moderator variables to two or three,

pendent variable and a single indicator o f the

either by limiting the number o f measures o f

moderator variable are multiplied t o form

each or, in the case o f multi-item scales, creat-

the interaction term, the latter will be highly

ing item parcels. An alternative approach is to

correlated with one or both o f the original

use the multigroup approach described earlier.

variables (Aiken & West, 1 9 9 1 ) . T h e magni-

For instance, we might divide a sample o f 1 0 0

tude o f these correlations (e.g., r2 and r3 in

participants into two groups o f 5 0 corre-

Figure 1 0 . 1 ) can be substantial (> . 8 0 ) and

sponding to those highest and lowest in atti-

give rise to the problem typically associated

tude accessibility. W e could then fit a model to

with

data from the two groups, a model in which

tions—inflated standard errors. Fortunately,

multicollinearity in prediction

equa-

attitude toward the object is modeled as a

there is a rather simple fix for this form o f

latent variable and specified as a cause o f

multicollinearity, which is an artifact o f scal-

behavior. Using equality constraints, we could

ing rather than a substantively meaningful

compare the strength o f the influence o f atti-

instance o f correlation. I f the distributions o f

tude on behavior for high and low accessibil-

the variables are normal, then

ity groups. If the strength o f this association,

scores on each variable from the mean on

subtracting

reflected in the path coefficient, differs across

that variable (i.e., centering) prior t o creating

groups, then we infer that accessibility moder-

the interaction term will result in a zero cor-

ates the attitude-behavior association. There

relation between the interaction term and the

are two drawbacks to this compromise strat-

independent and moderator variables.

egy. First, we have taken a measure o f accessi-

The magnitude o f the correlation between

bility that differentiates among respondents in

the independent variable and the moderator

terms o f milliseconds and reduced it to a

variable (e.g., rl in Figure 1 0 . 1 ) and the cor-

coarsely categorized variable with only two

relation between each and the dependent vari-

values. Second, although we have effectively

able (e.g., reflected in cl and c2 in Figure 1 0 . 1 )

modeled measurement error out o f the atti-

influence the likelihood o f detecting a signifi-

tude measures, we have not addressed the

cant moderated

effect

in the presence o f

229

230

DESIGN AND ANALYSIS measurement error. T o illustrate this point,

dependent variable are, statistically speaking,

assume that we have measures o f attitude and

easiest to detect.

attitude accessibility that, when

multiplied,

Considerations regarding the association

yield an interaction term with reliability o f

between m o d e r a t o r

. 7 0 . M o r e o v e r , assume that we wish

ables

to

are

somewhat

and

dependent

more

vari-

c o m p l e x . As

achieve statistical power o f . 8 0 for detecting a

noted early in the chapter, the m o d e r a t o r

medium effect o f Attitude x Accessibility on

hypothesis implies nothing a b o u t the associ-

behavior. H o w large must our sample be? If

ation between the m o d e r a t o r and dependent

attitude and attitude accessibility were not

variables. In fact, the differential status o f

correlated with each other but were strongly

the independent and m o d e r a t o r variables is

correlated with behavior, then we would need

easiest to defend when there is n o reason t o

a sample size exceeding 2 5 0 in order to realize

suspect that the m o d e r a t o r variable could

statistical power o f . 8 0 . If, on the other hand,

cause the dependent variable (i.e., when the

attitude and attitude accessibility were moder-

t w o are uncorrelated). T o the extent that

ately correlated with each other but neither

the m o d e r a t o r variable could be construed

was correlated with behavior, then we would

as a cause o f the dependent variable, then it

need a sample o f about 1 4 0 to achieve the

is better viewed as a second independent

same level o f statistical power. T h e basic prin-

variable that operates synergistically with

ciple is that tests o f moderated effects are most

the other independent variable t o produce

powerful when the independent and modera-

variability

tor variables are correlated with each other

(Carver, 1 9 8 9 ) .

but unrelated to the dependent variable.

in

the

dependent

variable

W e have noted that both the reliability o f

This rather simple principle gives rise to a

the interaction term and the statistical power

number o f important considerations. O n e

of the test o f a moderated effect are greater

concerns the associations between the inde-

when the independent and moderator vari-

pendent and moderator variables, attitude

ables are associated. I f the independent and

and attitude accessibility in our example, and

moderator variables are associated, a key

the dependent variable, behavior. If power is

concern is the nature o f that association. It is

maximized when these effects are zero, under

important to establish that the independent

what conditions might we expect them to be

variable is not causally associated with the

zero? Focusing first on the independent vari-

moderator

able, it is well known that when the effect o f

example, for attitude accessibility to qualify

an independent variable on a dependent vari-

as a m o d e r a t o r

able is moderated but the interaction effect is

association, there must not be a causal asso-

ignored, the independent and dependent vari-

ciation between attitude and attitude acces-

variable.

To

o f the

draw

on

our

attitude-behavior

able would be related in a curvilinear fashion.

sibility. I f an association between attitude

In the case o f a crossover interaction, in

and attitude accessibility could be construed

which the effect o f the independent variable

as causal, then it is possible that

on the dependent

accessibility is a mediator rather than a m o d -

variable reverses

when

attitude

moving from one extreme to the other o f a

erator o f the attitude-behavior association.

moderator variable, the linear effect o f the

As with the question o f whether a potential

independent variable on the dependent vari-

moderator

able, ignoring the moderated

strued as an independent variable, the issue

effect,

will

variable might better be con-

effects

c a n n o t be resolved statistically outside a

that involve a change in the direction o f the

research design that allows for a persuasive

association between an independent

test o f the directionality o f an association.

be near zero. As such, moderated

and

Mediation and Moderation they are measured. Even when a theoretical

STUMBLING BLOCKS

account is very specific regarding the status o f A c o m m i t m e n t t o the principles outlined in

key constructs in a model o f the process under

this chapter c a n n o t always be translated into

investigation, the placement o f variables rep-

a research study that gives evidence o f that

resenting these constructs in a statistical model

commitment. In this final section, we touch

is, with few exceptions, arbitrary when the

on four impediments to full implementation

variables all are measured at the same time.

of the measurement, design, and

analysis

O n e exception is the randomized experiment, which clearly distinguishes the

strategies we have proposed.

independent

variable from all other variables in the model.

Ambiguous

It is important to acknowledge, however, that

Theory

even in randomized experiments, the fact that

T h e reasoned approach to measurement

potential intervening and moderator variables

presupposes a reasonably detailed theoretical

typically are measured at the same time is an

account o f the phenomenon under investiga-

issue. In non-experimental designs, the place-

tion. T h e attitude-to-behavior process model

ment o f the independent variable is a concern

we used t o illustrate the approach is such an

as well. W e noted the considerable inferential

account. Unfortunately,

are

power gained by adding a second measure-

somewhat rare, leaving investigators t o , at

ment occasion, noting the importance o f a

such models

best, c o m b i n e the opportunistic

and

rea-

replicative rather than sequential measure-

soned approaches t o measurement. T h i s state

ment strategy. Apart from a replicative mea-

of affairs is problematic, for the determina-

surement

tion o f h o w a variable figures into a social

cannot be used to develop a definitive test o f a

process rarely can be made o n purely statisti-

model that includes mediated or moderated

cal grounds. I f theory is not available and

effects.

strategy, non-experimental

data

definitive statistical tests are not possible, then the placement o f variables within

a

model is arbitrary and, therefore, inferences are, at best, suggestive o f future research. Theory can be ambiguous in a different way. It is possible, for example, that a theoretical account proposes a set o f mediating mechanisms, but it is not clear what variables might stand in for the constructs in statistical tests o f a model. T h e most useful theoretical accounts provide information not only about how constructs are associated with each other but also about how those constructs are associated with variables that might represent them (Edwards & Bagozzi, 2 0 0 0 ) . Less useful are theories that accord a prominent role to constructs that have no evident concrete manifestation.

One-Shot Data W e have noted that a key consideration beyond how variables are measured is when

Limited Sample Size A concern when using statistical models that allow modeling o f measurement error is sample size. This is because the maximum likelihood estimation characteristic o f statistical approaches involving latent variables is a large-sample technique. Invariably, this concern leads to the question, H o w large is large enough? A growing literature suggests that "large" is not as substantial as once thought; however, it is equally clear that a sample that is large enough for tests o f one model would be inadequate for tests o f another. Under ideal circumstances—that is, normally distributed variables

and

an

approximately

correct

model—maximum likelihood performs reasonably well with samples as small as 5 0 (although stability can be an issue; see M a r s h &

Hau,

1999).

T h e ideal circumstances

231

232

DESIGN AND ANALYSIS are rare in practice, and a more reasonable

CONCLUSION

recommendation given typical circumstances would be samples o f at least 2 0 0 (Hoyle &c

W e have drawn a distinction between two

Kenny, 1 9 9 9 ) . W h e n it is not possible to

approaches to measurement in social psycho-

achieve a sample this large, then it is critical

logical research: the opportunistic approach, in

that all measures are highly reliable (r

> .90)

which all variables are measured on one occa-

in order to avoid the ill effects o f measurement

sion using a single method, and the reasoned

xx

approach, in which variables are assigned dif-

error.

ferent statuses in a model based on theory, and the strategy by which they are measured and

Limited Number of Indicators

treated in statistical analyses is governed by

A more general concern is the number o f

their status in the model. W e focused most of

measures, or indicators, o f key constructs

our attention on intervening and moderator

available for most statistical tests. Since the

variables, demonstrating that these variables

1 9 5 0 s , social psychologists have been aware

warrant particular scrutiny in light of their sen-

o f the virtues o f multiple indicators for teas-

sitivity to measurement error and complica-

ing apart variability attributable to the con-

tions that arise from assumptions regarding the

struct

error

temporal ordering of the variables. Substantive

introduced by the method o f measurement

theory is central to the reasoned approach, and

(Campbell & Fiske, 1 9 5 9 ) . Relatively recent

a recurring theme in the chapter is the fact that

advances in statistical software have brought

the absence o f a detailed theoretical account

techniques that can be used to model mea-

introduces a measure of arbitrariness into mea-

of

interest

and

systematic

surement error (e.g., structural equation mod-

surement, design, and analysis. T o overcome

eling) into the mainstream. Y e t , the inclusion

these obstacles to high-quality research, we

of multiple measures and, better still, multiple

urge investigators to (a) start with a well-

operational definitions o f key constructs in

developed theoretical account; (b) use this

social psychological research remains

account to distinguish among

the

independent,

exception rather than the rule. As we have

dependent, intervening, and moderator vari-

discussed in this chapter, if a construct serves

ables; (c) develop a measurement strategy that

as an intervening or moderator variable in a

is sensitive to the differing status of variables in

model, then the issue o f error in the opera-

the model; and (d) operationally define each

tional definition o f the construct takes center

construct with multiple indicators, preferably

stage. In the absence o f multiple measures or

representing different modes o f measurement.

multi-item scales whose subscales, individual

In our view, a single study that meets these

items, or arbitrary groups o f items (i.e.,

criteria stands to contribute more to

parcels) can serve as indicators, tests o f medi-

understanding o f the phenomenon

ated and moderated effects are significantly

investigation than several studies that take the

compromised.

opportunistic approach.

our

under

REFERENCES Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Thousand Oaks, CA: Sage. Baron, R. M., &c Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182.

Mediation

and

Moderation

Busemeyer, J . R., & Jones, L. D. (1983). Analysis of multiplicative combination rules when the causal variables are measured with error. Psychological Bulletin, 93, 5 4 9 - 5 6 2 . Cacioppo, J . T., Tassinary, L. G., & Berntson, G. G. (Eds.). (2000). Handbook of psychophysiology (2nd ed.). New York: Cambridge University Press. Campbell, D. T., &c Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Carver, C. S. (1989). How should multifaceted personality constructs be tested? Issues illustrated by self-monitoring, attributional style, and hardiness. Journal of Personality and Social Psychology, 56, 5 7 7 - 5 8 5 . Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/corre­ lation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum. Edwards, J . R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 3, 155-174. Evans, M . G. (1991). The problem of analyzing multiplicative composites: Ipnteractions revisited. American Psychologist, 46, 6-15. Fazio, R. H. (1990). Multiple processes by which attitudes guide behavior: The M O D E model as an integrative framework. In M. P. Zanna (Ed.), Advances in experi­ mental social psychology (Vol. 2 3 , pp. 75-109). New York: Academic Press. Fazio, R. H., Powell, M . C , & Williams, C. J . (1989). The role of attitude accessibility in the attitude-to-behavior process. Journal of Consumer Research, 16, 280-288. Fazio, R. H., 8c Williams, C. J . (1986). Attitude accessibility as a moderator of the attitude-perception and attitude-behavior relations: An investigation of the 1984 presidential election. Journal of Personality and Social Psychology, 51, 505-514. Greenwald, A. G., & Farnham, S. D. (2000). Using the Implicit Association Test to measure self-esteem and self-concept. Journal of Personality and Social Psychology, 79, 1022-1038. Hoyle, R. H., 8c Kenny, D. A. (1999). Sample size, reliability, and tests of statistical mediation. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 195-222). Thousand Oaks, CA: Sage. Kenny, D. A., & Judd, C. M. (1984). Estimating the nonlinear and interactive effects of latent variables. Psychological Bulletin, 96, 2 0 1 - 2 1 0 . Kimble, G. A. (1989). Psychology from the standpoint of a generalist. American Psychologist, 44, 4 9 1 - 4 9 9 . Lewin, K. (1951). Field theory in social science: Selected theoretical papers (D. Cartwright, Ed.). New York: Harper & Row. MacKinnon, D. P., Lockwood, C. M., Hoffman, J . M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83-104. Marsh, H. W., 8c Hau, K.-T. (1999). Confirmatory factor analysis: Strategies for small sample sizes. In R. H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 2 5 1 - 2 8 4 ) . Thousand Oaks, CA: Sage. McClelland, G. H., & Judd, C. M . (1993). Statistical difficulties of detecting interactions and moderator effects. Psychological Bulletin, 114, 3 7 6 - 3 9 0 . Pearl, J . (2000). Causality: Models, reasoning, and inference. New York: Cambridge University Press. Roskos-Ewoldsen, D. R., &c Fazio, R. H. (1992). On the orienting value of attitudes: Attitude accessibility as a determinant of an object's attraction of visual attention. Journal of Personality and Social Psychology, 63, 1 9 8 - 2 1 1 . Salmon, W. C. (1997). Causality and explanation. New York: Oxford University Press.

Part III DESIGN AND ANALYSIS Section C Research Designs: Deciding the Specific Approach for Testing the Research Question(s), Why, and How

11

C H A P T E R

Experimental Design and Causality in Social Psychological Research S. ALEXANDER HASLAM University of Exeter CRAIG MCGARTY The Australian National

University

INTRODUCTION

it something about us (our nationality, our gender)? W a s it something about the hostile

In the quest to enhance understanding, scien-

person (his or her personality, or low self-

tific research can have many goals—to mon-

esteem)? O r was it something about the situ-

itor, to describe, to predict, to explain. E a c h

ation (previous personal grievances, current

of these activities has a place in every scien-

social conflicts)?

tific discipline, and this is no less true o f social psychology. Indeed,

because social

In social psychology, as in other branches of

the discipline, all these activities

are

psychology is part o f our everyday experi-

important, and each presents its own chal-

ence, we each perform these activities rou-

lenges. In this chapter, we focus on

tinely as we go about our daily lives. F o r

activity that m a n y researchers consider most

example, if someone is hostile toward us, we

fundamental to social psychological investi-

are likely (a) to observe that person's future

gation and the one that presents the most

behavior carefully, (b) to tell our

profound

friends

difficulties—that

of

the

providing

exactly what happened to us, (c) to try to

social psychological explanation.

anticipate when or whether that person will

see, there are a number o f reasons why the

behave this w a y again, and (d) to try to find

task o f explanation proves difficult; how-

out exactly why the person was hostile. W a s

ever, t w o factors are basic. T h e first is the

As we will

AUTHORS' NOTE: We would like to thank Lucy O'Sullivan, Andrew Livingstone, Jolanda Jetten, Stephen Monsell, Tom Postmes, and the editors for their comments on various drafts of this chapter.

237

238

DESIGN AND ANALYSIS ease (and regularity) with which explanation

focus are (a) the IV(s), (b) the DV(s), and

is confused with description. T h e second is

(c) the participant sample.

the difficulty o f establishing causal

relations

Although variables and participants repre-

between two or more social elements: the

sent the essential ingredients o f any experi-

thing or things to be explained (the

depen­

ment, their quality alone does not guarantee

and the thing or

that the experiment will be effective. As with

things that provide the causal explanation

a cake, they have to be assembled appropri-

(the independent

ately—usually on the basis o f a tried and

dent variable[s]

or DVs)

variable[s] or TVs).

These problems constitute an important

tested recipe. It is on this assembly

process

scientific and philosophical topic in their

and some o f the basic recipes for experimen-

own right. F o r this reason, psychologists and

tal design that we focus in the third section o f

social psychologists have had quite a bit to

the chapter. In particular, we examine fea-

say about them, and we will discuss some o f

tures o f design that can threaten

their insights in more detail later.

validity by compromising researchers' ability

experimental

to interpret experimental results correctly and to draw appropriate inferences from them.

Designing Controlled Experiments: The Aims and Structure of This Chapter

T h e two forms o f threat on which we focus are (a) those that compromise the interpretation o f the relationship between manipulated

Although challenging, the task o f making causal statements (and o f detecting erroneous causal inferences) becomes easier when we realize that this rests upon an ability to conduct research in which the researcher has con­

and measured variables (threats to validity)

internal

and (b) those that compromise a

researcher's ability to generalize experimental findings to other non-experimental settings (threats to external

validity).

trol of the causal variable. T h e principal way

In the fourth section, we explore a range o f

in which this is achieved is by conducting

issues surrounding the interpretation of results

The

from experiments. W e start by considering the

business o f designing and conducting such

implications o f social psychological research

experiments is the focus of this chapter.

on causal inference and then go on to discuss

appropriately

designed

experiments.

In simple terms, the chapter looks at the

a number

o f specific methodological

and

strategies researchers can employ to reduce

theoretical issues that are the focus o f current

the uncertainty with which causal statements

interest and debate. In particular, we discuss

can be made on the basis o f experimental

the logic and limitations of mediational and

evidence. Along the way, it also identifies a

other forms o f correlational analysis.

number o f design features that can undermine such confidence and introduce uncertainty into the process o f explanation. T h e chapter is organized into four main

W H Y AND WHEN SHOULD WE DO EXPERIMENTS?

sections. T h e first looks at what we can hope to achieve when we conduct experiments and

Imagine a research program

also identifies situations in which experiments

researcher is interested in understanding the

are not appropriate. T h e second section dis-

social psychology o f prejudice toward minor-

cusses the key components

ity groups. This is an example that

o f any experiment

in which

a

we

and the choices that need to be made in order

develop throughout the chapter, so at the out-

to explore research questions as effectively as

set, it is worth thinking generally about the

possible. T h e three components on which we

sorts o f questions that are likely t o motivate

Experimental

Design and Causality

\

this researcher and t o which he or she will

Taylor, 1 9 8 4 ) , these researchers' distinctiveness

be seeking answers. T h e researcher may be

account argued that such stereotypes could

interested in understanding the rise o f anti-

develop because memory processes tend to

Semitism in 1 9 3 0 s Germany, for example, or

overrepresent distinctive behaviors performed

the reasons for the resurgence o f antiminority

by distinctive groups.

sentiment in extremist political parties contemporary

W e s t e r n democracies

in

(e.g.,

Augoustinos & Reynolds, 2 0 0 1 ) .

T o test this idea, H a m i l t o n and Gifford ( 1 9 7 6 , Experiment famous

1) conducted

experiment in which

a

now-

participants

As a starting point, the researcher may

were presented with 3 9 behavioral state-

hypothesize that individuals display prejudice

ments about different individuals w h o were

toward other groups because those individuals

identified as members o f one o f t w o groups,

have a dysfunctional personality (perhaps they

G r o u p A and Group B . T h e r e were 2 6 state-

are authoritarians or petty tyrants; Adorno,

ments about Group A, o f which 1 8 were

Frenkel-Brunswik,

favorable and 8 unfavorable, and 1 3 about

Levinson, &c

Sanford,

1 9 5 0 ; Altmeyer, 1 9 8 8 ; Ashforth, 1 9 9 4 ) . In

Group B , o f which 9 were favorable and

seeking to take this question further,

the

4 unfavorable. T h u s , numerically, Group B

form o f

was a minority, but the ratio o f favorable t o

research to conduct. Among other things, this

unfavorable statements was the same for

research must n o w decide what

means deciding whether or not it is possible

both groups. Despite this, on a range o f

and

measures, participants tended to

appropriate

to conduct

experimental

research.

represent

Group B less favorably than Group A. In other words, it appeared that subjects formed an

The Logic of Experiments In simple terms, experiments are distinguished

from

other

forms

illusory correlation between

minority

group membership and negative behavior. of

research

(e.g., surveys) by the fact that they involve

In controlled experiments, a manipulation takes

place

under

conditions

in

which

researchers have taken steps t o ensure that

attempts not only to measure variables but

the only difference between the groups o f

also to manipulate them. W h e r e survey-based

participants whose responses are compared

research into authoritarianism and prejudice

is in the variable that has been manipulated.

might measure levels o f authoritarianism and

In Hamilton and Gifford's ( 1 9 7 6 ) experi-

acts o f prejudice in a given community, an

ment, the only immediately obvious differ-

experiment might involve attempts to manip-

ence between Group A and G r o u p B was in

ulate authoritarianism and then to measure

the number o f statements about each, and

the impact o f this manipulation on people's

thus there appeared t o be strong grounds for

hostility toward others.

concluding that it was G r o u p B ' s minority

A recent example o f research that

is

directly relevant to our research question is

status that led to it being judged more negatively than G r o u p A.

provided by the illusory correlation research

The basic scientific logic o f the social psy-

of David Hamilton and his colleagues. T h i s

chological experiment is that it attempts to

research was

and

reduce the range o f possible explanations for

to

some difference between experimental condi-

often

tions to just two possibilities: (a) the effect o f

Gifford

initiated

by H a m i l t o n

( 1 9 7 6 ) as part o f an attempt

understand why

it is t h a t

people

develop negative stereotypes o f

minority

some combination o f the experimental factors

cognitive

and (b) chance. As is well known, statistical

approach to such issues (e.g., see Fiske &

logic never allows us to decide between these

groups.

In line w i t h

a social

239

240

DESIGN AND ANALYSIS two alternatives, but it does allow us to

some other variable that appears relevant to

estimate the degree to which chance or

the pattern o f results that have been observed.

random variation is a plausible explanation.

For example, in Hamilton and Gifford's

T o the extent that we can confidently reject

( 1 9 7 6 ) original study, the size o f the groups

chance as an explanation for some difference,

was confounded with the alphabetical superi-

we can be more confident that our experi-

ority o f the letters used to describe them

mental manipulation has produced the effect.

(McGarty & de la Haye, 1 9 9 7 ) . It may be the

This allows us to define the perfect experiment

case, then, that results did not reflect negative

as one in which we can confidently reject

reactions to smaller groups, but instead nega-

chance as an explanation o f any effect (this is

tive reactions to groups called B rather than A.

achieved through adequate statistical power)

M o r e subtly still, in all illusory correlation

and in which, at the same time, we can rule

studies, the total number o f statements about

out other interpretations or processes.

a group (A or B) is confounded with the number o f positive statements about it. Thus, it could be argued that the illusory correlation

Replication

effect reflects the fact that there are simply

However basic one's experimental design,

more positive statements about one group

perfection is rarely, if ever, achieved. T h i s is

than about the other (Smith, 1 9 9 1 ) . Indeed,

as such an

elaborating on such arguments, our own work

important part o f science. Replication is one

has suggested that the illusory correlation

way

effect partly reflects the fact that under condi-

why researchers view replication o f reducing

uncertainty

about

our

results. I f we can show the same effects under

tions where participants

are motivated

to

varying circumstances, then we can be m o r e

differentiate between the two groups (because

confident in rejecting chance as an explana-

the labeling implies that there is some differ-

tion o f our results and zero in on the experi-

ence between them) (Bless, Strack, 8c Schwarz,

mental manipulation as the causal factor.

1 9 9 3 ) , there are real differences

Indeed,

o f support for the conclusion that Group A is

one

reason

why

Hamilton

and

in the amount

Gifford's research is so widely cited in social

better than Group B rather than for the con-

psychology textbooks is that the results have

clusion that B is better than A. T h a t is, in the

proved easy to replicate. As a result, the illu-

Hamilton and Gifford study, 2 2 pieces of

sory correlation effect is widely regarded as

information support the view that A is good

both reliable and

and B is bad (the 18 positive statements about

robust.

Group A and the 4 negative statements about Group B) but only 1 7 support the view that B

Identifying Plausible

Confounds

T h e other benefit we derive from programs of research is the ability to rule out some spe-

is good and A is bad (the 9 positive statements about Group B and the 8 negative statements about Group A ) .

cific alternative explanations o f our findings.

As a result o f the ever-present danger o f

However, there are several notes o f caution

confounds, the universe o f possible social psy-

here. In particular, it is wise to bear in mind

chological experiments is not

that there is always an infinite number o f

divided into studies that have confounds and

meaningfully

alternative explanations for any given experi-

those that do not, but into studies where a

mental effect. T h a t is, it is always possible t o

plausible claim for the existence of a confound

assert that any I V that is remotely interesting

has been made and those where no such claim

enough to merit consideration in the social

has been m a d e

psychological literature is confounded with

sounds depressing, fortunately many possible

(as y e t ) .

1

Although

this

Experimental there

Design and Causality

are statistical

\

confounds can be eliminated on the basis o f

whether

past empirical research and theory. These help

between them. T h e existence o f such interac-

interactions

us to recognize when confounds that seem

tions is important because the claims we can

plausible are unlikely to be. Therefore, a

m a k e about the relationship between IVs and

sound theoretical knowledge o f any research

DVs

area is always necessary to rebut such claims

interactions are present (see Aiken 8c West,

and to winnow the full set o f confounds down

1 9 9 1 , for a detailed treatment o f this topic).

t o a manageable and useful array.

become much

more complex

when

In practice, then, experiments often involve

In the case o f the illusory correlation effect,

attempts to see whether particular

combina­

does it make theoretical sense to argue that

tions o f TVs produce particular effects. Classic

people have a preference for letter A over B?

examples in persuasion research (see Petty,

Should we conclude that this confound under-

1 9 9 7 ) show that the relationship

mines the internal validity o f the original

message complexity and persuasion depends

experiment (i.e., our confidence that the effect

on factors such as the intelligence o f the audi-

has occurred for the reasons proposed by the

ence and the presence or absence o f distrac-

researchers)? T h e answer depends on

our

tions. T h a t is, people may be more convinced

choice o f D V and the existence o f plausible

by complex messages, but only when they are

causal relationships between the

sufficiently intelligent to process the message

proposed

between

confound and that D V . F o r a confound to be

and there are no distractions present to inhibit

taken seriously, there must be some plausible

message processing.

path between that confounding factor (either

Experiments represent the best possible

alone or in interaction with other factors

w a y to disentangle these sorts o f questions in

manipulated in the experiment) and the D V .

social psychology, as in other

T h e only way t o arbitrate claims of confounds

inquiry. However, there are good reasons not

fields o f

is on the basis o f theoretical knowledge o f the

to use experiments for answering some ques-

relevant variables.

tions. W h a t are these?

Accordingly, if the D V in a piece o f illusory correlation research is liking, then familiarity with the literature on letter preference effects

When Not to Conduct

Experiments

tells us that the confound is a plausible cause

T h e two main reasons for not conducting

of the results. This is because research suggests

experimental research are that it is (a) not fea-

that, other things being equal, people prefer

sible or (b) not ethical to conduct a true exper-

groups labeled " A " to those labeled " B " (see

iment. T h e former is most obviously relevant

M c G a r t y 8c de la Haye, 1 9 9 7 ) . However, if

in cases where the causal variables cannot be

the D V is accurate recall, then the confound is

varied over the time course o f an experiment.

less likely to have caused the results, because

If the key variable controlling some process is

(to our knowledge) there is no literature or

relatively stable, then it may be impossible to

theory that supports a link between alphabet-

manipulate. For example, if authoritarianism

ical superiority and memory.

is a relatively enduring and stable aspect o f a person, then we may simply be unable to vary

Multiple Factors Importantly, experiments provide us with

it. W e might be able to manipulate an analog construct (e.g., concrete thinking) but still not be able to vary the key theoretical variable.

the opportunity t o consider the simultaneous

Similar limitations may apply for ethical

effects o f multiple factors. T h e primary ques-

reasons. It may be possible to vary some fac-

tion when we consider multiple factors is

tors, but it may not be ethical or desirable to

242

DESIGN AND ANALYSIS do so (see Kimmel, Chapter 3 , this volume).

never answer them all. Accordingly, they need

The level o f fear or anger that a person expe-

to be seen as just one tool in the researcher's

riences in an experiment may be quite easy to

arsenal, and they need to be used prudently

vary, but we nevertheless might be reluctant to

rather than impulsively, often in conjunction

make our participants either afraid or angry.

with other methods. For example, if one

Another bad reason for doing experiments

wanted to fully understand prejudice toward

arises from attempts to generalize results

minorities in contemporary society, it would

situations.

be foolish simply to conduct experiments—

Experiments are well suited to testing theo-

these would almost certainly need to be com-

ries, and it is these theories that we need to

plemented with the other research strategies

generalize, not the results o f the experiment

that are discussed in other chapters of this vol-

per se. A good example o f this distinction

ume (e.g., surveys, quasi-experiments, and case

directly t o other

settings and

is provided by the famous minimal group

studies; M a r k & Reichardt, Chapter 1 2 , this

studies o f Tajfel, Flament, Billig, and Bundy

volume; West, Biesanz, & K w o k , Chapter 1 3 ,

( 1 9 7 1 ) . T h e studies found that people w h o

this volume). Indeed, it is worth adding that it

were assigned to groups that had no prior

would also be foolish to confine one's thinking

meaning at all were willing t o discriminate in

to issues o f psychology (Reicher, 2 0 0 1 ) .

favor o f a member o f their in-group over a member o f an out-group, even though they did not k n o w the people in question and had no opportunity to benefit from this behavior. If one were to generalize directly from this finding to the world at large, one might conclude that people automatically favor their

EXPERIMENTAL COMPONENTS

Independent Variables (TVs) As noted above, the defining features o f

in-group over comparison out-groups. Indeed,

experiments are

(a) that

they attempt

to

researchers often use the minimal group find-

manipulate one or more variables, not just to

ings to make exactly this claim (see J o s t &

measure them, and (b) that the various levels

Elsbach, 2 0 0 1 ) . However, this is not the the-

of manipulated variables represent the only

oretical point that social identity researchers

difference between groups whose responses

were trying to make when they conducted the

are compared. W h e r e the responses o f two or

research, and neither is it a basic assumption

more different groups

of social identity theory (see M c G a r t y , 2 0 0 1 ;

best way t o achieve this is through

are compared,

the

random

o f participants t o experimental

Turner, 1 9 9 9 ) . T o generalize directly from

assignment

experimental

conditions in which they are exposed to

findings

rather

than

from

theory is to be guilty of naive empiricism (see

different levels o f the variable in question.

below), and although it is tempting to do this, it can lead to serious misunderstanding. The final bad reason for doing experiments hinges on a naive ideological commitment to

Randomization A

full explanation o f the reasons

for

the experimental method. As we have been at

randomly assigning participants to conditions

pains to point out elsewhere (Haslam &

appeals to a knowledge o f statistical principles

McGarty,

that were first elaborated by Fisher (e.g.,

1 9 9 8 , in press), being a good do

1 9 2 6 ) . T h e short explanation is that the pro-

experiments but about when you do them.

cedure ensures that experiments don't build in

Experiments help us to answer many impor-

differences

tant questions about a given topic, but they can

tially significant dimensions (e.g., in terms o f

researcher is not about whether

you

between groups—either on poten-

Experimental sex, age, intelligence, or personality) or on

Theoretical

Design and Causality

\

Relevance

seemingly trivial ones (eye color, hair length, weight). Accordingly, randomization involves

It follows from the above that all experi-

assignment o f partici-

ments must contain at least one manipulated

pants to conditions (for example, based on the

variable—an TV. O n the basis o f a theory or

toss of a coin).

hypothesis, this variable is predicted to have a

strictly unsystematic

An example can illustrate why it is impor-

causal effect on at least one measured outcome

tant to recognize that even the most trivial sys-

variable—a D V . T h e goals in selecting an IV

tematic difference can undermine a researcher's

for an experiment are quite straightforward:

ability to make an appropriate causal inference.

they are (a) to identify a variable that can be

Imagine that we conducted a study to examine

manipulated and (b) to ensure that this vari-

the impact of discrimination on collective self-

able corresponds as closely as possible to the

esteem in which we had two conditions: one

theoretical variable in which the researcher is

where people were given the opportunity to dis-

interested. However, satisfying these objectives

criminate (the experimental condition) and one

can be quite difficult—partly because there is

where they were not (the control condition)

often an inherent tension between them.

(see Oakes & Turner, 1 9 8 0 ) . W e might place

T o explain these points, let us develop an

people into different conditions on the basis of

example in which a researcher is interested in

their position on a roll, putting people whose

the social psychological underpinnings o f dis-

names are high in the alphabet in the control

crimination (e.g., H o g g & Sunderland, 1 9 9 1 ;

condition and those with names low in the

Hunter, Platow, Bell, Kypri, & Lewis, 1 9 9 7 ;

alphabet in the experimental condition. This

Long &c Spears, 1 9 9 7 ; O a k e s &

violation of the principle o f random assign-

1 9 8 0 ) . Social identity theorists (after Tajfel

Turner,

ment may seem relatively inconsequential,

and Turner [ 1 9 7 9 ] ) have argued that, in par-

and we may have good reasons for doing it

ticular social contexts (e.g., where groups are

(e.g., to minimize administrative confusion).

perceived to be in conflict and the relations

Nonetheless, the procedure may inadvertendy

between

build ethnic differences into the two groups, so

increase group-based self-esteem can increase

that people from

them

are unstable), a desire

to

one geographical region

people's willingness to engage in acts o f

(perhaps where names starting with Y and Z

discrimination (e.g., as found in the minimal

are common) are in the experimental condition

group

and people from another region in the control

Flament, et al. [ 1 9 7 1 ] ) .

studies

discussed

above;

Tajfel,

condition. In this case, any difference subse-

Among other things, this theory suggests

quently observed in the self-esteem o f partici-

that if group relations are unstable and people

pants in the two conditions might therefore be

are given an opportunity to discriminate against

due to differences in ethnicity rather

than

differences in physical reinforcement. T h e primary strength o f the experimental method is thus that, providing

an out-group, their collective self-esteem (CSE) will increase. However, self-esteem is not

participants

expected to increase if they are not given this opportunity. Alternatively, the theory

has

conditions,

been interpreted as suggesting that if people

researchers can be sure that effects observed

have low collective self-esteem, they will be

on the D V are due t o one o f two things: their

motivated to discriminate, but if they have

manipulation o f the I V or chance. In this

high

are

randomly

assigned

to

way, it gives researchers experimental

self-esteem they will not

(Hogg &

control

Abrams, 1 9 9 0 ; H o g g & Sunderland, 1 9 9 1 ;

both over what they investigate and over the

see Turner, 1 9 9 9 , for an extended discussion

causal interpretation o f their findings.

of these inferences and their limitations).

243

244

DESIGN AND ANALYSIS If we were t o design an experiment t o test

Manipulation

Checks

the second o f the above theoretical proposiHaving performed indirect manipulations

tions, it would need to manipulate people's self-esteem in order t o see whether this has

of

any effect on discrimination. As often occurs

researchers to show that those manipulations

this

sort,

it

often

is necessary

for

in social and other psychological research, an

have had the desired effect on the relevant

immediate problem that arises here is that it

theoretical variable. Researchers typically

is impossible to manipulate something like

achieve this by employing a

self-esteem directly.

W e cannot open up a

check

person's head and

switch the self-esteem

chology journals require these to be con-

manipulation

(indeed, many editors o f social psy-

synapse (if there were such a thing) from

ducted

as a m a t t e r

o f principle

before

"off" to " o n . " Similarly, we c a n n o t directly

accepting research for publication). A manip-

manipulate a whole host o f social psycholog-

ulation check is a D V , but unlike other D V s ,

ical variables such as m o o d , self-efficacy, and

it checks that the manipulation o f an I V has

personality. Indeed, as we noted above, even

had

if we could, it often would be ethically unac-

causal variable. F o r this reason, obtaining an

ceptable to do so (see Kimmel, Chapter 3 ,

effect on this type o f measure is not an end it

an effect

on a theoretically relevant

itself but rather a way o f ensuring that an

this volume; N i Bhrolchain, 2 0 0 1 ) . The problem here relates to general issues

experiment actually has been conducted (i.e.,

of research validity and psychological mea-

that the I V has been effectively manipulated).

surement. It arises from the fact that social

So, after making people listen either to a

psychological research generally is concerned

dirge or a dance-track, a researcher may ask

with making statements about

them h o w they feel in order to check that the

theoretical pro-

people w h o listened to the dance-track were

cesses—that are not directly observable. This

in a better m o o d . Likewise, having attempted

means that researchers often have to manipu-

to manipulate participants' collective self-

late theoretical variables indirectly,

esteem by giving false feedback about their

variables—usually

mental

states or

by using

IVs that they believe will have a specific

group's

impact on a given mental process or state. In

could ask them to complete a series o f self-

test performance,

the

researcher

other words, they have to m a k e decisions

esteem measures (e.g., C r o c k e r & Luhtanen,

the IV. Often

1 9 9 0 ) . If participants w h o were given posi-

this requires not only extensive reading in a

tive feedback tend t o have higher scores on

about h o w to operationalize

In

relevant scales, then this suggests that the

piloting, researchers conduct small-scale stud-

manipulation has been successful and that

ies to examine the impact o f multiple plausi-

they have "gotten t o first-base."

given area but also extensive piloting.

ble manipulations and select the one that appears most suitable.

Researchers usually are keen to obtain very big effects on manipulation checks. If

Piloting may lead researchers to conclude

they do not, they generally will seek to

that in a given study, the best way to manipu-

strengthen the manipulation o f the IV. This

late someone's m o o d is for the researcher to

might involve increasing its duration or inten-

play happy or sad music to that

sity, reinforcing it with additional manipula-

person.

Similarly, it may lead the researcher to manip-

tions, or simply using a different (superior)

ulate collective self-esteem by telling some

manipulation. Such steps often are necessary

research participants that their group has per-

because for experiments to have the best

formed poorly on a test and other participants

chance o f succeeding (i.e., for the IV to have

that their group has performed well.

an effect on the D V ) , the researcher needs to

Experimental

ensure that the manipulation

of the IV is as

Design and Causality

discrimination experiment, the

\

researchers

strong as possible. Indeed, if there were a first

will not be satisfied with the knowledge that

rule o f experimentation, this might be it.

they have changed people's collective selfesteem; instead, they want t o k n o w whether this change in self-esteem has had any effect

Between-

and Within-Subjects

Designs

on discrimination. T o do this, the experiment

In most o f the above examples, we have referred to between-subjects

designs in which

one group o f participants is exposed to one

obviously has t o include D V s — i n this case, measures o f discrimination—on which such effects might emerge.

level o f a given I V and another group is exposed t o another level (e.g., where

As with IVs, any D V that is selected needs

the

to be relevant to die theoretical argument that

study contains two experimental groups or

the researcher is examining. For example, in a

an experimental group and a control group).

study seeking to test social identity theory's

However, the manipulation o f IVs does

assertion that intergroup discrimination can

not have t o involve different groups o f partic-

increase self-esteem, it is important that the

ipants. It is also possible (and c o m m o n ) to

D V is a measure o f collective

conduct experiments using

based) self-esteem (e.g., as measured by agree-

within-subjects

(i.e., group-

designs in which the same participants are

ment with the statement " I feel good about my

exposed to different levels o f the I V (see West,

group") (Crocker & Luhtanen, 1 9 9 0 ) rather

Biesanz, & K w o k , Chapter 1 3 , this volume).

than personal self-esteem (e.g., as measured by

As we have seen, this approach typically is

agreement with the statement " I feel that I

adopted in illusory correlation research where

have a number o f good qualities") (Rosenberg,

the same participants make judgments o f

1 9 6 5 ) . This is because the theory makes it clear

both large and small stimulus groups.

that when people are acting as group members,

Several factors contribute to a researcher's decision t o manipulate

an I V within

or

their actions will affect their sense o f self defined at a collective rather than a personal

between subjects. T w o o f the main advan-

level (i.e., in terms o f social identity rather than

tages o f within-subjects manipulations

personal identity) (Turner, 1 9 9 9 , p. 2 4 ) .

are

(a) that they dramatically reduce the number of participants needed in any given experiScale

ment and (b) that they remove all the differences between the participants in different conditions that are produced by chance as a

Construction

Although many experiments

incorporate

only one or two D V s , it is c o m m o n for scores

result o f the random assignment o f different

on

participants to different conditions.

responses on multiple measures (see Wegener

these

variables t o

be composites o f

However, despite their apparent elegance,

8c Fabrigar, Chapter 7, this volume). By taking

within-subjects procedures also have draw-

multiple measures o f the same theoretical con-

backs. In particular, as we will see below,

struct (e.g., self-esteem, prejudice), researchers

they potentially introduce some quite serious

can be more confident in their characterization

threats to internal validity.

of participants with respect to that construct. For

Dependent Variables (DVs)

example, if you wanted

to find

out

whether potential employees were reliable, it is likely that you would want to examine a

O f course, the main "prize" in any experi-

number o f aspects of their behavior: D o they

ment is not to obtain effects on manipulation

turn up to work on time? Are they often absent

checks but to obtain them on other D V s . In a

for no reason? D o they meet deadlines?

245

246

DESIGN AND ANALYSIS As part o f this process, researchers again often conduct pilot studies in order to ensure that responses on multiple measures can be organized to form

and

reliable

less sensitive to changes in the IV. As

an

e x a m p l e , consider

a team

of

researchers that is interested in studying the

F o r example, in

effect o f group interaction on prejudice (e.g.,

developing a measure o f w o r k reliability, a

Haslam & Wilson, 2 0 0 0 ) . T o understand

researcher may find that arrival and depar-

why racially motivated

ture time measure "time-keeping," while an

likely to be committed by groups rather than

scales

and/or

coherent

relevant to a given issue, it often tends to be

ability

to

subscales.

remember

meet deadlines

appointments

measure

and

"dependability."

Accordingly, in their research, they measure

reliability using

one quantifying

two

time-keeping,

attacks are

more

by individuals, the researchers might hypothesize that

interaction among members o f

may

groups that are prejudiced has the effect o f

subscales,

polarizing and consensualizing their prejudice

the

other

dependability.

(see Haslam, Turner, O a k e s , M c G a r t y , & Reynolds,

Scale coherence usually is established in

1 9 9 8 ; Moscovici &

Zavalloni,

1 9 6 9 ) . In this way, interaction is expected t o

two phases. T h e first checks that responses

strengthen the prejudice o f already-prejudiced

on individual items are correlated, typically

groups

performed

exploratory

members. T o test this hypothesis, researchers

factor analysis to verify the existence o f a

might conduct a study in which they manipu-

by conducting

an

and

the

agreement

among

its

unitary dimension. This is then followed by

late the I V (interaction) by asking people to

computation o f a reliability coefficient such

perform a given task on their own or in a

as Cronbach's alpha. If these checks prove

group o f like-minded other people.

successful, responses on individual items are

The most dramatic demonstration would

then aggregated to create a single scale (see

then show that people in groups are much

Wegener & Fabrigar, Chapter 7, this volume

more

for more details about scale construction). In

extreme forms o f prejudice toward out-group

this way, Rosenberg's ( 1 9 6 5 ) scale comprises

members—for example, by subjecting them

10 items that are combined to form a single

to racial abuse and perhaps even physical vio-

measure of personal self-esteem, and Crocker

lence. Again, we can see that there are serious

and Luhtanen's ( 1 9 9 0 ) instrument comprises

ethical problems with this research. But even

likely t h a n

individuals

to

display

four 4-item subscales, each intended to mea-

if these didn't exist, it would be unlikely that

sure slightly different components o f collective

any experimental manipulation

o f the TV

self-esteem (referred to as private, public,

(interaction) would ever be strong enough to

membership, and identity).

bring

about

sizable change

in

the

DV

(number o f racial attacks) unless the study was going to include an awful lot of very The Relevance-Sensitivity

Trade-Off

intense interaction. So the D V isn't sensitive

As with TVs, the operationalization o f

enough to detect the TV's impact. Indeed,

D V s can involve a trade-off between theoret-

were even a small effect to emerge on the

ical relevance and practical considerations. In

dependent

the choice o f D V s , there is sometimes a con-

impressive indeed (Prentice & Miller, 1 9 9 2 ) .

flict between what experimenters would like to measure

and what they can measure.

conflict contributes to a

measure,

this w o u l d

be

very

T o deal with this problem, the researchers

This

would have to design a study with a much

relevance-sensitivity

more sensitive D V . They could see whether

& M c G a r t y , 1 9 9 8 ) , which

putting prejudiced people into a group and

reflects the fact that when a D V is highly

making them interact makes them more likely

trade-off(Haslam

Experimental

Design and Causality

\

to endorse racist statements or to express

T h e main strategy for establishing this link

racist attitudes. Even here, though, the scales

is to back up experimental research that pro-

may have to examine relatively subtle rather

duces effects on a given D V with additional

than more blatant

(i.e.,

research demonstrating a causal (not just a

"modern racism" rather than "old-fashioned

correlational) relationship between that vari-

racism")

Batts,

able and one o f greater relevance. In the prej-

1 9 8 1 ; for a discussion, see W a l k e r , 2 0 0 1 ) .

udice example, having found that interaction

forms o f racism

( M c C o n a h a y , Hardee,

&

But although the D V might n o w be sensitive

affects prejudicial attitudes, researchers could

enough t o detect changes in the I V , its link to

then

the theoretical variable o f interest is less direct

address the question o f whether prejudicial

and much more open to question. Clearly,

attitudes affect prejudicial acts. Indeed, ide-

conduct

additional

experiments

to

people may argue that there is only a weak

ally they (or someone else) would have estab-

relationship

lished the link between attitudes and behavior

between prejudicial

attitudes

and prejudicial behavior (e.g., Fishbein, 1 9 9 7 ;

before they conducted the experiment into the

LaPiere, 1 9 3 4 ) . In this case, it could be argued

effects o f interaction on attitudes. However, it

that the researchers have simply replaced one

remains the case that such links are often

research question with another.

taken for granted

The way in which researchers usually seek

rather than established

systematically.

to deal with the tension between relevance and sensitivity in their choice o f D V s is to select variables on which they can obtain effects (i.e.,

The Experimental

Sample

variables that are sensitive enough) and seek

As well as deciding what they are going to

to ensure that these are as relevant as possible.

manipulate and measure in an experiment,

However, in some cases this strategy opens

researchers must decide w h o the participants

their work up to a charge o f

will be (see also Miller, Chapter 5, this vol-

nonrelevance—

and these charges are commonly made against

ume; Shoda, Chapter 6, this volume). In many

experimentalists researching in all areas o f

cases, the choice o f a sample is unproblematic.

psychology. T h e charge is particularly likely to

For example, if researchers are interested in

be seen as justified if researchers frame their

looking at the impact o f organizational per-

work in terms o f variables that they never

formance on managers' ratings o f a person's

actually investigate. In the above example, the

leadership, they will almost certainly want to

researchers might make claims about

the

conduct their research on managers (e.g.,

importance o f their work for helping reduce

Meindl, 1 9 9 3 ) . Similarly, if researchers are

racial attacks (and obtain research funding on

investigating the impact o f a given teaching

this basis) but never actually establish the link

method on schoolchildren's attitudes toward

between their D V and this phenomenon.

members of various ethnic groups, then they

Just h o w big a problem this lack o f rele-

will want to include schoolchildren in their

vance is for experimental research has been a

study (e.g., Aronson, Blaney, Stephin, Sikes, &

matter o f debate for some time (dating back

Snapp, 1 9 7 8 ) .

to the so-called "crisis in social psychology"

But what if researchers are interested in

in the 1 9 7 0 s ) (e.g., see Israel & Tajfel, 1 9 7 2 ;

examining normal psychological processes

M c G a r t y & Haslam, 1 9 9 7 ) . Nevertheless,

in

most researchers would

W h o should they study then? If research

agree that it is

the

general

("normal")

population?

important to try to establish the relevance o f

practice is a guide, the most c o m m o n answer

an experiment's dependent measures to any

to this question is "students." Indeed, inves-

phenomenon in which they are interested.

tigations indicate that in about

8 0 % of

247

248

DESIGN AND ANALYSIS published research articles, the participants

encourage a researcher to conduct experiments

in social psychology experiments are first-

using other

specialized samples

wherever

year undergraduates (e.g., West, N e w s o m , &

possible

Fenaughty, 1 9 9 2 ) . In these cases, the choice

Unfortunately, though, samples o f certain

of participants seems t o have been pretty

types o f participants (e.g., C E O s , members o f

straightforward.

racist organizations) are often hard t o find

However, it seems appropriate to interro-

and

(e.g., Tajfel

often

unwilling

&

to

Fraser,

1978).

participate

in

gate experimentalists' reliance on first-year

research. M o r e o v e r , even if they consent to

students—if only t o answer the concerns that

participate, there may be particular problems

those (and other) students themselves often

associated with studying them (e.g., ethical

voice a b o u t

and practical).

the

apparent

limitations o f

this strategy (Chow, 1 9 9 5 ) . " H o w , "

they

M o r e important, though, experimentalists

ask, "can one hope to m a k e general state-

generally argue that it is possible to gain

ments about psychological processes among

significant insights into human nature

members o f society at large on the basis o f

the basis o f samples and situations that are

on

studies which focus on a class o f people that

not representative o f the breadth o f human

represents

society and experience (e.g., C h o w , 1 9 9 5 ;

only

a small subset

of

that

society—and one that, demographically, is

M o o k , 1 9 8 0 ; Turner, 1 9 8 1 ) . T h e arguments

not particularly representative o f the whole

marshaled t o support this point are quite

(tending to be wealthier, more

educated,

complex, but they rest on two main points:

more motivated, e t c . ) ? " T h i s question (and

representativeness

the criticism it implies) appears more c o m -

theory

pelling

volume; Shoda, Chapter 6 , this volume).

when

students

learn

that

the

and

the importance o f

(see also Miller, Chapter

5, this

researcher in question is interested in such things as attitudes, prejudice, group dynamics, and personality. T h e questions are asked

Representativeness

even more forcefully once one observes that

The first argument indefence of experimen-

student populations usually are studied in the

tal practice relates to the concept o f a repre­

laboratory rather than in the situations they

sentative sample. A core point here is that to be

encounter in their everyday lives (Vissers,

able to apply research findings to a particular

Heyne, Peters, &c Guerts, 2 0 0 1 ; see C o o k &

population

G r o o m , Chapter 2 , this volume). H o w can

experimental samples and situations do not

(e.g., most

" n o r m a l " people),

laboratory-based studies o f people w h o are

have to be representative o f that population in

generally intelligent, healthy, assertive, and

every respect. Instead, the experimental sample

tolerant tell us anything about the psycho-

only has to be representative o f that popula-

logical makeup o f the general

tion on theoretically relevant dimensions.

population,

whose actions take place in a world o f

If a researcher theorizes that group inter-

politics, poverty, oppression, struggle, and

action affects decision making by leading to

intolerance?

polarization (Moscovici & Zavalloni, 1 9 6 9 ) ,

M a n y non-experimental social psychol-

this means that the sample has to be repre-

ogists ask this same question, and go on to

sentative o f the population

argue that the conclusions inferred

from

research will be generalized only on dimen-

most experimental samples are necessarily

sions that relate to group membership, group

flawed. M a n y experimentalists also would

interaction,

argue that research should not be based

example, the participants need to be able t o

solely

on

student

samples. M o s t

would

and

group

t o which the

decisions.

For

be placed in groups, they need to be able to

Experimental

Design and Causality

\

interact, and they need to be able to make

measuring their responses on measures o f

decisions.

risk as defined which

The Importance

of

by the subpopulation

the sample

is drawn

from

(which may be

different for parents and for juvenile delin-

Theory

quents, for example). W e could then see if

A second point that is absolutely critical

these attitudes were more extreme after dis-

here, though, is that researchers must have a

cussion than they were before it. W e would

theoretical

not

then generalize our research to the larger

in order to m a k e

world not on the basis o f the experimental

(normally but

framework

always called a theory)

judgments about w h a t is (and w h a t is not) a

data itself (which would almost certainly

relevant

vary across samples, situations, and cultures)

dimension

o f the

experimental

sample and situation. N o t e t o o that this also

(Sohn, 1 9 9 2 ) , but on the basis of the

applies even if the sample is directly relevant

that it supports. At a later date, researchers

to the issue being studied. F o r example, in

could conduct further research that either

a

corroborated

study

o f children's

moral

reasoning,

researchers need to ensure that the children

or challenged that

theory

theory.

Indeed, this is exactly what happened in the

in their sample are representative o f the pop-

study o f group decision making in which

ulation t o which they wish to apply their

researchers initially thought that discussion

research, not just in terms o f age but also on

made all decisions more risky (Stoner, 1 9 6 1 )

dimensions o f intelligence and maturity. If

but went on to find that it had the effect o f

researchers do not attempt to ensure that

making them more polarized (i.e., more risky

their experimental subpopulation is represen-

if they were initially risky, m o r e cautious if

tative in this way and they attempt to gener-

they

alize any empirical findings

Turner,

they

obtain

straight to the social world at large, then, as

were

initially c a u t i o u s )

Hogg,

Davidson,

&

(McGarty, Wetherell,

1 9 9 2 ; M o s c o v i c i & Zavalloni, 1 9 6 9 ) .

we noted above, they are guilty o f naive empiricism. For

example, if researchers conduct

a

study with some students and find that they become m o r e left-wing after holding a group discussion

about

p o l i t i c s , they

may

be

Using Specialized

Samples

As we have seen, researchers often w a n t to conduct experiments using specialized sam-

tempted t o suggest that group discussion will

ples that are particularly relevant t o

make everybody more left-wing. However,

phenomena they are investigating. O n e rea-

the

this strategy is dangerous largely because it is

son for doing so is that it may be difficult to

merely a form o f induction—a

very weak

manipulate a given variable using a nonspe-

(though quite c o m m o n ) form o f scientific

cialized sample. In this case, the researcher

reasoning. T h i s important point is illustrated

may resort to quasi-experimental methodology in which participants are not randomly

schematically in Figure 1 1 . 1 . T h e o r y offers the scientific solution to this

assigned to conditions but are assigned on

problem because it allows experiments to test

the basis o f their preexisting membership o f

theoretical ideas using any sample and in any

a particular group (see M a r k & Reichardt,

situation where theoretically relevant vari-

Chapter 1 2 , this volume).

above

This strategy creates problems when seek-

example, we could test the theory that group

ing t o establish causality, however, because

discussion makes attitudes more risky by

participants in different experimental condi-

exposing any sample to group discussion and

tions are likely to differ on more than just the

ables can be manipulated.

In the

249

DESIGN AND ANALYSIS

250

Theory-Based Generalization

Empirical finding

1 Theory suggests empirical test

After holding a meeting a right-wing group makes more right-wing decisions

Generalization

2. Empirical finding supports theory

Theory Group interaction polarizes opinion

Group interaction polarizes opinion

T 3. Generalize on basis of theory

Naive Empiricism Empirical finding After holding a meeting a right-wing group makes more right-wing decisions

1. Generalize on basis of empirical finding

Theory

Generalization

Group interaction polarizes opinion

Groups make rightwing decisions

Figure 11.1

The Distinction Between Theory-Based Generalization and Naive Empiricism

I V o f interest. O n e way o f addressing these problems is by matching.

personality, and prejudice level. M a t c h i n g

Matching involves

basically involves researchers systematically

ensuring that different experimental groups

assigning participants to groups on the basis

do not differ appreciably on dimensions that

of differences in the variable o f interest (e.g.,

could be implicated in any effects obtained

personality, prejudice level) but seeking to

on

relevant

ensure that they do not differ systematically

M a t c h i n g is very c o m m o n in research

socioeconomic background). M a t c h i n g also

the

DV

(i.e.,

theoretically

variables).

on other key variables (e.g., age, sex, or

where it is almost impossible (as well as

can

unethical) to manipulate theoretically rele-

against which performance can be appraised.

be used to establish relevant

baselines

vant variables. Such variables include social,

For example, in a study to investigate work-

educational,

related

and

cultural

background;

stress, a researcher may

seek

to

Experimental

Design and Causality

\

obtain controls matched on variables such as

that an I V has had an effect than it is to

age, sex, and years o f education, in order

ascertain why it has had an effect.

stress o f a particular

Problems in establishing an experiment's

person or group in a given social setting

to gauge the relative

internal validity often occur when experi-

(e.g., shift workers) (Meers, M a a s s e n , &c

menters don't really manipulate w h a t they

Verhaagen, 1 9 9 4 ) . T h i s procedure allows the

think

researcher t o eliminate the variables

on

which the participants are matched as potential causes o f any differences in stress.

they're

manipulate

something

but

else.

actually

Earlier

mentioned the example o f Hamilton Gifford's

Although it sounds promising, matching is

manipulating

(1976)

illusory

we and

correlation

research, in which the researchers thought

never as effective as randomization in remov-

they were manipulating distinctiveness but in

ing differences between groups. This is partly

fact can be seen to have manipulated alpha-

because matching is very difficult t o do suc-

betical superiority, the number o f positive

cessfully,

statements about the groups, or the amount

especially if the

researcher

is

attempting t o match participants on more

of

than one or two variables.

about the groups.

support for differentiating

hypotheses

W h e r e alternative explanations o f experimental outcomes like this are identified, it is usually because the experiment involves an

EXPERIMENTAL ASSEMBLY

manipulation o f an extraneous

unintended

variable—in other words, a confounding. As

Threats to Internal Validity T h e previous two sections have outlined the purpose and ingredients o f experiments. Equally important are questions relating to the way experiments are put together and conducted—not least because it is very easy to construct bad experiments or to conduct experiments badly. In particular, a number of features o f design can compromise an experiment's

internal

researchers t o

interpret

validity,

leading

the effects

they

we

noted

a b o v e , this

is very

common

(indeed, in a sense, unavoidable) in experimental research, and many

disagreements

between researchers are based on differing views about the existence o f confounds and about

what

exactly

experimenters

have

manipulated in their experiments. Often, as we have seen, these disagreements are quite nuanced and hinge on subtle re-interpretation o f the status o f an IV.

obtain incorrectly. Uncertainty Dealing

With

Management

T h e general w a y in which issues o f inter-

Confounds

pretation are played out is through a process

As we noted above, internal validity is compromised when researchers think

that

of

uncertainty

(Haslam &

management

M c G a r t y , 2 0 0 1 , in press). In the first phase

their research findings show one thing but

o f this process, a particular

they in fact s h o w something else. In the case

result is established with a degree o f certainty

experimental

of experiments, this problem arises from the

and

fact that although particular findings allow

endorsed (e.g., in t e x t b o o k s ) . Subsequently,

researchers t o k n o w whether

though,

there is some-

thing to explain, they do not always allow

a particular

interpretation

is widely

other researchers increase uncer-

tainty by calling the received interpretation

researchers t o k n o w w h a t the explanation is.

into question. T h e field may then progress

This is because it is much easier t o ascertain

through attempts to resolve the uncertainty

251

252

DESIGN AND ANALYSIS that has arisen. T h i s cycle is one that has been played out in relation to most major topics in social psychology, and it is fair to say that most social psychologists establish their professional credentials by contributing to one or more o f these three phases o f uncertainty management. This typically is done in one o f two ways. T h e first is simply to conduct an additional study in which alternative interpretations are competitively tested—typically through independent manipulation o f multiple relevant variables. T h i s is h o w

researchers (e.g.,

Berndsen, Spears, van der Pligt, & M c G a r t y , 1 9 9 9 ; Haslam, M c G a r t y , & B r o w n , 1 9 9 6 ; McGarty

&

de

la

Haye,

1 9 9 7 ) have

addressed alternative interpretations o f the findings o f Hamilton and Gifford's ( 1 9 7 6 ) illusory correlation research that we discussed earlier.

Effects

O n e o f the main causes o f interpretational problems in experiments is the passage of time. This is a serious problem in experiments that have within-subjects designs in which participants are required to make a number o f responses. Here the basic principle o f experimental design—namely,

that

researchers need t o take steps to ensure that the only difference between groups o f participants whose responses are compared is in the variable that has been manipulated—is compromised by the fact that between the two observations, other things apart

from

will have occurred. Here

the manipulation

both the participants and their circumstances will have changed—and these changes, not the treatment,

may be responsible for the

change between the two observations (the pretest and the posttest).

Another way o f dealing with this problem is t o ensure in advance

that experiments

include manipulation checks. As we noted above, these can help establish whether indirect manipulations desired effect.

Maturation

o f I V s have had

Furthermore, if data

the from

manipulation checks are available, modern regression techniques allow researchers t o conduct path analyses

to test whether the

Any such temporal changes are generally referred to as instances o f maturation

effects.

T w o forms o f maturation effect are particularly important: practice

effects and fatigue

effects. B o t h relate t o the fact that by the time they complete a particular D V , participants already have had some experience o f completing it before.

effect o f an experimental manipulation can be ascribed t o the putative I V (see B a r o n & Kenny, 1 9 8 6 ; Hoyle & Robinson, Chapter 1 0 ,

History

this volume, for more details).

T i m e also exerts indirect effects on partic-

Effects

ipants because it is associated with changes

Ruling Out Alternative Hypotheses: Dealing With Specific Threats to Internal Validity Beyond this general point about problems

occurring in the world that they inhabit. Between any t w o observations, participants may be affected by any number o f things going on

in the

world—change

in

the

weather, the e c o n o m y , the political climate,

Stanley

and so on. Any such changes that take place

( 1 9 6 3 ) made a major contribution to the

over the course o f a study are referred to as

analysis of internal validity by cataloging

history

of

interpretation,

C a m p b e l l and

specific ways in which it could be threatened. In the remainder o f this section, we will

effects.

Maturation

effects

and

history

effects

threaten internal validity because they can

briefly discuss each o f these in turn, as well as

lead to systematic

the main ways to deal with them.

vations at different phases o f a study. T h i s is

differences

between

obser­

Experimental one reason why

participants

exposed

to

different levels o f a within-subjects variable

Design and Causality

\

collected by, where it is collected, and the methods used for collection.

are not perfectly matched. Sample Experiment

Effects

A final set o f threats to internal validity

O t h e r factors relating to the situation are

experimental

itself may mean that participants

systematically

Effects

different

at

different

relate t o strategies for selecting participant samples and the effects o f those strategies. T h e first o f these are selection

effects,

and

phases o f an experiment. First, when partici-

along lines suggested above, these arise when

pants are observed or asked t o respond for

participants are assigned t o different levels o f

a second time

(i.e., in a posttest), their

an I V o n the basis o f some specific criteria

responses may be affected by their awareness

(e.g., their age or their score on a particular

o f having responded previously (in a pretest).

test). As we have noted several times already,

Problems like this are k n o w n

the basic problem posed by selection effects

as

testing

effects. Testing effects are actually one aspect

is that because participants are not randomly

of the more general issue o f experimental

assigned t o conditions, any change in the D V

reactivity. T h i s refers to the fact that the pro-

may be due t o things associated

cess o f making

rather than the I V itself.

observations.

observations

can

change

I f reactivity affects partici-

with the I V

A m o r e subtle problem arises when an I V

pants' responses in experiments, it can cause

is manipulated within subjects and partici-

serious problems o f interpretation.

pants are selected because o f their extremity

W e will discuss t w o specific sources o f reactivity—experimenter bias and

demand

(or peculiarity) on the variable o f interest. T h e problem here is that due t o regression

to

one is unlikely to observe such

characteristics—in more detail below. This is

the mean,

because, unlike the other threats to internal

extremity again (for a further treatment o f

validity considered

this issue, see M a r k & Reichardt, Chapter 1 2 ,

in this section, these

problems call for special forms o f experi-

this volume).

mental control (e.g., using deception) that

A final threat posed by sampling strategies

are similar to those used to deal with threats

is the

to external validity. F o r this reason, even

effects. In the time between any two observa-

though they represents threats t o both inter-

tions, particular

nal and external validity, we will deal with

from a study. W h e r e this occurs, it may be

these problems in our discussion o f threats to

the characteristics o f the withdrawing partic-

external validity.

problem

of participant

mortality

participants m a y withdraw

ipants that are responsible for any effects

O n e basic form in which reactivity c o m -

rather than the experimental manipulation.

order

This is a particular problem in experiments

effects. These arise if the responses to treat-

that are conducted " o n l i n e " (see Birnbaum,

ments change depending

Chapter 1 6 , this volume).

m o n l y manifests

itself

is through

on the order

in

which the treatments occur. Other features

T h e basic problem posed by all these

of the experiment may also change between

threats to internal validity is the same—that

any t w o observations, and these

is,

instrumen­

they

produce

systematic

differences

tation effects undermine validity in much the

between experimental observations beyond

same way. Instrumentation refers to any fea-

those caused by manipulation o f the IV. As

ture o f the experiment related to the collec-

we have seen, these effects are often subtle

tion

and, indeed, quite intriguing. Faced with a

o f data—for

example, whom

it is

253

254

DESIGN AND ANALYSIS list o f potential experimental problems, you

manipulation o f the I V rather than from any

may also wonder why researchers ever go

of the threats to internal validity described

to the trouble o f trying t o conduct valid

above. As an example, consider researchers w h o

experiments.

are interested in establishing the ability o f a particular social contact program to reduce Experimental

Control

prejudice. T o do this, they could randomly

T h e good news is that despite the subtlety

assign participants t o experimental and con-

of many o f the above problems, all can be

trol conditions and start by measuring the

addressed (often quite easily) by appropriate

prejudice o f each group o f participants (due

experimental design. Indeed, the basic princi-

t o random assignment, there should be no

ple used to eliminate threats to validity is one

substantial difference between them). T h e y

that we have talked about at length:

experi­

could then m a k e people in the experimental

mental control. As we have pointed out, the

group take part in the social contact program

main way o f achieving experimental control

but leave the control group alone (this con-

is through a combination o f randomization

stitutes the manipulation o f the I V ) . Later,

and/or matching. F o r example, an effective

they could measure the prejudice levels o f

way o f dealing with order effects, practice

both groups again. I f the posttest shows that

effects, and fatigue effects is either (a) to

prejudice is much lower in the experimental

randomize the order in which participants

group than in the control group (and there

complete dependent measures, (b) t o m a k e

had been no difference between the groups

participants in both an experimental and a

before), then the researchers can correctly

control group complete all dependent mea-

attribute this difference to the positive effects

sures in the same order, or (c) t o systemati-

o f the program. Similarly, if prejudice is n o w

cally vary the order in which items are

clearly greater in the experimental group

presented, through

than in the control group, they would have

counterbalancing.

These points about experimental control can be demonstrated by describing one o f the

t o conclude that this form o f social contact is having a harmful effect.

main forms o f w h a t Campbell and Stanley ( 1 9 6 3 ) call a true experimental

design.

This

design deals effectively with all the above

Threats to External Validity

threats (see also M a r k & Reichardt, Chapter 12,

In the previous section, we discussed h o w

this volume). It involves supplementing a

the addition o f a between-subjects manipula-

basic pretest-treatment-posttest

design with

tion to a within-subjects experiment resulted

an additional pretest-posttest

control. T h i s

in an almost perfect basic experimental

control condition uses different participants

design. T h e r e is only one real problem with

but a pretest and posttest that are identical

this strategy, and this is that the effects o f

in every

respect

measurement

(e.g., including the same

procedures

and

the

same

experimenters).

testing across the t w o

observations (the

pretest and posttest) may interact

with the

experimental manipulation, so that partici-

If participants are randomly assigned t o

pants w h o are exposed t o a particular level o f

these two conditions, we can compare the

a treatment react to the first phase o f testing

results obtained at the posttest in the experi-

differently from those w h o are exposed to

mental condition with those obtained at the

another level (e.g., in a control condition

posttest in the control condition. Here a

where they receive n o treatment). T h i s reac-

significant difference must result from the

tivity does not undermine the experiment's

Experimental

Design and Causality

\

internal validity—because the effect is still

validity. T h e s e include (a) the reactivity o f

due t o the experimental

manipulation—but

participants both t o features o f experimental

it creates interpretative problems neverthe-

design (as discussed above) and t o the experi-

less. T h i s is because the results o f the study

menter's hypothesis and (b) the artificiality o f

m a y generalize only to situations

experimental settings ( C h o w , 1 9 9 5 ) .

where

people have been given the pretest. In other words, the pretest-posttest design can c o m promise external

Reactivity

validity.

For example, in our study looking at the

Research by Rosenthal (e.g., 1 9 6 6 ) sug-

impact o f social contact on prejudice, it m a y

gested that even if an experiment is believed

be the case that after participants have par-

to be testing a hypothesis in an evenhanded

ticipated in a contact program, they reinter-

way, researchers may lead participants

pret the meaning o f the items in prejudice

behave in the way that they want.

scales, so that they display less prejudice

example, this might occur because partici-

when

responding

on those scales at

the

pants

are sensitive to the

to For

experimenter's

posttest. T h e results o f this research would

power (Spears &c Smith, 2 0 0 1 ) and respond

thus tell us only about the effectiveness o f

to cues in the experiment that implicitly tell

contact as a means o f changing the interpre-

them what is expected o f them. A question

tation o f prejudice scales; they would not

like " D o you really w a n t to do t h a t ? " may

necessarily tell us anything about the effec-

lead participants t o assume that the correct

tiveness o f contact as a means o f reducing

answer is " N o " and to respond accordingly.

prejudice.

These cues are known as the experiment's

T o address the problem, a simple modifi-

demand

characteristics

(Orne, 1 9 6 2 ) ,

and

cation to the above design involves eliminat-

they can be thought o f as norms that partici-

ing the first observation, so that there is n o

pants conform to (or, on occasion, reject)

to

(Barber & Silver, 1 9 6 8 ) because they appear

react with. Indeed, this design is the basic

to be guides to behavior in the situation they

building

confront.

pretest for the experimental treatment block

of

most

experimental

research—largely because it is simple, neat, and effective.

Behavior in the form o f experimenter

bias

can have a similar effect. This refers to any

However, experimental pretests are not

behavior by the researcher that prevents a fair

the only things that threaten an experiment's

test o f the experimental hypothesis.

external validity. Earlier, we made the point

experimenter may react more positively to

that in order t o generalize from an experi-

responses that are to his or her liking (for

mental sample to another population,

the

example, by nodding or smiling). Less subtly,

sample must be representative o f the popula-

the researcher may " p r o d " participants so

tion on theoretically relevant

that they behave in the desired way in order

dimensions.

The

A related question concerns the process by

to

which researchers seek t o generalize from

Obviously, such cheating

experimental situations

worth o f any research, and the problems it

to situations beyond

the experiment. This is another c o m p l e x issue and one on

support the

experimental

hypothesis.

compromises the

creates will never be remedied by improvements to experimental design. In contrast, the

which a multitude o f opinions exists. Broadly

issue o f unintentional

speaking, debate centers on a number o f fea-

can be attended to quite easily—for example,

experimenter bias often

tures o f experimental design that are seen to

by using an experimenter w h o is blind to the

have the potential t o compromise external

condition in which participants are placed.

255

256

DESIGN AND ANALYSIS o f an

agree that experiments simplify the situations

experiment's hypothesis can jeopardize both

Because participants'

awareness

that confront people in their daily lives.

external and internal validity, social psychol-

Indeed, the experimental control that pro-

ogists often go to considerable trouble to

duces

conceal the purpose o f any experiment they

research its e x p l a n a t o r y

conduct. T h i s can be done by means o f

1 9 9 6 ; M o o k , 1 9 8 0 ) . But we argue

deception—either

misinforming participants

about the purpose o f an experiment or using concealment

t o avoid telling them the whole

truth. T h i s may involve presenting partici-

this simplicity gives

experimental

power

(Dawes, that

whether or not, in any given case, this practice

constitutes

empirical and

oversimplification

is

theoretical issue. In

an

other

words, it is something that can be established

pants with a plausible (but misleading) cover

only by considering each case on its merits

story that informs them that an experiment is

and doing additional research if necessary.

about one thing when really it is about another.

In

this

chapter,

we

have

typically

described simple experiments involving only

For the straightforward reason that these

one I V , but in fact, as suggested above, few

procedures involve a failure to be entirely

experiments examine variables in isolation.

truthful,

T h e y usually explore c o m p l e x interactions

their use raises obvious ethical

issues. M o r e o v e r , because the procedures are

between multiple

a c o m m o n feature o f experiments in social

tional features. I f a theory or explanatory

psychological

and

situa­

psychology, researchers in the field have a

principle is challenged by the findings that

particular need to address these issues. In

such research produces,

practical terms, it means that social psycho-

reject or modify that theory or principle

logical experiments are far more likely t o

accordingly. This should lead to a better

require systematic debriefing

of participants

than those in other fields—to explain the nature of, and reasons for, deception

researchers

can

theory, on the basis o f which it is more appropriate to generalize.

or

concealment (see Kimmel, Chapter 3 , this volume).

INFERRING CAUSAL RELATIONSHIPS F R O M EXPERIMENTAL RESEARCH

Artificiality T h e biggest threat to external validity concerns the ability o f researchers to generalize on the basis o f experiments that study psy-

Lessons From the Social Psychology of Causal Inference Research

in

social

psychology

often

chology in controlled laboratory situations

involves the assumption that, as they go

(Chow, 1 9 9 5 ) . W h e n researchers conduct

about their daily lives, people act like intu-

experiments, they try to manipulate theoreti-

itive or naive scientists (e.g., Kelley, 1 9 6 7 ) . In

cally relevant variables while holding

all

contrast, our present focus is on providing

others constant. But does this emphasis on

guidance for actual scientists. As it happens,

control deprive those variables o f their mean-

though, some o f the mistakes that can be

ing and richness (Gergen, 1 9 9 7 ; Leaf, 1 9 9 3 ) ?

made

in drawing causal inference

from

T h e arguments that experimentalists use

research are precisely those that confront

t o defend their practices are similar to those

decision makers and reasoners in everyday

that we presented in discussing the logic o f

life. It is therefore possible to " m i n e " the psy-

representative

chological literature on causal inference to

sampling. M o s t

researchers

Experimental

Design and Causality

\

identify problems that people run into as

present. W e must also be able t o distinguish

practicing researchers.

each variable from other variables. This is why we have spent so much space addressing

Detecting

issues o f extraneous variables, confounds,

Covariation

and interaction effects.

In both cognitive and social psychology,

T h e third qualification is that we need to

the primary focus in the literature on infer-

be sure that the causal relationship actually

ence has been on the detection o f covariation

holds. As we have already seen, we can quan-

(Cheng, 1 9 9 7 ; Kelley, 1 9 6 7 ; van Overwalle,

tify our confidence through the use o f appro-

1 9 9 8 ; see M c G a r t y , 1 9 9 9 , for a review). T h i s

priate statistics and reduce uncertainty about

is because one thing c a n n o t be the cause o f

relationships by replicating our experimental

another unless it covaries with it. Indeed,

procedures.

although correlation does n o t imply causation, causation does imply reasonably strong correlations (except in the case o f statistical suppression; see below). T h i s simple idea lies at the heart o f path analysis, structural equa-

Beyond We

Covariation

noted

that

the primary

focus o f

research on causal inference is on covariation

tion modeling, statistical mediation, and the

detection, but is it enough

use o f latent variables in factor analysis, and

covary?

we can use this same idea in considering the

n o t imply causation, a mere statement about

role o f cause in social psychological experi-

covariation is unlikely to provide a wholly

to say that factors

Precisely because correlation does

ments. J u s t as an inability to detect covaria-

satisfactory

tion accurately will have some consequence

M e d i n , and Gelman ( 1 9 9 5 ) have shown this

explanation.

Ahn,

Kalish,

for our understanding o f events in our every-

in the causal inference literature.

day lives, so the inability to detect covaria-

people are offered explanations that fail to

tion

undermines

go beyond covariation, they usually are left

It is possible for research t o qualify this

ered simple interpersonal events o f the form

link between causation and correlation in

" H a r r y hit J a c k " and were then given the

accurately

in

research

scientific understanding.

When

dissatisfied. Ahn et al.'s participants consid-

three important ways. First, a cause must

opportunity t o request further

occur before its consequences—the assump-

T h e y generally requested information o f the

tion o f causal order. O n e o f the attractions o f

form "Is Harry aggressive?" or " D o e s Harry

the experimental method is that it can allow

dislike J a c k ? "

rather

than

information.

covariational

us to guarantee the order o f our variables:

information o f the form " D o e s Harry hit

Manipulated IVs are never caused by D V s

other p e o p l e ? " or

(although this is not to say that the measure-

before?"

ment o f our D V s never changes the meaning that our experimental manipulations

have

for our participants). Second, we must be able to identify the

"Has

Harry

hit

Jack

This suggests that people want explanations that go beyond covariation and that point to underlying factors or processes. T h e producers and consumers o f social psycho-

presence o f the covarying factors. W e c a n n o t

logical and other research have the same

plausibly claim that t w o variables covary

preference for genuine explanation

unless we can identify precisely if and when

than mere description. M o r e o v e r , the charge

each o f the variables occurs. Indeed, this is

that the demonstration

why experiments use manipulation c h e c k s —

ship is "merely descriptive" (or redescriptive)

to tell us that a putative causal variable is

is o n e

that

can

apply

rather

o f some relationequally well

to

257

258

DESIGN AND ANALYSIS experimental and other methods. W h e n we

is important to m a k e a number o f basic

show the effect o f an I V on a D V , we may be

points here.

confident that the former has had an effect but still be left wondering about the process that produced the effect. W e can k n o w what

The Logic of

happened but not how and

T h e idea that causation implies correla-

why.

As an example o f the difficulty o f disen-

Mediation

tion lies at the center o f efforts to demon-

tangling explanation and description, we can

strate mediation. Essentially, if some third

return t o our

variable—the mediator,

earlier study

in which

a

mediating

variable,

researcher observes a relationship between

or MV—captures

personality and prejudice and goes on t o

an I V to a D V , there should be some overlap

the mechanism that links

hypothesize that individuals display preju-

between this M V and the I V in the prediction

dice

of the D V . F o r this reason, when the M V is

toward

members

of

other

groups

because those individuals have a dysfunc-

entered

into a regression equation,

there

tional personality. In this case, to demon-

should be a reduction in the strength o f the

strate

relationship between the I V and D V as well

that

their

personality

led

those

individuals to be prejudiced is no easy thing.

as evidence o f a clear relationship between

In particular, it cannot be established merely

the M V and the D V (Baron & Kenny, 1 9 8 6 ) .

by demonstrating that they were prejudiced

Experiments in social psychological n o w

and that their personality is o f a certain type.

c o m m o n l y e x a m i n e mediational

Indeed, it could not even be demonstrated by

ships by entering some hypothetical measure

relation-

showing that all people with a certain per-

of the mechanism into a regression equation.

sonality are prejudiced and that all people

As testament to this, B a r o n and Kenny's

with a different personality are unprejudiced

( 1 9 8 6 ) article on mediation has been cited

(i.e., that personality type is completely cor-

more than 3 , 0 0 0 times in the last 1 0 years,

related with hostility). In this example, our

and a high proportion o f these citations are

data may reflect a plethora o f other causal

in social psychology.

relations, including ones in which the causal sequence is reversed

(e.g., the possibility that

prejudiced people develop particular personalities) or those in which an association is produced by a third factor or tertiam

quid

Problems Although B a r o n and

of the

Interpretation conditions

Kenny ( 1 9 8 6 )

specified are

by

consistent

(e.g., the possibility that personality and prej-

with the existence o f mediation, there are a

udice are both products o f cultural back-

number

ground or social norms).

attempts to interpret the results o f media-

of

problems

associated

with

tional analysis. First, the conditions are also consistent with a range o f conditions where

Mediational Analysis

mediation does not hold. I f the I V , M V , and questions

D V are all affected by some fourth variable,

questions partly

then relationships that are spuriously consis-

T h e desire to answer " w h y " rather than only " w h a t "

explains why theory plays such an important

tent with mediation may be demonstrated.

role in science. It also explains why the idea

This possibility normally can be discounted

of statistical mediation,

a set o f methods for

by experimental manipulation o f the I V , so

identifying causal mechanisms, enjoys such

we will not discuss it further here. However,

popularity in the field. T h e topic is covered

if the M V is merely a good measure o f the I V

(see Hoyle &

(akin to a manipulation check), rather than a

R o b i n s o n , Chapter 1 0 , this volume), but it

measure o f a process created by the I V , then

elsewhere in this h a n d b o o k

Experimental

|

Design and Causality

there is a good chance that something that

reality into science fiction? Are laboratory

looks like mediation (but is actually

experiments

pseudo-

will be demonstrated.

mediation)

M o r e alarmingly, there are

distorted

caricatures o f the

external world that can offer only pallid and conditions

under which mediation should hold but the

trivial understanding o f the rich phenomena they seek t o investigate?

conditions specified by B a r o n and Kenny

In responding

to such questions, it is

( 1 9 8 6 ) will not reveal it. In particular, if

worth

some other variable suppresses the relation-

research to suggest that psychological pro-

ship between the I V and M V , then there will

cesses and the impact o f psychological states

be no evidence o f mediation unless this other

necessarily differ between experiments and

variable is measured in the study and entered

the so-called real world. T h i s is for

into the regression equation. Imagine, for

simple reason that experimental settings—

noting

that

there

is little or

no

the

example, that low self-esteem (IV) is hypoth-

even those o f the laboratory—are still a part

esized to create frustration ( M V ) , which in

of

turn creates aggression ( D V ) . If participants

experiments therefore can be reproduced in

that

real world.

Effects

produced

in

vary greatly on some variable that acts as a

the real world, and real world phenomena

suppressor

can

variable in that c o n t e x t (e.g.,

be r e p r o d u c e d

in

experiments.

As

power or emotional expressivity), then any

T u r n e r ( 1 9 8 1 ) put it, there is n o Great W a l l

relation

o f China between the two.

between

the

variables

may

be

masked in that setting. F o r example, differ-

Indeed, it is possible to turn arguments

ences in power might mask the relationship

about the irrelevance o f psychological experi-

between the I V and D V , and differences in

ments on their head. In most sciences, exper-

emotional expressivity m a y mask relation-

iments are used t o create ideal circumstances

ships that involve the M V .

that would not otherwise occur in real life. In

These observations suggest that

media-

psychology t o o , experiments are useful pre­ they control for the influence

tional techniques can be useful tools with

cisely because

which to augment one's analysis o f designed

o f extraneous variables that cloud the analy-

experiments. However, contrary to popular

sis o f everyday behavior.

opinion, they are not an elixir, capable o f

L o o k e d at in this way, where there is a

magically revealing latent process variables.

problem with generalizing from experimen-

They do not allow definitive statements about

tal research to the real world, it will often

causal relationships, and—to their credit—the

arise not from the fact that the research is

experts w h o have popularized these methods

invalid but from the fact that it is

have never claimed otherwise (for cautionary

O n e o f the main problems with experimental

notes, see also Sigall & Mills, 1 9 9 8 ) .

research is thus that researchers may be led

irrelevant.

to examine the impact o f particular variables CONCLUSION

because

they can, rather than because they

should.

F o r example, although social psy-

chology is frequently defined as the science o f Elaborating on the arguments in previous

social interaction, the proportion o f social

sections, we conclude this chapter by reflect-

psychological studies that actually involve

ing on the question o f whether or not the pri-

social interaction has declined dramatically

mary achievement o f the

over the last 3 0 years (from about 6 0 % o f

experiment—the

ability to m a k e causal statements on the basis

articles in the Journal

of control—is its ultimate downfall. Is it the

Social Psychology

case, as critics sometimes argue, that

1 9 9 8 ) (Haslam & M c G a r t y , 2 0 0 1 ; see also

an

obsession with control o f IVs turns social

of Personality

and

in 1 9 6 8 to around 2 0 % in

Haslam & Reicher, 2 0 0 2 ) .

259

260

DESIGN AND ANALYSIS Presumably, this is because studies

that

involve social interaction are harder to conduct, to analyze, and to publish (e.g., see Gonzalez & Griffin, Chapter 1 4 , this volume). But if topics are studied only because they are easy to investigate or because their findings are easy to publish, it will often be the research questions that are at fault and not the conclusions. Along these lines, in a recent review o f the

experiments is always constrained by the quality and utility of the ideas and theories they test. A trivial idea makes for a trivial experiment. But experiments can test powerful ideas and they can also be tools of revolutionary science. The more willing we are to put them to this use the more likely it is that we will rediscover and share in a science that is dynamic and challenging. (Haslam & McGarty, 2 0 0 1 , p. 18)

use o f the experimental method in 1 0 0 years of social psychological research, we followed Tajfel ( 1 9 7 2 ) in concluding that:

T o the extent that social psychology falls short o f its scientific goals, it is therefore not our experimental tools we should blame. By

It is not the case that experiments per se are bad or good, trivial or non-trivial. As with any other method, the quality and utility of

the same token, these tools should not, on their own,

be seen as sufficient for

discipline's progress.

NOTE 1. Note that Sigall and Mills (1998) contested this claim and argued that there are many examples of experimental findings for which there is no plausible alternative explanation. They provide one example—research by Zajonc ( 1 9 6 8 ) — where there appears to be no alternative explanation for the effects of the conceptual variable (number of exposures to stimuli) on the DV (preference for the stimuli). We would point out, however, that the conceptual variable here actually is familiarity, not number of exposures, and it is easy to craft alternative explanations for this "mere exposure" effect. For example, one could claim that the number of exposures reduces the surprise value of the stimulus set rather than familiarity with any single stimulus.

REFERENCES Adorno, T. W., Frenkel-Brunswik, E., Levinson, D. J . , & Sanford, R. N. (1950) The authoritarian personality. New York: Harper. Ahn, W. K., Kalish, C. W., Medin, D. L, & Gelman, S. A. (1995). The role of covariation versus mechanism information in causal attribution. Cognition, 54, 299-352. Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage. Altmeyer, B. (1988). Enemies of freedom: Understanding right-wing authoritarianism. San Francisco: Jossey-Bass. Aronson, E., Blaney, N., Stephin, C , Sikes, J . , & Snapp, M. (1978). The jigsaw classroom. Beverly Hills, CA: Sage. Ashforth, B. E. (1994). Petty tyranny organizations. Human Relations, 47, 755-778. Augoustinos, M., & Reynolds, K. (Eds.). (2001). Understanding prejudice, racism and social conflict. London: Sage.

our

Experimental

Design and

Barber, T. X . , 8c Silver, M . J . (1968). Fact, fiction, and the experimenter bias effect. Psychological Bulletin, 70, 1-29. Baron, R. M., 8c Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 6 0 2 - 6 1 9 . Berndsen, M., Spears, R., van der Pligt, J . , 8c McGarty, C. (1999). Determinants of intergroup differentiation in the illusory correlation task. British Journal of Social Psychology, 90, 2 0 1 - 2 2 0 . Bless, H., Strack, F., 8c Schwarz, N. (1993). The informative functions of research procedures: Bias and the logic of conversation. European Journal of Social Psychology, 23, 149-166. Campbell, D. T., 8c Stanley, J . C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally. Cheng, P. W. (1997). From covariation to causation: A causal power theory. Psychological Review, 104, 3 6 7 - 4 0 5 . Chow, S. L. (1995). In defence of experimental data: Data in a relativistic milieu. New Ideas in Psychology, 13, 2 5 9 - 2 7 9 . Crocker, J . , 8c Luhtanen, R. (1990). Collective self-esteem and ingroup bias. Journal of Personality and Social Psychology, 58, 60-67. Dawes, R. M . (1996). The purpose of experiments: Ecological validity versus comparing hypotheses. Behavioural and Brain Sciences, 19, 2 0 . Fishbein, M . (1997). Predicting, understanding and changing socially relevant behaviours. In C. McGarty & S. A. Haslam (Eds.), The message of social psy­ chology: Perspectives on mind in society (pp. 77-91). Oxford, UK: Blackwell. Fisher, R. A. (1926). The design of experiments (1st ed.). London: Oliver and Boyd. Fiske, S. T., 8c Taylor, S. E. (1984). Social cognition. Reading, MA: Addison-Wesley. Gergen, K. J . (1997). Social psychology as social construction: The emerging vision. In C. McGarty 8c S. A. Haslam (Eds.), The message of social psychology: Perspectives on mind in society (pp. 113-128). Oxford, UK: Blackwell. Hamilton, D. L., 8c Gifford, R. K. (1976). Illusory correlation in intergroup perception: A cognitive basis of stereotypic judgments. Journal of Experimental Social Psychology, 12, 392-407. Haslam, S. A., 8c McGarty, C. (1998). Doing psychology: An introduction to research methodology and statistics. London: Sage. Haslam, S. A., 8c McGarty, C. (2001). A hundred years of certitude? Social psychology, the experimental method and the management of scientific uncertainty. British Journal of Social Psychology, 40, 1-21. psychology: Haslam, S. A., 8c McGarty, C. (in press). Doing research in Understanding methods and statistics. London: Sage. Haslam, S. A., McGarty, C , 8c Brown, P. (1996). The search for differentiated meaning is a precursor to illusory correlation. Personality and Social Psychology Bulletin, 2 2 , 611-619. Haslam, S. A., 8c Reicher, S. (2002). A user's guide to "The Experiment"— Exploring the psychology of groups and power: Manual to accompany the BBC video. London: BBC Worldwide. Haslam, S. A., Turner, J . C , Oakes, P. J . , McGarty, C , 8c Reynolds, K. J . (1998). The group as a basis for emergent stereotype consensus. European Review of Social Psychology, 9, 2 0 3 - 2 3 9 . Haslam, S. A., 8c Wilson, A. (2000). Is prejudice really personal? The contribution of a group's shared stereotypes to intergroup prejudice. British Journal of Social Psychology, 3 9 , 4 5 - 6 3 . Hogg, M . A., 8c Abrams, D. (1990). Social motivation, self-esteem and social identity. In D. Abrams 8c M . A. Hogg (Eds.), Social identity theory: Constructive and critical advances (pp. 2 8 - 4 7 ) . London: Harvester Wheatsheaf.

Causality

DESIGN AND ANALYSIS Hogg, M . A., 8c Sunderland, J . (1991). Self-esteem and intergroup discrimination in the minimal group paradigm. British Journal of Social Psychology, 30, 51-62. Hunter, J . A., Platow, M . J . , Bell, L. M., Kypri, K., 8c Lewis, C. A. (1997). Intergroup bias and self-evaluation: Domain-specific self-esteem, threats to 36, identity and dimensional importance. British Journal of Social Psychology, 405-426. Israel, J . , 8c Tajfel, H. (Eds.). (1972). The context of social psychology: A critical assessment. London: Academic Press. Jost, J . T., & Elsbach, K. D. (2001). How status and power differences erode personal and social identities at work: A system justification critique of organizational applications of social identity theory. In M . A. Hogg & D. J . Terry (Eds.), Social identity processes in organizational contexts (pp. 181-196). Philadelphia: Psychology Press. Kelley, H. H. (1967). Attribution theory in social psychology. In D. Levine (Ed.), Nebraska Symposium on Motivation (Vol. 15, pp. 1 9 2 - 2 3 8 ) . Lincoln: University of Nebraska Press. LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13, 230-237. Leaf, R. C. (1993). Control, volition and the experimental method. New Ideas in Psychology, 11, 3-33. Long, K., 8c Spears, R. (1997). The self-esteem hypothesis revisited: Differentiation and the disaffected. In R. Spears, P. J . Oakes, N. Ellemers, & S. A. Haslam (Eds.), The social psychology of stereotyping and group life (pp. 2 9 6 - 3 1 7 ) . Oxford, UK: Blackwell. McConahay, J . B., Hardee, B. B., & Batts, V. (1981). Has racism declined in America? It depends on who is asking and what is being asked. Journal of Conflict Resolution, 25, 5 6 3 - 5 7 9 . McGarty, C. (1999). The categorization process in social psychology. London: Sage. McGarty, C. (2001). Social identity theory does not maintain that identification produces bias and self-categorization theory does not maintain that salience is identification. British Journal of Social Psychology, 40, 173-176. McGarty, C , 8c de la Haye, A. (1997). Stereotype formation: Beyond illusory correlation. In R. Spears, P. J . Oakes, N. Ellemers, 8c S. A. Haslam (Eds.), and group life (pp. 145-170). Oxford, The social psychology of stereotyping UK: Blackwell. McGarty, C , 8c Haslam, S. A. (Eds.). (1997). The message of social psychology: Perspectives on mind in society. Oxford, UK: Blackwell. McGarty, C , Turner, J . C , Hogg, M . A., Davidson, B., 8c Wetherell, M . S. (1992). Group polarization as conformity to the most prototypical group member. British Journal of Social Psychology, 31, 1-20. Meers, A., Maassen, A., 8c Verhaagen, P. (1994). Subjective health after six months and after four years of shift work. Ergonomics, 21, 857-859. Meindl, J . R. (1993). Reinventing leadership: A radical, social psychological approach. In J . K. Murnigham (Ed.), Social psychology in organizations: Advances in theory and research (pp. 89-118). Englewood Cliffs, NJ: Prentice Hall. Mook, D. G. (1980). In defense of external invalidity. American Psychologist, 38, 379-388. Moscovici, S., 8c Zavalloni, M . (1969). The group as a polarizer of attitudes. Journal of Personality and Social Psychology, 12, 125-135. Ni Bhrolchain, M . (2001). "Divorce effects" and causality in the social sciences. European Sociological Review, 17, 33-57.

Experimental

Design and

Oakes, P. J . , & Turner, J . C. (1980). Social categorization and intergroup behaviour: Does minimal intergroup discrimination make social identity more positive? European journal of Social Psychology, 10, 2 9 5 - 3 0 1 . Orne, M . (1962). On the social psychology of the psychological experiment with particular reference to demand characteristics and their implications. American Psychologist, 17, 7 7 6 - 7 8 3 . Petty, R. (1997). The evolution of theory and research in social psychology: From single to multiple effect and process models of persuasion In C. McGarty &C S. A. Haslam (Eds.), The message of social psychology: Perspectives on mind in society (pp. 2 6 8 - 2 9 0 ) . Oxford, UK: Blackwell. Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160-164. Reicher, S. D. (2001). Studying psychology studying racism. In M. Augoustinos & K. Reynolds (Eds.), Social psychological approaches to prejudice and racism (pp. 2 7 3 - 2 9 8 ) . London: Sage. Rosenberg, M . (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press. Rosenthal, R. (1966). Experimenter effects in behavioral research. New York: Appleton-Century-Crofts. Sigall, H., & Mills, J . (1998). Measures of independent variables are useful in social and Social psychology experiments but are they necessary? Personality Psychology Review, 2, 2 1 8 - 2 2 6 . Smith, E. R. (1991). Illusory correlation in a simulated exemplar-based memory. journal of Experimental Social Psychology, 27, 107-123. Sohn, D. (1992). Knowledge in psychological science: That of process or population? journal of Psychology, 126, 5-16. Spears, R., & Smith, H. J . (2001). Experiments as politics. Political Psychology, 22, 309-330. Stoner, J.A.F. (1961). A comparison of individual and group dimensions involving risk. Unpublished mater's thesis, School of Industrial Management, Massachusetts Institute of Technology. Tajfel, H. (1972). Experiments in a vacuum. In J . Israel & H. Tajfel (Eds.), The context of social psychology: A critical assessment (pp. 69-119). London: Academic Press. Tajfel, H., Flament, C., Billig, M . G., &C Bundy, R. F. ( 1 9 7 1 ) . Social categorization and intergroup behaviour. European journal of Social Psychology, 1, 149-177. Tajfel, H., & Fraser, C. (1978). Social psychology as social science. In H. Tajfel & C. Fraser (Eds.), Introducing social psychology: An analysis of individual reac­ tion and response (pp. 21-53). Harmondsworth, UK: Penguin. Tajfel, H., & Turner, J . C. (1979). An integrative theory of intergroup conflict. In W. G. Austin & S. Worchel (Eds.), The social psychology of intergroup rela­ tions (pp. 33-47). Monterey, CA: Brooks/Cole. Turner, J . C. (1981). Some considerations in generalizing experimental social psychology. In G. M . Stephenson & J . H. Davis (Eds.), Progress in applied social psychology (Vol. 1, pp. 3-34). Chichester, UK: Wiley. Turner, J . C. (1999). Some current themes in research on social identity and selfcategorization theories. In N. Ellemers, R. Spears, & B. Doosje (Eds.), Social identity: Context, commitment, content (pp. 6-34). Oxford, UK: Blackwell. van Overwalle, F. (1998). Causal explanation as constraint satisfaction: A critique and a feedforward alternative. Journal of Personality and Social Psychology, 74, 3 1 2 - 3 2 8 .

Causality

DESIGN AND ANALYSIS Vissers, G., Heyne, G., Peters, V., & Guerts, J . (2001). The validity of laboratory research in social and behavioral science. Quality and Quantity, 35, 129-145. Walker, I. (2001). The changing nature of racism: From "old" to "new." In M. Augoustinos & K. Reynolds (Eds.), Social psychological approaches to prejudice and racism (pp. 2 4 - 4 2 ) . London: Sage. West, S. G., Newsom, J . T., & Fenaughty, A. M . (1992). Publication trends in the Journal of Personality and Social Psychology: Stability and change in topics, methods, and theories across two decades. Personality and Social Psychology Bulletin, 18, 4 7 3 - 4 8 4 . Zajonc, R. B. (1968). Attitudinal effects of mere exposure. Journal of Personality and Social Psychology, 9(Monograph Suppl. 2, Pt. 2 ) .

CHAPTER

12

Quasi-Experimental and Correlational Designs Methods for the Real World When Random Assignment Isn't Feasible MELVIN M . MARK The Pennsylvania State University CHARLES S. REICHARDT University of Denver

I

s hindsight, as the expression would have

researcher might randomly assign participants

us believe, really 2 0 / 2 0 ? H o w much do

to receive either a positive or a negative m o o d

"rose-colored

induction, and then compare the participants'

glasses"

impair

one's

vision? C a n there be "power t o the people"

ratings o f a fictitious target person presented

when

band

on video. T h e primary advantage o f random-

For causal questions such as these, ran-

causal conclusions more reliably than do other

individuals

in a c o m m u n i t y

ized experiments is that they support credible

together to act collectively? domized experiments are perhaps the most

research designs.

widely used, and best, kind o f research design.

Nonetheless, researchers would be at a rel-

In the classic randomized experiment, individ-

ative disadvantage if they did not have other

uals are randomly assigned to one o f two (or

design options in their methodological tool-

more) conditions (see Haslam & M c G a r t y ,

kits. Alternative designs are useful for at least

Chapter 1 1 , this volume). F o r example, imag-

four reasons. First, randomized experiments

ine a social psychologist with the hypothesis

are not always feasible. F o r instance, it is

that positive m o o d works like "rose-colored

probably not practical to study the effects o f

glasses" to prevent people from seeing the

community activist groups with a randomized

unflattering characteristics o f a generally lik-

experiment. Second, for at least some research

able

questions, even if r a n d o m

person.

T o test

this

hypothesis,

a

assignment is

266

DESIGN AND ANALYSIS feasible, it may be possible only in special

conditions. In quasi-experimental designs, the

circumstances; yet, it may be important to see

prospective causal variable typically is called

what happens in circumstances where ran-

the "treatment" or "intervention," and the

dom assignment is infeasible. For example,

potential effect is often called the "outcome."

researchers can manipulate m o o d using a randomized experiment in the laboratory, but researchers might want to k n o w if similar findings occur in everyday settings, with natu-

A Counterfactual Conception of Causality

ral mood inductions, without the trappings o f

Quasi-experiments are intended to probe

an artificial research laboratory. Third, ethical

causal relations. T h u s , to understand them,

considerations may preclude random assign-

it is useful to understand what is meant by

ment. For example, ethics might

prohibit

causality. T h e literature on causality is com-

random assignment to a treatment condition

plex and vast. Fortunately, even a general

that imposes an emotionally forceful negative

appreciation o f this literature can enhance

event to assess its effects on hindsight. Fourth,

understanding o f the theory and practice o f

the questions that interest social psychologists

quasi-experimentation

are not always causal in nature. For example,

Biesanz, & K w o k , Chapter 1 3 , this volume).

questions about personality characteristics can be either causal or correlational.

(also

see

West,

Probably the most common conception of causality in the social sciences is based on the

In this chapter, we describe alternative

counterfactual definition of a causal effect.

research designs that can circumvent the lim-

According to the counterfactual definition, the

itations o f randomized experiments. W e do

effect of a treatment—as compared to an alter-

so by illustrating h o w we have used these

native treatment or a no-treatment control—is

designs to address the questions listed at the

the difference between (a) the outcome that

start o f the chapter. O u r presentation focuses

occurred after the treatment was administered,

primarily on quasi-experimental designs, but

to one or more participants in a given context;

we also consider correlational designs.

and (b) the outcome that would have

occurred

if, instead, an alternative treatment or no treatment had been administered to the same particQUASI-EXPERIMENTAL

ipants in the same context. As an example, take

DESIGNS: AN O V E R V I E W

the question o f whether positive mood acts like

T h e development o f the theory, methods, and

seeing a target's negative attributes. In part,

nomenclature of quasi-experimentation owes

addressing this causal question would require

"rose-colored glasses" and keeps people from

much to the work o f Donald T . Campbell and

us to observe a set o f participants who, after a

his associates (Campbell & Stanley, 1 9 6 6 ;

positive mood induction, are asked to rate the

C o o k & Campbell, 1 9 7 9 ; Shadish, C o o k , &c

target person.

Campbell, 2 0 0 2 ) . " Q u a s i " is Latin and means

would really like to do would be to observe the

"as if," "like," or "an approximation." Thus,

very same participants, exposed to the alterna-

a quasi-experiment is an approximation o f

tive treatment (a neutral or negative mood

an experiment, a "near experiment." Like

induction), but at the same time and in the same

randomized

Counterfactually, what

we

quasi-experiments

context as we observed them in the positive

are used to estimate the effects of one or more

mood condition. Such a difference is called the

treatments on one or more outcome variables.

ideal comparison.

The

experiments,

quasi-experiments

Although it defines what is meant by a

don't have random assignment to treatment

difference

is that

treatment effect, the ideal comparison cannot

Quasi-Experimental

and Correlational

Designs

267

be attained in practice. T h e ideal comparison

little attention in the present chapter. Instead,

is impossible because it requires implementing

our focus is on the comparisons used in various

two mutually exclusive treatments on

the

quasi-experimental and correlational designs,

same people at the same time. T h e research

and on the extent to which such comparisons

designs we use can only approximate the ideal

are good approximations to the ideal but

but unattainable comparison. Sometimes the

unattainable counterfactual comparison—or,

comparison a research design uses is a good

in other words, on the extent to which the

approximation to the ideal comparison. Other

designs are strong or weak with respect to

times it is not. T h e validity o f the results o f a

internal validity (Reichardt & M a r k , 1 9 9 8 ;

study depends on h o w good an approxima-

Shadish et al., 2 0 0 2 ) .

tion the research design is to the ideal but impossible counterfactual comparison. The counterfactual definition o f the effect o f a cause is central to thinking about causality.

From the Concept of Causality to Kinds of Comparisons

T w o additional considerations also are impor-

As the idea o f "counterfactual" implies,

tant. One is the acknowledgment that there are

the process o f drawing a causal inference is

real mechanisms that underlie cause-and-effect

inherently comparative. Thinking counter-

relationships. T h a t is, processes exist that

factually, one would like to compare the out-

mediate the relationship between treatment

come that occurred following a treatment

and outcome variables. Contemporary con-

with the outcome that would have

cern about underlying mechanisms is reflected

if the treatment had not been implemented

occurred

in the growing attention to structural equation

(or an alternative treatment had been imple-

modeling and other techniques that can be

mented). O f course, in the absence o f time

used to test the contribution o f hypothe-

travel, one cannot obtain this counterfactual

sized mediators

(see Hoyle &c R o b i n s o n ,

comparison on the same people in the same

Chapter 1 0 , this volume). A second important

setting at the same time. Something else has

consideration is the recognition that causal

to differ. F o r example, one can compare dif-

relationships are contingent. T h a t is, the effect

ferent people, some exposed to the treatment

of a treatment can vary substantially, depend-

and some not, at the same time.

ing upon such factors as the characteristics of

Although other forms o f comparison are

the person w h o receives the treatment and the

possible ( M a r k , 1 9 8 6 ; Reichardt, 2 0 0 0 ) , the

nature of the settings in which the treatment is

most c o m m o n quasi-experimental

implemented (House, 2 0 0 1 ) . Increased appre-

involve comparisons either across times

ciation o f the often complex, contingent nature

across different

groups

designs

of individuals.

or

In the

of causal relations is reflected in increased

following sections, we describe each o f these

attention in research to moderators, that is, to

types o f quasi-experimental designs and illus-

variables that modify the strength and perhaps

trate them with studies we have conducted.

the direction o f an effect. It is also reflected in the growing interest in meta-analysis (Wood & Christensen, Chapter 1 5 , this volume), which

COMPARISONS ACROSS TIME

allows investigators both to look for moderators that vary across studies and to search for

Can there be "power

relatively robust effects that hold across the

individuals

persons and settings in individual

act collectively?

studies.

to the people"

in a community

when

band together

Steiner and M a r k

to

(1985)

Although mediators, moderators, and meta-

were interested in whether

analysis are important, these topics receive

extent community-based activist groups are

and to

what

268

DESIGN AND ANALYSIS effective. Rather than attempt to simulate an

conclusions can be drawn about whether and

activist group in the laboratory, Steiner and

to what degree the treatment-as-manipulated

M a r k examined a community action group

makes

a

difference

in

the

outcome-as

that protested a bank's announced plan to

measured (Shadish et al., 2 0 0 2 ) . Threats t o

increase the mortgage interest rates for a sub-

internal validity refer to categories o f factors

set o f its customers. In addition t o a variety

that can lead t o inaccurate conclusions about

o f other actions, the community action group

this causal relationship. O n e

organized a "mass withdrawal," encourag-

threat t o internal validity, history,

ing people to stop doing business with the

the possibility that specific events, other than

b a n k . T h i s provided

to

the treatment, occurred between the pretest

address a question rarely studied in the liter-

and the posttest and caused a change in the

ature

an

opportunity

on community organizations:

How

important refers to

outcome of interest. F o r example, if the c o m munity's major employer had laid off its

effective are such groups' actions?

workers at the same time as the community action group staged its mass withdrawal, the

The One-Group,

Pretest-Posttest Design

decline in the bank's savings account holdings could have been the result o f the layoff

Let us imagine that Steiner and M a r k

rather than the community group's actions.

compared (a) the average monthly passbook

Other validity threats also could have

savings account holdings at the b a n k during

posed problems if Steiner and M a r k had used

the month before the mass withdrawal with

the

(b) the average holdings in the m o n t h after.

Maturation

Such a comparison is called a one-group,

rally occurring processes that arise over time

pretest-posttest design. T h i s design can be

within participants, could affect the outcome

represented schematically as

variable. In a long study, maturation includes

one-group,

pretest-posttest

design.

refers t o the possibility that natu-

changes associated with growing older, such as getting smarter or having fewer behavior

OXO

problems. In shorter studies, maturation can where X represents the treatment, O repre-

refer to changes such as becoming more tired

sents the outcome variable (or "an observa-

or getting hungrier. In some cases, maturation

t i o n " ) , and time passes from left t o right (e.g.,

includes long-term trends in a community or

the first observation—that is, the pretest—

other population. In the context o f the study

takes place before the treatment is imple-

of community action groups,

maturation

mented). If Steiner and M a r k had used this

includes the possibility that there was a rela-

design, they would have found a decline in

tively steady, long-term decline in total sav-

the balance o f savings accounts, with this

ings at the bank, perhaps due to a long-term

total declining roughly 1 0 % from the month

decline in people's saving patterns. Another

before the group's action to the month after.

threat

to

internal

validity,

sometimes

H o w confident could Steiner and M a r k have

described as a special case o f maturation, is

been that this drop was due t o the actions o f

seasonality.

the community action group?

change in an outcome variable is the result of

A number o f "threats to internal validity"

This refers to the possibility that

regular seasonal patterns. For example, sav-

m a k e it difficult to conclude that the activi-

ings can fluctuate from month to

ties o f the community group caused the

because o f gift buying during winter holidays

month

valid­

or because o f vacation spending during the

ity refers t o the extent to which accurate

summer. Regression toward the mean is also

decline in the bank's holdings. Internal

Quasi-Experimental

and Correlational Designs

a potential threat to validity; it refers t o the

M a r k chose not to use this simple design.

possibility that change in the outcome vari-

However, in special circumstances the con-

able occurs because the pretest level was

text in which the design is implemented may

unusually high or low and the outcome vari-

make the various threats to validity implausi-

able simply returned t o a more " n o r m a l "

ble (see, e.g., Eckert, 2 0 0 0 ) , or supplementary

level at the posttest.

information, apart from the research design,

Three other validity threats apply in gen-

may render the validity threats implausible

eral to the one-group, pretest-posttest design,

and thus allow confident conclusions. W e

but do not seem plausible in the simplified

return to such possibilities in a later section.

pretest-posttest version o f the Steiner and

In general, however, the one-group, pretest-

M a r k study. Testing refers to the possibility

posttest design seldom is adequate for testing

that the very act o f measurement at the pretest

hypotheses in social psychology. But more

causes a change in the outcome variable at the

complex

posttest. For example, in a one-group, pretest-

across times can sometimes suffice.

posttest study o f the effects o f an S A T prepa-

designs are described next.

designs

involving

comparisons Such

ration course, testing would be a problem if students scored higher on the posttest simply because the pretest familiarized them with the

Interrupted Time-Series

Designs

refers to the pos-

Instead o f the one-group pretest-posttest

sibility that a change in the measuring instru-

design, Steiner and M a r k ( 1 9 8 5 ) used the

test format. Instrumentation

effect.

simple interrupted time-series (ITS) design in

Instrumentation effects can arise from changes

their study o f the effectiveness o f the com-

in the definition o f an outcome variable or in

munity action group. I T S designs extend the

the measurement procedures. Attrition occurs

one-group, pretest-posttest

when participants drop out of the study. This

uses a single pretest and a single posttest

change in the sample can bias the estimate o f

observation, by adding observations both

the treatment effect.

before and after the treatment. Using the

ment is responsible for an apparent

Although the one-group, pretest-posttest design is often easy to implement in practice,

design,

which

same notation as earlier, the simple I T S design can be symbolized as

it is also c o m m o n l y subject to one or more o f O O O O O O X O O O O O O .

the preceding threats to internal validity. O u t of the long list o f validity threats that can plague this design—history, maturation, sea-

Figure 1 2 . 1 presents a subset o f the data

sonality, regression toward the mean, testing,

from

instrumentation, and attrition—one or more

action group called for the mass withdrawal

Steiner and M a r k . T h e community

often is sufficiently plausible that confident

campaign t o begin in July, 1 9 9 9 , which is

conclusions about the effect o f the treatment

labeled M o n t h 3 5 and denoted by a vertical

cannot be drawn. In the one-group, pretest-

line in the figure. T h e trend in savings bal-

posttest design, the pretest stands in for the

ances before the campaign is upward, as indi-

ideal counterfactual (i.e., what would have

cated by the regression line drawn through

happened if the treatment had not occurred),

the pretreatment observations. T h e trend in

but, for a variety o f reasons—summarized in

savings balances after the campaign started is

the long list o f internal validity

relatively flat, and there is a sharp drop right

threats

above—the pretest often does not provide a

at M o n t h 3 5 . T h e s e shifts in the regression

good approximation to the desired counter-

lines suggest that the mass withdrawal cam-

factual outcome. Accordingly, Steiner and

paign

w a s effective in reducing

savings

269

270

DESIGN AND ANALYSIS

7,500,000

7,000,000 w CD

O

6,500,000 CO

00 w c > co CO

6,000,000

5,500,000

5,000,000 1

10

20

30

40

50

60

Month Figure 12.1

balances, both immediately and increasingly over time.

an a b n o r m a l l y high or low

observation

shortly before the treatment will

T h e I T S design helps rule out several o f

appear

deviant relative to the other pretest observa-

the validity threats that plague the one-

tions. T h e threat o f testing also is generally

group, pretest-posttest design. F o r example,

controlled

for

in the I T S design.

With

maturation often is a plausible threat in the

repeated observations prior to the treatment,

one-group, pretest-posttest design. In the I T S

it is usually implausible that a sizable testing

design, in contrast, the effect o f maturation

effect would coincide with the intervention.

(e.g., a general upward o r downward trend

O n the other hand, the simple ITS design

in the observations) can be identified and

does not rule out all threats to internal valid-

taken

usually

ity. For example, history is often plausible. In

should be evident as an ongoing upward or

fact, history effects are as likely in the simple

downward

ITS design as in the one-group,

into a c c o u n t . M a t u r a t i o n trend,

and

a treatment

effect

appears as a rise or fall relative to the matu-

pretest-

posttest design, if the time interval between

rational trend. Similarly, regression toward

observations is the same. However, time-series

the mean is often plausible in the one-group,

designs often have shorter intervals between

pretest-posttest design but can be assessed

observations, making history effects less likely.

(and, therefore, ruled out) in an I T S design;

Instrumentation also can threaten the simple

Quasi-Experimental

and Correlational Designs

\

ITS design. In some cases, the introduction o f

account data over time from a bank in a

a treatment is associated with a change in the

nearby community that was not subject to the

way observations are defined or recorded.

community action group's activities. In gen-

This is c o m m o n , for example, in ITS studies o f

eral, adding a control group to the simple ITS

the impact o f legal changes, where a law that

design helps to control for history threats. For

changes the punishment

for a crime also

example, if an overall decline in the economy

o f the crime (e.g.,

had been responsible for the drop in savings

changes the definition

M a r s h , 1 9 8 5 ) . T h e threats o f history and

that Steiner and M a r k observed, a similar

instrumentation sometimes can be addressed

effect should have occurred in the savings

by careful analysis o f the historical record and

account holdings at a bank in a nearby com-

record-keeping procedures, respectively. For

munity. In this case, however, there would

example, Steiner and M a r k ( 1 9 8 5 ) consulted

have been a problem using a control time series

bank officials to assess whether there had been

from a nearby bank if people moved their sav-

changes in record keeping at the time o f the

ings from the targeted bank to the neighboring

community group's actions. Similarly, they

bank. (We return to the idea o f a control group

looked for general changes in the local econ-

in the more general discussion o f between-

omy and asked bank officials and community

group designs below.)

members about historical events that might

Comparisons other than a control group

have caused a downturn in savings. They

can be added to the simple ITS design, to try

found no evidence o f such changes. Rather

to reduce the plausibility o f problems o f

than relying solely on such detective work,

history and

researchers often attempt to control history by

option is known in the time-series literature

instrumentation.

One

design

using a more complex ITS design, most often

as the ITS with removed treatment design.

by adding a control group.

For instance, if the community action group in the Steiner and M a r k example had called

Interrupted Designs

off its actions and encouraged people

Time-Series

With a Control

to

return their savings t o the bank in question,

Group

the researchers could have assessed whether

O n e way to address threats to the validity

bank holdings went back up. History and

of the simple I T S design is to add time-series

instrumentation usually cannot explain such

observations from a group o f respondents

occurrences as both a decrease in savings

w h o were n o t exposed t o the

account balances at the start o f the mass with-

treatment.

Schematically, the design appears as

drawal campaign and a subsequent increase after the campaign ends, so they become less plausible as threats to validity. In another

O ^ O O C D O X O O O O O O O O O O O O

o o o o o o '

complex design, the I T S with nonequivalent dependent

The

top line represents time-series data

measures design, two

different

outcome variables are measured in time-series

were

form. O n e outcome measure is expected to be

exposed to the treatment, and the bottom line

influenced by the treatment, while the other is

from

a group o f participants

who

represents time-series data from a comparison

not; but both should be affected by the same

group that did not receive the treatment. T h e

validity threats. I f the first outcome changes

dashed line is used to show that these two

and the second does not, threats to validity

groups are nonequivalent (i.e., not created by

(which should have the same effects in the

random assignment). For example, Steiner and

two time series) become less plausible. F o r

M a r k ( 1 9 8 5 ) could have examined savings

example, if the community action group had

271

272

DESIGN AND ANALYSIS targeted savings accounts but not checking

for which

accounts, Steiner and M a r k could have used

responsible for not preparing for), they may

checking account data as a nonequivalent

conclude that it was not foreseeable. If the

dependent variable.

they feel responsible

(or

feel

negative outcome were so darned foreseeable, why wouldn't they have avoided it or at least prepared for it?!

Analysis

of

Time-Series

Interrupted Designs

T h e one-group, pretest-posttest design and the simple ITS design wouldn't be very useful

T h e statistical analysis o f ITS designs is

in addressing M a r k and Mellor's question. In

more complicated than the analysis o f tradi-

those designs, the effect o f the treatment is

tional experimental designs. T h e primary dif-

estimated by comparing the same individuals

ficulty with time-series observations is that

at different points in time, specifically by

data points adjacent in time are more likely to

comparing their standing on the outcome

be similar than data points more distant in

variable before the treatment

time. This phenomenon, called autocorrela-

standing on the outcome variable after the

with

their

tion, can lead to erroneous conclusions from

treatment. M a r k and Mellor's research ques-

traditional hypothesis tests and confidence

tion called instead for comparing

intervals. Several strategies can be used to

groups o f individuals w h o were subjected to

different

deal with autocorrelation. F o r longer time

different treatments. T h a t is, they needed to

series, perhaps the most c o m m o n approach

compare hindsight judgments from

involves the use o f autoregressive integrated

w h o were negatively affected by an event and

moving average ( A R I M A ) models. In essence,

hindsight judgments from people w h o were

the A R I M A approach calls for an empirical

not negatively affected.

people

determination o f the nature o f the autocorrelation, which allows statistical procedures to adjust for the problems that autocorrelation introduces. Additional methods o f analysis are available for shorter time series (see, e.g., M a r k , Reichardt, & Sanna, 2 0 0 0 ) .

The Regression-Discontinuity Design Although

the

randomized

experiment

often is the best design for comparing groups of individuals w h o receive different

treat-

ments, there are ethical concerns with ranCOMPARISONS ACROSS GROUPS

domly assigning people to a truly negative and

Is hindsight 20/20?

Hindsight bias refers to

self-relevant treatment—the

kind o f

treatment presumably most likely to eliminate

the tendency for people, once the outcome o f

hindsight

any event is known, to overestimate h o w pre-

opportunism

bias. Fortunately, reflecting

dictable the outcome was in foresight. In

quasi-experimental

the

that often characterizes good research,

Mark

and

other words, hindsight bias leads people to

Mellor found a natural event that allowed

believe that they " k n e w all along" that things

them to use a relatively strong alternative: the

would turn out as they did. Although hind-

quasi-experimental, between-groups

sight bias has been demonstrated in a wide

known as the regression-discontinuity design.

range o f circumstances, M a r k and Mellor (1991)

speculated that

design

The regression-discontinuity design uses a

self-serving biases

very different approach to assignment to treat-

should reduce or eliminate the hindsight bias

ment conditions from the randomized experi-

under certain conditions. In particular, when

ment. T h e regression-discontinuity

people directly experience a negative outcome

requires that participants be measured, prior

design

Quasi-Experimental

and Correlational Designs

to the treatment, on a special form o f pretest

half o f the figure reveal, hindsight bias (i.e.,

that we call the quantitative assignment vari-

the

stated

predictability

o f the

layoff)

able, or Q A V (Reichardt & M a r k , 1 9 9 8 ) . T h e

decreased

Q A V is used to assign participants to condi-

increased. M o r e importantly, there is a dra-

tions based on a specified cutoff value. Those

matic discontinuity in hindsight bias exactly

w h o score above the cutoff value are assigned

at the cutoff ( 1 9 years o f seniority). I f the lay-

in

both

groups

as

seniority

to one condition, and those w h o score below

off had n o effect on hindsight bias, the two

the cutoff are assigned to the other condition.

regression

Participants are subsequently measured on the

straight line. T h e downward

outcome variable, and the treatment effect is

regression line for the laid-off workers, com-

estimated as a discontinuity in the QAV-out-

pared t o the regression line for the survivors,

come relationship at the cutoff.

lines should

connect

as

one

shift in the

is evidence that the layoff caused a reduction

M a r k and M e l l o r ( 1 9 9 1 ) used the regres-

in hindsight bias. T h e design's name, "regres-

sion-discontinuity design to examine whether

sion-discontinuity," comes from the fact that

hindsight bias is attenuated among those w h o

the treatment effect is estimated by a discon-

experience a severely negative

tinuity between the t w o regression lines.

and

self-

relevant event. They addressed this question in

A graphical representation o f regression-

the context o f a company that laid off a large

discontinuity findings, such as Figure 1 2 . 2 ,

number o f workers. Whether or not a worker

highlights the source o f the design's inferen-

was laid off depended on the number o f years

tial strength. In general, n o plausible threats

he or she had been with the company, that is,

to validity exist that are likely to produce a

on seniority. In other words, seniority served

discontinuity precisely at the cutoff between

as the Q A V : Workers could be arrayed based

the treatment conditions. However, the esti-

on the number o f years o f seniority, and there

mate o f the treatment effect can be biased if

was a cutoff, specifically at 1 9 years, with

the relationship between the Q A V and the

individuals below the cutoff assigned to the

outcome variable is curvilinear but a linear

layoff group and people above the cutoff

relationship is fit to the data. T o address this

assigned to what M a r k and Mellor called the

problem, curvilinearity in the data should be

"survivor group"; that is, they were able to

modeled in the analysis. In short, statistical

continue working. T h e cutoff in this case was

analysis o f the regression-discontinuity design

chosen by the company, based on h o w many

typically involves predicting the

workers had to be laid off. T h e value o f the

variable in a series o f regression analyses,

regression-discontinuity design is that there is

where the predictors are the Q A V (trans-

outcome

no reason to believe a priori that the cutoff

formed by subtracting the cutoff value, so

would be related in any way to workers' judg-

that the treatment effect is estimated at the

ments about the foreseeability o f the layoff,

cutoff point), a dummy code

unless being laid off affected these judgments.

condition (e.g., laid off versus survivors), and

representing

a term representing the interaction o f condi-

Six weeks after the layoff, M a r k

and

Mellor surveyed laid-off workers and

sur-

tion and the Q A V . N o t e that the condition

vivors and asked them h o w predictable they

variable estimates the difference between con-

thought the layoff was in advance. Figure 1 2 . 2

ditions that exists over and above the rela-

shows the results. D a t a from those

who

tionship between the Q A V and the outcome.

were laid off are plotted on the left side o f the

In addition, polynomials o f the transformed

figure, and data from those w h o were not

Q A V and interaction are included; if they are

laid off are plotted on the right side. As the

not

regression lines fit through the data in each

Estimation o f the polynomials serves to test

significant,

they

are

dropped

out.

273

274

DESIGN AND ANALYSIS

3

c I

L

o j

0 Figure 12.2

20

Three-Year Averages

40

Years of Job Seniority

for the possibility o f a nonlinear relationship

merit could be developed, it could serve as

that could otherwise masquerade as a treat-

the Q A V for the

ment effect. In addition, researchers w h o use

design. However, in practice, the regression-

the regression-discontinuity design should be

discontinuity design has rarely been used

regression-discontinuity

aware that the design has substantially less

(Shadish et al., 2 0 0 2 ; T r o c h i m , 1 9 8 4 ) . Still, a

power

researcher w h o has the regression-discontinu-

than

a

randomized

experiment

(Cappelleri, Darlington, & T r o c h i m , 1 9 9 4 ) . The

regression-discontinuity

design

ity design in his or her toolkit may be sensi-

is

tized to look for those circumstances that fit

quasi-experimental

the requirements o f the design. F o r example,

designs. At its best, the design comes close to

large organizations' major personnel func-

the randomized experiment in terms o f allow-

tions, such as promotions and hiring, may

ing confident causal inferences, and it can be

meet the requirements

feasible in circumstances in which practical or

discontinuity design. If so, this could offer

among

the

strongest

o f the regression-

ethical considerations rule out random assign-

opportunities to study a variety o f social psy-

ment. In particular, random assignment may

chological phenomena, including affect, cop-

be unacceptable for treatments designed for

ing, expectations, and

those most in need (e.g., compensatory educa-

relatively strong quasi-experimental design in

hindsight,

using

a

tion programs) or most meritorious (e.g., pro-

a real world setting and with a relatively

motions). If an acceptable measure of need or

powerful treatment (Mellor & M a r k , 1 9 9 8 ) .

Quasi-Experimental

much

do

"rose-colored

glasses"

impair vision? Drawing on previous research on the effects o f m o o d (e.g., Schwarz & Clore, 1983),

Sinclair, M a r k , and

Clore

(1994)

hypothesized (in part) that, relative to people in a bad m o o d , people in a good m o o d would attend less to the strength o f the arguments in a persuasive message (and thus be more persuaded by a weak argument). Imagine that, to test this hypothesis, a researcher approaches students on a college campus on days with either notably pleasant or notably unpleasant weather. Based on the results o f Schwarz and Clore ( 1 9 8 3 ) , the researcher presumes (and verifies with

a manipulation

check)

that

people are in a more positive m o o d on days with pleasant weather. T h e student participants are asked to listen to a brief and weakly argued message advocating comprehensive e x a m s as a requirement

for

275

provide credible conclusions. Instead, the

Nonequivalent Group Designs How

and Correlational Designs

graduation.

Assume that, as predicted, the participants

internal validity threat o f initial selection differences

looms large. In the

preceding

example, different kinds o f people may tend to be out and about on pleasant days, as opposed to unpleasant days. F o r example, perhaps only highly dedicated students tend to be on campus on days with bad weather (making their way to class), while less motivated students spent time hanging around campus only on nice days. In short, the two groups could differ in any o f a large number o f ways, and these initial differences, rather than participants' m o o d , could account for observed differences on the outcome variable. A pretest measure is often added to the posttest-only nonequivalent group design to assess the size o f initial selection differences and thereby t o try t o take account o f their effects. T h e resulting, pretest-posttest nonequivalent group design can be depicted schematically as

approached on pleasant days are more persuaded by the message than participants on

OXO

the unpleasant days (as would be expected, if

O

O

people in a good m o o d are relatively inattentive to the strength o f an argument). This design is a posttest-only nonequivalent group design, shown schematically as XO O

It

would

likely

be

difficult

for

the

researcher in the preceding example to obtain a pretest measure, which would be an assessment o f students' reactions to weak arguments on an earlier day with weather.

However,

the

"neutral"

pretest-posttest

nonequivalent group design is c o m m o n in In this design, one group o f participants is

many research areas. For example, it is widely

exposed to the treatment, while another is not

used in studies o f educational and clinical

(or, alternatively, is exposed to a different

interventions (Lipsey & Wilson, 1 9 9 3 ) . Social

treatment condition). Both groups are then

psychologists can find this design used in

measured on the outcome variable. T h e design

applied research areas, addressing questions

is a quasi-experiment because the participants

such as the degree to which educational

are not randomly assigned to the treatment

interventions can reduce either

conditions. As a result, the participants in

(Signorella, Frieze, & Hershey, 1 9 9 6 ) or sus-

the treatment conditions are likely to be

ceptibility t o heuristics and biases (Lehman,

nonequivalent—hence, the name for the design.

Lempert, &c Nisbett, 1 9 8 8 ) .

prejudice

Although this elementary quasi-experiment

In some instances, the pretest-posttest non-

often is easy to implement, it usually does not

equivalent groups design may provide sufficient

276

DESIGN AND ANALYSIS evidence to allow a tentative conclusion about

confidence to the conclusions. Nonetheless,

the effects o f a treatment variable. In general,

there will always be some remaining uncer-

however, the validity threats that can plague

tainty as to whether selection has been ade-

this design deserve serious

quately controlled for.

consideration.

Although the pretest offers some possibility o f

T h e simplest analytic approach is to esti-

accounting for initial selection differences, in

mate the treatment effect as the difference

general there will always be at least some

between the treatment groups in the mean

remaining uncertainty as to whether selection

change from pretest to posttest scores. Called

has been adequately controlled for. In addi-

either a change-score or gain-score analysis,

tion to initial selection differences, the pretest-

this requires that operationally identical mea-

posttest

nonequivalent

groups

design

is

sures be used at the pretest and posttest. T h e

subject to a set o f validity threats such as

change-score analysis assumes that the mean

"selection by history" and

"selection by

change in each o f the treatment groups would

instrumentation," among others. Selection by

be the same in the absence o f a treatment

history, for example, refers to the possibility

effect. Although this assumption is reasonable

that the two groups are subject to different

under some circumstances, it can be seriously

historical forces. If so, even if the groups start

incorrect when the two groups are maturing

out close together on the outcome variable at

at different rates. Even if the groups start out

the pretest, they might end up further apart at

close together on the pretest, they can end up

the posttest simply because o f differential

farther apart at the posttest simply because of

history effects, even in the absence o f any

differential

treatment effect. Selection by

absence o f a treatment effect. This validity

instrumentation

refers to the possibility that the two conditions experience different

instrumentation

prob-

maturation

rates, even in the

threat is called selection by maturation. Analysis o f covariance ( A N C O V A )

is

lems. For instance, this might happen if one

another c o m m o n analytic approach. T o con-

observer is collecting data for the treatment

trol for the effects o f selection differences,

condition and another observer for the control

A N C O V A estimates the effect o f the treat-

(e.g., because they are at different sites), and

ment

one observer's standards change over time but

matched statistically on their pretest scores.

the other's don't.

T h e problem is that the treatment

by comparing

individuals

who

are

groups

may have initial selection differences on variables besides the pretest, in which case the Analysis

of the

Nonequivalent

Pretest-Posttest Group

results from A N C O V A will remain biased.

Design

T h e solution is to include in the statistical

A variety of statistical procedures are fre-

model all the variables that affect the posttest

quently used to estimate treatment effects in

and on which there are initial selection dif-

nonequivalent group designs. T h e task is the

ferences. But this raises a new

same in each case: to take account of the bias-

knowing

which

ing effects

included,

and

o f initial selection differences

variables

measuring

need them.

problem: to

be

Another

(Reichardt, 1 9 7 9 ) . T h e problem is that no ana-

problem with A N C O V A is that measure-

lytic procedure is guaranteed to adjust properly

ment error in the pretest, or in any other

for the effects of selection differences. As a

covariate, will bias the estimate o f the treat-

result, a common recommendation is to con-

ment effect. T h e effects o f such measurement

duct multiple analyses (that make

varying

error can be removed by using latent variable

assumptions about the nature o f selection dif-

models such as in structural equation model-

ferences), in search of convergent results to add

ing (see Hoyle & R o b i n s o n , Chapter 1 0 ,

Quasi-Experimental

and Correlational Designs

this volume), though such analyses require

and posttest assessments solves some problems,

multiple measures o f each covariate.

it can cause others, such as

Selection modeling is another

analytic

introducing

differences between the pretest and posttest

approach for estimating treatment effects in

samples. F o r example, people might be in

nonequivalent

group designs. In essence,

more o f a hurry or be more distracted on their

selection modeling fits two equations to the

way into a movie than on their way out, or

data. O n e equation models the process by

people might refuse to participate at one time

which individuals were selected into treat-

more than at the other.

ment conditions. T h e second equation is a causal model o f the posttest

scores

and

includes the results from the first equation.

Complex

Selection modeling is m o r e c o m p l e x statisti-

Group

cally than the other approaches noted above. Discussions o f these and related strategies are provided

analysis

in W i n s h i p

and

The

Nonequivalent Designs

credibility o f nonequivalent

group

designs often can be improved by making the designs more c o m p l e x . F o r example, Sinclair

M o r g a n ( 1 9 9 9 ) , Reichardt and M a r k ( 2 0 0 1 ) ,

etal.

and Shadish et al. ( 2 0 0 2 ) .

posttest-only nonequivalent group design to

(1994)

used

the inferentially

weak

assess the effects o f m o o d on the persuasiveness o f arguments, but they greatly enhanced The Pretest-Posttest Group

Design

With

Pretest and Posttest

the interpretability o f the results by embed-

Nonequivalent Separate

ding the design within a more elaborate set o f

Samples

comparisons.

In some quasi-experimental designs, the data from the pretest and posttest c o m e from independent

samples.

S c h e m a t i c a l l y , the

design is

As described earlier, previous research had shown that, relative to respondents in a positive m o o d , those in a negative m o o d are more likely to attend to the content o f persuasive messages. As a result, research has shown,

O ;xo

those in a negative m o o d show a much larger difference, in terms o f persuasion, between a

O ! O

weak

argument

and

a strong

argument.

Sinclair et al. were interested in whether these where the vertical line indicates that

the

effects o f m o o d on persuasion resulted from

pretest measure is collected on a different

affect-as-information

sample

1 9 8 3 ) , which posits that

than

the

posttest

measure.

For

(Schwarz & negative

Clore, mood

example, Forgas and M o y l a n ( 1 9 8 7 ) tested

serves as a cue that the environment is dan-

the effects o f m o o d on political and other

gerous, risky, or threatening, and therefore

judgments by approaching people outside a

calls for careful attention. Conversely, the

movie theater showing either a comedy or a

affect-as-information

serious drama. People were asked questions

positive m o o d serves as a cue that the envi-

either before they went in to see the movie or

ronment is nonthreatening, a condition that

after. Different sets o f respondents were ques-

elicits less detailed and careful processing.

tioned before and after the movie because the

hypothesis

proposes,

Sinclair et al. tested this hypothesis using

pretest measure o f m o o d could have alerted

(in

the respondents to the types o f changes that

group design that was described previously.

were expected. N o t e that although using dif-

T h a t is, they approached undergraduates on

ferent samples o f respondents for the pretest

either notably pleasant or notably unpleasant

part)

the posttest-only

nonequivalent

277

DESIGN AND ANALYSIS

278

No Attribution 7

­

6

­

5

­

4

­

External

•

CD

"D

Strong

3

Weak

\

Pleasant

Unpleasant

Pleasant

Unpleasant

Weather Figure 12.3

days and asked them to listen to a message

persuasive than the earlier, stripped-down

advocating the implementation o f compre-

version used t o illustrate the posttest-only

hensive final exams. In addition, Sinclair et al.

nonequivalent group design (which corre-

randomly assigned participants t o hear a mes-

sponds to the findings for the weak argument

sage with either weak or strong arguments.

conditions only). Comparing responses to

The left-hand side o f Figure 1 2 . 3 presents the

both strong and weak messages rules out, for

results—for the part o f the design described

example, the possibility that people are sim-

so

ply more negative generally in their evalua-

far. As shown

in the left portion o f

Figure 1 2 . 3 , relative to people w h o approached

on

a

pleasant

day,

were

tions on unpleasant days. T h e underlying

those

logic is akin to that o f the nonequivalent

approached on an unpleasant day were sub-

dependent measure, discussed previously in

stantially more persuaded by the strong mes-

the c o n t e x t o f interrupted time-series designs.

sage and substantially less persuaded by the

Sinclair e t a l . added even more compar-

weak message. T h a t is, participants on an

isons by manipulating another variable. They

unpleasant day were more sensitive to and

randomly assigned participants to either make

more strongly affected by the manipulation o f

or not make an external attribution regarding

the strength o f the argument.

the weather. Participants in the external attrileft-hand

bution condition were told that weather could

portion o f Figure 1 2 . 3 is considerably more

The

pattern o f results in the

affect moods and were asked to rate the

Quasi-Experimental weather for

so the researchers could

any "mood-weather

control

effects that could

about

and Correlational Designs

underlying

\

psychological processes.

T h a t a c o m p l e x pattern o f predictions and

no-attribution

findings can both enhance the credibility and

participants were not given this explanation

expand the scope o f causal inferences is one

before they listened to either the weak or

of the most important principles o f research

strong message. T h e attribution manipulation

design in general and quasi-experimentation

was included because previous research by

in particular (Campbell, 1 9 6 6 ; M a r k , 1 9 9 0 ;

Schwarz and Clore ( 1 9 8 3 ) had shown that

Reichardt, 2 0 0 0 ; T r o c h i m , 1 9 8 5 ) .

affect other responses." T h e

affect-as-information

effects

do not

279

occur

when people are led to make an external attribution for their m o o d . As shown in the righthand

panel o f Figure 1 2 . 3 , weather

CORRELATIONAL DESIGNS

and

argument strength did not interact for the par-

Unlike

ticipants w h o made an external attribution.

designs are not always used to assess causal-

The

complete pattern o f results

quasi-experiments,

correlational

from

ity. Correlational studies have long been used

Sinclair et al., as summarized in Figure 1 2 . 3 ,

to examine the factorial structure o f intelli-

provides a far more compelling case than

gence and personality, for example, with no

would a stripped-down version o f the study

causal

using only the posttest-only

nonequivalent

because this chapter focuses on assessing

group design. N o t only do the multiple com-

causality, our concern here is solely with the

parisons and complex pattern o f findings

use o f correlational designs t o draw infer-

increase confidence that one has observed a

ences about cause and effect.

conclusion

intended.

However,

cause-and-effect relationship, but the pattern

Correlational designs used to probe causal-

of results also helps clarify the psychological

ity are like quasi-experiments in that individu-

processes underlying that relationship. In the

als are not randomly assigned to treatment

case of Sinclair et al., the very different pattern

conditions. It is a bit more difficult to be clear

of effects for the external and no-attribution

about how correlational designs are

conditions supports the hypothesis that affect-

quasi-experiments, because there does

as-information is the underlying process.

appear to be a clear, universally agreed upon

unlike not

distinction. O n e seemingly c o m m o n approach is to say that, unlike in quasi-experiments, in

Summary

correlational designs there either is not a treathave

ment or the design does not allow confidence

quasi-experimental

that one variable is the treatment and the other

designs, from those that typically are weak

the outcome. In other words, correlational

inferentially, such as the one-group, pretest-

designs lack the standard features o f quasi-

In

the

reviewed

preceding a range

of

sections, we

posttest design, t o more rigorous designs that

experimental design, such as pretests (Shadish

allow relatively confident causal inference,

et al., 2 0 0 2 ) . O r we could focus on the way the

such as the

and

cause is measured: In correlational designs, the

interrupted time-series designs. W e have also

cause typically is a continuously scaled vari-

regression-discontinuity

considered c o m p l e x designs that c o m b i n e

able, whereas in quasi-experiments the cause

experimental and quasi-experimental design

typically is a discretely scaled variable (where

features. As illustrated by the Sinclair et al.

"discrete" means categorical, with relatively

( 1 9 9 4 ) example, such hybrid designs often

few categories). For instance, in the Sinclair

can provide credible evidence o f cause-and-

e t a l . ( 1 9 9 4 ) study o f the effect of mood,

effect relations while also providing evidence

weather—the indicator o f mood—was scaled



280

DESIGN AND ANALYSIS into two

or

implemented without a time lag between the

unpleasant. Thus, the design was characterized

discrete categories, pleasant

putative cause and effect. F o r example, in a

as a quasi-experiment. In contrast, weather

correlational study o f the effects o f stress on

could have been measured in a more continu-

physical health, a single questionnaire might

ous manner, perhaps leading the study to be

be used to ask respondents about their cur-

characterized as correlational. If the causal

rent levels o f both stress and physical health.

variable is measured continuously, the analysis

Such a design is called cross-sectional and

of data from correlational designs typically is

introduces two critical difficulties. First, cross-

conceptualized in terms of correlations, rather

sectional correlational designs are subject to

than

differences

the validity threat o f "ambiguous temporal

that are the typical focus in the analysis o f

precedence" (Shadish et al., 2 0 0 2 ) . T h a t is, the

quasi-experiments.

purported cause might instead be the effect.

in

terms

o f the

mean

For example, in studying the effects o f stress on physical health, a researcher might unwit-

Threats to the Validity of

Correlational Designs

tingly be assessing the effects

Correlational designs are conceptually similar

to

simpler

nonequivalent

quasi-experiments. As with

o f physical

health on stress. Second, the size o f a treatgroup

nonequivalent

ment effect is likely to vary with the interval between treatment implementation and outcome

measurement

( G o l l o b & Reichardt,

group designs, selection differences are the

1 9 8 7 , 1 9 9 1 ; Reichardt, 2 0 0 2 ) . F o r example,

most prominent threats t o validity in correla-

the effect o f stress on physical health presum-

tional designs. F o r example, as noted in the

ably varies depending on whether the time lag

previous example o f a study o f the effects o f

between a stressor and the measure of physical

m o o d on persuasion, students w h o tend to

health is a day, a week, a month, a year, or a

be on campus on pleasant days may differ

decade. W h e n stress and physical health are

from those on campus on unpleasant days,

assessed concurrently, it is not clear which

and these differences can bias the estimate o f

time lag the results apply to, if any.

the treatment effect. T h e threat to validity o f

T o avoid ambiguities about time lags, the

selection differences applies at least as much

measures o f the cause and effect in a correla-

in correlational designs as in nonequivalent

tional design must be tied to specific, distinct

group designs. In the literature on correla-

times. If data are collected at the same time,

tional designs, however, selection differences

one way t o insert a time lag between, say,

typically are labeled the problem o f omitted

stress and physical health is to have respon-

third variables.

dents report their current degree o f physical

Another notable disadvantage o f correla-

health and their level o f stress as it was at a

tional designs involves their handling o f

prior time. T h e problem is that retrospective

time lags. A prototypical feature o f quasi-

reports are notoriously inaccurate in ways

experimental designs is that the treatment is

that can bias the results (e.g., Schwarz &

implemented before the outcome is measured.

Oyserman, 2 0 0 1 ) . F o r example, those in

In fact, a time sequence between the cause

p o o r health may simply be more likely to

and

usual

recall negative stressors than those in good

schematic depictions o f quasi-experiments,

health. In general, longitudinal designs are

the effect

is embodied in the

where the X representing the treatment comes

better at inserting a reliable time lag between

t o the left of—that is, earlier in time than—

cause and effect, and it is encouraging to note

the O representing the outcome variable. In

that

c o n t r a s t , c o r r e l a t i o n a l designs often

increasing in frequency.

are

longitudinal

designs

appear

to

be

Quasi-Experimental

and Correlational Designs

|

As with quasi-experiments, the best way

The Analysis of Data From Correlational Designs

to reduce causal ambiguity in correlational analyses

A third variable is a variable that is corre-

is

with

design

elaborations.

Especially likely to be useful

are

hybrid

lated with the cause and influences the out-

designs that include nonequivalent dependent

come variable. For example, the variable o f

measures and the addition o f discrete compar-

financial resources is likely to be a third vari-

ison groups in which the treatment is expected

able in a correlational analysis o f the effects

to have different effects. Adding waves o f data

of stress on physical health, because financial

over time is also recommended

resources are likely to be related to stress and

among other advantages, multiple waves of

because,

to have an effect on physical health. As previ-

data enable researchers to use latent growth

ously noted, omitting third variables can bias

curve models with time-varying covariates

the analysis o f data from correlational designs

(e.g., Muthen, 1 9 9 1 ) , allowing comparisons to

in the same way that selection differences can

be drawn across times as well as across people,

introduce bias in quasi-experiments.

which is another useful

Structural equation modeling ( S E M ) is the

design-and-analysis

elaboration.

most widely used method to control for the effects

o f third

variables in correlational

designs (Hoyle & Robinson, Chapter 1 0 , this

B E Y O N D INDIVIDUAL STUDIES:

volume). T o control for the effects o f third

RESEARCH PROGRAMS,

variables, they must be measured and included

LINES O F RESEARCH, A N D

in the structural equation model. In the typical

RESEARCH SYNTHESES

correlational design, potential third variables are rampant, and it is exceptionally difficult to

O n e o f the c o m m o n shortcomings o f method-

be confident that they have all been measured

ological treatises is that they often focus only

and included in the analysis. As a result, it is

on individual studies, seemingly implying that

usually not possible to be confident that the

individual studies are the prime or only unit

effect o f the cause o f interest has been esti-

o f scientific progress. T o the contrary, most

mated

researchers realize that although

without

bias, perhaps

severe bias.

individual

Moreover, even if all third variables are iden-

studies are the building blocks, real progress

tified, they can at best be measured only falli-

in science usually comes from

bly. Measurement error in a third variable

evidence across multiple studies. Thus, our

converging

biases the treatment effect estimate. T o deal

discussion

with bias that results from measurement error,

correlational methods should be viewed in

latent variables are used in the structural equa-

this broader context o f the field.

of

quasi-experimental

and

tion model. Latent variable models require

O u r sense is that, in social psychology, it is

multiple measures o f each third variable and,

rare for a causal question t o be addressed only

in essence, use factor analysis to separate the

with

underlying construct o f interest from measure-

methods. Instead, quasi-experimental and cor-

quasi-experimental

or

correlational

ment error and other irrelevancies. If the cor-

relational methods are likely t o be used in con-

rect factor model has been specified, S E M

junction with experimental methods, either in

removes the bias resulting from measurement

a single investigator's research program or

error by controlling for the underlying con-

across investigators in a research area. In some

struct rather than for the fallible measures.

cases, quasi-experimental

However, the more serious problem o f omitted

methods serve as the forerunners o f experi-

third variables remains.

mental methods. F o r instance, correlational or

or correlational

DESIGN AND ANALYSIS

282

quasi-experimental findings may foster the

is,

development o f hypotheses grounded in real

Nevertheless, the retrospective and

the

regression-discontinuity

design.

world observations. In other cases, experi-

experimental nature o f the study left some

quasi-

mental studies come first but leave open ques-

ambiguity about whether the findings were

tions of construct and external validity ( C o o k

the

&

hypothesized.

Campbell, 1 9 7 9 ; Shadish e t a l . ,

2002).

result

o f self-protective motives,

as

F o r example, it is possible

Construct validity is concerned with the con-

(even if unlikely) that the results were instead

ceptual interpretation o f the treatment

and

due to differences in pre-layoff exposure to

outcome variables, that is, with the accuracy

information that coincided with the cutoff. A

of labeling, in abstract, conceptual terms, the

randomized experiment could further reduce

specific, concrete research operations

uncertainty about the causal relationship.

that

constitute the treatment and the outcome.

In fact, Louie and her colleagues (Louie,

External validity refers to the accuracy o f

1 9 9 9 ; Louie, Curren, & Harich, 2 0 0 0 ) and

inferences about the degree to which a finding

M a r k , Reiter, Eyssell, Cohen, and Mellor (in

about a causal relationship can be generali-

press) have replicated M a r k and Mellor's

zed to persons, settings, and times other than

( 1 9 9 1 ) quasi-experimental study with con-

those observed in a specific study. Questions

trolled experiments, also adding comparisons

about construct and external validity in experi-

that clarify the causal process. For example,

mental research are sometimes best addressed

Louie ( 1 9 9 9 ) replicated the finding that there

in real world settings that may not readily

is no hindsight bias among those w h o experi-

allow r a n d o m

ence a negative self-relevant outcome. Louie

assignment

to

conditions.

Iteration between experimental methods, on

also examined the attributions that people

one hand, and quasi-experimental or correla-

made, and the pattern o f attributions was

tional methods,

often

consistent with the operation o f self-serving

enhance construct and external validity. Such

on the other, will

processes. In general, measurement o f poten-

iteration has in fact taken place in several o f

tial process variables will be easier in a lab

the examples presented in this chapter.

experiment, such as those carried out by

As described in the section on the regression-

Louie and her colleagues, than in a quasi-

discontinuity design, M a r k and M e l l o r ( 1 9 9 1 )

experiment in a field setting. Louie ( 1 9 9 9 )

took advantage o f a company's w o r k layoff

also showed that a manipulation intended to

to examine whether a highly self-relevant,

constrain self-serving biases eliminated hind-

negative outcome reduces the magnitude o f

sight bias among those w h o experienced a

hindsight bias. M a r k and M e l l o r were drawn

positive self-relevant outcome. These experi-

to the layoff as a research site in part because

mental replications, with their use o f proce-

it allowed them to examine hindsight in the

dures to clarify underlying causal processes,

c o n t e x t o f an event that

are important both for increasing confidence

important

was

extremely

for the respondents. This, they

that

the effect

Mark

and

Mellor

(1991)

believed, would allow them to examine the

observed was causal and for demonstrating

impact o f self-protective mechanisms

that it was the result o f self-serving biases.

that

had not been addressed adequately in the large experimental literature on

hindsight

bias, which in general had relied on judg-

In similar fashion, the Sinclair et al. ( 1 9 9 4 ) study,

which

used

a

quasi-experimental

manipulation o f m o o d , was conducted in the

ments about relatively trivial events. In addi-

context o f an ongoing line o f research, to

tion, because the workers were laid off based

which many investigators have contributed,

on seniority, M a r k and M e l l o r were able to

on the effects o f m o o d and the processes that

use a strong quasi-experimental design, that

underlie m o o d effects. In many studies, m o o d

Quasi-Experimental

and Correlational Designs

has been experimentally manipulated, using a

can be used in settings or with populations

variety o f procedures such as exposing people

where randomized

to film clips, asking them to recall happy or

infeasible or impractical. They also can allow

sad memories, having them listen to different

social psychologists t o investigate the effects

styles o f music, having them work in a pleas-

o f interventions that are both more intense

ant or unpleasant r o o m , or giving them posi-

and more unobtrusive than would be possi-

experiments are either

tive or negative feedback. Sinclair and his

ble in randomized experiments. W e believe,

colleagues could easily have used such a pro-

however,

cedure to randomly assign people to m o o d

seen as a potential complement to, rather

conditions. However, several considerations

than a substitute for, experimental methods.

that

quasi-experiments

are

best

led them to use a hybrid (quasi-experimental and experimental) design. First, most research on the effects o f transitory m o o d states used some form o f experimental mood induction, and researchers should attempt to add "mul-

CONCLUSIONS In closing, we offer a few overall conclusions.

tiplism" on those aspects o f research that are most homogenous

(Cook,

1985).

Second,

also for reasons o f multiplism, there are benefits to having at least some studies in a line of research take place outside laboratory settings. In part, the benefit is that nonlaboratory settings tend to avoid, or at least reduce, the possibility that findings are due to artifacts such as demand characteristics. In addition, a naturalistic manipulation

of mood

enhances construct validity, by helping to ensure that the m o o d effects in the literature can be obtained with everyday determinants of mood, and not just as a result o f the par-

Design Matters Both correlational and the weaker quasiexperimental designs will in general be susceptible to threats to internal validity that preclude confident causal inference. But even weak designs may be valuable for hypothesis generation and when causal inference is not a key research goal. And stronger quasi-experimental designs, such as the regression-discontinuity and interrupted time-series designs, often can provide credible causal conclusions.

ticular kind (or intensity) o f mood induction used in the laboratory. Third, because o f the experimental parts o f the hybrid

design,

Sinclair and his colleagues were optimistic that interpretable results would arise even if the mood manipulation were quasi-experimental. Fourth, Schwarz and Clore ( 1 9 8 3 ) had shown that weather could serve effectively as a naturalistic m o o d induction.

But Design Is Not Everything As illustrated

by Steiner and

Mark's

( 1 9 8 5 ) consultation with b a n k officials and others about changes in record keeping and sources o f history effects,

supplementary

information can help in assessing the plausibility o f threats t o validity. In addition, just because a threat is generally plausible for a

In short, we do not intend this chapter to

given design does not mean that it will

serve as an unabashed advocacy o f the use o f

always be plausible for that

quasi-experimental and correlational meth-

example, maturation might not be a plausi-

design. For

ods in social psychology. Such methods, if

ble explanation for changes in attitudes dur-

used judiciously, bring advantages relative to

ing a 10-minute experiment. Often, existing

the traditional experimental methods

evidence and c o m m o n sense can be helpful

that

appear t o predominate in mainstream social

in assessing whether a threat t o validity is

psychology. Quasi-experiments

plausible.

sometimes

283

284

DESIGN AND ANALYSIS design can yield credible results, and in some

The Pattern of Observed Effects Also Matters In the absence o f random

c i r c u m s t a n c e s , the assignment,

causal inference often can be enhanced byadding comparisons that produce a c o m p l e x pattern o f results, thereby making alternative explanations less plausible. This is the case for the Sinclair et al. ( 1 9 9 4 ) study, where it is

implementation

of a

strong design can be severely compromised. But in all cases, causal inference is fallible. Research reports should explicitly acknowledge the plausible threats to validity that are present and critically discuss h o w well these threats have been taken into account.

difficult to imagine a threat to internal validity that could explain the predicted pattern. As an added benefit, predicting and observing a c o m p l e x pattern will often help clarify the mediating process, which also adds confidence to inferences about cause and effect.

Finally, Social Psychologists Should Not Be One-Trick Ponies T h e randomized experiment is a wonderful method, generally considered—for good reason—the gold standard in causal research. At the same time, all methods have limitations. Quasi-experimental

Recognizing and Reporting Threats to Validity Matters Too

methods,

especially

stronger quasi-experimental designs, can bring important

contributions to "critical multi-

T h e credibility o f causal inference depends

plism" (Cook, 1 9 8 5 ) , especially when consid-

on both the research design and the analysis

ered in the context o f programs o f research

of data. In some circumstances, a

that also include experimental methods.

weak

REFERENCES Campbell, D. T. (1966). Pattern matching as an essential in distal knowing. In K. R. Hammond (Ed.), The psychology of Egon Brunswik (pp. 81-106). New York: Holt, Rinehart & Winston. and quasi-experimental Campbell, D. T., &c Stanley, J . C. (1966). Experimental designs for research. Skokie, IL: Rand McNally. Cappelleri, J . C , Darlington, R. B., &C Trochim, W. M. K. (1994). Power analysis of cutoff-based randomized clinical trials. Evaluation Review, 18, 141-152. Cook, T. D. (1985). Postpositivist critical multiplism. In R. L. Shotland & M. M . Mark (Eds.), Social science and social policy (pp. 2 1 - 6 2 ) . Beverly Hills, CA: Sage. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Skokie, IL: Rand McNally. Eckert, W. A. (2000). Situational enhancement of design validity: The case of training evaluation at the World Bank Institute. American Journal of Evaluation, 21, 185-193. Forgas, J . P., & Moylan, S. (1987). After the movies: The effects of transient Bulletin, mood states on social judgments. Personality and Social Psychology 13, 4 7 8 - 4 8 9 . Gollob, H. F., & Reichardt, C. S. (1987). Taking account of time lags in causal [Special section on structural equation modeling], models. Child Development 58, 80-92.

Quasi-Experimental

and Correlational

Gollob, H. F., &C Reichardt, C. S. (1991). Interpreting and estimating indirect effects assuming time lags really matter. In L. M. Collins 8c J . L. Horn (Eds.), Best meth­ ods for the analysis of change: Recent advances, unanswered questions, future directions (pp. 243-259). Washington, DC: American Psychological Association. House, E. R. (2001). Unfinished business: Causes and values. American Journal of Evaluation, 2 2 ( 3 ) , 3 0 9 - 3 1 5 . Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988). The effects of graduate training on reasoning: Formal discipline and thinking about everyday-life events. American Psychologist, 43, 4 3 1 - 4 4 2 . Lipsey, M. W., &c Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209. Louie, T. A. (1999). Decision makers' hindsight bias after receiving favorable and unfavorable feedback. Journal of Applied Psychology, 84, 2 9 - 4 1 . Louie, T. A., Curren, M . T., 8c Harich, K. R. (2000). "I knew we would win": Hindsight bias for favorable and unfavorable team decision outcomes. Journal of Applied Psychology, 85, 2 6 4 - 2 7 2 . Mark, M . M. (1986). Validity typologies and the logic and practice of quasiexperimentation. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis (pp. 4 7 - 6 6 ) . San Francisco: Jossey-Bass. Mark, M. M. (1990). From program theory to tests of program theory. In L. Bickman (Ed.), New directions for program evaluation: Program theory in pro­ gram evaluation (pp. 37-51). San Francisco: Jossey-Bass. Mark, M . M., &c Mellor, S. (1991). The effect of the self-relevance of an event on hindsight bias: The foreseeability of a layoff. Journal of Applied Psychology, 76, 569-577. Mark, M . M., Reichardt, C. S., &c Sanna, L. J . (2000). Time series designs and analyses. In H. E. A. Tinsley & S. R. Brown (Eds.), Handbook of applied mul­ tivariate statistics and mathematical modeling (pp. 3 5 3 - 3 8 9 ) . New York: Academic Press. Mark, M . M., Reiter, R., Eyssell, K. M., Cohen, L., & Mellor, S. (in press). "I couldn't have seen it coming": The impact of negative self-relevant outcomes on retrospections about foreseeability. Memory. Marsh, J . C. (1985). Obstacles and opportunities in the use of research on rape legislation. In R. L. Shotland 8t M. M. Mark (Eds.), Social science and social policy (pp. 2 9 5 - 3 1 0 ) . Beverly Hills, CA: Sage. Mellor, S., &c Mark, M . M. (1998). A quasi-experimental design for studies on the impact of administrative decisions: Applications and extensions of the regression-discontinuity design. Organizational Research Methods, 1, 3 1 5 - 3 3 3 . Muthen, B. (1991). Analysis of longitudinal data using latent variable models with varying parameters. In L. M . Collins & J . L. Horn (Eds.), Best methods for the analysis of change: Recent advances, unanswered questions, future directions (pp. 1-17). Washington, DC: American Psychological Association. Reichardt, C. S. (1979). The statistical analysis of data from nonequivalent group designs. In T. D. Cook &c D. T. Campbell, Quasi-experimentation: Design and analysis issues for field settings (pp. 147-205). Skokie, IL: Rand McNally. Reichardt, C. S. (2000). A typology of strategies for ruling out threats to validity. In L. Bickman (Ed.), Research design: Donald Campbell's legacy (Vol. 2, pp. 89-115). Thousand Oaks, CA: Sage. Reichatdt, C. S. ( 2 0 0 2 ) . The priority of just-identified, recursive models. Psychological Methods, 7, 3 0 7 - 3 1 5 . Reichardt, C. S., & Mark, M . M. (1998). Quasi-experimentation. In L. Bickman & D. Rog (Eds.), Handbook of applied social research (pp. 193-228). Newbury Park, CA: Sage.

Designs

285

DESIGN AND ANALYSIS Reichardt, C. S., 8c Mark, M . M . (2001). Nonequivalent group designs. In N. Smelser 8c P. Baltes (Eds.), International encyclopedia of the social and behavioral sciences (pp. 1 0 6 5 5 - 1 0 6 6 0 ) . Oxford, UK: Elsevier. Schwarz, N., 8c Clore, G. L. (1983). Mood, misattribution, and judgments of well-being: Informative and directive functions of affective states. Journal of Personality and Social Psychology, 45, 5 1 3 - 5 2 3 . Schwarz, N., 8c Oyserman, D. (2001). Asking questions about behavior: Cognition, communication and questionnaire construction. American Journal of Evaluation, 22, 127-160. Shadish, W. R., Cook, T. D., 8c Campbell, D. T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Boston: Houghton Mifflin. Signorella, M. L., Frieze, I. H., 8c Hershey, S. W. (1996). Single-sex versus mixedsex classes and gender schemata in children and adolescents: A longitudinal comparison. Psychology of Women Quarterly, 20, 599-607. Sinclair, R. L., Mark, M. M., 8c Clore, G. L. (1994). Mood-related persuasion depends on (mis)attributions. Social Cognition, 12, 309-326. Steiner, D., 8c Mark, M. M. (1985). The impact of a community action group: An illustration of the potential of time series analysis for the study of community groups. American Journal of Community Psychology, 13, 13-30. Trochim, W. M . K. (1984). Research design for program evaluation: The regres­ sion-discontinuity approach. Newbury Park, CA: Sage. Trochim, W. M . K. (1985). Pattern matching, validity, and conceptualization in program evaluation. Evaluation Review, 9, 575-604. Winship, C , 8c Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659-707.

CHAPTER

13

Within-Subject and Longitudinal Experiments Design and Analysis Issues STEPHEN G . WEST Arizona State University JEREMY C. BIESANZ University of

Wisconsin-Madison

OI-MAN KWOK Arizona State University

I

n this chapter, we consider within-subject

M R I measures o f cortical activity), there are

and

measure)

very large individual differences on the out-

experimental designs and their use in

come variable and a subtle effect that requires

longitudinal

(repeated

emerging

a high power test is sought, or a special pop-

questions in social psychology. In the within-

ulation o f limited size is o f focal interest (e.g.,

subject design, each participant receives two

decision making by accountants or physi-

or more treatments. Following each treat-

cians). Within-subject designs assume that

ment, the outcome variable is assessed. Such

each treatment has only a short-term effect,

answering

both traditional

and

designs are particularly useful in research

so that treatments delivered later in the experi-

contexts in social psychology in which the

ment are not affected by earlier treatments. In

outcome measure is expensive to collect (e.g.,

contrast, in the longitudinal

experimental

AUTHORS' NOTE: We thank Leona Aiken, Patrick Curran, and the editors for their comments on an earlier version of this chapter. Correspondence should be directed to Stephen G. West, Psychology Department, Arizona State University, Tempe, AZ 8 5 2 8 7 - 1 1 0 4 or via e-mail to [email protected].

287

288

DESIGN AND ANALYSIS design, each participant

is given a single

we identified 2 5 ( 8 % ) studies that

JPSP,

treatment and then measured on the outcome

included a pretest and posttest, another 3 2

variable at multiple time points. Here the

(11%)

focus is on the longer-term changes in the

waves, and another 5 ( 2 % ) using both a

using three or more

measurement

treatment effect and the processes related to

pretest and two or more subsequent waves

the development o f an outcome over time.

of

Although important examples o f within-

measurement.

subject designs have been published in such

ple measurement

diverse

the

areas

of

social

psychology

as

O f note,

the

majority

( 6 8 % ) o f the 3 7 total studies involving multiwaves were reported

"Personality Processes and

in

Individual

expectancy effects, person perception, and atti-

Differences" section. In comparison to earlier

tude change, such designs traditionally have

findings by West, Newsom, and Fenaughty

not been commonly used in social psychology.

( 1 9 9 2 ) , these findings seem to indicate that a

A decade ago, Keren ( 1 9 9 3 ) observed that dif-

small but increased interest in longitudinal

ferent substantive areas of psychology varied

studies is occurring, primarily among person-

greatly in their utilization of between- versus

ality researchers and some basic social psy-

within-subject designs. Areas such as psy-

chologists

chophysics and perception strongly empha-

research. Few social psychologists are taking

conducting

non-experimental

areas

advantage o f the power o f longitudinal experi-

such as personality and social psychology

ments with repeated measurements to study

strongly emphasized between-subject designs.

processes over time.

sized within-subject designs, whereas

W e surveyed the two sections o f the 2 0 0 0 and 2 0 0 1 volumes o f the Journal

of

1

Because o f space constraints, our focus in

Personality

this chapter will be limited to experimental

and Social Psychology (/PSP; 3 0 3 articles) and

contexts with quantitative outcome variables

the Journal of Experimental

in either basic laboratory or applied settings,

Social

Psychology

(JESP; 8 0 articles) for studies involving within-

conditions that characterize most social psy-

subject designs. Consistent with Keren's obser-

chological research (West, N e w s o m , et al.,

vation, we were able to locate very few studies

1 9 9 2 ) . A companion piece (Biesanz, West, &

(< 2 % ) in which a central theoretical variable

K w o k , in press) considers approaches to lon-

of interest was manipulated within subjects.

gitudinal research in naturalistic settings that

Some additional

more often typify personality research.

studies included

within-

subject design features, but these typically were controls for incidental factors such as order of presentation of multiple exemplars of stimuli or multiple dependent variables. Although social psychology's emphasis on

between-

subject designs is often appropriate, withinsubject designs offer

advantages

in some

research areas.

PERSPECTIVES O N CAUSAL INFERENCE T h e increased prominence o f time in withinsubject and longitudinal designs raises both new issues and new opportunities. T o aid our

M a j o r advances have occurred in longitu-

understanding, it will be helpful to draw on

dinal research that permit improved studies o f

the perspectives

change over time. Personality and social psy-

Holland, 1 9 8 6 ; Rubin, 1 9 7 8 ) and Donald

chologists have primarily used

Campbell (e.g., Campbell & Stanley, 1 9 6 6 ;

traditional

of Donald

Rubin

(e.g.,

approaches and are just beginning to make

Shadish, C o o k , & Campbell, 2 0 0 2 ) and their

limited use o f exciting new design and analy-

colleagues on

sis techniques in this area. In the 3 0 3 articles

M a r k & Reichardt, Chapter 1 2 , this volume).

published in the 2 0 0 0 and 2 0 0 1 volumes o f

Our presentation begins with the simple case

causal inference

(see also

Within-Subject and Longitudinal Experiments

\

in which there are two treatment conditions, a

which t o consider three design elements that

treatment condition T and a control condition

we introduce below.

C. T h e outcome variable is measured

follow-

ing the treatment. For example, in classic research on aggression, T might be a frustra-

T H R E E DESIGN ELEMENTS

tion (e.g., the individual participant * is prevented

from

solving

a

problem

by

a

confederate), C might be n o frustration (i is allowed to solve the problem), and the outcome variable Y is a measure o f aggression, such as the intensity o f electric shocks ostensibly delivered to the confederate. Keep in mind that both the T and C conditions may have potential positive or negative effects on the outcome. For example, placebo conditions often lead to effects on outcome variables in well-controlled pharmaceutical experiments. Rubin's causal model (see Holland, 1 9 8 6 ) begins with a consideration o f the ideal case under

which

a

causal

effect

could

be

observed. Suppose we have a single participant i w h o will receive the treatment. T h e causal

is defined

effect

as the

difference

between the participant's outcome, Y in the T

treatment

condition and

the

participant's

outcome, Y , in the control condition conc

ducted

under identical circumstances. In

symbols, the causal effect is Y - Y . N o w , in T

c

practice, this ideal case can never be achieved because "identical circumstances" can never be achieved. F o r example, in our experiment on aggression, achieving identical circumstances would require that the researcher give the treatment (the frustrating

experi-

ence) to participant / at the beginning o f the experiment and then measure the partici-

Basic Within-Subject

Design

Both the treatment and control conditions may be given to the participant. As one example, participant

/ would

initially be

exposed t o treatment condition T and the outcome would be measured. Following this, the participant / would be exposed t o control condition C and the outcome would be measured. T h i s within-subject design gives us the causal effect as Y - Y provided that two T

strong

assumptions

assumption, temporal

c

are

met.

stability,

The

first

means that

the effect o f the treatment c a n n o t depend on time—the magnitude o f the treatment effect must be identical regardless o f when the T and C conditions are delivered. Campbell and Stanley ( 1 9 6 6 ) provided a practical list o f c o m m o n threats t o this assumption; the list is presented in T a b l e 1 3 . 1 (section A ) . T h e second assumption, causal

transience,

means that the effects o f each treatment do not persist over time. Otherwise stated, the effect o f each treatment (e.g., the frustration) is expected t o be only temporary, with the participant's level on the o u t c o m e o f interest as well as levels o f other motivational, emotional, and cognitive states and physiological states returning to the baseline levels prior to the introduction o f the second treatment.

pant's level o f aggression at the end o f the experiment, following which the researcher would return the participant to the identical

Unit Homogeneity

m o m e n t at the beginning o f the experiment

Alternatively, researchers can assume that

and n o w give the same participant i the con-

the participants (experimental units) are iden-

trol condition (no frustration) and then mea-

tical. T h e causal effect is n o w Y

sure the participant's level o f aggression at

where the number in parentheses indicates

the end o f the experiment. Despite the impos-

the participant (unit) that receives each treat-

sibility o f its ever being achieved, the ideal

ment condition. T h i s approach is often viable

case serves as a useful reference point from

in experiments in engineering; however, in

T ( 1 )

- Y

C ( 2 )

,

289

290

|

DESIGN AND ANALYSIS

Table 13.1

Some Threats to Internal Validity in Within-Subject and Longitudinal Designs

A. Threats associated with temporal stability (after Campbell & Stanley, 1966) 1. History. Specific events may occur between measurements that affect the outcome. Attitudes toward raising tuition may change following a tuition increase proposal by college regents. 2. Maturation. Within-participant processes unrelated to the treatment may affect the outcome. Children's cognitive abilities increase as they age, and students' mood states are more positive on weekends. 3. Testing. Scores on subsequent measures may be affected by prior measurement. Reporting depressed mood on a first measurement may increase the likelihood of reporting depressed mood on the next measurement. 4. Instrumentation. Changes in the capabilities of the measuring instrument may occur across measurements. Aggressive acts at age 4 (e.g., biting) are different from aggressive acts at age 16 (e.g., fighting with weapons). Measures may have ceiling or floor effects such that change cannot be detected. 5. Statistical Regression. Participants selected on the basis of the extreme high or low scores at Time 1 may score closer to the mean at Time 2 even in the absence of treatment. A group of students selected because they have 1 day of illness Year 1 will have on average more than 1 day of illness Year 2. B. Violations of the stable unit treatment value assumption (SUTVA) (after Cook & Campbell, 1979; West, Biesanz, & Pitts, 2 0 0 0 ) 6. Atypical Reactions of Participants. Participants aware of the nature of other experimental conditions may change their response on the outcome variable as a result of this knowledge. Patients in medical research sometimes have given up when they learned or inferred that they were assigned to the control group. 7. Treatment Diffusion or Compensatory Treatment by Treatment Providers. In applied research, control group participants may receive treatment from another source (e.g., medication from black market) or treatment staff may provide compensatory services (e.g., extra nursing care).

research with humans, it is rare that partici-

background variables, whether measured or

pants can be assumed to be identical. Even

unmeasured. Consequently, following treat-

monozygotic (identical) twins may differ in

ment, the difference between the mean o f the

their life experiences, attitudes, relationship

treatment group and the mean o f the control

status, current m o o d state, and so on.

group on the outcome o f interest, M T

M, c

will be an unbiased estimate o f the magni-

Randomization

tude o f the causal effect. Randomization shifts the focus from com-

T h e key feature o f randomization is that

parisons involving a single participant to com-

assignment to treatment conditions is gener-

parisons involving a sample of participants.

ated by a random process such as flipping a

W e can no longer make causal statements

2

coin, so that each participant has an identical

about any individual participant. W e can only

chance o f being assigned to the T or C group.

compare the average outcome for the partici-

T h e important result o f this process is that at

pants receiving the treatment with the average

the beginning o f the experiment, prior t o the

outcome for the participants receiving the con-

receipt o f any treatment, the mean scores

trol condition. Second, the causal effect is now

of the participants in the T and C groups

an unbiased estimate o f the true causal effect.

will be approximately equal on all possible

In any particular replication o f the experiment,

Within-Subject and Longitudinal Experiments

\

this estimate is likely to be too high or too low.

This

T h e researcher can know only that the estimate

treatment levels allows presentation o f the

more

applied

example

with

four

of the causal effect would on average be

full range o f approaches to controlling for

correct in the hypothetical case o f the same

order effects. These approaches c a n n o t be

experiment being exactly replicated a very

distinguished with only t w o treatments.

large number o f times. Third, randomization

Professional t a x accountants were pre-

does not provide assurance that participants

sented with a series o f descriptions o f t a x

are equated at the beginning of the experiment,

cases and were asked about the likelihood

even on average, if some participants are not

that they would take a "shady" but not illegal

present for the measurement o f the outcome

deduction

variable. Participant loss (attrition) from the

measured on a scale from 0 % to 1 0 0 % . In

experiment potentially can seriously under-

t w o cases, there was a high probability that

mine the strength of the causal inferences.

taking the deduction would lead to a govern-

In research with human participants, the

for each case. Likelihood was

ment audit, whereas in the other two cases

researcher must also assume that neither the

this probability was low (audit probability).

randomization process n o r the participant's

In t w o cases, the taxpayer preferred t o take

(or others') awareness o f other participants'

audit risks, whereas in the other two cases

treatment

the taxpayer preferred t o avoid government

conditions affect

the

outcome.

This assumption is k n o w n as the stable unit

audits (risk preference). T h u s , the design was

treatment value assumption, or S U T V A . In

a 2 (Audit Probability) x 2 (Risk Preference)

the typical 1-hour social psychological experi-

factorial:

ment conducted by trained researchers with

T 2 = audit low; risk high; T 3 = audit high,

T l = audit

low,

risk

low;

individual participants w h o do not leave the

risk low; T 4 = audit high, risk high.

laboratory, current evidence suggests that S U T V A does not appear t o be a major problem. However, in basic and applied experiments that take place across days, weeks, or

Approaches to Controlling for Order of Treatment Conditions

years, the possibility that subjects in different treatment conditions affect each other becomes far

more

Random

Ordering

likely. T a b l e 1 3 . 1 , section B ,

O n e approach to controlling for order

presents some illustrations (see also W e s t ,

effects is t o generate several different random

Biesanz, et al., 2 0 0 0 ) .

orders o f the four

treatment

conditions.

Participants are then randomly assigned t o one o f the orders. W i t h four treatment conditions, there are 4 ! (4 factorial) or 2 4 differ-

WITHIN-SUBJECT EXPERIMENTS

ent unique orders in which the treatments Within-subject experiments in practice typi-

might be presented. Ideally, the use o f all

cally c o m b i n e the design elements o f ran-

possible orders with multiple replications o f

domization with within-subject design

or

each order both balances potential artifac-

unit homogeneity. T h i s combination helps

tual effects a n d provides some ability to test

minimize the possibility o f potential viola-

for order effects. N o r m a l l y in practice, how-

tions o f the assumptions o f temporal stability

ever, the researcher can implement only a

and causal transience. As a means o f illus-

small random subset o f the possible orders.

trating the ideas in this section, we will use

F o r e x a m p l e , using a t a b l e o f

an example patterned after an experiment by

numbers, we generated the following four

Kaplan, Reckers, W e s t , and Boyd

orders o f treatment conditions.

(1988).

random

292

DESIGN AND ANALYSIS

Measurement 2

1 Order Order Order Order

1 2 3 4

Occasion

T4 T3 T3 Tl

Tl Tl T2 T2

(Trial)

3

4

T2 T2 T4 T3

T3 T4 Tl T4

treatments on the second trial ( T l and T 2 — low audit), but not for treatments on the first trial ( T 3 and T 4 — h i g h audit) and

would

lead to a possible bias in the estimate o f the causal effect.

Latin Square

Designs

W h e n it can be implemented, the Latin If the magnitude

o f the causal

effect

square

design

often

will be the

In Latin square

optimal

changes over time (violation o f temporal sta-

approach.

bility), then the estimate o f the causal effect

treatment condition appears once on each

designs, each

may be biased because not all the treatment

trial across orders (rows) and once in each

conditions appear equally on each trial—the

order across trials (columns). F o r our t a x

design is not balanced. For example, T 3

deduction example with four treatments, one

appears twice on Trial 1 and never on Trial 2 .

possible Latin square is shown below.

Consequently, unless participants can be randomly assigned to all possible random orders, this approach does not necessarily eliminate

Measurement

Occasion

(Trial)

bias in any particular experiment.

1

2

3

4

T4 T3 T2 Tl

T2 Tl T3 T4

Tl T2 T4 T3

T3 T4 Tl T2

Counterbalancing Counterbalancing is a very simple approach

Order Order Order Order

1 2 3 4

to achieving a more balanced design. In counterbalancing, one random order o f presenta-

This

balance o f treatment

conditions

tion o f the treatments is generated, and then a

means that most forms o f violation o f the

second reversed order o f presentation is then

temporal

created, as illustrated below.

assumptions will have minimal effects on the

stability and

causal

transience

estimate o f the causal effect. F o r example, if there is a "practice effect" such that any Measurement

Occasion

(Trial)

treatment will have a larger effect on Trial 2 than Trial 1, then this effect is constant

1

2

3

4

T4 T3

Tl T2

T2 Tl

T3 T4

estimate o f the causal effect o f treatment will

In some contexts, counterbalancing may

produces a larger effect on Trial 2 than on

be a sufficient method o f controlling for tem-

Trial 1, the estimate o f the causal effect will

Order 1 Order 2

across the four treatment conditions and the not be biased. O n the other hand, if there is a differential practice effect such that only T l

poral changes in the magnitude o f the causal

be biased. Latin square designs provide an

effect. O n the other hand, counterbalancing

excellent method o f addressing temporal sta-

does not lead to a fully balanced design in

bility but cannot rule out all forms o f viola-

which each treatment condition appears on

tions o f causal transience. If it is believed that

each trial. F o r example, if there were a large

the effect o f a treatment may carry over to

increase in the magnitude o f the causal effect

the following trial, the researcher

from Trial 1 to Trial 2 (e.g., a " w a r m - u p "

extend the time between trials or introduce

effect),

intervening activities between trials.

this effect

would

be present

for

should

Within-Subject Table 13.2

and Longitudinal

Experiments

j

Analysis of Variance Source Table for Latin Square Design

Source

df

Between subjects Order Error between Within subjects Treatment Audit Probability Risk Audit x Risk Trials Latin Square residual Error within

19 3 16 60 3 1 1 1 3 6 48

MS 6.50 4.45 8.53 12.00 4.54 9.05 1.02 0.39 1.18

F

P

1.43

ns

10.17 3.85 7.67 0.86 0.33

.0025 .06 .008 ns ns

NOTE: The overall treatment effect has 3 degrees of freedom (df) and is partitioned into an audit probability main effect, a risk main effect, and an Audit Probability x Risk interaction, with 1 df each. The rows containing these effects are offset.

Latin square designs also permit researchers

square residual effect provide two separate

to perform partial tests of the assumptions. T o

partial tests o f violations o f the assumptions

illustrate, we simulated a data set for our tax

o f temporal stability and causal transience.

deduction example in which five participants

T h e between-subjects order effect also reflects

(total n = 2 0 ) were randomly assigned to each

any failure o f randomization

of the four orders o f presentation in the Latin

equate the subjects in the four groups that

square above. The results o f the illustrative

receive different orders o f the treatment con-

analysis are presented in Table 1 3 . 2 .

to precisely

ditions. As a within-subjects effect, the test o f

T h e substantive effects o f interest are the

the Latin square residual or (AB) (see Winer,

audit probability main effect, the risk main

Brown, & Michels, 1 9 9 1 , pp. 7 0 3 - 7 0 4 ) will

effect, and the Audit Probability x Risk inter-

nearly always provide a far more stringent

action. T h e statistically significant

test o f the assumptions o f temporal stability

Audit

Probability x Risk interaction suggests that

and

causal transience. T h e

nonsignificant

the taxpayer's risk preference and the risk o f

effects indicate that violations o f the assump-

audit combine to determine the likelihood o f

tions were not detected in our illustration.

taking a shady t a x deduction. In the present

Cochran and C o x ( 1 9 5 7 ) and Winer et al.

illustration, the interaction is a synergistic

( 1 9 9 1 ) provide more complete presentations.

one, such that the combination o f a low probability o f being audited and a taxpayer willing to take an audit risk produces a substantially higher likelihood o f taking shady M

T 3

deduction

= 35, M

T 4

(M

T 1

=

64,

M

T 2

the

= 39,

= 39).

In addition to the substantive effects o f

Randomized Matched Designs (Predictor Sort Designs) Thus far, we have assumed that the same participant receives each treatment condition. Another w a y o f constructing

a

"within-

interest, checks on the assumptions o f the

subject" design is to use the design element of unit

design are provided. T h e nonsignificant trials

homogeneity. In the approach using random-

effect indicates that the likelihood o f taking

ized matched designs (also known as predictor

the deduction does not change over trials. T h e

sort designs), pairs, triples, or quadruples o f

between-subjects order effect and the Latin

participants w h o have similar characteristics

293

294

DESIGN AND ANALYSIS are identified and matched. T h e matching may

effect will be constant for all participants. In

be performed on the basis o f a single pretest

other cases, researchers may hypothesize that

variable, or it may be performed

the magnitude o f the treatment effect will

using a

composite o f several variables. Randomization

vary depending on the participant's level on

then takes place within each matched set. T o

a background variable—a moderator

illustrate, let us return to our example. Suppose

or

that accountants' fiscal conservatism has a

interaction

Treatment

x

Background

(see also Hoyle &

effect

Variable Robinson,

strong association with their propensity to rec-

Chapter 1 0 , this volume). Background vari-

ommend shady deductions. From a large pool

ables may include continuous variables such

of potential participants, we can identify sev-

as personality traits (e.g., neuroticism), atti-

eral sets o f four participants (blocks) who have

tudes, or aptitudes (e.g., I Q ) , or categorical

similar scores on a pretest measure of fiscal

variables such as gender or attachment style.

conservatism

In m o d e r a t o r

and

then

randomly

assign

the participants within each block to one o f the

designs,

a

analyses

strong

in

within-subject

(usually

reasonable)

is made that the

participant's

four treatment conditions. T o illustrate, the

assumption

four participants with the highest scores on fis-

level on the background variable does not

cal conservatism are identified as block 1, the

change during the experiment.

next four highest are identified as block 2 , and so on. Within each block, a participant is ran-

Consider a simple two-condition

within-

subjects design in which each participant is

domly assigned to each of the four treatment

asked to evaluate an equally qualified in-

conditions. In the statistical analysis, each

group and out-group member on the outcome

block o f participants is treated as if it were a

variable of suitability for employment. The

single participant measured under each o f the

researcher has also collected a

four treatment conditions, but because differ-

measure o f each participant's level of preju-

continuous

ent participants receive each treatment, this

dice prior to the experiment. T h e experi-

design rules out potential threats of temporal

menter hypothesizes that there will be no

stability and causal transience. T o the extent

relationship between level o f prejudice and the

that the participants within each block are

evaluation for the in-group member, but that

closely matched and matching variable(s) are

the out-group member will receive an increas-

strongly related to the outcome variable, this

ingly negative evaluation as the participant's

design will approach the statistical power o f an

level o f prejudice increases. W i t h only two

actual within-subject design. In some cases, the

within-subject treatment conditions (in-versus

gain in statistical power can be substantial.

out-group),

Student ( 1 9 3 1 ) showed that an early random-

Treatment x Background Variable interaction

ized experiment on 1 0 , 0 0 0 children comparing

effect can be tested using a simple difference

their gains in height and weight from con-

score method

sumption

o f pasteurized

versus raw

the treatment

3

effect

and

the

(Judd, Kenny, & McClelland,

milk

2 0 0 1 ) . T h e difference between the outcomes

would have achieved the same statistical power

for each participant in the in-group and out-

if 5 0 matched pairs o f identical twins had been

group conditions, D = Y, - Y

randomly

puted. T h e n , the continuous

N

assigned to treatment

conditions

instead.

OUT>

is first commeasure o f

prejudice, X , must be centered (i.e., put in deviation form, X = X - M , where X c

Moderator Effects

x

c

is the

centered variable, X is the original measure of prejudice, and M

x

is the mean o f the sample

T h e designs we have considered so far

on the measure o f prejudice). Finally, the

assume that the magnitude o f the treatment

difference score is regressed on the centered

Within-Subject

and Longitudinal

Experiments

\

prejudice score, X , using the regression

sets o f difference scores for each participant:

equation D=b

D

c

0

test of b

+ b^X . In this equation, the c

is equivalent to the test of the

l

= M j - M for the difference between the

M

2

scores on the mediator and D

= 0

G

- 0

1

2

for

Prejudice x In-group versus Out-group inter-

the difference between the scores on the out-

action, and the test o f b corresponds to the

c o m e variable in the two treatment groups.

in-group versus out-group effect, so long as

The

prejudice is centered. With more complex

regressed on the difference in the mediators,

0

difference in the o u t c o m e s is

then

. T h e test o f b represents the

designs (e.g., more than two treatment groups),

D

moderator effects are most easily tested within

test o f the final condition for mediation. In

the framework of the general linear model

addition, if b does not differ from 0 , then the

(Maxwell & Delaney, 2 0 0 3 ) . These models can

results are consistent with a model in which

be extended to designs with more than one

the entire treatment effect operates through

background variable and more than two levels

the mediating variable.

of within-subjects treatment conditions.

=b +b D 0

l

M

x

0

Tests

Mediation

of

mediation

involve

strong

assumptions

(Holland, 1 9 8 8 ; M a c K i n n o n ,

Lockwood,

Hoffman,

West,

&

Sheets,

2 0 0 2 ) . T h e assumptions o f temporal stability

Mediation

occurs

when

a

treatment

causes changes in a mediating (immediate

o

outcome),

which,

and causal transience must be met. T h e medi-

variable

ator must be measured without error and be

turn,

causally prior t o the outcome. T h e relation-

in

causes changes in the outcome variable,

ship between the mediator

T —> M —» O . F o r example, frustration may

must be linear; the variances o f the outcome

and

outcome

be hypothesized to lead to specific areas o f

variable in the two treatment groups must be

cortical

equal in the population. M e t h o d s for testing

activation

(measured

by M R I ) ,

which, in turn, lead to aggression. Baron and

mediation with m o r e than two

treatment

Kenny ( 1 9 8 6 ) developed a framework

groups, models involving both

mediation

for

testing mediation in between-subject designs,

and moderation, and methods for when the

which we will utilize here. T h e B a r o n and

assumptions o f the basic model are violated

Kenny framework recently was extended by

are described by Judd et al. ( 2 0 0 1 ) .

Judd et al. ( 2 0 0 1 ) to within-subject designs. W e consider only designs in which there are two treatment groups (see also Hoyle

&

R o b i n s o n , Chapter 1 0 , this volume). In the Judd et al. framework, three condi-

Summary and Conclusions Within-subject experimental designs provide a relatively strong basis for estimating the

tions are needed in a within-subject design t o

causal effects o f treatments. The primary issues

show that the data are consistent with a

that must be addressed are the assumptions o f

mediational model. First, the treatment must

temporal stability and causal transience. W e

produce changes in the outcome variable,

presented several strategies that reduce, but do

T —» O . Second, the treatment must produce

not fully eliminate, threats to internal validity.

changes in the mediator, T —> M . These first

Consequently, if researchers base the selection

two conditions can be tested directly using a

of a research design solely on the criterion of

repeated measures t test or A N O V A . Finally,

internal validity, then simple

randomized

the mediator must be shown to account for a

between-subjects experiments conducted in the

significant

laboratory often will be the method of choice.

portion

of

the

relationship

between the treatment and the o u t c o m e . T h e final condition can be tested by creating t w o

Earlier, w e n o t e d

that

within-subject

designs are rarely reported in two o f the

295

296

DESIGN AND ANALYSIS leading journals in social psychology. W e

achieve a desired level o f statistical power

also observed that they are c o m m o n in some

(e.g., . 8 0 ) in a within-subjects design with

areas o f psychology, such as perception and

two treatment groups and a correlation o f .5

psychophysics (Keren, 1 9 9 3 ) . H o w can we

between-subjects' responses in the t w o treat-

account for this?

ment conditions is only 2 5 % o f the number

First, many social psychological manipulations, particularly when employed in con-

needed

in

the

corresponding

between-

subjects design (Venter & M a x w e l l , 1 9 9 9 ) .

substantial

Some emerging research areas are likely to

deception, m a k e it very difficult or impossi-

place a high priority on this advantage. As

ble to return participants to their original

noted earlier, researchers w h o

baseline state. Even under more favorable

questions requiring specialized populations

texts

using

minor

or

more

investigate

within-

and researchers for w h o m data collection is

subject designs (e.g., Latin square designs;

very expensive will benefit greatly from the

randomized matched designs) yield internal

use o f within-subject designs. In addition,

validity c o m p a r a b l e

when multiple replications o f each treatment

conditions in which the stronger

to

between-subjects

laboratory experiments, the within-subject

condition are possible within subject, the

designs require more careful thought on the

causal effect may once again be estimated

part o f the researcher t o identify potential

separately for each participant rather than

threats and to incorporate design features

for the entire sample (see Anderson, 1 9 8 2 ) . Given the increasing realization that statisti-

that help rule out those specific threats. Second, social psychological researchers

cal models that assume a constant treatment

have not historically placed a high priority on

effect across different individuals may often

designs that achieve high levels o f statistical

be unrealistic, the fact that such designs per-

power (see Rossi, 1 9 9 0 ) . T h e easy availability

mit tests o f this assumption should increase

of large numbers o f our favorite research

their use. O u r hope is that researchers will

participants, college undergraduates, at many

rediscover the careful thinking o f a distin-

large universities may reduce the

guished previous generation o f researchers

perceived

need for designs that maximize statistical

concerning the advantages

power.

tages o f within-subject designs.

and

disadvan-

Third, much o f the careful thinking about within-subject designs was originally undertaken by experimental psychologists such as

LONGITUDINAL EXPERIMENTS

Benton Underwood and psychological statisticians such as Ben W i n e r in the 1 9 6 0 s . T h e

In the basic two-group between-subjects ran-

insights o f this earlier era have been crowded

domized experiment depicted in Figure 1 3 . 1

out o f the curriculum in psychology and are

(section A ) , the researcher randomly assigns

not being transmitted to new generations o f

participants to the treatment or control group.

researchers (see Aiken, West, Sechrest, &

Following the delivery o f the treatment, the

Reno, 1990).

researcher collects the outcome variable from

There are some developing counter forces

each participant.

Longitudinal

experiments

that may presage a reinvigoration o f the

extend this basic design by assessing the out-

within-subject design.

come variable on more than one measurement

One

o f the

chief

advantages o f the within-subject design is its

occasion. As we will illustrate later, the

efficiency in achieving adequate statistical

repeated measurements provide a far greater

power with the minimum number o f partici-

ability to examine the temporal course o f the

pants. T h e number o f participants needed to

effects o f social psychological manipulations.

Within-Subject and Longitudinal A.

Randomized posttest only Posttest M,

Treatment Control B.

Experiments

Randomized pretest-posttest design Pretest M, M,

Posttest M M

Treatment Control

2

2

C. Solomon four-group design Pretest M, M 1

D.

Treatment Control

Posttest M M

Treatment Control

M, M,

2

Longitudinal experiment (Four Waves)

Treatment Control Figure 13.1

2

Measurement Wave (Trial) 1 2 3 4 M, M M M M, M M M 2

3

4

2

3

4

Some Randomized Experimental Designs

NOTE: All participants are randomly assigned to treatment groups. The same outcome measurement is collected at each measurement wave. Measurement wave is indicated by a subscript (1, 2, 3, 4 ) .

Given

that

randomization

has

occurred

successfully, the two major threats to the

regular basis. In these longitudinal

designs

with multiple measurement occasions, par-

inference o f the causal effect are attrition and

ticipant

violations o f S U T V A .

threat to the validity o f the causal inferences.

attrition

b e c o m e s an

important

O f particular concern is differential attrition from the treatment and control groups. Even if the rate o f attrition is identical in the T and

Threats to Causal Inference

C groups, there m a y still be reason for con-

Attrition

cern. F o r example, in an educational experifrom

ment testing a new curriculum, if the least

measurement, can occur for a variety o f rea-

intelligent students drop out o f the T group

Attrition,

or loss o f participants

sons, particularly when the experiment is

and the most intelligent students drop out o f

extended in time and when participants leave

the C group, any results showing higher stan-

the laboratory. In some basic and

many

dardized test scores in the treatment relative

applied experiments in social psychology,

t o the control group would be highly suspect.

participants are measured daily, once a week, once a month, once a year, or on a less

The

best solution to problems o f attri-

tion is t o prevent it from occurring. This

297

298

DESIGN AND ANALYSIS strategy requires careful planning during the

pretest. O u t c o m e data collected during early

development o f the experimental protocol. A

measurement periods following treatment can

considerable craft knowledge o f techniques

provide useful information to adjust estimates

for keeping participants involved in longitu-

of treatment effects during later measurement

dinal research has developed over the past

periods, after attrition has occurred. In addi-

few decades (see Ribisl et al., 1 9 9 6 ) . These

tion, W e s t and

methods require that experimental staff have

methods of estimating upper and lower bounds

Sagarin ( 2 0 0 0 )

described

multiple ways o f locating participants, send

to treatment

participants reminders, and have plans for

that attrition may be associated with impor-

alternative data collection procedures if a

tant, but unmeasured, variables.

effects

given the possibility

participant does not show up or moves to another location. O f importance, participants in the control group should receive just

SUTVA

as much attention as those in the treatment

O f most concern in longitudinal experi-

group. Multiyear longitudinal experiments

ments in social psychology is the second part

using these retention procedures often have

of S U T V A , the possibility that participants

been able t o limit attrition t o less than 5 % o f

communicate about

the original sample.

dures, potentially contaminating the esti-

experimental proce-

In cases in which attrition cannot be pre-

mates o f the causal effects. Traditionally,

vented, statistical analyses may be conducted

social psychologists have followed the proce-

to probe the effects o f attrition and adjust for

dure o f swearing participants to secrecy,

the effects o f attrition. If pretest measures can

although the effectiveness o f these proce-

be collected on all participants, these measures

dures in experiments over extended time

provide critical information about the nature

periods has not been examined carefully. T h e

of the participants w h o do not complete

use o f subtle treatments o f which the partici-

posttest measurement (attriters). Prediction o f

pant may not be fully aware and procedures

attrition status

(1 = completer; 0 = attriter)

that minimize information about the exis-

from (a) the pretest score, (b) the treatment

tence o f other treatment conditions also may

condition, and (c) the interaction o f pretest

be helpful in many cases. W h e n treatments

and treatment in a logistic regression analysis

are not subtle, most effective will be proce-

helps researchers understand the nature o f the

dures that geographically and/or temporally

attrition problem (West & Sagarin, 2 0 0 0 ) . A

separate participants.

main effect o f the pretest score indicates that the researcher should be very cautious in generalizing the results o f the experiment to the full population from which the participants

Some Longitudinal Experimental Designs

were sampled, whereas an interaction between

In our presentation below, we consider

the pretest and the treatment indicates that the

several traditional and new designs and analytic

nature o f the participants w h o dropped out

approaches for longitudinal

differed in the T and C conditions. This latter

designs.

experimental

finding indicates a potentially serious threat to causal inference. W h e n attrition

does

occur, the application o f analyses based on modern

missing data

theory

(Schafer

&

Pre-Post The

Randomized

pre-post

Experiment

randomized

experiment

Graham, 2 0 0 2 ) can adjust treatment effects

depicted in Figure 1 3 . 1 (section B) adds to

for the effects o f any variables measured at

the basic randomized design the feature o f a

Within-Subject and Longitudinal Experiments pretest on the o u t c o m e variable. T h u s , this

statistically equating groups

design combines the design features o f ran-

pretest differences.

on

\

observed

domization and within-subject design. T h i s design has long been used in some areas o f social psychology (e.g., attitude change). T h e

Solomon

Four-Group

Design

chief advantage o f the pre-post design is that,

One potential criticism o f the pre-post ran-

given a fixed number o f participants, the sta-

domized experiment is that generalization o f

tistical power typically is higher than in the

the causal effect may be limited to contexts in

posttest-only design. T o the extent that there

which a pretest measurement is taken. For

is a relationship between the pretest and the

example, the pretest measurement may sensi-

posttest (e.g., in an attitude change experi-

tize the participants to the subsequent treat-

ment, more positive attitudes at pretest are

ment, making it more effective. O r , repeated

associated with m o r e positive attitudes at

testing may alter the participant's response

posttest), the ability to detect true treatment

(see T a b l e 1 3 . 1 , section A ) . T o address such

effects will be increased. F o r example, to

criticisms,

detect a standardized

S o l o m o n four-group

effect

size d - . 5 0 ,

researchers

can

employ

design illustrated

the in

where d is the difference in standard devia-

Figure 1 3 . 1 (section C ) , which represents a

tions between the treatment

combination o f the simple posttest-only ran-

and

control

groups (which Cohen, 1 9 8 8 , describes as a

domized experiment and the pretest-posttest

moderate effect size), the researcher will need

randomized

1 2 6 participants ( 6 3 T , 6 3 C) to be able to

domly assigned to one o f the four groups, and

reject the null hypothesis 8 0 % o f the time.

the magnitude of the treatment effect in the

However,

. 6 0 correlation

posttest-only versus pretest-posttest measure-

between the pretest and the posttest, this

ment conditions can be compared. In cases

number

reduction).

in which the estimates o f the causal effect

Statistical procedures decrease the magnitude

in the two measurement conditions differ,

of the error variance by removing the portion

researchers must decide whether the posttest-

if there falls

to

is a 82

(36%

experiment. Subjects are

ran-

that is accounted for by the subjects' pretest

only or the pretest-posttest measurement con-

scores. W h e n the relationship between the

dition best represents the context to which

pretest and posttest is linear and the strength

they wish to generalize. Typically, the posttest-

of this relationship does not differ in the

only estimate will be chosen. In cases in which

treatment and control conditions, then analysis

they do not differ, the estimates o f the causal

of covariance provides the proper adjustment.

effects from the two measurement conditions

In

can

other cases, m o r e c o m p l e x statistical

be

combined

using

meta-analytic

models are required (see Huitema, 1 9 8 0 ;

procedures suggested by Braver and Braver

Little, An, J o h a n n s , &

( 1 9 8 8 ) or through missing data techniques

Giordani,

2000,

M a x w e l l & Delaney, 2 0 0 3 ) .

(Schafer & Graham, 2 0 0 2 ) .

A second key advantage o f the pre-post randomized experiment is often overlooked (Shadish et al., 2 0 0 2 ) . As we noted earlier, if pretest measures can be collected on all par-

Multiwave In

Longitudinal

Experiment

multiwave longitudinal experiments,

ticipants, these measures provide critical

participants are randomly assigned to treat-

information that can be used to study the

ment conditions and then measured on the

effects o f attrition and t o provide improved

outcome variable at several time points fol-

estimates o f the causal effect both by using

lowing

modern

section D ) . Longitudinal designs and analy-

missing data

techniques and

by

treatment

(see

Figure

13.1,

299

300

|

DESIGN AND ANALYSIS

Table 13.3

Univariate Analysis of Variance of Disliking Ratings

Source Treatment (in-group vs. out-group) Error between Trials Treatment x Trials Error within

df

MS

1 76 3 3 228

37,206 861 760 450 108

43.2