Handbook of Research on Teaching (Third Edition)

Table of contents :
Contents
Preface
Part 1 Theory and Method of Research on Teaching
I Paradigms and Research Programs in the Study of Teaching: A Contemporary Perspective Lee S. Shulman
2. Philosophy of Research on Teaching: Three Aspects Gary D Fenstermacher
3 Measurement of Teaching Richard J. Shavelson, Noreen M. Webb, and Leigh Burstein
4. Quantitative Methods in Research on Teaching Robert L. Linn
5. Qualitative Methods in Research on Teaching Frederick Erickson
6. Observation as Inquiry and Method Carolyn M. Evertsen & Judith L. Green
7. Syntheses of Research on Teaching Herbert J. Wal berg
8. Theory, Methods, Knowledge, and Research on Teaching Bruce J. Biddle & Donald S. Anderson
Part 2 Research on Teach i ng and Teachers
9. Teachers' Thought Processes Christopher M. Clark & Penelope L. Peterson
10. Students' Thought Processes Merlin C. Wittrock
11. The Teaching of Learning Strategies Claire E. Weinstein & Richard E. Mayer
12. Teacher Behavior and Student Achievement Jere Brophy & Thomas L. Good
13. Teaching Functions Barak Rosenshine and Robert Stevens
14. Classroom Organization and Management Walter Doyle
15. Classroom Discourse Courtney B. Cazden
16. Media in Teaching Richard E. Clark & Gavriel Salomon
17. Philosophy and Teaching Maxine G reene
Part 3 The Social and Institutional Context of Teaching
18. The Cultures of Teaching Sharon Feiman-Nemser and Robert E. Floden
19. Research on Teacher Education Judith E. Lanier
20. School Effects Thomas L. Good & Jere E. Brophy
Part 4 Adapting Teaching to Differences Among Learners
21. Adapting Teaching to Individual Differences Among Learners Lyn Corna & Richard E. Snow
22. Teaching Creative and Gifted Learners E. Paul Torrance
23. Teaching Bilingual Learners Lily Wong Fillmore
24. Special Educational Research on Mildly Handicapped Learners Donald L. Mac Millan, Barbara K. Keogh & Reginald L. Jones
Part 5 Research on the Teach i ng of S u bjects and G rad e Levels
25. Research on Early Childhood and Elementary School Teaching Prog rams Jane A. Stallings & Deborah Stipek
26. Research on Teaching in Higher Education Michael J. Dunkin
27. Research on Written Composition Marlene Scardamalia & Carl Bereiter
28. Research on Teaching Reading Robert Calfee & Priscilla Drum
29. Research on Teaching and Learning Mathematics: Two Disciplines of Scientific Inquiry Thomas A. Romberg & Thomas P. Carpenter
30. Research on Natural Sciences Richard T. White & Richard P. Tisher
31 . Research on Teaching Arts and Aesthetics Beverly J . Jones & June King Mcfee
32. Moral Education and Values Education : The Discourse Perspective Fritz K. Oser
33. Research on Teaching Social Studies Beverly J. Armento
34. Research on Professional Education Sarah M . Dinham & Frank T. Stritter
35. Research in Teaching in the Armed Forces Harold F. O'Nei l, J r., Clinton L. Anderson & Jeanne A. Freeman
About the Contributors
Indexes
Subject Index
Name Index

Citation preview

HANDBOOK OF RESEARCH ON TEACHING Thi rd Edition

Editor

Merlin C. Wittrock

Editorial Advisory Board Appointed by the American Educational Research Association

Marianne Amare/ Beverly Armento David Berliner

Geraldine Clifford Walter Doyle

Frank Farley

Gary Fenstermacher

Good Reginald Jones Thomas

Richard Shave/son Lee Shulman

HANDBOOK OF RESEARCH ON TEACHING Thi rd Ed ition A PROJECT OF THE AMERICAN EDUCATIONAL RESEARCH ASSOCIATION

Edited by

Merlin C. Wittrock

MACMILLAN PUBLISHING COMPANY A Division of Macmillan, Inc. NEW YORK

Collier Macmillan Publishers LONDON

Copyright © 1986by the American Educational Research Association.

All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the Publisher.

Macmillan Publishing Company A Division of Macmillan; Inc. 866Third Avenue, New York, N. Y. 1002 2 Collier Macmillan Canada, Inc. Library of Congress Catalog Card Number:85-4866 Printed in the United States of America printing number 56789 10

Library of Congress Cataloging in Publication Data Main entry under title: H andbook of research on teaching. "A project of the American Educational Research Association." Includes bibliographies and index. 1. Education-Research-United States. I. Wittrock, M. C. (Merlin C.), 1931. II. American Educational Research Association. III. Second handbook of research on teaching. 371.1'07'072 854866 LB1028H. 31 51985 ISBN 0 -02-900310-5

Contents ix

Preface Acknowledgments

xi

PART 1: THEORY AND METHOD OF RESEARCH ON TEACHING 1.

Paradigms and Research Programs in the Study of Teaching: A Contemporary Perspective Lee S. Shulman

2. Philosophy of Research on Teaching: Three Aspects Gary D Fenstermacher 3.

Measurement of Teaching Richard J. Shave/son, Noreen M. Webb, and Leigh Burstein

4. Quantitative Methods in Research on Teaching Robert L. Linn

3 37

50 92

5.

Qualitative Methods in Research on Teaching Frederick Erickson

119

6.

Observation as Inquiry and Method Carolyn M. Evertson and Judith L. Green

162

Syntheses of Research on Teaching Herbert J. Walberg

214

Theory, Methods, Knowledge, and Research on Teaching Bruce J. Biddle and Donald S. Anderson

230

7. 8.

PART 2: RESEARCH ON TEACHING AND TEACHERS 9.

Teachers' Thought Processes Christopher M. Clark and Penelope L. Peterson

255

10. Students' Thought Processess Merlin C. Wittrock

297

11. The Teaching of Learning Strategies Claire F. Weinstein and Richard F. Mayer

315 V

vi

CONTENTS

12. Teacher Behavior and Student Achievement Jere E. Brophy and Thomas L. Good

328

13. Teaching Functions Barak Rosenshine and Robert Stevens

376

14. Classroom Organization and Management Walter Doyle

392

15. Classroom Discourse Courtney B. Cazden

432

16. Media in Teaching Richard E. Clark and Gavriel Salomon

464

17. Philosophy and Teaching Maxine Greene

479

PART 3: THE SOCIAL AND INSTITUTIONAL CONTEXT OF TEACHING 18. The Cultures of Teaching Sharon Feiman-Nemser and Robert E. Floden

505

19. Research on Teacher Education Judith E. Lanier with Judith W. Little

527

20. School Effects Thomas L. Good and Jere E. Brophy

570

PART 4: ADAPTING TEACHING TO DIFFERENCES AMONG LEARNERS 21. Adapting Teaching to Individual Differences Among Learners Lyn Corno and Richard E. Snow

605

22. Teaching Creative and Gifted Learners E. Paul Torrance

630

23. Teaching Bilingual Learners Lily Wong Fillmore with Concepcion Valadez

648

24. Special Educational Research on Mildly Handicapped Learners Donald L. MacMillan, Barbara K. Keogh and Reginald L. Jones

686

PART 5: RESEARCH ON THE TEACHING OF SUBJECTS AND GRADE LEVELS 25. Research on Early Childhood and Elementary School Teaching Programs Jane A. Stallings and Deborah Stipek

727

26. Research on Teaching in Higher Education Michael J. Dunkin with Jennifer Barnes

754

27. Research on Written Composition Marlene Scardamalia and Carl Bereiter

778

CONTENTS

vii

28.

Research on Teaching Reading Robert Calfee and Priscilla Drum

804

29.

Research on Teaching and Learning Mathematics: Two Disciplines of Scientific Inquiry Thomas A. Romberg and Thomas P. Carpenter

850

30. Research on Natural Sciences Richard T. White and Richard P. Tisher

874

31.

Research on Teaching Arts and Aesthetics Beverly J. Jones and June King McFee

906

32.

Moral Education and Values Education: The Discourse Perspective Fritz K. Oser

917

33.

Research on Teaching Social Studies Beverly J. Armento

942

34.

Research on Professional Education Sarah M. Dinham and Frank T. Stritter

952

35.

Research on Teaching in the Armed Forces Harold F. O'Neil, Clinton L. Anderson, and Jeanne A. Freeman

971

About the Contributors

988

Indexes

995

Preface Research on teaching has flourished since the publication of the Second Handbook of Research on Teaching. Since then, traditional lines of inquiry matured and emerging areas of research evolved. These old and new areas of research led to chapters in the third edition that have no counterparts in the two earlier editions of the Handbook. In addition, methods for studying, observing, and analyzing teaching and thought processes of learners grew and developed. From these areas of study and methods of research came data that added to our knowledge about teaching and to our interpretations of familiar concepts. In addition to these empirical advances, important conceptual approaches opened lines of inquiry that led to an understanding of the cognitive processes of teachers and learners that mediate the effects of teaching upon student achievement. These recent advances in empirical research, combined with improved conceptualizations, advanced our under­ standing of teaching and our ability to explain and to demonstrate how teaching can be improved.

Historical Notes These strong statements about data, methods, and conceptualizations of research, especially about the significance of research on teaching, indicate progress made since the publication of the last edition of the Handbook. In the preface to the Second Handbook of Research on Teaching, Robert Travers discussed his disappointment in the lack of advance in substantive knowledge about teaching. He described the difficulty chapter authors had in finding significant research to report in the Handbook, a difficulty even greater than that encountered by the chapter authors of the first edition of the Handbook of Research on Teaching. He also mentioned that the research on teaching was often not cumulative across areas of study or even within an area, lacking the coherence of integrative theories and models of teaching. On a more positive note, Travers discussed the then new areas of research on teaching that offered promise. He mentioned the technology of classroom management. He wrote also of the influence of educational sociology, and of the contributions of the following areas of study. Behavior modifica­ tion was beginning to affect teaching. Piagetian concepts about learning were also beginning to make their way into research on teaching. The study of preschool educational programs offered significant findings and promise, as did the research on computer assisted instruction. At the time he wrote, the young field of research on teaching was growing rapidly. Theories of teaching were attempting to catch up with it, organize it, and develop conceptualizations and models of it. Dunkin a,nd Biddle's influential book, A Study of Teaching, was soon to be published. Process-product research on teaching was the dominant research program. Programmed instruc­ tion was a popular instructional technique, and behavioristic models of learning were the most common conceptualizations of how teaching influenced achievement. The study of cognition had not yet made much impact on research on teaching.

Recent Advances in Research In the Handbook of Research on Teaching, Third Edition, none of the chapter authors had difficulty in finding significant work to report either in substantive areas or in the methodologies of research. Some of the significant research they report occurs in areas that have no analogues in chapters of the ix

X

PREFACE earlier editions of the Handbook. The chapter "Teachers' Thought Processes," for example, reports data and a conceptualization of teaching that emphasize how teachers' thoughts mediate between classrooom processes and student behaviors. Most of the work in this area began in 1973. Another chapter, "Students' Thought Processes," also has no parallel in the earlier editions of the Handbook. In this chapter, research on student cognitive and affective processes, such as attention, motivation, and comprehension, offers new interpretations of teaching processes, such as practice, time-on-task, teacher praise, reinforcement, and the active role of learners in constructing meaning from classroom teaching. For example, teacher praise seems to influence the attention of many students in the classroom. By observing one child being rewarded, many students learn the teachers' objectives and-intentions. From the study of student attributional processes, reinforcement seems to function, not automatically, but primarily when learners attribute it to their own activity. Research on learning from text and on reading and writing indicates that comprehension involves students actively building relations between knowledge or experience and the text. Other chapters report related data and models. The chapter "The Teaching of Learning Strategies" discusses the mental processes learners use to remember and to understand information and subjects taught in school. The chapter on teaching in the armed forces introduces research on intelligent computer-assisted instruction, including beginning attempts to model or simulate learning using the computer to represent our knowledge about a learner's responses to teaching. In the chapter on mathematics learning, which is, of course, not a new chapter topic, research on learner's strategies of adding and subtracting is described and used to imply how teaching might be improved using this knowledge. The chapter on written composition studies related processes and their implications for the teaching of writing. In the section on theory and methods of research on teaching, methodological advances are presented. In the chapter "Measurement of Teaching," for example, the authors present methods for measuring students' and teachers' thought processes. In the chapter on quantitative methods, sophisticated techniques of multivariate analyses of teaching are summarized. In the chapter "Qualitative Methods in Research on Teaching," we read about the influence of anthropology upon the methods of research on teaching. Last of all, the newly emerging programs and paradigms of research on teaching are discussed in the first chapter of the Handbook. This chapter on research programs provides an excellent introduction into the concepts and models that underlie the recent and encouraging progress in the research on teaching reported throughout the Handbook.

The Development of the Handbook Frank Farley, President of AERA, asked me to edit the Handbook of Research on Teaching, Third Edition. With his advice, and after talking with numerous people in AERA, especially in the field of research on teaching, I nominated and President Farley appointed the Handbook Editorial Board:

Marianne Amarel, Beverly Armento, David Berliner, Geraldine Clifford, Walter Doyle, Frank Farley, Gary Fenstermacher, Thomas Good, Reginald Jones, Richard Shavelson, and Lee Shulman. After several months of preliminary work, in August of 1981 this Editorial Board met for three days to design the Handbook, to set policies for its preparation, to suggest chapter authors, and to make recommendations about the obligations of the Handbook publisher. Among the more important recommendations and policies of the Editorial Board were the following. First, the board suggested structuring the Handbook to best represent the advancing state of knowledge about teaching, including the incorporation of new chapters needed to reflect recent and important lines of inquiry. Second, the board asked me to appoint at least one, usually two, reviewers for each chapter. These reviewers, as do their counterparts in other AERA publications, would provide comments and feedback to the chapter authors from colleagues specializing in their same fields of research. After the meeting of the Editorial Board, the authors of the Handbook chapters were promptly invited, as were the reviewers for each chapter. Each chapter author and each chapter reviewer was told that we had four objectives or intentions for each chapter. First, we wanted the chapter to include, but also to go beyond, the important function of summarizing or reviewing research, theories, and methods of research on teaching. We wanted chapter authors to emphasize a

xi

PREFACE conceptual understanding of the research on teaching that will show the readers how the studies, theories, and findings relate to one another. We wanted to provide an integrated discussion of research. Second, we wanted to convey what is known about research on teaching, and, where appropriate, what is known about teaching. Third; we wanted to provide useful theoretical explanations of the research findings. Fourth, we wanted to provide an organized coverage of the appropriate subject matter. All chapters were developed according to these criteria.

ACKNOWLEDGMENTS The preparation of this Handbook involved excellent work by many people. I am greatly indebted to them for all of their contributions to the Handbook. I thank Frank Farley for his guidance in the planning of the Handbook. The members of the Editorial Board deserve full credit and special recognition for all of their contributions to the focus and structure of the Handbook, for their suggestions about the chapter authors and reviewers, and for the development of the policies that guided the preparation of this volume. The chapter authors and the chapter reviewers, of course, deserve thanks for preparing the manuscripts that comprise the Handbook. Anita King, Amy Shaunessey, and William Russell, of AERA, contributed to the preparation of the Handbook from its conception to its completion. Elyse Dubin and Charles Smith, of Macmillan Publishing Company, supervised the publication of the book. Elyse Dubin and her staff worked carefully, transforming the edited manuscripts into the finished volume. I also want to thank people at UCLA, including Deans John Goodlad and C. Wayne Gordon for their support. Special thanks go to Christine Carrillo, Barbara Trelease, and Joyce Amslow for their help over several years with all the correspondence regarding the preparation of the Handbook, and to Nancy Wittrock for the many hours she donated to the editing of the Handbook. Merlin C. Wittrock Los Angeles, California

HANDBOOK OF RESEARCH ON TEACHING Third Edition

Part 1 Theory and Method of Research on Teaching

I

Paradigms and Research Programs in the Study of Teaching: A Contemporary Perspective Lee S. Shulman Stanford University

studies summarized in this volume, it is essential that the reader understand the questions that have been asked and the manner in which those questions have been framed, both conceptually and methodologically. Research on teaching, like most other fields of study, is not the work of individual scholars working alone and idiosyncratically. Indeed, most research is conducted in the context of scientific communities, "invisible colleges'�-�f scholars who share similar conce tions of ro er questions, methods, techniques, and forms of explanation. To understand why research is formulated in a particular fashion, one needs to locate the investigation among the alternative approaches to inquiry that characterize a field. A goal of this chapter will be to describe the diverse communities of scholars, practitioners, and policymakers that comprise, or in whose interests are defined, the activities and universe of research on teaching. The term most frequently employed to describe such research communities, and the conceptions of problem and method they share, is J.!.aradigm. The term has been used in several ways. In his chapter "Paradigms for Research on Teaching" prepared for the first Handbook of Research on Teaching under his editor­ ship, Gage referred to paradigms as "moaels, patterns, or sche­ mata. Paradigms are not theories; they are rather ways of thinking or patterns for research that, when carried out, can lead to the develo ment of theor "(Gage, 1963, p. 95). Writing during the infancy of this field of research, Gage drew most of his models from psychology or other behavioral sciences, rather than from the study of teaching itself. He was describing how models might be used in the study of teaching, not how they had already been employed. An important sign of the vigor of the field Gage was then fathering is the multiplicity of models from the study of teaching itself that we can now describe some

Introduction and Overview This is a chapter about alternatives. It deals with the alternative ways in which the women and men who study teaching go about their tasks. We conduct research in a field to make sense of it, to et smarter about it, erha s to learn how to erform more adeptly within it. Those who investigate teaching are in­ volved in concerted attempts to understand the phenomena of teaching, to learn how to improve its performance, to discover better ways of preparing individuals who wish to teach. This handbook presents the approaches and results of research on teaching, both to inform readers regarding the current state of theoretical knowledge and practical understanding in the field and to guide future efforts by scholars to add to that fund of understanding. The purpose of this chapter is to serve as a reader's guide to the field of research on teaching, especially to the research pro­ grams that direct, model, or point the ways for research on teaching. The premise behind this chapter is that the field of research on teaching has produced, and will continue to yield, growing bodies of knowledge. But knowledge does not grow naturally or inexorably. It is produced through the inquiries of scholars - empiricists, theorists, practitioners - and is there­ fo!e a func!�o_nof th_iii!!d_s__ �Lq!J��i!funs asked, roblems os.ed, and issues framed by those who do research. To understand the findings and methods of research on teaching, therefore, re­ quires that the reader appreciate the varieties of ways in which such questions are formulated. The framing of a research ques­ !i.Q!!,_like that of an attorney in a court of law, limits the ran e of ermissible res onses and refigures the character of possible outcomes. Simply put, to interpret the findings of the many

The author thanks reviewers Richard Shavelson (U.C.L.A.) and N. L. Gage (Stanford University) and editorial consultants Walter Doyle and Marianne Amarel for their helpful suggestions.

3

4

LEE S. SHULMAN

twenty years later. More recently, Doyle (1978; 1983) has written lucidly on the paradigms for research on teaching. The most famous use of"paradigm" is that of Thomas Kuhn, whose Structure of Scientific Revolutions (1970) is a classic of contemporary history of science that has become part of the common parlance and prevailing views of nearly all members of the social and natural science communities. Since one of his friendliest critics (Masterman, 1970) identified some twenty­ two different uses of"paradigm" in Kuhn's book, I will refrain from attempting a succinct definition at this point. I prefer to .employ the concept of a research program (Lakatos, 1970) to describe the genres of inquiry found in the study of teaching, rather than the Kuhnian conception of a paradigm. Neverthe­ less, .the two terms are used interchangeably in most of the chapter. The argument of this chapter is that each of the extant re­ search programs grows out of a particular perspective, a bias of either convention or discipline, necessarily illuminating some part of the field of teaching while ignoring the rest. The danger for any field of social science or educational research "iies in.its· l)Otential corruption (or worse, trivialization) by a single para­ digmatic view. In this manner, the social sciences and education can be seen as quite different from Kuhn's conception of a mature paradigmatic discipline in the natural sciences, which is ostensibly characterized by a single dominant paradigm whose principles define "normal science" for that field of study. I will therefore argue that a healthy current trend is the emer­ gence of more complex research designs and research programs that include concern for a wide range of determinants influenc­ ing teaching practice and its consequences. These "hybric!" de­ signs, Fhich mix. �xperim�nt with ethnography, mult� regressions with multiple case studies, process-product designs with analyses of student mediation, surveys with personal dia­ ries, are exciting new developments in the study of teachin� But they present serious dangers as well. They can become utter chaos if not informed by an understanding of the types of kpowledge roduced b these different a roaches. However, the al­ ·ternative strategy that reduces the richness of teaching to nothing more than the atomism of a multiple variable design may be even worse. This chapter will thus discuss several alter­ native ways of thinking about "grand strategies" for research on teaching, for programs of research properly construed rather than individual, one-shot investigations. The chapter will begin with a discussion of the general char­ acter of research programs or paradigms, those conceptions of problem and procedure that members of a research community share and in terms of which they pursue their inquiries and exercise their gatekeeping. After examining the general conception of research pro­ grams, a synoptic map of research on the teaching field will be presented. In terms of that map, the various research programs that constitute the field will be described and discussed. This general model will be followed by detailed discussions of the dominant competing (and complementary) research programs currently pursued in the study of teaching. The next section will discuss the prospects for this field of study, in light of its current progress and present dangers, and in the spirit of contemporary critiques of social science method and theory as exemplified in the work of Cronbach (1975;

1982). Finally, a set of recommendations and anticipations re­ garding future research programs will be presented. We begin with the matter of research programs or paradigms.

Paradigms and Research Programs

How should teaching be studied? Where does one begin? In what terms can questions be put? Although logically the range and diversity of answers to these questions is vast, in practice, any given scholar appears to operate within a fairly limited repertoire of alternatives. Thus, some researchers always begin with the assumption that their ti'sk.Is to relate, whether experi1:!}ell_!.�ly or descriptively, variations in the measured achieve­ ment or attitudes of pupils to variations in the -observed behav1o_r 9f teac:li�r:�- Additional wrinkles may be added to the­ design - use of individual pupil data as against classroom mean scores, use of pupil- or teacher-characteristic data as mediating variables - but the fundamental character of the questions re­ mains unchanged. Other scholars are equally focused on still other formulations, whether involving classroom discourse, teacher cognitions, the sense pupils make of instruction, or the social organization of classrooms via task or activity structures. Once committed to a particular line o_Lresearch, the indivi1 WISC & WRATREADING & WRATMATH

Classify: ED

Classify: Regular

Behavior

WISC - WRATREADING & WISC - WRATMATH

WRATREADING - WRATMATH < 20 Classify: ED

Classify: LD

Both < 24

Either ::: 24

Classify: Regular

WRATMATH & WRATREADING

> 80 Classify: Regular

Classify: LD

Fig. 3.2. Model of placement reco m mendations built from p rocess tracing d ata (Cadwell & Leary, 1 982).

From the verbal protocols, an analysis was made of each teacher-consultant's decision process and charted. Figure 3.2 shows a flow chart for one teacher-consultant's placement deci­ sions. The flow chart indicates the order in which the teacher requested information: behavior rating, IQ, achievement, sex, and SES. The behavior variable predominated, entering the de­ cision process in two ways: (a) first as an overall screen - if behavior is potentially dangerous to the student or other stu­ dents ( = 1) then classify him or her as educationally disabled; and (b) as an indicator, along with low reading and mathe­ matics achievement, for classifying the student for special education. From this flow chart, a computer program was written to simulate each consultant's decision processes. The decisions based on the computer program for each of the 50 students were compared with the teacher's placement decisions. For the 25 students for whom verbal report data were collected and upon whom the simulation was based, the model and the teacher-consultant agreed on 88% of the placement decisions. For the remaining 25, 85% agreement was obtained. According to Cadwell and Leary: An examination of those profiles over which there was disagreement indicated that the disagreement resulted more from the subject's [teacher-consultant's] inconsistent use of a strategy than an inade-

quacy in the process model. . . . However, since process-tracing models do not contain an error component, there was no means for incorporating the subject's inconsistencies when applying the placement policy." (1982, p. 10)

(Here we disagree. The computer program could have incor­ porated a stochastic process reflecting these inconsistencies. The problem is to ascertain the magnitude and function of this process.) The teacher-consultant's placement decisions also were discriminated on the information about student intelligence, achievement, sex, and SES using discriminant analysis. The most important variable in discriminating among the teacher­ consultant's placements, far and away, was evidence of problematic behavior. In addition, reading, mathematics achievement, and sex also reliably discriminated among place­ ments. The statistical model placed 82% of the hypothetical stu­ dents into the same classrooms as did the teacher-consultant. Both methods, then, were similar in identifying what infor­ mation was needed to simulate the teacher-consultant's judg­ ment. Both replicated the teacher-consultant's placement decisions with about the same accuracy, above 80%. The models, however, disagreed about some variables used to make the placement decisions and the way in which informa­ tion was combined to reach those decisions. First, the statistical

86

RICHARD J. SHAVELSON, NOREEN M. WEBB, AND LEIGH BURSTEIN

model included the variable, sex, while the process-tracing data gave no indication that the teacher-consultant considered this information. And the statistical model ignored data on intelli­ gence while the process-tracing model included it. Second, the statistical model applied the same strategy to all data while the process-tracing model used different placement strategies for different levels of problematic behavior. And third, the com­ puter model did not contain an error component reflecting variation in the consistency with which the teacher-consultant applied strategy, while the statistical model did. The two methods, then, are similar in that they show a rea­ sonable amount of consistency in the variables identified for decision making, the relative importance ofthese variables, and, with notable exceptions, how those variables were combined (Table 3.12). In the abstract, the two methods appear to measure the same construct. Nevertheless, one caveat is in order. The similarity between the two methods might be as much as function of the requirements of simple computer programs such as those used by Cadwell and Leary (1 982) and Einhorn et al. ( 1979) as they are of the abstract similarity of data obtained with the methods. This is because such programs require data to be segmented into components and integrated back into a sequential, step­ wise process. As Cadwell and Leary (1982) pointed out, their program did not capture the categorization processes that their teacher - consultants might reasonably be expected to use, both because of the requirement to report their thoughts verbally (words and linear sequences of words) and the need to translate the protocols into a simple sequential rather than a more complicated categorization program for the computer.

Concluding Remarks Normally a chapter on measurement would contain relatively pure psychometrics, presenting the latest developments in met­ rics and scaling (e.g., item response theory), reliability (e.g., gen­ eralizability theory or item response theory), and validity (e.g., construct validation theory). However, in originally conceiving the chapter in this manner, we discovered inadequacies. In the end, we realized that any discussion of psychometrics cannot, and should not, be divorced from the phenomena they are used to model. As we hope we have made quite clear, substantive knowledge is a precursor to measuring the attributes of teach­ ing. If not, psychometrically sophisticated analyses are likely to miss their target. This chapter treats just the tip of the measurement iceberg for a number of reasons. First, the domain of measurement in re­ search on teaching is enormous; far too broad for us to treat in any depth. Hence, our decision to deal with just three areas - measurement of teacher effectiveness, classroom process, and teacher cognition. Second, each topic within the domain is terribly complex. This chapter raises as many questions about the measurement of effectiveness, process, and cognition as it answers. Third, attempting to review the literature on the measure­ ment of teaching is like trying to hit a moving target. Research is already underway using measurement techniques we have presented here and exploiting many of the measurement issues we raised (e.g., the various lines of instructional content

research, the research of Rogosa et al. (in press) on teacher behavior, and Cadwell's work on teacher ratings). Finally, research on teaching is a psychometrician's nirvana and nemesis at the same time. As the research progresses into new areas at an explosive rate, measurement issues abound. There is practically no limit to the measurement problems in need of solution. But the problems are not easily dealt with by straightforward psychometric techniques based on simplified assumptions. Rather, these techniques must be built on a bed­ rock of substantive knowledge and the complexities that such knowledge reveals.

REFERENCES Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole Publishing Company. Anderson, L. M. (198 1 ). Short-term student responses to classroom in­ struction. The Elementary School Journal, 82, 97- 108. Anderson, L. M., Evertson, C. M., & Brophy, J. E. (1978). The First­ Grade Reading Group Study: Technical report of experimental and process-outcome relationships (Report No. 4070). Austin TX: Un­ iversity of Texas, R & D Center for Teacher Education. Anderson, L. M., Evertson, C. M., & Brophy, J. E. ( 1979). An experi­ mental study of effective teaching in first-grade reading groups. Ele­ mentary School Journal, 79, 193-223. Anderson, L. M., Morgan, R. 0., Evertson, C. M., & Brophy, J. E. ( 1978).Context effects and stability of teacher behaviors in an experi­ mental study offirst grade reading group instruction. Paper presented at the Annual Meeting of the American Educational Research Asso­ ciation, Toronto. Anderson, R. C. ( 1977). The notion of schemata and the educational enterprise: General discussion of the conference. In R. C. Anderson, R. S. Spiro, & W. E. Montague (Eds.), Schooling and the acquisition of knowledge. Hillsdale, NJ: Lawrence Erlbaum. Anderson, R. C., Spiro, R. S., & Montague, W. E. (1977). Schooling and the acquisition of knowledge. Hillsdale, NJ: Lawrence Erlbaum. Baker, E. L., & Herman, J. L. (1983). Task structure design: Beyond linkage. Journal of Educational Measurement, 20, 149-164. Barr, R., & DFeeben, R. ( 1977). Instruction in classrooms. In L. S. Shulman (Ed.), Review of Research in Education (Vol. 5). ltasca, IL: F. E. Peacock. Barr, R., & Dreeben, R. ( 1983). How schools work. Chicago: University of Chicago Press. Berliner, D. C. (1980). Studying instruction in the elementary class­ room. In R. Dreeben and J. A. Thomas (Eds.), The analysis ofeduca­ tional productivity: Vol. 1. Issues in microanalysis. Cambridge, MA: Ballinger. Birenbaum, M., & Tatsuoka, K. K. ( 1980). The use of information from wrong responses in measuring students' achievement (Research Re­ port 80-1 ONR). Urbana : University of Illinois Computer-Based Education Research Laboratory. Block, J. H., & Burns, R. B. (1976). Mastery learning. In L. S. Shulman (Ed.), Review of research in education (Vol. 4). Itasca, IL: F. E. Peacock. Bloom, B. S. (1953). Thought processes in lectures and discussions. Journal of General Education, 7, 160- 170. Bloom, B. S. ( 1976). Human characteristics and school learning. New York: McGraw-Hill. Bock, R. D. (1960). Components of variance analysis as a structural and a discriminal analysis for psychological tests. British Journal of Sta­ tistical Psychology, 13, 1 5 1 - 163. Bock, R. D. ( 1976). Basic issues in the measurement of change. In D. N. M. de Gruijter & L. J. T. van der Kamp (Eds.), Advances in psychol­ ogical and educational measurement. New York : Wiley. Bock, R. D., & Bargmann, R. E. (1966). Analysis of covariance struc­ tures. Psychometrika, 31, 507-533. Bock, R. D., & Lieberman, M. ( 1970). Fitting a response model for N dichotomously scored items. Psychometrika, 40, 5-32.

MEASUREMENT OF TEACHING

Borich, G. D., Malitz, D., & Kugle, C. L. ( 1978). Convergent and dis­ criminant validity of five classroom observation systems : Testing a model. Journal of Educational Psychology, 70, 1 19- 128 Borko, H. ( 1978). Factors contributing to teachers' preinstructional de­ cisions about classroom management and long-term objectives. Un­ published doctoral dissertation, University of California, Los An­ geles. Borko, H., & Cadwell, J. ( 1982). Individual differences in teachers' deci­ sion strategies : An investigation of classroom organization and management decisions. Journal of Educational Psychology, 74, 598-610. Borko, H., & Shavelson, R. J. (1983). Speculations on teacher educa­ tion: Implications of research on teachers' cognitions. Journal of Education for Teaching, 9, 210-224. Brennan, R. L. (1980). Applications of generalizability theory. In R. A. Berk (ed.), Criterion-referenced measurement : State of the art. Baltimore : Johns Hopkins Press. Brennan, R. L. ( 1983). Elements of Generalizability Theory. Iowa City, IA: The American College Testing Program. Brophy, J. E. (1973). Stability in teacher effectiveness. American Educa­ tional Research Journal, 10, 245-252. Brophy, J. E. (1975). The student as the unit of analysis (Report No. 7512). Austin: University of Texas, Research and Development Center for Teacher Education. Brophy, J. E. (1979).Teacher behavior and its effects. Journal of Educa­ tional Psychology, 71(6), 733-750. Brophy, J. E., Coulter, C. L., Crawford, W. J., Evertson, C. M., & King, C. E. ( 1975). Classroom observation scales : Stability across time and context and relationships with student learning gains. Journal of Educational Psychology, 67(6), 873-881. Brophy, J.E., & Evertson, C. M. ( 1974). Process- product correlations in the Texas Teacher Effectiveness Study : Final report (Report No. 744). Austin: University of Texas, R & D Center for Teacher Educa­ tion. Brophy, J. E., & Evertson, C. M. ( 1977). Teacher behavior and student learning in second and third grades. In G. D. Borich & K. S. Fenton (Eds.), The appraisal of teaching : Concepts and process. Reading, MA: Addison-Wesley. Brophy, J. E., Evertson, C. M., Crawford, J., King, C., & Senior, K. (1975). Stability of measures of classroom process behaviors across three different contexts in second and third grade classrooms. Cata­ log of Selected Documents in Psychology, 5, 338. Brown, B. W., & Saks, D. H. ( 1980). Production technologies and re­ source allocations within classrooms and schools : Theory and mea­ surement. In R. Deeben & J. A. Thomas (Eds.), The analysis of educational productivity: Vol. ]. Issues in microanalysis. Cambridge, MA: Ballinger. Brown, B. W., & Saks, D. H. (198 1). Microeconomics of schooling. In D. C. Berliner (Ed.), Review of research in education (Vol. 9). Washington, DC: American Educational Research Association. Brown, B. W., & Saks, D. H. (1983).An economic approach to measuring the effects of instructional time on student learning. Paper presented at the annual meeting of the American Educational Research Association, Montreal. Brown, J. S., & Burton, R. R. ( 1978). Diagnostic model for procedural bugs in basic mathematical skills. Cognitive Science, 2, 155-1 92. Brunswik, E. (1943). Organismic achievement and environmental pro­ bability. Psychological Review, 50, 255-272. Bryk, A. S., Strenio, J. F., & Weisberg, H. I. (1980). A method for esti­ mating treatment effects when individuals are growing. Journal of Educational Statistics, 5, 5-34. Bryk, A. S., & Weisberg, H. I. (1977). Use of the nonequivalent control group design when subjects are growing. Psychological Bulletin, 84, 950-962. Burstein, L. (1980a). The role of levels of analysis in the specification of educational effects. In R.Dreeben and J. A. Thomas (Eds.), Analysis of educational productivity: Vol. I. Issues in microanalysis (pp. 1 19-94). Cambridge, MA: Ballinger. Burstein, L. ( 1980b). Analyzing multilevel educational data: The choice of an analytical model rather than a unit of analysis. In E. Baker and E. Quellmalz (Eds.), Design, analysis, andpolicy in testing and evaluation (pp. 8 1 -94). Beverly Hill: Sage Publications.

87

Burstein, L. (in press). Units of analysis. In International Encyclopaedia of Education: Research and Studies, Oxford, U.K.: Pergamon. Burstein, L., & Linn, R. L. (198 1). Analysis of educational effects from a multilevel perspective: Disentangling between- and within-class rela­ tionships on mathematics performance (CSE Report Series 1 72). Los

Angeles : University of California, Center for the Study of Evalua­ tion. Burstein, L., Linn, R. L., & Capell, E. S. (1978). Analyzing multilevel data in the presence of heterogeneous within-class regressions. Jour­ nal of Educational Statistics, 3(4), 347-383. Burstein, L., Miller, M. D., & Linn, R. L. ( 198 1). The use of within­ group slopes as indices of group outcomes (CSE Report Series 171). Los Angeles: University of California, Center for the Study of Eval­ uation. Byers, J. L., & Evans, T. E. (1980). Using a lens-model analysis to identify factors in teachingjudgement. Research Series No. 73. East Lansing: Michigan State University, Institute for Research on Teaching. Cadwell, J. ( 1979). Regression models of teacher judgment and de­ cision-making. Unpublished doctoral dissertation, University of California, Los Angeles. Cadwell, J., & Leary, L. F. (1982, March). The cognitive processes un­ derlying the identification of children with learning and behavior problems. Paper presented at the annual meeting of the American Educational Research Association, New York. Calfee, R. C. (1981). Cognitive psychology and educational practice. In D. C. Berliner (Ed.), Review of research in education (Vol. 9). Washington, DC: American Educational Research Association. Calkins, D., Borich, G.D., Pascone, M., & Kugle, C. L. (1977). General­ izability of teacher behaviors across classroom observation systems.

Paper presented at the annual meeting of the American Educational research Association, New York. Cardine!, J., & Tourneur, Y. (1974, July-August). The facets of differen­ ciation [sic] and generalization in test theory [shortened English version of the original test: Une theorie des tests pedagogiques]. Paper presented at the 1 8th Congress of the International Associa­ tion of Applied Psychology, Montreal. Cardinet, J., & Tourneur, Y. ( 1977). How to structure and measure edu­ cational objectives in periodic surveys. In R. Summer (Ed.), Moni­ toring national standards of attainment in schools. Slough, UK: National Foundation for Educational Research in England and Wales. Cardinet, J., Tourneur, Y., & Alla!, L. (1976a). The generalizability of surveys of educational outcomes. In D. N. M. De Gruijter & L. J. T. van der Damp (Eds.), Advances in Psychological and Educational Measurement (pp. 1 85-198). New York : Wiley. Cardine!, J., Tourneur, Y., & Alla!, L. ( 1976b). The symmetry of generalizability theory: Applications to educational measurement. Journal of Educational Measurement, 13, 1 19- 135. Cardinet, J., Tourneur, Y., & Alla!, L. ( 1981). Extension of generaliz­ ability theory and its applications in educational measurement. Journal ofEducational Measurement, 18, 1 83-204. Carroll, J. B. (1963). A model for school learning. Teachers College Re­ cord, 64, 723-733. Colbert, C. D., (1979). Classroom teacher behavior and instructional or­ ganizational patterns. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. Coleman, J. S., Campbell, E. G., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1966). Equality ofEducational Opportunity. Washington, DC: U.S. Department of Health, Educa­ tion, and Welfare, Office of Education. Cone, R. (1978). Teachers' classroom management decisions: A study of interactive decision-making. Unpublished doctoral dissertation, Un­ iversity of California, Los Angeles. Cooley, W. W. (1981). Understanding achievement test variables. In N. Sovik, H. M. Eike! & A. Lysne (Eds.), On individualized instruction: Theories and research (pp. 27-38). Oslo: Universitetsforlagel. Cooley, W. W., & Leinhardt, G. ( 1980). The instructional dimensions study. Educational Evaluation and Policy Analysis, 2, 7-25. Cooley, W. W., & Lohnes, P. R. ( 1976). Evaluation research in educa­ tion: Theory, principles, and practices. New York: Irvington. Copeland, W. D. ( 1979, March). Teaching/learning behaviors and the demands of the classroom environment: An observational study. Paper

88

RICHARD J. SHA VELSON, NOREEN M. WEBB, AND LEIGH BURSTEIN

presented at the annual meeting of the American Educational Re­ search Association, San Francisco. Cornbleth, C., & Korth, W. (1979, April). Instructional context and indi­ vidual differences in pupil involvement in learning activity. Paper pre­ sented at the annual meeting of the American Educational Research Association, San Francisco. Cornbleth, C., & Korth, W. (1980). Context factors and individual dif­ ferences in pupil involvement in learning activity. Journal of Educa­ tional Research, 73, 318-323. Como, L. (1979). A hierar�hical analysis of selected naturally occurring aptitude-treatment interactions in the third grade. American Educa­ tional Research Journal, 16(4), 391 -409. Crawford, J., Brophy, J. E., Evertson, C. M., & Coulter, C. L. (1977). Classroom dyadic interaction: Factor structure of process variables and achievement correlates. Journal of Educational Psychology, 69, 761-772. Crawford, W. J., & Stallings, J. (1978). Experimental ejfec_ts of in-service teacher training derived from process-product correlations in the primary grades. Paper presented at the annual meeting of the American Educational Research Association, Toronto. Cronbach, L. J. (with assistance of Deken, J. E. and Webb, N.) (1976). Research on classrooms and schools: Formulation of questions, de­ signs and analysis (Occasional paper). Stanford, CA: Stanford Eval­ uation Consortium. Cronbach, L. J., & Furby, L. (1970). How should we measure "change"-or should we? Psychological Bulletin, 74, 68-80. Cronbach, L. J., Gieser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: The theory of general­ izabilityfor scores and pro.files. New York: John Wiley. Cronbach, L. J., Rajaratnam, N., & Gieser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 1 37-163. Cronbach, L. J., & Snow, R. W. (1977). Aptitudes and instructional meth­ ods. New York: Irvington. Cronbach, L. J., & Webb, N. (1975). Between-class and within-class effects in a reported aptitude x treatment interactions: Reanalysis of a study by G. L. Anderson. Journal ofEducational Psychology, 67, 717-724. Curtis, M. E., & Glaser, R. (1983). Reading theory and the assessment of reading achievement. Journal ofEducational Measurement, 20(2), 133-147. Dahllof, U. S. (1971). Ability grouping, content validity, and curriculum process analysis. New York : Teachers College Press, Columbia University. Davis, R. B. (1979). Error analysis in high school mathematics, conceived as information processing pathology. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. Davis, R. B., & McKnight, C. (1976). Conceptual, heuristic, and algor­ ithmic approaches in mathematics teaching. Journal of Children's Mathematical Behavior, 1 (Supplement), 271-286. Davis, R. B., & McKnight, C. (1979). Modeling the processes of mathe­ matical thinking. Journal of Children's Mathematical Behavior, 2, 91-1 13. Dawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 18, 95-106. de Finetti, B. (1964). Foresight: Its logical laws, its subjective sources. In H. E. Kyburg & G. E. Smokier (Eds.), Studies in Subjective Probabi­ lity. New York: John Wiley. Dorr-Bremme, D. (1982). Assessing students: Teachers' routine prac­ tices and reasoning. In J. Burry, J. Catterall, B. Choppin, & D. Dorr­ Bremme (Eds.), Testing in the• nation's schools and districts: How much? ™iat kinds? To what ends? At what costs? (Report No. 194, pp. 18-51). Los Angeles: l.Tniversity of California, Center for the Study of Evaluation. Doyle, W. (1977). Paradigms for research on teacher effectiveness. In L. Shulman (Ed.), Review of research in education (Vol. 5). Itasca, IL: F. E. Peacock. Doyle, W. (1983). Academic work. Review of Educational Research, 53(2) 159-199. Duda, R. 0., & Shartliffe, E. H. (1983). Expert systems research. Science, 220, 261-268.

Dunkin, M., & Biddle, B. (1974). The study ofteaching. New York: Holt, Rinehart & Winston. Ebbesen, E. B., & Konecni, V. J. (1980). On the external validity of decision-making research: What do we know about decisions in the real world? In T. S. Wallsten (Ed.), Cognitive processes in choice and decision behavior. Hillside, NJ: Lawrence Erlbaum. Eggert, W. (1978, March). Teacher process variables and pupil products. Paper presented at the annual meeting of the American Educational Research Association. Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Per­ sistence of the illusion of validity. Psychological Review, 85, 395-416. Einhorn, H. J., & Hogarth, R. M. (1981). Behavioral desision theory: Process ofjudgement and choice. Annual Review of Psychology, 32, 53-58. Einhorn, H. J., Kleinmuntz, D. N., & Kleinmuntz, B. (1979). Linear regression and process-tracing models of judgment. Psychological Review, 86, 465-485. Emmer, E. T., Evertson, C. M., & Brophy, J. E. (1979). Stability of teacher effects in junior high classrooms. American Educational Re­ search Journal, 16(1), 71-75. Emmer, E. T., Evertson, C. M., Sanford, J., & Clements, B. S. (1982). Improving classroom management: An experimental study in junior high classrooms. Austin: University of Texas, R & D Center for Teacher Education. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psycho­ logical Review, 87, 215-251. Erlich, 0., & Shavelson, R. J. (1976, September). The application ofgen­ eralizability theory to the study ofteaching. Beginning teacher evalua­ tion study (Tech. Report No. 76-9-1). San Francisco: Far West Laboratory. Erlich, 0., & Shavelson, R. J. (1978). The search for correlations be­ tween measures of teacher behavior and student achievement: Mea­ surement problem, conceptualization problem, or both? Journal of Educational Measurement, 15, 77-89. Erlich, 0., & Borich, G. (1979). Measuring classroom interactions: How many occasions are required to measure them reliably? Jour­ nal of Educational Measurement, 16, 1 1- 18. Evertson, C. M., Anderson, C. W., Anderson, L. M., & Brophy, J. E. (1980). Relationships between classroom behaviors and student outcomes in junior high mathematics and English classes. American Educational Research Journal, 17, 43-60. Evertson, C. M., Anderson, L. M., & Brophy, J. E. (1978a). Process-out­ come relationships in the Texas junior high school study: Overview of methodology and rationale. Paper presented at the annual meeting of the American Educational Research Association, Toronto. Evertson, C. M., Anderson, L. M., & Brophy, J. E. (1978b). Texas Junior High Study: Final report of process-outcome relationships (Report No. 4061). Austin: University of Texas, R & D Center for Teacher Education. Evertson, C. M., Emmer, E. T., & Brophy, J. E. (1980). Predictors of effective teaching in junior high mathematics classrooms. Journal for Research in Mathematics Education, 1 1 , 167-178. Evertson, C. M., Emmer, E. T., & Clements, B. S. (1980). Junior high classroom organization study: Summary of training procedures and methodology (Research and Development Report No. 6101). Austin: University of Texas, R & D Center for Teacher Education. Evertson, C. M., Emmer, E. T., Sanford, J., & Clements, B. S. (1982). Improving classroom management: An experimental study in elemen­ tary classrooms. Austin : University of Texas, R & D Center for Teacher Education. Evertson, C. M., & Hickman, R. C. (1981, March). The tasks ofteaching classes ofvaried group composition. Austin: University of Texas, R & D Center for Teacher Education. Evertson, C. M., & Veldman, D. J. (1981). Changes over time in process measures of classroom behavior. Journal ofEducational Psychology, 73, 1 56-163. Fiedler, M. L. (1975). Bidirectionality of influence in classroom interac­ tion. Journal ofEducational Psychology, 67, 735-744. Filby, N. N., & Dishaw, M. M. (1976). Refinement ofreading and math­ ematics tests through an analysis of reactivity. Beginning teacher evaluation study (Tech. Rep. No. III-b). San Francisco: Far West Laboratory.

MEASUREMENT OF TEACHING

Fisher, C., Filby, N., Marliave, R., Cahen, L. Dishaw, M., Moore, J., & Berliner, D. (1978). Teaching behaviors, academic learning time, and student achievement. (Phase Ill-b, final report) Beginning teacher evaluation study. San Francisco: Far West Laboratory. Fitzgerald, J., Wright, E. N., Eason, G., & Shapson, S. (1978). The effects of subject of instruction on the behavior of teachers and pupils. Paper

prepared for the annual meeting of the American Educational Re­ search Association, Toronto. Floden, R. E., Freeman, D. J., Porter, A. C., & Schmidt, W. H. ( 1980). Don't they all measure the same thing? Consequences of selecting standardized tests. In E. L. Baker & E. Quellmalz (Eds.). Design, analysis and policy in testing and evaluation. Beverly Hills: Sage Pub­ lications, 109- 120. Floden, R. E., Porter, A. C., Schmidt, W. H., Freeman, D. J., & Schwille, J. R. ( 1981). Responses to curriculum pressures : A policy-capturing study of teacher decisions about context. Journal of Educational Psychology, 73, 129-141. Frick, T., & Semmel, M. I. (1978). Observer agreement and reliabilities of classroom observational measures. Review of Educational Re­ search, 48, 157-184. Gaier, E. L. (1954). A study of memory under conditions of stimulated recall. Journal of General Psychology, 50, 147-1 53. Gil, D. (March 1980). The decision-making and diagnostic processes of classroom teachers (Research Series No. 71). East Lansing: Michi­ gan State University, Institute for Research on Teaching. Glaser, R. (Ed.). (1978). Advances in instructional psychology (Vol. 1). Hillsdale, NJ: Lawrence Erlbaum. Glass, G. V., & Stanley, J. C. (1970). Statistical methods in education and psychology. Englewood Cliffs, NJ: Prentice-Hall. Goldberg, L. R. (1976). Man versus model of man: Just how conflicting is that evidence? Organizational Behavior and Human Performance, 16, 1 3-22. Good, T. L. (1979). Teacher effectiveness in the elementary school: What we know about it now. Journal of Teacher Education, 30(2), 52-64. Good, T. L. (n.d.). Research on teaching, unpublished manuscript. Good, T. L., & Beckerman, T. M. (1978). Time-on-task: A naturalistic study in sixth-grade classrooms. Elementary School Journal, 78, 193-201. Good, T. L., Cooper, H. M., & Blakey, S. L. (1980). Classroom interac­ tion as a function of teacher expectations, student sex, and time of year. Journal of Educational Psychology, 72, 378-387. Good, T. L., & Grouws, D. A. (1975). Process-product relationships in fourth-grade mathematics classrooms (Final report, National Insti­ tute of Education Grant NEG-OO-3-0123). Columbia: University of Missouri, College of Education. Good, T. L., & Grouws, D. A. (1977). Teaching effects: A process­ product study in fourth-grade mathematics classrooms. Journal of Teacher Education, 28, 49-53. Good, T. L., & Grouws, D. A. (1979). The Missouri Mathematics Effec­ tiveness Project: An experimental study in fourth-grade classrooms. Journal of Educational Psychology, 71, 355-362. Good, T. L., & Grouws, D. A. (1981, May). Experimental research in secondary mathematics classrooms: Working with Teachers (Final re­ port of National Institute of Education Grant NIE-G-79-0103). Green, J. L., & Smith, D. C. (1983). Teaching and learning: A linguistic perspective. Elementary School Journal, 83(4). Greene, J. C. (1980). Individual teacher/class effects in aptitude treat­ ment interactions. American Educational Research Journal, 17, 291 -302. Greeno, J. G. (1976). Indefinite goals in well-structured problems. Psy­ chological Review, 83, 473-491. Guilford, J. P., & Frutcher, B. (1978). Fundamental statistics in psychol­ ogy and education (6th ed.). New York: McGraw-Hill. Haertel, E., & Calfee, R. (1983). School achievement: Thinking about what to test. Journal of Educational Measurement, 20(2), 1 19-132. Haller, E. (1975). Pupil influence on teacher socialization: A socio­ linguistic study. Journal of Educational Psychology, 67, 735-744. Hamilton, S. F. (1983). The social side of schooling: Ecological studies of classrooms and schools. Elementary School Journal, 83(4), 313-334.

89

Hammond, K. R., & Adleman, L. ( 1976). Science, values and human judgement. Science, 194, 389-396. Harnisch, D. L. (1983). Item response patterns : Applications for educational practice. Journal of Educational Measurement, 20(2), 191-206. Harnisch, D. L., & Linn, R. L. (1982). Identifications of aberrant re­ sponse patterns in a test of mathematics. Paper presented at the an­ nual meeting of the American Educational Research Association, Los Angeles. Harnischfeger, A., & Wiley, D. E. ( 1976). The teaching-learning process in elementary schools: A synoptic view. Curriculum Inquiry, 6, 6-43. Hildreth, C., & Houck, J. P. (1968). Some estimators for a linear model with random coefficients. Journal of the American Statistical Asso­ ciation, 63, 584-595. Hoepfner, R. ( 1978). Achievement test selection for program evaluation. In M. J. Wargo & D. R. Green (Eds.), Achievement testing of disad­ vantaged and minority students for educational program evaluation.

Monterey, CA: CTB/McGraw-Hill. Hoffman, P. J. (1960). The paramorphic representation of clinical judg­ ment. Psychological Bulletin, 57, 1 16- 131. Hoffman, P. J., Slavic, P., & Rorer, L. G. (1968). An analysis-of-vari­ ance model for the assessment of configural cue utilization in clini­ cal judgment. Psychological Bulletin, 69, 338-349. Hopkins, K. D. ( 1982). The unit of analysis : Group mean vs. individual observation. American Educational Research Journal, 19(1), 5-18. Rusen, T. (Ed.). (1967). International study ofachievfment in mathemat­ ics: Comparison of twelve countries (Vols. 1 & 2). New York : John Wiley. Jenkins, J. R., & Pany, D. ( 1976). Curriculum biases in reading achieve­ ment tests (Tech. Rep. No. 16). Urbana-Champaign : University of Illinois, Center for the Study of Reading. Jiireskog, K. G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 32, 443-182. Jiireskog, K. G. (1974). Analyzing psychological data by structural ana­ lysis of covariance matrices. In D. H. Krantz, R. C. Atkinson, R. D. Luce, & P. Suppes (Eds.), Contemporary developments in mathemati­ cal psychology (Vol. 2). New York : W. H. Freeman & Company. Karweit, N., & Slavin, R. E. (1981). Measurement and modeling choices in studies of time and learning. American Educational Research Journal, 18, 157- 1 72. Klein, S. S. (1971). Student influence on teacher behavior. American Educational Research Journal, 8, 403-421. Kropp, F. F., Stoker, H. W., & Bashaw, W. L. (1966, February). The construction and validation of tests of the cognitive processes as de­ scribed in the Taxonomy of Education Objectives. (U.�. Office of

Education Contract OE-4-10-019). Tallahassee; Florida State Un­ iversity. Lashley, K. S. (1923). The behavioristic interpretation of consciousness (Part 2). Psychological Review, 30, 329-353. Leinhardt, G. (1978). Applying a classroom process model to instruc­ tional evaluation. Curriculum Inquiry, 8, 155-186. Leinhardt, G. ( 1983). Overlap: Testing whether it's taught. In G. F. Madaus (Ed.), The courts, validity, and minimum competancy testing. Hingham, MA: Kluwer-Nijhoff Publishing. Leinhardt, G., & Seewald, A. M. (1981). Overlap: What's tested, what's taught? Journal of Educational Measurement, 1 8, 85-96. Linn, R. L., & Burstein, L. (1977). Descriptors of aggregates (CSE Re­ port Series). Los Angeles : University of California, Center for the Study of Evaluation. Lohnes, P. (1972). Statistical descriptors of school classes. American Educational Research Journal, 9, 547-556. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ : Lawrence Erlbaum. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley. Luce, S. R., & Hoge, R. D. (1978). Relations among teacher rankings, pupil-teacher interactions, and academic achievement: A test of the teacher expectancy hypothesis. American Educational Research Journal, 1 5, 489-500. Lundgren, V. P. ( 1972). Frame factors and the teaching process. Stock­ holm: Almquist & Wiksell. Magnusson, D. (1966). Test theory. Reading, MA: Addison-Wesley.

90

RICHARD J. SHAVELSON, NOREEN M. WEBB, AND LEIGH BURSTEIN

Marliave, R., Fisher, C. W., & Dishaw, M. M. (1978, February). Aca­ demic learning time and student achievement in the B-C period. Begin­ ning teacher evaluation study (Technical Note Series, Technical Note

V-2a, February). Martin, J., Veldman, D. J., & Anderson, L. M.(1980). Within-class rela­ tionships between student achievement and teacher behaviors. American Educational Research Journal, 17( 4), 479-490. Marx, R. W. ( 1983). Student perception in classrooms. Educational Psy­ chologist, 18(3), 145-165. Marzano, W. A. ( 1973). Determining the reliability of the Distar instruc­ tional system observation instrument. Unpublished thesis, University of Illinois, Urbana-Champaign. McConnell, J. W., & Bowers, N. D. ( 1979). A comparison of high-infer­ ence and /ow-inference measures of teacher behaviors as predictors of pupil attitudes and achievements. Paper presented at the annual

meeting of the American Educational Research Association, San Francisco. McDonald, F., & Elias, P. ( 1976). The effects of teacher performance on pupil learning. Beginning Teacher Evaluation Study (Phase II, final report, Vol. I). Princeton, NJ : Educational Testing Service. McGaw, B., Wardrop, J. L., & Bunda, M. A. (1972). Classroom obser­ vation schemes: Where are the errors? American Educational Re­ search Journal, 9, 1 3-27. McNair, K. ( 1978- 1979). Capturing inflight decisions: Thoughts while teaching. Educational Research Quarterly, 3, 26-42. Medley, D. M. (1977). Teacher competence and teacher effectiveness: A review of process-product research. Washington, DC: American As­ sociation of Colleges for Teacher Education. Medley, D. M., & Mitzel, H. E. (1963). Measuring classroom behavior by systematic observation. In Gage, N. L. (Ed.), Handbook of Re­ search on Teaching. Chicago, IL: Rand McNally. Miller, M. D. (1981). Measuring between-group differences in instruction. Unpublished doctoral dissertation, University of California, Los Angeles. Miller, M. D. (in press). Item response and instructional coverage. Jour­ nal of Education Measurement.

Muthen, B. ( 1978). Contributions to factor analysis of dichotomous variables. Psychometrika, 43, 551 -560. Muthen, B. (1983). Latent variable structural equation modeling with categorical data. Journal of Econometrics, 22, 43-65. Muthen, B., & Christoffersson, A. (1981). Simultaneous factor analysis of dichotomous variables in several groups. Psychometrika, 46, 407-419. Muthen, L. (1983). The estimation of variance components for dichoto­ mous dependent variables: Applications to test theory. Unpublished doctoral dissertation, University of California, Los Angeles. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259. Noble, C. G., & Nolan, J. P. (1976). Effect of student verbal behavior on classroom teacher behavior. Journal of Educational Psychology, 68, 342-346. Payne, J. W. (1982). Contingent decision behavior. Psychological Bulle­ tin, 92, 382-402. Peterson, P. L. (1977). Interactive effects of student anxiety, achievement orientation and teacher behavior on student achieve­ ment and attitude. Journal of Educational Psychology, 69, 779792. Peterson, P. L., & Clark, C. M. (1978). Teachers' reports of their cogni­ tive processes during teaching. American Educational Research Journal, 1 5, 555-565. Peterson, P. L., & Janicki, T. (1979). Individual characteristics and chil­ dren's learning in large-group and small-group approaches. Journal of Educational Psychology, 71, 677-687. Peterson, P. L., Marx, R. W., & Clark, C. M. (1 978). Teacher planning, teacher behavior and student achievement. American Educational Research Journal, 1 5, 417-432. Plewis, I. (1981). Using longitudinal data to model teachers' ratings of classroom behavior as a dynamic process. Journal of Educational Statistics, 6, 237-255.

l

Postman, L., & Tolman, E. C. ( 1959). Brunswik's probabilistic function­ alism. In S. Kock (Ed.), Psychology: A study of a science (Vol. 1). New York: McGraw-Hill. Program on Teaching Effectiveness (November 1976). Afactorially de­ signed experiment on teacher structuring, soliciting, and reacting (Re­ search and Development Memo #47). Stanford Center for Research and Development in Teaching. Pullis, M., & Cadwell, J. (1982). The influence of children's tempera­ ment characteristics on teachers' decision strategies. American Edu­ cational Research Journal, 19, 165- 181. Resnick, L., & Ford, W. (1981). The psychology of mathematics for in­ struction. Hillsdale, NJ: Lawrence Erlbaum. Rogosa, D. E. (1980). Comparisons of some procedures for analyzing longitudinal panel data. Journal of Economics and Business, 32, 1 36- 151. Rogosa, D. E., Brandt, D., & Zimkowski, M. ( 1982). A growth curve approach to the measurement of change. Psychological Bulletin, 90, 726-748. Rogosa, D. E., Floden, R., & Willett, J. B. (in press). Assessing the stabi­ lity of teacher behavior. Journal of Educational Psychology. Romberg, T. A. (1983). A common curriculum for mathematics, In G. D. Fenstermacher & J. I. Goodlad (Eds.). Individual differences and the common curriculum: Eighty-second yearbook of the National So­ ciety for the Study of Education (pp. 121- 1 59). Chicago: University

of Chicago Press. Rosenshine, B. (1979). Content, time, and direct instruction. In P. Peter­ son and H. Walberg (Eds.), Research on teaching: Concepts,findings, and implications. Berkeley, CA : McCutchan. Rosenshine, B. ( 1983). Teaching functions in instructional programs. Elementary School Journal, 83(4), 335-352. Rosenshine, B., & Berliner, D. C. ( 1978). Academic engaged time. British Journal of Teacher Education, 4, 3- 16. Rosenshine, B., & Furst, N. (1973). The use of direct observation to study teaching. In R. M. W. Travers (Ed.), Second Handbook ofRe­ search on Teaching. Chicago: Rand McNally. Rowley, G. L. (1976). The reliability of observational measures. American Educational Research Journal, 1 3, 51 -59. Rowley, G. L. (1978). The relationship of reliability in classroom research to the amount of observation: An extension of the Spearman-Brown formula. Journal of Educational Measurement, 1 5, 165-180. Rumelhart, D. E., & Ortony, A. (1977). The representation of knowl­ edge in memory. In R. C. Anderson, R. J. Spiro, & W. E. Montague (Eds.), Schooling and the acquisition of knowledge. Hillsdale, NJ: Lawrence Erlbaum. Russo, N. A. (1978). The effects of student characteristics, educational beliefs and instructional task on teachers' preinstructional decisions in reading and math. Unpublished doctoral dissertation, University of

California, Los Angeles. Schmidt, W, H. ( 1969). Covariance structure analysis of the multivariate random effects model. Dissertation, University of Chicago. Schmidt, W. H. (1983). Content biases in achievement tests. Journal of Educational Measurement, 20(2), 165-178. Schneider, W., & Treiber, B. (1984). Classroom differences in the deter­ mination of achievement changes. American Educational Research Journal, 21(1), 195-2 1 1 . Schwille, J., Porter, A., Belli, G., Floden, R., Freeman, D., Knappen, L., Kuhs, T., & Schmidt, W. (1983). Teachers as policy brokers in the content of elementary school mathematics. In L. S. Shulman & G. Sykes (Eds.), Handbook of teaching policy. New York : Longman. Shavelson, R. J. (1973). What is 'the' basic teaching skill? Journal of Teacher Education, 14, 144- 1 5 1 . Shavelson, R. J. (1972). Some aspects of the correspondence between content structure and cognitive structure in physics instruction. Journal of Educational Psychology, 63, 225-234. Shavelson, R. J. ( 1976). Teachers' decision making. In N. L. Gage (Ed.), The psychology of teaching methods, Seventy-fifth Yearbook of the National Society for the Study of Education (Part I). Chicago: Un­ iversity of Chicago Press. Shavelson, R. J. (1983). Review of research on teachers' pedagogical judgments, plans, and decisions. Elementary School Journal, 83(4).

MEASUREMENT OF TEACHING

Shavelson, R. J., Caldwell, J., & Izu, T. (1977). Teachers' sensitivity to the reliability of information in making pedagogical decisions. American Educational Research Journal, 14, 83-97. Shavelson, R. J., & Dempsey-Atwood, N. (1976). Generalizability of measures of teaching behavior. Review of Educational Research, 46, 553-6 1 1 . Shavelson, R . J., & Stern, P . (1981). Research on teachers' pedagogical thoughts, judgments, decisions, and behavior. Review ofEducational Research, 51, 455-498. Shavelson, R. J., & Webb, N. M. ( 1981). Generalizability theory: 1973-1 980. British Journal of Mathematical and Statistical Psychol­ ogy, 34, 133- 166. Sherman, T. N., & Cornier, W. H. (1974). An investigation of the in­ fluence of student behavior on teacher behavior. Journal of Applied Behavior Analysis, 7, 1 1 -21. Shulman, L. S., & Elstein, A. S. (1975). Studies of problem solving, judg­ ment, and decision making: Implications for educational research. In F. N. Kerlinger (Ed.), Review of Research in Education (Vol. 3). Itasca, IL: Peacock. Slavin, R. E. (1983). Student level analysis in classroom experiments: An extension of the Cornfield-Tukey bridge. Paper presented at the an­ nual meeting of the American Educational Research Association, Montreal. Slovic, P., Fischhoff, B., & Lichtenstein, S. (1977). Behavioral decision theory. Annual Review ofPsychology, 28, 1 -39. Smith, B. 0. (1983). Some comments on educational research in the twentieth century. Elementary School Journal, 83(4), 488-492. Snow, R. J., Federico, P. A., & Montague, W. E.(Eds.), ( 1980). Aptitude, learning and instruction (Vols. l & 2). Hillsdale, NJ: Lawrence Erlbaum. Soar, R. S., & Soar, R. M., (1972). An empirical analysis of selected follow through programs : An example of a process approach to evaluation. In I. Gordon (Ed.), Early Childhood Education. Chicago: National Society for the Study of Education. Stallings, J. A. (1976). How instructional processes relate to child out­ comes in a national study of follow through. Journal of Teacher Edu­ cation, 27, 43-47. Stallings, J. A., & Kaskowitz, D. (1 974). Follow through classroom ob­ servation evaluation 1972-1973. (SRI Project URU-7370). Menlo Park, CA: SRI International. Swamy, P. A. V. B. (1970). Efficient inference in a random coefficient regression model. Econometrics, 38, 3 1 1-328. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with mis­ conceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354. Tatsuoka, K. K., & Linn, R. L. (in press). Indices for detecting unusual patterns : Links between two general approaches and potential ap­ plications. Applied Psychological Measurement, 7(1), 8 1 -96. Tatsuoka, K. K., & Tatsuoka, M. M. (1982). Detection of aberrant re­ sponse patterns and their effect on dimensionality. Journal of Edu­ cational Statistics, 7, 2 1 5-231 . Tatsuoka, K . K., & Tatsaoka, M . M . (1983). Spotting erroneous rules of operation by the individual consistency index. Journal of Educa­ tional Measurement, 20(3), 221-230. Tikunoff, W., Berliner, D. C., & Rist, R. (1975). An ethnographic study of the forty classrooms of the Beginning Teacher Evaluation Study known sample (Tech. Rep. 75-10-5). San Francisco : Far West Labor­

atory. Tourneur, Y. (1978).Les Objectifs du domaine cognitif, 2me partie theorie des tests. Ministere de !'Education Nationale et de la Culture Fran­ caise, Universite de l'Etat a Mons, Faculte des Sciences Psycho­ Pedagogiques.

91

Tourneur, Y., & Cardinet, J. (1979). Analyse de variance et theorie de la generalizabilite: Guide pour la realisation des calceuls [Doc 790. 803/ (CT/9)]. Universite de l'Etat a Mons. Tuckwell, N. B. (n.d.-a). Content analysis of stimulated recall protocols (Occasional Paper Series Tech. Rep. #80-2-2, [a]). Edmonton, Canada: University of Alberta, Center for Research in Teaching, Faculty of Education. Tuckwell, N. B. (n.d.-b). Stimulated recall: Theoretical perspectives and practical and technical considerations (Occasional Paper Series Technical Report 8-2-3, [b]). Edmonton, Canada : University of Alberta. Van Roekel, J. L., & Patriarca, L.(April 1979). Do reading and learning disability clinicians differ in their diagnosis and remediation? Paper presented at the Annual Conference of the International Reading Association, Atlanta. Veldman, D. J., & Brophy, J. E. (1974). Measuring teacher effects on pupil achievement. Journal of Educational Psychology, 66(3), 319-324. Vinsonhaler, J. F. ( 1979). The consistency of reading diagnosis. Research Series No. 28. East Lansing: Institute for Research on Teaching, Michigan State University. Walker, D. F., & Schaffarzik, J. (1974). Comparing curricula. Review of Educational Research, 44(1), 83-1 1 1. Webb, N. M. (1980). Group process: The key to learning in groups. In K. H. Roberts and L. Burstein (Eds.), Issues in aggregation, New directions for methodology of social and behavioral science (Vol. 6), 77-87. Webb, N. M. (1982). Group composition, group interaction and achievement in cooperative small groups. Journal of Educational Psychology, 74, 475-484. Webb, N. M. ( 1980). Implementation of BTES interaction activities in junior high school mathematics (Final report submitted to the California Commission for Teacher Preparation and Licensing). Webb, N. M., & Shavelson, R. J. (1981). Multivariate generalizability of general educational development ratings. Journal of Educational Measurement, 1 8, 1 3-22. Webb, N. M., Shavelson, R. J., & Maddahian, E. (1983, June). Multivar­ iate generalizability theory. In L. J. Fyans, Jr. (Ed.), Generalizability theory: Inferences and practical applications (New Directions for Testing and Measurement, No. 1 8). San Francisco: Jossey-Bass. Weinshank, A. (June 1 979). An observational study of the relationship between diagnosis and remediation in reading. Research Series No. 72. East Lansing: Institute for Research on Teaching, Michigan State University. Weinstein, R. (1983). Perceptions of schooling. Elementary School Jour­ nal, 83(4), 287-312. Wiley, D. E. (1970). Design and analysis of evaluation studies. In M. C. Wittrock & D. E. Wiley (Eds.), The evaluation of instruction. New York: Holt, Rinehart & Winston. Wiley, D. E., Schmidt, W. H., & Bramble, W. J.( 1973). Studies of a class of covariance structure models. Journal of the American Statistical Association, 1973, 68, 3 1 7-323. Wohlwill, J. S. (1973). The study of behavior development. New York: Academic Press. Wright, C. J., & Nuthall, G. (1970). Relationships between teacher be­ haviors and pupil achievement in three experimental elementary sci­ ence le.ssons. American Educational Research Journal, 7, 477-492. Yinger, R. J., Clark, C. M., & Mondo!, M. M. (1979). Selecting instruc­ tional activities: A policy-capturing analysis. Research Series No. 103. East Lansing: Michigan State University, Institute for Re­ search on Teaching.

4.

.

Q u a n t i tat i ve M et h od s 1 n Research o n Teac h i n g Ro bert L. L i n n

University of Illinois at Urbana-Champaign

The division of methods into the categories considered in this chapter and the following chapter on qualitative methods is convenient. Quantitative methods are generally associated with systematic measurement, experimental and quasi-experimental methods, statistical analysis, and mathematical models. Quali­ tative methods, on the other hand, are associated with natural­ istic observation, case studies, ethnography, and narrative reports. It should be recognized, however, that the boundaries between the methods sometimes get blurred. The person using quantitative methods must make many qualitative decisions regarding the questions to pose, the design to implement, the measures to use, the analytical procedures to employ, and the interpretations to stress. The person who relies on qualitative methods often finds certain quantitative sum­ maries, classifications, and analyses a useful part of the research (cf. Light, 1973). Methods in both categories can contribute to what Cronbach and Suppes (1969) have described as "discip­ lined inquiry." Nonetheless the emphases, traditions, and ideals are different. The "two cultures" (Stenhouse, 1978) more often appear to be conflicting than complementary. Yet, results relying on both categories of methods contribute to the understanding of teaching and therefore have complementary value even if there is no deliberate effort of the type recom­ mended by Saxe and Fine ( 1 979) to make them complementary. Although the title of this chapter provides a convenient distinction, it may suggest a more ambitious effort than was intended or undertaken. The scope is certainly much more lim­ ited than the gamut of available quantitative methods, the con­ sideration of which would require a grandiose project indeed. Measurement is a crucial element in the use of quantitative methods. Fortunately, other chapters deal with measurement and while some measurement issues are also addressed here, the coverage is limited to issues as they relate to the analysis and interpretation of results. Properties of certain commonly used metrics for reporting achievement test data, for example, often

affect interpretations and are reviewed for this reason. Simi­ larly, measurement error plays an important role in explana­ tory models and attempts to adjust for preexisting differences between groups, and is discussed in those contexts. But no attempt is made to provide a comprehensive treatment of measurement theory or techniques. Questions of design are central components of the topics covered in this chapter. But it is not a chapter on design, in the tradition of either Fisher ( 1925) or Campbell and Stanley ( 1963), though concepts from those traditions are heavily relied upon. The power of randomization is recognized, but greater attention is given to quasi-experimental designs that have more frequent applicability in research on teaching. The discussion owes much to the conceptualization provided by Campbell and Stanley (1963) and updates and extensions of that work (e.g., Cook & Campbell, 1976, 1 979). Their types of validity and identification of threats to validity provide important logical underpinnings for the quantitative methods that are discussed. Here the emphasis is more on the quantification and analytical methods associated with the designs, however. Statistical analysis is also a vital part of the discussion, but the chapter is not a survey of statistical techniques and develop­ ments, per se. Rather, the emphasis is on the logic of certain techniques and their utility for the analysis of data obtained in commonly used paradigms for the study of teaching and its effects. The chapter starts with a section on considerations in the choice of a design, in which the emphasis is on trade-offs and the implications for analysis and interpretability. This is fol­ lowed by the major section of the chapter, in which analyses of data from randomized experiments, various nonequivalent con­ trol group designs, single-group designs, and designs involving aggregate data are considered. The final section of this chapter is devoted to the quantitative analysis of summary results from previous research.

The author thanks reviewers Richard M. Jaeger (University of North Caroline - Greensboro) and David Rogosa (Stanford University).

92

93

QUANTITATIVE METHODS Some Basic Considerations in Design, Interpretation, and Analysis: An Overview Although description is sometimes the end goal, and always an important part of quantitative methods, causal inferences are of primary concern in this chapter. Some would question both the feasibility and the desirability of making causal inferences. In­ deed, the editor of the Second Handbook of Research on Teach­ ing, Robert M. W. Travers, has recently recommended (Travers, 1981) that we "drop that word cause and bring educational re­ search out of the middle ages" (p. 32). Travers' recommenda­ tion was based upon his conclusion that "discussions of causal relationships in research are pre-Newtonian" (p. 32), i.e., based on "Aristotle's concept of pushfrom behind" (.p. 32). He further argued that "modern scientists do not use the concept of cause, except during chatty moments" (p. 32). The idea of functional relationships was offered as one of the available substitutes that is better than the concept of cause. Causation is a controversial topic. Philosophical debates about causation are plentiful. Cook and Campbell ( 1979, pp. 9-36) have discussed the relationship of their work to some of the philosophical traditions. That discussion will not be re­ peated here, nor will an analysis of the theoretical arguments be attempted. Such an analysis is left to others who are better equipped to undertake the task (cf. Ennis, 1973). Before pro­ ceeding to a discussion of methods, however, it is desirable to consider briefly the three major points made by Ennis (1982) in a rejoiner to Travers' recommendation that we abandon the concept of causality. Ennis acknowledged that the analysis of the concept of cause remains quite controversial, but argued that the concept as or­ dinarily used is not limited to the mechanistic notion of a "push from behind." As Ennis argued, "it is legitimate and essential for educational researchers to have causal concerns" and to pursue "questions about what brings about, and what has brought about matters of educational significance" (Ennis, 1982, p. 27). Such investigations are concerned with general and specific causal statements, respectively, and need not involve the mechanistic notion of a physical push. Ennis persuasively argued that we cannot eliminate the con­ cept of cause merely by doing away with the word "cause." In­ vestigations of whether reduced class size leads to improved student achievement, of whether peer tutoring helps either the tutor or the person being tutored, of who benefits from round­ robin reading groups, or of teacher behaviors that increase time-on-task are all concerned with cause, whether or not the word is used. Of course, being concerned about cause is no guarantee of success, or that the causal statements are valid. Empirical assumptions are always required: Even the most tightly controlled experimental design cannnot eliminate all al­ ternative explanations (Ennis, 1973, p. 9). Recognition of the fallibility of causal statements produced from the results of any design leads to an emphasis on the search for "plausible rival hypotheses" (cf. Cook and Campbell, 1979, pp. 20-25) and an emphasis on falsification (Popper, 1959). It is also important to distinguish between proximate and fundamental causes and to recognize that general causal statements need to be qualified by an indication of frequency by "specifying a condition that either

sometimes is, often is, usually is, always is, etc., a sufficient con­ dition for the bringing about of a specified sort of effect" (Ennis, 1973, p. 10).

A Comparison of Two Treatments Whether judged by means of a randomized experiment, a quasi­ experiment, or merely a "thought experiment" (Cronbach, 1982, p. 120), comparison of alternative conditions is funda­ mental to causal inferences. Rubin (1974) has provided a for­ mulation for the comparison of treatments that defines a specific causal effect in a manner consistent with the ordinary meaning of the concept. His formulation is useful in analyzing the assumptions required in making causal inferences from comparative studies and in comparing alternative designs and analyses. Using E and C to denote the treatments to be com­ pared, Rubin ( 1974) defined "the causal effect of the E versus C treatment on Y for a particular trial (i.e., a particular unit and associated times t 1 , t 2 ) as follows:" Let y(E) be the value of Y measured at t2 on the unit, given that the unit received the experimental Treatment E initiated at t1 ; Let y(C) be the value of Y measured at t2 on the unit given that the unit received the control Treatment C initiated at t1 ; Then y(E) - y(C) is the causal effect of E versus C treatment on Y for that trial, that is, for that particular unit and the times t 1 , t 2• (p. 689) Rubin's definition provides a useful standard for evaluating designs f?r the comparison of two treatments. It is obvious, however, that it is impossible to measure directly the specific effect for any unit because only one of the two treatments can be introduced at t 1 • If y(E) is obtained for a particular unit (e.g., individual classroom, school) then y(C) cannot be obtained for that unit, and vice versa. Two time intervals t 1 to t2 and t3 to t4 · ht be used with, say, C introduced at t 1 and E at t 3 while mtg y(C) and y(E) are obtained at times t2 and t4, respectively. But there are many plausible rival hypotheses (e.g., maturation, carryover effects) that must be eliminated in order for such a design to be a reasonable approximation to Rubin's conceptual definition. The more common operational approximation to the defini­ tion is to assign E to one unit and C to another. But, of course, Y 1 (E) may not equal yi (E) and y 1 (C) may not equal yi(C), where the subscripts identify the units. The outcome of the experiment is either y1 (E) - yi (C), or Yi(E) - y1 (C). Neither quantity is necessarily equal to the specific causal effect for either unit, y;(E) - y;(C); i = 1 or 2. The dilemma of not being able to obtain y(E) and y(C) on the same unit led Rubin (1974) to define the "'typical' causal effect of the E versus C treatment for M trials . . . [as] the average (mean) causal effect for the M trials" (p. 690). This average still cannot be directly assessed because some units receive E while others receive C, and no unit can receive both for a given t 1 , t2 . If N units receive E and another N units receive C, M = 2N, then the "typical causal effect" is 1

2N

1

2N

y (E) (C), � = 2N -� ; - 2N _L Y; l-1

i= 1

94

ROBERT L. LINN

but the available data only allows for the calculation of

where it is assumed that the units are ordered such that the first N receive E and the second N receive C. RANDOMIZATION

Random assignment cannot guarantee that d will equal �- Only one of the (2N)! possible allocations of the "randomization set" is observed, and none of the possible ds resulting from a parti­ cular allocation is necessarily equal to �- But there are, as Rubin has shown, two important benefits of randomization. First, d is an unbiased estimator of �, and, second, " precise probabilistic statements can be made indicating how unusual the observed" (Rubin, 1 974, p. 693) d is under any hypothesized �- Without randomization, the possibility of bias due to prior differences on an uncontrolled "third variable" can seldom, if ever, be ruled out as an alternative explanation of the results. These important advantages of randomization identified by Rubin apply not only to the completely randomized experi­ ment, but to randomized block designs in which 2N matched pairs are created and one member of each pair is randomly se­ lected to receive E and the other to receive C. The latter design may also have an advantage of greater precision. IMPROVING ESTIMATES

Blocking is one example of an approach to improving the esti­ mate of �- To the extent that the variable(s) used to form matched pairs may increase the within-block homogeneity on Y, the precision of the estimates will be improved (cf. Feldt, 1958). For example, in a study where the dependent variable is student achievement, students with equal scores on a previously administered ability test would generally be expected to have more similar achievement than students with different ability test scores. By forming matched pairs on the ability test scores and randomly assigning one member of each pair to the E and C conditions, the effect can be estimated with greater precision. Often other adjustment procedures, e.g., use of gain scores or covariate adjustments, have great advantage, whether the ad­ justment variables causally affect Y or are only correlated with Y. The choice of variables (e.g., prior achievement, age, height) on which to make an adjustment, and the way in which those adjustments are to be made often involves trade-offs between desirable properties of unbiasedness and the magnitude of the error variance. Rubin (1974) shows that subtracting any prior score X; from y;(E) or Y;(C), depending on which treatment the unit receives, preserves the property of unbiasedness, but not all prior variables make sense. If y is a measure of achievement, then x = prior achievement makes sense; running speed, pulse rate, and height do not. "Clearly, in order to make an intelligent adjustment for extra information, we cannot be guided solely by the concept of un­ biasedness over the randomization set. We need some model for the effects of prior variables in order to use their values in an intelligent way" (Rubin, 1 974, p. 696). This conclusion of Rubin's may be illustrated by a simple example in which the values of y;(E) and Y;(C) are assumed to be known, but different

Table 4. 1 . Hypothetical Values on a Prior Measure, X, and Outcomes Under Two Treatment Conditions for Four Experimental Units Unit

X;

Y;(E)"

Y;(C) b

Y;(E) - Y;(CJ

I

10 20 30 40

8 14

2 10 18 26

6 4 2 0

2 3 4

20

26


0 are referred to as censored selection. In some situations, however, neither x nor y are available for units with U � 0. The latter situation is re­ ferred to as truncated selection, and obviously poses more diffi­ cult estimation problems. A more general model for J groups and a maximum likeli­ hood estimation technique was developed by Muthen and Joreskog (in press). Estimation techniques for both the censored and the truncated selection cases are provided. The basic fea­ tures of the latter model are briefly described below for the sim­ ple case of a single dependent variable and a single covariate assuming a single selection relationship. As noted by Muthen and Joreskog, generalizations to a multivariate system and multivariate selection relations are possible. The causal model for the dependent variable in group j,j= 1, 2, . . . J, is:

Selection into group j occurs if an unmeasured selection vari­ able, Ui, is greater than 0, and the selection variable is related to the observed covariate by

for the jth group. As the result of the selection process, the co­ variance between g and e in any group may be nonzero, that is, a-�. =I= 0. Muthen and Joreskog (in press) used simulated data for two groups to evaluate their estimation procedure and compare it to an ordinary ANCOVA. Members of Group 1, the control group, were assumed to be randomly sampled from an unse­ lected population in which the following causal model related y to x,

and selection into the experimental group is based upon U > 0, where U = 0 - l.0x + g.

Variances of e and g are assumed to equal 1.0, the covariance between g and e to equal - .5 and x to be normally destributed. In other words, the simulated experimental treatment had an effect on y, changing both the slope and intercept of the regres­ sion, but the units selected for the treatment generally had lower values of x, since g - x had to be greater than zero for a unit to be selected into the experimental group. A standard analysis of covariance with 4000 simulated con­ trol units and 2037 experimental units showed no significant difference between treatments, with a pooled within-group slope of .81 and intercepts of - .42 and - .43 for the control and experimental groups, respectively. In contrast, the Muthen and Joreskog estimation correctly indicated a significant positive effect of the experimental treatment for both the censored and truncated selection situations. For the censored case, the esti­ mates of the within-group slopes and intercepts were quite close to the original values of the simulation. Estimates assuming a truncated selection case were not as good, but were still con­ siderably better than those from a standard ANCOVA. Figure 4.3 shows the differences between the true simulation model regressions and the estimates obtained· from three ana­ lyses for each treatment group. Panel A of Figure 4.3 shows differences for a standard ANCOVA while Panels B and C show the Muthen and Joreskog differences for the censored and truncated selection situations, respectively. A perfect outcome would yield horizonal lines with a difference of zero for both treatment groups. As can be seen, the results for the censored case are quite satisfactory. Those for the ANCOVA are not.

y= - .4 + .8x + e,

COMPLETE DISCRIMINANT AND IDEAL COVARIATE

where the mean and variance of x were O and 1 respectively, and the variance of e was .9. The causal model for the experimental condition, group 2, was assumed to be

Cronbach et al. (1977) have presented a conceptual formulation that provides a basis for analyzing the potential sources of bias of various adjustments for preexisting differences between non­ randomly formed groups. They consider two hypothetical vari­ ables : the complete discriminant and the ideal covariate. The

y= 0 + l.0x + e, .2

(/) Cl) (.) C: Cl) -

� i5

.2

0

---------■

(/) Cl)

.2

0 1-. _ _.... __ ....._ �� � � -�-

�-2

� i5

- .4

0

.5 Covariate A. ANCOVA

1 .0

0

u C: � - 2 ....___ ____ I �

u C:

2

(/) Cl)

---------■

- .4

-.4

- .6 __________, 0

.5 Covariate 8. Censored Model

1 .0

- . 6 ..______..______,

0

1 .0 .5 Covariate C. Truncated Model

Fig. 4.3. Difference between the true simulation model regressions and estimates obtained from three analyses (based on Muthen and Joreskog, 1 983). (Legend : --- = experimental group ; - - - = control group.)

-106

ROBERT L. LINN

complete discriminant, D, serves a role similar to U in the above formulation. It is the variable that contains the best possible information regarding selection differences between groups. In other words, no other variable would add to the ability to pre­ dict group membership beyond that possible from knowledge of D. The ideal covariate, C, is the best possible predictor of the dependent variable without knowledge of treatment conditions. Within a treatment condition, prediction of the dependent vari­ able could not be improved by the addition of any prior infor­ mation to that contained in C. It is possible that C would differ from one treatment group to another, but this possibility will be ignored here. If either D or C were known, it could be used as the covariate in ANCOVA to produce unbiased estimates of the treatment effect. In practice, of course, neither D nor C is known. Instead, some measures of variable, X, or set of variables, is available for use as the covariate. Substituting X for D or C generally leads to biased estimates of the treatment effects. The magnitude of the bias depends on the relationships of X to the unknown variables D and C. The Cronbach et al. (1977) analysis shows that the bias in the estimates due to using X as a covariate may be in either direc­ tion rather than just in favor of the group with higher scores on X, as is often presumed. Their analysis also shows the limits on the amount of bias as a function of the relationship of X to the hypothetical variables D and C. As noted by Cook and Campbell (1979), "the implication is that knowledge about the relationship of an arbitrary covariate to both these constructs can be used to fix limits on the size of the bias" (p. 194). In practice, the relationship of X with D and with C is uncer­ tain. However, reasonable ranges of correlations of X with D and X with C may be hypothesized. The range of hypothesized correlations may, in turn, be used to set limits on the amount of bias in the treatment effect estimates. Thus, a range of estimates of the size of the treatment effect, rather than a point estimate, would be produced. This approach is illustrated by Reichardt (1979). REGRESSION-DISCONTINUITY DESIGN

As is illustrated by the Cronbach et al. (1977) analysis, know­ ledge of the basis of assignment to treatment groups is the key to obtaining unbiased estimates of treatment effects. The surest way to gain such knowledge is to control the process of assign­ ment. The regression-discontinuity design derives its strength from the fact that the basis of assignment is known. Units are assigned to treatments solely on the basis of a pretest or other measure obtained prior to treatment. Units below a cutting score on the premeasure are assigned to one treatment condi­ tion and those above are assigned to a second treatment condi­ tion. Commonly, the group with scores below the cutting score receives the experimental treatment and the higher-scoring group serves as the comparison, but logic applies as well to the opposite assignment or to any two treatments. With assignment determined solely on the basis of the pre­ measure, that measure is necessarily the complete covariate in the terminology of Cronbach et al. (1977). If the assumptions of ANCOVA are satisfied, then using as the covariate the premea-

Q) :0 (1)

>-� c

Q) 'O C Q) a. Q)

Experimental � ... Control Covariate

Fig. 4.4. Illustration of regression on discontinuity analysis. (Solid lines are the estimated within-group regressions for the observed range of scores in each group ; dashed lines are extrapolations into the range of scores of the other group.)

sure on which assignment was based will yield unbiased esti­ mates of the treatment effect (Goldberger, 1972a; Rubin, 1977). Measurement error in the covariate does not bias the ANCOVA estimates in a regression-discontinuity design. As stated by Cook and Campbell (1979) "it is knowledge of the selection process that makes the regression-discontinuity de­ sign so potentially amenable to causal interpretation" (p.1 38). The logic of the regression-discontinuity design is illustrated in Figure 4.4. Units with pretest scores below the cutting score receive the experimental treatment and those above receive the comparison treatment. The solid lines represent the within­ group regressions over the range of observed pretest scores within each group. The dashed lines show the extrapolations of those within-group regressions into the range of pretest scores of units receiving the other treatment. The estimated differential effect of the treatment is the difference in the predicted scores on the dependent variable evaluated at the cutting score of the pretest. The example in Figure 4.4 is straightforward, but may be overly simplistic. There is, of course, no guarantee that the with­ in-group regressions are parallel. Unequal slopes are not neces­ sarily a problem; indeed, they may be of considerable substantive interest. However, they may also be the result of a potentially more difficult problem, that of nonlinearity. Floor and ceiling effects on the dependent variable can result in non­ linearity and can seriously distort the estimates based on a lin­ ear model. The potential for distortion is illustrated in Figure 4.5. The dashed line shows the true nonlinear regression assum­ ing no treatment effect. The solid lines represent the fitted linear regression effect which erroneously suggests a positive effect of the treatment for the group with higher pretest scores. Problems such as those illustrated in Figure 4.5 have been encountered in applications of the regression-discontinuity

QUANTITATIVE METHODS

107

Single Group Designs

(]) "O C (]) a. (])

Experimental � .. Control Covariate Fig. 4.5. Illustration of effects of nonlinearity in the regres­ sion-discontinuity design. (Solid lines are estimated linear regressions within groups; dashed line is the true nonlinear regression.)

design within the context of the Title I Evaluation and Report­ ing System (TIERS). Model C of Tiers is the regression-conti­ nuity design and was originally preferred to the more frequently used norm-referenced evaluation model, Model A (Tallmadge & Wood, 1976). Subsequent investigations of Model C, how­ ever, found that Model C often yields biased and erratic results (Stewart, 1980; Tallmadge & Wood, 1981). Nonlinearity result­ ing from floor effects for low-scoring Title I students and ceiling effects for high-scoring non-Title I students is a major factor in the unsatisfactory results of Model C. In his analysis of designs where assignment to treatment con­ ditions is based solely on a covariate, Rubin ( 1977) concluded that "if the conditional expectations are parallel and linear, the proper regression adjustment is the simple covariance adjust­ ment. However, since the quality of the resulting estimates may be sensitive to the adequacy of the underlying model, it is wise to search for non-parallelism and nonlinearity in these condi­ tional expectations" (p. 1). Checks for nonlinearity are greatly facilitated by having distributions of scores on the covariate overlap for the two groups, which is contrary to the usual re­ gression discontinuity design. As stated by Rubin, "the most defensible analysis of the data requires the distribution of the covariate to overlap in the two groups; without overlap, the analysis relies on assumptions that cannot be checked using the data at hand" ( 1 977, p. 22). This caution suggests a variation in the regression-discontinuity design in which scores on the co­ variate are used to define three regions : Units in the first region are assigned to the experimental treatment, units in the third to the control, and units in the middle region are randomly as­ signed to treatments. The existence of data for both treatments in the middle score range on the covariate provides a direct estimate of the r.:onditional expectations without the need for extrapolation.

The absence of a comparison group severely limits the ability of the investigator to make causal inferences. On the other hand, premature experimentation can be costly and have little chance of payoff. Brophy (1979) put it stronger, in suggesting that pre­ mature experimentation can be "counterproductive. " He went on to suggest that "before rushing to make a classroom process variable the independent variable to be manipulated in an experiment, it is worth exploring the correlational relation itself at greater length in order to develop 'grounded theory' . . . to guide experimentation" (Brophy, 1 979, p. 740). This advice is similar to that given earlier by Rosenshine and Furst (1973), who argued for programmatic research starting with observa­ tion followed by correlational studies and experimental studies. The potential range of variables is great. Correlational stu­ dies can often serve to narrow the range by identifying likely candidates for experimental manipulation. Dunkin and Biddle ( 1974) proposed a model with four categories of variables that need to be considered in studies of teaching. Their model pro­ vides a useful framework for organizing variables and guiding correlational studies. They distinguish between presage, con­ text, process, and product variables. Included among presage variables are indicators of teacher formative experiences (e.g., SES, sex, and age), teacher training experiences (e.g., university, practice teaching, special training), and teacher properties (e.g., teaching skills, and ability and personality measures). Context variables include measures of student formative experiences, student properties (e.g., pretest scores), school and community contexts (e.g., school size, ethnic composition), and classroom contexts (e.g., size, facilities, materials). Process variables are measures of teacher and student classroom behaviors. Finally, product variables are short- and long-term measures of student knowledge, skills, and attitude (Dunkin & Biddle, 1974, pp. 36-48). ZERO-ORDER CORRELATION COEFFICIENTS

Almost 30 years ago Herbert Simon (1954) aptly observed : Even in the first course in statistics, the slogan "Correlation is no proof of causation ! " is imprinted firmly in the mind of the aspiring statistician or social scientist. It is possible that he leaves the course (and many subsequent courses) with no very clear ideas as to what is proved by correlation, but he never cea,ses to be on guard against "spurious" correlation, that master of imposture who is always rep­ resenting himself as "true" correlation. (p. 467)

Simon goes on to discuss the usual alternative explanations of an observed correlation between X and Y: that X causes Y; that Y causes X; that X and Y are both caused by a third vari­ able, Z; or that the "causal effect of X on Y (or vice versa) operates through Z" (Simon, 1 954, p. 468). He then proceeds to present a detailed analysis of the standard approach to testing for spuriousness, by controlling for Z through partial correla­ tions or other related approaches. Simon's analysis leads to the clear conclusion that whether one is dealing with zero-order correlations or partial correlations, causal interpretations nec­ essarily rest on the acceptance of certain a priori assumptions. He concludes that "correlation is proof of causation in the two­ variable case if we are willing to make the assumptions of time

108

ROBERT L. LINN

precedence and non-correlation oferror terms" (Simon, 1 954, pp. 472-473). Of course, the catch is that these strong assumptions are sel­ dom considered tenable. It is important to bear in mind, how­ ever, that strong assumptions are just as necessary in making causal interpretations from more complicated analysis, whether in the form of partial correlations, regression analyses, or struc­ tural equation models. Duncan ( 1975), who is well known for his contributions to causal modeling (cf. Duncan, 1966; Gold­ berger & Duncan, 1973), has cautioned that " one can never infer the causal ordering of two or more variables knowing only the values of the correlations (or even the partial correlations !)" (p. 20). Causal reasoning must be in the other direction, from a model to implications for correlations which have causal inter­ pretations only within the context of the model and its assump­ tions, as Duncan (1975) observed : Knowing the causal ordering, or, more precisely, the causal model linking the variables, we can sometimes infer something about cor­ relations. Or, assuming a model for the sake of argument we can express its properties in terms of correlations and (sometimes) find one or more conditions that must hold if the model is true but are potentially subject to refutation by empirical evidence. . . . Failure to reject a model, however, does not require one to accept it, for the reason . . . that some other model(s) will always be consistent with the same set of data. (p. 20) Given the limitations of causal modeling with correlational data, it is hardly surprising that some of the leaders in research on teaching eschew "simplistic models that 'explain' variance in student learning by entering predictor variables into multiple regression or path analysis" (Brophy, 1979, p. 741). Preference is given to using correlational analysis in simple form for the generation of hypotheses for experimental study. Simple corre­ lations of individual process variables with student outcome (e.g., achievement) or change variables, or two-variable regres­ sions with student pretest scores as the second predictor along with each process variable, are considered more useful for hy­ pothesis generation in Brophy's judgment than more complex models. This approach is illustrated by Evertson, Anderson, Anderson, & Brophy's (1979, 1 980) analysis of several hundred individual observation variables. Subsequent experimental research by Emmer, Evertson, Sanford, & Clements (1982) illustrates the potential benefits of using the results of exploratory correlational analyses to pro­ vide hypotheses and focus for experimental studies. It is not obvious, however, that the search of hundreds of zero-order correlation coefficients or beta weights in two-predictor regres­ sions had advantages over other correlational analysis strate­ gies using more complex regression analyses and causal modeling. The inclusion of student pretest scores as a predictor, along with each individual process variable, in the search for relationships with student posttest scores reflects an implicit concern that the simple correlation between process and post­ test may be spurious. But there are other potential variables in the presage and context categories described by Dunkin and Biddle (1974) that, if taken into account, might also lead to the conclusion that the relationship of a particular process variable

to student performance is spurious. The analytical procedures described below are intended to explore such possibilities. MULTIPLE REGRESSION: PARTITIONING VARIANCE

Multiple regression and correlational (MRC) analysis is hardly new, but as Cohen (1982) recently observed , the view of MRC " has changed radically during the past decade when articles (e.g., Cohen, 1 968 ; Darlington, 1 968; Overall & Speigel, 1969) and later textbooks (e.g., Cohen & Cohen, 1 975; Kerlinger & Pedhazur, 1973; McNeil, Kelly & McNeil, 1975; Namboodiri, Carter & Blalock, 1 975; Ward & Jennings, 1 973) appeared in which MRC . . . was presented as a very powerful and flexible general data-analytic method" (p. 41). This "new look," as Cohen (1982) calls it, includes a number of features relatively unfamiliar a decade ago, such as uses of coded variables (e.g., dummy variables, effect codes), polynomial and product terms (cf. Glass, 1968; Cohen, 1978 ; Sockloff, 1 976), hierarchical re­ gression, and the partitioning of variance (cf. Mood, 1971 ; Wisler, 1969). Here the focus is on the latter of these: hierarchi­ cal regression and partitioning variance. The Dunkin and Biddle (1974) categorization of variables is used to illustrate the partitioning of variance for hierarchical blocks of variables. Table 4.5 lists as "major blocks" of vari­ ables, Dunkin and Biddle's four major categories. Minor blocks are groups of variables within each major block and, finally, examples of individual variables within each block are listed. For simplicity, it is assumed that only one product variable, student achievement on a posttest, is of interest. In order to partition the variance into portions unique to each block, joint for blocks of variables (also called common terms), and that portion due to error (or unexplained), a var­ iety of multiple correlations are computed. Differences between squared multiple correlations are then used to partition the Table 4.5. Blocks of Variables in Dunkin and Biddle's ( 1 974) Model for the Study of Teaching Maj or Blocks

M i nor Blocks

Example of Ind ividual Variables

A. Presage

a. Teacher Formative Experiences b. Teacher Trai n i ng Experiences C. Teacher Properties

al. a2. bI . b2. CI. c2.

Social Class Age U n iversity Trai n i ng Program Teacher Skills Teacher Abi l ity

B. Context

d . Student Formative Experiences e. Pupil Properties f. School & Com m u n ity Contexts g . Classroom Contexts

dI. d2. eI. e2. fI. f2. gI. g2.

Social Class Sex Pretest Student Attitude School Size Ethnic Com position Class Size Materials

C. Process

h. Teacher Classroom Behaviors i. Student Classroom Behaviors

hI. h2. iI. i2.

Teacher Enthusiasm Corrects Homework Engaged Time Student Questions

D. Product

j . Student Performance

iI.

Posttest

Note. Adapted from Figure 3.1 of Dunkin & Biddle (1974).

109

QUANTITATIVE METHODS

Table 4.6. I l lustration of the Partitioning of Variance for the Dunkin and Biddle ( 1 974) Model

Esti mate

Component Unique Block A: U(A) Block B : U(B) Block C : U(C) Joint Blocks A & B : J(A, B) Blocks A & C : J(A, C) Blocks B & C : J(B, C) Blocks A, B & C : J(A, B, C) Error o r Unexplained

Results for Example•

SMC(A, B, C) - SMC(B, C) SMC(A, B, C) - SMC(A, C) SMC(A, B, C) - SMC(A, B)

.02 .32 .05

SMC(A, B, C) - SMC(C) - U(A) - U(B) SMC(A, B, C) - SMC(B) - U(A) - U(C) SMC(A, B, C) - SMC(A) - U(B) - U(C) SMC(A, B, C) - U(A) - U(B) - U(C) -}(A, B) - j(A, C) - J(B, C) 1 .0 - SMC(A, B, C)

.06 .03 .13 .19 .20

• Results are computed assuming the following values of the relevant squared multiple correlations: SMC(A) = .3; SMC(B) = .7; SMC(C) = .4; SMC(A, B) = .75; SMC(A, C) = .48; SMC(B, C) = .78; SMC(A, B, C) = .8.

variance. Detailed rules for partitioning the variance are pro­ vided by Kerlinger and Pedhazur (1973, pp. 297-305) and by Pedhazur (1982, pp. 194-208), but here the procedure will be illustrated only by example. The technique is often referred to as commonality analysis. A major application of this technique is described in Mayeske et al. (1972). The unique variance attributed to each of the three major blocks of predictors (pressage, context, and process) is the dif­ ference between the squared multiple correlation with all pre­ dictors included in the regression, SMC(A,B,C) and the squared multiple correlation with all variables except those in the block of interest included. For example, the unique variance for the process block, U(C), is U(C) = SMC(A,B,C) - SMC(A,B). Joint terms, or variance common to two blocks and not unique to either, and the error or unexplained variance are obtained from other combinations of squared multiple correlations, as shown in Table 4.6. The last column of Table 4.6 shows results for a hypothetical example using the squared multiple correlations listed at the bottom of the table. From those numbers it would be concluded that, while 40 % of the variance in posttest scores is predictable from process variables (SMC(C) = .4), only 5 % of the variance is uniquely associated with that block of variables (U(C) = .05). The remaining 35 % is common with A (3 %), with B (13 %), or with A and B (19 %). Similar analyses could be conducted at the level of minor blocks of variables or individual variables. How­ ever, the number and interpretation of common terms would be quite unwieldy. Hence, the attention to minor blocks and/ or individual variables is apt to be limited to the estimated uniquenesses. A potential problem, not apparent in the hypothetical results in Table 4.6, is that negative estimates of common or joint terms can and often do occur. For example, if SMC(A) were .5 instead of .3 but all other squared multiple correlations on Table 4.6 remained unchanged, the joint termfor Blocks B and C, J (B,C) would equal - .07. Such negative estimates of "variance accounted for" are uninterpretable.

PATH ANALYSIS/STRUCTURAL EQUATION MODELS

In the past ten or fifteen years the use of linear causal models, called path analysis or structural equation models, for analyz­ ing correlational data has grown substantially. Duncan (1966) introduced the basic ideas of path analysis to many social scientists, and since that time causal modeling has been widely discussed and applied to a wide range of social science and edu­ cational problems (cf. Anderson, 1978; Anderson & Evans, 1974; Duncan, 1975; Goldberger, 1972b; Goldberger & Dun­ can, 1973; James, Mulaik, & Brett, 1982; Jencks et el., 1972; Kenny, 1979; Land, 1969; Wolfe, 1977). Although the estima­ tion procedures that are now available (e.g., Joreskog and Sor­ bom's LISREL V, 1981) are considerably more powerful and sophisticated than those available to early workers, the funda­ mental ideas of path analysis were rather fully developed in the 1920s by Sewall Wright (1921). Path diagrams are useful representations of presumed mod­ els, which show, with arrows, the hypothesized causal connec­ tions between variables and underlying assumptions regarding disturbance terms. A simple, four-variable path diagram is illus­ trated in Figure 4.6. A single variable from each of Dunkin and Biddle's (1974) categories is posited for simplicity. The curved, double-headed arrow between the pressage and context vari­ ables represents a simple, unanalyzed correlation. That is, no casual direction is presumed and no attempt is made to explain this correlation as spurious. The straight single-headed arrows, on the other hand, depict presumed causal relations. Thus, pro­ cess is seen to be presumed to be affected by both pressage and context, and product is affected by all three of the other vari­ ables. The fact that process is not completely determined by pressage and context is depicted by the unobserved variable U 1 , which is known as a disturbance term. The disturbance term for product is U 2 . The fact that there is no direct connection between pressage or context and the disturbance terms or between the two distur­ bance terms corresponds to critical assumptions of this model, namely that U 1 and U 2 are uncorrelated with each other and with pressage and context. Using X 1 , X2 , and X3 for pres sage, content and process, respectively, and Y for product, the

110

ROBERT L. LINN

path diagram in Figure 4.6 corresponds to the following two equations:

Table 4.7. lntercorrelations, Standardized Partial Regression Coefficients, and M ultiple Correlations for Hypothetical Example Correspond i ng to Figure 4.6.

X 3 = P1 X 1 + P 2 X 2 + q 1 U1 ,

lntercorrelations

Y = p 3 X 1 + p4 X 2 + P s X 3 + q 2 U 2 , where the ps and qs are path coefficients. With a simple model such as in Figure 4.6, in which the Xs and Y are all observed variables, the path coefficients are simply equal to standardized partial regression coefficients. That is, p 1 and p2 are the "beta" weights in the regression of X 3 on X 1 and X2 , and p3 , p4, and p5 are the beta weights in the regression of Y on X 1 , X 2, and X 3 . The qs are equal to the square roots of one minus the corresponding squared multiple correlations. There are often advantages, however, to using the corresponding unstandardized weights which, unlike the stan­ dardized weights, do not depend on sample variability (cf. Blalock, 1967; Boudon, 1968). In more complicated models, where some of the variables are unobserved but data are available for fallible indicators, and/or there is reciprocal causation hypothesized in the model (e.g., achievement affects self-concept and self-concept affects achie­ vement), estimation procedures other than ordinary least square regression are needed. For example, two-stage least squares (e.g., James & Singh, 1978) or the least squares or maxi­ mum likelihood analysis of covariance structures (e.g., Joreskog and Sorbom, 1981) may be used in such situations. Before con­ sidering these complications, however, the basic procedure is illustrated by a numerical example for the simple model in Figure 4.6. Intercorrelations among the four variables in the model in Figure 4.6 are listed in the top of Table 4.7 for a hypothetical example. Those correlations yield the multiple correlations and standardized partial regression coefficients (i.e., path coeffi­ cients) shown in the bottom half of Table 4.7. From those re­ sults, ignoring sampling error, it would be concluded that X 1 and X2 each have a small positive effect on X 3 , but that X 3 is largely determined by other factors, q2 = .966. Both X2 and X 3 are seen to have small positive effects on Y(P4 = P5 = . 1 ), while X 1 has a considerably larger effect (P3 = .5). Again, however, other factors, not included in the model but depicted by the disturbance term, have still larger effects on Y(q2 = .817). Correlations between observed variables are decomposed into a sum of products of path coefficients and unexplained cor-

Pressage

Product

Context

Fig. 4.6. Diagram based on Dunkin and Biddle's ( 1 974) vari­ able classification scheme.

Variables

x,

X2

X3

y

x,

1 .000 .400 . 1 80 .558

.400 1 .000 .240 .324

. 1 80 .240 1 .000 .2 1 4

.558 .324 .2 1 4 1 .000

X2 X3

y

Partial Regression Coefficients Dependent Variable

x,

X2

X3

.I

.2

y

.5

.I

X3 .I

u, .966

U2

M u ltiple Correlation

.8 1 7

.257 .577

relations. The fundamental equation for recursive models (i.e., models in which the causal linkages run in only one direction) is

where the Ps are all the direct path coefficients to variable Yand the ps are correlations. For example, the correlation between Y and X 1 is Py! = pyl + Py2 P 1 2 + py J P t J = . 5 + (. 1)(.4) + (.1)(. 18) = .558 The correlation between X 1 and X 3 could also be decomposed into = .1 + (.2)(.4) = .18 but the correlation p 1 2 is left unexplained. Thus, a complete de­ composition of Py i is

All the arrows, except the curved arrow representing the unex­ plained correlation between X 1 and X2 , flow in one direction. That is, for example, there is no arrow pointing back from Y to one of the explanatory variables, X 1 , X2 or X3 . Such a model is called recursive. Nonrecursive models, which allow for recipro­ cal causation, are also possible, but as noted above, require more complex estimation procedures plus more complex de­ composition. Nonrecursive models will not be considered here, but discussions and examples may be found in Anderson (1978), Duncan (1975), Joreskog and Sorbom (1981), and Kenny (1979). A model such as the one in Figure 4.6 suffers from a number of limitations. The path coefficients are consistent with, indeed they reproduce perfectly, the observed correlations among the variables. However, many other models would also fit the data. As was noted above, "one can never infer the causal ordering of

111

QUANTITATIVE METHODS

two or more variables knowing only the values of the correla­ tions" (Duncan, 1 966, p. 20). Sometimes a model may be re­ jected, however, because it implies restraints on correlations which, if not satisfied, make that model untenable. For example, if it were assumed that there was no direct path from context to process in the model in Figure 4.6 (i.e., that p 32 = 0, and the arrow from context to process was deleted), the correlation be­ tween process and context, p2 3 , would be implied to equal the product of two other correlations. Specifically, P23

= P 1 2 P 1 3,

which is equivalent to saying that the partial regression weight of X 2 in the regression of X3 on X 1 and X2 is zero. If that partial regression weight were significantly different from,zero, the modified model could be rejected. There is no way, however, to reject the original model on the basis of correlational data, because any set of correlations among four variables could be reproduced by the model. This is so because the model in Fig­ ure 4.6 is just identified; that is, there are no constraints of the values of any of the correlations imposed by the model. Hence, the model could not be rejected on the basis of correlational data. The modified model with p3 2 = 0, on the other hand, is said to be overidentified. The implied constraint in the correlations stated above provides the potential means of rejecting the mod­ el. Overidentified models obviously provide greater potential for learning from the data than just-identified or underidentified models (i.e., models where there is not a unique solution for the model coefficients). The model depicted in Figure 4.6 is also limited by the failure to take into account measurement error. Errors of measure­ ment in the regressors lead to biased regression coefficients (Cochran, 1968). For example, if /3y, is the standardized regres­ sion weight for the regression of Y on an error-free measur_e, T, then th� corresponding coefficient for the regression of Y on X, a fallible indicator of T, is

where P xx· , is the reliability of X. The above expression is based on the assumptions of classical test theory, that is, X = T+ E,

where E is the error of measurement and is assumed to be un­ correlated with both T and Y. Since the reliability of X is less than one, the regression coefficient for X is obviously less than the corresponding value for T. Thus, if the interest is in the effect of the construct T on Y, then that effect will be under­ estimated by /3yx · With more than one regressor in the equation, errors of measurement will again result in biased coefficients, but the direction of the bias is not necessarily in the direction of under­ estimation, Some coefficients may be biased toward overstating the magnitude of an effect, while others tend to understate an effect. A measurement model must be added to the structural model to remove the biases due to errors of measurement.

Fig. 4.7.

Path diagram with latent variables, each measured

by two fallible indicators.

MEASUREMENT MODELS WITH MULTIPLE INDICATORS

The LISREL model (Joreskog and Sorbom, 198 1 ) may be used to illustrate the use of measurement models in combination with a structural model involving unmeasured constructs. A simple model involving three unobserved constructs, each with two fallible indicators, is shown in Figure 4.7. Circled Greek letters represent the unobserved constructs, and the arrows connecting the circles show the presumed structural model. The observed fallible indicator variables are indicated by the letters in the squares. Uncircled cis and e s denote errors of measure­ ment, which are assumed to be mutually uncorrelated and un­ correlated with all the unobserved constructs. Uncircled (s are disturbance terms in the structural model. To make the model in Figure 4. 7 more concrete, X 1 and X2 may be considered classroom means on two tests administered at the beginning of the year and are seen as indicators of aver­ age ability, Y1 and Y2 are two measures of teacher implemen­ tation of an instructional program, say, from teacher logs and classroom observations; and 11 1 is the underlying implementa­ tion construct. Finally, Y3 and Y4 are classroom means on two posttests and 11 2 is average achievement. The path of primary interest is between implementation, 11 1 , and achievement, 11 2 • The measurement model in Figure 4.7 can be represented by the following six equations:

e;

X1

=

l 1 e + ci 1 ,

X2

=

l2 e + tS 2 ,

Y1 =

A 3 1'/ 1

+ e1

Y2 =

A4 1'/ 1

+ e2 ,

Y3 = Asl'/2 + e3 , Y4

= A6 '12 + e4,

112

ROBERT L. LINN

where the bs and es are assumed to be mutually uncorrelated and uncorrelated with � and the '7S. Since the scale of � and the '7S is arbitrary, 1 1 , 13 , and 15 are set equal to 1.0 in order to identify the model. The remaining ls and variances of the bs and es can then be estimated from the variances and covar­ iances of the observed variables, the Xs andYs. The structural model is defined by the following two equa­ tions: '1 1 = Y 1 � + C1 , '12 = Y2 � + P11 1 + C2 , where the disturbance terms ( 1 and (2 are assumed to be uncor­ related with each other and with all other latent variables in the model. The estimate of p provides the desired estimate of the direct effect of program implementation on achievement. Parameter estimates for the above model could be obtained using LISREL V from the six-by-six variance-covariance matrix for the observed variables. The use of multiple measures, as illustrated by the above model, has substantial theoretical advantages because it pro­ vides a means of avoiding distortions in the parameter esti­ mates due to errors of measurement. From a practical point of view, however, it is the exception rather than the rule that multiple measures of each construct are feasible to obtain.

Aggregation and the Unit of Analysis In studies of teaching, measures are typically obtained at two or more levels. There are at least measures of individual students within a classroom and of teachers and/or classroom-level vari­ ables. There may also be measures at other levels of aggrega­ tion, for example, school or school district characterizations, and community characteristics. The data are inherently multi­ level. Too frequently, however, the hierarchical nature of the data is ignored in analysis. Cronbach (1976) has forcefully argued that analyses of edu­ cational data often look at the wrong question and produce unjustified conclusions due to the failure to attend to questions of the unit of analysis and the hierarchical organization of the data. According to Cronbach (1976): The majority of studies of educational effects - whether classroom experiments, or evaluations of programs, or surveys - have collect­ ed and analyzed data in ways that conceal more than they reveal. The established methods have generated false conclusions in many studies. (p. 1 ) The fact that correlation and regression analyses may yield quite different results when data are aggregated at different levels is hardly a new discovery. Warnings by Thorndike (1939) and Robinson (1950) that correlations between group means should not be interpreted as relationships at the level of indi­ viduals are well known. But there is less understanding of how to match the level of analysis to the question or how best to conduct multilevel analyses. Brophy (1979) has proposed that studies of teacher behavior and its effects should "use the teacher as the unit of analysis (not the school or school district)" (p. 739). This is sensible ad-

vice; and better than the alternative, sometimes used, of attri­ buting teacher and classroom characteristics to each student in a classroom and using the individual student as the unit of ana­ lysis. But, as was earlier recognized by Brophy (1975), some questions concern relationships within classrooms. The effects of variation of teacher-student interactions within a classroom are concealed by between-classroom analyses of means. As stated by Burstein (1980), "lumping together (through aggre­ ation to the class level) data from different teacher-student interactions may mask the varying quality of the interactions in the same way that Veldman and Brophy (1974) argued that school-level analyses do for teacher differences" (p. 188). Recognition that both between- and within-group analyses may be needed to address the questions of concern in a study of teaching has led several authors (e.g., Burstein, 1980; Cooley, Bond, & Mao, 1981; Cronbach, 1976; Cronbach & Webb, 1975) to recommend routine use of both types of analysis or a multilevel analysis that includes both between- and within­ group effects. Cronbach (1976), for example, suggests that sta­ tistical analyses that pool individuals without regard to group­ ing are apt to be misleading. He goes on to suggest that "the more reasonable analysis is to relate variables within groups, . . . and then analyze group level variables across groups" (p. 1.3). Burstein (1980) has summarized results of an application of Cronbach's analysis strategy that was conducted by Martin (1978). Martin reanalyzed data from Anderson, Everston, and Brophy's (1978) study of first-grade reading groups. Three lev­ els: classrooms, reading groups within classrooms, and students within reading groups were used in Martin's reanalysis. Sum­ mary results from Martin's dissertation that were reported by Burstein (1980) show a significant effect for "new-question feed­ back" at the between-class level but not at the other two levels. "Nonvolunteer selection" into reading groups, on the other hand, had a sizeable significant effect at the level of reading groups within classes but no effect at the between-class or stu­ dent-within-reading-group levels. Results such as these led Bur­ stein (1980) to conclude that "one need not choose a 'correct' unit of analysis in teacher effectiveness research. On the con­ trary, analyses at multiple levels combined with theory-guided interpretations have much to offer the substantive researcher" (p. 191). Cronbach (1976, pp. 3.1-3.11) proposed a decomposition of an individual's score on a dependent variable of interest into the following components: 1. 2. 3. 4. 5.

the between-group predicted the between-group residual the common within-group predicted the specific within-group predicted the individual student residual

For example, if Y;i is the score on a posttest for student i in group j, X ii , the corresponding pretest score, and ½ a measure of the teacher in classroom j, then the five above components may be identified as follows: 1. Y. . + b8x(.X.i - X. .) + bBT( '½ - T.), 2. (� - Y. .) - b8x(.X.i - X..) - bBT( ½ - T.), 3. bw (Xii - X),

113

QUANTITATIVE METHODS

4. (bi - bw)(Xii - X), and 5.

e,j,

where means are denoted by bars and dots for the subscripts over which averages are taken, b8x and b8r are between-class regression coefficients, bw is the pooled within-class regression coefficient, bi is the regression coefficient within classj, and e,i is the individual residual. If the teacher variables were measured specifically to each student, then terms involving T, parallel to those involving X, could be added to Components 3 and 4. A between-class analysis would focus only on the first two components. For many studies this may be the most appropri­ ate focus. Certainly, teacher effects as indicated by b8r are of major interest in any study of teaching, as is the amount of residual variation between classes, indicated by the second component. The search for additional process variables at the classroom level that reduce the second component is an ob­ vious part of research on teaching. The third and fourth components are also of potential sub­ stantive interest, however, particularly if they include process variables that are linked to individual students. The pooled "within-group slope reflects the tendency, across all groups, of students above the group average to do better or worse on the outcome measure than the rest of the group" (Burstein, 1 980, p. 217). The specific within-group slopes have the potential of re­ flecting differences in process between groups, that tend to in­ crease or decrease the dependency of student outcomes on ini-

I I

I

I

I

I I I

I

I

I

I I

I

I

I

I

I I

I

I

I

I

.5

I

0

I

I

I

I

I

I

I

I

!

I I

I

I

I

I

:

I

I

I I

I I

I

I I I

l

I

I

I

I

I

I

I

'I I

I

I

I

Pooled Slope

I

I I

tial status. Hence, vanat10n between classes in the specific within-group slopes is itself worthy of study and, if possible, explanation in terms of teacher behaviors (Burstein, 1 980; Burstein, Linn, & Capell, 1 978). McDonald and Elias' (1976) beginning teacher effectiveness study included analyses of specific within-class slopes from re­ gressions of posttest on pretest scores. They concluded that "there is substantial variability among classes with regard both to slopes and intercepts of the regression lines" (p. 615). An inspection of the within-class regressions shows a substantial range indeed. For 33 classes with reading data, the slopes ranged from a low of .60 to a high of 1.21. The corresponding figures for the 37 classes with mathematics test data are .06 and 1.44. Ninety-percent confidence intervals for the within-class re­ gressions for the 37 classes with mathematics test results are shown in Figure 4.8. The solid vertical line in the figure corre­ sponds to the value of the pooled within-class slope. It is appar­ ent from an inspection of Figure 4.8 that there is considerable instability in the specific within-group slopes due to the small number of students within a class. Consequently the confidence intervals for many classes are quite wide. Nonetheless, a few of the confidence intervals do not overlap and the pooled within­ class slope is not included in the 90 % confidence intervals for 5 of the classes. The important unanswered question is whether differences in within-class slopes may reflect differences in class­ room process.

I

I

:

I

I

!

I

I I

I

I I

I

I I I

I I

I I

I

I I !

-,

I

I I I

I

I

I

I

.5

Slopes

1 .0

I

1 .5

2.0

Fig. 4.8. N inety-percent confidence intervals for regressions of posttest in mathematics on pretest in mathematics for 37 se­

cond-grade classrooms (data from M cDonald and Elias, 1 976) .

114

ROBERT L. LINN

Meta-Analysis The quantitative methods discussed above are applicable to in­ dividual studies. The usefulness of quantitative analysis of data collected in a research study has long been recognized. Glass (1976) and others (e.g., Hunter, Schmidt, & Jackson, 1982; Jackson, 1980; Rosenthal, 1978) have persuasively argued that quantitative methods are just as important to the review and analysis of results across studies. According to Glass : most of us were trained to analyze complex relationships among variables in the primary analysis of research data. But at the higher level, where variance, non-uniformity and uncertainty are no less evident, we too often substitute literary exposition for quantitative rigor. The proper integration of research requires the same statisti­ cal methods that are applied in primary data analysis. (p. 6) It is not easy to look at the results of 1 00 independent studies with varying and often apparently conflicting results and reach sound conclusions regarding effects and true variation in effects without the help of quantitative analysis. The problem is akin to trying to summarize the test results for 1 00 individuals in varying experimental conditions without the use of summary statistics. Yet, when Jackson (1980) randomly sampled 36 re­ views in leading educational, psychological, and sociological journals, he found that only 3 of them cited summary statistics for the studies reviewed. In the last few years, however, Glass's notions of meta-analysis have gained much favor. The power of the approach and of related approaches developed indepen­ dently by Schmidt and Hunter (1977) can be seen in the strength of the support provided for conclusions in a number of areas where previous reviews had yielded little (cf. Glass & Smith, 1979; Hartley, 1977; Hunter, 1 980; Hunter, Schmidt, & Hunter, 1979; Kulik, Kulik, & Cohen, 1979; Pearlman, 1982; Smith & Glass, 1977). Two books (Glass, McGaw, & Smith, 198 1 ; Hunter, Schmidt, & Jackson, 1982) on meta-analysis have recently been published. The reader interested in greater detail is referred to those two books and to McGraw (in press). See also Walberg's chapter in this volume for further discus­ sion of meta-analysis.

Vote Counting An early approach to quantification in reviews has come to be known as "vote counting." As described by Light and Smith ( 1971), studies are categorized as showing a significantly posi­ tive effect, a significantly negative effect, or no significant effect. "The number of studies falling into each of these three catego­ ries is then simply tallied" (Light & Smith, 1971, p. 433). Vote counting is a step in the right direction and has been widely used, but it has limitations. As noted by Hunter, Schmidt, and Jackson ( 1982) it " is biased in favor of large sam­ ple studies which may show only small effect sizes " (p. 22). More importantly, Hedges and Olkin (1980) have shown that vote counting methods have very poor power that actually de­ creases as the number of studies increases.

p- Values An alternative to vote-counting that was suggested by Rosen­ thal ( 1978) is to combine the probabilities (p-values) from inde-

pendent studies. This approach has an advantage over simple vote counting, in that it uses information available in p-values within either the range of " significant" or "nonsignificant" re­ sults. As noted by Rosenthal and Rubin ( 1979), "a p of .05 tells about the same story as a p of .06 if both results are in the same direction and the studies are of similar size (p. 1 165). In vote counting, however, the two studies would be tallied in different categories. An obvious limitation of cumulation p-values is that they de­ pend not only on effect sizes but on sample sizes. It is important to know not only the existence of an effect, but its size. Thus, it is desirable to go beyond probabilities, to estimates of the size and variability of effects across studies. Another concern that applies not only to vote-counting and p-value methods but to the methods considered below, is the possibility that significant results are more apt to get published, and hence be available for review, than nonsignificant results. The "lost studies " might alter the probability and the estimate of the average effect size if they were available. Rosenthal (1979) has proposed an ingenious method for dealing with the ques­ tion of studies that may have been conducted but never re­ ported because the results were nonsignificant. He suggested that reviewers calculate the number of " new, filed or unre­ trieved" (p. 640) studies, which on average showed null results, and which would be required to bring the estimated overall probability of an effect down to a barely significant level. If the number of such studies is larger than could plausibly be ex­ pected to have been conducted, the conclusion based on the probability estimate from available studies is strengthened. Rosenthal and Rubin's ( 1978) estimate that about 65,000 lost studies would be required to lower the p-value to the barely significant level in their review of interpersonal expectancy effects, for example, makes the lost-study explanation of the findings entirely implausible.

Effect Sizes Glass (1976, 1977, 1980) has emphasized the importance of esti­ mating the effect size of each study and quantitatively analyzing the effect sizes across studies. The mean, variance, and relation­ ship of effect sizes to coded study chracteristics provide the primary basis for synthesizing research findings and drawing conclusions. For studies comparing experimental and control groups, the effect size for study i, ES;, is defined by

where YE and Yc are the sample means for the experimental and control groups, respectively, and S is the standard deviation in the control group or the pooled within-group standard devia­ tion. Alternatively, correlations are used as estimates of effect sizes. Glass ( 1977) reports a variety of formulas that may be used to convert various statistics that may be available in the reports of individual studies into estimates of effect sizes. Glass's meta-analysis approach has been applied to several areas of research relevant to teaching. Redfield and Rousseau (1981) found that, contrary to the mixed and inconclusive inter­ pretations of a previous narrative review, the meta-analysis of

QUANTITATIVE METHODS 20 studies of teachers' use of higher cognitive questions had positive effects on student achievement. Hartley (1977) used the approach to synthesize the findings of studies of individually paced instruction and Kulik, Kulik, and Cohen (1979) have used it to analyze results of 75 studies of Keller's (1968) person­ alized system of instruction. Possibly the most elaborate appli­ cation of the general approach is Glass and Smith's (1979) analysis of the effects of class size on student achievement. De­ spite a long history of inconclusive and conflicting reviews, the power of the quantitative analyses used by Glass and Smith enabled them to conclude that "a clear and strong relation­ ship between class size and achievement has emerged" (1979, p. 15). Recent statistical work related to the use of effect size esti­ mates (e.g., Hedges, 1981, 1982; Rosenthal and Rubin, 1982) has moved beyond simple descriptive statistics based upon esti­ mated effect sizes to provide statistical theory for the analysis. Distinguishing between population and sample estimates of ef­ fect sizes enabled Hedges (1981) to obtain a distribution of effect sizes, which he used to show that Glass's estimate has a small-sample bias. He also derived an unbiased estimator, which he showed has a smaller variance. More recently, Hedges (1982) presented a large-sample test for the homogeneity of ef­ fect sizes across studies. Rosenthal and Rubin (1982) have also provided an approximate significance test "of the heterogeneity of effect sizes for two or more independent studies for any type of effect size assuming estimates of its variance are available" (p. 503).

Explaining the Variance of Effect Sizes Schmidt and Hunter (1977) independently developed methods of meta-analysis within the context of validity-generalization that share many of the features of Glass's approach (see, for example, Hunter, Schmidt, & Jackson, 1981; Linn & Dunbar, in press; Pearlman, 1982, for reviews of methodological de­ velopments and applications stemming from the Schmidt and Hunter approach to meta-analysis). The principal difference between the Schmidt-Hunter and Glass approaches is that the variance of the estimated effect sizes is typically unanalyzed in the Glass approach, whereas Schmidt and Hunter place con­ siderable emphasis on explaining that variance in terms of various statistical artifacts. "These artifacts include: (1) sampling error, (2) study differences in reliability of independent and dependent measures, (3) study differences in range restric­ tion, (4) study differences in instrument·validity, and (5) compu­ tational, typographical and transcription errors" (Hunter, Schmidt, & Jackson, 1982, p. 28). A variety of methods have been developed for estimating the amount of variance in effect size estimates that is attributable to the first three of these arti­ facts (cf. Hunter, Schmidt, & Jackson, 1981; Linn & Dunbar, in press). The meta-analysis techniques advocated by Schmidt and Hunter have had a profound effect on the interpretation of val­ idity study results in personnel psychology. The approach has applicability in many other areas of research, including research on teaching. Applications of the Glass and Schmidt-Hunter meta-analytic techniques hold the promise of learning much more from previous research than has heretofore been feasible.

115

Conclusion Quantitative analysis of primary research data and of results of previous research is a vital part of research on teaching. But quantitative methods cannot stand alone. Statistical adjust­ ments are a poor substitute for adequate research design. The choice of what is worth studying and the appropriate reach of generalizations that almost always take us beyond the "limited reach of internal validity" (Cronbach, 1982, p. 112 ff.) necessar­ ily involve qualitative judgments. The utility of highly quantita­ tive causal modeling requires string theory that is often lacking. On the other hand, theories need the power of quantitative analysis in order to be refined. As Cooley et. al. (1981) noted: This is a kind of chicken and egg situation. The more we understand phenomena, the better the models will be; the better the models, the better will be our understanding. The way to improve both is to use models in our educational research. Make them explicit. Get them out in the open so that they can be critically examined and im­ proved as better thinking and better data are brought to bear on them. Through such an iterative process, we will finally begin to achieve the kinds of convincing models that are so desperately needed if we are to improve quantitative approaches to the study of educational phenomena. (p. 80)

REFERENCES Anderson, J. G. (1978). Causal models in educational research: Nonre­ cursive models. American Educational Research Journal, 15, 8 1 -97. Anderson, J. G., & Evans, F. B. (1974). Causal models in educational research : Recursive models. American Educational Research Jour­ nal, 11, 29-39. Anderson, L., Evertson, C., & Brophy, J. (1978). The first grade reading group study : Technical report of experimental and process- outcome relationships. Austin : University of Texas, R & D Center for Teacher Education. Baglin, R. F. (1981). Does "nationally" normed really mean nationally? Journal of Educational Measurement, 18, 97-108. Barnow, B. S. (1973). The effects ofHead Start and socioeconomic status on cognitive development of disadvantaged children. Unpublished doctoral dissertation, University of Wisconsin, Madison. Barnow, B. S., Cain, G. G., & Goldberger, A. S. (1980). Issues in the analysis of selectivity bias. In E. Stromsdorfer & G. Farkus (Eds.), Evaluation Studies Review Annual (Vol. 5). Beverly Hills: Sage Publications. Bentler, P. M. (1976). Multistructure statistical model applied to factor analysis. Multivariate Behavioral Research, 11, 3-25. Bentler, P. M. (1980). Multivariate analysis with latent variables; Causal models. Annual Review of Psychology, 31, 419-456. Bentler, P. M., & Woodward, J. A. (1980). A Head Start reevaluation: Positive effects are not yet determined. In E. L. Baker & E. S. Quell­ malz (Eds.), Educational testing and evaluation: Design, analysis and policy. Beverly Hills: Sage Publications. Berk, R. A., Ray, S. C., & Cooley, T. F. (1982). Selection biases in socio­ logical data (Project Report, National Institute of Justice Grant No 80-IJ-CX-0037). Santa Barbara: University of California. Blalock, H. M., Jr. (1967). Causal inferences, closed populations, and measures of association. American Political Science Review, 61, 1 30-136. Boudon, R. (1968). A new look at correlation analysis. In H. M. Bla­ lock, Jr., & A. B. Blalock (Eds.), Methodology in social research. New York : McGraw-Hill. Brophy, J. E. (1975). The student as the unit of analysis (Research Rep. No. 75- 1 2). Austin: University of Texas. Brophy, J. E. (1979). Teacher behavior and its effects. Journal ofEduca­ tional Psychology, 71, 733- 750. Burstein, L. (1980). The analysis of multilevel data in educational re­ search and evaluation. In D. C. Berliner (Ed.), Review ofResearch in Education, 8, 1 58-233.

116

ROBERT L. LINN

Burstein, L., Linn, R. L., & Capell, F. J. (1978). Analyzing multilevel data in the presence of heterogeneous within-class regressions. Journal of Educational Statistics, 3, 347-383. Campbell, D. T. (1967). The effects of college on students: Proposing a quasi-experimental approach (Research Report). Evanston, IL: Northwestern University. Campbell, D. T., & Erlebacher, A. E. (1970). How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful. In J. Hellmuth (Ed.), Compensatory educa­ tion: A national debate: Vol. 3. Disadvantaged child. New York : Brunner/Maze!. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi­ experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally. Cochran, W. G. (1968). Errors of measurement in statistics. Techno­ metrics, JO, 637-666. Cochran, W. G., & Cox, G. M. (1957). Experimental designs (2nd ed.). New York: John Wiley. Cohen, J. (1968). Multiple regression as a general data analytic system. Psychological Bulletin, 70, 426-443. Cohen, J. (1978). Partialed products are interactions; partialed powers are curve components. Psychological Bulletin, 85, 858-866. Cohen, J. (1982). "New-look" multiple regression/correlational analy­ sis and the analysis of variance/covariance. In G. Keren (Ed.), Sta­ tistical and methodological issues in psychology and social sciences research. Hillsdale, NJ: Lawrence Erlbaum Associates. Cohen, J., & Cohen, P. (1975). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates. Coleman, J. S., & Karweit, N. (1970). Measures of school performance. Santa Monica, CA : Rand Corporation. Cook, T. D., & Campbell, D. T. (1976). The design and conduct of quasi-experiments and true experiments in field settings. In M. Dun­ nette (Ed.), Handbook of industrial and organizational psychology. Chicago: Rand McNally. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation : Design and analysis issuesforfield settings. Chicago: Rand McNally. Cooley, W. W., Bond, L., & Mao, B. (1981). Analyzing multilevel data. In R. A. Berk (Ed.), Educational evaluation methodology: The state of the art. Baltimore: Johns Hopkins University Press. Cox, D. R. (1957). The use of a concomitant variable in selecting an experimental design. Biometrika, 44, 1 50-1 58. Cronbach, L. J. (1976, July) Research on classrooms and schools: Formulation of questions, design, and analysis (Occasional paper). Stanford, CA: Stanford University, Stanford Evaluation Consor­ tium. Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass. Cronbach, L. J., & Furby, L. (1970). How we should measure "change" - or should we? Psychological Bulletin, 74, 68-80. Cronbach, L. J., Rogosa, D. R., Floden, R. E., & Price, G. G. (1977). Analysis of covariance in nonrandomized experiments; Parameters affecting bias (Occasional paper). Stanford, CA: Stanford Univer­ sity, Stanford Evaluation Consortium. Cronbach, L. J., & Snow, R. E. (1976). Aptitudes and instructional methods. New York: Irvington. Cronbach, L. J., & Suppes, P. (Eds.). (1969). Research for to­ morrow's schools: Disciplined inquiry for education. New York: Macmillan. Cronbach, L. J., & Webb, N. (1975). Between-class and within-class effects in a reported aptitude x treatment interaction: Reanalysis of a study by G. L. Anderson. Journal of Educational Psychology, 67, 717-727. Darlington, R. B. (1968). Multiple regression in psychological research. Psychological Bulletin, 69, 161 -182. David, J. L., & Pelavin, S. H. (1977). Research on the effectiveness of compensatory education programs: A reanalysis ofdata (SRI Project Report URU-4425). Menlo Park, CA: SRI International. DuBois, P. H. (1957). Multivariate correlational analysis. New York: Harper. Duncan, 0. D. (1966). Path analysis: Sociological examples. American Journal of Sociology, 72, 1 - 1 6.

Duncan, 0. D. (1975). Introduction to structural equation models. New York: Academic Press. Dunkin, M., & Biddle, B. (1974). The study of teaching. New York: Holt, Rinehart & Winston. Elashoff, J. D. (1969). Analysis of covariance: A delicate instrument. American Educational Research Journal, 6, 383-401. Emmer, E. T., Evertson, C., Sanford, J., & Clements, B. S. (1982). Im­ proving classroom management: An experimental study in junior high classrooms (R & D Rep.) Austin: University of Texas, R & D Center for Teacher Education. Ennis, R. H. (1973). On causality. Educational Researcher, 2, 4- 1 1 . Ennis, R. H . (1982). Abandon causality? Educational Researcher, J J, 25-27. Evertson, C. M., Anderson, C. W., Anderson, L. M., & Brophy, J. E. (1979). Predictors of student outcomes in junior high mathematics and English classes (Rep. No. 4066). Austin: University of Texas, R & D Center for Teacher Education. Evertson, C. M., Anderson, C. W., Anderson, L. M., & Brophy, J. E. (1980). Relationships between classroom behaviors and student outcomes in junior high mathematics and English classes. American Educational Research Journal, 1 7, 43-60. Feldt, L. S. (1958). A comparison of the precision of three experimental designs employing a concomitant variable. Psychometrika, 23, 335-353. Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver and Boyd. Glass, G. V. (1968). Correlations with products of variables: Statistical formulation and implications for methodology. American Educa­ tional Research Journal, 5, 721-728. Glass, G. V. (1976). Primary, secondary and meta analysis of research. Educational Researcher, 5(1 1), 3-8. Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351 -379. Glass, G. V. (1980). Summarizing effect sizes. In R. Rosenthal (Ed.), New directions for methodology of social and behavioral science: Quantitative assessment of research domains. San Francisco: Jossey-Bass. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills: Sage Publications. Glass, G. V., & Smith, M. L. (1979). Meta-analysis of research on the relationship of class-size and achievement. Educational Evaluation and Policy Analysis, 1, 2-16. Goldberger, A. S. (1972a). Selection bias in evaluating treatment effects: Someformal illustrations (Discussion paper). Madison: Un­ iversity of Wisconsin, Institute for Research on Poverty. Goldberger, A. S. (1972b). Structural equation methods in the social sciences. Econometrika, 40, 979-1001. Goldberger, A. S. (1979). Methods for eliminating selection bias. Mad­ ison: University of Wisconsin, Department of Economics. Goldberger, A. S. (1981). Linear regression after selection. Journal of Econometrics, 15, 357-366. Goldberger, A. S., & Duncan, 0. D. (1973). Structural equation models in the social sciences: Specification, estimation, and testing. New York: Seminar Press. Good, T. L., & Grouws, D. A. (1979). The Missouri Mathematics Effec­ tiveness Project : An experimental study in fourth-grade classrooms. Journal of Educational Psychology, 71, 355-362. Hartley, S. S. (1977). Meta-analysis of the effects of individually paced instruction in mathematics. Unpublished doctoral dissertation, University of Colorado, Boulder. Heckman, J. J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. Annuals of Economic and Social Measurement, 5, 475-492. Heckman, J. J. (1979). Sample selection bias as specification error. Econometrika, 47, 1 53-161. Heckman, J. J. (1980). Sample selection bias as a specification error. In J. P. Smith (Ed.), Female labor supply: Theory and estimation. Princeton, NJ: Princeton University Press. Hedges, L. V. (1981 ). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.

QUANTITATIVE METHODS

Hedges, L. V. (1982). Estimation of effect size from a series of indepen­ dent experiments. Psychological Bulletin, 92, 490-499. Hedges, L. V., & Olkin, I. (1980). Vote-counting methods in research synthesis. Psychological Bulletin, 88, 359-369. Hunter, J. E. (1980). Validity generalization for 12,000 jobs: An applica­ tion of synthetic validity and validity generalization to the General Aptitude Test Battery (GATB). Washington, D.C.: U.S. Depart­ ment of Labor, U.S. Employment Service. Hunter, J. E., Schmidt, F. L., & Hunter, R. (1979). Differential validity of employment tests by race : A comprehensive review and analysis. Psychological Bulletin, 86, 721 -735. Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982). Meta analysis : Cumulating research findings across studies. Beverly Hills: Sage Publications. Jackson, G. B. (1980). Methods for integrative reviews. Review of Edu­ cational Research, 50, 438-459. Jaeger, R. M. (1979). The effect of test selection on Title I project im­ pact. Educational Evaluation and Policy Analysis, 1, 33-40. James, L. R., Mulaik, S. A, & Brett, J. M. (1982). Causal ana­ lysis: Assumptions, models, and data. Beverly Hills: Sage Pub­ lications. James, L. R., & Singh, B. K. (1978). An introduction to the logic, as­ sumptions and basic analytic procedures of two stage least squares. Psychological Bulletin, 85, 1 104-1 122. Jencks, C., Smith, M., Acland, H., Bane, M. J., Cohen, D., Gintis, H., Hayns, B., & Michelsohn, S. (1972). Inequality: A Reassessment of the Effect of Family and Schooling in America. New York: Basic Books. Jiireskog, K. G. (1973). A general model for estimating a linear struc­ tural equation system. In A. S. Goldberger & 0. D. Duncan (Eds.), Structural equation models in the social sciences: Specification, esti­ mation and testing. New York: Seminar Press. Jiireskog, K. G., & Si:irbom, D. (1981). LISREL V: Analysis of linear structural relationships by maximum likelihood and least squares (Research Rep. 8 1-8). Uppsala, Sweden: University of Uppsala, Department of Statistics. Jurs, S. G., & Glass, G. V. (1971). The effect of experimental mortality on the internal and external validity of the randomized comparative experiment. Journal of Experimental Education, 40, 62-66. Keller, F. S. (1968). Good-bye teacher. Journal of Applied Behavior Analysis, 1, 79-89. Kempthorne, 0. (1952). The design and analysis of experiments. New York : John Wiley. Kenny, D. A. (1975). A quasi-experimental approach to assessing treat­ ment effects in the nonequivalent control group design. Psychologi­ cal Bulletin, 82, 345-362. Kenny, D. A. (1979). Correlation and causality. New York: John Wiley. Kerlinger, F. N., & Pedhazur, E. J. (1973). Multiple regression in beha­ vioral research. New York: Holt, Rinehart & Winston. Kirk, R. E. ( 1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Monterey, CA: Brooks/Cole. Kulik, J. A., Kulik, C. C., & Cohen, P. A. (1979). A meta-analysis of outcome studies of Keller's personalized system of instruction. American Psychologist, 34, 307-3 18. Land, K. C. (1969). Principles of path analysis. In E. F. Borgotta & G. W. Bohnstedt (Eds.), Sociological Methodology 1969. San Fran­ cisco: Jossey-Bass. Light, R. J. (1973). Issues in the analysis of qualitative data. In R. M. W. Travers (Ed.), Second handbook of research on teaching. Chicago: Rand McNally. Light, R. J., & Smith, P. V. (1971). Accumulating evidence: Procedures for resolving contradictions among different reseach studies. Har­ vard Educational Review, 41, 429-471. Linn, R. L. (1981). Measuring pretest-posttest changes. In R. A. Berk (Ed.), Educational evaluation methodology: The state of the art. Bal­ timore, MD: Johns Hopkins University Press. Linn, R. L., & Dunbar, S. B. (in press). Validity generalization and pre­ dictive bias. In R. A. Berk (Ed.), Performance assessment: The state of the art. Baltimore: Johns Hopkins University Press. Linn, R. L., Dunbar, S. B., Harnisch, D. L., & Hastings, C. N. (1982). The validity of the Title I Evaluation and Reporting System. Evaluation Studies Review Annual, 7, 427-442.

117

Linn, R. L., & Slinde, J. A. (1977). The determination of the significance of change between pre- and posttesting periods. Review of Educa­ tional Research, 47, 121 -150. Linn, R. L., & Werts, C. E. (1977). Analysis implications of the choice of a structural model in the nonequivalent control group design. Psy­ chological Bulletin, 84, 299-324. Lord, F. M. (1960). Large-sample covariance analysis when the control variable is fallible. Journal of the American Statistical Association, 55, 307-321. Lord, F. M. (1967). A paradox in the interpretation of group compar­ isons. Psychological Bulletin, 68, 304-305. Magidson, J. (1977). Toward a causal model approach for adjusting for preexisting differences in the nonequivalent control group situation : A general alternative to ANCOV A. Evaluation Quarterly, 1, 399-420. Manning, W. H., & DuBois, E. H. (1962). Correlational methods in research on human subjects. Perceptual Motor Skills, 15, 287-321. Martin, D. (1978). The unit of analysis problem in teacher effectiveness research. Unpublished doctoral dissertation, University of Texas at Austin. Mayeske, G. W., Wisler, C. E., Beaton, A. E., Jr., Weinfeld, E. D., Cohen, W. M., Okada, T., Proschek, J. M., & Tabler, K. A. (1972). A study of our nation's schools. Washington, DC: U.S. Government Printing Office. McDonald, F. J., & Elias, P. J. (1976). Beginning Teacher Evaluation Study : Vol. I. The effects of teaching performance on pupil learning (Phase II, final report). Princeton, NJ: Educational Testing Service. McGaw, B. (in press). Meta-analysis. In T. Husen & T. N. Postleth­ waite (Eds.), International encyclopedia of education: Research and studies. Oxford: Pergamon. McNeil, K. A., Kelly, F. J., & McNeil, J. T. (1975). Testing research hypotheses using multiple linear regression. Carbondale: Southern Illinois University Press. Mood, A. M. (197 1). Partitioning variance in multiple regression ana­ lyses as a tool for developing learning models. American Educational Research Journal, 8, 1 9 1-202. Muthen, B., & Ji:ireskog, K. G. (1983). Selectivity problems in quasi­ experimental studies. Evaluation Review, 7, 1 39-174. Namboodiri, N. K., Carter, L. F., & Blalock, H. M., Jr. (1975). Applied multivariate analysis and experimental designs. New York: McGraw-Hill. Olsen, R. (1980). Approximating a truncated normal regression with the method of moments. Econometrika, 48, 1099- 1 105. Overall, J. E., & Spiegel, D. K. (1969). Concerning least squares analysis of experimental data. Psychological Bulletin, 72, 31 1 -322. Pearlman, K. (1982). The Bayesian approach to validity generalization : A systematic examination of the robustness ofprocedures and conclu­ sions. Unpublished doctoral dissertation, Catholic University of America, Washington, DC. Pedhazur, E. J. (1982). Multiple regression in behavioral research : Ex­ planation and prediction (2nd ed.). New York: Holt, Rinehart & Winston. Popper, K. R. (1959). The logic of scientific discovery. New York: Basic Books. Porter, A. C. (1967). The effects of using fallible variables in the analysis of covariance. Unpublished doctoral dissertation, University of Wisconsin, Madison. Porter, A. C., & Chibucos, T. R. (1974). Selecting analysis strategies. In G. D. Boruch (Ed.), Evaluating educational programs and products. Englewood Cliffs, NJ: Educational Technology Publications. Prescott, G. A. (1973). Manual for interpreting : Metropolitan Achieve­ ment Test. New York: Harcourt Brace Jovanovich. Redfield, D. L., & Rousseau, E. W. (198 1). A meta-analysis of experi­ mental research on teacher questioning behavior. Review of Educa­ tional Research, 51, 237-245. Reichardt, C. S. (1979). The design and analysis of the nonequivalent group quasi-experiment. Unpublished doctoral dissertation, North­ western University, Evanston, IL. Reicken, H. W., Boruch, R. F., Campbell, D. T., Coplan, W., Glennan, T. K., Pratt, J., Rees, A., & Williams, W. (1974). Social experimenta­ tion : A methodfor planning and evaluating social innovations. New York: Academic Press.

118

ROBERT L. LINN

Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351-357. Rogosa, D., •Brandt, D., & Zimowski, M. ( 1982). A growth curve ap­ proach to the measurement of change. Psychological Bulletin, 92, 726-748. Rosenshine, B., & Furst, N. (1 973). The use of direct observation to study teaching. In R. M. W. Travers (Ed.), Second handbook of re­ search on teaching. Chicago: Rand McNally. Rosenshine, B., & McGraw, B. (1972). Issues in assessing teacher accountability in public education. Phi Delta Kappan, 53, 640-643. Rosenthal, R. (1978). Combining results of independent studies. Psy­ chological Bulletin, 85, 185-193. Rosenthal, R. (1979). The "file-drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641. Rosenthal, R., & Rubin, D. B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 3, 377-415. Rosenthal, R., & Rubin, D. B. ( 1979). Comparing significance levels of independent studies. Psychological Bulletin, 86, 1 165-1 1 68. Rosenthal, R., & Rubin, D. B. (1982). Comparison effect sizes of inde­ pendent studies. Psychological Bulletin, 92, 500-504. Rubin, D. B. (1974). Estimating causal effects of treatments in ran­ domized and nonrandomized studies. Journal of Educational Psychology, 66, 688-701. Rubin, D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics, 2, 1 -26. Saxe, L., & Fine, M. (1979). Expanding our view of control groups in evaluations. In L. Datta & R. Perloff (Eds.), Improving evaluations. Beverly Hills: Sage Publications. Scheffe, H. (1959). The analysis of variance. New York: John Wiley. Schmidt, F. L., & Hunter, J. E. ( 1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. Schmidt, F. L., Hunter, J. E., & Pearlman, K. (1981). Task differences as moderators of aptitude test validity in selection: A red herring. Journal of Applied Psychology, 66, 166-1 85. Simon, H. A. (1954). Spurious correlation: A causal interpretation. Journal of the American Statistical Association, 49, 467-479. Smith, H. J. ( 1957). Interpretation of adjusted treatment means and regression in analysis of covariance. Biometrics, 13, 282-308.

Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760. Sockloff, A. L. ( 1976). The analysis of nonlinearity via linear regression with polynomial and product terms. Review of Educational Re­ search, 46, 267-291. Siirbom, D. ( 1981). Structural equation models with structured means. In K. G. Jiireskog & H. Wold (Eds.), Systems under indirect observa­ tion: Causality, structure and prediction. Amsterdam : North Hol­ land Publishing. Stenhouse, L. ( 1978). Some limitations of the use of objectives. In D. Hamilton et al. (Eds.), Beyond the numbers game. Berkeley, CA: McCutchan. Stewart, B. L. ( 1980). The regression model in Title I evaluation. Moun­ tain View, CA: RMC Research. Tallmadge, G. K., & Wood, C. T. (1976). User's Guide: ESEA Title I evaluation and reporting system. Mountain View, CA: RMC Research. Tallmadge, G. K., & Wood, C. T. (1981). User's Guide: ESEA Title I evaluation and reporting system. Mountain View, CA: R.M.C. Research. Thorndike, E. L. (1939). On the fallacy of imputing the correlations found for groups to the individuals or smaller groups composing them. American Journal of Psychology, 52, 122-124. Travers, R. M. W. (1981). Letter to the editor. Educational Researcher, JO, 32. Veldman, D. J., & Brophy, J. E. (1974). Measuring teacher effects on student achievement. Journal of Educational Psychology, 66, 3 19-324. Ward, J. H., Jr., & Jennings, E. (1973). Introduction to linear models. Englewood Cliffs, NJ : Prentice-Hall. Winer, B. J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw-Hill. Wisler, C. E. (1969). Partitioning the explained variation in a regression analysis. In G. W. Mayeske et al., A study of our nation's schools. Washington, DC: U.S. Department of Health, Education, & Wel­ fare, Office of Education. Wolfe, L. M. (1977). An introduction to path analysis. Multiple Linear Regression Viewpoints, 8, 36-61. Wright, S. ( 1921). Correlation and causation. Journal of Agricultural Research, 20, 557-585.

5.

Q u a l itat i ve M et h o d s i n Researc h o n Teac h i n g Fred e r ick Er i c kson

Michigan State University

General and abstract ideas are the source of the greatest errors of mankind. J. J. Rousseau What is General Nature? is there such a Thing ?/What is General Knowledge ? is there such a Thing ?/Strictly Speaking All Knowledge is Particular. W. Blake

Introduction and Overview

it avoids the connotation of defining these approaches as essen­ tially nonquantitative (a connotation that is carried by the term qualitative), since quantification of particular sorts can often be employed in the work; and (c) it points to the key feature of family resemblance among the various approaches - central re­ search interest in human meaning in social life and in its eluci­ dation and exposition by the researcher. The issue of using as a basic validity criterion the immediate and local meanings of actions, as defined from the actors' point of view, is crucial in distinguishing interpretive participant ob­ servational research from another observational technique with which interpretive research approaches are often confused, so-called rich description. Since the last decades of the 19th century, the data collection technique of continuous narrative description - a play-by-play account of what an observer sees observed persons doing - has been used in social and behav­ ioral research. The technique was first used by psychologists in child study and then by anthropologists and sociologists doing community studies. It is important to emphasize at the outset that the use of con­ tinuous narrative description as a technique - what can less formally be called " writing like crazy" - does not necessarily mean that the research being conducted is interpretive or qualitative, in a fundamental sense. What makes such work

This chapter reviews basic issues of theory and method in ap­ proaches to research on teaching that are alternatively called ethnographic, qualitative, participant observational, case study, symbolic interactionist, phenomenological, constructivist, or interpretive. These approaches are all slightly different, but each bears strong family resemblance to the others. The set of related approaches is relatively new in the field of research on teaching. The approaches have emerged as signifi­ cant in the decade of the 1960s in England and in the 1 970s in the United States, Australia, New Zealand, and Germany. Be­ cause interest in these approaches is so recent, the previous edi­ tions of the Handbook of Research on Teaching do not contain a chapter devoted to participant observational research. Accord­ ingly, this chapter attempts to describe research methods and their theoretical presuppositions in considerable detail and does not attempt an exhaustive review of the rapidly growing literature in the field. Such a review will be appropriate for the next edition of this handbook. From this point on I will use the term interpretive to refer to the whole family of approaches to participant observational re­ search. I adopt this term for three reasons : (a) It is more inclu­ sive than many of the others (e.g., ethnography, case study); (b)

The author thanks reviewers Hugh Mehan (University of California at San Diego) and Raymond P. McDermott (Teachers College, Columbia University). Preparation of this chapter was supported in part through the Institute for Research on Teaching, College of Education, Michigan State University. The Institute for Research on Teaching is funded primarily by the Program for Teaching and Instruction of the National Institute of Education, United States Department of Education. The opinions expressed in this publication do not necessarily reflect the position, policy, or endorsement of the National Institute of Education. (Contract No. 400-81-0014). 119

120

FREDERICK ERICKSON

interpretive or qualitative is a matter of substantive focus and intent, rather than of procedure in data collection, that is, a research technique does not constitute a research method. The technique of continuous narrative description can be used by researchers with a positivist and behaviorist orientation that deliberately excludes from research interest the immediate meanings of actions from the actors' point of view. Continuous narrative description can also be used by researchers with a nonpositivist, interpretive orientation, in which the immediate (often intuitive) meanings of actions to the actors involved are of central interest. The presuppositions and conclusions of these two types of research are very different, and the content of the narrative description that is written differs as well. If two ob­ servers with these differing orientations were placed in the same spot to observe what was ostensibly the "same" behavior per­ formed by the "same" individuals, the observers would write substantively differing accounts of what had happened, choos­ ing differing kinds of verbs, nouns, adverbs, and adjectives to characterize the actions that were described. The reader should note that in making these assertions I am taking a different position from Green and Evertson (this vol­ ume), who emphasize certain commonalities across various ap­ proaches to direct observation. Their comprehensive review of a wide range of methods of classroom observation (including some of the methods discussed here) does not emphasize the discontinuities in theoretical presupposition that obtain across the two major types of approaches to classroom research, posi­ tivist/behaviorist and interpretive. This chapter emphasizes those discontinuities. Green and Evertson are relatively opti­ mistic about the possibility of combining disparate methods and orientations in classroom observation. I am more pessimis­ tic about that possibility, and have become increasingly so in the last few years. Reasonable people can disagree on such mat­ ters. The reader should compare the two chapters, keeping in mind the differences in perspective that characterize our two discussions, which run along lines that are somewhat similar but are nonetheless distinct. From my point of view, the primary significance of interpre­ tive approaches to research on teaching concerns issues of con­ tent rather than issues of procedure. Interests in interpretive content lead the researcher to search for methods that will be appropriate for study of that content. If interpretive research on classroom teaching is to play a significant role in educational research, it will be because of what interpretive research has to say about its central substantive concerns: (a) the nature of classrooms as socially and culturally organized environments for learning, (b) the nature of teaching as one, but only one, aspect of the reflexive learning environment, and (c) the nature (and content) of the meaning-perspectives of teacher and learner as intrinsic to the educational process. The theoretical conceptions that define the primary phenomena of interest in the interpretive study of teaching are very different from those that underlie the earlier, mainstream approaches to the study of teaching. These distinctive features of the interpretive perspec­ tive will be considered throughout this essay. This is not quite to say that this is a situation of competing paradigms in research on teaching, if paradigms are thought of in the sense used by Kuhn (1962) to refer to an integrated set of theoretical presuppositions that lead the researcher to see the

world of one's research interest in a particular way. A paradigm is metaphysical. A scientific theoretical view, according to Kuhn, becomes in practical usage an ontology. The current conflict in research on teaching is not one of competing para­ digms, I would argue, not because the competing views do not differ ontologically, but simply because as Lakatos (1978) and others have argued for the natural sciences - and especially for the social sciences - paradigms do not actually compete in scientific discourse. Old paradigms are rarely replaced by falsifi­ cation. Rather the older and the newer paradigms tend to coex­ ist, as in the survival of Newtonian physics, which can be used for some purposes, despite the competition of the Einsteinian physics, which for other purposes has superseded it. Especially in the social sciences, paradigms don't die; they develop vari­ cose veins and get fitted with cardiac pacemakers. The perspec­ tive of standard research on teaching and the interpretive perspective are indeed rival theories - rival research pro­ grams - even if it is unlikely that the latter will totally super­ sede the former. I have not attempted a comprehensive review of the field, nor have I attempted to present a modal perspective on interpretive research. There is much disagreement among interpretive re­ searchers about the proper conduct of their work and its theo­ retical foundations. Given this lack of consensus, which is greater than that in the more standard approaches to research on teaching, it would be inappropriate for me to attempt to speak on behalf of all interpretive researchers. Accordingly, this chapter emphasizes those aspects of theory and method that are most salient in my own work. In substance, my work is an at­ tempt to combine close analysis of fine details of behavior and meaning in everyday social interaction with analysis of the wider societal context - the field of broader social influences - within which the face-to-face interaction takes place. In method, my work is an attempt to be empirical without being positivist; to be rigorous and systematic in investigating the slippery phenomena of everyday interaction and its connec­ tions, through the medium of subjective meaning, with the wider social world. The chapter begins with an overview of interpretive ap­ proaches and the kinds of research questions that are of central interest in sµch work. The next section reviews the intellectual roots of interpretive research, the development of firsthand par­ ticipant observation as a research method, and the underpin­ nings of that development in particular kinds of social theory and practical concern. The third section traces the implications of this general theoretical orientation for the study of classroom teaching. Then the discussion turns to consider issues of meth­ od. There is a section on data collection and analysis, and a section on data analysis and the preparation of written reports. These sections will address the reasons that data analysis inheres in the data collection phase of research as well as in the reporting phase. The chapter concludes with a discussion of implications of interpretive approaches for future research on teaching.

Overview Interpretive, participant observational fieldwork has been used in the social sciences as a research method for about seventy

QUALITATIVE METHODS IN RESEARCH ON TEACHING years. Fieldwork research involves (a) intensive, long-term par­ ticipation in a field setting; (b) careful recording of what hap­ pens in the setting by writing field notes and collecting other kinds of documentary evidence (e.g., memos, records, examples of student work, audiotapes, videotapes); and (c) subsequent analytic reflection on the documentary record obtained in the field, and reporting by means of detailed description, using nar­ rative vignettes and direct quotes from interviews, as well as by more general description in the form of analytic charts, sum­ mary tables, and descriptive statistics. Interpretive fieldwork research involves being unusually thorough and reflective in noticing and describing everyday events in the field setting, and in attempting to identify the significance of actions in the events from the various points of view of the actors themselves. Fieldwork methods are sometimes thought to be radically inductive, but that is a misleading characterization. It is true that specific categories for observation are not determined in advance of entering the field setting as a participant observer. It is also true that the researcher always identifies conceptual is­ sues of research interest before entering the field setting. In field­ work, induction and deduction are in constant dialogue. As a result, the researcher pursues deliberate lines of inquiry while in the field, even though the specific terms of inquiry may change in response to the distinctive character of events in the field setting. The specific terms of inquiry may also be reconstrued in response to changes in the fieldworker's perceptions and under­ standings of events and their organization during the time spent in the field. Interpretive methods using participant observational field­ work are most appropriate when one needs to known more about: 1. The specific structure of occurrences rather than their gen­ eral character and overall distribution. (What does the de­ cision to leave teaching as a profession look like for partic­ ular teachers involved?) What is happening in a particular place rather than across a number of places? (If survey data indicate that the rate of leaving teaching was lowest in a particular American city, we might first want to know what was going on there before looking at other cities with aver­ age teacher leaving rates.) 2. The meaning-perspectives of the particular actors in the particular events. (What, specifically, were the points of view of particular teachers as they made their decisions to leave teaching?) 3. The location of naturally occurring points of contrast that can be observed as natural experiments when we are unable logistically or ethically to meet experimental conditions of consistency of intervention and of control over other in­ fluences on the setting. (We can't hold constant the condi­ tions that might influence teachers to want to leave teaching, and we can't try to cause them to want to leave.) 4. The identification of specific causal linkages that were not identified by experimental methods, and the development of new theories about causes and other influences on the patterns that are identified in survey data or experiments. Fieldwork is best at answering the following questions (on these questions, and the ensuing discussion, see Erickson,

121

Florio, & Buschman, 1980, of which these remarks are a paraphrase): 1. What is happening, specifically, in social action that takes place in this particular setting? 2. What do these actions mean to the actors involved in them, at the moment the actions took place? 3. How are the happenings organized in patterns of social or­ ganization and learned cultural principles for the conduct of everyday life- how, in other words, are people in the immediate setting consistently present to each other as environments for one another's meaningful actions? 4. How is what is happening in this setting as a whole (i.e., the classroom) related to happenings at other system levels outside and inside the setting (e.g., the school building, a child's family, the school system, federal government man­ dates regarding mainstreaming)? 5. How do the ways everyday life in this setting is organized compare with other ways of organizing social life in a wide range of settings in other places and at other times? Answers to such questions are often needed in educational research. They are needed for five reasons. The first reason con­ cerns the invisibility of everyday life. "What is happening here?" may seem a trivial question at first glance. It is not trivial since everyday life is largely invisible to us (because of its familiarity and because of its contradictions, which people may not want to face). We do not realize the patterns in our actions as we perform them. The anthropologist Clyde Kluckhohn illustrated this point with an aphorism: "The fish would be the last creature to discover water." Fieldwork research on teaching, through its inherent reflectiveness, helps researchers and teachers to make the familiar strange and interesting again (see Erickson, 1984). The commonplace becomes problematic. What is happening can become visible, and it can be docu­ mented systematically. A second reason these questions are not trivial is the need for specific understanding through documentation of concrete details of practice. Answering the question, "What is happening?" with a general answer often is not very useful. "The teacher (or stu­ dents) in this classroom is (are) on-task" often doesn't tell us the specific details that are needed in order to understand what is being done, especially if one is attempting to understand the points of view of the actors involved. Nor is an answer like the following sufficient, usually: "The teacher is using behavior modification techniques effectively." This does not tell how, specifically, the teacher used which techniques with which children, nor what the researcher's criterion of effectiveness was. Similarly, the statement "The school district implemented a program to increase student time-on-task" does not tell enough about the extent and kind of implementation so that if test scores or other outcome measures did or did not show change, that putative "outcome" could reasonably be attributed to the putative "treatment." "What was the treatment?" is often a useful question in research on teaching. Interpretive fieldwork research can answer such a question in an adequately specific way. A third reason these questions are not trivial concerns the need to consider the local meanings that happenings have for the people involved in them. Surface similarities in behavior are

122

FREDERICK ERICKSON

sometimes misleading in educational research. In different classrooms, schools, and communities, events that seem ostensi­ bly the same may have distinctly differing local meanings. Di­ rect questioning of students by a teacher, for example, may be seen as rude and punitive in one setting, yet perfectly appropri­ ate in another. Within a given setting, a certain behavior like direct questioning may be appropriate for some children at one moment, and inappropriate at the next, from a given teacher's point of view at that time in the event being observed. When a research issue involves considering the distinctive local mean­ ings that actors have for actors in the scene at the moment, fieldwork is an appropriate method. A fourth reason these main research questions of fieldwork are not trivial concerns the need for comparative understanding of different social settings. Considering the relations between a setting and its wider social environments helps to clarify what is happening in the local setting itself. The observation "Teachers don't ask for extra materials; they just keep using the same old texts and workbooks for each subject" may be factually accu­ rate, but this could be interpreted quite differently depending on contextual circumstances. If school system-wide regula­ tions made ordering supplementary materials very difficult in a particular school district, then the teachers' actions could not simply be attributed to the spontaneous generation of local meanings by participants in the local scene-the "teacher cul­ ture" at that particular school. What the teachers do at the classroom and building level is influenced by what happens in wider spheres of social organization and cultural patterning. These wider spheres of influence must also be taken into ac­ count when investigating the narrower circumstances of the lo­ cal scene. The same applies to relationships of influence across settings at the same system level, such as the classroom and the home. Behavior that may be considered inappropriate in school may be seen as quite appropriate and reasonable in community and family life. For example, children may be encouraged in the family to be generous in helping one another; in the classroom this may be seen by the teacher as attempts at cheating. A fifth reason for the importance of this set of questions con­ cerns the need for comparative understanding beyond the imme­ diate circumstances of the local setting. There is a temptation on the part of researchers and school practitioners alike to think of what is happening in the standard operating procedures of everyday life as the way things must and ought to be, always and everywhere. Contrasting life in United States classrooms to school life in other societies, and to life in other institutional contexts such as hospitals and factories, broadens one's sense of the range of possibilities for organizing teaching and learning effectively in human groups. Knowing about other ways of or­ ganizing formal and nonformal education by looking back into human history, and by looking across to other contemporary societies around the world, can shed new light on the local hap­ penings in a particular school. The fieldworker asks continually while in the field setting, "How does what is happening here compare with what happens in other places?" Awareness of this does not necessarily lead to immediate practical solutions in planning change. The compar­ ative perspective does inform attempts at planning change, however. By taking a comparative perspective people can dis­ tinguish the spuriously distinctive and the genuinely distinctive

features of their own circumstances. That can lead them to be at once more realistic and more imaginative than they would otherwise have been in thinking about change. To conclude, the central questions of interpretive research concern issues that are neither obvious nor trivial. They con­ cern issues of human choice and meaning, and in that sense they concern issues of improvement in educational practice. Even though the stance of the fieldworker is not manifestly evalua­ tive, and even though the research questions do not take the form "Which teaching practices are most effective?" issues of effectiveness are crucial in interpretive research. The definitions of effectiveness that derive from the theoretical stance and em­ pirical findings of interpretive research differ from those found in the more usual approaches to educational research and de­ velopment. The program of interpretive research is to subject to critical scrutiny every assumption about meaning in any set­ ting, including assumptions about desirable aims and defini­ tions of effectiveness in teaching. This critical stance toward human meaning derives from theoretical presuppositions that will be reviewed in considerable detail in the next section of this chapter. Intellectual Roots and Assumptions of Interpretive Research on Teaching

Roots in Western European Intellectual History Interpretive research and its guiding theory developed out of interest in the lives and perspectives of people in society who had little or no voice. The late 18th century saw the emergence of this concern. Medieval social theorists had stressed the dig­ nity of manual labor, but with the collapse of the medieval world view in the 16th and 17th centuries the lower classes had come to be portrayed in terms that were at best paternalistic. One sees this paternalism in baroque theater and opera. Peasants and house servants were depicted in a one-di­ mensional way as uncouth and brutish. They were not reflec­ tive, although they may have been capable of a kind of venal cleverness in manipulating their overseers, masters, and mistresses. Examples of this are found in the servant characters of Moliere and in the farm characters of J. S. Bach's Peasant Cantata, written in 1742. Beaumarchais' The Barber of Seville, written in 1781, is distinctive precisely because it presented one of the first sympathetic characterizations of a servant figure. The dangerous implications of such a portrayal were immedia­ tely recognized by the French censors, who prevented its perfor­ mance for three years after it had been written. The perspective represented by The Barber of Seville had been preceded in France by the writings of Rousseau. In England, this new per­ spective had been prefigured by the mid-18th-century English novelists. Interest by intellectuals in the life-world (Lebenswelt) of the poor-especially the rural poor-continued to grow in the early 19th century as exemplified by the brothers Grimm, who elicited folklore from German peasants. Their work emerged simultaneously with the development of the early romantic movement in literature, in which commoners were positively

QUALITATIVE METHODS IN RESEARCH ON TEACHING

portrayed. Folklore research presupposed that the illiterate country people who were being interviewed possessed a genuine aesthetic sense and a true folk wisdom, in spite of the peasants' lack of formal education and lack of "cultivated" appreciation of the polite art forms that were practiced among the upper classes. Concerns for social reform often accompanied this in­ terest in the intelligence and talent of the untutored rural poor. Innovations in pedagogy were also related to this shift in the view of the poor, for example, the schools established in Switzerland by Pestalozzi to teach children who had hitherto been considered unteachable. Later in the 19th century, attention of reformers shifted from the rural poor to the working-class populations of the growing industrial towns (for a parallel discussion of this development, see Bogdan and Biklen, 1982, pp. 4-11). In England, Charles Booth documented the everyday lives of children and adults at work in factories and at home in slum neighborhoods (see Webb, 1926). Similar attention was paid to the urban poor in the United States by the "muckraking" journalists Jacob Riis (How the Other Half Lives, 1890) and Lincoln Steffins (1904), and in the novels of Upton Sinclair, for example, The Jungle (1906). Another line of interest developed in the late 19th century in kinds of unlettered people who lacked power and about whom little was known. These were the nonliterate peoples of the European-controlled colonial territories of Africa and Asia, which were burgeoning by the end of the 19th century. Travel­ ers' accounts of such people had been written since the begin­ nings of European exploration in the 16th century. By the late 19th century such accounts were becoming more detailed and complete. They were receiving scientific attention from the emerging field of anthropology. Anthropologists termed these accounts ethnography, a monograph-length description of the lifeways of people who were ethnoi, the ancient Greek term for "others" - barbarians who were not Greek. Anthropologists had begun to send out their students to collect ethnographic information themselves, rather than relying for the information on the books written by colonial administrators, soldiers, and other travelers. In 1914, one of these students, Bronislaw Malinowski, was interned in the Trobriand Archipelago by the British govern­ ment while he was on an ethnographic expedition. Malinowski was a student at Oxford University and had been sent to the Britsh colonies by his teachers. He was also Polish; a subject of the Austro-Hungarian Empire. On that ground he was suspect as a spy by some officials of the British colonial administration. Forced to stay in the Trobriands more than twice as long as he had intended, during his detention Malinowski developed a closer, more intimate view of the everyday lifeways and mean­ ing-perspectives of a primitive society than had any previous ethnographer, whether traveler or social scientist. Malinowski's account, when published in 1922, revolutionized the field of so­ cial anthropology by the specificity of its descriptive reporting and by the sensitivity of the insights presented about the beliefs and perspectives of the Trobrianders. Malinowski (1935, 1922/ 1966) reported insights not only about explicit cultural know­ ledge. Information on explicit culture had been elicited from informants by earlier researchers who used interviewing strat­ egies derived from the work of the early folklorists. In addition

123

Malinowski reported his inferences about the Trobriander's im­ plicit cultural knowledge- beliefs and perspectives that were so customary for the Trobrianders that they were held outside conscious awareness and thus could not be readily articulated by informants. By combining long-term participant observa­ tion with sensitive interviewing, Malinowski claimed, he was able to identify aspects of the Trobrianders' world view that they themselves were unable to articulate. Many anthropologists attacked the claims of Malinowskian ethnography as too subjective and unscientific. Others were very taken by it. It squared with the insights of Freudian psy­ chology that people knew much more than they were able to say. Freud's perspective, in turn, was consonant with the much broader intellectual and artistic movement of expressionism, which emphasized the enigmatic and inarticulate dark side of human experience, harking back to a similar emphasis among the early Romantics. Malinowski was a product of this Jate19th-century intellectual milieu, and the postwar disillusion­ ment with the values of rational liberal thinking made the 1920s an especially apt time for the reception of Malinowski's position. Malinowski's intellectual milieu in his formative years was not only that of the late 19th century in general, but that of German intellectual perspectives in particular. These bear men­ tion here, for they involve presuppositions about the nature of human society and consequently about the nature of the social sciences. German social theory, as taught in universities of the time, made a sharp distinction between natural science, Naturwissenschaft, and what can be translated "human science," or "moral science," Geisteswissenschaft. The latter term, which literally means, "science of spirit," was dis­ tinguished from natural science on the grounds that humans differ from other animals and from inanimate entities in their capacity to make and share meaning. Sense-making and mean­ ing were the spiritual or moral aspect of human existence that differed from the material existence of the rest of the natural order. Because of this added dimension, it was argued, humans living together must be studied in terms of the sense they make of one another in their social arrangements. The etymological metaphor of spirit as an entity that under­ lies this sharp distinction between the natural and the human sciences recalls an analogous metaphor in the term psychology, which in the Greek literally means "systematic knowledge about the soul." The terms Geisteswissenschaft and psychology remind us that in the mid-19th century, as social and behavioral sciences began to be defined as distinctive fields, there was as yet no commitment to define them as positive sciences modeled after the physical sciences. That came later. The chief proponent of the distinction between the natural and the human sciences was the German historian and social philosopher Wilhelm Dilthey (1883/1976c, 1914/1976a). He argued (1914/1976b) that the methods of the human sciences should be hermeneutical, or interpretive (from the Greek term for "interpreter"), with the aim of discovering and communicat­ ing the meaning-perspectives of the people studied, as an inter­ preter does when translating the discourse of a speaker or writer. Dilthey's position was adopted by many later German social scientists and philosophers, notably Weber (1922/1978) and Husserl (1936/1970). A somewhat similar position was

124

FREDERICK ERICKSON

taken by Dilthey's contemporary, Marx, especially in his early writings, for example, the Theses on Feuerbach (1959 ed.). Despite Marx's emphasis on material conditions as determin­ ing norms, beliefs, and values, he was centrally concerned with the content of the meaning-perspectives so determined. Indeed, a fundamental point of Marx's is the historical embeddedness of consciousness - the assumption that one's view of self and of the world is profoundly shaped in and through the concrete circumstances of daily living in one's specific situation of life. Subsequent Marxist social theorists have presumed that pro­ found differences in meaning-perspective will vary with social class position, and that presumption extends to any other spe­ cial life situation, for example, that due to one's gender status, race, and the like. We can assume that Malinowski was influenced by basic as­ sumptions in German social theory of his day. Those assump­ tions were contrary to those of French thinkers about society, notably Comte and Durkheim. Comte, in the mid-19th century, proposed a positivist science of society, modeled after the physi­ cal sciences, in which causal relations were assumed to be ana­ logous to those of mechanics in Newtonian physics (Comte, 1875/1968). Durkheim, Comte's pupil, may or may not have adopted the metaphor of society as a machine, but in attempt­ ing to contradict the notion that the individual is the fundamen­ tal unit of society, he argued that society must be treated as an entity in itself- a reality sui generis. Such a position can easily be interpreted as a view of society as an organism or machine. At any rate, what was central for Durkheim was not the mean­ ing-perspectives of actors in society, but the "social facts" of their behaviors (Durkheim 1895/1958). This stands in sharp contrast to the German intellectual tradition of social theory. (On the relations of presuppositions in social theory to method­ ology in the social sciences, see the book-length comparative review of functionalist, interpretive, Marxist, and existentialist positions in Burrell & Morgan, 1979; and see also Winch, 1958, and Giddens, 1982.) A final influence on American participant observational re­ search was the development of American descriptive linguistics during the 1920s. In studying native American languages linguists were discovering aspects of language structure­ sound patterns and grammar- that had never been considered in traditional grammar and philology based on Indo-European languages. These new aspects of language structure were regu­ lar and predictable in speech, but the speakers themselves were unaware of the structures they had learned to produce so regu­ larly. Here was another domain in which was evident the ex­ istence of implicit principles of order that influenced human behavior outside the consciousness of those influenced. This intellectual environment was the context for the training of Margaret Mead at Columbia University, with which she was associated for the rest of her career. Her study Coming of Age In Samoa (1928), again controversial as of this writing, can be con­ sidered the first monograph-length educational ethnography. It is significant that it dealt with teaching and learning outside schools. By the mid-1920s, with the urban sociology of Robert Park at the University of Chicago, ethnography came home. Students of Park and Burgess (e.g., Wirth, 1928; Zorbaugh, 1929) used sustained participant observation and informal interviewing as

a means of studying the everyday lives and values of natural groups of urban residents (mostly working class migrants from

Europe), who in Chicago and other major American cities were distributed in residential territories defined by an intersection of geography, ethnicity, and social class. In the 1930s a whole American community, Newburyport, Massachusetts ("Yankee City"), was studied by Malinowskian fieldwork methods, under the direction of the anthropologist W. Lloyd Warner at Har­ vard (Warner & Lunt, 1941). In a study that was greatly in­ fluenced by Warner, Whyte (1955) identified another kind of natural group as a unit of analysis, a group of late adolescent males in an urban working-class Italian neighborhood. Here the community was studied out from the gang of young men, rather than the usual anthropologist's way of trying to study the community as a whole, which Whyte found to be theoreti­ cally and logistically impossible in an urban setting. After World War II, ethnographers began to turn directly to issues of education, under the leadership of Spindler (1955) at Stanford and Kimball (1974) at Teachers College, Columbia. Both were influenced considerably by the work of Margaret Mead, and Kimball had been a student of Warner's. Chicago school sociology also made a significant contribution to ethno­ graphic work in a study of a cohort of medical students under the direction of Everett Hughes (cf. Becker, Geer, Hughes, & Strauss, 1961). An institutionalized network for researchers with these interests was initiated in 1968, with the formation of the Council on Anthropology and Education as a member or­ ganization of the American Anthropological Association. Also during the 1960s much qualitative work on teaching was done in England, under the leadership of Stenhouse and his asso­ ciates (Stenhouse, 1978; MacDonald & Walker, 1974; Walker & Adelman, 1975; Elliott & Adelman, 1976). The next major impetus for ethnographic study of education, and the last to be noted in this discussion, came in the early 1970s with the crea­ tion of the National Institute of Education. Key staff there established policies that not only allowed funding for ethnographic study, but which in some instances encouraged work to be done in schools along those lines. Often in that work the unit of analysis was the school community, with classroom teaching receiving peripheral attention. At the same time, how­ ever, studies were begun in which teaching in school classrooms was the central phenomenon of research interest. (For addition­ al discussion of the role of NIE, see Cadzen, this volume). To conclude where we began, it is important to remember that qualitative research that centers its attention on classroom teaching is a very recent phenomenon in educational research. The key questions in such research are: "What is happening here, specifically? What do these happenings mean to the people engaged in them?" The specifics of action and of meaning-perspectives of actors in interpretive research are often those that are overlooked in other approaches to research. There are three major reasons for this. One is that the people who hold and share the meaning­ perspectives that are of interest are those who are themselves overlooked, as relatively powerless members of society. This is the case for teachers and students in American public schools, as it was for the working class in postmedieval Europe. (See the discussion on the powerlessness of teachers in Lanier & Little, this volume). A second reason that these meaning-perspectives

QUALITATIVE METHODS IN RESEARCH ON TEACHING

are not represented is that they are often held outside conscious awareness by those who hold them, and thus are not explicitly articulated. A third reason is that it is precisely the meaning­ perspectives of actors in social life that are viewed theoretically in more usual approaches to educational research as either peripheral to the center of research interest, or as essentially irrelevant - part of the "subjectivity" that must be eliminated if systematic, "objective" inquiry is to be done. In the section that follows we will explore further the signifi­ cance to research on teaching of the meaning-perspectives of teachers and students, and we will consider reasons why stan­ dard educational research does not, for the most part, take ac­ count of these phenomena in the design and conduct of studies of teaching and learning in classrooms.

Theoretical Assumptions of Interpretive Research on Teaching Let us begin by considering some well-documented findings from survey research - test data on school achievement and measured intelligence. These are perplexing findings. They can be interpreted differently depending upon one's theoretical orientation. 1 . In the United States there are large differences across indi­ viduals in school achievement and measured intelligence, according to the class, race, gender, and language back­ ground of the individuals. Moreover, these differences per­ sist across generations. 2. Test score data accumulated in the recent process-product research on teaching show differences across different class­ rooms in the achievement of elementary pupils who are similarly at risk for school failure because of their class, race, gender, or language background. 3. The same test data also show differences in achievement and measured intelligence among individual children in each classroom. These findings from survey data suggest that while the likeli­ hood of low school achievement by low-socioeconomic-status children and others at risk may be powerfully influenced by large-scale social processes (i.e., handicaps due to one's position in society) and individual differences (i.e., measured intelli­ gence), the school achievement of such children is amenable to considerable influence by individual teachers at the classroom level. Teachers, then, can and do make a difference for educa­ tional equity. Finding Number 2 seems to argue against the contention of critics on the radical left that social revolution is a necessary precondition for improvement of school performance by the children of the poor in America. Finding Number 2 also contradicts the contention of liberals that environmental depri­ vation, especially the home child-rearing environment, ac­ counts for the low school achievement of such pupils, since outside-school conditions presumably did not change for those students at risk who nonetheless did better academically with some teachers than with others. Findings Number 1 and 3 con­ tradict the contention of moderate and radical conservatives that current school practices involve social sorting that is fair (i.e., assigning pupils to different curricular tracks on the basis

125

of measured achievement) and that the route to educational improvement lies in simply applying current sorting practices more rigidly. The radical left, of course, can dismiss finding Number 2 on the grounds that the differences in measured achievement of pupils that can be attributed to teacher influence are only slight, and are thus trivial. The radical right can dismiss findings Number 1 and 3 on the grounds that children of low socioeco­ nomic status (SES), or racial/linguistic/cultural minority back­ ground, or females are genetically inferior to white upper-class males, which accounts for their low achievement and measured intelliegnce. Both of these counterarguments, from the left and from the right, have been used to dismiss the significance of these three findings. If one does not dismiss the findings on those grounds, however, the three findings taken together are paradoxical. It would seem that children's achievement can vary from year to year and from teacher to teacher with no other things changing in those children's lives. Yet children at risk, overall, perform significantly less well in American schools than do children not at risk. Positive teacher influence on the achievement of children at risk seems to be the exception rather than the rule, and risk here refers to the children's social background, not to the individual child's intrinsic ability. (In­ deed, from an interpretive perspective it is meaningless to speak of a child's intrinsic ability, since the child is always found in a social environment, and since the child's performance and adults' assessments of the child's performance both influence one another continually. Rather we can say that a child's as­ sessed ability is socially constructed. It is a product of the child's social situation - the social system - rather than an attribute of that person.) The findings from the United States are more paradoxical in the light of international school science achievement data (see Comber & Keeves, 1 973, pp. 251, 259) which show that in some developed countries, such as Flemish-speaking Belgium, Italy, Sweden, and Finland, the correlation between social class back­ ground and school achievement is much lower than it is in the United States or Great Britain, and that the correlation is lower in Japan than in the United States or Britain although not so low in Japan as in the countries listed first in this sentence. How are we to understand why these survey data show the patterns that they do? Is the social construction of student performance and assessed ability different in Italy from that of the United States? What might we do to foster higher achievement by low­ achieving groups of pupils in the United States? How are we to understand the nature of teaching in the light of these findings? The survey data themselves do not tell us. They must be inter­ preted in the context of theoretical presuppositions about the nature of schools, teaching, children, and classroom life, and about the nature of cause in human social life in general. These assumptions, we have already noted, differ fundamentally be­ tween the standard approaches to research on teaching and the interpretive approaches that are the topic of this chapter. Perhaps the most basic difference between the interpretive and the standard approaches to research on teaching lies in their assumptions about the nature of cause in human social relations. This recalls the distinction made earlier between the natural sciences and the human sciences (Naturwissenschaft and Geisteswissenschaft).

126

FREDERICK ERICKSON

In the natural sciences causation can be thought of in me­ chanical, or chemical, or biological terms. Mechanical cause, as in Newtonian physics, involves relationships between force and matter, and physical linkages through which force is exerted ­ one billiard ball striking another, or the piston in a combustion engine linked by cams to the drive shaft. Chemical cause in­ volves energy transfer in combination between atoms of differ­ ent elements. Biological cause is both mechanical and chemical. Relations among organisms are also ecological, that is, causal relations are not linear in one direction, but because of the com­ plexity of interaction among organisms within and across spe­ cies, cause is multidirectional. Thus the notion of cause and effect in biology is much more complex than that of physics or chemistry. But the differences in conceptions of cause in the natural sciences are essentially those of degree rather than of kind. Even in biology there is an underlying assumption of uni­ formity in nature. Given conditions x, a bacterium or a chicken is likely to behave in much the same way on two different occa­ sions. The same is true in physics or chemistry, more or less, despite post-Einsteinian thinking. Under conditions x, one cal­ orie of heat can be considered the same entity on the surface of the sun and on the surface of the earth, and under conditions x, one atom of oxygen and two of hydrogen will combine in the same way today as they do the next day. The assumption of uniformity of nature, and of mechanical, chemical, and biological metaphors for causal relations among individual entities is taken over from the natural sciences in positivist social and behavioral sciences. Animals and atoms can be said to behave, and do so fairly consistently in similar circumstances. Humans can be said to behave as well, and can be observed to be doing so quite consistently under similar cir­ cumstances. Moreover, one person's behavior toward another can be said to cause change in the state of another person. Me­ chanical, chemical, and ecological metaphors can be used to understand these causal relations, thinking of humans in so­ ciety as a machine, or as an organism, or as an ecosystem of inanimate and animate entities. Classrooms and teaching have been studied from this per­ spective, especially by educational psychologists, and also by some positivist sociologists. Linear causal models are often em­ ployed, behavior is observed, and causal relations among var­ ious behavioral variables are inferred ; for example, certain patterns of questioning or of motivational statements by the teacher are studied to see if they cause certain changes in test­ taking behavior by children. In educational psychology this perspective derives from a kind of hybrid behaviorism, in the sense that what counts is the researcher's judgment of what an observable behavior means, rather than the actors' definitions of meaning. Such behaviorist or "behavioralist" presuppositions about the fixed and obvious meanings of certain types of actions by teachers underlie the so­ called process-product approach to research on teacher effec­ tiveness (cf. Dunkin & Biddle, 1974), which was at first cor­ relational and later became experimental. In educational sociology an analogous perspective derived from the shift toward positivism that occurred in American so­ ciology after World War II. Social facts were seen as causing other social facts, by relations akin to that of mechanical link­ age. These linkages were monitored by large-scale correlational

survey research (e.g., Coleman et al., 1966) and subsequent reanalyses of that data set (Mosteller & Moynihan, 1972, and Jencks et al., 1979). The main guiding metaphors for most educational research on teacher and school effectiveness from the 1950s through the 1970s became the metaphor of the classroom as something like a Skinner box and the metaphor of school systems and the wider society as something like linked parts of a large, inter­ nally differentiated machine. In neither metaphor is the notion of mind necessary. The phenomenological perspective of the persons behaving is not a feature of the theoretical models the metaphors represent. Interpretive researchers take a very different view of the na­ ture of uniformity and of cause in social life. The behavioral uniformity from day to day that can be observed for an indivi­ dual, and among individuals in groups, is seen not as evidence of underlying, essential uniformity among entities, but as an illusion - a social construction akin to the illusion of assessed ability as an attribute of the person assessed. Humans, the inter­ pretive perspective asserts, create meaningful interpretations of the physical and behavioral objects that surround them in the environment. We take action toward the objects that surround us in the light of our interpretations of meaningfulness. Those interpretations, once made, we take as real - actual qualities of the objects we perceive. Thus, once a child is assessed as having low ability, we assume not only that the entity low ability ac­ tually exists, but that it is actually an attribute of that child. We do not question such assumptions, once made. We cannot do so, or as actors in the world we would always be inundated by masses of uninterpretable detail and would be continually tan­ talized by the need to hold all inference and background as­ sumption in abeyance. We handle the problem of having to be, for practical purposes, naive realists-believers in the taken­ for-granted reality we perceive at first glance - by continually taking the leap of faith that is necessary. We see the ordinary world as if it were real, according to the meanings we impute to it. The previous discussion elaborates on the point made in the previous section on the emergence of a distinction between the natural sciences, Naturwissenschaften, and the human sciences, Geisteswissenschaften. This line of thinking, explicated by Dil­ they, and continued with Weber (1922/1978) and Schutz (1971), is exemplified in current writing in the philosophy of social sci­ ence by Berger and Luckmann (1967), Winch (1958), and Gid­ dens (1976), among many others. To be sure, there is much more apparent uniformity in hu­ man social life. Through culture humans share learned systems for defining meaning, and in given situations of practical action humans often seem to have created similar meaning interpreta­ tions. But these surface similarities mask an underlying diversi­ ty; in a given situation of action one cannot assume that the behaviors of two individuals, physical acts with similar form, have the same meaning to the two individuals. The possibility is always present that different individuals may have differing interpretations of the meaning of what, in physical form, appear to be the same or similar objects or behaviors. Thus a crucial analytic distinction in interpretive research is that between behavior, the physical act, and action, which is the physical behavior plus the meaning interpretations held by

QUALITATIVE METHODS IN RESEARCH ON TEACHING

the actor and those with whom the actor is engaged in inter­ action. The object of interpretive social research is action, not behav­ ior. This is because of the assumption made about the nature of cause in social life. If people take action on the grounds of their interpretations of the actions of others, then meaning-interpre­ tations themselves are causal for humans. This is not true in nature, and so in natural science meaning from the point of view of the actor is not something the scientist must discover. The billiard ball does not make sense of its environment. But the human actor in society does, and different humans make sense differently. They impute symbolic meaning to others' ac­ tions and take their own actions in accord with the meaning interpretations they have made. Thus the nature of cause in hu­ man society becomes very different from the nature of cause in the physical and biological world, and so does the nature of uniformity in repeated social actions. Because such actions are grounded in choices of meaning interpretation, they are always open to the possibility of reinterpretation and change. This can be seen in examples of symbolic action such as pub­ lic executions. Such an event is conducted with the aim not only to punish a particular offender but to coerce a confession of guilt or remorse in order to deter others from committing simi­ lar offences. The intentions are both physical and social : to kill the offender and to do so in such a way as to influence public opinion. The physical death that occurs can be seen to result from a physical cause that disrupts the biochemical organiza­ tion of the body as a living system. The reactions of the offender and audience, however, are not "caused" by the physical inter­ vention itself, but are matters of meaning interpretation, eman­ ating from the various points of view of differing actors in the event. Consider the case of Joan of Arc. Some soldier's hand thrust a lighted torch into the pile of wood whose subsequent com­ bustion killed Joan, who was tied to a stake. Looking at the crude physical and behavioral "facts" in this sequence of events, one can say that the result of what the soldier did was a matter of physics, chemistry, and biology. But the soldier's behavior did not cause Joan to cry out, denying the charge of witchcraft, insisting that the voices she had heard were those of angels rather than demons. She persisted in justifying her military resistance to the English as a response to the will of God. Such persistence was social action, entailing a choice of meaning in­ terpretation. Some of the witnesses at the scene accepted Joan's interpretation rather than that of her English judges. As the story of her death spread, French nobles and commoners united in intensified resistance to the English. French morale was in­ creased rather than decreased, which frustrated the intent of the execution on grounds of witchcraft. Meaning interpretations - rather than physical or chemical processes - were what were causal in this sequence of social ac­ tions and reactions. These interpretations were the result of hu­ man choices, made at successive links in the chain of social interaction. Had the soldier refused to light the fire because he was persuaded of Joan's innocence, the judges might have cho­ sen to relent or to have executed the soldier as well. Had Joan admitted guilt to the charge of witchcraft, the reaction of the French armies might have been different. But even if she had confessed publicly, the witnesses and subsequent audience

127

might have discounted her admission as the result of coercion. Thus the French might still have continued to resist the English, enraged by what they had come to see as a symbol of abhorrent injustice. We can see how it makes sense to claim that prediction and control, in the tradition of natural science, is not possible in systems of relations where cause is mediated by systems of sym­ bols. The martyr breaches the symbolic order of a degradation ceremony, turning the tables upon those whose intention is to degrade. Martyrs are exceptional but they are not unique. That martyrdom occurs at all points to the intrinsic fragility of the usual regularities of social life, grounded as they are in choices of meaning in the interpretation of symbols. The case of the execution of Joan of Arc shows how interpre­ tive sense-making can be seen as fundamentally constitutive in human social life. Because of that assumption, interpretive re­ search maintains that causal explanation in the domain of hu­ man social life cannot rest simply upon observed similarities between prior and subsequent behaviors, even if the correla­ tions among those behaviors appear to be very strong, and ex­ perimental conditions obtain. Rather, an explanation of cause in human action must include identification of the meaning­ interpretation of the actor. " Objective" analysis (i.e., systematic analysis) of " subjective" meaning is thus of the essence in social research, including research on teaching, in the view of interpre­ tive researchers. The interpretive point of view leads to research questions of a fundamentally different sort from those posed by standard re­ search on teaching. Rather than ask which behaviors by teachers are positively correlated with student gains on tests of achievement, the interpretive researcher asks "What are the conditions of meaning that students and teachers create to­ gether, as some students appear to learn and others don't? Are there differences in the meaning-perspectives of teachers and students in classrooms characterized by higher achievement and more positive morale? How is it that it can make sense to students to learn in one situation and not in another? How are these meaning systems created and sustained in daily interac­ tion?" These are questions of basic significance in the study of peda­ gogy. They put mind back in the picture, in the central place it now occupies in cognitive psychology. The mental life of teachers and learners has again beome crucially significant for the study of teaching (Shulman, 1981, and Shulman, this vol­ ume), and from an interpretive point of view mind is present not merely as a set of " mediating variables" between the major independent and dependent variables of teaching - the in­ puts and outputs. Sense-making is the heart of the matter, the medium of teaching and learning that is also the message. Interpretive, participant observational fieldwork research, in addition to a central concern with mind and with subjective meaning, is concerned with the relation between meaning-per­ spectives of actors and the ecological circumstances of action in which they find themselves. This is to say that the notion of the social is central in fieldwork research. In a classic statement Weber (1922/1978) defined social action: " A social relationship may be said to exist when several people reciprocally adjust their behavior to each other with respect to the meaning which they give to it, and when this reciprocal adjustment determines

128

FREDERICK ERICKSON

the form which it takes" (p. 30). Standing somewhere, for example, is a behavior. Standing in line, however, is social ac­ tion, according to Weber's definition, because it is meaningfully oriented to the actions of others in the scene- others standing in the line, and in the case of a typical school classroom, the teacher in charge, who has told the children to stand in line. All these others in the scene are part of ego's social ecology. Pat­ terns in that ecology are defined by implicit and explicit cultural understandings about relationships of proper rights and obliga­ tions, as well as by conflicts of interests across individuals and groups in access to certain rights. Thus, for example, in the standing-in-line scene, the official rights of the person occupy­ ing the status of teacher differ from the rights of those persons occupying the status of student. Moreover, there is an addition­ al dimension of difference in rights and obligations, that be­ tween the official (formal) set of rights and obligations and the unofficial one. Officially, all children in the line have the obliga­ tion to obey the teacher's command to stand in line. Unoffi­ cially, however, some children may have rights to obey more casually than others. These differences among the children can be thought of as an unofficial, informal social system within which status (one's social position in relation to others) and role (the set of rights and obligations that accrues to a particu­ lar status) are defined differently from the ways they are defined in the official, formal system. A basic assumption in interpretive theory of social organiza­ tion is that the formal and informal social systems operate si­ multaneously, that is, persons in everyday life take action together in terms of both official and unofficial definitions of status and role. A basic criticism of standard research on teach­ ing that follows from this theoretical assumption is that, to the extent that teacher and student roles are accounted for in the predetermined coding categories by which classroom observa­ tion is done, the category systems take no account of the unoffi­ cial, informal dimensions of role and status in the classroom. This is not only to miss the irony and humor of the paradoxical mixing of the two dimensions of classroom life, but to miss the essence of the social and cognitive organization of a classroom as a learning environment. Classrooms, like all settings in for­ mal organizations, are places in which the formal and informal systems continually intertwine. Teaching, as instructional lead­ ership, consists in managing the warp and woof of both di­ mensions in dealing with children and their engagement with subject matter. To attempt to analyze classroom interaction by observing only the warp threads and ignoring the woof threads is to misrepresent fundamentally the process of pedagogy. The focus on social ecology - its process and structure- is intrinsic in interpretive social research on teaching. The re­ searcher seeks to understand the ways in which teachers and students, in their actions together, constitute environments for one another. The fieldwork researcher pays close attention to this when observing in a classroom, and his or her fieldnotes are filled with observations that document the social and cultural organization of the events that are observed, on the assumption that the organization of meaning-in-action is at once the learn­ ing environment and the content to be learned. All human groups have some form of social organization. While it is universal that regularly interacting sets of individuals possess the capacity to construct cultural norms by which their

social ecology is organized- face to face, and in wider spheres up and out to the level of the society as a whole- the particular forms that this social organization takes are specific to the set of individuals involved. Thus we can say that social organization has both a local and a nonlocal character. Let us consider the local nature of social organization first, and then the nonlocal nature of it. Interpretive social research presumes that the meanings-in­ action that are shared by members of a set of individuals who interact recurrently through time are local in at least two senses. First, they are local in that they are distinctive to that particular set of individuals, who as they interact across time come to share certain specific local understandings and traditions - a distinctive microculture. Such microcultures are characteristric of all human groups whose members recurrently associate. These are so-called natural groups, which are the typical unit of analysis studied by fieldwork researchers. The ubiquity of these natural group-specific microcultures can be illustrated by the following example. Compare and contrast the daily routines of two upper-middle-class white American families, one of which is characterized by serious emotional pathology, and the other of which is not. As the family systems are viewed from the out­ side, on the surface, patterns in the conduct of everyday life may seem very similar. Both families live in the same suburb, next door to one another. Both houses have dining rooms. Both families use paper towels in the kitchen and buy Izod sports shirts for their children. Yet in one family there is deep trouble, and in the other there is not. The microculture of one nuclear family regarding child-rear­ ing and other aspects of family life differs in at least some re­ spects from that of a family living next door that is identical to the first in ethnicity, class position, age of parents and children, and other general demographic features. These differences, al­ though small, are not at all trivial. They can have profound significance for the successful conduct of daily life. This is at­ tested to by the personal experience of marriage, in which ego learns that ego's in-laws and spouse hold somewhat different assumptions from those held by ego about the normal conduct of daily life, and that these others may be just as deeply con­ vinced as is ego of the inherent rightness of their own customary ways of doing things. The same is true for school classrooms. Interpretive re­ searchers presume that microcultures will differ from one class­ room to the next, no matter what degree of similarity in general demographic features obtains between the two rooms, which may be located literally next door or across the hall from one another. Just as the adjacent suburban families differed, so two classrooms can differ in the meaning-perspectives held by the teacher and students, despite the surface similarities between the two rooms. Almost every American elementary school classroom today has fluorescent lights. Regulations governing the amount of floor space that must be provided for a given number of children mandate that American classrooms will be roughly the same size. Regulations mandate a roughly similar ratio of adults to students. Entering virtually any elementary school classroom one will see arithmetic workbooks, a pub­ lished basal reading series, a chalkboard, dittoed work sheets, some books to read, crayons for coloring, paste, and scissors. Roughly the same level of skills is taught at the various grade

QUALITATIVE METHODS IN RESEARCH ON TEACHING

levels throughout the country. How then to account for the substantial differences in patterns of student achievement across different classrooms? It may be that the differences in organization that we need to be interested in are quite small indeed, and radically local - little differences in everyday class­ room life that make a big difference for student learning, subtly different meaning-perspectives in which it makes sense to stu­ dents to learn in one classroom and does not make sense to learn in another classroom, from a student's point of view. Meanings-in-action are assumed by some interpretive re­ searchers to be local in a second and more radical sense, that of the locality of moment-to-moment enactment of social action in real time. Today's enactment of breakfast in a family differs from yesterday's, and in conversation during today's breakfast, the content and process of one person's turn at speaking and the reaction of the audience to what is said will differ from that of the next turn at speaking, and audience reaction. Life is con­ tinually being lived anew, even in the most recurrent of custom­ ary events. This is assumed to be true of school classrooms as well. Positivist research on teaching presumes that history repeats itself; that what can be learned from past events can generalize to future events- in the same setting and in different settings. Interpretive researchers are more cautious in their assumptions. They see, as do experienced teachers, that yesterday's reading group was not quite the same as today's, and that this moment in the reading group is not the same as the next moment. What constitutes appropriate and intelligible social action in class­ rooms and all other natural human groups is the capacity for a set of individuals to live together successfully in the midst of the current moment, reacting to the moment just past and expect­ ing the next moment to come. This is the world of lived exper­ ience, the life-world (Lebenswelt). The life-world of teacher and students in a classroom is that of the present moment. They traverse the present moment together across time as surf­ boarders who ride the crest of a wave together with linked arms. It is a delicate interactional balancing act. If any one in the set wavers or stumbles, all in the set are affected. Each individual in the set has a particular point of view from within the action as the action changes from moment to mo­ ment. During the course of the enactment of recurrent types of events (e.g., breakfast, math lessons) some of these individual perspectives come to be intersubjectively shared among the members of the interacting set. Members come to approximate one another's perspectives, in at least a rough correspondence among the individually differing points of view even though these are not identical. Since each individual in the set is unique, however, the specific content of shared understandings at any given moment and across moments and days is unique to that particular set of individuals. Thus within a given moment in the enactment of an event and during the overall course of shared life together, particular sets of individuals come to hold distinc­ tive local meanings-in-action. These meanings are also nonlocal in origin. Face-to-face social relations do indeed have a life of their own, but the materials for the construction of that life are not all created at the moment, within the scene. One nonlocal influence on local action is culture, which can be defined in cognitive terms as learned and shared standards for perceiving, believing, acting,

129

and evaluating the actions of others (see the discussion in Goodenough, 1981, pp. 62ff.). Cultural learning profoundly shapes what we notice as well as what we believe, at levels out­ side conscious awareness as well as within awareness. Students in an ordinary American classroom speak English, a culturally learned language system that connects the students with the lives of others across space and time, back before the Norman Conquest. Much of what they know of the language is outside conscious awareness - that is, children come to school know­ ing how to use grammatical constructions of which they de­ velop a reflective awareness only after some years of schooling. Students in American classrooms ieam a particular cuitural tradition of mathematical reasoning, and an arithmetic symbol system derived from Arabia. These cultural traditions are non­ local in provenience. Another source of nonlocal influence is the perception that local members have of interests or constraints in the world be­ yond the horizon of their face-to-face reiations. In a school classroom these influences may come from the teacher next door, from parents, from the principal, from institutionalized procedures in the federal government regarding the allocation of special resources to the classroom. There is indeed a social structure within which classroom life is embedded. In that sense Durkheim was right- society is a reality in itself, and there are social facts of which local actors take account. Here, however, the interpretive researcher parts company with Durkheim, for the issue is how to take account of the reality of nonlocal culture and society without assuming me­ chanistic causal linkages between these outside realities and the realities of social relations face to face. Interpretive research takes a somewhat nominalist position on this issue of ontology: Society and culture do exist, but not in a reified state. Social class position, for example, does not "cause" school achieve­ ment - people influence that, in specific interactional occasions. The structure of the English language system does not " cause" the way specific people speak- the speaker makes use of the system in individually distinctive ways. The principai's memo that an achievement test is to be given Friday morning does not "cause" the teacher's hand motions as he or she passes out the test booklets - that behavior is the result of meaning-interpre­ tations and choices, deliberate and nondeliberate, that the teacher has made, including the choice not to ignore the memo's injunction. The task of interpretive research, then, is to discover the spe­ cific ways in which local and nonlocal forms of social organiza­ tion and culture relate to the activities of specific persons in making choices and conducting social action together. For classroom research this means discovering how the choices and actions of all the members constitute an enacted curriculum- a learning environment. Teachers and students in their interac­ tion together are able to (a) make use of learned meaning ac­ quired and shared through acculturation (not only language system and mathematical system, but other systems such as po­ litical ideology, ethnic and class subcultures, assumptions about gender roles, definitions of proper role relationships be­ tween adults and children, and the like); (b) take account of the actions of others outside the immediate scene, making sense of them as structure points (or better, as production resources) around which they can construct local action; (c) learn new

130

FREDERICK ERICKSON

culturally shared meanings through face-to-face interaction; and (d) create meanings, given the unique exigencies of practical action in the moment. Some of these emergent solutions become institutionalized as distinct local traditions. Others of these created meanings are improvised in uniquely concerted ways, given the unique per­ spectives ofjust that local set of interacting individuals. Indeed, even what can be called "following rules" can be seen as involv­ ing more than passive compliance to external constraints. Indi­ viduals are not identically socialized automatons performing according to learned algorithmic routines for behavior (such blind rule followers are described by Garfinkel, 1967, as "judg­ mental dopes" and "cultural dopes," pp. 67-68). Rather, they are persons who act together and make sense, according to the cultural "rules" which as they enact, they vivify in situationally specific ways. Thus the local microcultures are not static. The microculture can be drawn upon by the members of the local group as they assign meanings to their daily action, but because of the constant, intense dialogue between the interpretive per­ spective provided by the microculture, the exigencies of practi­ cal action in the unique historical circumstances of the present moment, and the differences in perspective among members of the interacting group, the ways in which the evolving microcul­ ture can influence the actions of group members is a dynamic process in which change is constant. From this perspective, in research on teaching it is the surface similarities across classrooms or across reading groups that seem trivial and illusory, rather than as in the standard perspec­ tive, in which the local differences we have been describing are seen as trivial - as uninterestingly "molecular" variation that can be ignored in the analysis of general characteristics of effec­ tive teaching. Mainstream positivist research on teaching searches for general characteristics of the analytically general­ ized effective teacher. From an interpretive point of view, how­ ever, effective teaching is seen not as a set of generalized attributes of a teacher or of students. Rather, effective teaching is seen as occurring in the particular and concrete circum­ stances of the practice of a specific teacher with a specific set of students "this year," "this day," and "this moment" Uust after a fire drill). This is not to say that interpretive research is not interested in the discovery of universals, but that it takes a different route to their discovery, given the assumptions about the state of na­ ture in social life that interpretive researchers make. The search is not for abstract universals arrived at by statistical generaliza­ tion from a sample to a population, but for concrete universals, arrived at by studying a specific case in great detail and then comparing it with other cases studied in equally great detail. The assumption is that when we see a particular instance of a teacher teaching, some aspects of what occurs are absolutely generic, that is, they apply cross-culturally and across human history to all teaching situations. This would be true despite tremendous variation in those situations - teaching that occurs outside school, teaching in other societies, teaching in which the teacher is much younger than the learners, teaching in Urdu, in Finnish, or in a mathematical language, teaching narrowly con­ strued cognitive skills, or broadly construed social attitudes and beliefs. Despite this variation some aspects of what occurs in any human teaching situation will generalize to all other sit­ uations of teaching. Other aspects of what occurs in a given

instance of teaching are specific to the historical and cultural circumstances of that type of situation. Still other aspects of what occurs are unique to that particular event, and to the par­ ticular individuals engaged in it. The task of the analyst is to uncover the different layers of universality and particularity that are confronted in the specific case at hand - what is broadly universal, what generalizes to other similar situations, what is unique to the given instance. This can only be done, interpretive researchers maintain, by at­ tending to the details of the concrete case at hand. Thus the primary concern of interpretive research is particularizability, rather than generalizability. One discovers universals as mani­ fested concretely and specifically, not in abstraction and gener­ ality (see the discussion in Hamilton, 1980). Among anthropologists this point is made in the distinction between ethnography, the detailed study of a particular society or social unit, and ethnology, the comparative study of differing societies, or social units. The basis for valid ethnological comparison, however, is the evidence found in detailed ethnographic case studies, not in data derived from surveys. (See the discussion in Hymes, 1982, who asserts that educational ethnology rather than ethnography is the most fundamental task for interpretive fieldwork research in education.) In linguistics, from which the term concrete universal comes, the point is made in distinguishing between universal and spe­ cific structural features of human languages. One cannot study the topic of human language in general. One finds in nature only specific human languages. Only by detailed understanding of the workings of a specific language, followed by comparative analysis of each language considered as a system in its own right, can one distinguish what is universal from what is specific to a given language. One can begin to distinguish the universal from the specific by comparing languages with differing struc­ tural properties, for example, Navaho, Ojibwa, Urdu, Chinese, Yoruba, Finnish, Greek, English, but only if one understands very thoroughly the organization of each language as a distinct system, through developing a fully specified model of each sys­ tem. Partial models of each system, and a sampling procedure that randomly selected a few sentences from each of the lan­ guages would not be an adequate empirical base for studying human language as a general category. Interpretive social research on teaching presumes that the same obtains for teachers and classrooms. Each instance of a classroom is seen as its own unique system, which nonetheless displays universal properties of teaching. These properties are manifested in the concrete, however, not in the abstract. Such concrete universals must be studied each in its own right. This does not necessarily mean studying classrooms one by one. But it does presume that the discovery of fully specified models of the organization of teaching and learning in a given classroom must precede the testing of generalization of those models to other classrooms. The paradox is that to achieve valid discov­ ery of universals one must stay very close to concrete cases. For interpretive researchers, then, the central focus of pro­ cess-product research on teaching on the production of gen­ eralizable knowledge seems inappropriate. The following quotation from Brophy (1979) illustrates the way in which the concern for generalization drives the enterprise to process-pro­ duct research: "A study involving 20 classrooms studied for 20 hours each is almost certainly going to be more valuable than a

QUALITATIVE METHODS IN RESEARCH ON TEACHING

study of a single classroom for 400 hours or a study of 400 classrooms for one hour each, other things being equal (i.e., sophistication of research design)" (p. 743). That could only be true if fully specified models had been developed, and if class­ rooms were generically similar enough that subtle variations across them were trivial in what for lack of better terms we can call the social and cognitive organization of teaching and learning.

The Mainstream Perspective in Research on Teaching The history of mainstream positivist research on teaching for the past 20 years is one of analytical bootstrapping with very partial theoretical models of the teaching process, on the as­ sumptions that what was generic across classrooms would emerge across studies, and that the subtle variations across classrooms were trivial and could be washed out of the analysis as error variance. This approach to studying teacher effectiveness can be seen as a borrowing by American educational researchers of an ap­ plied natural science model for research and development ex­ emplified by agricultural experimentation. Research and development using a positivist natural science approach is possible in agriculture because of the uniformity of the phenomena that are considered. While the chemical compo­ sition of the soil may vary from one field to the next, and weather conditions may vary from year to year, the fundamen­ tal variables that are considered - chemicals, genetic structures of plants, the biochemistry of plant metabolism and growth - are constant enough in form and bounded enough in scope that it is possible to conduct research and development by the operations of repeated measurement, prediction, and controlled experimental intervention. This is research by means of the design and testing of "treatments " whose effects can be monitored and whose working can be explained by references to a theoretical apparatus of covering laws. In the first Hand­ book of Research on Teaching it was just such theory and re­ search design that was called for in the introductory chapter by Gage ( 1 963) - the positivist model of science borrowed from the natural sciences of psychology, with Hempel providing the fundamental rationale in philosophy of science (see the discus­ sion in Smith, 1 979). The first Handbook contained what since became the classic article on experimental design (Campbell & Stanley, 1966), according to which an agricultural kind of re­ search and development could be conducted. Campbell ex­ tended these recommendations in later proposals for large-scale program development. These were interventions that could be studied as quasi-experiments (Cook & Campbell, 1 979). Twenty years later it seems that there is so much variation across classrooms, and so much variation in the implementa­ tion of "treatments" themselves that large-scale program eval­ uation by quasi-experimental methods is very problematic. As that became apparent in study after study Campbell himself ( 1 978) and Cronbach (1975) called for the use of more natural­ istic observational methods - case studies done by participant observers, or "documentation" studies, which would give a de­ tailed view of the actual structure and process of program im­ plementation. At the same time, Bronfenbrenner ( 1 977) was

131

calling for an "ecological" approach to the study of child devel­ opment, considering the child in the context of family and com­ munity life. These approaches, while advocating the use of methods other than those of the experiment or the social survey (testing and measurement in education are considered here as one form of survey research), still did not consider going be­ yond the bounds of the fundamental natural science paradigm for educational research, with its underlying assumption of the uniformity of nature in social life. A story similar to that for attempts at large-scale program evaluation can be seen in recent research on teacher effective­ ness, in which the classroom was the unit of analysis, rather than the program. This so-called "process-product" research (the term is that of Dunkin & Biddle, 1 974) developed during the late 1 960s and early 1970s (see the review of major studies in Brophy & Good, this volume). The last fifteen years of this work can be seen as a search for an increasingly specific look at causal linkages between teacher effectiveness, as measured by end-of-the-year student gain scores on standardized achievement tests, and particular teach­ ing practices. The teaching practices were monitored firsthand by observ­ ers who noted the occurrence of various types of predetermined teacher behaviors and student behaviors (e.g., teacher ques­ tions, teacher praise, teacher reprimand, student "on-task" be­ havior, student " off-task " behavior). In this approach, called systematic classroom observation, the types of behavior of inter­ est for observation were chosen according to their theoretical significance. What was "systematic" was the use of predeter­ mined categories themselves. This was to assure uniformity of observation (reliability) across times of observation in the same classroom and across different classrooms. The concern for re­ liability of measurement reflected the positivist assumptions behind the research. As the work has progressed, coding categories for a while became more specific and differentiated. Then as certain vari­ ables (such as student on-task behavior) seemed to correlate highly with gains in student test scores across multiple studies, the observational systems focused more and more on theoreti­ cally salient types of student and teacher behavior, which were generalized functions. Subsequent experimental "treatment" studies indicated that when teachers increased certain behaviors that were found in the correlational studies to be asscociated with increased stu­ dent achievement gains, those gains occurred in the experimen­ tal classrooms. (See the review in Brophy & Good, this volume. Students in the classrooms receiving the experimental treat­ ments in some cases achieved higher scores on standardized tests than did children in control group classrooms in which the frequency of the recommended teacher behaviors was much less. This is hopeful news for educators. It suggests that an agri­ cultural model for inquiry into educational productivity is an appropriate one. In the model the teacher, as Mother Nature, provides the fertilizer, light, and water that enable the students, as plants, to grow tall and strong. All this seems quite straightforward. Why then might any other form of research on teaching be necessary? Interpretive participant observational research is very labor intensive, while observation by use of predetermined coding categories is much

132

FREDERICK ERICKSON

less so. It would seem that there is no need for interpretive re­ search, or any other. The findings on teacher effectiveness seem to be all in. That would be a premature conclusion, however. The case for interpretive research is pointed to by some interesting anomalies in the process-product work. One such anomaly lies in the corpus of process-product data itself. Apparently, in correlational studies of the same teacher across school years, the stability of teacher effects on student achievement is not high (see the discussion in Brophy and Good, this volume. This could be due to a number of influences, for which there is no evidence in the correlational data sets, for example, teachers teaching somewhat differently with each new set of students, stress in the teacher's life outside school (birth of child, death in family, divorce, remarriage), stress or change in the school itself (introduction of new reading series, change of principal). The process-product data do not indicate why teacher influence seems to vary from year to year. Another anomaly is that in spite of evidence that indicates that certain teacher behaviors can influence students to learn more, and in spite of experience that shows that teachers can be trained to use those behaviors more frequently, teachers do not always persist in using the recommended behaviors. Sometimes they do, but sometimes they do not. An example of this is Rowe's finding (1974) that waiting longer for student answers produces more reflective answers by students. Teachers can be told this, and trained to pause for a longer "wait-time," yet after a few months they go back to using shorter wait-time in lesson dia­ logue with students. One wonders if wait-time might not have negative meaning to teachers in the concrete circumstances of conducting classroom discussion. Such a concrete, enacted meaning might override whatever more abstract and decon­ textualized meaning that wait-time behavior might have as gen­ erally correlated positively with student learning. How do teachers make sense such that a behavior like wait-time seems sociolinguistically inappropriate? What are the intuitions about interaction against which doing wait-time behavior runs counter? How might these intuitions be changed - or is there another behavioral means that might provide a less counterin­ tuitive route to the same ends? Those are questions about the specifics of practice that derive from the perspectives of inter­ pretive research. These kinds of anomalies suggest that while the standard work has produced some insights about general characteristics of effective training, we may have learned about all that is possi­ ble by proceeding with that theoretical frame of reference, and the methods that derive from it.

An Interpretive Perspective on Teacher Effectiveness The use of predetermined coding categories by process-product researchers presupposes uniformity of relationships between the form of a behavior and its meaning, such that the observer can recognize the meaning of a behavior time after time. Imagine a student sitting at a desk, looking out the window. What does this mean? Is the student on-task or off-task? We must infer meaning from the observed behavior. What are the

grounds for such inferences? When they must be made in split­ second judgments by coders, what evidence do we have that such inferences about meaning are valid? The fundamental problem with the standard approach to observational research on teacher effectiveness, from an interpretive perspective, is that its evidence base is invalid. Surface appearances are taken as valid indicators of intended meaning. In consequence, what are claimed to be low-inference observational judgments are in fact highly inferential. Once the data are coded there is no way to retrieve the original behavioral evidence to test the validity of the inferences made about the behavior's meaning (see the dis­ cussion on this point by Mehan, 1979). No matter how strong the correlations appear to be in such data sets, a good possibilty always exists that such correlations are spurious, if relation­ ships between behavioral form and social meaning are as vari­ able as interpretive researchers claim them to be. Moreover, if such variability is inherent in social life and thus omnipresent in classrooms, experiments that purport to manipulate teacher and student behaviors, so giobally defined, are likely to be shot through with confounding relationships between putative "treatment" conditions, "control" conditions, and "outcomes" that invalidate the causal inferences that are made. The standard research on teacher effectiveness could only proceed as it has done on the presupposition of uniformity of nature in social life that follows from adopting natural science models for social scientific inquiry. Interpretive research makes very different assumptions. It looks for variability in relation­ ships between behavioral form and intended meaning in class­ room interaction. Moreover, interpretive research on teaching repeatedly discovers locally distinctive patterns of performed social identity-of enacted statuses and their attendant role re­ lationships, such that a phenomenon like time-on-task is locally meaningful in terms of the particular performed social identities of the actual students spending time on the academic tasks as­ signed to them. If Mary, a high achiever, is observed by the teacher to be off-task at a given moment, this may mean some­ thing quite different from Sam, a problem student, being ob­ served as off-task in the same moment. One of Sam's obligations as a problem student (who is perceived as often be­ ing off-task) may be to be constantly on-task (since this will be "good for him"). Mary, on the other hand, who as a high achiever is perceived as (by definition) being on-task most of the time, does not have Sam's obligation to be constantly on­ task. Indeed Mary has earned the right to take occasional "breaks" - time off-task. One is reminded of the differences in work rights and obligations between hourly wage employees, who punch a time clock, and salaried workers, who do not. Yet even the role distinction between Sam and Mary is not entirely absolute. Some mornings, if Sam is having an unusually good day (i.e., if he appears to be working diligently and constantly) he may have earned, for that morning, the right to take a break, like Mary, the salaried worker. The contrast between the interpretive and the standard per­ spectives can be further illustrated by considering classroom social organization in terms of the metaphor of a chess game. Standard research on teacher effectiveness presupposes a stan­ dard board (curriculum and aims), a standard set of chess pieces (statuses of teacher and student), and a standard set of rules of procedure that govern the relations among the pieces

QUALITATIVE METHODS IN RESEARCH ON TEACHING

(roles of teacher and student) that are appropriate, that is, pos­ sible within the game. Interpretive researchers presume that the board itself, the number and shapes of its "squares" - places to be in the curriculum- will vary from one classroom to the next, although on the one hand, with the publication of textbooks for reading and arithmetic with teacher's manuals and accompany­ ing worksheets for students, and on the other hand, with ac­ countability systems for management by objectives and continual achievement testing, there seems to be more pressure for uniformity of curriculum and aims than there was a genera­ tion ago. Even if one grants a superficial uniformity of the board itself, when one comes to the direct observation of actual playings of the game - observation that is unmediated by pre­ determined coding categories- one finds that the types of pieces vary from game to game. In one game there are many pawns, few knights, and no bishops. In another game there are no pawns, many knights, and many bishops. Since each type of piece is allowed to move differently on the board, the system of possible movements - the system of social relations changes from game to game. Moreover, some interpretive re­ searchers would argue that the differences among games, as they are actually played, are even more profound than the dif­ ferences that would obtain if it were only a matter of having a different board or different pieces from one game to the next. If within a given game, neither the board nor the pieces are them­ selves entirely fixed - if the definitions of aims, curriculum, and the social identities and roles of teachers and students are con­ stantly emergent in negotiation within the action of teaching and learning itself - then the school classroom is indeed a fun­ damentally different kind of social universe than the stable, fixed, and unidimensional one presupposed by positivist re­ search on teaching. It is as if in the chess game, the white bishop has a mistress, the red king knows this but the white king doesn't, and the red king chooses at some times to take advan­ tage of his knowledge to pressure the white bishop through blackmail, while at other times the red king chooses to ignore the white bishop's secret. To see the school classroom as a chess game that is multidimensional, filled with paradox and contra­ diction from moment to moment and from day to day, is to see the school classroom, and teaching, as a game of real life. The study of classrooms, interpretive researchers would argue, is a matter of social topology rather than social geometry. A central task for interpretive, participant-observational re­ search on teaching is to enable researchers and practitioners to become much more specific in their understanding of the inher­ ent variation from classroom to classroom. This means building better theory about the social and cognitive organization of particular forms of classroom life as immediate environments for student learning. Conclusions drawn from process-product research can sug­ gest in general terms what to do to improve student achieve­ ment, but these general recommendations give neither the researcher nor the practitioner any information about how, specifically, to do what is called for. Some examples of recom­ mended teaching behaviors are found in a recent review article by Rosenshine (1983, p. 338): • Proceed in small steps (if necessary) but at a rapid pace. • Use high frequency of questions and overt student practice.

133

• Give feedback to students, particularly when they are cor­ rect but hesitant. • Make corrections by simplifying questions, giving clues, ex­ plaining or reviewing steps, or reteaching lost steps. • Ensure student engagement during seatwork (i.e. teacher or aide monitoring). The teaching functions called for by Rosenshine are global; for example, give feedback, simplify questions to correct, insure student engagement during seatwork (i.e., insure time-on-task). The functions could be performed in a myriad of different ways, appropriately and inappropriately, on differing occasions. How to understand what might be appropriate and what not, in spe­ cific cases, goes beyond the bounds of standard research on teacher effectiveness. In considering issues of teacher effectiveness interpretive re­ searchers might ask, "How is time-on-task manifested in differ­ ent classrooms and at different times by different students within a given room? What is clear feedback, from differing stu­ dent points of view and teacher points of view? Does any one of the possible ways of giving clear feedback actually take place in the concrete circumstances of face-to-face communication or in writing between teacher and student? For that matter, if rela­ tionships between teacher and student are fully interactional (i.e., reciprocal), how do students give teachers clear feedback? How do student actions influence teacher productivity - the teacher's time-on-task?" To conclude, there are three very serious problems with stan­ dard process-product research on relationships between class­ room interaction and student achievement. The first problem is that the work proceeds from an inadequate notion of interac­ tion- one-way causal influence as a behavioral phenomenon - rather than reciprocal exchange of phenomenologically meaningful action. The second problem is that the standard work gives an extremely reduced view of classroom process. Its use of predetermined coding categories as a means of primary data collection gives no clear detailed evidence about the spe­ cific classroom processes that are claimed to lead to desired outcomes. The third problem is that the product studied is too narrowly defined- usually as end-of-the-year achievement test scores. With the standard approach to the study of teacher ef­ fectiveness having provided so reduced and one-dimensional a view of classroom processes, classroom products, and class­ room interaction itself, it is not unreasonable to claim that the final word has not been spoken on this issue in research on teaching. From an interpretive point of view, teacher effectiveness is a matter of the nature of the social organization of classroom life - what we have called the enacted curriculum- whose con­ struction is largely, but not exclusively, the responsibility of the teacher as instructional leader. This is a matter of local meaning and local politics, of teaching as rhetoric (persuasion), and of student assent as the grounds of legitimacy for such persuasion and leadership by a teacher. As Doyle (1979) puts it in a felici­ tious phrase, students in classrooms are not the "passive recip­ ients of instructional treatments" (p. 203). In sum, issues of local politics at the classroom level seem to be at the heart of educational decision making by teachers and by students. Moreover, one can use the notions of politics and

134

FREDERICK ERICKSON

persuasion to consider an essential activity of schools as institu­ tions, that of social sorting.

Power, Politics, and the Sorting Functions of Teaching The sorting activities of schools occupy central interest in inter­ pretive research on teaching. In developed countries the avail­ ability of universal public schooling is a means justifying the allocation of individuals across generations across the range of occupational slots available in the society. Conservative sociol­ ogists, such as Parsons ( 1959), liberals such as Clignet and Foster ( 1966), and radical sociologists such as Willis ( 1977) and Bourdieu and Passeron ( 1 977) agree on the importance of this sorting function. Opinion differs over whether or not a particu­ lar society's school sorting procedures are justifiable or not, on grounds of fairness to individuals and groups. According to lib­ eral social theory, school sorting procedures would be fair if they were universalistic, that is, if sorting criteria applied to in­ dividuals as individuals, along dimensions of comparison that apply universally to all persons regardless of such attributes of status as gender, race, social class, or religious preference. From the early work (e.g., Henry, 1 963) through the recent work of Willis ( 1977), much fieldwork research in education has been concerned with identifying the particularistic bias inherent in the putatively universalistic standard operating procedures of schools. At the classroom level, fieldwork has investigated the particularistic bi.as that is implicit in the kinds of environ­ ments that are established by teachers. The presumption is that the low school achievement of social and cultural minority stu­ dents is better explained by considering the character of the classroom learning environment than by attributing the typical pattern of school failure of those children to deficiencies in individual intelligence and motivation. For anthropologists especially it has seemed odd that in de­ veloped societies the school failure rate is so high among the majority of the population, who are of working-class or under­ class status. This pattern stands in sharp contrast to that found in various nonliterate societies, in which almost everyone in the society acquires the knowledge and skills necessary for survival according to the pattern of adaptation developed in the particu­ lar society. That may have been true in nonliterate soci­ eties for the five million years of human evolution. Wolcott ( 1982) quotes Gearing in a question to modern societies : " It's something of a wonder that anyone ever learns anything. But given that they do, then we can also ask why everybody doesn't learn everything?" (See also Gearing & Sangree, 1 979, p. 1.) Is this because socialization of the young is done more effectively in nonliterate societies? Is the apparent difference in the success of teaching and learning somehow due to the difference in scale between large developed societies and small nonliterate ones? Or might this difference also have to do with something about the institutionalization of teaching and learning in the school as a formal organization? Or, because the school failure rate is highest among the lower classes, is there something wrong with them? One possibility is that lower class and minority populations are genetically inferior; that across generations gene pools have developed in these populations that produce, on the average, an

overrepresentation of individuals of lower intelligence than those found in populations of white, upper-middle-class Ameri­ cans. This genetic deficit theory was proposed in the late 1 960s by Jensen ( 1969). Another possible explanation lies in a family socialization deficit hypothesis. If the life circumstances of the poor are dif­ ficult and if their vision of life possibilities is limited, families of the poor may not provide children with the amounts of intellec­ tual stimulation and motivation for achievement that middle­ class families provide. Socialization deficit hypotheses were proposed during the 1960s under the labels of "cultural de­ privation," " linguistic deprivation," and "family disorganiza­ tion" (e.g. Riessman, 1 962). It was argued that school subjects and intelligence tests required abstract thought and that lower­ class families developed only concrete reasoning skills in their children. Numerous studies were conducted by child develop­ ment researchers in which invidious comparisons were made between the child-rearing patterns of lower class and middle­ class families (e.g., Hess & Shipman, 1 965). In the United States and Great Britain a large body of litera­ ture developed that criticized the genetic and socialization defi­ cit hypotheses, characterizing them as "blaming the victim" (see Keddie, 1 973). The argument over socialization deficit hypotheses tended to be conducted across disciplinary lines. Much of the deficit-oriented research and the prescriptions for teaching practice that followed from it were done by psycholo­ gists in the fields of education and child development. Much of the critique of the deficit hypotheses came from anthropolo­ gists, sociologists, and linguists. Cole, a notable exception, was a cognitive psychologist who conducted cross-cultural research that showed that nonschooled people often simply did not know the point of school-like tasks used in intelligence tests, by which they could be assessed as mentally deficient when in fact they were just using a different way of making sense (Cole & Scribner, 1974; Scribner & Cole, 1981). Anthropologists and sociologists with linguistic training found the school failure rate among low-SES and minority pop­ ulations in developed societies especially odd, in light of what was coming to be known about the cognitive demands of first­ language acquisition by children. It was apparent that virtually every child who is not severely impaired physically or neurolog­ ically comes to school at age 5 having mastered the basic struc­ ture of the language spoken at home, its grammar and sound system. Linguists had contended that less prestigious regional, social class, and racial dialects were no less cognitively complex than the standard language spoken in school (cf. Labov, 1 972, and Erickson, 1984). Modern language-acquisition theory viewed mastering the grammar and sound system of a language as necessarily requir­ ing complex, abstract, cognitive abilities, even though the thinking that took place was outside conscious awareness. Given that mastery of the speaking knowledge of a language was far more cognitively complex than beginning to learn to read the written form of that language, how was it that many children appeared to have great difficulty with simple, begin­ ning reading? Children of low SES and of ethnic, racial, and cultural minority background could be seen, in school and out­ side it in the home and local community, to be able to speak much better than they could read. What might account for this?

QUALITATIVE METHODS IN RESEARCH ON TEACHING

One line of explanation, proposed by anthropologists, lin­ guists, and by some sociologists, was that subtle subcultural differences between the community and the school led to inter­ actional difficulties, misunderstanding, and negative attribu­ tions between teachers and students in the classroom. The preponderance of this work identified specific cultural differ­ ences between teachers of majority group background and low­ SES, minority group children. The cultural differences consisted principally in implicit assumptions, learned outside conscious awareness in everyday life in the home and in the community, about the appropriate conduct of face-to-face interaction. Some of the basic properties in the organization of interaction that were investigated (often through comparative studies of chil­ dren's lives at home and at school) were phonological and gram­ matical dialect features in children's speech that teachers had difficulty understanding (Piestrup, 1973), children's means of showing attention and understanding through nonverbal be­ havior such as gaze and nodding (Erickson, 1979), and differ­ ences in the organization of turn-taking in conversation that lead to overlapping of speakers or to long pauses between turns (Watson-Gegeo & Boggs, 1977 ; Shultz, Florio, & Erickson, 1982). Mehan ( 1979) published a study of question-answer se­ quences in school lessons that revealed the tremendous com­ plexity involved in managing such conversation. His analysis suggested the possibility of miscommunication due to different cultural expectations for the fine tuning of classroom discourse. More global aspects of interaction patterns that differed be­ tween home and school were also identified. These had to do with the cultural organization of social relationships in com­ munication, that is, with foundational definitions of appropri­ ateness in leadership and followership, in adult roles and in child roles. Among the topics investigated were differing cul­ tural assumptions about the appropriateness of indirectness and directness (a) in the exercise of social control and in the use of a "spotlight" of public attention by asking content questions of named individuals (Philips, 1 982; Erickson & Mohatt, 1 982; (b) in the very situation of an adult asking "teacher-like" ques­ tions of a child - questions the child can presume the adult al­ ready knows the answer to (Heath, 1982); (c) in the differences in assumptions about the appropriateness of competitiveness and in cultural definitions of students offering and receiving help from one another as showing laudable concern for others, or as cheating ; and (d) in cultural notions of appropriateness of humor and mock aggression in discourse (Lein, 1 975). Taken together, cultural differences between home and school that have been identified at the level of basic structural properties in the organization of interaction, and at the level of global differences in assumptions about appropriate role rela­ tionships between adults and children, involve fundamental building blocks, as it were, of the conduct of classroom interac­ tion as a medium for subject matter instruction and for the in­ culcation of culturally specific values - definitions of honesty, seriousness of purpose, respect, initiative, achievement, kindli­ ness, reasonableness. When students act in ways that do not match the classroom teacher's cultural expectations, the chil­ dren's behavior can be perceived by teachers as frustrating, con­ fusing, and sometimes frightening. Given the teachers' and the students' recurring difficulties in interacting together from day

135

to day, an adversarial relationship is likely to be set up between the teacher and the student. This would inhibit the teacher's ability to learn from the students - to assess accurately what the students know, what they want educationally, and what they intend interpersonally in social relations with the teacher. Recent work in Alaska and Hawaii appears to support the cultural difference hypothesis. In both cases, as teachers have interacted with students in the classroom in ways that resemble those that are culturally appropriate in the home and commun­ ity, student achievement on standardized tests increases dra­ matically. The Alaskan study (Barnhardt, 1982) reports the situation in a small village school in the Alaskan interior. Achievement by Athabaskan Alaskan native children of the vil­ lage was low until Alaskan native teachers began to teach in the three classrooms of the school: Grades 1 -2, 3-4, and 5-6. After the native teachers arrived student achievement rose dramati­ cally in all three classrooms. Subsequent participant observa­ tion and videotape analysis revealed that the teachers organized instruction and interacted with students in ways that were culturally appropriate. Exercise of social control was for the most part very indirect, and the teachers usually avoided public reinforcement - not only avoiding negative reinforce­ ment of children's actions, but avoiding overt positive reinforce­ ment as well. These patterns are typical of child-rearing in the community, and resemble those reported in Oregon and North­ ern Ontario by Philips ( 1982) and by Erickson and Mohatt ( 1982). The patterns found in the Athabaskan classrooms resemble patterns documented in Alaskan Eskimo classrooms by Collier (1973). From this study it is not absolutely clear that the cultural patterns of instruction were the main influence on increased student achievement, since the native teachers at the school were also lifelong residents of the village, and their presence in the role of teacher may have increased rapport with parents and changed the climate of family and community expectations for children's school achievement. Still the evidence is highly sug­ gestive that not only were the new teachers local natives, but they also taught children in forms of interaction that resembled those that were appropriate in family and community life out­ side the school. In the Hawaiian case, evidence supporting the cultural differ­ ence hypothesis is even more clear than in the Alaskan case. In an innovative school program developed for native Hawaiian children researchers discovered that when the children were al­ lowed to use overlapping speech while discussing reading sto­ ries in reading groups, their reading achievement rose. Previous ethnographic research had established that overlapping speak­ ing turns was characteristic of certain kinds of conversations in the community (Au & Jordan, 1 980). In subsequent experimen­ tal research (Au & Mason, 1 98 1 ), material of equivalent diffi­ culty was taught under two different conditions of social organization of discourse. In one condition the teacher allowed the students to overlap one another's speaking turns while dis­ cussing the reading story. In the other condition the teacher did not allow overlapping speech during the discussion of the story. The children's achievement was clearly higher under the first condition, in terms of proximal indices of achievement, such as error rates during the lesson, and in scores on tests admin­ istered directly after the lesson. In subsequent development

136

FREDERICK ERICKSON

work this alternative procedure for teaching reading, which incorporates culturally congruent discourse patterns into the overall design for reading pedagogy, is now being implemented in public school classrooms with native Hawaiian children. Similar positive results in student achievement have occurred, as indicated both by proximal indices of achievement and in end-of-the-year scores on standardized tests. What might account for these results, theoretically? One line of explanation concerns the nature of face-to-face interaction as a learning task environment. In interaction in school lessons a dimension of culturally patterned social organization (patterns for turn-taking, listening behavior, and the like) always coexists with the dimension of the logical organization of the informa­ tion content of the subject matter. The two dimensions- social organization and subject matter organization - are always re­ flexively intertwined in the enactment of a lesson. (For full dis­ cussion, see Erickson, 1982b, 1982c.) One reason that cultural congruence in the social organization of interaction in lessons seems to lead to higher student achievement may be that when the social organization of lesson interaction happens in ways that are culturally customary- already mastered through over­ learning in daily life outside school - this simplifies the task en­ vironment of the lesson, allowing children to concentrate more fully on the subject matter content. In other words, lessons may be easier for children when their social organization dimension is clear and familiar. This theory of lesson interaction as a social and cognitive task environment may provide an alternative explanation to the finding that highly ritualized lesson interaction formats ap­ pear to lead to higher achievement by cultural minority chil­ dren even if the lesson formats are not congruent with cultural patterns for the social organization of interaction that are found in the student's home and community. Stallings and Kaskowitz (1974) report this finding in the evaluation of alter­ native models for Follow Through. The DISTAR instruction format, highly ritualized and not culture-specific, seems to result in higher student achievement, even for cultural minority populations for which the lesson format is quite culturally alien, as in the case of native Americans. The DISTAR format may have this result, not because it happens to fit a direct instruction model, but because the format by its very ritualization is so clear and easy to learn that it is soon mastered by children. Once learned, the ritual format would simplify the lesson as a task environment. Indeed, the communication of a teacher's expectations for the conduct of interaction in ways that are clear and predictable, and the establishment of implicit or explicit consensus between the teacher and students that these ways of interacting are just, may be the fundamental feature that characterizes both the cul­ turally incongruent teaching strategies, such as DISTAR, and the culturally congruent ones, such as the Kamehameha read­ ing program in Hawaii. If clarity is of the essence, and if clarity can be achieved by instructional means that are culture-specific and culturally congruent, as well as by means that are culturally incongruent, then a wider range of policy options becomes available for improving the academic performance of cultural minority students. The cultural difference hypothesis assumes that differences in expectation for the conduct of interaction are a systematic

source of breakdowns in interaction that is analogous to the notion of linguistic interference in second-language acquisition. When features of the grammar and sound system of two lan­ guages differ, one can predict the likely recurrence of certain types of structural errors. For example, if in some language other than English the /th/ sound does not occur, but the /d/ sound does occur, one can predict that a speaker will con­ sistently say "dis" for "this" when speaking English. Or if in some language other than English gender reference is signaled by some means other than alternative pronouns, one can pre­ dict that the speaker will substitute "he" for "she"and vice versa when speaking English. Analogously, one can predict that a teacher who is not used to overlapping speech in conversation (such as that found among working-class native Hawaiians and Italian-Americans) will interpret the overlapping talk as inter­ ruption, even though the children do not interpret the behavior of overlapping talk as an interruption. It follows, then, that cul­ turally congruent social organization of instruction can reduce the situations of interactional interference that occur in the classroom, and that the reduction of these interactional difficul­ ties increases student opportunity to learn and decreases mis­ understanding between teacher and student. But if interactional interference, by itself, is the chief factor that inhibits student learning, how can one explain the results of the Follow Through Evaluation? How does one explain reports of other instances of culturally incongruent instruction that ap­ pear to raise student achievement? One such instance is re­ ported in a case study of a residential school for Alaskan natives in which instruction was conducted in culturally incongruent ways and yet in which student motivation and achievement were high (cf. Kleinfeld, 1979). It would seem that interactional difficulty and miscommunication are not simply a matter of structural interference between cultural patterns of the com­ munity and of the school. A possible explanation lies in considering as a political phe­ nomenon the local microculture and social organization of classroom life and its relation to student learning. If we think of classroom teaching and learning as a matter of local politics, some relationships between cultural difference or similarity, social relationships among teachers and students, and student learning begin to appear. These relationships are much less clear when we think of teaching and learning as a matter of individual psychology (whether behaviorist, cognitive, or so­ cial), or even in terms of the sociology and anthropology of the classroom as an ecosystem. When we consider individual func­ tioning in the context of a sociocultural ecosystem, we have a framework for an anatomy of classroom teaching and learning. When we consider the dynamic operation of the ecosystem as a political process we have a physiology of teaching and learning. Central to such a framework are the concepts of power, au­ thority, influence, competing interest, legitimacy, assent, and dissent. Power, as the ability to coerce the actions of others, is poten­ tially possessed both by teachers and by students in the class­ room. Authority, the legitimate exercise of power and focus of socially sanctioned knowledge and judgment, resides officially with the teacher. Influence, the unsanctioned capacity to exer­ cise power, resides with students. Every person who has at­ tempted to teach faces the reality of student influence in relation

QUALITATIVE METHODS IN RESEARCH ON TEACHING

to teacher authority. Even in institutional arrangments of schooling that vest the teacher with virtually unchecked au­ thority (traditional religious instruction being a vivid case in point), the exercise of that authority in the absence of student assent can at best lead to outward conformity to the teacher's will, that is, in a teaching situation the student always possesses the ability to resist by refusing to learn what the teacher intends should be learned. The teaching-learning transaction, then, can be seen as an inherently political and rhetorical situation, in which at least the implicit consent of the governed must be gained by the governor through persuasion. The teacher must somehow persuade the followers that his or her guidance is legitimate and in the student's own interest. If the student per­ ceives his or her interest to be fundamentally in conflict with that of the teacher, and if the student resists the teacher by with­ holding learning, the teacher is unable to teach. Thus in the classroom social system as a political economy, the power to withhold the currency that is essential to the system - student learning - ultimately resides with the student. This is true even if student resistance is covert and the student does not engage in more overt forms of protest. The interactional sabotage we call "discipline problems" can be seen as a form of interactional judo - control of the ostensibly stronger party by the ostensibly weaker one. A crucial question for educational research then becomes : What are the conditions of micropolitics in the social organiza­ tion of classroom life that set off a contest of wills between teacher and students in which the students refuse to learn what the teacher intends to teach? Some student failure may indeed be due to lack of student ability or motivation that lies outside the teacher's ability to change it as the conventional wisdom of educators and educational psychologists suggests. But some student failure may be more accurately seen as a matter of mi­ cropolitical resistance. The overrepresentation of student fail­ ure to learn simple knowledge and skill among low-SES and cultural minority populations of students is suggestive in this regard. The interpretation of school failure as evidence of self-defeat­ ing resistance rather than as evidence of inadequacy on the part of students has been most consistently maintained by British sociologists of education, who see the production of student failure in schools as necessary for the maintenance of the exist­ ing class structure in society. In a recent review (1983) Giroux surveys this work. He makes the critical point that it is impor­ tant to restrict the notion of resistance and not use the term loosely to refer to any sort of inappropriate or self-defeating action by a student or by a teacher. In the United States this position has been asserted in a series of papers by McDermott ( 1974, 1977) and McDermott and Gospodinoff (1981) that criticize both the family socialization deficit hypothesis and the cultural difference hypothesis as ex­ planations for school failure. His argument derives in part from psychiatrists' theories accounting for the generation of psycho­ pathology in family relationships. One of McDermott's princi­ pal sources was Scheflen's adaptation ( 1960) of Bateson, Jackson, Haley, and Weakland's (1956/1972) theory of the in­ teractional "double bind "as the cause of schizophrenia in chil­ dren. An analogue to Scheflen's position is found in the work of Laing ( 1970). In the pathological family certain family members

137

become locked in patterns of regressive relations with other family members. The situation is not caused by the action of any single individual. The entire family system - its locally ne­ gotiated and maintained system of statuses and roles - sup­ ports and maintains the adversarial relationship between the parties who are manifestly at odds. To return to the chess meta­ phor for the social organization of face-to-face relations, pawns, knights, and kings by their patterned actions enable each other to act in concert. We see similar situations in which individuals become locked into relationships that are mutually punitive or are mutually destructive in other ways : in bad marriages, in alcoholic or abusive families, in recurrent difficulties in relations between a supervisor and a subordinate in a work group. Over time, interpersonal conflict develops a history. It ramifies throughout the whole social unit of interacting individuals. McDermott contends that this is what happens in school class­ rooms, among teachers and students who, for the most part unwittingly, are mutually failing one another. The student can be seen as playing an active role in this as student and teacher collaborate in producing a situation in which the student achieves school failure (McDermott, 1974). Another source of McDermott's position comes from recent work on interethnic relations in two-person interview situations (Erickson, 1975, Erickson & Shultz, 1982). Shultz and I found that in interethnic and interracial interviews between junior college counselors and students, certain kinds of cultural differences in communi­ cation behavior (e.g., differences in signaling attention and un­ derstanding through nonverbal listening behavior) were associated with other kinds of interactional trouble and nega­ tive interpersonal attribution in some interviews, conducted by a given counselor, yet in other interviews conducted by that counselor with students whose ethnic or racial background and cultural communication style matched that of the students with whom the counselor had had trouble, the same features of cul­ turally differing communication behavior that had led to trou­ ble with one student did not lead to serious, ramifying interactional difficulty with another student: Even though mo­ mentary difficulty due to culturally differing behavior styles could be observed, the trouble that occurred was soon recov­ ered from. It did not escalate the way it did in other interethnic or interracial interviews conducted by the same counselor. This suggested a micropolitics of cultural difference in interaction. (For a disscussion of the role of culture difference in the larger­ scale politics of interethnic relations see Barth, 1969.) Under some circumstances, cultural differences in communication pat­ terns became a resource for interpersonal conflict, while in other circumstances the same kinds of behaviors were not reacted to and made use of as a resource for conflict. A simple interference explanation for the negative effects of cultural dif­ ference on the conduct of interaction was inadequate to explain our data. The significance of cultural difference as an inhibiting factor in classroom teaching and learning may be that cultural differ­ ence can function as a risk factor. As a source of relatively small interactional difficulties, cultural differences can become re­ sources for the construction of much more large-scale and wide­ spread conflict between teachers and students. It is in this sense that cultural differences along the lines of social class, ethnicity, race, gender, and handicapping conditions can play a role in the

138

FREDERICK ERICKSON

creation of classroom situations in which some students with­ hold learning as a form of resistance to teachers. To summarize, a wide range of explanations exist for the high rates of school failure in developed societies. At one ex­ treme are explanations that presume a radical individual deter­ minism. These explanations identify some deficit in the individual learner, whether due to genetic inheritance or to en­ vironmental socialization, as the primary cause of school failure by students. A related set of explanations identifies similar defi­ cits in the individual teacher, identifying the teacher as pri­ marily responsible for student failure. This is the implication of the process-product research on teaching which suggests as a policy conclusion that individual teachers who are instruction­ ally ineffective need either to be retrained or to be removed from the classroom. At another extreme are explanations that presume a radical contextual or societal determinism as the source of school fail­ ure. These explanations identify the inequitable distribution of power and privilege in society as the root cause of school failure of low-SES and cultural minority children. Schools function, these theorists argue, as passive sorting mechanisms that repro­ duce the social class position of individuals from one generation to the next. (See, e.g., Bowles & Gintis, 1 976.) In the absence of widespread change in power relations along the lines of social class, race, and gender, school achievement patterns will not change and any minor changes in achievement at the classroom level that might result fron remedial work with teachers and students are at best trivial and at worst pernicious, since they would mask the need for fundamental social change. Currently the aim of boring drill and denial of opportunities. for indepen­ dent reasoning in classrooms is to produce a docile work force from the compliant students, and to justify the existence of a permanently unemployed underclass made up of noncompliant students, who resisted by refusing to learn. A middle position is taken on this issue by many interpretive researchers. Such a position attempts to acknowledge the real­ ity of individual differences in aptitude and motivation for learning, the reality of cultural differences and their micropoliti­ cal significance, as it varies from classroom to classroom, the reality of the sorting functions of schools, as well as the reality of the functions of schools to stimulate learning and broaden opportunity among students whose life circumstances are lim­ ited, and who are thus at risk for school failure. Many interpre­ tive researchers acknowledge that the tension for the classroom teacher between the responsibility to be the student's impartial judge and the responsibility to be the student's advocate is an inherent source of tension in the role of the teacher as it is insti­ tutionally defined in the United States. This contradiction in the teacher role, identified more than a generation ago by Waller ( 1932), can be seen as genuinely inherent as a paradox that is not reducible. In Japan and in other educational systems in which examinations are centralized and are administered by an agency external to the classroom, the teacher's role is not so contradictory as it is in the United States. In Japan the teacher prepares the student for the examinations, functioning as the student's advocate throughout the process of elementary and secondary schooling (Vogel, 1965). No univocal social theory, whether conservative, liberal, or radical, provides by itself an adequate explanation for phe-

nomena of school achievement in the United States. According to conservative social theory the essential inferiority of the lower classes as less intelligent and hard working than the higher classes is evidenced by the manifest class differences in school achievement. According to liberal social theory, inequi­ ties currently exist in school sorting practices, but if sorting were done more objectively according to universalistic judg­ ment criteria the schools could provide equality, or equity, of opportunity. According to radical social theory, changes in school sorting practices can only result after fundamental social change since the current sorting practices serve to legitimate current class division and are thus maintained at the school system and classroom level by pressures from the wider society. Interpretive research accommodates the reality of the local organization of teaching and learning at the classroom level, together with the reality of external pressures on the organiza­ tion of the classroom. Both levels of organization must be en­ compassed, theoretically and empirically, in an account of the micropolitics of classroom organization, within which low-SES minority children fare more or less well from one classroom to the next. This suggests a set of issues and questions for future interpre­ tive research on teaching. We must ask what are the specific features of social organization and meaning that arise on a given classroom ecosystem; the enacted hidden curriculum of social organization and the enacted manifest curriculum of sub­ ject matter organization, which must be considered together. We can ask about the relation of this enacted curriculum to the range of kinds and amounts of student learning that take place. Learning here can include cognitive learning of subject matter, but the notion of learning would not be limited to this single aspect. In order to study these issues it is necessary to ask what the specific conditions are by which teachers and students construct local social organization in ways that increase or de­ crease differing specific kinds and amounts of student resistance to learning the manifest curriculum. We can consider how this situationally embedded, locally produced resistance to learning (and other kinds of difficulty in learning) varies by class, race, ethnicity, and gender. We can consider the ways in which the situationally embedded informal classroom social system -the statuses and roles available to students in it, the locally pro­ duced resistance to learning and other sources of learning diffi­ culty - is organized in relation to statuses external to the classroom, such as class, race, ethnic, and gender identity. We can consider how all the local social organization relates to nonlocal, external sources of influence and how the settings ex­ terior to the classroom are influenced by what happens inside the room. We can ask how all this relates to student learning and to teacher morale. Such study can begin with the assumption that learning and teaching are intrinsic to the biological and social foundations of human adaptation, within the life cycle and across generations. Learning, it can be assumed, is not optional for humans, and we would not expect it to be so for students in classrooms. The basic issue is not that some students learn and others do not. We can assume that all students are learning something. The basic issue is that many students, for a variety of different rea­ sons, do not appear to be learning what the teacher and the school claim to be teaching. Both the claims regarding what is

QUALITATIVE METHODS IN RESEARCH ON TEACHING

being taught and the claims regarding what is being learned need to be scrutinized, in the context of the wider societal in­ fluences and the local meaning systems that are created as teachers and students influence one another in the teaching and learning environment of enacted curriculum, the specifics of which must be identified because they vary from classroom to classroom as does student achievement. The core issues in teacher and student effectiveness concern meaningfulness - the grounds for legitimacy and mutual assent - rather than causa­ tion in a mechanical sense. The inquiry involves a search for interpretive understanding of the ways in which particular indi­ viduals engage in constructing patterns of action and meaning by which they enable one another to accomplish desired (or undesired) ends. The particular means they construct for col­ laboratively accomplishing those ends are expected to vary across each specific classroom, and within classrooms from year to year. There may be universal principles of organization by which people collectively foster or inhibit the accomplishment of their stated goals. These, it is assumed, can only be discova ered by studying particular instances in close detail, since the universal principles are realized in ways that are locally unique. Finally, such study must also link the immediacy of the local lives of students and teachers, inside and outside the classroom, to nonlocal and general aspects of social structure and culture. For the interpretive participant observational researcher, this must be done by looking out from the classroom to the wider world, as well as looking in to the classroom from the wider world, as functionalists and radical critics both tend to do. One of the most immediate places to look outside the class­ room is the student's own family and local community. Many claims are made about the influence of the home and commun­ ity on the child's learning in the classroom. It may be that the main reason for differences in student achievement according to class and parents' educational background is that parents who are higher in SES and in education know how to coach their children in school subjects, while parents who are lower in SES and in experience with educational success themselves do not know how to do this coaching. It may be that parents who are higher in SES and in educational attainment model the mean­ ingfulness and usefulness of knowledge and skill that are acquired in school, while parents of lower SES and educational attainment do not model this. It may be that, as Ogbu asserts (1978), parents and other members of a caste-like minority group with a pariah status in the society, such as American blacks, communicate a sense of hopelessness to their school-age children, while parents and other members of a non-caste-like minority group (such as the American Chinese) communicate the more hopeful belief that school success is the route to adult success. The caste-like minority student, according to Ogbu, by failing to strive, plays an active role in achieving school failure. (There is something that strikes one intuitively as realistic in this hypothesis, in its portrayal of the low-achieving student as an active agent in the construction of his or her own victim status. In that sense Ogbu's hypothesis is reminiscent of that of McDermott.) Currently, however, there is no substantial body of empirical evidence against which to judge these claims. To test any of these assertions we would need specific knowledge of the life experiences of students who vary in educational achievement

139

within the same classroom, and within class, racial, ethnic, and gender categories. There are a few studies that have begun to investigate students' lives outside school in relation to their lives inside school, notably Heath (1983), as well as studies mentioned earlier in this discussion. These studies have not fo­ cused specifically on variation in classroom achievement within as well as between demographic background factors such as class, race, ethnicity, and gender, among students from the same classroom. To test Ogbu's hypothesis, for example, we would need to follow high-achieving and low-achieving American black students from the same classroom, as well as high-achiev­ ing Chinese-American students from the same classroom, to see if the high-achieving black and Chinese-American students were receiving a qualitatively different set of implicit and explic­ it messages about achievement attribution than were the low­ achieving black American students. It hardly seems possible that the low achievement pattern among low-SES black stu­ dents is explained by so simple a matter as the presence of sig­ nificant others who say to the student, implicitly or explicitly, "You can't make it no matter what you do." How consistent are such messages? How do they relate to the message in the mean­ ing system of the classroom? How does Ogbu's hypothesis ac­ count for the fact that in some classrooms low-SES black students do better than they do in other classrooms? The rela­ tionships between what happens in students' lives outside and inside the classroom, and the relations between that and school achievement are not at all clear. The same is true for teachers. In recent work, Cusick ( 1980) asserts that teachers in two public high schools he studied voted on curriculum issues differently depending on the ways in which a particular curriculum decision might affect a second job they held or a strong avocational interest they had. We do not know the ways in which teachers who are parents of children may be influenced in their teaching by the demands (and rewards) of family life, nor how this might vary with the age of the teacher's children, nor how the situation of teachers who are parents might contrast with that of teachers who are not parents. A host of new questions can be raised concerning the ways in which different kinds of teachers and students make sense of the differ­ ences between the concrete circumstances of their lives outside and inside school.

Data Collection Of all the aspects offieldwork research, data collection has been the most discussed in the literature on methods. In the interest of economy of exposition, this discussion of issues in data col­ lection is kept to a minimum. I will review major themes and issues in data collection and will refer the reader to main sources in the literature for discussion at greater length. One approach to data collection in the field is to make it as intuitive - or as radically inductive - as possible. The convic­ tion is that with long-term, intensive participant observation in a field setting, begun with no prior conceptual expectations that might limit the fieldworker's openness to the uniqueness of ex­ perience in the setting, an intuitive sense of relevant research questions and of conclusions regarding pattern will emerge by induction. From this point of view, fieldwork is seen as an

140

FREDERICK ERICKSON

almost mystical process, essentially unteachable. The best preparation is solid grounding in substantive courses in anthropology and/or sociology. After learning relevant sub­ stantive theory and after reviewing empirical results of field­ work research the novice researcher proceeds to the field and does fieldwork. Anthropologists, especially, have set forth this mystical con­ ception of fieldwork as unteachable. It is said of Alfred Kroeber that when a doctoral student came for advise on fieldwork research methods before embarking on a study of a native American society somewhere in California, Kroeber made the following comments: 1. First, find your Indians (i.e., don't study the wrong group by mistake). 2. Pads of paper and pencils are very useful. 3. Be sure to take a frying pan, but don't loan it to anyone; you may not get it back. This is an extremely romantic notion of fieldwork. One enters the field with no preconceptions, and learns the methods by doing them (as one can learn to swim by being thrown in the pool). After tremendous emotional stress one finally induces grounded analytic categories. The likelihood of stress is in­ creased if one experiences not only emotional trauma while in the field, but contracts an exotic, debilitating disease such as malaria. Only after returning home does one crack the code of local lifeways and solve the interactive analytic puzzle. Another approach to data collection is to make the process as deliberative as possible. That is what is argued for here. There is no warrant, in contemporary philosophy of science and cognitive psychology, for the romantic conception of fieldwork, in which the fieldworker arrives in the setting with a tabula rasa mind, carrying only a toothbrush and hunting knife. One can argue that there are no pure inductions. We always bring to experience frames of interpretation, or schemata. From this point of view the task of fieldwork is to become more and more reflectively aware of the frames of interpretation of those we observe, and of our own culturally learned frames of interpreta­ tion we brought with us to the setting. This is to develop a distinctive view of both sides of the fence, what Bohannon ( 1963, pp. 7-8) has characterized as the stereoscopic social vision of the ethnographer. When we consider fieldwork as a process of deliberate in­ quiry in a setting (cf. Pelto & Pelto, 1 977; Levine, Gallimore, Weisner, & Turner 1 980) we can see the participant observer's conduct of data collection as progressive problem solving, in which issues of sampling, hypothesis generation, and hypothe­ sis testing go hand in hand. Fieldworkers' daily presence in the setting is guided by deliberate decisions about sampling and by intuitive reactions as well. When and where these observers go, whom they talk to and watch, with whom they participate in daily activities more actively and with whom they participate with a more distanced observational stance - all these involve strategic decisions about the nature of the key research ques­ tions and working hypotheses of the study. All research decisions are not deliberate, however. Because of this the toothbrush and hunting-knife school has a valid point in reminding us of the importance of induction, intuition, and

intensive firsthand presence in the setting. From the point of view of a more deliberative conception of fieldwork, however, the central issue of method is to bring research questions and data collection into a consistent relationship, albeit an evolving one. This is possible, we argue here, without placing shackles on intuition and serendipity. Framing research questions explicitly and seeking relevant data deliberately enable and empower intuition, rather than stifle it. In the absence of a deliberative approach to fieldwork some typical problems of inadequate evidence emerge at the stage of , data analysis after leaving the field. These are problems that 'might have been avoided had different strategic decisions been made at the stage of data collection, when midcourse correction was still possible. There are five major types of evidentiary inadequacy. 1 . Inadequate amounts of evidence. The researcher has too lit­ tle evidence to warrant certain key assertions. The field­ worker's daily round did not include the scenes in which evidence could have been collected that would have con­ firmed the assertion. 2. Inadequate variety in kinds of evidence. The researcher fails to have evidence across a range of different kinds of sources (e.g., direct observation, interviewing, site documents) to warrant key assertions through triangulation. The re­ searcher did not seek triangulating data while in the field. 3. Faulty interpretive status of evidence. The researcher fails to have understood the key aspects of the complexity of action or of meaning perspectives held by actors in the setting. (Participation was not long enough or intensive enough in key recurrent scenes, and/or interviewing and direct obser­ vation did not complement one another, and/or the re­ searcher was deceived by informants who lied and faked because they did not trust the researcher or did not agree with the researcher's aims.) 4. Inadequate disconfirming evidence. The researcher lacks data that might disconfirm a key assertion. Moreover, the researcher lacks evidence that a deliberate-search was made for potentially disconfirming data while in the field setting. This weakens the plausibility of the absence of disconfirm­ ing evidence and leaves the researcher liable to charges of seeking only evidence that would support favorite interpre­ tations. (The researcher, not having realized while in the field the relations between research questions and data col­ lection, failed to identify a key assertion during fieldwork, and consequently failed to search for evidence that might disconfirm the assertion or that might stand as discrepant cases, whose analysis while in the field might shed new light on the assertion and its theoretical presuppositions.) 5. Inadequate discrepant case analysis. The researcher did not scrutinize the set of disconfirming instances, examining each instance (i.e., discrepant case) and comparing it with the confirming instances to determine which features of the disconfirming case were the same or different from the anal­ ogous features of the confirming cases. Such comparative feature analysis often reveals flaws in the original assertion, which if rewritten can account for the discrepant cases as well as accounting for those initially thought to have been confirming instances. The remaining members of the set of

QUALITATIVE METHODS IN RESEARCH ON TEACHING

disconfirming instances are genuinely discrepant cases that are not accounted for by the assertion. (Discrepant case an­ alysis will be illustrated by classroom examples in the next major section of the chapter on data analysis and report writing. The point to be noted here is that discrepant case analysis enables the researcher to refine and adjust major assertions and their theoretical presuppositions. If such an­ alysis is not done two types of errors can result: (a) the researcher rejects an assertion prematurely, by counting as similar all instances that seem, upon first analysis, to be disconfirming ones, and/or (b) the researcher fails to refine, and adjust the assertions that appeared at the first analysis' to be confirmed by the data.)

Issues of Site Entry and of Research Ethics Potentially good fieldwork research can be compromised from the outset by inadequate negotiation of entry in the field setting. This leads to problems of data quality and of research ethics. The researcher's interest is in the broadest possible kinds and amounts of access. Given the potential problems of evidentiary adequacy noted above, the fieldworker wants ideally to be able to observe anywhere in the setting at any time, and to be able to interview any member of the setting on any topic. This may or may not be in the best interests of those in the setting. Issues of special interest and special risk arise, not only between mem­ bers of the setting and its outside constituencies (school district staff and its local community, the state education department, federal education agencies); these issues of special interest and risk also arise within and across system levels in the organiza­ tion (the interests of teachers in relation to a principal, the inter­ ests of students in relation to teachers). Two basic ethical principles apply. Those studied; especially those studied as focal research subjects, need to be (a) as in­ formed as possible of the purposes and activities of research that will occur, and of any burdens (additional work load) or risks that may be entailed for them by being studied. Focal re­ search subjects also need to be (b) protected as much as possi­ ble from risks. The risks involved can be minimal. Their nature is not that of physical risk, as in some medical experiments. Psychological and social risks (embarrassment and/or liability to administrative sanction) are usually the kind entailed in fieldwork research. Still, the risks of psychological and social harm can be substantial when fieldwork is done by an institu­ tionally naive researcher who has not adequately anticipated the range of different kinds of harm to which persons of varying social position in the setting are potentially liable. Liability to risk is often greatest between members of differ­ ing interest groups in the local setting. Reporting to a general scientific audience usually does not expose local people to risk. Rather, it is reporting in the local setting that needs to be con­ sidered in the negotiation of access to information about indivi­ duals in the setting. The researcher is in a perplexing situation. He or she needs to have done an ethnography of the setting in order to anticipate the range of risks and other burdens that will be involved for those studied. While it is not possible, at the outset, to antic-

141

ipate all the ethical issues that will emerge, it is possible to antic­ ipate many of them and to negotiate about them with those interest groups in the setting whose existence and whose cir­ cumstances are apparent at the outset. In school settings, these general classes of group interest are students, teachers, parents, principals, central administrators. Some of the differing inter­ ests of these differing classes can be identified in advance. It is usually wise, for example, to guarantee to teachers that certain kinds of information about their teaching will not be available to their immediate supervisors, and that information about a student's home life will not be available to teachers. Such infor­ mation about teachers and about homes, however, might not expose individuals to risk if reported in the aggregate - all or many teachers, all or many homes. In special local circumstances the usual risks attendant to a given institutional position may not exist. For example, the in­ formation that a teacher does not use the basal reader as directed by the system-wide mandates for the reading program might count against the teacher in one principal's eyes, while for another principal with another point of view that same information would count in the teacher's favor. The researcher is wise to negotiate strict protection of infor­ mation at the outset of a study. When special circumstances warrant, these agreements can be changed to be more flexible later in the research process. Usually, however, it is more diffi­ cult to restrict access of higher-ups in the system to certain kinds of information later in the research process. The basic ethical principle is to protect the particular inter­ ests of especially vulnerable participants in the setting. Focal subjects are especially vulnerable, as are those who are single occupants of an institutional status (e.g., there is only one prin­ cipal and many teachers, but only one kindergarten teacher). People who are not focal in a study may be less at risk. (For example, if one is studying low-achieving students who are girls, high-achieving boys in the room are less likely to be at risk. They would appear in descriptive reports as part of the back­ ground, not in the foreground of the report.) In a study I directed in a large urban school system, for example (Cazden, Carrasco, Maldonado-Guzman, & Erickson, 1980), we were conducting a full year's participant observation and videotap­ ing with Hispanic bilingual teachers who were not fully certi­ fied. The teachers did not have tenure and were on annually renewable contracts. In that instance we negotiated an agree­ ment with the building principal, the district superintendent, and an associate superintendent in the central office that not only would the researchers never be asked to show their field notes to an administrator, but that no adminstrator would ask the researchers for oral characterizations of the teachers being studied or for access to the videotapes for any purposes of evaluation. The researchers anticipated showing some video­ tape footage to the principal and the faculty in staff meetings, but planned to do this in a second year, after data had been col­ lected, and after the teachers had reviewed the videotape foot­ age to be shown and had given consent that others see the footage. In any event, no footage would ever be shown anyone but the research staff without the teachers' consent. In addition, however, we had negotiated written agreement with adminis� trators that they would not even ask to see teacher-cleared foot­ age until after the next annual contract had been signed by the

142

FREDERICK ERICKSON

teachers. This did not totally eliminate risk to the teachers, but it minimized the risk of informal coercion while the teachers were being studied. As it happened, no requests for information or tape viewing were made by administrators during the year, but it was prudent to have anticipated this in negotiating entry. The researcher is wise to take great care in being explicit about uses of information and access to it, because it is in the researcher's interest to have as much access in the setting as is possible under conditions of high trust and rapport. Access in itself is of no use to the researcher without the opportunity to develop trust and rapport. The very process of explicit entry negotiation with all categories of persons likely to be affected by the research can create the conditions of trust that are neces­ sary. In consequence, we can see that ethical responsibility and scientific adequacy must go hand in hand in fieldwork research. If research subjects consent freely to be studied and if they do so having been informed of the purposes of research and the possi­ ble risks to them, as well as the possible benefits, then deception and faking are minimized, as is passive resistance to the re­ searcher's presence. In sum, negotiation of entry is a complex process. It begins with the first letter or telephone call to the site. It continues throughout the course of research, and continues after the re­ searcher has left the site, during later data analysis and report­ ing. Careful negotiation of entry that enables research access under conditions that are fair both to the research subjects and to the researcher establishes the grounds for building rapport and trust. Without such grounds mutual trust becomes proble­ matic and this compromises the researcher's capacity to identify and analyze the meaning-perspectives of those in the setting. There is considerable discussion of entry negotiation in the literature of fieldwork research methods. See especially Agar (1980, pp. 42-62), Bogdan and Biklen (1982, pp. 120-125), Schatzman and Strauss (1974, pp. 18-33), and Wax (1971, pp. 15-20, 84-93, 143-174).

Developing a Collaborative Relationship with Focal Informants Trust and rapport in fieldwork are not simply a matter of nice­ ness; a noncoercive, mutually rewarding relationship with key informants is essential if the researcher is to gain valid insights into the informant's point of view. Since gaining a sense of the perspective of the informant is crucial to the success of the re­ search enterprise, it is necessary to establish trust and to main­ tain it throughout the course of the study. One source of difficulty with trust is the tendency for infor­ mants to assume, whatever the researcher's presentation of the purposes of research was during the initial stages of negotiation of entry, that the researcher's purposes are in some way evalua­ tive. It is often necessary to reinterpret the purposes of research a number of times to the same informant. In addition, it is nec­ essary to explain the purposes of the study to each new infor­ mant one meets. If the new informant is not a potential key informant, a brief recounting of the study's purposes may suf­ fice, but it is often wise to give each new informant one meets a full explanation of the study's purposes because one cannot anticipate fully at the outset which informants will become key

later in the study. It is useful for the researcher to have virtually memorized a brief statement of the study's purposes, the proce­ dures that will take place, and the steps taken to maximize con­ fidentiality and minimize risk. If material cannot be kept confidential the informant(s) need to know that. If the re­ searcher will be present in a given scene in the role of partici­ pant observer, the informants need to know that in that scene their actions and comments are "on the record," even if the researcher is not writing notes or making a recording. If infor­ mants wish actions to remain off the record they need to under­ stand clearly that it is up to them to request that of the participant observer. Informant concerns about the observer's evaluative perspec­ tive make great sense, given the ubiquity of observation for eva­ luative purposes in schools. In an ultimate sense, the researcher's purposes are indeed evaluative, for to portray peo­ ple's actions in narrative reports is to theorize about the organi­ zation of those actions, and evaluation is inherent in any theory. We will return to this point in the next section of the chapter. The researcher can expect that informants will test the assur­ ances of confidentiality, nonjudgmental perspective, and other ethical considerations that were negotiated by the researcher. This testing usually happens early in the relationship with a new informant. The informant may ask for an evaluative com­ ment, or may reveal some harmless piece of information to the researcher and then check the organization's rumor network to see if the researcher revealed the item of information to anyone else in the setting. When doing team research it is very important for all mem­ bers of the research team to adhere strictly to a basic ground rule: Never make comments to other team members about any­ thing observed in the site, while you are at the site. Side conver­ sations between research team members on site can be over­ heard by participants in the site, sometimes with disastrous consequences for the credibility of the research team. An excellent way to establish and maintain trust in a setting is to involve the informants directly in the research, as collabo­ rators with the researcher(s). Some issues of trust come up at the outset of attempts by the researchers to develop a partner­ ship in research with key informants. The informants may per­ ceive such attempts by researchers as manipulative, since the self-perception of classroom teachers, especially, is that they are not experts; if anyone is the expert in a partnership between researchers and classroom teachers, the teachers may assume that it is the researchers who are the experts. This perception on the part of the teachers may persist despite disclaimers by the researchers. In time, however, a genuine partnership can de­ velop, in which the teacher and the researcher begin to frame research questions jointly and to collect data jointly. (On the process of collaborative fieldwork research on teaching, see Florio & Walsh, 1980.) In developing initial rapport, as well as in establishing a col­ laborative relationship with key informants, it is necessary that the researcher have a clear idea of the major research questions guiding the inquiry, and the likely data collection procedures that will be used to pursue the lines of inquiry suggested by the study's guiding questions. This presupposes the deliberative conception of the fieldwork research process that was alluded to earlier in this discussion.

QUALITATIVE METHODS IN RESEARCH ON TEACHING

Data Collection as an Inquiry Process Inquiry begins in the field with the research questions that guide the study. Three issues are crucial at the outset: (a) identi­ fying the full range of variation in modes of formal and informal social organization (role relationships) and meaning-perspec­ tives; (b) collecting recurrent instances of events across a wide range of events in the setting, so that the typicality or atypicality of certain event types with their attendant characteristic social organization can later be established; and (c) looking at events occurring at any system level (e.g., the classroom, the school, the reading group) in the context of events occurring at the next higher and next lower system levels. This means that if one were observing reading groups because a guiding research question led one to focus on issues in the teaching of reading, one would look at least at constituent events within the reading group event (sets of topically connected turns at speaking, teacher moves, student moves) and at events at the classroom level -for example, formal and informal status hierarchies of stu­ dents in the classroom as a whole, characteristic ways the teacher has of organizing interaction witq students in events other than reading. Ideally, one would wish to look at an even wider range of system levels for possible connections of influ­ ence-for example, building-level influences, exerted by other teachers and by the principal, that might affect the teacher's teaching, and influences from children's lives outside school in their homes and in other community settings that might in­ fluence their actions in the reading group. In fieldwork one never considers a single system level in isolation from other lev­ els; that is a basic feature of the sociocultural theory from which participant observational methods derive. In order to determine the full range of variation in social or­ ganizational arrangements, meaning-perspectives, and connec­ tions of influence within and across system levels in the setting and its surrounding environments, it is necessary to begin ob­ servation and interviewing in the most comprehensive fashion possible. Later in the research process one moves in successive stages to more restricted observational focus. The progressive problem solving of fieldwork entails a pro­ cess of sequential sampling. Because of the wide-angle view taken at the outset, it can be seen as "observing without any preconceptions," but that is a misleading characterization. Pre­ conceptions and guiding questions are present from the outset, but the researcher does not presume at the outset to know where, specifically, the initial questions might lead next. In con­ sequence the researcher begins with the most comprehensive possible survey of the setting and its surrounding environments. Concretely, that means that the researcher plans deliberately to spend time in particular places, at particular times. For ex­ ample, in studying a classroom, one would first begin by seek­ ing an overall sense of the neighborhood school community by collecting written information on the school community (e.g., census data), walking and driving around the community, and stopping in local shops. One would then enter the classroom, observing for complete days, from before the students arrive until they leave at the end of the day. Having identified the full range of events that occurred in the day, and having, through repeated observation, begun to establish the relative frequency of occurrence of the various event types, the researcher can be-

143

gin to focus on those events that are of central interest in the study. The researcher would begin to restrict the range of times and places at which observation occurred. Periodically, how­ ever, the researcher would want to return to more comprehen­ sive sampling, in order to restore breadth of perspective, and in order to collect more instances of events across the full range of events that occurred in the setting. This provides additional warrant for claims the researcher might later want to make re­ garding the typicality and atypicality (high and low frequency) of certain event types, or of certain role relationships within an event or range of events. As the researcher focuses on a more restrictive range of events within the setting, the researcher also begins to look for possible connections of influence between the setting and its surrounding environments. Unlike standard community eth­ nography, in which one begins with the whole community as the unit of analysis and moves progressively to investigate sub­ units within the community, in educational fieldwork one usually moves relatively quickly, after a brief general survey of the community, to continuous, focused study of a given educa­ tional setting (e.g., the classroom, the math lesson). After con­ siderable study of the focal setting the researcher moves out again to investigate its surrounding environments. The analytic task is to follow lines of influence out the classroom door into the surrounding environments. Cues to these lines of influence are found in site documents (e.g., memos enjoining certain ac­ tions within the classroom) and in comments of members in the setting (teachers, students) about those aspects of their lives outside the immediate setting that influence what takes place there. Informants are usually not fully aware consciously of the full range and depth of these influences, which include cultur­ ally learned and taken-for-granted assumptions about proper conduct of social relations, content of subject matter, human nature, and attitudes that shape one's definitions of what work, play, trustworthiness, academic ability, and the like might look like when encountered in everyday life in the classroom. With time, the fieldworker's notions of the phenomena that are most relevant to the study become clearer and clearer. In the final stages of fieldwork research, the focus may be very restricted indeed, as research questions and working hypoth­ eses become more and more specific. The process of fieldwork research as deliberate inquiry has been described by some anthropologists (e.g., Agar, 1980; Dobbert, 1982; Dorr­ Bremme, 1984; Goetz, & LeCompte, 1984; Levine et al., 1980; Pelto & Pelto, 1977), and by numerous sociologists (e.g., Glaser & Strauss, 1979; Lofland, 1976; Schatzman & Strauss, 1974). This process has been described for studies of teaching by Erickson (1973, 1977), and by Mehan (1979), among others. Limits of space preclude full discussion here. Some elabora­ tion on the nature of the inquiry process itself, however, is appropriate.

The Boundedly Rational Process of Problem Solving in Fieldwork In fieldwork the researcher is attempting to come to understand events whose structure is too complex to be apprehended all at once, given the limits on human information-processing capacity. These limits-what Simon (1957) calls bounded

144

FREDERICK ERICKSON

rationality-are compensated for in participant observation by spending time in the field setting. The participant observer, present in particular spaces and times in the field setting, waits for particular types of recurrent events to keep happening (e.g., disputes over land tenure, deaths, births, preparing the main meal of the day, seeing the next client at the unemployment office, having a reading lesson, exchanging turns at reading aloud, handing in a written exer­ cise). The researcher may seek out particular sites within a field setting where a particular type ofevent is most likely to happen. This gives the participant observer a situation analogous to that of the subject in a learning experiment- the opportunity to have multiple trials at mastering a recurringly presented task. In this case the task is that of learning how to observe analyti­ cally a particular type of event, and how to make records (field notes, audio and video recordings) of the actions that occur in the events, for purposes of more careful study later. Across each trial at observing a recurrent event the partici­ pant observer can alter slightly the focus of analytic attention. each time attending to some features of what is occurring and not attending to others. The observer can also vary the focus of attention in rereading field notes taken during the event, and in writing these up in expanded form after the day's observation has been completed. A fundamental principle is that this subse­ quent reflection and write-up, which usually takes at least as long as the time spent initially in observation, needs to be com­ pleted before returning to the field setting to do further observa­ tion. This means that the researcher needs to anticipate spending time writing up notes- a full day's observation in a classroom would need to be followed by a full day's (or night's) period of write-up. Write-up stimulates recall and enables the researcher to add information to that contained in the unela­ borated, raw notes. Write-up also stimulates analytic induction and reflection on relevant theory and bodies of research litera­ ture. There is no substitute for the reflection during fieldwork that comes from time spent with the original field notes, writing them up in a more complete form, with analytic insights re­ corded in them. In spite ofthe limits on information-processing capacity, time over time observation and reflection enable the observer to de­ velop an interpretive model of the organization of the events observed. These models are progressively constructed across a series of partial observations in a process that is analogous to a learning experiment in which the learner is presented a series of trials. Two sets ofprocedural decisions by the fieldworker have spe­ cial importance for correcting what is traditionally thought of as bias in sampling and observation -(a) the decisions the ob­ server makes about where to be in space and in time in the field setting, and (b) the decisions the observer makes about the foci of attention in any one occasion of observation. The former decisions affect the overall sampling of events that the partici­ pant observer makes. The latter decisions affect the complete­ ness and analytic adequacy ofobservations made cumulatively across a set of trials. A major strength of participant observation is the opportu­ nity to learn through active participation - one can test one's theory of the organization of an event by trying out various kinds of participation in it. A major limitation in fieldwork is

the partialness ofthe view ofany single event. There is, in conse­ quence, a tendency toward bias in sampling that favors the fre­ quently occurring event types since those are the ones one comes to understand most fully across time. We will discuss the consequences of the bias in the next sec­ tion of the paper. There is also another sense in which a bias toward the typical is present in fieldwork research. Given the limits on what can be attended to during any one observational trial, the observer may come to be dominated early on by a focus on an emerging theory of organization that is being in­ duced. As that happens the fieldworker may attend, while ob­ serving, mainly to those aspects of action that confirm the induced theory, overlooking other aspects of action that might be noted as data according to which the emergent theory might be disconfirmed. Thus potentially disconfirming evidence is less likely to be recorded in the fieldnotes than is the potentially confirming evidence. The researcher's tendency to leap to conclusions inductively early in the research process can be called the problem of prema­ ture typification. This problem makes it necessary to conduct (while in the field, and in subsequent reflection after leaving the field) deliberate searches for disconfirming evidence in the form of discrepant cases- instances of the phenomena of interest whose organization does not fit the terms of one's emerging theory. (On the importance of discrepant case analysis, see Mehan, 1979, and the classic statement of Lindesmith, 1974.) Another way to reduce the bias ofpremature typification and the bias toward emphasis on analysis of recurrent events at the expense of analysis of rare events is to include machine record­ ing in the research process. Audio or audiovisual records offre­ quent and rare events in the setting and in its surrounding environments provide the researcher with the opportunity to revisit events vicariously through playback at later times. Re­ cording of naturally occurring interaction in events does not substitute for firsthand participant observation and recording by means offieldnotes. Still, such recordings, subjected to syste­ matic analysis, can provide a valuable additional data source in fieldwork research. It is appropriate to discuss briefly the spe­ cial nature of machine recording and analysis in the process of fieldwork research. The discussion will anticipate slightly that to be covered in the next section on data analysis, but it is ap­ propriate here to highlight contrasts with the kinds of progres­ sive problem solving that are possible during firsthand participant observation in the field setting. The use of machine recording as a primary data resource in fieldwork research has been called "microethnography" by Erickson (1975, 1976, 1982a), "constitutive ethnography" by Mehan (1979), and "sociolinguistic microanalysis" by Gumperz (1982). The micro­ ethnographic research process has been described in detail by Erickson and Shultz (1977/81), by Erickson and Wilson (1982), and by Erickson (1982a). Machine recording and analysis differ from participant ob­ servation in one crucial respect. Unlike the participant observer the analyst of audiovisual or audio documentary records does not wait in the setting for instances ofa particular event type to occur. In reviewing the machine-recorded documentary evi­ dence, the analyst is freed from the limits of the participant ob­ server's embedding in the sequential occurrence ofevents in real time and space. The researcher indexes the whole recorded cor-

QUALITATIVE METHODS IN RESEARCH ON TEACHING

pus, identifying all the major named events recorded (e.g., les­ sons, meetings, recess periods, parent-teacher conferences) and identifying as well the presence in certain events of key infor­ mants. Then the researcher searches back and f orth through the entire recorded corpus f or instances of frequent and rare events, moving as it were back and f orth through time and space to identify analogous instances. This is analogous to the initial survey of the setting and its surrounding environments that the participant observer does by making choices of where to be in time and space in the setting. Then the researcher identifies a particular set of instances from the recorded corpus that are of special interest. The researcher at this point is able to revisit this set of instances vicariously by replaying them. The capacity to revisit the same event vicariously f or repeated observations is the chief innovation made possible by the use of machine re­ cordings in fieldwork research. The innovation has distinctive strengths and limitations. The first strength is the capacity for completeness of analysis. Because of the (theoretically) unlimited opportunity for revisit­ ing the recorded instance by replaying it, the instance can be observed from a variety of attentional foci and analytic perspec­ tives. This enables a much more thorough description than those that can be prepared by a participant observer from field notes. A second strength is the potential to reduce the dependence of the observer on primitive analytic typification. Because the in­ stance can be replayed, the observer has opportunity for deli­ beration. She can hold in abeyance interpretive judgements of the functions (meanings) of the actions observed. Often in parti­ cipant observation these interpretive inferences can be faulty, especially at the early stages of fieldwork. In microethnographic analysis of a film or videotape the op­ portunity to look and listen more than once relieves the observ­ er's tendency to leap too soon to analytic induction. This independence from the limits of real time in observation pro­ duces a prof ound qualitative difference in the conduct of in­ quiry. That difference is analogous to the contrast between spoken and written discourse, the latter being amenable to revi­ sion and to much more detailed preplanning than the f ormer (on this contrast, see Goody, 1977; Ong, 1977). A third strength, in the analysis of machine recordings, is that it reduces the dependence of the observer on frequently occurring events as the best sources of data. For the analyst of a machine recording, especially an audiovisual one, the rare event can be studied quite thoroughly through repeated reviewing. That op­ portunity is not available to the participant observer in dealing with rare events. There are two main limitations in the use of machine records as a primary data source. The most fundamental limitation is that in replaying a tape the analyst can only interact with it vicariously. Thus the vicarious experience of an event which is microethnography's greatest strength is also its greatest weak­ ness. The researcher has no opportunity to test emerging theor­ ies by trying them out as an active participant in the scenes being observed. That opportunity is one of the hallmarks of participant observation. Another limitation in using machine recordings as a primary data source is that in order to make sense of the recorded ma­ terial the analyst usually needs to have access to contextual in-

145

formation that is not available on the recording itself. The recorded event is embedded in a variety of contexts - in the life histories and social networks of the participants in the events, and in the broader societal circumstances of the events, includ­ ing the relevance of ethnic, social class, and cultural group membership of the participants for the ways in which they or­ ganize their conduct together in the recorded event. Knowing the ways in which the event that occurs face to face fits into the web of influences within the setting and between the setting and its wider environments (including the total society in which the setting resides) can be crucially important in helping the re­ searcher to come to an interpretive understanding of the imme­ diately local organization of interaction within the event, or to an interpretive understanding of the significance of the event's having happened in a particular way rather than some other possible way. Sometimes the broader contexts around an event may not have much influence on the conduct of social relations face to face within the event. In such cases, an analysis is not invalidated by the absence of broader ethnographic framing. But when the influence of the wider world is strong on the im­ mediate scenes of face-to-face interaction in the classroom, and when those influences place systematic constraint on the con­ duct of relations in the classroom, then the absence of broader contextual framing from general ethnographic fieldwork can invalidate the analysis. Both limitations of microethnography - the absence of participation as a means of learning, and the absence of con­ textual information beyond the frame of the recording ­ can be overcome by combining regular ethnography with microethnography.

Data Analysis and Reporting Writing the Report There are nine main elements of a report of fieldwork research : 1. 2. 3. 4. 5. 6. 7. 8. 9.

Empirical assertions Analytic narrative vignettes Quotes from fieldnotes Quotes from interviews Synoptic data reports (maps, frequency tables, figures) Interpretive commentary framing particular description Interpretive commentary framing general description Theoretical discussion Report of the natural history of inquiry in the study

Each of these elements will be discussed in turn. Separately and together they allow a reader to do three things. First, they allow the reader to experience vicariously the setting that is described, and to confront instances of key assertions and analytic con­ structs. Second, these elements allow the reader to survey the full range of evidence on which the author's interpretive analy­ sis is based. Third, they allow the reader to consider the theoret­ ical and personal grounds of the author's perspective as it changed during the course of the study. Access to all these ele­ ments allows the reader to function as a coanalyst of the case reported. The absence of any one of the elements, or inadequacy

146

FREDERICK ERICKSON

in any of them, limits the reader's ability to understand the case and to judge the validity of the author's interpretive analysis. GENERATING AND TESTING ASSERTIONS

A report of fieldwork research contains empirical assertions that vary in scope and in level of inference. One basic task of data analysis is to generate these assertions, largely through in­ duction. This is done by searching the data corpus - reviewing the full set of field notes, interview notes or audiotapes, site documents, and audiovisual recordings. Another basic task is to establish an evidentiary warrant for the assertions one wishes to make. This is done by reviewing the data corpus repeatedly to test the validity of the assertions that were generated, seeking disconfirming evidence as well as confirming evidence. Here are some examples of the kinds of assertions that a field­ work research report might include: 1. There are two major groups of children in the classroom, from the teacher's and the students' points of view: good readers and bad readers. 2. Usually, good readers receive higher-order skills emphasis in reading instruction (i.e., emphasis on comprehension) while bad readers receive lower-order skills emphasis in reading instruction (emphasis on decoding). 3. There are two main subgroups of bad readers; those who try, and those who don't try. 4. Not trying, from this particular teacher's point of view, con­ sists in lack of persistence in small group reading lessons, and in lack of accuracy in doing reading seatwork. Not try­ ing does not consist in not completing seatwork, nor in talking with other students during periods of seatwork. Such behavior is not conceived of by the teacher as not trying. Nor does making an occasional error in seatwork count as not trying. What counts as not trying in seatwork behavior is frequent small errors - frequent in each day's work, and from day to day. 5. The two subgroups of bad readers receive the same kind of reading instruction, but those who try get some of the same privileges that good readers get (e.g., a wider choice of books to read, taking a message to the office, an interesting assignment). 6. Bad readers who don't try are usually treated differently by classmates from bad readers who do try. Bad readers who don't try are often not chosen by classmates to play a game, or to trade food with at lunch time, while bad readers who do try are chosen for these relationships by their good read­ ing classmates as often as the good readers choose fellow good readers. 7. Two exceptions to the pattern described in Assertion 6 are two boys who are physically adept and skilled in sports. These boys, both bad readers who do not try, are chosen often by other boys for sports teams on the playground. 8. With the bad readers who don't try the teacher has a re­ gressive social relationship, that is, in almost every face-to­ face encounter with the teacher, regardless of the subject matter, and in nonacademic activities as well, the teacher reacts to these children negatively in some way, and the children act inappropriately in some way. There is one ex-

ception to this. That child is a girl who comes from what the teacher considers to be a very disorganized family. The teacher suspects child abuse in this family but has not yet reported this to the principal. These assertions about major lines of division in the class in

social identity (status) and role (rights and obligations in rela­

tion to others) vary in scope and in level of inference. The asser­ tion (1) that two major categories of students exist, good readers and bad readers, is broad in scope, but relatively low in level of inference. The assertion (4) about what, from this teacher's point of view, counts as not trying is relatively narrow in scope and low in inferential level. The assertion (8) that the teacher's relationship with the bad readers who don't try is re­ gressive is both broad in scope and high in inferential level, since the assertion covers all sorts of encounters with the teacher that occur across the school day, and involves the inference that both parties are contributing to the trouble they are in. Assertions such as these are generated during the course of fieldwork as noted in the previous section of this chapter. After the researcher has left the field site, such assertions are tested and retested against the data base: the corpus of fieldnotes, interview protocols, site documents (in this case including samples of students' written work), and perhaps audiotapes or videotapes of naturally occurring classroom events. To test the evidentiary warrant for an assertion the re­ searcher conducts a systematic search of the entire data corpus, looking for disconfirming and confirming evidence, keeping in mind the need to reframe the assertions as the analysis pro­ ceeds. For example, in testing Assertion 2 above, concerning the different kinds of reading instruction given to those students the teacher considers to be good readers and bad readers, the re­ searcher would first search the data corpus for all instances of formal reading instruction. If the students were divided into dif­ ferent groups by skill level, all instances of formal reading in­ struction in those groups would be examined to see whether the teacher's emphasis was on higher-order or lower-order skills. Any discrepant cases, that is, higher-order skills instruction given to a low-performance reading group, or lower-order in­ struction given to a high-performance group, would be identi­ fied. If the discrepant cases outnumbered those that fitted the assertion, the assertion would not be warranted by the data. Even if most of the cases fitted the assertion, the discrepant in­ stances would be noted for subsequent analysis. After reviewing the behavioral evidence, the researcher might also review other kinds of evidence. If the teacher had commented on formal reading instruction in interviews, the interview transcripts or audiotapes would be reviewed to see what they might reveal about the teacher's beliefs about the kinds of reading instruc­ tion that were appropriate for children at different skill levels. In Assertion 3 the teacher distinguishes between two types of bad readers, those who try and those who don't try. To test this assertion and to discover the attributes that distinguish in­ dividuals in the two categories, interview data would be re­ viewed first. Did the teacher mention this distinction, in so many words, in formal interviews? What about in informal comments made to the researcher during transitions between classroom activities? Review of the fieldnotes would reveal this.

QUALITATIVE METHODS IN RESEARCH ON TEACHING

The fieldnotes might also show if the teacher invoked the dis­ tinction in addressing students in the classroom (e.g., "John, you're just not trying ") or in writing a comment on � student's paper. The researcher would search the fieldn�tes, videotapes, and samples of student written work for any mstance of such action by the teacher. In addition the researcher would look for more subtle indicators of the teacher's perspective on trying and on who trys: tone of voice and facial expression in addressing those who don't try, amounts of time given to complete the work or to comply with a directive to close the book and line up for recess. Did the teacher, finally, not hold a not-trying student accountable for completing a task, or for not only finishing it, but doing it well? Traces of evidence for these issues and questions would ap­ pear in the field notes. It should be emphasized that these are indeed traces - fragments that must be pieced together into mosaic representations that are inherently incomplete. Conclu­ sive proof is often not possible, especially from data derived from fieldnotes. Yet some lines of interpretation can be shown to be more robust than others. On that admittedly shaky ground must rest the possibility of intellectual integrity and credibility in interpretive research. Assertion 5 claims that all bad readers receive the same kind of formal reading instruction, but that those who try receive a different kind of informal reading instruction (a wider choice of books to read during free time, an interesting writing assign­ ment). Assertion 5 also claims that bad readers who try also receive other privileges, such as taking a note to the office, lead­ ing a song, collecting student papers, while bad readers who do not try do not receive these privileges. To test the warrant for these claims, the researcher would look in the data corpus be­ yond the instances of formal reading instruction. Here the unit of analysis might be mention of individuals in the fieldnotes. The researcher might identify the full set of individuals who are bad readers who don't try, and the full set of those who do try, and search the fieldnotes for instances of description of what those children were doing. Then the researcher would compare what the notes reported for the two sets of bad readers -what were their differences in privileges, in access to activities that were positively valued by children in the room. Notice that the general concept of privileges would have to be specified in terms of local meanings that were distinctive to that classroom. There would need to be descriptive evidence in the fieldnotes, or in interviews with children, that such activities as taking a note to the office or collecting student papers were indeed valued posi­ tively. Descriptive evidence for this might consist of a number of instances reported in the fieldnotes in which many students vol­ unteered for such activities, or expressed disappointment that they were not chosen. The researcher continues to assume that local meanings and values are not self-evident in the data, nor can they be assumed to generalize from one room to the next. It may be true that most grade-school-age children in America like to collect papers. That might not be true in this particular classroom, however, for some locally distinctive reasons. As the data corpus is searched the researcher continually looks for disconfirming evidence. Are there any instances in which a bad reader who doesn't try receives privileges that are similar to those received by bad readers who do try? These dis­ crepant cases are noted. After the general search is completed

147

the researcher returns to the discrepant cases for closer investi­ gation. The results of such an investigation are reported in As­ sertions 6 and 7, concerning the ways that students treat the bad readers who don't try. The general pattern was reported in Assertion 6; bad readers who don't try are usually treated nega­ tively by their peers. Two students are treated positively, however, in special circumstances. These are athletically adept boys, who are chosen for team sports in the playground. Evi­ dence for this assertion would come from the discrepant case analysis. There might have been many instances of these two boys being chosen. The circumstances would be limited to team sports in the playground, however, and the positive reaction of peers is easily explained in terms of the interests of building a winning team. Another example of a finding from discrepant case analysis is seen in Assertion 8, concerning the one bad reader who doesn't try, with whom the teacher does not have a regressive relation­ ship. The explanation for this case involves higher levels of in­ ference than does the explanation for the athletically adept boys being chosen by their peers. Why would this girl be treated dif­ ferently from the other bad readers who don't try? The answer lies not in a causal analysis-in a mechanical sense of causa­ tion - but in an exploration of the teacher's perspective. What does this child mean to the teacher? How does that differ from what other bad readers who don't try mean to the teacher? Var­ ious comparison cases can be identified. The researcher might consider all the other girls in the set bad readers who don't try to see what the child's gender status might mean to the teacher. The researcher might consider all the other children, boys and girls, for whom child abuse is suspected. If there were another bad reader who doesn't try who may be abused at home, that would be an appropriate comparison case. If that child were also a girl, and did not receive special privileges, that would form the ideal contrast set. In that case the researcher might ask, "What else might explain this?" Further investigation of the fieldnotes, teacher interviews, and videotapes might reveal that there are differences in general classroom demeanor be­ tween the two children - the one who does not receive special privileges is less polite than the other, or less funny, or less sad looking. The preceding discussion illustrates the kind of ana­ lytic detective work by which the researcher discovers the subtle shadings of distinctions in social organization and mean­ ing-perspective according to which classroom life is organized as a learning environment for the children and for the teacher. Discrepant cases are especially useful in illuminating these lo­ cally distinctive subtleties. A deliberate search for disconfirming evidence is essential to the process of inquiry, as is the deliberate framing of assertions to be tested against the data corpus. This is classic analysis, termed analytic induction in the literature on fieldwork methods. Much of this induction takes place during fieldwork, but much of it remains to be discovered after leaving the field. A good rule of thumb is to plan to spend at least as much time in analysis and write-up after fieldwork as one spent in collecting evidence during fieldwork. In reviewing the fieldnotes and other data sources to generate and test assertions the researcher is looking for key linkages among various items of data. A key linkage is key in that it is of central significance for the major assertions the researcher wants to make. The key linkage is linking in that it connects up

148

FREDERICK ERICKSON

many items of data as analogous instances of the same phe­ nomenon. To have said, for example, that the teacher dis­ tinguishes between good readers and bad readers is to link all occurrences in the total data set in which the teacher treated bad readers differently from good readers, in reading instruc­ tion and in any other classroom activities. The distinction be­ tween bad and good readers is key because many instances of one kind of treatment by the teacher toward the bad readers and another kind of treatment toward the good readers can be linked together in the review of the fieldnotes. Instances from interviews can also be linked, in relation to this distinction, to­ gether with instances of teacher-student interaction from the fieldnotes. In searching for key linkages the researcher is looking for patterns of generalization within the case at hand, rather than for generalization from one case or setting to another. Generali­ zation within the case occurs at different levels, which differ in the scope of applicability of the generalization. In our example we saw that the distinction between good and bad readers gen­ eralized most broadly for formal reading instruction. This means that upon review ofthe data corpus it was apparent that there were no instances ofhigher-order formal instruction given to bad readers and no instances of lower-order formal instruc­ tion given to good readers. For informal instruction, however, the broadest generalization does not hold. A subsidiary distinc­ tion must be made to account for the patterns in the data; the distinction between bad readers who try and those who don't try. We found upon review of all instances of informal reading instruction (this set included certain kinds of writing assign­ ments and privileges for choosing books for free reading) that the bad readers who tried were treated the same as good read­ ers, while the bad readers who didn't try were treated differently from both the good readers and the bad readers who tried. Moreover that pattern of differential treatment generalized be­ yond informal reading instruction to many other instances of interaction between the teacher and the bad readers who didn't try, with some notable exceptions. But the exceptions were a

minority ofinstances, and they applied to a limited subset ofthe bad readers who didn't try. An appropriate metaphor for this kind of pattern discovery and testing is to think of the entire data set (fieldnotes, inter­ views, site documents, videotapes) as a large cardboard box, filled with pieces of paper on which appear items of data. The key linkage is an analytic construct that ties strings to these various items of data. Up and down a hierarchy of general and subsidiary linkages, some of the strings attach to other strings. The task of pattern analysis is to discover and test those link­ ages that make the largest possible number of connections to items of data in the corpus. When one pulls on the top string, one wants as many subsidiary strings as possible to be attached to data. The strongest assertions are those that have the most strings attached to them, across the widest possible range of sources and kinds ofdata. If an assertion is warranted not only by many instances of items of data from the fieldnotes, but by items from interviews and from site documents, one can be more confident of that assertion than one would be of an asser­ tion that was warranted by only one data source or kind, re­ gardless of how many instances of that kind of data one could link together analytically. The notion of key linkage is illus­ trated in Figure 5.1. It should be noted that this kind of analysis requires a sub­ stantial number of analogous instances for comparison. Rare events are not handled well by the method of analytic induc­ tion. In consequence, there is a bias toward the typical in field­ work research at the stage of data analysis after one has left the field setting - a bias that is analogous to that discussed in the previous section, at the stage of progressive problem solving during data collection. It is important to remember that in this approach to research, frequently occurring events can come to be understood better than can rare events. Audiovisual record­ ing of rare events can reduce this problem somewhat, since the recording permits vicarious "revisiting," but even this does not entirely eliminate the problem. In judging the validity of a re­ searcher's analysis, then, it is important to keep in mind how

FN = Field Note Excerpt; IC = Interview Comment; SD = Site Document (memo, poster) ; MR = Machine Recording (audiotape, videotape)

Fig. $.1 .

Key l i n kages between data and assertions.

QUALITATIVE METHODS IN RESEARCH ON TEACHING

much of the author's argument hangs upon the interpretive an­ alysis of rare events. The best case for validity, it would seem, rests with assertions that account for patterns found across both frequent and rare events. In conducting such analysis and reporting it, the researcher's aim is not proof, in a causal sense, but the demonstration of plausibility, which as Campbell argues ( 1978) is the appropriate aim for most social research. The aim is to persuade the au­ dience that an adequate evidentiary warrant exists for the asser­ tions made, that patterns of generalization within the data set are indeed as the researcher claims they are. Much fieldwork research can be faulted on this point. The analysis may be correct. It may even be believable on intuitive grounds, in the sense that we say " This rings true" after reading the report. If systematic evidence to warrant the assertions is not presented, however, the researcher is justly open to criticism of the analysis as merely anecdotal. Such criticism can be refuted by conducting the kind of systematic data analysis described here, and by reporting the evidence for the assertions in the ways that will be discussed in the sections to come. It should be clear from this discussion that the corpus of materials collected in the field are not data themselves, but re­ sources for data. Fieldnotes, videotapes, and site documents are not data. Even interview transcripts are not data. All these are documentary materials from which data must be constructed through some formal means of analysis. We will conclude the discussion of data analysis by describing the means by which the data resources are converted into items of data. The process of converting documentary resources into data begins with multiple readings of the entire set of fieldnotes. Then the researcher may find it useful to make one or two pho­ tocopied sets of the whole corpus of fieldnotes. These copies can be used in the kinds of data searches described above. Analo­ gous instances of phenomena of interest can be circled in colored ink. These may be instances of a whole event (e.g., a reading lesson), or of a constituent episode or phase within an event (e.g., the start-up phase of the reading lesson), or of a transaction between focal individuals (e.g., reprimand by the teacher to a bad reader who doesn't try). A number of searches through the notes can be made in this fashion, to identify evi­ dence for or against the major assertions the researcher wishes to make, circling instances in different colors of ink, depending on the assertion the instance relates to. At this point some re­ searchers cut up a second copy of the fieldnotes and tape the various instances to large file cards, which then can be sorted further as the analysis proceeds. One can easily envision the use of microcomputers at this stage, or at earlier stages in the re­ view of fieldnotes. As of this writing the use of microcomputers in this kind of data analysis is just beginning and there seem to be no major discussions of this in the literature. It would seem wise, however, to use the computer for retrieval tasks later in the search process rather than at the outset. Reading through the actual notes page by page provides the researcher with a more holistic conception of the content of the fieldnotes than that which would be possible with the more partial view pro­ vided by computerized data retrieval. Reading the notes "by hand" provides more opportunity to encounter unexpected dis­ confirming evidence and to discover unanticipated side issues that can be pursued in subsequent readings.

149

To conclude, the basic units of analysis in the process of ana­ lytic induction are instances of action in events that take place between persons with particular statuses in the scene, and in­ stances of comments on the significance of these commonplace actions, and on broader aspects of meaning and belief, from the perspectives of the various actors involved in the events. The instances of actions are derived from review of the fieldnotes and from review of machine recordings. The instances of com­ ments are derived from analysis of formal and informal inter­ views with informants. The basic units in the process of data analysis are also the basic elements of the written report of the study. Instances of social action are reported as narrative vignettes or as direct quotes from the fieldnotes. Instances of interview comments are quoted in interpretive commentary that accompanies the por­ tions of analytic narrative that are presented. Reporting these details can be called particular description. This is the essential core of a report of fieldwork research. Without particular de­ scription to instance and warrant one's key assertions, the reader must take the author's assertions on faith. Particular de­ scription is supported by more synoptic surveys of patterns in the basic units of analysis. This can be termed general descrip­ tion. A third major type of content in a report of fieldwork research is interpretive commentary. Such commentary is interpolated between particular and general description to help the reader make connections between the details that are being reported and the more abstract argument being made in the set of key assertions that are reported. The researcher has two main aims in writing the report: to make clear to the reader what is meant by the various asser­ tions, and to display the evidentiary warrant for the assertions. The aim of clarification is achieved by instantiation in particu­ lar description. Analytic narrative vignettes and direct quotes from interviews make clear the particulars of the patterns of social organization and meaning-perspective that are contained in the assertions. The aim of providing evidentiary warrant for the assertions is achieved by reporting both particular descrip­ tion and general description. A single narrative vignette or a quote from an interview provides documentary evidence that what the assertion claimed to have happened did occur at least once. General description accompanying the more particular description provides evidence for the relative frequency of oc­ currence of a given phenomenon. General description is also used to display the breadth of evidence that warrants an asser­ tion - the range of different kinds of evidence that speak to that assertion. In the next sections we turn to consider the major types of content in a report on fieldwork research. Particular descrip­ tion is discussed first, then general description, and then the interpretive commentary that accompanies the presentation of descriptive evidence. PARTICULAR DESCRIPTION: ANALYTIC NARRATIVE AND QUOTES

Analytic narrative is the foundation of an effective report of fieldwork research. The narrative vignette is a vivid portrayal of the conduct of an event of everyday life, in which the sights and sounds of what was being said and done are described in the

150

FREDERICK ERICKSON

natural sequence of their occurrence in real time. The moment­ to-moment style of description in a narrative vignette gives the reader a sense of being there in the scene. As a literary form the vignette is very old; it was termed prosographia by Greek rhe­ toricians, who recommended that orators include richly de­ scriptive vignettes in their speeches to persuade the audience that the orator's general assertions were true in particular cases. In the fieldwork research report the narrative vignette has functions that are rhetorical, analytic, and evidentiary. The vig­ nette persuades the reader that things were in the setting as the author claims they were, because the sense of immediate pres­ ence captures the reader's attention, and because the concrete particulars of the events reported in the vignette instantiate the general analytic concepts (patterns of culture and social organi­ zation) the author is using to organize the research report. Such narrative is analytic - certain features of social action and meaning are highlighted, others are presented less prominently or not mentioned at all. This is an extremely important point; the vignette has rhetor­ ical functions in the report. The task of the narrator is twofold. The first task is didactic. The meaning of everyday life is con­ tained in its particulars and to convey this to a reader the narra­ tor must ground the more abstract analytic concepts of the study in concrete particulars - specific actions taken by specific people together. A richly descriptive narrative vignette, pro­ perly constructed, does this. The second task of the narrator is rhetorical, by providing adequate evidence that the author has made a valid analysis of what the happenings meant from the point of view of the actors in the event. The particular descrip­ tion contained in the analytic narrative vignette both explains to the reader the author's analytic constructs by instantiation and convinces the reader that such an event could and did hap­ pen that way. It is the task of more general, synoptic description (charts, tables of frequency of occurrence) to persuade the reader that the event described was typical, that is, that one can generalize from this instance to other analogous instances in the author's data corpus. We will consider synoptic description more fully later in this discussion. It follows that both the author and the critical reader should pay close attention to the details of the narrative and to features of its construction. The narrative vignette is based on fieldnotes taken as the events happened and then written up shortly there­ after. The vignette is a more elaborated, literarily polished ver­ sion of the account found in the fieldnotes. By the time the vig­ nette is written up the author has developed an interpretive perspective, implicitly or explicitly. The way the vignette is writ­ ten up should match the author's interpretive purposes and should communicate that perspective clearly to the reader. The vignette fulfills its purpose in the report to the extent that its construction as a narrative presents to the reader a clear picture of the interpretive point the author intends by telling the vig­ nette. Even the most richly detailed vignette is a reduced ac­ count, clearer than life. Some features are selected in from the tremendous complexity of the original event (which as we have already seen contains more information bits than any observer could attend to and note down in the first place, let alone report later) and other features are selected out of the narrative re­ port. Thus the vignette does not represent the original event itself, for this is impossible. The vignette is an abstraction; an

analytic caricature (of a friendly sort) in which some details are sketched in and others are left out; some features are sharpened and heightened in their portrayal (as in cartoonists' emphasis on Richard Nixon's nose and 5 o'clock shadow) and other fea­ tures are softened, or left to merge with the background. Two potential aspects of contrast in the narrative convey the empathetic caricature that highlights the author's interpretive perspective: (a) variation in the density of the texture of descrip­ tion (across a sequence of events in time, some of which are described in great detail and some of which are glossed over in summary detail), and (b) variation in the alternatively possible terms used for describing the action itself (selecting some parti­ cular nouns, verbs, adverbs, and adjectives rather than others, and by this selection pointing to the locally meaningful roles, statuses, and intentions of the actors in the event). A story can be an accurate report of a series of events, yet not portray the meaning of the actions from the perspectives taken by the actors in the event. Here is a version of a familiar story in which the basic terms of a summary narrative do not convey meaningfulness, from the point of view of the characters in the story. A young man walked along a country road and met an older man. They quarreled and the young man killed the other. The young man went on to a city, where he met an older woman and married her. Then the young man put his eyes out and left the city.

This version does not tell us about roles, statuses, and the ap­ propriateness of actions, given those roles and statuses. The older man was not just any older man, but was Oedipus' father and king of the city. The woman was queen of the city and Oedipus' mother. Thus, actions that generally could be de­ scribed as killing and marrying entailed parricide and incest, at more specific levels of meaning. In sum, richness of detail in and of itself does not make a vignette ethnographically valid. Rather, it is the combination of richness and interpretive perspective that makes the account valid. Such a valid account is not simply a description; it is an analysis. Within the details of the story, selected carefully, is contained a statement of a theory of organization and meaning of the events described. We can see that the analytic narrative vignette, like any other conceptually powerful and rhetorically effective instrument, is a potentially dangerous tool that can be used to mislead as well as to inform. The interpretive validity of the form a narrative vignette takes cannot be demonstrated within the vignette itself. Such demonstration is the work of accompanying interpretive commentary, and of reporting other instances of analogous events. In an effective report of fieldwork, key assertions are not left undocumented by vignettes, and single vignettes are not left to stand by themselves as evidence. Rather, interpretive connec­ tions are made across vignettes, and between the vignettes and other more summary forms of description, such as frequency tables. Direct quotes from those observed are another means of con­ veying to the reader the point of view of those who were stu­ died. These quotes may come from formal interviews, from more informal talks with the fieldworker on the run (as when during the transition between one classroom event and the next the teacher might say, "Did you see what Sam just did?"), or in

QUALITATIVE METHODS IN RESEARCH ON TEACHING

a chat at lunch. Quotes may also come from fieldnotes, from what was recorded about the speech of teacher or students, from audiotapes or videotapes made in fieldnotes, or from tran­ scriptions of audiotapes or videotapes made in the classrooms. Another form for reporting narrative vignettes is the written­ up fieldnotes themselves. These can be quoted directly in the report, with the date they were originally taken included. Often a series of excerpts from the notes, written on different days, can warrant the claim that a particular way an event happened was typical - that the pattern shown in the first excerpt from the notes (or shown in a fully finished vignette) did in fact happen often in the setting. This demonstrates generalizability within the corpus, substantiating such statements as " Usually when Sam and Mary didn't finish their seatwork the teacher over­ looked it, but when Ralph didn't finish he was almost always called to account." Direct quotes from fieldnotes can also be used to show changes in the fieldworker's perspective across time. This use will be disscussed as we consider below the report of the evolu­ tion of inquiry in the study. I have discussed the reporting functions of vignettes, quotes from fieldnotes, and quotes from the speech of participants studied. I have said that to tell a story in a certain way is to present a theory of the organization of the events described, and to portray the significance of the events to those involved in them. There is another purpose in choosing and writing up de­ scriptive narratives and quotes. This is to stimulate analysis early on in the organization of data on the way to writing the report. Much has been written about the process of rereading fieldnotes and reviewing audiovisual records to generate an­ alytic categories and to discover assertions and key linkages between one set of events observed and others observed (see Agar, 1 980, pp. 137- 173; Becker, 1958; Bogdan & Biklen, 1982, pp. 1 55- 162; Dorr-Bremme, 1 984, pp. 1 5 1- 1 66 ; Goetz & Lecompte, 1984, pp. 1 64-207; Levine et al., 1 980; McCall & Simmons, 1 969; Schatzman & Strauss, 1 974, pp. 108- 127: see especially Miles & Huberman, 1984, pp. 79-283.) The Leap to Narration. A way to stimulate the analysis is to force oneself, after an initial reading through of the whole cor­ pus of fieldnotes and other data sources, to make an assertion, choose an excerpt from the fieldnotes that instantiates the as­ sertion, and write up a narrative vignette reporting the key event chosen. In the very process of making the choices (first, of which event to report, second, of the alternative regarding de­ scriptive density and the use of alternative descriptive terms) the author becomes more explicit in understanding the theoreti­ cal "loading"of the key event that was chosen. Later in the analysis the author may conclude that this was not the best instance of the assertion, or that the assertion itself was flawed in some way. Forcing oneself at the outset to make the choices entailed in jumping into storytelling can be a way of bringing to explicit awareness the analytic distinctions and perspectives that were emerging for the author during the course of the time spent in the field. This awareness can be pushed still further by rewriting the vignette in such a way that it makes an interpre­ tive point that is substantially different from the point of the first version, while recounting the same events reported in the first vignette. Choosing a key event early in the analysis after

151

leaving the field, and then writing it up as two different vignettes that make two differing interpretive points, is a standard assign­ ment when I teach data analysis and write-up. It teaches the skills of using alternative density of descriptive detail and delib­ erate choice of descriptive terms to highlight the interpretive point being made. It also forces the analyst to begin to make deliberate analytic decisions. By forcing the analyst to choose a key event, it brings to awareness latent, intuitive judgments the analyst has already made about salient patterns in the data. Once brought to awareness these judgments can be reflected upon critically. GENERAL DESCRIPTION

The main function of reporting general descriptive data is to establish the generalizability of patterns that were illustrated in particular description through analytic narrative vignettes and direct quotes. Having presented a particular instance it is ne­ cessary to show the reader the typicality or atypicality of that instance - where it fits into the overall distribution of occur­ rences within the data corpus. Failing to demonstrate these pat­ terns of distribution - to show generalization within the corpus - is perhaps the most serious flaw in much reporting of fieldwork research. The vignette shows that a certain pattern of social relationships can happen in the setting, but simply to re­ port a richly descriptive, vivid vignette, or to assert in interpre­ tive commentary that accompanies the vignette that this instance was typical or was significant for some other reason (e.g., it was an interesting discrepant case) does not demon­ strate to the reader the validity of such assertions about the significance of the instance. This can only be done by citing analogous instances - linking the key event to others like it or different from it - and (a) by reporting those linked instances in the form of vignettes and (b) by showing in summary fashion the overall distribution of instances in the data corpus. General descriptive data are reported synoptically; that is, they are presented so that they can be seen together at one time. One kind of synoptic reporting medium is the simple frequency table showing raw frequencies, whose patterns of distribution are apparent by inspection. Because the frequencey data that are tabulated are often nominal rather than ordinal, two- and three-way contingency tables are often an appropriate way to show patterns to the reader. Fienberg ( 1977) recommends three-way contingency tables as an especially appropriate way to display these patterns. Occasionally one may wish to apply inferential statistical tests of significance to the data. Usually, given the conditions of data collection, nonparametric tests, such as chi-square and the Mann-Whitney two-tailed test of rank-order correlation are more appropriate than are parametric statistics. The paramet­ ric approaches are usually inappropriate for an additional rea­ son. In standard inferential statistics we assume as analysts that we do not know the pattern of distribution within a sample. One of the tasks of statistical manipulation is to discover what the patterns of distribution are. In the analysis of fieldwork data, pattern discovery is done qualitatively. The frequency tables presented to the reader are usually a tabulation of quali­ tative judgments using nominal scales (or ofjudgments on ordi­ nal scales involving very low levels of inference in assigning

152

FREDERICK ERICKSON

rank or series position to a given instance). In preparing the table as a reporting medium the analyst already knows the pat­ tern offrequency distribution. The analysis is already done; the table merely reports the results ofanalysis in a synoptic form. In consequence, manipulation of the data by elaborate inferential statistical methods such as multivariate analysis, multidimen­ sional scaling, or other forms of factor analysis is usually not necessary or appropriate. One example ofan appropriate use of multidimensional scaling is in a study whose multiple methods of data collection included a questionnaire survey of teenagers, some ofwhom had dropped out ofschools, others ofwhom had not (see Jacob & Sanday, 1976.)

data. It is often necessary in the space of a few adjoining para­ graphs (or even sentences) to be very specific descriptively and quite general interpretively. Some beginning fieldworkers re­ solve this tension by presenting particular description only, with a minimum of interpretation, thus giving any reader but the author a case ofsemantic and conceptual indigestion- too much richness. Other students attempt to resolve the tension by adopting a voice of medium-general description - neither con­ crete enough nor abstract enough. This resembles somewhat the "scientific" discourse style of journals in which positivist research is reported. In this voice the author fails to ground the generalizations in particulars and fails as well to take the gener­ alizations far enough theoretically.

INTERPRETIVE COMMENTARY

Accompanying commentary frames the reporting of particular and general description. This commentary appears as three types : interpretation that precedes and follows an instance of particular description in the text, theoretical discussion that points to the more general significance ofthe patterns identified in the events that were reported, and an account ofthe changes that occurred in the author's point of view during the course of the inquiry. The interpretive commentary that precedes and follows an instance of particular description is necessary to guide the reader to see the analytic type of which the instance is a con­ crete token. An instance of an analytic narrative vignette or an instance of an extended direct quote contains rich descriptive detail that is multivocal in meaning. Especially in the vignette, but also in quotes from interviews, there is much more semantic content in the text than can be seen at first reading by the au­ dience. Interpretive commentary thus points the reader to those details that are salient for the author, and to the meaning-inter­ pretations of the author. Interpretive commentary also fills in the information beyond the story itself that is necessary for the reader to interpret the story in a way similar to that of the author. Foreshadowing commentary, like a set of road signs encountered while driving, enables the reader to anticipate the general patterns that are to be encountered in the particulars of the narrative to come. Commentary that follows the particular vignette or quote stimulates the retrospective interpretation of the reader. Both the anticipatory and the subsequent commen­ tary are necessary if the reader is not to be lost in a thicket of uninterpretable detail. Writing this commentary is necessary for the author as well, since it is precisely the reflective aware­ ness that enables one to write such commentary that enables the writer to be an analyst as well as reporter. A key vignette is relatively easy to select and to write up. What is much more difficult is to probe analytically the significance of the concrete details reported, and the various layers ofmeaning contained in the narrative. Beginning fieldwork researchers find that the most difficult task in reporting is to learn to comment on the details of the narratives that are presented, using elaborated expository prose to frame the narrative contents of the report. In fact, alternation between the extreme particularity of de­ tail found in the vignette (or in an exact citation from fieldnotes, or in a direct quote from an interview) and the more general voice of the accompanying interpretive commentary is a diffi­ cult shift to become accustomed to as a reporter of fieldwork

NATURAL HISTORY OF INQUIRY

Very often, in article-length reports of fieldwork research, and even in monograph-length reports, the author does not include a discussion ofthe ways in which the key concepts in the analy­ sis evolved or unexpected patterns were encountered during the time spent in the field setting and in subsequent reflection. This is an unfortunate omission from a report of fieldwork research, since the plausibility of the author's final interpretation is greatly enhanced by showing the reader that the author's think­ ing did indeed change during the course of study. Fieldwork research presumes that distinctive local meanings will be present in the field setting. These are meanings that could not be fully anticipated by armchair theorizing ("opera­ tionalization ofindicators ofvariables") before entering the set­ ting. Because these unknown local meanings and unrecognized dimensions of the research problem cannot be known at the outset, fieldwork is necessary. But as we pointed out in the pre­ vious section, the fieldwork researcher is always guided by a general set of research interests, and often by a set of quite spe­ cific research questions. Given the complexity of the pheno­ mena to be observed in the setting there is ample opportunity to be selective in collecting evidence - finding only those data that confirm the author's initial hunches and ideological commit­ ments. We have discussed the research process as the search for fal­ sification. It is the author's responsibility to document this pro­ cess for the reader, to show in considerable detail (a) that the author was open to perceiving, recording, and reflecting on evi­ dence that would disconfirm the author's preconceived notions and commitments (as evidenced by the fact that the author's thinking and data collection did change during the course ofthe study); and (b) specific ways in which the changes in interpre­ tive perspective took place. A good way to show this is by writ­ ing a first-person account of the evolution of inquiry before, during, and after fieldwork. Primary evidence for this change in perspective comes from the corpus of fieldnotes and the original research proposed. If the notes contained formal statements ofthe research questions, dated quotes from the notes can reveal changes in the questions across time. The final set ofresearch questions and issues can be contrasted with those found in the initial proposal. Dated ana­ lytic memos written "to the file" during the course of the re­ search can be an additional source of evidence for changes in the researcher's interpretive perspective. A synoptic chart can

QUALITATIVE METHODS IN RESEARCH ON TEACHING

be an effective way to display these changes in thinking. Anoth­ er way in which the fieldnotes can be used to illustrate shifts in interpretive perspective is by analyzing the texture of descrip­ tion found in the notes before and after major intellectual turn­ ing points in the inquiry. AUDIENCES AND THEIR DIVERSE INTERESTS

We have discussed generic issues ofaudience interest in the pre­ vious sections on data collection, analysis, and reporting. For any reader, the report must (a) be intelligible in the relations drawn between concrete detail and the more abstract level of assertions and arguments made by linked assertions; (b) display the range of evidence that warrants the assertions the author makes ; and (c) make explicit the author's own interpretive stance and the grounds of that stance in substantive theory and in personal commitments. Presenting all this enables the reader to act as a coanalyst with the author. In addition to these general concerns of a universal reader, there are more particular concerns of specific audiences that should be kept in mind by the researcher in preparing a report. There are at least four major types of audiences, each with a distinctive set of salient interests : (a) the general scientific com­ munity (fellow researchers), (b) policymakers (central adminis­ trators in the school district, state and federal officials), (c) the general community of practitioners (teachers, principals, and teacher educators), and (d) members of the local community that were studied (teachers, students, building principal, par­ ents). It is often advisable to prepare different research reports to address the specific concerns of the different audiences. Let us review these differing kinds of concerns. The most salient concerns ofthe general scientific community involve the scientific interest and adequacy of the study. If the problem addressed in the study is seen as intellectually signifi­ cant and if the assertions and interpretations that were made seem warranted by the evidence presented, the interests of the scientific community are met. These interests are essentially technical. The most salient concerns of policymakers have to do with generating policy options and making choices among them. Here the central interest is not in the technical adequacy or intrinsic scientific merit of the study, but in how the study can inform the current decision situation of the policymaker. Given the labor-intensive, long-term nature of fieldwork research, it might appear that this approach is usually oflittle use to policy­ makers in current decision making, since by the time the field­ work study was completed, the decision situation would have changed considerably (cf. Mulhauser, 1975). A more appropriate role for fieldwork research in relation to policy audiences is to inform the generation of options by pointing to aspects of the practical work situation that may have been overlooked by policymakers. A fieldwork research report can help the policymaker see what a specific aspect ofthe work of teaching looks like up close; what the practical con­ straints and opportunities for action are in the everyday world of life in a specific classroom, in a specific school building, in a specific community. This is why fieldwork studies ofimplemen­ tation have been useful; they identify unintended consequences

153

of implementation, unanticipated barriers to it, unrecognized reasons why it was successful in a particular setting. Such reporting can help policymakers develop new conceptions of policy and generate a wider range of options than they had previously conceived of. To say to a policymaker caught in trying to decide between option x and option y that options a and b are also available can be of considerable benefit. Lest the picture be painted too rosily, however, it should be noted that fieldwork research, by uncovering practical details of everyday work life that may be inhibiting or fostering the imple­ mentation ofgenerally defined policies, can present information to policymakers that they find frustrating. Indeed, the core per­ spective of fieldwork research can be seen as fundamentally at odds with the core perspective of policymakers. The core per­ spective of fieldwork research is that fine shadings of local meaning and social organization are of primary importance in understanding teaching. The core perspective ofgeneral policy­ makers is that these fine shadings oflocal detail are less impor­ tant than the general similarities across settings ; at best this is insignificant random error, at worst it is troublesome noise in the system that needs to be eliminated ifthe system is to operate more effectively. The conclusions of fieldwork research usually point to the rationality of what is often considered organizationally irra­ tional behavior by administrators and policymakers. The main message of a fieldwork research report to a policy audience, then, is likely to be one of cognitive dissonance. It is important that the researcher realize this in designing a report for a policy audience. The most salient concerns of a general audience of practi­ tioners have to do with deciding whether or not the situation described in the report has any bearing on the situation oftheir own practice. The interests of practitioners can be pejoratively characterized as a desire for tips and cookbook recipes- pres­ criptions about "what works." This notion of what practi­ tioners want is too simplistic . Practitioners may say they want tips, but experienced practitioners understand that the useful­ ness and appropriateness of any prescriptions for practice must be judged in relation to the specific circumstances of practice in their own setting. Thus the interest in learning by positive and negative example from a case study presupposes that the case is in some ways comparable to one's own situation. Another way to state this is to say that a central concern for practitioners is the generalizability of the findings of the study. This is not a matter of statistical generalization so much as it is a matter of logical generalization (the distinction is that of Hamilton, 1980). The responsibility for judgment about logical generalization resides with the reader rather than with the researcher. The reader must examine the circumstances of the case to determine the ways in which the case fits the circum­ stances of the reader's own situation. Practitioners can learn from a case study even if the circum­ stances of the case do not match those of their own situation. This is possible for the practitioner only if the circumstances of the case were described clearly and specifically by the re­ searcher. Thus the problem of inadequate specificity mentioned in the previous section is a technical issue in data collection and reporting that affects the usefulness of the case study to audiences of practitioners.

154

FREDERICK ERICKSON

For the last type of audience to be considered here, members of the local community that was studied, issues of central con­ cern are of a different order from those of the previous types of audiences that have been considered. We use the term commun­ ity loosely here to refer to the network of persons who interact directly, or no more than one or two steps removed from direct interaction, in the delivery ofinstruction to children. That set of persons includes the teachers and students themselves, the building principal and any other building-level staff, staff who visit the school to deliver instruction or to supervise teachers, and the parents ofthe children. In a small school district this set might also include the central office administrators, school board members, and leaders ofkey interest groups ofcitizens in the school district. For these members ofthe local school community there are a variety ofpersonal concerns with the information the fieldwork research report contains. They are not only concerned with the scientific interest and adequacy ofthe study. The issue ofgener­ alization to their own situation does not apply to them, since this is their case. What does apply, powerfully, is that individu­ ally and in various collectivities, their personal and institutional reputations are at stake in their portrayal by the researcher in the report. It is extremely important that the researcher keep this in mind in preparing a report, or a set ofreports, to indivi­ duals and groups within the local community that was studied. If the information presented in the report is to be of use to them -if they are to be able to learn by taking a slightly more distanced view of their own practice - the reports must be sen­ sitive to the variety of personal and institutional interests that are at stake in the kinds ofinformation that are presented about people's actions and thoughts, and in the ways these thoughts and actions are characterized in the reports. We can distinguish four major types ofinformation that have different kinds ofsensitivity in reports to audiences ofmembers of the local community. These four types, or domains, of infor­ mation lie along two major lines of contrast: (a) that between information that is news, or not news, and (b) that which if known would be positively or neutrally regarded, or negatively regarded. The distinction between news and that which is not news in­ volves conscious awareness by the audience of the information contained in the report. Much of what is contained in a report of fieldwork research that is written to a general, nonlocal au­ dience is information that local members already know. Be­ cause ofthe transparency ofeveryday life to those involved in it, however, a good deal of the contents of the fieldwork report may be perceived as news by members oflocal audiences. What is news to one local member may not be news to another; for example, what the teacher knows as a commonplace reality of everyday work in the classroom may not be known at all (or may be understood in a different way) by the principal or by parents. The distinction between information that might be negatively regarded and that which might be positively or neutrally re­ garded points to the differential sensitivity of different informa­ tion to different actors in the local setting. One way to view social organization is in terms of patterns of access to or exclu­ sion from certain kinds of information. In organization theory since Weber we have seen that lines of power and influence are

Fig. 5.2.

Information Known

Information U n known

Positively (or Neutrally) Regarded

1

3

Negatively Regarded

2

4

Types of i nformation in reports to local aud i ences.

drawn along lines of differential access to information. Thus basic political interests are at stake in the revelation or conceal­ ment of certain items of information among local audiences. The researcher needs to keep these interests in mind in prepar­ ing reports for local audiences. Figure 5.2 can be a useful heuristic in making strategic deci­ sions about what to include in a report to a local audience, and how to characterize what is presented in the report. The con­ tingency table displays four major types of information across the two dimensions of contrast just discussed: Type 1 informa­ tion is that which is already known to some or all of the mem­ bers of the local setting, and which is either positively regarded, or is neutrally regarded, that is, not negatively regarded. This type of information is sensitive not because it might jeopardize the reputation of any individual or group in the setting; rather, Type 1 information puts the reputation ofthe researcher at risk, since it can be dismissed as trivial by local members, for whom this information is not news. They may see such information as unimportant, unless it is carefully framed in the report. An example of such information is the assertion that there are approximately 27 children to every adult in early grades classrooms in which the teacher does not have a full-time aide. On the face ofit, this seems an obvious bit ofinformation that is also trivial. One can imagine a local audience reading this in a report and thinking, "This researcher spent 6 months in our building, and that's all she has to say about what early grades classrooms are like? Of course, children vastly outnumber adults in classrooms; everyone knows that." The fact that an early grade classroom is a crowded social space in which children vastly outnumber adults is not at all trivial. It is one of the most important social facts about class­ rooms, considered as an institutionalized set ofrelations among persons. In no other setting in daily life does one encounter such an adult-child ratio, sustained each day for approximately 5 hours. In addition, authority is radically asymmetric in these crowded conditions, as illustrated by an observation of Som­ mer (1969, p. 99) who notes that despite the crowding teachers have 50 times more free space than do students, and teachers also have more freedom to move about in their classroom terri­ tory. Profound realities about the daily work ofteaching-real­ ities involving information overload and what might be called person overload- are entailed in this statement about the adult-child ratio in the early grades classroom. To report such

QUALITATIVE METHODS IN RESEARCH ON TEACHING

an item of information to a local audience without consider­ able interpretive framing to highlight the significance of the social fact, however, is to risk having the credibility of the re­ searcher's work dismissed by the local audience. The researcher needs to attend to this when presenting Type 1 information in a report to the local audiences, and also when writing to general audiences of practitioners and policymakers. Type 2 information is that which is already known to all or to some local members, and which is negatively regarded. This is perhaps the most sensitive type of information to present in a report to a local audience. Often members ofthe local commun­ ity have developed informal or formal social organizations around such inf ormation - ways of avoiding dealing with it, ways ofconcealing it. This type ofinformation is the proverbial skeleton in the closet that everyone knows is there. An example is an item of information regarding one reason teachers' work takes the form it does in a particular building is because the principal is an alcoholic. Everyone in the building knows this, and most of the staff are involved in covering for this principal. Considerable effort goes into this social organizational work, effort which could be made use of in more educationally pro­ ductive ways. For a general audience of practitioners or policymakers, in­ formation about the principal's addiction problem might be a social fact that is crucial to interpretation ofsocial organization in the setting. Moreover, since alcoholism among school staff is not an isolated phenomenon, general audiences may find the opportunity to reflect on this issue in this case study very useful as a stimulus to taking a slightly more distanced view of the phenomenon of addiction and its influence on social organiza­ tion in their own setting. If strict confidentiality can be maintained, the Type 2 information would not be at all inappropriate for reporting to a general audience. To a local audience- at least to some individuals and groups in the local setting - such information could be very provocative. Usually it seems wise to censor entirely such information in reports to local audiences, unless the information is essential to a key as­ sertion or interpretation in the study. In that case, the informa­ tion might be reported to some actors in the setting and not to others. It is impossible to stress too strongly the need for care in presenting Type 2 information in reports in the local setting. Type 3 information is that which is not known by members of the local community and which, if known, would be positively regarded. This type of information presents the least problems in reporting of all the four types discussed here. Still, some stra­ tegic considerations apply in presenting Type 3 information. Members of the local community are pleased by this informa­ tion ; they discover things that work in their setting that they were not aware of, or they discover new aspects of what works, and this helps them think about the organization ofother activ­ ities that don't work the way they want them to. Thus, the pre­ sentation of Type 3 inf ormation has important teaching functions in the report to a local audience. One teaching function is that Type 3 material can be used as positive reinforcement in the report. Type 3 material is a reward to the reader who is a member of the local community - it can influence the reader to continue reading the report. Moreover, Type 3 information is justly rewarding. The researcher is not

155

simply flattering those studied; they have a right to know this type of information about themselves. Because of the function of Type 3 material as positive reinforcement, and because it does not jeopardize the individual and collective self-esteem of members of the local community, Type 3 information can be alternated in the report with Type 4 information, which is nega­ tively regarded, once known. Type 3 information can prepare the way for members of the setting to begin thinking about more unpleasant information, or to think about Type 1 infor­ mation, which if properly framed can stimulate new insights into the taken-for-granted, but which, since it is not news, does not stimulate the pleasurable reaction that follows from learn­ ing Type 3 information. Since Type 3 material is genuinely good news it is an important pedagogical resource for the researcher in preparing a report for an audience of those who were studied. Another more substantive function of presenting Type 3 ma­ terial is that it can stimulate deeper reflection into everyday practice in the setting- new aspects of the organization of social relations and meaning-perspectives of participants that reduce student resistance to learning, or that make for greater clarity and comprehensiveness in the presentation of subject matter, or that foster greater justice in the relations between teacher and student, and among students, or that channel in­ trinsic interest and motivation on the part of students and teachers. These are basic issues in the organization of instruc­ tion in classrooms and in a school building as a whole that warrant reflection by participants in that setting. To stimulate this reflection Type 3 information should be highlighted in the report, possibly by reporting some of it first, possibly also by reporting Type 3 information on a topic that the researcher has discovered is of current interest in the setting. In the report the underlying organizational issues should be stressed, for exam­ ple, the local definitions ofjustice in social relations, the local definitions of intrinsic interest, the features of presentation of subject matter that are locally regarded as clear and under­ standable. Having raised such issues by reporting items of Type 3 material, the researcher can then use those topics in the report as a bridge f or the presentation of Type 4 material. Because of the connection between Type 3 and Type 4 material within a common topic, the sensitivity of Type 4 material can be reduced, by placing it in the context of information that is positively regarded. Type 4 material is that which is not currently known in the setting, and which if known would be negatively regarded. This kind ofinformation needs to be handled with care in the report, but it is by no means impossible to present to local audiences. It is not so potentially explosive as Type 2 material, since partici­ pants in the setting, being unaware of it, have not organized ways to keep it away from the light of scrutiny. Still, Type 4 information can at the least threaten the self-esteem of individ­ uals and ofnetworks of individuals in the setting. In some cases this information can be extremely sensitive. It may be wise to exclude some of this information from reports to some of the audiences in the setting. What one might say in a confidential report to the teacher that was studied, for example, one might want to leave out of the report that was to be read by the princi­ pal, or by parents. To make these kinds of strategic decisions the researcher needs to apply the same standards for assessing

156

FREDERICK ERICKSON

risk to those studied that were discussed in the previous section under the topic of negotiating entry. Those with the least power in a setting are usually those least at risk for embarrassment or administrative sanction by the revelation of Type 4 informa­ tion - but not always. The researcher's basic ethnographic analysis of the setting can shed light on these strategic issues in reporting. The researcher's role in the setting is also an important con­ sideration in dealing with Type 4 information. If the researcher were doing advocacy research on behalf of a particular interest group in the setting (e.g., teachers, parents) one might present more Type 4 information, or one might handle it less tenderly than if one had negotiated access to the setting by agreeing to address the interests of a broader array of interest groups. Even if Type 4 information is not especially sensitive for key individuals in the setting it still should be handled judiciously in a report to a local audience. The reasons for this are pedagogi­ cal as well as ethical. Some of these reasons were introduced in the discussion of Type 3 material. Generic issues that can be illustrated by negative example with Type 4 material can be presented first as positive examples by reporting Type 3 mater­ ial. This makes the relatively bitter pill of Type 4 information easier to swallow, and can reduce the defensiveness of members of the local community to the more unpleasant aspects of the report. It would be pedagogically ineffective to introduce an important topic in the report by presenting a whole series of instances of Type 4 information, in vivid narrative vignettes and striking quotes from interviews. This would be hard for people to take, and when without dissimulation a researcher can pre­ sent bad news in the context of good news, this seems the much wiser course. To conclude, reporting to local audiences can be thought of as a process of teaching the findings. In doing so, the researcher takes an active stance toward the audience, considering the au­ dience as client. As in any teaching, it is crucial to assess the current state of the learner's knowledge and to assess the sensit­ ivity of the learning task-its potential for embarrassment or difficulty. All learning involves risk, and in designing instruc­ tion any teacher needs to consider the risks from the student's point of view. In this sense, teaching requires an ethnographic perspective toward the learners and toward the learning envi­ ronment. Because of this the fieldwork researcher is in an un­ usually good position to be a teacher of the findings of the study, if he or she chooses to take on that role with audiences of those who were studied. The very process of data collection and analysis that makes it possible to report valid information at the end of a fieldwork study provides the data relevant to deci­ sions about how to teach the findings. In applied research, and especially in reporting to local au­ diences of those studied, the single report in the form of a mon­ ograph-length ethnography is probably obsolete. Multiple re­ ports of varying length, each designed to address the specific interests of a specific audience, are usually more appropriate. Nor is writing the only medium of reporting. In the local setting oral reports at meetings, and mixing oral and written reporting in workshops, can be effective ways to teach the findings. For an individual teacher that was studied, reviewing a set of field notes or a videotape with the researcher can be a valuable experience in continuing education.

Whether the report to a local audience takes oral or written form the researcher should invite critical reactions from the au­ dience. Dialogue between the researcher and those studied pro­ vides the researcher with an opportunity to learn as well as providing those studied an opportunity to learn. The validity check that comes from this dialogue can be of great value to the researcher. It can inform revisions in reports. It can stimulate the researcher to rethink the basic analysis itself. One of the problems in reporting to general audiences is that the writer does not usually get this kind of feedback. In teaching the findings to local audiences, such dialogue is intrinsic to the reporting process.

Conclusion: Toward Teachers as Researchers Interpretive research is concerned with the specifics of meaning and action in social life that takes place in concrete scenes of face-to-face interaction, and that takes place in the wider so­ ciety surrounding the scene of action. The conduct of interpre­ tive research on teaching involves intense and ideally long-term participant observation in an educational setting, followed by deliberate and long-term reflection on what was seen there. That reflection entails the observer's deliberate scrutiny of his or her own interpretive point of view, and of its sources in for­ mal theory, culturally learned ways of seeing, and personal value commitments. As the participant observer learns more about the world out there he or she learns more about himself or herself. The results of interpretive research are of special interest to teachers, who share similar concerns with the interpretive re­ searcher. Teachers too are concerned with specifics of local meaning and local action; that is the stuff of life in daily class­ room practice. In a recent review article titled "Toward a More Effective Model of Research on Teaching," Bolster (1983) ar­ gues on the same grounds presented in this chapter. He uses the terms sociolinguistic and symbolic interactionist as labels for in­ terpretive approaches to research on teaching, and argues that these approaches have special relevance for classroom teachers. Bolster's argument is especially telling because of his situa­ tion as an experienced public school teacher who for 20 years was also simultaneously a university-based teacher educator. Bolster's career as a teacher has been a prototype for new roles for experienced teachers; he has shown one way of being a mas­ ter teacher. He spent mornings as a junior high school social studies teacher in Newton, Massachusetts, and spent after­ noons as a professor of education at Harvard University. For him the radically local character of classroom teaching was a compelling reality. He found that interpretive research took ac­ count of that dimension of teaching practice (Bolster, 1983): The more I became aware of and experienced with this methodol­ ogy, the more I became convinced that of all the models of research I knew, this model has the greatest potential for generating know­ ledge that is both useful and interesting to teachers . . . this approach focuses on situated meanings which incorporate the various reac­ tions and perspectives of students. In common with the teachers' perspective, it assumes the multiple causation of events: the class-

QUALITATIVE METHODS IN RESEARCH ON TEACHING

room is viewed as a complex social system in which both direct and indirect influences operate. Unanticipated contingencies potentially illuminate rather than confound understanding since reaction to the unexpected often highlights the salient meanings assigned to what is normal. Most important of all, symbolic interactionist research in class­ rooms necessarily relies heavily on the teacher's interpretation of events. The relationship between teacher and researcher as col­ leagues, therefore, is more perceptive than political, and each has individual and professional reasons for nourishing and extending it. (pp. 305-306)

The inherent logic of the interpretive perspective in research on teaching leads to collaboration between the teacher and the researcher. The research subject joins in the enterprise of study, potentially as a full partner. In some of the most recent work (e.g. Florio & Walsh, 1980) the classroom teacher's own re­ search questions - about particular children, about the organi­ zation of particular activities- become the focus of the study. It is but a few steps beyond this f or the classroom teacher to become the researcher in his or her own right. As Hymes notes (1982), interpretive research methods are intrinsically democratic; one does not need special training to be able to understand the results of such research, nor does one need arcane skills in order to conduct it. Fieldwork research requires skills of observation, comparison, contrast, and reflection that all humans possess. In order to get through life we must all do interpretive fieldwork. What professional interpretive re­ searchers do is to make use of the ordinary skills of observation and reflection in especially systematic and deliberate ways. Classroom teachers can do this as well, by reflecting on their own practice. Their role is not that of the participant observer who comes from the outside world to visit, but that of an un­ usually observant participant who deliberates inside the scene of action. A future for interpretive research on teaching could be that the university-based researcher gradually works him or herself out of a job. That is a slight rhetorical exaggeration ; there is still a need for outsiders' views of classrooms and of teaching prac­ tice. In some ways the teacher's very closeness to practice, and the complexity of the classroom as a stimulus-rich environment, are liabilities f or reflection. Kluckhohn's aphorism, mentioned at the outset of this chapter, can be recalled here at its close : The fish might indeed be the last creature to discover water. The university-based researcher can provide valuable dis­ tance- assistance to the classroom teacher in making the fam­ iliar strange, and interesting. This can be done by the univer­ sity-based researcher as a consultant, as a continuing educator of experienced teachers. The teacher, as classroom-based re­ searcher, can learn to ask his or her own questions, to look at everyday experience as data in answering those questions, to seek disconfirming evidence, to consider discrepant cases, to entertain alternative interpretations. That, one could argue, is what the truly effective teacher might do anyway. The capacity to reflect critically on one's own practice, and to articulate that reflection to oneself and to others, can be thought of as an essential mastery that should be possessed by a master teacher. Teachers in public schools have not been asked, as part of their job description, to reflect on their own practice, to deepen

157

their conceptions of it, and to communicate their insights to others. As the teaching role is currently defined in schools there are external limits on the capacity of a teacher to reflect criti­ cally on his or her own practice. There is neither time available, nor an institutionalized audience for such reflection. The lack of these opportunities is indicative of the relative powerlessness of the profession outside the walls of the classroom. This is not the case, to the same degree, in other professions. For physicians and lawyers, and even for some nonelite profes­ sions more similar to teaching such as social work, it is routine for practitioners to characterize their own practice, both for purposes of basic clinical research and for the evaluation of their services. For example, surgeons often dictate a narrative description of the procedures used during an operation. After this narrative is transcribed and filed it can be reviewed by col­ leagues for purposes of evaluation, and it is available as docu­ mentation in the event of a malpractice suit. The social worker writes process notes on interviews with clients; these notes are then available for evaluation and consultation by supervisors. By contrast, the teacher's own account of his or her practice has no official place in the discourse of schooling, particularly in teacher evaluation, staff development, and/or debates about the master teacher role and merit pay that have been stimulated by recent reports and proposals for educational reform (e.g., National Commission on Excellence in Education, 1983 ; Goodlad, 1984). If classroom teaching in elementary and secondary schools is to come of age as a profession- if the role of teacher is not to continue to be institutionally infantilized - then teachers need to take the adult responsibility of investigating their own prac­ tice systematically and critically, by methods that are appropri­ ate to their practice. Teachers currently are being held increasingly accountable by others f or their actions in the class­ room. They need as well to hold themselves accountable for what they do, and to hold themselves accountable for the depth of their insight into their actions as teachers. Time needs to be made available in the school day f or teachers to do this. Any­ thing less than that basic kind of institutional change is to per­ petuate the passivity that has characterized the teaching profession in its relations with administrative supervisors and the public at large. Interpretive research on teaching, conducted by teachers with outside-classroom colleagues to provide both support and challenge, could contribute in no small way to the American schoolteacher's transition to adulthood as a profes­ sional. It is appropriate to conclude with a narrative vignette and some interpretive commentary on it. During 1981 and 1982 a set of jokes became popular in the United States concerning activities that "Real Men" did or did not engage in, because the activities were considered effete and unmasculine. The first of the jokes to have currency was "Real Men don't eat quiche." In the winter of 1982 a well-known practitioner of process­ product research on teaching sent brief notes to colleagues around the country containing the f ollowing one-liner, which the researcher claimed to have found very amusing : "Real Men don't do ethnography." On a winter morning as I arrived at the mailboxes of the staff of the Institute f or Research on Teaching where I work, two of my IRT colleagues gleefully showed me the note, which had just

158

FREDERICK ERICKSON

arrived in the mail. I too found the joke amusing, but not for the same reasons as those that may have been held by the author of the note. One reason the joke seemed funny to me was because of its apparent presuppositions about power in social research-pre­ suppositions of which the author of the note seems to have been unaware. One presupposition is that of prediction and control as the aim of nomothetic science. Another presupposition in­ volves the power relations that are presumed to obtain between the social scientist, as the producer of assertions whose authori­ tativeness rests on their predictive power, and the various au­ diences to which those assertions are addressed. In the case of research on teaching, these audiences include government offi­ cials, curriculum developers, school administrators, teachers, parents, and the general citizenry. The primary role of researchers on teaching who are Real Men, as defined by the standard approach to educational re­ search and dissemination, seems to be to make statements about the general effectiveness of various teaching practices. The primary audience for these statements is those placed in relatively high positions in the hierarchy of educational policy formation and implementation: federal and state officials, cur­ riculum developers and publishers, local school administrators (especially central office staff who relate directly to the school board), and teacher educators. It is the role of these audiences to communicate the prescriptions of research on teaching re­ garding effective practice to the primary service deliverers in the school system-teachers and building principals. These service deliverers, in turn, are to communicate the prescriptions to indi­ vidual parents as justifications for the classroom practices of the teacher who teaches their child. Perhaps Real Men don't do interpretive research on teaching because they are unwittingly, or wittingly, committed to exist­ ing power relations between technical experts and managers, and the front-line service providers and receivers of services in the institution of American schooling. In those existing ar­ rangements, statements about "what works" in general, derived from positivist research conducted across many classrooms, can carry considerable weight with audiences of higher-level de­ cision makers. Even if such statements turn out in the long run to have been wrong, in the short run they serve to support belief in the fundamental uniformity of practice in teaching. Such be­ lief is functional for decision makers, since it justifies uniform treatment by general policy mandates that are created by centralized decision making and implemented in "top-down" fashion within the existing social order. More decentralized, "bottom-up" decision making that granted more autonomy to front-line service providers in the system would change the cur­ rent distribution of power in educational institutions. The ap­ propriateness of such "bottom-up" change strategies is suggested in the theoretical orientations and in the growing body of empirical findings of interpretive research on teaching. ·Such research, while it does not claim to speak in a voice of univocal, positive truth, can make useful suggestions about the practice of teaching, since although no interpretive assertions can be conclusively proven, some lines of interpretation can be shown systematically to be false. Paradoxically, the chief useful­ ness of interpretive research for the improvement of teaching practice may be its challenge to the notion that certain truths

can be found, and in its call to reconstrue fundamentally our notions of the nature of the practical in teaching. Interpretive research on teaching, then, is not only an alter­ native method, but an alternative view of how society works, and of how schools, classrooms, teachers, and students work in society. Real Men may be justified in not wanting to do ethno­ graphy, for in the absence of general belief in social science as a means of determining positive truth it is difficult for the social scientist to operate in society as a mandarin, or philosopher­ king. The key issue for that kind of Real Man in educational research may not be teacher effectiveness, but researcher effect­ iveness, narrowly construed. Another view of the future is that it could be a place in which interpretive work is a creatively subversive activity in the field of education. Real Women and Men who were schoolteachers, principals, parents, and students, as well as those who were uni­ versity-based scholars, might find themselves doing ethno­ graphy, or whatever one might want to call it, as a form of continuing education and institutional transformation in research on teaching.

REFERENCES Agar, M. (1980). 1he professional stranger: An informal introduction to ethnography. New York: Academic Press. Au, K. H., & Jordan, C. ( 1980). Teaching reading to Hawaiian children: Finding a culturally appropriate solution. In H. Trueba, G. P. Guthrie, & D. H. Au (Eds.), Culture in the bilingual classroom. Rowley, MA: Newbury House. Au, K. H., & Mason, J. (198 1). Social organizational factors in learning to read : The balance of rights hypothesis. Reading Research Quar­ terly, 1 7(1), 1 1 5- 1 52. Barnhardt, C. ( 1982). "Tuning-in": Athabaskan teachers and Athabas­ kan students. In R. Barnhardt (Ed.), Cross-cultural issues in Alaskan education (Vol. 2). Fairbanks: University of Alaska, Center for Cross-Cultural Studies. Barth, F. (1969). Ethnic groups and boundaries: 1he social organization of culture difference. Boston: Little, Brown. Bateson, G., Jackson, D., Haley, J., & Weakland, J. (1956). Toward a theory of schizophrenia. Behavioral Science, l. (Reprinted in G. Bateson, Ed., Steps toward an ecology of mind. New York: Ballantine Books, 1972). Becker, H. S. (1958). Problems of inference and proof in participant observation. American Sociological Review, 23, 652-660. (Reprinted in G. J. McCall & J. L. Simmons, Eds., Issues in participant observa­ tion: A text and reader, pp. 245-254. Reading, MA: Addison­ Wesley, 1969). Becker, H. S., Geer, B., Hug:1,-,_ •c. C .. � Strauss, A. (1961). The boys in white: Student culture in medicai school. Chicago: University of Chicago Press. Berger, P., & Luckmann, P. ( 1967). 1he social construction of reality. New York: Anchor Books. Bogdan, R. D., & Biklen, S. K. (1982). Qualitative researchfor education: An introduction to theory and methods. Boston : Allyn & Bacon. Bohannon, P. (1 963). Social anthropology. New York: Holt, Rinehart & Winston. Bolster, A. S. (1983). Toward a more effective model of research on teaching. Harvard Educational Review, 53(3), 294-308. Bourdieu, P., & Passerson, J. C. (1977). Reproduction: In education, so­ ciety and culture. Beverly Hills: Sage Publications. Bowles, S., & Gintis, H. (1976). Schooling in capitalistic America: Educa­ tional reform and the contradictions of economic life. New York : Basic Books. Bronfenbrenner, U. (1977). Reality and research in ecology of human development. .Proceedings of the American Philosophical Society, 1 19(6), 439-469.

QUALITATIVE METHODS IN RESEARCH ON TEACHING

Brophy, J. E. (1979). Teacher behavior and its effects. Journal of Educa­ tional Psychology, 7(6), 733- 750. Burrell, G., & Morgan, G. (1979). Sociological paradigms and organiza­ tional analysis. London: Heinemann. Campbell, D. T. ( 1978). Qualitative knowing in action research. In M. Brenner, P. Marsh, & M. Brenner (Eds.), The social context of method. New York: St. Martin's. Campbell, D. T., & Stanley, J. C. ( 1966). Experimental and quasi-experi­ mental designs for research. Chicago : Rand McNally. Cazden, C., Carrasco, R., Maldonado-Guzman, A., & Erickson, F. (1980). The contribution of ethnographic research to bicultural bilingual education. In J. Alatis (Ed.). Current issues in bilingual education. Washington, DC: Georgetown University Press. Clignet, R., & Foster, P. ( 1966). 1hefortunatefew: A study of secondary schools and students in the Ivory Coast. Evanston, IL: Northwestern University Press. Cole, M., & Scribner, S. (1974). Culture and thought: A psychological introduction. New York: John Wiley. Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1966). Equality of education opportunity. Washington, DC: U.S. Government Printing Office. Collier, J. (1973). Alaskan Eskimo education. New York: Holt, Rinehart & Winston. Comber, L. C., & Keeves, J. P. (1973). Science education in nineteen countries. New York: John Wiley. Comte, A. (1968). System of positive polity. New York: B. Franklin. (Original work published 1875) Cook, T. D. & Campbell, D. T. (1 979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychol­ ogy. American Psychologist, 30(2), 1 16-127. Cusich, P. A. (1980). A study ofnetworks among the professional staffs of two secondary schools (Final Tech. Rep. No. 400-79-0004). Wash­ ington, DC : National Institute of Education. Dilthey, W. (1976a). The construction of the historical world in the human studies. In. H. P. Richman (Ed.), W Dilthey: Selected writ­ ings (pp. I 70-206). Cambridge: Cambridge University Press. (Originally published in W. Dilthey, Gesammelte Schriften, Vol. 7, pp. 79-88, 1 30- 1 66. Leipzig, 1914). Dilthey, W. ( 1976b). The development of hermeneutics. In H. P. Rich­ man (Ed.), W Dilthey: Selected »titings (pp. 247-260). Cambridge: Cambridge University Press. (Originally published in W. Dilthey, Gesammelte Schriften, Vol. 5, pp. 317-337. Leipzig, 1914). Dilthey, W. (1976c). Introduction to the human studies: The relation­ ship of the human studies to the sciences. In H. P. Richman (Ed.), W Dilthey: Selected writings (pp. 1 63-167). Cambridge: Cambridge University Press. (Originally published in W. Dilthey, Gesammelte Schriften, Vol. 1, pp. 14-21. Leipzig, 1 883). Dobbert, M. L. (1982). Ethnographic research : Theory and application for modern schools and societies. New York: Praeger. Dorr-Bremme, D. ( 1984). An introduction to practical.fieldwork from an ethnographic perspective (Grant No. NIE-G-83-0001). Los Angeles; University of California, Center for the Study of Evaluation. Doyle, W. (1979). Classroom tasks and student abilities. In P. L. Peter­ son & H. J. Walberg (Eds.), Research on teaching: Concepts, .find­ ings, and implications. Berkeley, CA: McCutchan. Dunkin, M. J., & Biddle, B. (1974). The study of teaching. New York: Holt, Rinehart & Winston. Durkheim, E. (1958). The rules ofsociological method. Glencoe, IL: Free Press. (Original work published 1 895). Elliott, J., & Adelman, C. (1976). Classroom action research (Ford Teaching Project, Unit 2 Research Methods). Norwich, England: University of East Anglia, Centre for Applied Research in Education. Erickson, F. (1973). What makes school ethnography 'ethnographic'? Council of Anthropology and Education Quarterly, 4(2), 10-19. Erickson, F. (1975). Gatekeeping and the melting pot: Interaction in counseling encounters. Harvard Educational Review, 45(1), 44-70. Erickson, F. (1976). Gatekeeping encounters: A social selection process. In P. R. Sanday (Ed.), Anthropology and the public interest. New York: Academic Press.

159

Erickson, F. ( 1977). Some approaches to inquiry in school/community ethnography. Anthropology and Education Quarterly, 8(3), 58-69. Erickson, F. ( 1979). Talking down: Some cultural sources of miscom­ munication in inter-racial interviews. In A. Wolfgang (Ed.), Re­ search in nonverbal communication. New York: Academic Press. Erickson, F. (1982a). The analysis of audiovisual records as a primary data source. In A. Grimshaw (Ed.), Sound-image records in social interaction research. Special issue of the Journal of Sociological Methods and Research, 1 1(2), 21 3-232. Erickson, F. (1982b). Classroom discourse as improvisation : Relation­ ships between academic task structure and social participation structure in lessons. In L. C. Wilkinson (Ed.), Communicating in the classroom. New York: Academic Press. Erickson, F. (1982c). Taught cognitive learning in its immediate environments: A neglected topic in the anthropology of education. Anthropology and Education Quarterly, 13(2), 149- 1 80. Erickson, F. ( 1984). Rhetoric, anecdote, and rhapsody : Coherence stra­ tegies in a conversation among Black American adolescents. In D. Tannen (Ed.), Coherence in spoken and written discourse (pp. 8 1- 1 54). Norwood, NJ: Ablex. Erickson, F., Florio, S., & Buschman, J. ( 1980). Fieldwork in educational research (Occasional Paper No. 36). East Lansing: Michigan State University, Institute for Research on Teaching. Erickson, F., & Mohatt, G. (1982). The cultural organization of partici­ pation structures in two classrooms of Indian students. In G. Spindler (Ed.), Doing the ethnography ofschooling. New York: Holt, Rinehart & Winston. Erickson, F., & Schultz, J. (1981). When is a context? Some issues and methods in the analysis of social competence. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings. Norwood, NJ: Ablex. (Originally published in The Quarterly News­ letter of the Institute for Comparative Human Development, 1977, 1(2), 5-10.) Erickson, F., & Schultz, J. ( 1982). The counselor as gatekeeper: Social interaction in interviews. New York: Academic Press. Erickson, F., & Wilson, J. (I 982). Sights and sounds of life in schools: A resource guide to film and videotape for research and education. (Re­ search Series No. 125). East Lansing: Michigan State University, Institute for Research on Teaching. Fienberg, S. E. ( 1977). The collection and analysis of ethnographic data in educational research. Anthropology and Education Quarterly, 8(2), 50-57. Florio, S., & Walsh, M. (1980). The teacher as colleague in classroom research. In H. Trueba, G. Guthrie, & K. Au (Eds.), Culture in the bilingual classroom: Studies in classroom ethnography. Rowley, MA: Newbury House. Gage, N. L. (Ed.). ( 1963). Handbook of research on teaching. Chicago: Rand McNally. Garfinkel, H. (1967). Studies of the routine grounds of everyday activities. Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall. Gearing, F., & Sangree, L. (1 979). Toward a cultural theory of education and schooling. The Hague: Moulton. Giddens, A. (1976). New rules of sociological method: A positive critique of interpretive sociologies. London : Hutchinson. Giddens, A. ( 1982). Pro.files and critiques in social theory. Berkeley: University of California Press. Giroux, H. (1983). Theories of reproduction and resistance in the new sociology of education : A critical analysis. Harvard Educational Review, 53(3), 257-293. Glaser, B., & Strauss, A. ( 1979). The discovery ofgrounded theory: Stra­ tegies for qualitative research. Hawthorne, NY: Aldine. Goetz, J. P., & Lecompte, M. D. (1984). Ethnography and qualitative design in educational research. New York : Academic Press. Goodenough, W. ( 1981). Culture, language, and society. Menlo Park, CA : Benjamin-Cummings. Goodlad, J. l. (1984). A place called school: Prospectsfor thefuture. New York: McGraw-Hill. Goody, J. (1977). The domestication of the savage mind. Cambridge, England: Cambridge University Press. Gumperz, J. J. (1 982). Discourse strategies. Cambridge, England : Cam­ bridge University Press.

160

FREDERICK ERICKSON

Hamilton, D. ( 1980). Generalization in the education sciences : Prob­ lems and purposes. In T. Popkewitz & R. Tabachnick (Eds.), The study of schooling: Field-based methodologies in education research. New York: Praeger. Heath, S. B. ( 1982). Questioning at home and at school : A comparative study. In G. Spindler (Ed.), Doing the ethnography ofschooling. New York: Holt, Rinehart & Winston. Heath, S. B. (1 983). Ways with words: language, life, and work in communities and classrooms. Cambridge: Cambridge University Press. Henry, J. (1963). Culture against man. New York: Random House. Hess, R., & Shipman, V. (1965). Early experience and the socialization of cognitive modes in children. Child Development, 34, 869-886. Husserl, E. ( 1970). The crisis of European sciences and transcendental phenomenology (D. Carr, Trans.). Evanston, IL: Northwestern University Press. (Original published 1 936). Hymes, D. ( 1982). Ethnographic monitoring. In H. T. Treuba, G. P. Guthrie, & K. H. Au (Eds.), Culture in the bilingual classroom. Rowley, MA: Newbury House. Jacob, E., & Sanday, P. R. ( 1976). Dropping out: A strategy for coping with cultural pluralism. In P. R. Sanday (Ed.), Anthropology and the public interest: Fieldwork and the theory. New York: Academic Press. Jencks, C., Bartlett, S., Corcoran, M., Crouse, J., Eaglesfield, D., Jack­ son, G., MCielland, K., Mueser, P., Olneck, M., Schwartz, J., Ward, S., & Williams, J. ( 1979). l¼o gets ahead? 1he determinants of eco­ nomic success in America. New York: Basic Books. Jensen, A. R. ( 1969). How much can we boost IQ and scholastic achievement? Harvard Educational Review, 39(1), 1 -123. Keddie, N. (1973). The myth of cultural deprivation. Harmondsworth, Middlesex: Penguin. Kimball, S. T. ( 1974). Culture and the educative process. New York: Columbia University, Teachers College Press. Kleinfeld, J. S. ( 1979). Eskimo school on the Andreafsky. New York: Praeger. Kuhn, T. ( 1962). The structure of scientific revolutions. Princeton, NJ: Princeton University Press. Labov, W. ( 1972). language in the inner city. Philadelphia: University of Pennsylvania Press. Laing, R. D. ( 1970). Knots. New York: Pantheon. Lakatos, I. ( 1978). The methodology of scientific research programmes. Cambridge, England: Cambridge University Press. Lein, L. (1975). You were talkin' though, oh yes, you was: Black mi­ grant children: Their speech at home and school. Anthropology and Education Quarterly, 6(4), 1 - 1 1. Levine, H. G., Gallimore, R., Weisner, T., & Turner, J. L. (1980). Teach­ ing participant-observational methods: A skills-building approach. Anthropology and Education Quarterly, 11, 38-54. Lindesmith, A. R. ( 1947). Addiction and opiates. Chicago : Aldine. Lofland, J. ( 1976). Doing social life: 1he qualitative study ofhuman inter­ action in natural settings. New York: ; ohn Wiley. MacDonald, B., & Walker, R. (Eds )_ ( I 974). SAFARI: Innovation, evaluation, research, and the proble1;1 of control. Norwich, England: University of East Anglia, Centre for Applied Research in Educa­ tion. Malinowski, B. ( 1935). Coral gardens and their magic. London: Allen & Unwin. Malinowski, B. (1961). Argonauts of the western Pacific. New York: Dutton. (Original work published 1922). Marx, K. ( 1959). Theses on Feuerbach. In S. Feuer (Ed.), Marx and Engels: Basic writings on politics and philosophy. Garden City, NY: Doubleday Anchor. McCall, G. L., & Simmons, J. L. ( 1969). Issues in participant observa­ tion: A text and reader. Reading, MA: Addison-Wesley. McDermott, R. P. (1974). Achieving school failure: An anthropological approach to illiteracy and social stratification. In G. D. Spindler (Ed.), Education and culture process. New York: Holt, Rinehart & Winston. McDermott, R. P. ( 1977). School relations as contexts for learning in school. Harvard Educational Review, 47, 298-313. McDermott, R. P., & Gospodinoff, K. (198 1 ). Social contexts for ethnic borders and school failure. In H. T. Trueba, G. P. Guthrie, & K. H.

Au (Eds.), Culture and the bilingual classroom. Rowley, MA: Newbury House. Mead, M. ( 1928). Coming of age in Samoa. New York: Morrow. Mehan, H. ( 1979). l£arning lessons: Social organization in the classroom. Cambridge, MA: Harvard University Press. Miles, M. B., & Huberman, A. B. ( 1984). Qualitative data analysis. Be­ verly Hills: Sage Publications. Mosteller, F., & Moynihan, D. P. (Eds.). ( 1972). On equality of educa­ tional opportunity. New York: Random House. Mulhauser, F. ( 1 975). Ethnography and policy making: The case of education. Human Organization, 34(3), 3 1 1-315. National Commission on Excellence in Education. ( 1983). A nation at risk: The imperative for educational reform. Washington, DC: U.S. Department of Education. Ogbu, J. ( 1978). Minority education and caste: The American system in cross-cultural perspective. New York: Academic Press. Ong, W. ( 1977). Interfaces of the word. Ithaca, NY: Cornell University Press. Parsons, T. ( 1959). The school class as a social system. Harvard Educa­ tional Review, 29(4), 297-3 1 8. Pelto, P., & Pelto, G. ( 1977). Anthropological research : The structure of inquiry. New York: Harcourt Brace. Philips, S. U. ( 1982). The invisible culture: Communication in classroom and community on the Warm Springs Indian Reservation. New York: Longman. Piestrup, A. ( 1973). Black dialect interference and accommodation of reading instruction in.first grade (Monograph No. 4). Berkeley, CA: Language Behavior Research Laboratory. Riessman, F. ( 1962). The culturally deprived child. New York: Harper & Row. Riis, J. ( 1890). How the other half lives. New York: Scribner's. Rosenshine, B. (1983). Teaching functions in instructional programs. Elementary School Journal, 83(4), 335-351 . Rowe, M. B. ( 1974). Wait-time and rewards a s instructional variables. Journal of Research in Science Teaching, 11, 8 1 -94. Schatzman, L., & Strauss, A. ( 1974). Field research : Strategies for a natural sociology. Englewood Cliffs, NJ : Prentice-Hall. Scheflen, A. E. ( 1960). Regressive one-to-one relationships. Psychiatric Quarterly, 23, 692-709. Schutz, A. (1971). Collected papers I: The problem ofsocial reality. The Hague : Martinus Nijhoff. Scribner, S., & Cole, M. (1981). The psychology of literacy. Cambridge, MA: Harvard University Press. Shulman, L. S. (198 1 ). Recent developments in the study of teaching. In B. Tabachnik, T. S. Popkewitz, & P. B. Szekely (Eds.), Studying teaching and learning. New York: Praeger. Shultz, J. J., Florio, S., & Erickson, F. ( 1982). Where's the floor? Aspects of the cultural organization of social relationships in communica­ tion at home and at school. In P. Gilmore & A. A. Glatthorn (Eds.), Children in and out of school: Ethnography and education. Wash­ ington, DC: Center for Applied Linguistics. Simon, H. A. (1957). Models of man. New York: John Wiley. Sinclair, U. ( 1906). Thejungle. New York : Doubleday, Page. Smith, L. M. ( 1979). An evolving logic of participant observation, edu­ cational ethnography, and other case studies. In L. Shulman (Ed.), Review of research in education. Itasca, IL: F. E. Peacock. Sommer, R. ( 1969). Personal space. Englewood Cliffs, NJ: Prentice­ Hall. Spindler, G. D. (Ed.). ( 1955). Education and anthropology. Stanford, CA : Stanford University Press. Stallings, J. A., & Kaskowitz, D. ( 1974). Fol/ow-through classroom ob­ servation evaluation 1972- 1973. Menlo Park, CA: Stanford Re­ search Institute. Steffens, L. ( 1904). The shame of the cities. New York: McClure, Phillips. Stenhouse, L. ( 1978). Case study and case records: Towards a contem­ porary history of education. British Educational Research Journal, 4(2), 2 1 -39. Vogel, E. (1965). Japan's new middle class. Berkeley : University of Cali­ fornia Press. Walker, R., & Adelman, C. ( 1 975). A guide to classroom observation. London: Methuen.

QUALITATIVE METHODS IN RESEARCH ON TEACHING

Waller, W. (1932). The sociology of teaching. New York: John Wiley. Warner, W. L., & Lunt, P. S. (1941). The social life of a modern com­ munity. New Haven, CT: Yale University Press. Watson-Gegeo, K. A., & Boggs, S. T. (1977). From verbal play to talk story: The role of routine in speech events among Hawaiian children. In S. Ervin-Tripp & C. Mitchell-Kernan (Eds.), Child discourse. New York: Academic Press. Wax, R. H. (1971). Doing fieldwork: Warnings and advice. Chicago: University of Chicago Press. Webb, B., (1926). My apprenziceship. New York: Longmans, Green. Weber, M. (1978). The nature of social activity. In W. C. Runciman (Ed.). Weber: Selection in translation (pp. 7-32). Cambridge:

\

161

Cambridge University Press. (Originally published in Wirtschaft und gesellschaft, Vol. l, pp. 1-14. Tiibingen, 1922). Whyte, W. F. (1955). Street corner society (2nd ed.). Chicago: Univer­ sity of Chicago Press. Willis, P. E. (1977). Learning to labour: How working class kids get working class jobs. Aldershot, Hampshire: Saxon House. Winch, P. (1958). The idea of a social science and its relation to philo­ sophy. London: Routledge and Kegan Paul. Wirth, L. (1928). The ghetto. Chicago: University of Chicago Press. Wolcott, H. (1982). The anthropology of learning. Anthropology and Education Quarterly, 13(2), 83-108. Zorbaugh, H. (1929). The gold coast and the slum. Chicago: University of Chicago Press.

6. Observation as Inquiry and Method Carolyn M. Evertsen Peabody College, Vanderbilt University

Judith L. Green The Ohio State University

Introduction

behaviors related to student outcomes using a series of related descriptive, correlational, and experimental studies. This ap­ proach is often referred to as a process-product approach (Doyle, 1977; Dunkin & Biddle, 1974; Koehler, 1978). Another source of influence in this phase was the availability of funds for evaluation studies of Follow-Through projects from the U.S. Office of Education and for large- and small-scale studies of teaching from the newly formed National Institute of Education. Phase Four parallels Phase Three in terms of time and can be defined as a period of expansion, alternative approaches, theor­ etical and methodological advances, and convergence across re­ search directions in the use of observational techniques to study teaching (ca. 1972 to present). This phase has two beginning points: The publication of Functions of Language in the Class­ room (Cazden, John, & Hymes, 1972) and the USOE/NIE pan­ els on the study of teaching. The former resulted from a series of interdisciplinary conferences sponsored by the U.S. Office of Education (1965 & 1966) held for the purpose of suggesting priorities for research on children's language and its relation­ ship to school success (Cazden, this volume; Cazden, John, & Hymes, 1972). Therefore, the publication of Functions of Lan­ guage in the Classroom marks the beginnings of the linguistic approach to the study of teaching-learning processes. This ap­ proach was solidified in the panel reports with the publication of Report 5, T eaching as a Linguistic Process in a Cultural Set­ ting (National Institute of Education, 1974b). The USOE/NIE panels on the study of teaching were convened to help the National Institute of Education set a research agenda for the

Observation, as an approach to study educational processes and issues, has a rich and varied history. One way to view this history is to consider the directions taken in past research. Using this approach, four overlapping phases can be identified. Phase One can be viewed as an exploratory phase in which the question of interest was whether teacher-student interactions and other related classroom and instructional behaviors could be reliably and validly identified (ca. 1939-1963). This phase begins with some of the earliest systematic studies of instruc­ tional processes (see Amidon & Hough, 1967) and ends with the publication of the chapter by Medley & Mitzel in the first Handbook of Research on Teaching (Gage, 1963a). Phase Two can be characterized as a period of instrument development, and of descriptive, experimental, and training stu­ dies (ca. 1958-1973). This phase begins with the emergence of studies using category systems (Amidon & Hough, 1967; Simon & Boyer, 1970a; 1970b) and of issues about paradigms for the study of teaching (Gage, 1963b). It ends with publication of chapters by Rosenshine and Furst and by Gordon and Jester in the Second Handbook of Research on Teaching (Travers, 1973). Phase Three is described as a period in which studies of teacher effects explored teacher behaviors that relate to student outcome performance generally on standardized tests (ca. 1973 to present). This phase was influenced by several events. The beginning of this phase is chronicled by Rosenshine and Furst in the Second Handbook of Research on Teaching (Travers, 1973), in which they called for systematic exploration of teacher

The authors thank reviewers Hermine H. Marshall (University of California- Berkeley) and Celia Genishi (The University of Texas-Austin). Preparation of this chapter was supported in part by Peabody College, Vanderbilt, The Ohio State University, and the University of Delaware. The authors also wish to thank Rita Fillos, James Hiebert, Gladys Knott, Virginia Koehler, Daniel Neale, Don Sanders, and Cynthia Wallat for their comments on earlier drafts; however, the opinions expressed and any responsibility for errors or omissions belong solely to the authors.

162

OBSERVATION AS INQUIRY AND METHOD next decade. These reports laid the groundwork for other perspectives on the study of teaching (e.g., human interaction, clinical information processing, theory development; see Berliner & Koehler, 1983). While Phase Four is continuing, its history has been chronicled recently in special issues of the Elementary School Journal and the Educational Psychologist ("Research on Teaching," 1983a, 1983b), and in chapters throughout this volume. Across all four phases numerous methods have been de­ scribed for use in observing teaching-learning processes in edu­ cational settings (e.g. Amidon & Hough, 1967; Barnes, Britton, & Rosen, 1971; Dunkin & Biddle, 1974; Erickson & Wilson, 1982; Evertson & Holley, 1981; Flanders, 1970; Genishi, 1983a; Gilmore & Glatthorn, 1982; Good, 1983; Gordon & Jester, 1973; Green, 1983b; Green & Wallat, 1981a; Hamilton, 1983; Heap, 1980b; 1983; Lundgren, 1977; Medley & Mitzel 1963; Mehan, 1979a; Rosenshine & Furst, 1973; Shavelson, 1983; Simon & Boyer, 1970a, 1970b; Sinclair & Coulthard, 1975; Spindler, 1982; Stallings, 1977; Stubbs & Delamont, 1976; Trueba, Guthrie, & Au, 1981; Weinstein, 1983; Wilkinson, 1982; Woods and Hammersley, 1977). However, little syste­ matic discussion exists on the nature of observation both as method and as an inquiry process (Fassnacht, 1982). Given the varied nature of observational research and the expansion of methodological directions identified in Phase Four (1972-pre­ sent), such a discussion is needed in order to guide the decision making of those interested in systematic observation in educa­ tional settings. The purpose of this chapter is to explore the nature of obser­ vation as a research approach and to provide a framework for making informed decisions about design and implementation of observational research. That is, the intent of this chapter is to explore the nature of observational inquiry and methods re­ lated to this inquiry process. The framework was derived from a review of work on the nature of observation; consideration of observational tools used in past research; discussion of ap­ proaches to observation that have emerged in the last decade for use in the study of teaching-learning processes;_and discus­ sions between the authors of this chapter and groups of junior researchers -doctoral students at the University of Delaware and at The Ohio State University-interested in designing and implementing systematic observational research projects. This chapter is complemented by discussions of research methods and design processes presented in other chapters in this volume. The information in the chapter is presented in two parts. Part One focuses on a discussion of generic and specific issues re­ lated to the nature of observation and observational tools. Part Two discusses observation as an inquiry process and de­ scribes the framework for selecting, designing, and implement­ ing observation of educational practices and issues.

The Nature of Observation and Observational Tools The answer to the question "What is observation?" depends on the purpose for which the person is asking the question. Is the person a classroom teacher interested in observing student per­ formance during ongoing lessons? Is the person a teacher or counselor interested in observing student behavior in order to

163

supplement test information so that a comprehensive profile of ability and performance can be developed before placing a stu­ dent in a special program? Is the person a researcher interested in using observation to study intellectual development, effective instruction, classroom climate, and so forth? Is the person a developmental psychologist interested in observing student ability to conserve? Each of these people will engage in some form of systematic, deliberate observation; however, the specific observational pro­ cess will vary. Different foci will be selected; observations will be conducted in different environments; different events will be sampled; the duration of the observation will vary; different means of recording data will be used; different rules for evi­ dence will exist. In other words, the program of research (a series of conceptually and theoretically linked studies) will differ (see Shulman, this volume, for a discussion of "program of research"). The differing purposes lead to differences in strategies for ob­ servation, levels of systematization, and levels of formality. These factors lead to differences in design and implementation. In other words, the purpose of the observation influences what is observed, how it is observed, who gets observed, when obser­ vation takes place, where it takes place, how observations are recorded, what observations are recorded, how data are ana­ lyzed, and how data are used. In addition, the purpose of an observation is related to the theory, beliefs, assumptions, and/ or past experiences of the person who is doing the observation. These factors form the frame of reference of the observer and influence the decision-making as well as the observational pro­ cess (e.g. Dunkin & Biddle, 1974; Fassnacht, 1982; Power, 1977; Shulman, 1981). While differences exist, generic issues and principles can be identified when the question of what is meant by observation and observational methods is explored on a general level. In the remainder of this section, the generic issues involved in doing observation in educational settings will be discussed. The issues involved in such observation will be considered separately from the theoretical or conceptual frame guiding specific studies or approaches. In the section on observation as inquiry, the rela­ tionship of the theoretical or conceptual frame and related methods will be considered. The separation of the discussion of method from theory guiding individual studies (a) permits the development of concepts about the nature of observation as a strategy for representing aspects of reality and (b) avoids issues related to the use of observation as a method of research in a given discipline or field of study. This approach was selected for heuristic purposes to avoid contrasts of a limited set of ap­ proaches (e.g. qualitative vs. quantitative; experimental vs. de­ scriptive; normative vs. interpretive). The purpose of this approach is to highlight issues involved in doing adequate ob­ servation, to describe the complexity of those issues, to consider tools available, and to provide guidelines for what must be con­ sidered regardless of frame of reference.

Observation: A Multifaceted Phenomenon Observation is an everyday event. It is part of the psychology of perception, and as such, it is a tacit part of the everyday func­ tioning of individuals as they negotiate the events of daily life.

CAROLYN M. EVERTSON AND JUDITH L. GREEN

164

Not all observation that occurs in daily life is tacit. Observation also occurs more deliberately and systematically when the sit­ uation demands such behavior. For example, more deliberate and systematic observations are required of children playing baseball and other games. The player must observe the patterns of play, the direction of the ball, the actions of others, and the objects in the path of play in order to participate appropriately or navigate the field of play. These observations, while more systematic than ordinary observations in everyday life, are fre­ quently tacit. This type of more systematic observation is also required in more formal situations such as classrooms. Students must observe the social and academic expectations of who can talk- when, where, how, and for what purpose -in order to participate appropriately in learning activities. Teachers must observe more systematically to maintain the flow of a lesson, the management of instruction, and the involvement of stu­ dents, as well as to obtain informal and formative assessment of student performance, lesson development, and program direction. In daily life, systematic observations are undertaken, not to answer specific questions, but to establish, maintain, check, suspend, and participate in everyday events (Blumer, 1969). Such observations are related to the demands of the situation. However, when observation is used to answer a stated question, it must be deliberate and systematic. In addition, it must be a conscious process that can be explicated so that others may assess its adequacy and understand the process. Observations used for research and overt decision-making processes in edu­ cational settings are more formal and externalized than obser­ vation in daily life. One way to think of these different aspects of observation is as a continuum in which everyday, tacit observa­ tions are the least formal; systematic situation-specific, every­ day tacit observations are more formal, but are still not under overt, conscious control; and deliberate and systematic ques­ tion-specific observations are the most formal of the three and under the most direct control. This way of conceptualizing observation, illustrated in Figure 6.1, suggests that observation for research and deci­ sion making in educational settings (deliberate, systematic, question-specific observation) is both similar to everyday ob­ servational processes and different from such processes. One major difference is the reason for observing. Other differences will become evident in the sections that follow.

I l I ----i---r-----rEveryday Tacit

Less

Fo,m al

Observations

Everyday Deliberate Systematic

Situation­ Specific Observations

Fig. 6.1 Continuum of observation types.

Deliberate Systematic

H;ghly

Fo,m al

Question­ Specific Observations

Observation as a Research and Decision-Making Process As suggested previously, observation for research and decision­ making purposes is closely tied to the question of why one ob­ serves. Observations for observation's sake (i.e., random obser­ vations) cannot be added together in the way children put beads on a string. Such observations are not cumulative and do not constitute evidence. The question would have to be raised: Why was the observation done? What purpose does it serve? Since observations are phenomena that occur in a specific �on­ text or setting, there would be no systematic means of aggrega­ ting information from such observations. The purpose of the observation guides what will be done, how it will be used, and what can be obtained. Karl Popper (1963), as cited in Stubbs, Robinson, and Twite (1979), makes the point: Twenty-five years ago I tried to bring home the same point to a group of physics students in Vienna by beginning a lecture with the following instructions: "Take pencil and paper; carefully observe, and write down what you have observed!" They asked, of course, what I wanted them to observe. Clearly the instruction "Observe!" is absurd. (It is not even idiomatic, unless the object of the transitive verb can be taken as understood.) Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language, with property words; it presupposes similarity and classification, which in its turn presupposes interest, points of view, and problems. (p. 21)

Implicit in Popper's definition is the assumption that the ob­ server is the first instrument of observation. In other words, the task or object selected, the observer's frame of reference, and the purpose of the observation, among other factors, will influence what will be perceived, recorded, analyzed, and ultimately de­ scribed by the observer. While this is true, in observational re­ search in the behavioral and social sciences, the observer often augments the observation process by using a tool or instrument to focus or guide observation (e.g., a sign system, a specimen record, category system, field notes). Another way to think about this is that the perceptual system of the observer is the first tool used by the observer and that this tool is influenced by the observer's own goals, biases, frame of reference, and abili­ ties. An observation tool or lens (e.g., an observation instru­ ment or system) further influences and constrains what will be observed, recorded, analyzed, and described. Therefore, obser­ vation is a mediated process on several levels - the level of the observer as a person with biases, beliefs, training, and ability, and the level of the instrument or tool used to make and record an observation. This tool also has a point of view, bias, struc­ ture, and so forth. The issues related to the mediation and representation processes must be considered since as Fassnacht (1982) suggests: A statement about real events is always a statement mediated by a particular representational mechanism in a particular context. A representation can have many different qualities, depending on the mechanism or the organism involved in the representational process. (p. 7)

He goes on to conclude that because of these issues "absolute independence of a mechanism is a contradiction in terms." The

OBSERVATION AS INQUIRY AND METHOD

issues raised above deserve further exploration since one pur­ pose of observation in educational settings is to obtain a de­ scription or representation of events, processes, and phenomena as well as factors that influence these phenomena. Such information is necessary for understanding and improving schooling, instruction, and learning. The issues raised above re­ quire consideration of (a) observation as a means of represent­ ing reality in educational settings ; (b) observation as a contextualized process; (c) mechanisms or tools for recording and storing observations; and (d) factors involved in observa­ tion : units of observation/aggregation of data; sampling; and sources of error. Observation as a Means of Representing Reality The remainder of this chapter will focus on observations that are systematic, deliberate, and question-specific. Such observa­ tions are formal in structure and situation-specific. As will be shown, such observations are undertaken in a variety of ways using a variety of representational systems. While others have viewed this diversity as chaos (Rosenshine & Furst, 1973), we argue that the variety of options is a potential strength for re­ searchers interested in the study of teaching. Weinstein (1983) points to the value of such diversity: It is difficult to maintain that one view of classroom events is more accurate than another. Rather, we must learn from each perspective, identify matches and mismatches among perspectives, and examine relationships between perceptions and behavior. . . . By investigating several perspectives in each study, we will improve our understand­ ing of the social reality of classrooms. (p. 306)

Mead (1975) made a similar point when she suggested that one reason an anthropologist enters the field is to obtain "a" grammar, not "the" grammar of a people or an event. Fassnacht (1982) also made a similar point. He suggests: Even the representational mechanism with the finest analysis now known does not guarantee that with a different mechanism a finer analysis of reality would not be possible. (p. S)

What these scholars are suggesting is that any tool (mechan­ ism) used or approach to observation of an event (process, set of behaviors, group context) provides only one representation or view of the phenomena under study. Weinstein ( 1983) sug­ gests that this characteristic of observations and approaches be used as a strength and be incorporated into studies. Recently, this approach to integrating multiple perspective analysis in a single project has been undertaken by researchers from a wide variety of disciplinary groundings and perspectives interested in the study of teaching-learning processes (e.g. Bloome, 1983; Cole, Griffin, & Newman, 1979; Cook-Gumperz, in press; Evertson, Anderson, & Clements, 1980; Frederiksen, 198 1 ; Green & Harker, 1982; Green, Harker, & Wallat, in press; Marston & Zimmerer, 1979; Morine-Dershimer & Tenenberg, 198 1 ; Weinstien, 1983; among others). These studies demon­ strate the richness of information and the complementarity of views that can be obtained by systematically building on differ-

165

ent approaches. They suggest that the selectivity issue need not be a detriment; they demonstrate what can be learned by acknowledging this issue and developing a program of obser­ vational research that incorporates a variety of perspectives. The selectivity issue operates beyond the level of perspective or approach. Selectivity is an inevitable characteristic of any tool, representational system, or program of research. It is not possible to record all aspects of reality with any given system or tool or in any single research project. Selectivity is also reflected in the decisions a researcher or decision maker must make. For example, researchers and decision makers must select (a) the question for study; (b) a setting in which to observe (e.g. play­ ground, classroom); (c) a slice of reality to observe within the setting (e.g. games played by boys vs. girls; reading groups; group discussions; question asking by teachers during language arts lessons ; the first hour of school; the beginning of a lesson); (d) a tool or combination of tools to record and store the segment of reality for study (e.g., category system and narrative record; rating scale, category system, and checklist; field notes and vid­ eotape record; audiotape record, still camera pictures, and field notes); (e) procedures for observing (e.g., where to stand, when to observe, how many observers to use, in what order to use specified tools); (f) the subjects to be observed or the locus of observation (e.g., individual, group, event, behavior type, stra­ tegy); (g) the analysis procedures appropriate for the question and the record obtained; and (h) the method of reporting the data and information extracted from the observation record (e.g., written report, graphic representation, protocol tape). In other words, selectivity is a part of the entire decision-making, design, and implementation process involved in observational research. Rather than control for selectivity that appears to be impos­ sible, one strategy could be to report in more detail than exists presently in most studies the parameters of the study and the nature of the decisions made. Another way to think about these issues is to consider the reporting of decisions that contribute to selectivity as demographic information. This would provide a clearer picture of what was observed, how, where, when, and for what purposes (Herbert & Attridge, 1975; Shaver, 1983; see also Corsaro, 1985, for an illustrative example). This strategy may avoid some of the pitfalls that led to the lack of replication in studies of teaching reported in Dunkin & Biddle (1974). The information would provide the basis for determining whether two observational studies were indeed equivalent, and whether the variation observed across studies was a result of the proce­ dures used in the study or the portion of reality selected. Closely related to the selectivity issue is the fact that reality cannot be directly apprehended (Fassnacht, 1982). Since obser­ vations require a representational mechanism and the mechan­ ism contains selective elements, the representation is mediated by the tool used as well as by the representational process. That is, the representation or description obtained is dependent on the instrument used to record the observation and the way in which the data were collected (e.g., length of observation ; events sampled). In addition, the process surrounding the ob­ servation also provides a degree of mediation of reality (Stubbs et al., 1979). In other words, "Truth" can never be known. What the researcher and decision maker attempt to do is to collect sufficient and appropriate evidence to ensure that the description is as accurate as possible given the representational

166

CAROLYN M. EVERTSON AND JUDITH L. GREEN

process used. (Additional information related to these issues will be considered in the section on sources of error.) In this section, several factors that influence the nature of the description or representation of reality were discussed briefly. On a general level, description was shown to be related to both the system for recording the observations and the process of observational research itself. In the discussion below, a closer look at various aspects of the observational process will be con­ sidered. The purpose of this discussion is to describe options available and to clarify issues that influence the nature and quality of observational research.

VALUE CLUSTER Set of community values characteristically enacted in a corresponding set of culturally defined, behavioral domains

I

NETWORK TYPE (Open and Closed)

DOMAIN Cluster of social situations typically constrained by a common set of behavioral rules

I

Cluster of role relationships defined by extent to which they are governed by a single (or multiple) set of community values

SOCIAL SITUATION

Observation as a Contextualized Process One factor that influences what is observed is the way in which a "context" is defined. A review of research on teaching sug­ gests that a variety of definitions of context exist. That is, the issue of context has been discussed from a variety of perspec­ tives (e.g., Barr & Dreeban, 1983; Bloome & Green, 1982; Brophy & Evertson, 1978; Cazden, 1983; Cook-Gumperz & Gumperz, 1976; Erickson & Shultz, 198 1 ; Evertson & Veld­ man, 198 1 ; Fishman, 1972; Good, Cooper, & Blakey, 1980; Green, 1 983b ; Hamilton, 1983; McDermott, 1976; Ogbu, 198 1 ; Philips, 1972, 1982; Popkewitz, Tabachick, & Zeichner, 1979; Scribner & Cole, 1 98 1 ; Taylor, 1 983). From this work a series of issues related to ways of considering context can be extracted: local context as embedded in larger levels of context; historical context of the setting; historical context of the specific event under study; and context of the research approach. Consideration of these issues related to context can (a) help researchers understand how individuals and groups acquire knowledge from everyday activities and events in both formal and informal educational settings, (b) help researchers deter­ mine factors constraining and supporting performance in a given context, and (c) lead to understandings of what makes two contexts functionally equivalent (Erickson & Shultz, 198 1 ; Florio & Shultz, 1979). As will be shown below, the task is to match the question with the context and the instrument with the phenomenon, context, and question. Local Context as Embedded in Broader Levels of Context. The discussion to this point has considered only the immediate event in the immediate setting. However, the context in the im­ mediate or local setting (Erickson, 1979) is only one context that surrounds the observed event. If a systems approach is taken, the local context can be seen as embedded in and in­ fluenced by larger contexts (Barr & Dreeban, 1983; Bloome & Green, 1982; Carey, Harste, & Smith, 198 1 ; Erickson & Shultz, 1981). The reading group is embedded in the classroom; the classroom is embedded in the school ; the school in the district; the district in the community; and so forth. These other con­ texts occur simultaneously and impact on the immediate con­ text. For example, they frame how people will participate (e.g., Au, 1980; Fishman, 1976; Michaels, 198 1 ; Scollon & Scollon, 1984; Tannen, 1979), what resources people use and what re­ sources are available (Barr & Dreeban, 1983; Heap, 1980b), and often how people will use or perceive what occurs (Ander­ son, 198 1 ; Morine-Dershimer & Tenenberg, 198 1 ; Morine-Der­ shimer, 1982;). In other words, links exist between micro and

Encounter defined by intersection of setting, time, and role relationship

�

I

I

SETTING

I

ROLE RELATIONSHIP Set of culturally defined mutual rights and obligations

I.-,

I

INTERACTION TYPE (Personal and Transactional) Function of interaction defined by degree to which participants in social situation stress the mutual rights and obligations of their role relationship

SPEECH (Events and Acts)

>----'

j

Fig. 6.2. Relations h i p among some constructs employed in socioli ngu istic analysis. From " How Can We Measure the Roles Which a Bili ngual's Languages Play i n His Everyday Behavior ? " by R. Cooper. Copyright 1 976 by La Linguistique, P. U . F. Paris. Repri nted by perm ission .

macro levels of context and are signaled through performance, materials available, practices, and so forth. In Figure 6.2, Cooper (1968 [as cited by Fishman, 1976]) demonstrates the links between micro- and macro-constructs for a sociolinguistic analysis. Cooper's example also relates to a previous point. Weinstein (1983) suggested the need to consider multiple perspectives to provide alternative descriptions or representations so that the matches and mismatches in perspective could be explored and more holistic representations of events obtained. Cooper sug­ gests that the micro and macro factors and the links between these factors should be considered and that such considera­ tions may require a full range of methods and analytic proce­ dures. (For other discussions of micro-macro issues in educa­ tion see Bloome & Green, 1982; Hamilton, 1983; Ogbu, 198 1.) This brief discussion highlights the need to consider multiple levels of context as well as the level of context used in a specific observational study. In addition, it suggests the need to account for relevant aspects of other contexts that impact on the specific context under study. The consideration of context issues is an important part of making inferences about processes and how those generalize.

OBSERVATION AS INQUIRY AND METHOD

Historical Context of the Setting. Another type of context that frames local ones is the historical context of the setting. For example, schools have histories and one way to view this history is as a set of expectations, traditions, networks, and lines of communication (Hymes, 198 1 ; Little, 198 1, 1982). The historical context of the setting is not directly observable. How­ ever, to answer the question of why an event or structure oc­ curred where it did or when it did, the researcher may have to explore the historical context. For example, past practice or policy may determine why reading rather than mathematics instruction occurs from 9:00 to 10:00 in every classroom in a school. The decision may be based on past practice, because it has always been that way. In order to generalize about the time segmentation of the day in a particular school or about the meaning of an event, the history behind the observed phenom­ enon needs to be considered. The Historical Context of the Event. Recent work on classroom processes has shown that events in classrooms have histories (e.g., Cazden, 1976; Cochran-Smith, 1984; Emmer, Evertson, & Anderson, 1980; Erickson & Shultz, 198 1 ; Evertson & Emmer, 1982; Mehan, 1979a; Smith & Geoffrey, 1968; Wallat & Green, 1979, 1982). This work focuses on how processes (e.g., social norms, management of activities) unfold over time. It shows that the first few weeks of school are critical periods in the establishment of routines, expectations, classroom structures, and so forth. If the question is, What is reading? How does mathematics occur? What are the group practices? What is the nature of curriculum delivery, organization, and development? or similar questions about teaching-learning processes and re­ lationships, then the development of these processes over time must be considered and the beginnings of the processes (events) must be captured. The historical context of local events is important to consider given the findings that some processes (events, practices) re­ main stable over time while others vary (Eder, 1982; Evertson & Veldman, 198 1 ; Good et al., 1980; Wallat & Green, 1982). That is, the students may be adhering to a set of rules, expecta­ tions, or procedures established prior to the day of observation. Some of these may be stable over time and, therefore, the ob­ server recording these behaviors would get a representative set of behaviors. Some of the behaviors might be situation specific, content specific, or time specific. Without knowing which be­ haviors were stable and representative of the event under study, and which were idiosyncratic to the specific day and time, the researcher will be constrained in what can be generalized about this teacher, pupil, or class. This work suggests that a match between sampling and the question must be carefully considered and that the behavior in any one context or event within a context may be related to prior experiences with similar events. The questions that need to be considered include, What are the ties between contexts? What behaviors are stable over time and therefore representa­ tive of the expectations (procedures, practices), and which vary? When should sampling be done and at what points, in order to know that observations are representative? In other words, the issue of context is related to the issue of sampling and the repre­ sentativeness of the data and findings.

167

Exclusive Approaches ------- Inclusive Approaches

Fig. 6.3. Incl usive and exclusive approaches to context.

Context as Determined by the Research Approach. One last as­ pect of context will be reviewed. This aspect is related to the research approach, that is, different approaches may sample dif­ ferent amounts and levels of context, and the decisions about context sampling may be related to the theory used, the beliefs of the researcher, and the tools used, as well as the questions asked. One way to view this issue is as a continuum. On one end of the continuum, context factors are controlled or minimized. At the other end of the continuum, contexts are not controlled and as many aspects of the context as can be obtained are in­ cluded in the data collection. Figure 6.3 provides a graphic rep­ resentation of this continuum. Approaches toward the exclusive end seek to simplify what is collected, to reduce the " noise" in the system so that specific types of phenomena or behaviors are considered (e.g., question asking; teacher-student interactions). Researchers using approaches at this end of the continuum (e.g. category systems, checklists, rating scales) tend to be concerned with discerning behavioral laws or normative information. They seek to study a large number of classrooms with an effi­ cient, reliable approach. Generally, what is observed is prede­ termined; rules for observation are routinized to permit interobserver agreement. At this end of the continuum, many of the cues that offer redundancy to interpretations (Snow, 1974) are reduced or eliminated in the measuring instrument. Se­ lectivity at this end of the continuum is deliberate. That is, the researcher has made deliberate decisions about the unit used in the observation system (e.g., checklists or category systems have a limited number of items to observe and record). Only the items on the instrument are observed and recorded. Although instruments may be modified later as a result of what the inves­ tigator learns about the setting, no new units are added during the period of observation. As the researcher moves toward the other end of the contin­ uum, larger and larger segments of context and greater numbers of context variables are considered. At the farthest end, the inclusive end, no attempt is made to exclude any aspect of context on a deliberate basis. Rather, researchers attempt to sample a wide slice of everyday life or reality. They often at­ tempt to sample multiple levels of context. Decisions about what to observe and record, when examining less than the total setting (e.g. group, culture, event over the year, etc.), are princi­ pled decisions (Heath, 1982). The decisions frame not only what slice of context will be sampled, but also what will be sampled within the context (e.g., Erickson & Wilson, 1982; Green, 1983b; Heath, 1982). Recording at the inclusive end of the continuum is usually made by one or more tools including audiotapes, videotapes, movie cameras, still cameras, narrative records (Erickson & Wilson, 1982). Records at the inclusive end are more dependent upon technological advances to record and freeze segments of life and everyday events. In contrast, records at the exclusive end tend to reduce descriptions to categories of behavior re­ corded during the observation period (on-line recording). Se­ lectivity at the inclusive end of the continuum, therefore, is less constrained. Coding or representation at this end is done

168

CAROLYN M. EVERTSON AND JUDITH L. GREEN

retrospectively; the degree of distance from the "live" data is less at the inclusive end. Finally, rules for data collection are adaptive (Corsaro, 1985; Erickson, 1977; Heath, 1982; Spindler, 1982; Spradley, 1 980). Principled decisions are made within the observational period and during the project, based on need identified during the project. A general set of guidelines exist governing observation. New techniques or combinations of techniques are selected and used to answer questions. While systematic, the observation process at this end is broader and more flexible than at the exclusive end of the continuum. While the two ends of the continuum have tended to be mu­ tually exclusive in the past, there is no theoretical reason that the two ends cannot work together. For example, from research at the inclusive end of the continuum, naturally occurring vari­ ables, patterns of behavior, and events can be identified. The investigator would then be able to construct a category system or checklist to focus observation on specific aspects within the context. The information obtained with the focused tool could then be examined at the more inclusive end. This procedure can allow the investigator to change lenses and to move back and forth across and within levels of context. To pursue the lens notion, an analogy for the context issues above is a microscope (Green, 1983b). At the highest power, the microscope lens focuses on the details on the slide and the gen­ eral field is obscured. As the power becomes weaker, more and more of the field or general context is included. At the lowest power, the details are obscured and the field becomes the focus. The different powers allow the investigator to explain different aspects of the phenomenon and move back and forth across powers. This procedure permits exploration of the observed phenomenon under different levels of focus. On the continuum, then, the exclusive end is often considered the higher power. It decontextualizes behaviors and focuses on the occurrence of specific behaviors. The field surrounding these behaviors is gen­ erally ignored except for general demographic information. As the researcher moves toward the inclusive end, more and more of the field or context is considered. The studies selected for in­ depth discussion in the section of this chapter dealing with ob­ servation as inquiry illustrate various aspects of the continuum and the way that researchers may move across the ends of this continuum. The discussion of context above is meant to be illustrative rather than all-inclusive. The topic requires greater treatment, not only of context and selectivity issues, but of the levels of context and their relationship to analysis procedures and sta­ tistical treatment (e.g., Burstein, 1980; Martin, Veldman, & An­ derson, 1980; Snow, 1974; and chapters in this volume). Other context issues that have been considered in recent research in­ clude intrapersonal versus interpersonal contexts (e.g., Bloome & Green, 1984; Cazden, 1983; Heap, 1980b), the interrelated­ ness of levels of context of schooling (Barr, in press; Barr & Dreeban, 1983), and functions of language used as context for instruction (e.g., Collins, 1983, in press; Marshall & Green, 1978; Mehan, 1979a; Pinnell, 1977). By considering the different types and levels of context, future research can address more systematically questions of match and mismatch of performance in different contexts, of factors influencing performance in a given setting, and of differing per­ spectives and levels of description of a phenomenon.

Snow ( 1974), building on the work of Brunswik (1956), arti­ culates the issues involved: Brunswik seems to have felt that intelligent human beings were ac­ tive, flexible, adaptive processors of information available in a prob­ abilistic, partially redundant environment. Above all, then, the experimenter must adapt his research methodology to fit this form of phenomenon, rather than trying to force the phenomenon to adapt to the experimenter. (p. 266)

Snow further states : Brunswik would say that we should continue to use systematic de­ sign for what it is good for. But he would note that we have so far failed to adjust that methodology to fit the adaptive, probabilistic functioning of human behavior or to derive molar descriptions of that behavior in multidimensional natural situations. (p. 269)

To paraphrase this point for observational research, the re­ searcher's goal is to freeze everyday activities as well as experi­ mental activities so that they can be examined systematically. To do this, the researcher selects a tool to obtain a representa­ tion of reality. The crucial point is to select a tool that fits the phenomenon under study and the level of context that is appro­ priate. In other words, a researcher must choose, construct, or adapt an observational tool, method, process, and program ap­ propriate to the question asked, the context surrounding the phenomenon, and the nature of the phenomenon. In this case, the task is to use a tool that will permit the researcher to record and store the active, flexible, adaptive processes that occur in teaching-learning situations and to consider the contextualiza­ tion cues (Cook-Gumperz & Gumperz, 1976 ; Corsaro, 1985) that form the basis of inference by both the participants in the situation and the researcher (e.g., verbal, nonverbal, and para­ linguistic features of language). This information then be­ comes the basis for the construction of data, for the reconstruc­ tion of the event, and for the analysis of data.

Systems for Recording and Storing Observational Data The systems used to record, store, and represent observations in this section are described in general terms. The intent of this section is not to review the literally hundreds of instruments/ systems available or to prescribe which systems are best. That task has been addressed elsewhere (e.g. Dunkin & Biddle, 1974; Rosenshine & Furst, 1973; Simon & Boyer, 1970a, 1970b). Rather the purpose of the discussion that follows is to provide a way of classifying the major types of systems used, to describe their general characteristics, and to explore how data are collected. However, before presenting the classification framework, the premises on which the classification is based will be discussed. Four premises underlie the classification of tools/instruments. First, the plethora of instruments can be sorted into a finite set of classes. For the purpose of this discussion, four classes of collection procedures have been identified: categorical systems, descriptive systems, narrative systems, and technological re­ cords. Second, each classification of systems/records can be identified by a unique set of characteristics which can then be

169

OBSERVATION AS INQUIRY AND METHOD used to discriminate among the different instruments (see Table 6.1). Third, all of the instruments are selective and all �re used to freeze behavior, events, and processes for later analysis. The key is not which is best but which is most suited for the question under study and which will adequately represent the segment of reality being observed. Fourth, the characteristics of the systems can be considered separately from the perspective or theoretical orientation that frames the particular study. In other words, the characteristics can be determined generically. This last premise means that the nature of the system is dis­ tinguished from the items in the system (e.g. verbal praise, teacher warmth, higher-order questions), since the items or units of observation are -related to the perspective of the researcher. (For other classification systems for observational methods see Genishi, 1983a; Gordon & Jester, 1973; Rosenshine & Furst, 1973; Simon & Boyer, 1970a 1970b; Wright, 1960.) The classification framework in Table 6.1 presents four types of information about each system: (a) general classification of the nature of the system; (b) types of systems; (c) methods of recording data found within each type of system, and (d) gen­ eral goals of the users. Each of these will be considered in turn as they appear in Table 6.1. The order of presentation is related to the context continuum presented previously (see Figure 6.3). The systems listed at the far left of Table 6.1 would be placed at

the far left or exclusive end of the context continuum. Systems to the right of categorical systems include larger and larger seg­ ments of context and, therefore, will be found at the more inclu­ sive end of the continuum. Just as it is possible to move across or to consider different levels of context, it is possible to com­ bine more than one system within the same study. (See the sec­ tion on observation as inquiry for an illustration of this point.) The key is to match the type of system selected with the ques­ tion and phenomena under study at the particular point in the study. Nature of the System. The question of whether a system was a

closed one or an open one was considered. Closed systems are those that contain a finite number of preset categories or units of observation (e.g. teacher criticism, student response to con­ vergent questions). The categories for a closed system are mutually exclusive and defined in advance to reflect philoso­ phical, theoretical, empirically derived, or experience-based be­ liefs about the nature of the process, event, or group under study (cf. Dunkin & Biddle, 1974). Observations are confined to identifying and recording behaviors contained within the sys­ tem itself. The observation system is self-contained (i.e., no new categories may be added to the system during observation periods even though revisions of the system may be done later). Categorical systems are considered complete or closed systems.

Table 6-1 . Ways of Recording and Storing O bservations: Broad Classifications Category Systems

Nature of

Systems

Types of

Systems

Methods of

Record ing

Goals of Users

Descriptive Systems

Narrative Systems

Technological Records

Closed system

Open system

Open system

Open system

Always has preset categories.

May have preset categories.

No p reset categories.

No preset categories.

Sam ples of behaviors, events, processes that occur with i n a given time period. Boundaries of events are often ignored. Focus on behavior in general.

Meaning is viewed as context specific. Boundaries of events are considered d u ring as wel l as prior to observations. Samples behaviors, events, processes that occur with naturally occurring boundaries.

Meaning is viewed as context specific. Boundaries of events are considered d u ring as well as prior to observation for live record ing. Sam ples behaviors that occur with i n natural ly occu rring boundaries.

Meaning is viewed as context specific. Sam ples behaviors, events, processes that occur within a given time period or a given event. No atte mpt to filter or mediate what is observed.

Category, sign, checkl ist, rating scales

Structured descriptive analysis systems

Specimen record, d iaries, anecdotal records

Sti l l pictu res, videotape, audiotape

Selected behaviors coded on form using tal l i es, numeric representations, ratings. Records one behavior at a time. Used live and on line. Behaviors coded on special form.

Selected behaviors recorded using verbal symbols and/ or transcription. Records m u ltiple aspects of behaviors. Also considers broad segment of event. Generally used with permanent record (e.g., aud iotape, videotape).

Broad segments of events recorded orally or i n written form. Observation recorded in everyday syntax.

Potentially the widest " lens. " An unfi ltered record of all behaviors and events that occur i n front of the camera or within pickup of m icrophone. Depend i ng on the focus of the researcher the lens can be wide or narrow.

Study a wide range of classrooms to obtain normative data, identify laws of teach ing, general ized across cases. Less interest in ind ividual variation within cases.

Obtain detailed descriptions of observed phenomena, explain unfolding processes, identify generic princi ples from explorations of specific situations, and generalize within cases as well as compare findings across cases.

Obtain detailed descriptions of observed phenomena, explain u nfold i ng processes, identify generic principles and patterns of behavior in specific situations. Goal is to understand specific case and to compare findings across cases.

Obtain a permanent record of event to be recorded. Decisions on what to record are related to goals of researcher and questions under consideration. The pu rpose is to freeze the event i n time for analysis at a later point i n time.

170

CAROLYN M. EVERTSON AND JUDITH L. GREEN

An open system uses a wider lens. These systems may have a range of preset categories used to describe behaviors (or other types of phenomena), or may contain categories generated from observed patterns. The categories within an open system differ from those within the more closed systems. Individual be­ haviors/phenomena in an open system are often recorded using more than one category. For example, behaviors may have more than one function; therefore, the categories may be com­ bined to reflect a variety of co-occurring functions (e.g. a teacher's message can focus students and simultaneously signal control ; the delivery provides the multiple cues). Variables re­ flecting patterns of behavior are obtained (e.g. initiation­ response-evaluation ; cf. Mehan, 1 979a). The patterns can be described in terms of both functions and effects. Within the types of open systems, differences occur. Descrip­ tive systems are more constrained than either narrative systems or technological records. Descriptive systems may have preset categories as described above and also categories generated by the data. The narrative systems and the technological records have no predetermined categories. The categories for these sys­ tems/records are derived from analysis of the data after obser­ vations, not during collection. Since permanent records are used (e.g. written, audio, or video), identification of categories of behavior, patterns of behavior, and construction of variables is possible retrospectively. Such variables can be reliably and validly identified (Erickson, 1979). Another way to view open systems is that they can be generative; that is, new variables can be identified throughout the analysis process. The number and type of variables are grounded in (suggested by) observed pat­ terns in the data as well as the theoretical framework guiding analysis. One difference, then, between closed and open systems can be seen in the point of identification of categories. First, in closed systems, categories are selected a priori. In the more open sys­ tems, categories are generally drawn from the raw data. In both systems, theoretical, philosophical, or experience-based deci­ sions are used to construct the categories. A second difference is found in the role of the observer. The observer using a closed system is limited to recording only those items listed. With open systems, the observer captures a wider slice of context. The ob­ server's perceptions, training, and framework determine what will be recorded. The notable exception to this is the techno­ logical records. Technological records augment the observation process by providing a permanent record of the events. At the point of analysis, the observer as the instrument of observation becomes a key factor. Whether one chooses an open or a closed system may depend upon the question, the philosophy, the theory, and/or the point of observation within the study. Each type of system/record serves different purposes and produces different descriptions. Methods of Recording Data. The second factor that was consid­ ered in distinguishing between and in classifying the systems was the means of recording data. Data from categorical systems are recorded live and on line in general. That is, observed be­ haviors are recorded on the coding forms as they occur, either as ratings at the end of a set period of time, as numeric symbols for the behavior observed (e.g., 4 represents 'direct questions'; cf. Flanders, 1970), or as a tally on a checklist. Each behavior is

generally recorded in only one category. As indicated in Table 6. 1 , behaviors, events, and processes are sampled during a given time period. Two types of boundaries for the period of observa­ tion have been used; time and event. When time is the sampling unit the boundaries of specific events are ignored. For example, the first hour of school might be recorded beginning at 9: 00 and continuing until 10:00. When event is the sampling unit, the observation begins with the onset of the event and ends with its closure. That is, in the real time of the classroom, the observer uses the cues given by participants that the event has begun and ended. For example, recording begins when the teacher calls the Bluebirds to the reading group and ends when the teacher dismisses them. Within the period of observation two other types of record­ ing procedures are used. First, tallies are made of each occur­ rence of a specific behavior, event, or process. Only those behaviors designated within the category system or checklist are recorded. Second, the behaviors on the system are recorded at preset time intervals (e.g. Flanders, 1970 - every 3 seconds; Stallings, 1983 - alternating 5-minute segments during interac­ tions). Table 6.2 summarizes the differences among instruments in this classification. Each type of recording procedure pro­ duces different types of raw data and leads to different descrip­ tions. (See the section on units of observation for additional discussion of categorical systems.) The descriptive systems are generally used in combination with technological records (e.g., audiotapes, videotapes). De­ scriptive systems are applied to such records in one of two ways. First, descriptive systems that have been formalized and have predetermined categories to guide description are applied to either the video record or to a combination of video record and transcript (e.g., Adelman, 198 1 ; Bellack, Hyman, Smith, & Klie­ bard, 1966; Elliott, 1976; Frederiksen, 1 98 1 ; Green & Wallat, 1979, 198 1b; Mehan, 1979a; Sinclair & Coulthard, 1975 ; Stubbs, 1983). The behaviors are then recorded in a way that reflects the communicative or pedagogical structure of the un­ folding event. Generally, this requires patterns of behavior to be identified and represented systematically. Second, transcrip­ tions are made and usually linguistic symbols are used to identi­ fy stress patterns, turns, tied sequences, and so forth (e.g. Cochran-Smith, 1984; Collins, 1983; Erickson, 1982; Heap, 1983; Mishler, 1984; Ochs, 1979; Stubbs et al., 1979; Tannen, 1979). Both of these descriptive approaches use a retrospective analysis of specific aspects of the total recording. These systems, however, continually refer to the broader record of the event recorded by technological records. The period of observation may vary in length; however, data are recorded within naturally occurring boundaries of events or contexts. Meaning in these systems is considered to be situation specific. (See section on units of observation for additional discussion of descriptive systems.) With narrative systems descriptions are recorded using spoken or written language. An observer either writes a narra­ tive description of the unfolding event or orally records a verbal description on a tape recorder. The recordings can be made live, as in the case of critical incident records and specimen records, or can be reflective, as in the case of diaries or journal records. Table 6.3 describes the differences among the systems. The pe­ riod of observation may vary depending upon the purposes of

171

OBSERVATION AS INQUIRY AND METHOD

Table 6.2. Categorical Approaches: Types of Recording Systems and General Characteristics Category Systems

1 . Contain preset categories in which all

occu rring behavior must be recorded in one of the categories.

Checklist Sign Systems

Rating Scales

1 . Contain preset categories in which only

1 . Contain preset categories for which weighted j udgements are req u i red (e.g., 1 low, 3 moderate, 5 high).

the specified behaviors are recorded, not all occurring behavior.

=

=

=

2. Essentially a classification system ;

2. Essentially a classification system ;

2. Essentially a system using interval scales (e.g., ordinal : determi nation of greater or lesser ; interval : determi nation of equal ity of differences between points).

3. Generally used in settings as events

3. Generally used in settings as events

3 . Generally used at the end of an observation period to s u m marize cumu lative d i rect observations. Observer need not be in the setting when using.

4. Amenable to record ing smal ler u n its of behavior which require low inference.

4. Amenable to recording smaller un its of behavior which req u i re low inference.

4. Amenable to assessing high-i nference or global constructs (e.g., teacher warmth ; emotional intensity).

5. Time based in that behaviors are recorded at designated intervals. The period of time for record ing multipl ied by the nu mber of tallies equals the approximate length of observation. (e.g. a 5-m i n ute observation in which codes are assigned every 5 seconds yields 1 2 per m i nute and 60 per 5-minute observation). Events may be the locus of observation, but time sam p l i ng is the domi nant proced ure.

5. Can be time based o r event based. That is, a general ti me period (period of observation) is set or an event in which observation will be undertaken is specified. Total n u mber of tallies is not necessarily related to the length of the observation period.

5. Time for record ing is not relevant except that too long an interval between observation and assessment can yield loss of information.

nominal scale.

nominal scale.

unfold. Can be used with video or audio records.

unfold. Can be used with video or audio records.

(For an in-depth coverage of rating scales see Remmers, 1 963 ; see Fassnacht, 1 982 on d i mensional scales.)

Table 6.3. Types of N arrative Systems Diary/Journal Records

Critical Incident

Specimen Description

Field Notes/Descriptive N otes

1 . Retrospective written records of either ones's own behavior/experience or that of others.

1 . On-l ine or retrospective

1 . On-line, more systematic and intensive record ing of events. Aim is to record all behavior d u ring a designated t i me period in an u n i nterru pted stream and in detail.

1 . Written account of what the

2. Useful for chronicling longitudinal information about individ uals, groups, activities, and so forth.

2. Used to gain specific i nformation in descriptive form that bears on a q u estion or q u estions of interest (e.g., to monitor i m plementation of school-wide rules).

2. Useful for record ing " everything " in sequence and unselectively about what subjects do, say, and what the situation is. May be l ess useful for record ing longitudnal information u n l ess many records are collected over time.

2. Used to record information 'obtained d u ring partici pant observation.

3. No particular systematic

3. Recorder is usually a person

3. Req u i res an observer trained

3. Requi res an observer trained

trai n i ng needed. Presumption of reasonable use of written language. Diarykeeper must understand the q u estion of interest and the theoretical framework to record appropriate information.

record used to record relevant behavior or incidents that add ress an area or topic of i nterest.

in the setting i n a position to observe d i rectly. Req u i res (a) definition of field of observation, person, place, situation, behavior to be set up in advance ; (b) definition of behavioral criteria to determine which i n cidents are critical.

i n the conceptual framework or tradition guiding the observations.

investigator sees, hears, experiences, and thi nks in the cou rse of collecting and reflecting. Refers to all data collected i n the field i n the course of the study.

in field work techniq ues and i n the conceptual framework or trad ition guiding the observation. Trai ning insures consistency of record ing of observations.

172

CAROLYN M. EVERTSON AND JUDITH L. GREEN

the study. The duration of the observation can be a single event, a critical incident, or a longer period (e.g., the first day of school, the first hour of school, the first 2 weeks of school for the first hour). As in the case of descriptive systems, researchers using narrative systems consider meaning to be situation specific. The data recorded are constrained by the observer's perceptual framework, acuity, training, and written or oral fluency. (See the discussion on narrative systems in the section on units of observation for additional information.) Technological records are live recordings of events, processes, and groups. Records are obtained by using electronic devices that make permanent records (e.g. videotapes, audiotapes, vid­ eodiscs). This record is raw data and must be acted upon sys­ tematically in order to construct data or representations of events. Technological records can be used in combination with any of the other systems depending upon the researcher's goals and questions. The decisions involved in placement of the camera, in gaining access, in selecting the locus of observation are some of the factors that influence the types of informa­ tion that can be obtained from these records. (See Erickson & Wilson, 1982 for a comprehensive discussion.) Periods of re­ cording will vary depending upon the question under study. (See the section on units of observation for additional informa­ tion about technological records.) General Goals of the User. The final class of distinctions that will be considered involve the goals of the investigator. Users of categorical systems are generally concerned with studying a wide range of classrooms in order to obtain normative data and to identify general laws of teaching. Little focus is given to indi­ vidual cases, except to identify contrastive groups (e.g. more­ and less-effective classroom managers). Within-case variations are less often considered. Users of descriptive systems are generally concerned with ob­ taining a detailed description of observed phenomena in order to explain developing processes, and to identify generic princi­ ples by exploring specific processes. The purpose of this work is to explore both within and across occurrences (cases) of specific phenomena (type case analyses) and to contrast findings in one setting with those in other settings (Erickson & Shultz, 198 1 ; Florio & Shultz, 1979; Green & Harker, 1982). Of concern are questions such as how two contexts are functionally equivalent; what behaviors are stable across situations; and which are situation specific. In other words, issues of functioning within a case and across cases from a comparative perspective are considered. Narrative System users are also concerned with obtaining de­ tailed descriptions of observed phenomena in order to explain unfolding processes and to identify generic principles and pat­ terns of behavior within specific events. The goal is to under­ stand not only what is occurring but to identify factors that influence the occurrence of this behaviors. Like descriptive sys­ tems, the key is to assemble a broad picture of the phenomena under consideration. Differences in intent influence what will be done and how it will be done (see Table 6.3). For example, a school psychologist using a specimen record or a critical inci­ dent technique will record behaviors to obtain data about a

specific student who needs an intervention program or about an ongoing problem. In contrast, an ethnographer using a specimen record will record (e.g. descriptive field notes) dif­ ferent information, information about the nature of the class­ room or phenomena as a social process or social system. Finally, the goal of the user of the technological record is to obtain a permanent recording of an event or phenomenon so that it can be studied in greater depth at a later time. The goal and theoretical framework of the researcher as well as the ques­ tion under study will influence what is recorded. The observation systems described above are tools that can be used to study a wide variety of educational phenomena. Each system, however, will record a different segment of reality; each obtains a different level and type of description of the ob­ served phenomena; each stores information in different forms; and each permits retrieval of different types of information. In other words, the structures of the different instruments con­ strain and influence the nature of the information that can be obtained about the observed phenomena. The range of different types of instruments described above means that the researcher has a variety of options. The task is to select from this range of methods or tools the one or ones best suited to both the ques­ tion and the phenomena under study.

The Systems, Units of Observation, and Aggregation of Data The discussion above described the general structure of obser­ vation instruments. In this section, the nature of these systems will be elaborated further and the nature of units of observation found within observational systems will be considered. The issue of what units of observation are is complex. Units of ob­ servation must be considered in a variety of ways. First, a di­ versity of units exists. For example, Fassnacht ( 1982), in an extensive treatise on observation theory and practice, identified nineteen different units. Table 6.4 provides a brief description of these. The description of units in Table 6.4, while useful in demon­ strating the diversity of units of observation, is not meant to be a definitive statement of units or a set of definitions for the dis­ cussion that follows. This discussion is meant to serve primarily as a point of departure for the discussion of units and their relationship to systems of observation. Second, each unit of observation represents an independent variable in that it is not manipulated overtly. However, during data analysis any given unit can function as a dependent vari­ able for analysis purposes. For example, in a correlational study, measures (units of observation) of student engagement and success in class activities could be correlated with certain teacher behaviors (Fisher et al., 1980). In a descriptive study the researcher could explore shifts in behaviors (e.g., ways of bid­ ding for a turn at talk) within and across lessons to determine whether students had acquired a particular social rule or norm of behavior (Hammersley, 1974; Stoffan-Roth, 1981). Third, units of observation can be aggregated during the data analysis phase of the study to form larger classes of units. For example, praise (Brophy, 1979) and attention (May, 1981) have been found to be multifaceted phenomena. Fourth, by identifying patterns across units of observation, new descriptive units can

OBSERVATION AS INQUIRY AND METHOD

173

Table 6.4. Terms for Behavioral U nits Type of U n it

General Definition

1 . Natural un its

Detected through the perceptual system and reflected in natural language (Barker & Wright, 1 955). Perceived as breaks in streams of behavior.

2. U n its of behavior

Can be described as a natural unit also. Laws of perceiving forms are also valid for behavioral forms. Just as there are forms in the world of objects, there are dynamic and temporal configu rations (e.g., d i rection or speed of movement, position of the body).

3. Ind uctive vs. ded uctive

Refers to the process by which u nits are constructed . For ind uctive u n its, one starts with the behavior and attempts to classify (e.g., ethology). For ded uctive un its, u n its are derived from theory, hypotheses, or logical propositions.

4. Di rectly observable vs. i nferred

Two types are distinguished : those that are on princi ple i nvisible (e.g. a person's intentions, emotions, thoughts), and those that are invisible d ue to circu mstances (e.g., instances when the observer cannot see the behavior because of an obsta­ cle). Both instances req u i re inference. The issues involved here are not so much whether inference should be used but at what stage i n the data collection.

5. Descriptive vs. evaluative

The former notes concrete behaviors and suspends j udgments. The latter sum mar­ izes and assesses a series of behaviors (e.g., one can observe a child for 30 m i nutes and conclude he is angry, or one can record the concrete behaviors which could lead to that judgment.)

6. Phenomenological

Behaviors that have the same form (Brannigan & H u m phries, 1 972).

7. Morphological

Simi lar to a phenomenological u n it with emphasis on the formal or structural aspects of the behavior as criteria for constructing.

8. U n its based on factor analysis

U n its are based on d i mensions emerging from the statistical analysis (see Em mer & Peck, 1 973).

9 . Discrete vs. continuous

This refers to the extent to· wh ich it is possible to count or measure behaviors. The issue of discreteness vs. conti nuousness arises with the use of rating scales.

1 0. S i m ple vs. complex

Two views exist of these types of u n its. The first views the uniqueness of the h u man cortex to analyze complex events and relies on ratings of complex concepts (Langer, Schulz, & Thun, 1 974). The second is the construction of complex u n its out of al ready observed si m pie ones (Richards & Bernal, 1 972).

1 1 . Indices as u nits

Composed of various indicators d rawn together. Mentioned more freq uently in the sociological literature than in the psychological l iterature.

1 2. Reductionist

These are the result of fi nding the smallest unit of meaning, not necessarily the smal l est observable u n it. To be meaningful, it m ust have a particular meaning for the observer of the behavior or evoke a particular response in a partner.

1 3 . Causal

Behaviors with a common cause are regarded as identical.

1 4. Functional

A u n it d efined with respect to its effect or context. The emphasis is on the i mpor­ tance of the context.

1 5. Situations as u n its

If one is concerned with behavior which is to some extent rule bound and has recurrent elements, then situations can be viewed as units. More precisely such a unit should be comprised of both situation and behavior. Barker and Wright (1 955) regard their " behavior setting " as such a u n it.

1 6. Molecular 17. Molar

Both terms are taken from Barker and Wright (1955). Molecular u n its are known as actones and molar u n its are actions (e.g., molecular: perspiration ; molar : hurry­ ing to school).

1 8. Time units

These refer to time intervals of time-sam pling methods and to any ti me-derived measures used for behavior observation.

1 9. Action un its or events

Conceptually sim ilar to behavior unit and natural u n its, but is distinguished by its form and content.

Note. Adapted from Theory and Practice of Observing Behaviour (pp.76-81) by G. Fassnacht, 1982, London: Academic Press. Copyright 1982 by Academic Press. Reprinted by permission.

174

CAROLYN M. EVERTSON AND JUDITH L. GREEN

be constructed. For example, Mehan ( 1979a) identified a pat­ tern of initiation-response-evaluation. This unit is composed of a series of smaller units, adjacency pairs (e.g., initiation-re­ sponse; response-evaluation). The issue of units of observation, therefore, is related to both data collection and data analysis. The units selected constrain what can be reported. They are lexical items (words/mor­ phemes) in a system of language (Fassnacht, 1982.) The rela­ tionships between and among units, as specified by the framework guiding the systems, make up the grammar of the system. An observational system, therefore, is a language sys­ tem. That is, it determines what can be described and which aspects of the phenomena are described. Fassnacht (1982) cap­ tures the issues succinctly : Decisions about units are of great importance insofar as they estab­ lish principles with regard to the statements that can be made about a topic before anything has been discovered about it. By deciding on certain units, the nature of the relationships that can subsequently be discovered is defined. One can neither discover nor construct anything beyond the limits imposed by these units. The unit defines, so to speak, the intellectual limits of possible statements and only allows relationships within the context. (p.57)

This statement highlights how the selection of units is a cen­ tral issue in observational research. It also emphasizes the rela­ tionship between the choice of unit and the framework guiding the study as well as the relationship between the choice of unit and the nature of the description obtained. The choice of units depends on the theoretical, philosophical, experience, or com­ mitment basis of the framework that guides the observational study (Dunkin & Biddle, 1974). Different frameworks require and lead to different units. These frameworks reflect different foci. Another way to think about this issue is that "different units emerge depending on whether one bases them on their structural or functional aspects, their content or the way they are collected, their purpose or their relationship with other units " (Fassnacht, 1982, p. 76). Before proceeding to the discus­ sion of the relationship between different types of systems and units of observation, two additional facts about units un­ covered during the exploration of this topic will be presented. These facts and the discussion above form a general frame for the considerations of units and systems that follow. First, no simple dichotomy or continuum can be used to classify or ex­ plore the different units. Units tend to overlap; they may repre­ sent different levels of context; different levels may exist within a general category (e.g. within natural units, behavioral units, de­ ductive units); units may take a variety of forms and may in­ clude a range of different types of content. Second, some units are used to record information and are determined on an a pri­ ori basis. Other units are constructed from smaller recorded units to permit analysis of data. Still others are constructed dur­ ing analysis and are products of the analysis. To illustrate some of the points, one type of unit, natural units, will be explored without reference to specific instruments. The remainder of the units will not be considered individually, but will be considered by system type : categorical systems, de­ scriptive systems, narrative systems, and technological records. No attempt will be made to cover each unit identified by Fass-

nacht (1982) presented in Table 6.4 or to adhere to those defini­ tions. Rather they serve as a point of departure. Natural units are defined by Fassnacht (1982) as breaks in streams of behavior (cf. Barker, 1963, 1968; Barker & Wright, 1955). Where the stream of behavior being observed is seg­ mented depends on the theoretical orientation or the concep­ tualization of the process that frames the recording, analysis, and/or construction of the units of observation. Fassnacht (1982) argues that what makes a natural unit natural is not that it is reality itself, but that reality has determined the unit. An­ other way to view this type of unit is that it was not constructed by the observer first, but rather the observer described a natur­ ally occurring event. A natural unit, therefore, is a phenomenon that is perceived by members of a particular social group or culture as real. For example, in many classrooms, reading groups are natural units. They are organized by teachers in the environment and are understood to be segments in the school day by all members of the classroom, school, and community. They exist in a particular time slot in the school day; they have specific rights and obligations for participation associated with them; they have specific content associated with them; they have particular configurations of participants from the total classroom group; and they have been created to meet specific goals. These units are defined by the reality of classroom life. Such units can exist on both a micro/molecular level and a macro/molar level. The community, the school, the classroom, and even the reading group are molar units or micro units rela­ tive to each other and other segments of reality. In classrooms, reading groups may be seen as micro or macro. They are less than the total composite of a classroom, therefore, they do not reflect all of classroom life. The reading group defined this way is a micro unit. However, in contrast to a peer tutorial situation or an individual student working alone, reading groups would be molar units. In comparison to the community or the school, classrooms are micro units. The size of the unit varies with the focus selected. The same unit can be both micro or macro de­ pending on the framework used and the way the unit is concep­ tualized. All of these units are natural units. Natural units can also exist beyond the setting or episode level. A natural unit for a sociolinguist interested in studying classroom communication and classroom life might be narra­ tive forms of discourse (e.g., Cazden this volume; Cochran­ Smith, 1984; Cook-Gumperz, in press ; Michaels, 198 1 ; Scollon & Scollon, 1984), instances of requests (Wilkinson & Calcula­ tor, 1982), or ways students get help from teachers (Cooper, Ayres-Lopez, & Marquis, 198 1 ; Merritt, 1982). They can also be sequences of talk, questioning patterns, norms for participa­ tion, the nature of peer interaction, and nonverbal behavior. (For examples, see edited volumes by Borman, 1982; Cazden et al., 1972; Cook-Gumperz, in press; Garnica & King, 1979; Gil­ more & Glatthorn, 1982; Green, Harker, & Wallat, in press; Green & Wallat, 1981a; Hymes, 198 1 ; Trueba et al., 198 1 ; Wil­ kinson, 1982). Natural units can vary depending on the theoret­ ical frame used. Natural units for the ecological psychologist (e.g., Barker, 1963, 1968; Barker & Wright, 1955; Gump, 1969; Moos, 1976) will differ from those used by the ethologist (e.g. Blurton-Jones, 1972; Lorenz, 198 1). The units identified by re­ searchers from these disciplines will differ from those units used by educational psychologists, linguists, social psychologists,

OBSERVATION AS INQUIRY AND METHOD

clinical psychologists, sociologists, behaviorists, phenomeno­ logists, and so forth. The brief discussion above suggests that units can overlap and can co-occur. They are determined within the specific framework guiding the study or observation. They will differ in size, scope, content, level, function, and structure. In preparing for observation, careful consideration must be given to which units are used and to how they are to be determined and con­ structed. In the remainder of this section, different types of units associated with each of the different classifications of observa­ tion systems will be highlighted: categorical systems, descrip­ tive systems, narrative systems, and technological records. This discussion is not intended to be all-inclusive; rather, illustrative examples of units from each of these systems will be provided. Also included will be a discussion of some of the issues related to units from the various systems. Before turning to the discus­ sion of the units within each classification, one additional point should be made. The classification of systems presented in this chapter was constructed to provide a framework for distin­ guishing among the ways of recording observations that have been used in research on teaching. The first class of systems is usually referred to in the literature as "category systems" (cate­ gorical systems in the classification in this chapter) and the nature and structure of these has been well documented else­ where (e.g., Dunkin & Biddle, 1974; Gordon & Jester, 1973; Rosenshine & Furst, 1973; Simon & Boyer, 1970). The other classes of systems were formulated to reflect the nature and structure of those systems. In addition, narrative and descrip­ tive systems and technological records tend to be relatively recent developments in educational research and have not as yet received extensive treatment. For this reason the discussion of these systems is more elaborated than that for categorical systems. UNITS AND CATEGORICAL SYSTEMS

As suggested in Tables 6.1 and 6.2, categorical systems are closed systems with units of observation specified on an a priori basis. A limited set of units is used with any particular instru­ ment and units are tallied in some manner. The units within a checklist, category system, or rating scale can be viewed in a variety of ways. First, the units are generally derived deductively. Researchers have used a variety of perspec­ tives to select and/or construct the units for the system. While in the past units have generally been deductively derived, there is no theoretical or conceptual reason that units for a category system, checklist, or ration scale could not be determined in­ ductively. That is, the items for these categorical systems could be derived from data from an earlier study or from a current study. If a technological record was used (e.g., videotape or au­ diotape) then a category system or checklist could be developed from inductively derived units. This inductively derived system could then be applied to the technological record. In this way the findings of one or more studies that identified a series of patterns could be used to design a category system, checklist, or rating scale. Second, units in category systems and checklists generally re­ flect a behavioral stance (e.g., Berliner, 1978; Brophy & Good, 1970b). That is, dynamic, temporal configurations are perceived

175

as behavior units (Fassnacht, 1982). Each behavioral unit tends to represent a wide variety of forms. For example, the verbal dimensions ofteaching are reflected in a group of ten categories in the Flanders Interaction Analysis System (Flanders, 1970). Each category is a behavioral unit and represents a different, discrete type of behavior. For example, all student talk was placed in either a category labeled " Student initiation" or in one labeled "student talk." Other researchers have differen­ tiated these categories to a greater degree (e.g., Ober, Bentley, & Miller, 1 971). Teacher talk in Flanders' system was represented by seven categories. Four categories were assumed to represent teachers' indirect influence (e.g., accepts feelings, praises or en­ courages, accepts or uses ideas of students, and asks questions). Three categories were assumed to reflect teachers' direct in­ fluence (e.g., lectures, gives directions, and criticizes or justifies authority). The final category represented silence or confusion. This system was designed to be used on line (live). Regardless of content, all teacher and student behaviors were coded in one of the ten categories. While greater numbers of behaviors or units were used in other category systems, the format of these systems and the assumptions underlying many of them are similar to the Flanders' System (See Simon & Boyer, 1970a, 1970b for a gen­ eral review of category systems; see Medley, Soar, & Coker, 1984; Hough, 1980a, 1980b, 1980c and Stallings, 1983 for exam­ ples of more recent systems.) Third, units within category systems and checklists are also discrete units and simple units. Each behavior observed is coded in only one category. One exception, however, is with sign systems where the observer indicates all behaviors that apply. In addition, the units within these systems reflect only directly observed behaviors. Finally, in contrast to category systems and checklists, rating systems include continuous rather than discrete units. For ex­ ample, one type of rating scale uses a semantic differential for­ mat (Osgood, Suci, & Tannenbaum, 1957). This approach creates a continuous unit (e.g., individual attention by teacher to group instruction by teacher). The observer places a mark on a point between the two units to reflect which variable occurred most frequently or the intensity of the occurrence of the vari­ able. (For examples of this approach see Marshall, Hartsough, Green, & Lawrence, 1977; Marshall & Weinstein, 1982; and Remmers, 1 963 for a detailed discussion). Other types of rating scales require the observer to record a single rating (e.g., per­ centage of occurrence of a behavior; degree or intensity of occurrence - high, medium or low). Rating scales may also include units based on factor analysis (Emmer & Peck, 1973) and units that require some degree of observer inference. To summarize, a variety of units of observation have been used for category systems in general : deductive units, behav­ ioral units, and discrete units. Category systems and checklists have also included simple units, molecular units, and directly observable units. Rating systems add continuous units, units based on factor analysis, and inferred units. Additionally, they can include micro or molar units. As the foregoing discussion demonstrates, no single unit is related to all systems classified as categorical systems; rather, different units are found in various systems. While units vary by system, some aspects of categorical systems are the same. In categorical systems units are deter­ mined and the relationship between units is specified prior

176

CAROLYN M. EVERTSON AND JUDITH L. GREEN

to data collection and analysis. Therefore, the construction of new units is not done during collection and rarely done during analysis. DESCRIPTIVE SYSTEMS

As noted in Table 6.1, descriptive systems are open systems. While these may have preset categories, the categories can be combined in a variety of ways in order to construct systematic descriptions of evolving lessons and to segment streams of be­ havior. Streams of behavior (both verbal and nonverbal) are recorded using symbols of everyday language. In some in­ stances, the verbal symbols are then translated into other types of symbols (e.g., dance notation, graphic forms) that reflect the actions, forms, or functions of the observed behaviors. Central to the descriptive system is the use of transcriptions of the flow of talk in the lesson or observed context. The need to have transcriptions of talk and/or actions means that descriptive systems are not used for on-line recordings. Re­ gardless of the ways data are coded or recorded in written form, all descriptive systems depend upon permanent records of ob­ served events. That is, descriptive systems are used in conjunc­ tion with technological records. In most instances, the person who will be analyzing and coding data was also involved in making the technological record. The technological record pro­ vides the basis for (a) in-depth analyses of evolving streams of behavior, (b) identification of patterns of behavior within the ongoing sequences, (c) identification and construction of new units of observation, (d) testing of identified patterns across time and situations, and (e) analyses of data from a variety of complementary perspectives. In addition, the existence of a per­ manent record allows the researcher to use the record with participants in a stimulated recall format to obtain the partici­ pants' perspectives on what was occurring and to validate the observed patterns in a form of triangulation (e.g., Elliott, 1976; Morine-Dershimer, 1981, in press-a, in press-b). Units of observation related to descriptive systems are both deductive and inductive. Such units can be selected on an a priori basis or derived from the analysis procedures. The a priori units are both natural and micro units. They are based on a conceptualization of what teachers do; that is, on what types of pedagogical moves they make (e.g., Bellack et al., 1966; Namuddu, 1982); on a theoretical perspective about the nature of communication (e.g., research on conversational analysis, ethnography of communication, ethnomethodology, socio­ linguistics, and sociology of language); or on nonverbal actions (e.g., Birdwhistell, 1970; Duncan & Fiske, 1977; Duncan & Niederehe, 1974; Galloway, 1971, 1984; Hall, 1966; Harper, Wiens, & Matarazzo, 1978; Knapp, 1978; Scheflen, 1972; Woolfolk & Brooks, 1983). The units used to guide descriptions are drawn from theoretical research in different disciplines. In addition, the relationships between units are also specified on a conceptual or theoretical base. For example, from conversa­ tional analysis, discourse analysis, ethnography of communica­ tion, and sociolinguistics come the descriptors of linguistic elements of conversation. These elements become units of anal­ ysis on a micro level. Units include, but are not limited to chunks of discourse (e.g., conversations or episodes, utterancesi sentences, phrases, words, phonemes, speech events, speech

acts, etc.). Rules of discourse and syntax help specify the rela­ tionships between elements. The type of unit and the relationship that exists between units are related to the question under study as well as the theo­ retical framework in which the analysis is grounded (e.g., conversational analysis, sociolinguistics, psycholinguistics). For example, using the construct of adjacency pairs in conversation, Mehan (1979a) found that many teacher-student interaction sequences had a structure that he labeled the 1-R-E (initia­ tion-response-evaluation) sequence. This unit may be a simple question, followed by a response which is then followed by an evaluative statement about the response or which can be a series of exchanges tied to each other through adjacency pairs (Sacks, Schegloff, & Jefferson, 1974). This sequence begins with an initiation and proceeds through a series of turns until the initiator receives the expected response as indicated by the na­ ture of the evaluation (Mehan, 1979a, 1979b). The micro units are tied together in ways specified by communication or con­ versational theory (e.g., through cohesion ties, adjacency pairs, etc). In linguistically based descriptive approaches, then, the micro units are units constructed to allow researchers to talk about elements of conversation, and the relationships between the units are specified by the theoretical frame guiding the study. For a related discussion of linguistic units see the discus­ sion of criteria for adequate observation at the end of the first section of this chapter. Within descriptive systems, micro units are used to construct more molar units. (For examples of systematic work using this framework see Adelman, 1981; Au, 1980; Barnes & Todd, 1977; Barr, in press; Barr & Dreeban, 1983; Bellack et al., 1966; Bloome, 1981, in press; Cochran-Smith, 1984; Collins, 1983, in press; Edwards & Furlong, 1978; Frederiksen, 1981; Gilmore, in press; Green et al., in press; Green & Wallat, 1981b; Lund­ gren, 1977; Mehan, 1979a; Sinclair & Coulthard, 1975; Stubbs, 1983; Wilkinson, 1982.) In other words, the micro units are dis­ crete units that can be combined to form more molar units that are inductively determined. The level of analysis depends on the theoretical framework guiding the individual study. However, as suggested above, all of the descriptive systems depend on permanent records and some form of transcriptions of these records. Transcriptions of audio and videotape records contain both micro units and natural units. How the transcription is made depends on the theoretical requirements of the researcher. Transcription is a principled process (Cochran-Smith, 1984; Heap, 1980a; Mishler, 1984; Ochs, 1979; Stubbs et al., 1979). For example, some researchers require transcripts with detailed information about pauses, intonation patterns, stress patterns, false starts, hesitations, and other such paralinguistic and con­ textualization cues (e.g., Collins, 1983, in press; Cook-Gumperz & Gumperz, 1976; Corsaro, 1985). Others can use transcripts with less detail. For some researchers transcriptions can be made on a turn-by-turn-at-speaking basis. That is, notations are made on the transcript each time a new speaker begins to talk. For some researchers transcriptions must be made on less than a turn-by-turn level. That is, the transcriptions must reflect the propositional structure of talk (e.g., Frederiksen, 1975, 1981; Green & Harker, 1982; Harker, in press) or the individual social message units (e.g., Bloome, 1981; Green & Wallat, 1979,

OBSERVATION AS INQUIRY AND METHOD

198 1b). The way in which the units in a transcript will be formalized will depend on the theoretical and conceptual needs of the researcher as well as the nature of the process and the level of the process under study. In addition, some transcription processes require the addition of nonverbal information. As­ pects of the form as well as the content of the transcription are units of observation. Transcriptions are natural units and are also the basis for, or rather the data for, the construction of other types of units. In the process of transcription the way talk is displayed is a theoretical statement about what units are im­ portant and about the relationship of units. A number of other units of observation are related to descrip­ tive systems : functional units, situations as units, inferred units. First, functional units will be considered. One way to think about functional units is to consider the purpose the observed behavior serves. For example, not all talk presented in question form is a request for information or is meant to elicit informa­ tion. Some questions actually serve an imperative function or are forms of indirect requests (Ramirez, in press; Sinclair & Coulthard, 1975 ; Wilkinson & Calculator, 1982). Consider­ ation of how the talk unit is functioning permits construction of categories that become functional units. This view of functional units is constructed by exploring the goals of patterns of ac­ tions. When the goals of larger patterns of action are consid­ ered, situations as units can be identified (e.g, letter writing is occurring, students are playing a game). From this perspective, situations can be considered as constructed by what people are doing, how they are doing it, and what definitions they have of these actions. The structural and functional aspects of these units can be inferred by considering how participants hold each other accountable for the actions and meanings that occur (cf. Barker, 1963, 1968 ; Erickson & Shultz, 1981 ; McDermott, 1 976; Philips, 1972, 1982). By considering the rights and obliga­ tions for participation and the structural aspects of situations, researchers are able to construct and explore situations as units of observation (e.g., Barker & Wright, 1955; Bossert, 1979; Erickson & Shultz, 198 1 ; Florio & Shultz, 1979; Green & Harker, 1982; Philips, 1972, 1982 ; Wallat & Green, 1982). The last units to be explored are inferred units. These units are a central part of the inquiry process used in conjunction with most descriptive systems. In most studies in which descrip­ tive systems are used, higher level or molar units are con­ structed from groupings of micro/molecular units. The basis for such groupings may be theoretical or conceptual frameworks that specify the relationship between units, or they can be in­ ferred from patterns of observed actions or behaviors. The pro­ cess of making inferences is principled and the basis of such inferences is a part of the inquiry process related to the con­ struction of grounded theory (cf. Glaser & Strauss, 1967). As suggested above, by observing ways people engage in everyday events, how tasks are structured, how people hold each other accountable, and what definitions they hold for tasks, re­ searchers can extract natural units and develop descriptions of ongoing events. Inferred units, therefore, are a central part of descriptive systems. The differences in units are often due to the differences in the frameworks guiding the inferencing process (e.g., anthropology, cognitive science, linguistics, developmen­ tal psychology, ecological psychology, educational psychology,

177

social psychology, sociology, and phenomenology, among others. To summarize, a variety of units of observation have been used with descriptive systems : inductive units, deductive units, natural units, discrete units, inferred units, units of behavior, molecular units, molar units, functional units, and situations as units. While not all-inclusive, this list is representative of the different units generally found in these relatively new types of observation and analysis systems. They are products of the cur­ rent phase of observational studies, Phase Four: the expansion and consolidation phase referred to at the beginning of this chapter. Descriptive systems tend to be approaches in which units are both specified on an a priori basis and are constructed during the inquiry/observational process or analysis process. In other words, descriptive systems tend to be based on a retro­ spective analysis of recorded events. This approach to observa­ tion permits in-depth, detailed description and analysis of evolving events and conversations within events and explora­ tion of the comparability of events, observations, and so forth. It also permits construction of variables and units of observa­ tion that can be used within the corpus of recorded events to explore the nature of observed phenomena and patterns within the data. Descriptive systems, therefore, are integrally tied to technological records. NARRATIVE SYSTEMS

As noted in Tables 6. 1 and 6.2, narrative systems are open sys­ tems, have no predetermined categories, and record broad seg­ ments of events or behaviors in oral or written form. While each of the types of narrative systems - diaries/journals, critical inci­ dent records, specimen records, and field notes - require oral and /or written records, the ways they are recorded or collected make the form, content, and scope of these systems different from both categorical systems and descriptive systems. Differ­ ences are related, in part, to the fact that the observer is the primary instrument of observation. That is, what is recorded is not necessarily specified on an a priori basis ; rather what is recorded depends largely on the observer's perceptual system and ability to capture in everyday language what is observed. In other words, researchers' perceptions and training, as well as their oral and written language abilities, influence what will be recorded and how it will be recorded. The narrative approach, therefore, is dependent on the individual doing the observing. Units of observation related to narrative systems are both similar to and different from those for categorical systems and descriptive systems. As in the case of categorical and descriptive systems, the locus of observation is predetermined. The observ­ er decides who will be observed, what will be observed, and where and when observations will occur. The observer records information in everyday language. A narrative record, then, is a form of natural unit or rather a record of naturally occurring actions, events, behaviors, and so forth. In this way, a narrative record is more like a technological record than either a categor­ ical system or a descriptive system. The observer in making a narrative record records a wide slice of life. No attempt is made to filter what occurred in any systematic way although a specific lens can be used (eg., management of events, sequences of teacher-student interaction). A narrative record, therefore,

178

CAROLYN M. EVERTSON AND JUDITH L. GREEN

becomes a permanent record of what was observed (e.g., Becker, 1970; Corsaro, 1985 ; Denham & Lieberman, 1 980; Evertson, Anderson, & Clements, 1980; Evertson, Emmer, & Clements, 1 980; Marshall & Weinstein, 1 982; McCall & Simmons, 1969; Pelto & Pelto, 1977; Sevigny, 198 1 ; Spradley, 1980). The locus of observation is a deductively derived unit. That is, the nature of the question, past research, and past units can be used to select macro level/molar units such as situations as units, actions as units, units of behavior, phenomenological units, and functional units. These units can serve as loci of observation within a given setting. They can also be derived inductively from the narrative records. The type of unit selected and the information recorded depend on the conceptual or theoretical framework guiding the observation. Narrative records are constructed in two ways: during obser­ vation and after observation. Three of the four types of narra­ tive systems described in Tables 6. 1 and 6.2 use narrative records made during the period of observation: critical incident records, specimen records, and field notes. Diary or journal re­ cords and some forms of field notes are made after the event. Each of these will be considered in turn. Since there is a rela­ tionship between the way a record is made of the observation and the units that can be derived from the record, the discussion will explore issues in recording related to each system. Critical incident records are made either on line or in a ret­ rospective manner. The observer records information in narra­ tive form about the implementation of a particular practice (e.g., a school-wide set of rules) or a particular type of behavior (e.g., disruptive behavior). The recorder uses a specific frame to guide what is recorded (e.g., the definition of behaviors to be observed). Discrete units such as place, person, situation, and type of behavior to be observed are set in advance. Patterns of behavior are recorded in everyday language. These patterns are extracted from the text to create observation units. Observation units are inferred from the behaviors recorded on the narrative record. These patterns become descriptors of the incident ob­ served. The narrative record is a context for interpreting and inferring these units; the records are not data in and of them­ selves. Additionally, the patterns identified can be classified to construct more molar level units. Units, therefore, are both de­ ductively and inductively derived. They can include behaviors, situations, actions, conversations, structure of phenomena, and so forth. The observer is the critical element, since the observer is the instrument of observation and recording. The structure of the record stems from the observer's training, theoretical per­ spective, and the type of information required in the study. The critical incident record, therefore, can be viewed as a con­ strained system that records a specific slice of reality, one de­ fined in advance and guided by a specific framework or theory. Specimen descriptions are more detailed than critical incident records. The aim of this type of recording is to obtain descrip­ tions of events that are systematic and intensive on-line records. The observer records the behavior that occurs during a desig­ nated time period. The goal is to obtain an uninterrupted stream of behavior with as much detail as possible. Recording is not intentionally selective. The observer records what a selected subject does and says, as well as the information about the set­ ting. Other individuals are recorded only in relationship to the

person selected for observation. The recorder attempts to make a chronological record of all of the main steps in any action (e.g., goes to the blackboard; writes an answer to the problem correctly; returns to seat). Interpretation is avoided. Informa­ tion is recorded from a participant perspective (cf. Barker & Wright, 1955). Once the specimen record has been made, the observer seg­ ments the streams of behavior into episodes that reflect the action or situation. Episodes are constructed by viewing behaviors in an episode as having a consistent direction, as be­ ing goal directed (cf. Barker & Wright, 1955). Behaviors, there­ fore, are viewed as leading the "subject towards a particular behavioral goal" (Fassnacht, 1982, p. 1 75). Episodes are natural units. They can also be considered, or can contain, behavior units, action units, and/or situation units. As with categorical and descriptive systems, specimen records permit the identifica­ tion and construction of a variety of types of units. Each con­ tributes different information about the observed phenomena and each provides a different type and level of description. Field notes, like specimen records and critical incident re­ ports, depend on the observer as the instrument of observation to obtain narrative records. Field notes are generally associated with participant observation in anthropology (e.g, Pelto & Pelto, 1977; Sanday, 1976 ; Spradley, 1980) and in sociology (Becker, 1970; McCall & Simmons, 1969). Particip:c.nt observa­ tion has been used in ethnographic studies of educational processes and settings. To understand the types of units that can be identified or constructed from narrative records made during participant observation, a brief discussion of the nature of this method needs to be considered. Participant observation is used to study the ways of living of a culture or social group. Participant ob­ servation ranges from active to passive participation (Spradley, 1980). Active participation means that the observer becomes involved in events and records events after the fact. This type of participant observation allows the observer to capture the in­ sider's perspective, to record the events as they were perceived by a participant in the event. Passive participant observation means that the observer does not participate in the events but remains an outsider to the event or the setting. In both in­ stances, the observer records the ways of living of the social group under study. What gets observed and recorded depends on both the framework guiding the participant observation and the type of ethnography used - e.g., comprehensive, topic centered, and hypothesis testing (Hymes, 1 98 1). Comprehensive ethnography explores an entire society or social group and its relationship to the larger society (e.g., Ogbu, 198 1). A topic-centered ethno­ graphic approach focuses on an aspect of a social group or so­ ciety - e.g., literacy (Collins, in press; Gilmore, in press; Heath, 1982, 1983 ; Szwed, 1977; Taylor, 1983) or schooling (Hymes, 198 1 ; Noblit, 1982; Smith & Geoffrey, 1968; Spindler, 1982). Hypothesis-testing ethnography is an ethnographic study framed by a specific theory. For example, the Whitings (Whit­ ing, 1 963) used socialization theory to frame a study of sociali­ zation practices in six cultures. Cook-Gumperz, Gumperz, and Simons (1981) used discourse theory, reading theory, conversa­ tional analysis theory, and other related work to frame a study of communication in classrooms and the ways in which

OBSERVATION AS INQUIRY AND METHOD communication leads to evaluation of performance. (For addi­ tional information see the discussion on observation as inquiry later in this chapter.) While these types of ethnography may appear discrete, in reality they represent different points on a continuum. Additionally, they may co-occur. For example, a single study may have different levels of context sampled, use past theory to frame aspects of the study (e.g., communication between participants; reading processes), and select specific phenomena to explore in more depth (e.g., narrative perfor­ mance, reading groups, playgrounds). Each of these approaches uses narrative records to freeze the events being observed. However, given the difference in pur­ pose, the type of information will vary to some degree. For ex­ ample, those working with teacher-student communication will require more information about the type of talk that occurred. The observer focusing on teacher-student talk will want to sup­ plement the narrative record of the event with a technological record. Participant observation, then, like the other narrative approaches and categorical and descriptive systems, is in­ fluenced by the questions being asked and the framework guiding the study. Like specimen records, once the narrative records are made, the streams of behavior are indexed (e.g., Erickson, Cazden, Carrasco, & Guzman, 1978- 198 1 ; Griffin, as cited in Merritt & Humphrey, 198 1 ). That is, events and episodes are identified. Information about who is participating, what contexts were created by participants, the nature of the events, and so forth are transcribed for each record. The indexing permits identifica­ tion of potentially functionally equivalent events and processes (Erickson & Shultz, 198 1 ; Florio & Shultz, 1979) across days and observations within the corpus. The indexed information may contain information on a variety of types of units : phenomenological units, situations as units, behaviors as units, action units. The specific types of units indexed will depend on the goals of the researcher. While indexing has been considered theoretically for field notes and technological records (audio­ and videotapes), this procedure is an important step in all ob­ servational studies. The units within narrative systems are dependent on the de­ scriptions recorded. While general information recorded on field notes is information about the unfolding events, observers often record other information about the nature of the process (e.g., Corsaro, 198 1 ; Sevigny, 198 1 ). Corsaro (1981) reports tak­ ing four types of field notes : personal notes, methodological notes, theoretical notes, and descriptive notes. The latter type of notes, descriptive notes, are those reported above. Methodologi­ cal notes refer to issues related to placement of equipment, positioning of the observer, gaining access to the setting, and recording information. The observer records all decisions made during the inquiry cycle. Personal notes refer to personal obser­ vations or reactions and to things to remember or consider. T heoretical notes refer to ties to theory and to observed pat­ terns. The observer records hypotheses generated within the sit­ uation and makes notes on patterns observed in one event so that they can be explored in other situations. The four sets of notes help to provide systematic information about the inquiry process and help frame the analyses of the data. Field notes, therefore, are records in everyday language of observed phenomena, methodological decisions, theoretical ob-

179

servations, and other relevant information. In anthropology and sociology, field notes also refer to historical records, and other information obtained in the field (e.g., Pelto & Pelto, 1977). Units of observation are constructed from information re­ corded on the field notes. Units of observation, therefore, are generally derived inductively. A wide variety of units can be derived by extracting patterns of behavior or segments of goal­ directed behavior. These patterns can be sequences of behavior, situations, actions, or structures. In addition, the units observed in one observation or setting can be used to guide decisions about what to observe in subsequent sessions. While field notes are generally used within an anthropo­ logical or sociological perspective, recent work on the study of teaching has also begun to use this form of narrative record (e.g., Evertson, Anderson, & Clements, 1980; Evertson, Emmer, & Clements, 1980; Fisher et al., 1978; Marshall & Weinstein, 1982). More information will be presented about these ap­ proaches in the section on observation as inquiry. The final records and their related units are journals and diaries. These records are made retrospectively. They are not recorded live but rather capture the person's recalled knowl­ edge of the event recorded (Florio & Clark, 1982; Yinger & Clark, 198 1 ). These records permit longitudinal studies of events and people. As suggested in Table 6.3, records tend to be written, but no theoretical or conceptual reason exists that they could not be oral records (audiotape records). As with the other types of narrative records, the frame guiding the journal or dia­ ry writer influences what will be recorded and the observations depend on the perceptions and the recall of the writer/observer. Patterns of behavior or situations are extracted from the text, thus creating units inferred from the narrative records. Units that guide collections (e.g., who, what, when) are derived deduc­ tively from past work and from the framework guiding the re­ cording of information. As it is in the case of other narratives, units can also be derived inductively. Which units can be identi­ fied depends on the frame of the person writing the journal and the purpose for keeping it. In summary, narrative records record unfolding events in varying degrees of detail. They can be written on-line or live in situ or can be written after the event. The information is taken down in everyday language in a chronological manner. Variety of types of units can be derived from the data. They can be derived both inductively and deductively. Units include : natural units, deductive units, inductive units, behavior units, situations as units, phenomenogical units, action units, directly observable units, and inferred units. TECHNOLOGICAL RECORDS

As indicated in Table 6. 1, technological records include audio­ tape, videotape or videodisc records, and camera records. These instruments are open systems and tend to capture the widest segment of reality with little intervention of the observer. Once the observer sets the system in motion (e.g., audiotape or video­ tape), the events that occur in front of the camera lens or around the pick-up of the microphone are collected unselec­ tively. Records may be made of specific events or behaviors. Therefore, they would involve deductive units, behavior units,

180

CAROLYN M. EVERTSON AND JUDITH L. GREEN

and situational units that are set on an a priori basis. They are also potential sources of other types of units. Which units will be derived depends on the nature of the analysis used and the framework guiding the study. Technological records may be used in conjunction with all of the other systems presented previously. The technological re­ cord may be used in situ with the other systems or may be a record to wnich the other systems are applied. In general then, units described for the previous systems can be derived from technological records. The permanence of the record makes multiple analyses, approaches, and the identification of a wide variety of complementary units or variables possible (e.g., Morine-Dershimer, in press-a, in press-b; Ramirez, in press; Shuy, in press; Tenenberg, in press). SUMMARY

In the sections above the range or options available for describ­ ing phenomena were presented briefly. Each provides a differ­ ent interpretation and representation of reality, since reality can "only be apprehended indirectly via signs and mechanisms of representation" (Fassnacht, 1982, p. 64). As noted above, inter­ pretations and representations are the products of the concep­ tual framework guiding the study (Brophy, 1983; Delamont & Hamilton, 1976; Doyle, 1977; Sanders, 198 1 ; Shulman, 198 1 ). As Hamilton (n.d.) points out, "different frames of reference cre­ ate different patterns of explanation " (p. 8). Units from this per­ spective "are inventions, not discoveries; interpretations, not descriptions " (Hamilton, n.d., p. 5). Units can also be thought of as samples of different aspects of reality as well as different levels of reality. Units, therefore, are variables constructed to help the observer reflect on various aspects of observed phenomena.

Issues in Sampling The discussion of units of observation is also related to a discus­ sion of sampling issues because each unit represents a selected aspect of reality, not the whole of reality. In this section three additional ways of viewing sampling will be presented : (a) issues of what is sampled, when and where; (b) issues of when sampling decisions are made; and (c) time issues in sampling. When and Where. Berliner (1976) notes that " part of the answer is knowing when and where to observe" (p. 8). The requirement is to select an appropriate time and place to observe within the curriculum or stream of behavior. Selection of the wrong time and place wastes time and can invalidate findings. The problem is to select the time, place, and locus of observation that match the question under study. Herbert & Attridge (1975) identify five areas of decision making and sampling: (a) number of sub­ jects; (b) total length ohime of observation; (c) distribution of observation time; (d) likelihood of observations being represen­ tative of the phenomena under study; and (e) alternatives/op­ tions. Each of these decisions needs to be informed by the theoretical framework, past research, and/or a pilot or "orien­ tation to the setting" phase. As discussed in the following sec­ tion, without the latter, the researcher cannot insure the

representativeness of the sample. The issue of what is sampled and where is a part of the broader issues of validity. The issues above are also related to variability and stability of the behavior or phenomena under study, the environment in which the observation takes place, the characteristics of the subjects, and the nature of the inquiry cycle adopted. Problems may arise at all stages of an investigation and each decision may alter the validity of information obtained at other points in a study (Herbert & Attridge, 1975). A study conceived of this way is not based on a static, rigid approach, but rather on a dynamic one with a series of decision points, each of which is related to and influences what is collected and how it is collected. This view of sampling decisions leads to the second issue to be con­ sidered: when sampling decisions are made. (Additional factors related to sampling issues are presented by Linn, this volume and by Erickson, this volume.) When Sampling Decisions Are Made. The two factors presented below were extracted from the review of the different observa­ tional systems discussed previously. From this review two gen­ eral inquiry approaches with different sampling procedures were identified. These approaches can be thought of as differ­ ences in the inquiry cycle used by investigators. The first ap­ proach involves decisions made on an a priori basis. The second involves decisions made throughout the inquiry process. These approaches do not form a dichotomy; rather they can be con­ sidered as different ends of a continuum, illustrated in Figure 6.4, similar to the exclusive and inclusive context continuum presented earlier in this chapter. Investigators that make all decisions on an a priori basis se­ lect the units of observation, the locus of observation, and de­ termine the length of time and place of observation in advance. Issues of timing of observations and selection of events occur prior to beginning the study. Researchers who use this ap­ proach must be familiar with the setting to insure that what will be observed is representative of the phenomena to be studied. In addition, the researcher must ascertain whether or not the phenomena will occur in the context selected for study, and ifit does occur, whether or not it occurs with sufficient frequency to be reliably identified (Shavelson & Dempsey-Atwood, 1976). Without considering the setting and the frequency of occur­ rence of the events selected for observation, the researcher may encounter problems with the validity and stability of the find­ ings. For example, Wallat, Green, Conlin, & Haramis (1981) report the case of a teacher who was asked to record for 20 minutes a day the time spent in whole-group instruction. The teacher complied and recorded the time spent with children evaluating progress in the learning centers and directing chil­ dren to center work. This was the only period in which the teacher met with the total group. The researcher, who had never visited the class, concluded that the teacher was the most dog­ matic and authoritarian teacher in the study. Had he visited the classroom, he would have seen the teacher operate skillfully in A l l decisions Decisions are made are ma de on ------- t h roughout t he stud . y an a priori basis . Fig. 6.4. Sam p l i ng contin u u m .

OBSERVATION AS INQUIRY AND METHOD

an open structure setting. While this incident is an exception, it serves to highlight the need to be familiar with the setting and to represent accurately the nature of the phenomena to be stu­ died. The inquiry cycle for those studies at the far left end of the continuum is one in which the locus of observation and the timing of the observations are set in advance. After these are set, the researcher then gains access to the site(s) needed to carry out the study in a reliable, valid way, This approach, like all observational studies, requires that the observer gain access to the site and develop a level of trust and rapport with the partici­ pants. The observer needs to negotiate a contract with safe­ guards for the participants, thereby coming to an agreement with the participants that includes general human subjects pro­ tection and specifies how the information will be used as well as who will have access to the data and the results (e.g., Erickson & Wilson, 1982; Kimmel, 1981). This procedure helps to avoid situations such as those reported by Wallat et al. (1981), cited above. The research process or cycle at the far left is a multistep process in which the steps are taken to insure the validity of the decisions and choices in sampling. At the far right of the continuum, sampling issues are encoun­ tered at various points within the inquiry cycle. The researcher begins with a general question and selects the setting(s) for study. The researcher then seeks to gain access to the setting, to develop trust and rapport with the participants, and to nego­ tiate a contract to insure confidentiality and validity. In this type of inquiry cycle, general questions are asked, instruments are selected, collection procedures designed, data collected, and analyses begun. This process is then used to refine questions, select or add new collection procedures and loci of observation, and guide continued analyses. The cycle is reflexive; that is, it is not linear. Rather these stages tend to co-occur and are used to inform each other (e.g., Spradley, 1980). While there are many similarities between these two ap­ proaches, one of the major differences is the size of the general sample. The studies at the left end of the continuum tend to employ a larger number of cases than those at the far right. Investigators who engage in research at the far left are generally interested in testing hypotheses and in the identification of gen­ eral laws of behavior or norms of performance. Therefore, they must sample a large enough number of cases in order to generalize to other populations and settings. In contrast, those at the far right are generally interested in in-depth studies of a small number of cases. The purpose of these studies is to explore selected processes or settings in as much detail as possible, to generate hypotheses about the nature of these processes/ settings, and then to test what is occurring both within and across cases. Studies have recently been undertaken that also combine these approaches. These studies tend to be longitudinal or pro­ grammatic in nature. In such studies substudies are added, vari­ ous aspects explored, and questions generated and refined. Sampling decisions within such work vary with the component under exploration (e.g., Cole, Griffin, & Newman, 1979; see also various studies by Cook-Gumperz, Gumperz, & Simons, and Evertson et al. in the section of this chapter dealing with obser­ vation as inquiry). Another body of work that fits between two general approaches is work in which multiple studies are done

181

on the same body of data. This work is a form of secondary analysis in which the first study can be approached from either end of the continuum. The later studies are grounded in the earlier study, but may explore additional aspects of the phe­ nomenon. For example, Merritt (1982; Merritt & Humphrey, 1981) elected to explore how students obtained help from teachers and how teachers gave help in a variety of contexts. Merritt reanalyzed data collected under a broader ethno­ graphic study (Shuy & Griffin, 1981).

Time Sampling. Whether the researcher uses a normative ap­ proach, an in-depth approach, or a combination with a recur­ sive inquiry cycle, time sampling and event sampling are the two most frequently used ways of selectively recording informa­ tion. Time is used in at least three ways. First, time is used to specify the general boundaries of the observation period (e.g., the whole day, 1 hour, or 30 minutes} Second, time is used to designate the specified interval of recording of certain behaviors with an on-line category system (e.g., every 5 minutes; cf. Stallings, 1983). Third, small segments of time can be desig­ nated within a general period of observation. All occurrences of observed behaviors or events are then recorded using a check­ list or rating scale. In addition, rating scales can also register the intensity of a particular event as well as the frequency of occur­ rence. The first time-sampling method, the period of observation, in­ volves a series of decisions. The investigator must determine the length of a specific day's observations, the sequencing of obser­ vations, and the distribution of observations over time. This first time-sampling method, therefore, allows one to collect in­ formation over time and over many occasions. The periods of observation are kept stable. If the period of time is the same, depending upon the observational system, one can explore what kinds of events, behaviors, actions, demands, constraints, and processes occur in this time period. The nature of the peri­ od of observation will be determined by the question under study. For example, if the research question focuses on how school gets started, then the investigator must sample the be­ ginning of the day of the first days of school. If the question concerns effective reading instruction, then the researcher would sample both formal and informal reading events in class­ rooms in which instruction occurred (e.g., Griffin, 1977). If the question has to do with the quality of teacher-pupil academic contacts, the period of observation must encompass a represen­ tative sample of academic activities both over the day and over the week. Finally, if the goal is to make a statement about the nature of the process on a general level, then the investigator would have to sample events over time (e.g., throughout the year). Longitudinal samples are needed to answer the question of variability and stability of behaviors. The researcher must determine whether to sample a given period of time and ignore the boundaries of events or to sample within an event. The decision about this sampling issue will in­ fluence the type of system selected and the type of questions that can be explored. For example, some systems such as narra­ tive and descriptive systems and technological records will per­ mit ease of exploration of longitudinal questions. The category system or sign system, unless special adjustments and designa­ tions is made to note boundaries of events, are not as sensitive

182

CAROLYN M. EVERTSON AND JUDITH L. GREEN

to these issues. In addition, coding strategies (e.g., reducing ac­ tions or behaviors to a symbol) do not readily permit explora­ tion of detailed processes over time. The second time-sampling method, short-interval sampling, permits exploration of the duration of time a specific behavior occurred (e.g., 70 % of all talk was teacher talk; 20 % higher­ order questions were asked). It does not permit retrieval of the actual number of instances of a specific behavior. Therefore, we cannot tell whether a frequency code is a single instance of a behavior for 15 seconds or five instances each lasting 3 seconds. The distribution of specific behaviors and the shifts in distribu­ tion and influence are not retrievable (e.g., Bales & Strodtbeck, 1967). The second time-sampling method is related to the first approach in that the observer needs a length of time to record (e.g., 30 minutes), a specific point to begin (e.g., the beginning of school at 8: 30), and an interval in which to record the frequency of behaviors (e.g., every 3 or 5 seconds). The problem for the researcher is to determine how many intervals must be sampled to be representative and how long the observation period should be. Small segments of time sampled within the general period of observation through the use of checklists and rating scales per­ mit exploration of the actual occurrence of a specific behavior or event (e.g., 45 % or all behaviors observed were teacher-ini­ tiated questions; praise did or did not occur during the 10 min­ utes of observation of oral reading). This approach does not permit exploration of (a) the duration of a given behavior un­ less specifically indicated as an item (e.g, indicate how long attention to task lasted- 2 minutes, 3 minutes, 4 minutes, 5 minutes, or more), or (b) what preceded or followed a given behavior or event. In other words, a given behavior is decon­ textualized from the stream of behavior. The problem for the investigator is to determine the number of sampling sessions within the overall period of observation, the length of these sessions, and which instruments to use. Event Sampling. The final issue of sampling to be discussed is event sampling.. In contrast to time sampling, event sampling specifies a locus of observation (e.g., reading groups, whole class activities) but does not specify the length of observation. The length of the observation period is determined by the naturally occurring boundaries related to the event in the ongoing situa­ tion of the classroom. Event sampling is a topic-centered ap­ proach. For example, if one is interested in the nature of children's oral reading in middle grades, it is not necessary to record the beginning of school. However, it will be necessary to sample all events in which oral reading occurs (e.g., reading groups, social studies lessons, story reading, music, science les­ sons, menu reading, etc.). The researcher must select a locus of observation and then determine all possible situations in which the phenomenon occurs. Once this has been determined, events can be sampled in a representative manner. This discussion of sampling briefly illustrates the nature of the decisions that must be considered and the interrelatedness of many of these decisions. Sampling issues are related to issues of stability and validity. The way one samples within a given observation study is related to the questions asked and the framework guiding the observations. Finally, sampling is re­ lated to the degree of confidence that a reader can have in the

interpretations and findings of the study. The question is, does the sample provide a high degree of confidence that reality has been represented validly?

Representative Sources of Error: Limits on Certainty in Observation In previous sections, the general purpose of observation and generic factors involved in designing and implementing systematic observation were discussed. The goal of observa­ tional research was defined as the representation of specific seg­ ments of reality. The representational process was shown to be mediated by the mechanism or system selected as the means of representing reality. In this section, one last set of factors will be considered. These factors focus on the identification of sources of error and ways of reducing such errors. The notion of sources of error was selected for consideration since it underlies issues of reliability, validity, and rigor in ob­ servation and places limits on certainty as to what can be ob­ tained. Space does not permit in-depth discussion of each of these issues; therefore, a more general focus on the issue of sources of error was selected since reliability and validity have been addressed elsewhere in this volume. (See also Bracht & Glass, 1968; Cohen, 1960; Erickson, 1979; Flanders, 1967; Frick & Semmel, 1978; Genishi, 1983a; Heap, 1980a; Herbert & Attridge, 1975; Kugle, 1978; Light, 1971; McCutcheon, 1981; McDermott & Hood, 1982; Shavelson & Dempsey-Atwood, 1976; Snow, 1974, among others.) Reliability and validity will be considered within the general issue of sources of error. SOURCES OF ERROR: AN OVERVIEW

The perspective on observational research as a means of repre­ senting and exploring reality adopted in this chapter means that the cause of error can never be found in the segment of reality. Sources of error are found in the representational system or process (cf. Fassnacht, 1982). Table 6.5 presents a synthesis of representative sources of error drawn from several sources (e.g. Erickson, 1979; Fassnacht, 1982; Frick & Semmel, 1978; Herbert & Attridge, 1975; McGaw, Wardrop, & Bunda, 1972; Miller, 1974). This list, while not all-inclusive, represents the range of types of errors that can lead to false representations of reality and inva­ lid descriptions. Analysis of this list suggests two things. First, the different systems described previously are subject to differ­ ent types of errors. For example, issues of central tendency and leniency are applicable to categorical systems that base descrip­ tions on on-line ratings or codings. On the other hand, logical errors and errors in sampling are applicable to all systems. Second, the list of errors can be categorized. Errors are related to (a) observers, (b) the system for obtaining the representation, (c) the framework or assumptions about the phenomena under study, and (d) the procedures used to collect data. Fassnacht (1982) argues that errors are an inherent part of the perceptual process. This fact has led investigators to adopt a variety of measures in an effort to reduce the possibility of error, to insure rigor in observation, and to obtain an adequate repre­ sentation of reality. Two such measures will be considered

183

OBSERVATION AS INQUIRY AND METHOD

Table 6.5 Sources of Error in Observational Research Type of Error

Definition

1. Central tendency

When using rating scales, the observer tends toward the subjective midpoint when judging a series of stimuli.

2. Leniency or generosity

When using rating scales for which a "yes,""sometimes,""rarely," or"no" is required, the observer tends to be lenient or generous.

3. Primacy or recency effects

Observer's initial impressions have a distorting effect on later judg­ ments.

4. Logical errors

Observer makes judgment errors based on theoretical, experiential, or commitment-based assumptions (e.g., the assumption that be­ cause a teacher shows warmth to a class, she/he is also instruction­ ally effective).

5. Failure to acknowledge self

The influence of the observer on the setting is overlooked. The inves­ tigator's role may lead to the establishment of particular expecta­ tions. Judgments can be made in accordance with these expectations.

6. Classification of observations

Construction of macro categories loses fine distinctions. Such cate­ gories permit quantification, but lose information about the pro­ cess and fine-grained differences.

7. Generalization of unique behavior

Judgments may be based on evidence from an unrepresentative sample. Can lead to false conclusions or incorrect classifications of people or events.

8. Nested interests and values of observer

Findings become value-laden or otherwise distorted because of un­ checked personal bias.

9. Failure to consider perspective of the observed

For investigators interested in a clear picture of everyday life, failure to obtain participants' perspectives may leak to identification of unvalidated factors, processes, or variables (Erickson & Wilson, 1982).

10. Unrepresentative sampling

Errors may occur based on samples which do not represent the gen­ eral group of behaviors, that do not occur frequently enough to be observed reliably, or are inconsistent with the theory guiding the observations.

11. Reactions of the observed

Reactions of participants being observed can distort the process or phenomena being observed (e.g., teachers who are anxious about being observed may behave differently than they would at a calmer time).

12. Failure to account for situation or context

Leads to incorrect conclusions from assumptions of functional equiva­ lents (e.g., reading time 1 = reading time 2). Can lead to overlook­ ing what is being taught, changes in activities, variations in rights and obligations for participation, hence can distort conclusions.

13. Poorly designed observation systems

Leads to problems with reliability and validity.

14. Lack of consideration for the speed of relevant action

Errors may occur based on the omission of crucial features because of the rapidity of actions in the classroom.

15. Lack of consideration for the simultaneity of relevant action

Errors may occur based on failure to account for more than one activ­ ity occurring at a time; more than one message being sent at a time (e.g., use of different channels-verbal and nonverbal); and more than one function of a message at a time.

16. Lack of consideration of goal-directed or purposive nature of human activity.

False conclusion that a behavior lacks stability because of failure to consider the purposes of human behavior.

17. Failure to insure against observer drift

Errors caused by changes in the way the observer uses a system as time goes on. Can lead to obtaining descriptions that do not match the original categories or that vary from each other (Kugle, 1978).

184

CAROLYN M. EVERTSON AND JUDITH L. GREEN

below: (a) measures of reliability and (b) proposed sets of criteria for various aspects of the observational process. RELIABILITY: A SOURCE OF POTENTIAL ERROR

Issues of reliability are complex and are tied to issues of validity (see Erickson's chapter, this volume for a discussion of validity). Herbert & Attridge (1975) define the relationship as follows: Generically validity refers to the degree to which the measures ob­ tained by an instrument actually describe what they purport to de­ scribe; reliability refers to the accuracy and consistency with which they do so. (p. 6)

On a general level, reliability is involved with the reduction of possible sources of error, as will be shown in this section.

Given the diversity of systems, no single approach to reliabil­ ity assessment is adequate. Rather a variety of issues must be considered. In Table 6.6, a set of questions extracted from the work of Frick & Semmel (1978) and others is presented to help clarify the issues in reliability. As indicated in Table 6.6, different systems require different approaches to reliability. The two most general approaches were defined by Frick and Semmel (1978). They suggest that when considering reliability in observational research, it is im­ portant to make conceptual distinctions between two indices of reliability: reliability coefficients (score reliability) and observer agreement coefficients (percentage of agreement). Both of these indices are related to a quantitative approach to reliability and require the quantification of data before they can be deter­ mined. A discussion of the statistical treatment of each of these measures is beyond the scope of this discussion (see, for

Table 6.6. Questions and Issues in Reliability Questions When should observer agreement be measured?

Related Issues 1. Prior to data collection (Frick & Semmel, 1978). 2. Training does not guarantee against observer skill deterioration as data collection proceeds (Frick & Semmel, 1978; Kugle, 1978). 3. Calculations of degree to which observer disagreement limits re­ liability should be done after the study.

On what kinds of data should observer agreement be calculated?

1. Agreement should be computed on the same unit(s) of behavior that will be used in the data analysis. 2. Agreement should be computed on subcategories of behavior as well as the larger, subsuming categories.

With whom should agreement be obtained?

1. High interobserver agreement may not mean agref!ment with the original categories, because systematic misinterpretation can exist even with high agreement. 2. Observers' scores should also be compared with a criterion. This is known as criterion-related agreement.

Under what conditions should agreement be calculated?

1. Coding in the setting may differ from coding of unambiguous sam­ ples in a laboratory or training session (also see Johnson & Bolstad, 1973). 2. Ways to heighten observer vigilance and maintain accountability should be considered.

How can agreement be measured?

1. Intraclass correlation coefficients Useful after a study is completed, but impractical during or before. Highly affected by the variance between subjects. 2. Simple percentage agreement. Drawbacks are that low frequencies in some categories and high frequencies in others may make interpretations ambiguous. Does not account for false inflation due to chance agreement. (See Co­ hen, 1960; Flanders, 1967; Light, 1971 ; Scott, 1955 for examples of ways to compute agreement coefficients.)

Which agreement coefficient is appropriate?

1. Dependent upon the type of observation system, number of cate­ gories, type of data, unit(s) of analysis, and purpose. 2. If nominal comparisons cannot be obtained, then marginal agree­ ment methods should be used. 3. If there are only a few categories and/or frequency distributions are unequal, then correction for chance agreement should be made. 4. The definition of" items" changes with systems. The probability of occurrence of the items must be considered.

OBSERVATION AS INQUIRY AND METHOD

example, Cohen, 1960; Flanders, 1967; Light, 1971; McGaw et al., 1972); however, issues that affect the determination of this type of reliability are presented instead (e.g., observer drift, training, units used for reliability, frequency of occurrence of units). Consideration of the points raised in Table 6.6 can help eliminate or avoid errors in determination of both reliability coefficients and observer agreement coefficients. Erickson (1979) proposes another type of reliability, reliabil­ ity for descriptive studies. The first issue relates to the value of description for classroom research. He argues that the value of description does not lie essentially in its"richness," nor in its capacity to evoke in us a vivid sense of"being there." Nor does the"discovery of new independent and dependent variables" in itself get us anywhere, nec­ essarily. Before classroom research can proceed further, researchers need languages of description at the level of primary data collection which make contact with the theories of action that are being used in moment-to-moment decision making by participants in the events they observe and describe. (p. 4) Erickson's first concern, therefore, relates to the interrela­ tionship of description, theory, and language. This concern echoes a similar concern voiced by Popper (1963, as cited in Stubbs, Robinson & Twite, 1979; see p. 164 of this chapter). Popper suggests that all observation "and its description pre­ supposes a descriptive language with property words; it presup­ poses similarity and classification which in its turn presupposes interest, points of view, and problems" (p. 21). For Erickson and Popper, therefore, issues of reliability are related to obtain­ ing adequate description as well as validity. Erickson elaborates the issues of reliability. He suggests that "descriptive reliability is indeed desirable in the interest of showing plausibility in judgment, even though . . . descriptive validity is the logically antecedent problem in the construction of an adequate descriptive language" (p. 19). He notes two ways that reliability can be obtained for descriptive data. Since this area has received little or no treatment in the research field in general, this issue will be treated in greater detail than the other types of reliability discussed above. First, when audiovisual records are made, the researcher can use an "instant replay" to demonstrate reliability in descrip­ tion. This approach, according to Erickson (1979), is analogous to the instant replay by a sports announcer on television. The instant replay can be thought of as a"folk" means of demon­ strating validity and reliability in description. In classroom re­ search, the external memory of an audiovisual record, together with the running notes obtained as a participant observer, provide the researcher with the same evidentiary resource the television football announcer has in the instant replay. The audiovisual record and replay capability provide the qualitative researcher an opportunity to make the data base public. (p. 19) This evidence, he suggests, can be shared with teachers and other participants working in partnership with the researcher. This approach is a form of triangulation of data or perspectives (also see Adelman & Walker, 1975 and Elliot?, 1976). Second, primary evidence can be shared with other re­ searchers or with other members of the community (see Bloome, 1981 for an example of the use of a community

185

advisory board to insure reliability and validity of data and inferences about data). With regard to this approach, Erickson (1979) states: In the process of this sharing, descriptive research becomes no longer a matter of strictly private judgment and opinion. The epi­ thets"mere journalism" or"fiction" may still be flung, but they can no longer stick, for the grounds of evidence can be made clear in the arena of public knowledge. The claims made in descriptive state­ ments can be disconfirrned, at least at the level of inference involved in the language of primary data collection. The propositions in­ volved in descriptive accounts are no longer incorrigible ones (and the incorrigibility of narrative description as reported in field stud­ ies has been a serious problem up to now, especially for educational researchers trained in a tradition of positivism in scientific research). The capacity for instant replay opens up the possibility of interac­ tional research which is not positivistic, yet is still rigorously empiri­ cal. (pp. 19-20) This discussion highlights a wide range of issues that must be considered when determining the reliability of a study. In addi­ tion, reliability was shown to be a possible source of error. That is, because a reliability coefficient is presented does not mean that the information is valid, that the coefficient was determined in an appropriate manner, or that the representation of reality is accurate. The consumer of observational research must go beyond the score and ask how reliability was determined and explore the relationship between reliability and validity, since it is possible to measure behaviors reliably that have low validity with regard to the question under study or the segment of re­ ality observed. For example, one could conceivably construct an instrument to measure intelligence by having children throw stones as far as they could. It would be possible to obtain a high correlation between how far stones were thrown on one occa­ sion and how far they were thrown on a second occasion, but this would hardly constitute a valid measure of intelligence. The foregoing discussion of reliability can be thought of as a framework or set of guidelines for finding appropriate ways to determine reliability. That is, the questions and the related issues that were raised can guide the researcher in designing, selecting, and applying appropriate measures of reliability. For additional discussion of these and other issues (e.g., internal and external validity), see Le Compte and Goetz (1982); Hansen (1979); and Pelto and Pelto (1977), among others. TOWARD CRITERIA FOR ADEQUATE OBSERVATION

As in the case of reliability, no single set of criteria exist to date to guide observation in general. Rather specific criteria have been proposed for different types of systems. The purpose of such criteria is to insure the rigor associated with a particular approach and to provide a framework for informed decision making for design and implementation of an observational study. Regardless of the approach, the investigator must answer a general set of questions. The questions include: who, what, when, where, and how to observe as well as why to observe. While these questions appear obvious, what is not so obvious is how they are interrelated and how they are influenced by the

186

CAROLYN M. EVERTSON AND JUDITH L. GREEN

questions under study. Additionally, what is not evident in looking at these questions alone are the ways in which the theoretical (philosophical, experience-based, or commitment­ based) perspective influences the questions asked, the data col­ lected, the level(s) of analysis, and the descriptions obtained. To illustrate this relationship, the work of two researchers who have explored this issue will be considered (Genishi, 1983b; Hymes, 1977). A graphic outline of these issues is provided by Genishi derived from her work in exploring verbal interactions in the classroom (Genishi & Di Paolo, 1982). Figure 6.5 shows the relationship among various aspects of the research process as conceptualized by Genishi ( 1983b). As indicated in Figure 6.5, the questions asked affect the methods of data collection, the units of analysis, and the types of coding systems used as well as the level of transcription. This conceptualization provides a framework for making decisions about what to do and how to do it. While designed for analysis of verbal interaction in classrooms from a sociolinguistic, psy­ cholinguistic, or child language framework, the relationships among parts and the decisions to be made are the same as for other disciplines. This framework, therefore, is a general one to guide decision making in observational studies. Hymes (1977) demonstrates how the area of concern in­ fluences what questions will be asked and, therefore, what data will be collected and how those data will be analyzed. Writing for the research approach known as linguistic ethnography, he states that: If one begins with social life, then the linguistic aspects of ethnography require one to ask : What are the communicative means, verbal and other, hy which this bit of social life is conducted and interpreted? What is their mode of organization from the standpoint of verbal repertoires or codes? Can one speak of appropriate and inappropriate, better or worse, uses of these means? How are the skills entailed by the means acquired, and to whom are they accessible? (p. 93)

In contrast, if one starts from language in one's study, the eth­ nography of linguistic work requires one to ask another set of questions: Who employs these verbal means, to what ends, when, where, and how? What organization do they have from the standpoint of patterns of social life? (p. 93)

While stated for linguistic approaches, the Hymes (1977) example suggests that a change in lenses leads to a shift in the focus of the questions or to another set of questions. The key is to match the questions and the behaviors observed. This is true whether the approach is behavioral, phenomenological, or eco­ logical, and so forth. The Genishi ( 1983b) and Hymes ( 1977) examples point out that one type of criteria that must be considered is the match between the focus, the research questions, the data collection, the manner of displaying data, and the analysis. In other words, they demonstrate the need to consider the internal as well as the external validity of the study. Another factor they point to is the

Premise: Data Collection and Analysis Involve Decisions About: RESEARCH QUESTIONS What are children accom plishi ng through verbal interaction?* What kin ds of arguments do childre n have? How do teachers give instructions for student seatwo rk? How do children take conversational turns during dramatic play? What intonation patterns are associ ated with student engag ement d u ring science lesson s ? Determine LEVEL OF

ANALYSIS

Affects Methods of Data Collection : Post h oc notetaking* Speci men description (with paper and pencil) Audiotaping Videotaping Affects Coding System Selected : Them es* Frames Narrati ves Categ ories Determi nes Unit of Analysis: Chunks of discourse* (e.g., speech events) Episo des Utterances/propositions/speech acts Phrases Words Phone mes Determines TRANSCRIPTION Degree of detail in transcription of data should suit the selected level of analysis. Fig. 6.5. Studying classroom verbal i nteraction. Adapted from Genishi (1 983b). * Examples are arranged from most global to most h ighly focused.

OBSERVATION AS INQUIRY AND METHOD

need to consider the level of analysis that is appropriate for the question under study. For example, if a researcher is concerned with identifying the relationship between classroom manage­ ment procedures and student achievement (cf. Evertson and her colleagues), then detailed analyses of teacher-student talk may not be an appropriate beginning point. This level may be ap­ propriate at a later point (e.g., Evertson, 1984; Green & Weade, 1984) when a contrast between effective and less effective class­ room managers is desired in order to explore differences in models of management and management of lesson content. (For other examples see Barr & Dreeban, 1983 ; Burstein, 1980; Green et al., in press; Morine-Dershimer, in press-a, in press-b.) The issue of level of analysis has implications for factors such as sampling, units of observation, and the nature of the systems selected. These issues are also related to factors involved in secondary analyses (e.g., Koehler, in press; Rentel, in press). The discussion to this point has dealt with general issues. In the remainder of this section, two specific sets of criteria will be presented. The first set by Herbert and Attridge (1975) are aimed at researchers whose approaches require the use of cate­ gorical systems, systems to the far left of the context continuum proposed in Figure 6.3. The second set by Spindler (1982) is aimed at investigators whose approaches require an ethno­ graphic or more descriptive approach, systems that fall to the right of the context continuum. While contrasting on one level, these two sets of criteria can serve as a point of departure for researchers regardless of orientation. Each set of criteria is de­ signed to insure that the observation will be systematic, that information will be obtained in a reliable and valid manner, that information will be easily transmitted to others, and that others can reconstruct the research process. Herbert and Attridge (1975) have identified a series of criteria for use in designing and implementing categorical systems. In addition to the design issues, Herbert and Attridge address the process of training and coding manual development. Table 6.7 presents a summary of the criteria by area. While this set of criteria was designed for use with categorical systems, many of the items are applicable to all types of obser­ vational studies. That is, many of the items in the validity sec­ tion apply across approaches. For example, in all studies, terms should be clearly defined, the basis for the terms (theory or other bases) should be specified and used consistently, items should represent dimensions being studied, and ground rules for implementation and categorization for all units should be specified. All of the items in the section on inference can be applied to other systems. In contrast, the two items in the con­ text section will be specific to approaches at the context-exclu­ sive end of the continuum in Figure 6.3. These criteria can be viewed as points of departure rather than as a set of rules that are set in stone. Their goal is to assure rigor and internal and external validity of studies in addition to replicability. The criteria or characteristics proposed by Spinder (1982) apply to studies at the context-inclusive end of the continuum. These criteria help to define anthroethnography (ethnography based on anthropological perspectives applied to the study of education). Spindler's criteria, therefore, serve both to define this approach and to clarify the relationship between perspec­ tive and data collection. Table 6.8 provides a summary of these characteristics/criteria.

187

The criteria above will guide data collection and help to spec­ ify the perspective guiding the approach. They also indicate the nature of the inquiry cycle. That is, they suggest that the obser­ vation has a specific theoretical perspective (transcultural socio­ cultural participant oriented); it is goal driven (to make explicit what is implicit in sociocultural knowledge generated in the set­ ting); questions and hypotheses are generated within the observa­ tion/ethnographic cycle- inquiry is dynamic (instruments, codes, observation schedules, etc., are selected as needed within the study for the specific questions raised during the study), and technical devices are used in addition to other instruments, codes, and so forth. The discussion above is illustrative rather than definitive. Different approaches will require different types of criteria. The fact that the majority of criteria presented in this section focus on linguistic or ethnographic approaches stems from the fact that these approaches are relatively recent to the study of the classroom and educational processes. Researchers who have used or developed these approaches, as with any new direction, have had to specify criteria for other investigators not grounded in the approaches. These criteria, however, raise a basic issue: What are the criteria for each approach? In order to examine observational research approaches, to develop a theory or theories of observation for research purposes, and to insure sys­ tematic observation, researchers need to address the issue of what makes observation systematic both within a specific ap­ proach, study, or program of research as well as across ap­ proaches. In this way, the consumer will be able to understand the segment of reality each addresses best. Additionally, such criteria will help researchers show how a particular method or instrument was used and adapted. In the sections above, the argument was made that a method or tool could be described independently from the framework of its selection and the units of observation. The criteria pre­ sented confirm this but also suggest that within a specific ap­ proach a given tool (e.g. narrative, interview, technological record) will be used in specific ways; it will be adapted to the needs of the study. For example, stimulated recall was used to explore teachers' perceptions in one study by Morine and Val­ lance (1975) and adapted to collect both teacher and student perspectives in a later study by Morine-Dershimer and Tenen­ berg, ( 1981). Stimulated recall is a tool; the framework guiding it determines how it will be used and what information it will be used to collect.

Summary The discussions above were designed to raise generic questions about the nature of observation and observational processes. The discussion began with the consideration of the nature of observation as an everyday process as well as a research- or question-directed process. Factors involved in using observa­ tion as a research approach were discussed: context of observa­ tion, systems for recording and storing observations, units of observation, sampling, and sources of error. Observation was shown to be a process mediated by a variety of factors. Obser­ vational research was shown to be a multifaceted process, a process in which different goals require different ways of recording and storing data. Finally, observation as a research

188

CAROLYN M. EVERTSON AND JUDITH L. GREEN

Table 6.7. Criteria for Observation Systems and Manuals Area

Criteria

Subcategory

1. I DENTI FYI NG : Enabling selection of appropriate instrument

1.1 1.2 1.3 1.4 1.5

2. VALIDITY: Specifying the provisions of evidence that allow both developer and user to decide if the instrument represents the events it claims to

ITEM CHARACTERISTICS

2.1 2.2 2.3 2.4 2.5 2.6

I N FERENCE

2.7 2.8 2.9

2.10 CONTEXT

2.11 2.12

3. PRACTICALITY : Concerning the ease of implementation, its acceptability to those under study, complexity of data gathering mechanisms, training procedures, etc.

OBSERVER EFFECT

2.13

RELIABILITY

2.14

VALI DITY

2.15

I NSTRUMENT ITEMS

3.1 3.2 3.3

OBSERVERS

COLLECTION AND RECORDING OF DATA

Name of instrument should identify focus and pur­ pose. I nstrument should be accompanied by a statement of purpose. Behaviors, subjects, and content should be specified. Intended applications should be stated. Situations in which the instrument should not be used should be specified. All terms should be clearly defined. If derived from theory, terms should be defined con­ sistently with their use in the theory. Items should be exhaustive of the dimension (s) of be­ havior under study. Items should be representative of the dimensions of behavior under study. Items should be mutually exclusive. Ground rules for implementation and for categoriza­ tion of borderline/unusual behaviors should be speci­ fied. Items should be as low in degree of observer infer­ ence as the complexity of the behavior(s) will permit. The degree to which observer inference is present and controlfed should be explicated. Guidelines as to the kinds of inferences that can and should be made should accompany the system to re­ duce unwarranted assumptions/applications of the findings. Statistical and other methods of inferential treatment which are recommended for use should be specified. The problems of context must be recognized and the degree and kind of context brought to bear in the instrument must be explicated. Methods of reducing/controlling use of context by ob­ servers must be explicated. The effect of observers, equipment, and procedures on the observational setting should be explicated. The types of reliability assessed, meaning, and condi­ tions under which they were determined should be recorded. Instruments should be accompanied by methods used to test their validity, results obtained, and purpose for which they apply. Items comprising the instrument should be relevant to its purposes. Codes should be simple and easy to make. Categories and codes should be easily learned.

3.4 3.5

Special qualifications of observers should be noted. Training procedures, steps, durations, and results should accompany instruments. Manuals, tapes, and training devices should be available. 3.6 Manuals should recommend number, location, and functions of observers, coders, technicians, and staff needed. 3.7 Data collection and recording procedures must accompany the instrument. 3.8 Observation unit recommended should be specified (e.g., length of time). 3.9 Coding unit recommended should be specified. 3.10 Procedures for analyzing data should be described. 3.11 Recommendations for data transmission and display techniques for the instruments should be described. 3.12 Costs likely to be incurred in use of instrument should be noted.

Note. From "A Guide for Developers and Users of Observation Systems and Manuals" by J. Herbert and C. Attridge, 1975, American Education Research Journal, 12(1), pp. 1-20. Copyright by American Educational Research Association. Reprinted by permission.

OBSERVATION AS INQUIRY AND METHOD

Table 6.8. Criteria for Adequate Ethnography

Guidelines 1 . Observations are contextualized. 2. Hypotheses and q uestions emerge as the study proceeeds in the setting selected for observation. 3. Observation takes place over a long period and is repetitive (1 year appears to be minimum). 4. The native view of reality is brought out by inferences from ob­ servation and by various forms of ethnographic inquiry. 5. A major part of the ethnographic task is to understand what so­ ciocultural knowledge participants bring to and generate in the social setting being studied. 6. Instruments, codes, schedules, questionnaires, agenda for i nter­ views, and so forth are generated in the field as a result of obser­ vations and ethnographic inquiry. 7. A transcu ltural perspective is present, though frequently as an unstated assumption. 8. The task is to make explicit what is implicit and tacit to inform­ ants and participants in the social setting being studied. 9. Inquiry and observation m ust d isturb as little as possible. 1 0. The conversational management of the i nterview or eliciting in­ teraction must be carried out so as to promote the unfolding of emic1 cultural information in its most heu ristic, natural form. 1 1 . Any form of technical device that will enable the ethnographer to col lect more live data-i mmediate, natural, detailed be­ havior-will be used, such as camera, audiotapes, and videotapes. Nole. Adapted from Doing che Ethnography of Schooling by G. Spindler. Copyright © 1982 by CBS College Publishing. Reprinted by permission.

By "emic" is meant the view from within the culture, the folk view, in terms of native categories. 1

approach was shown to be a systematic process guided by the frame of reference of the researcher. In the last section of this chapter, the question of observation as an inquiry process will be considered. Since the ways in which an observational study will be designed and implemented depend on the questions under consideration and the frame of reference guiding the study, no prescription for approaches is possible. Therefore, in the following section three different re­ search approaches will be highlighted to demonstrate how frame of reference combines with methods/tools to create unique observational studies. Finally, in this section, a set of steps constructed by a group of young researchers attempting to design and implement their first observational studies will be described. This section, while it closes the chapter, will actually constitute a beginning. Therefore, in the section that follows, work will be presented that represents both advanced programs of research and beginning points for new researchers.

Observation as Inquiry In the previous sections, the discussion of observation focused on methods and factors involved in the design and implementa­ tion of observational research. The discussion was generic in nature; that is, no consideration was given to observation used in specific studies or to the design and implementation of such studies. In this section, observation as a process of inquiry will be explored and examples of ways this process can be realized will be presented. The purpose of this section is to help the reader begin to make plans for conducting observational stud-

189

ies. To facilitate this process a series of concrete examples are presented. The conceptualization of observation as an inquiry process, that is, as a systematic and appropriate approach to research on educational processes, events, and issues in educational set­ tings, involves more than simply using a particular system. The design and implementation of an observational study requires the researcher to make a series of systematic decisions about who, what, when, and where to observe, in addition to answer­ ing the question of how. In addition, the researcher selecting observation as the principal way of collecting information about processes, events, and issues must be concerned about sampling, representativeness, and systematicity. These issues must be considered in all types of studies that use observation. In other words, whether one uses observation as a way of cap­ turing and representing reality in, say, a statistical, hypothesis­ testing study, a descriptive study, a correlational study, or an ethnographic study, the decisions will be similar. What will differ are the ways in which the studies will be designed, the theoretical grounding, the instruments used to sample and re­ present reality, the units of observation, the locus of observa­ tion, the sample size, the orchestration of collection procedures, and the sampling procedures. The following discussion by Corsaro (1985) provides a con­ crete illustration of some of the major aspects of the decision­ making process. The issues raised in this example will be ap­ plied to the representative examples of observational research presented in the remainder of this chapter. Therefore, this ex­ ample serves as a general introduction to this inquiry section. In this example, factors discussed in the first section are high­ lighted. Earlier I defined interactive episode as the collection or sampling unit. Actual sampling procedures involved decisions regarding which units (interactive episodes) to sample from the continuous flow of interaction in the nursery school. The interactive episodes recorded in field notes and on videotape were representative of typi­ cal activities in the setting and had potential for the development of theoretical propositions. In short, sampling decisions were both rep­ resentative an d theoretical.

When a sample is representative it captures the overall texture of the setting under study. Since it is impossible to record all interac­ tion in a given setting, field researchers often attempt to insure rep­ resentativeness by collecting data across several dimensions including people, places, time and activities (cf., Denzin, 1977; Me­ han, 1979a; Reiss, 1971 ; and Schatzman & Strauss, 1973). A typical strategy is to draw up an inventory of the range and number of elements in each of the dimensions in early phases of field entry and then modify the inventory as needed at various points in data collec­ tion. When using more than one data collection technique (e.g., ob­ servational field notes, interviewing, audiovisual recording, etc.) data obtained at one point in time with one technique can serve as a basis for identifying dimensions in the setting which are then sampled using another technique later in the collection process. I used data collected during the period of concealed observation to develop an inventory of the participants, activities, social-ecolog­ ical areas and schedule of the nursery school. The inventory then guided the sampling of interactive episodes in participant observa­ tion during the next three months. At the end of this period, I re­ viewed the field notes, and added and deleted items to and from the inventory. The revised inventory was then used to guide sampling

190

CAROLYN M. EVERTSON AND JUDITH L. GREEN

decisions in later participant observation and for the audiovisual recording of interactive episodes. Once the videotaping phase of the research began, I checked the representativeness of the sample of audiovisual data in two ways. First, after each taping session, I reviewed and summarized the data. In the summaries, I specified the number of episodes, where and when they occurred, their duration, the participants, and the nature of the activity. At the end of each week, I compared the summaries with the sampling inventory and checked off items (features of be­ havior recorded in the episodes) as they were collected. Secondly, on several occasions, I asked the teachers to look over the summaries and to respond to their representativeness of typical activities in the school. The teachers' responses, as well as my comparison of the audiovisual data with field notes, resulted in the collection of addi­ tional videotaped episodes which improved the overall representa­ tiveness of the recordings. . . . As I have pointed out, sampling procedures in field research should be both reliable and representative. But the nature of ethno­ graphic research demands that sampling procedures also be reactive to developments in the course of research. As Hymes notes, "for many ethnographers, it is of the essence of the method that it is a dialectical, or feed-back (interactive-reactive) method. It is of the essence of the method that initial questions may change during the course of inquiry" (Hymes, 1978: 8). Thus, sampling procedures must, in part, reflect the dialectical nature of ethnographic research. It is precisely along these lines that ethnographic research differs from hypothesis-testing approaches. It is not that field research is more theoretically oriented than hypothesis-testing approaches (i.e., surveys, experiments, etc.), but rather field research differs in that theoretical concerns guide sampling decisions both initially and throughout the study. The ethnographer must be sensitive to theoret­ ical leads as they emerge, and pursue them through theoretically directed sampling procedures. (in press, pp. 32-34) Corsaro has identified several issues that can be used to un­ derstand inquiry. First, decision points for sampling, instru­ mentation, and question generation and refinement may occur at different points. In ethnographic studies, these decisions occur prior to and throughout the study. In individual studies from other approaches (e.g., descriptive, sociolinguistic, narra­ tive) this pattern of decision making can also be identified. This pattern can also be seen when programs of research are consid­ ered. That is, findings and procedures in one study may be used to inform those in another or as the basis for modification in later studies that are linked together to explore a question or set of equations systematically. The second issue raised by Corsaro is the question of the history of a study. In the ethnographic study, decisions had both local histories, histories within the study, and distant histories, histories of research approaches, questions, and findings from past work. The history of a single study must also be considered. In programmatic research, the history of a given study is the sum of preceding work as well as the theoretical grounding. For example, to understand why a particular instrument or data collection procedure was used may require knowledge of earlier work within the program. The distinctions made by Corsaro about the differences in decision points for sampling and design in different types of studies suggests the need to extend the earlier discussions on this and related issues (see Figures 6.2 and 6.4). The issue is not simply when decisions are made within a single study but when and how decisions are made both within and across studies. When decision making is viewed from this broader perspective,

the varied approaches appear more similar than different. The key is to make decisions in systematic and principled ways, and to modify procedures based on questions, local context, and knowledge gained from past work (work either at an earlier point in the study or in an earlier study). In considering the types of studies to include as examples of the inquiry process these distinctions were considered. The studies that were selected as illustrative examples had to meet several criteria. First, they had to represent different observa­ tional approaches to the study of teaching-learning processes in educational settings. Second, they had to represent program­ matic research. Third, they had to represent different theoretical orientations. Fourth, they had to represent different research (e.g., experimental, descriptive, enthnographic, etc.). Fifth, they had to use multiple perspectives and multiple approaches to represent reality within a study; and sixth, they had to represent different types of decision making (e.g., initial cycles of decision making across studies, cycles of decision making within studies, and combinations of these approaches). Four sets of studies were identified and serve as illustrations: work on classroom management and organization by Evertson (Peabody College, Vanderbilt University) and her colleagues, Jere Brophy and Linda Anderson (Michigan State University); Edmund Emmer et al. (Research and Development Center for Teacher Education, The University of Texas- Austin); work by Marshall and Weinstein on classroom processes and student perceptions (University of California - Berkeley); work on par­ ticipant perspectives of classroom discourse by Morine-Der­ shimer (Syracuse University) and Tenenberg (California State University - Hayward) and their colleagues in a number of different institutions (e.g., Shuy, Georgetown; Ramirez, State University of New York); and work by Cook-Gumperz, Gumperz, and Simons and their colleagues (Michaels, Harvard; Collins, Temple; Murphy, University of California ­ Berkeley) on discourse processes within and across educational settings in the School/Home Ethnography Project at the University of California, Berkeley. Each of these bodies of work will be explored; each serves to illustrate a variety of different aspects of the decision making and inquiry process involved in using observation as the pri­ mary means of representing reality. As a whole, the four sets of studies represent a range of ways of orchestrating a limited set of tools and procedures to match the question under study and to obtain representations of specific aspects of the reality of life in classrooms and other educational settings. These studies highlight the design and implementation pro­ cess and help to specify some of the aspects of the research pro­ cess related to observation as inquiry. In addition, these studies represent extensively funded projects (all were funded by federal agencies USOE, NIE, & NIMH). They do not address the ques­ tion of how to begin or what an individual researcher might do. Therefore, the final set of questions to be addressed in this sec­ tion will be those relating to a general framework for conduct­ ing observational research. This framework was derived from consideration of the issues raised in this chapter and from work by a group of novice researchers at the University of Delaware and Ohio State University. These junior researchers explored both the steps involved in doing observational research and the ways to conceptualize these processes graphically.

OBSERVATION AS INQUIRY AND METHOD

Example 1 : The Descriptive­ Correlational-Experimental Cycle One way to view the work of Evertson and her colleagues across the various studies is to consider this set of studies as representing the descriptive-correlational-experimental cycle suggested by Rosenshine and Furst (1973). Figure 6.6 provides a brief graphic representation of this 15-year program. (For more detailed information about each study, the reader is re­ ferred to the citations provided.) While the cycles are presented linearly, each cycle itself is actually composed of two descrip­ tive-correlational-experimental loops and a series of interac­ tive-reactive decisions across studies (cf. Corsaro, 1985). These loops overlap; that is, they share studies in common and the decisions within Loop 1 influenced decisions in Loop 2. In addi­ tion, the questions raised in one study are explored in subse­ quent studies; methods used in a study are often modified and extended in studies that follow ; and findings in one study are tested and/or explored in later studies at different grade levels. Therefore, as the researchers moved from one study to the next, they interacted with data, procedures, and questions, and reacted to the problems, issues and findings. This interactive­ reactive process led to modifications in subsequent studies (e.g., instruments were modified, collection procedures added, and hypotheses generated and tested). Background of the Studies. The cycle began as a response to

several factors. First, the Coleman Report (Coleman et al., 1966) and later reanalyses of these data by Jencks et al. (1972) minimized the contributions of teachers to student academic achievement. Aside from statistical and methodological flaws which masked rather than revealed teacher effects, the studies contributed to the general attitudes that teacher effects on their students were less important than other factors such as socio­ economic status. The Coleman study did not include systematic observation of teacher behaviors, and findings were based on school measures, data available from school records, and aver­ ages across classrooms. In other words, individual teachers' contributions to student achievement were not considered. Sec­ ond, relationships between teachers' classroom behaviors and student achievement had not been carefully explored. What ap­ peared to be needed were systematic studies of teachers teach­ ing in natural settings. Third, simultaneous with these research issues was the trend toward developing curricula that were "teacher-proof," that is, designed to convey the content regard­ less of the skill of the teacher. Fourth, earlier studies of teachers' classroom behavior showed a lack of stability from one year to the next (Rosenshine, 1970). This lack of stability appeared to be a major obstacle in developing a sound data base that related teacher behavior to student outcomes. The response to the issues discussed above was the design and development of the Texas Teacher Effectiveness Project (TTEP) starting in 1970 (Brophy, 1973; Good, Biddle, & Bro­ phy, 1975; Veldman & Brophy, 1974). The intent of the series of studies described in Loop 1 was to build a data base by con­ ducting large-scale field studies of teachers' classroom behavior and relating these characteristics and processes to measures of student learning gain. In addition some of the methodological problems were addressed by developing new methods of obser-

191

vation and by identifying a stable group of teachers who had consistent effects on their students' achievement across years. This brief discussion of the background of this cycle of stud­ ies demonstrates the need to consider the historical context in which the studies were developed. In order to understand a given study, it may be necessary to explore what was occurring historically in education and the broader society that influenced the development of a study. This information also serves as the framework for a given study or set of studies. The historical perspective will also become clear as the dif­ ferent studies in the cycle are considered. This program can be divided into a series of loops, or related studies. Therefore, to understand what was done and why, it is important to trace the links between and across studies. Another way to think about an individual study is to consider it within a particular research loop that is grounded in a program of research. From this per­ spective, each study has a history and a place in history. To understand the findings and observational approaches in one study and to utilize these findings or approaches, it may be nec­ essary to consider the study in its historical perspective (re­ search history) as well as in the historical perspective of the times. LOOP I : RESEARCH ISSUES AND DESIGN OF LATER INQUIRY

Space does not permit adequate discussion of each study; there­ fore the focus of the discussion of this set of studies and the remainder of the studies in this and subsequent sections will focus on the links between studies and factors contributing to decision making within a program of research. Loop 1 consists of the Texas Teacher Effectiveness Study (TTEP) and the First Grade Reading Group Study (FGRGS). The TTEP was the first descriptive-correlational study; the teachers observed were those identified as consistent in their effects on student achieve­ ment over a 3-4-year period. As indicated in both Figure 6.6 and Table 6.9, this study used a variety of data collection methods to obtain a representation of classroom behavior (Brophy & Evertson, 1974; 1976). From the TTEP and research literature on teaching in pre­ school (Blank, 1973) as well as Jere Brophy's program development work at Southwest Educational Development Laboratory, a manual was developed delineating 22 principles for small-group management and instruction. These principles formed the basis for the treatment condition in the FGRGS. The FGRGS was the first experimental study in this program of research (Anderson, Evertson, & Brophy, 1979). The TTEP and FGRGS form the descriptive-correlational-experimental loop. This loop ends with the FGRGS; however, the information obtained in this set of studies informs work in Loop 2. Three factors led to the cessation of Loop I and to modifications of work in Loop 2. First, one of the consistent findings across the TTEP and FGRGS was that management and organization factors were related to student achievement. Second, the use of category systems did not permit in-depth exploration of the managerial and organizational process. The behaviors and var­ iables obtained from the category systems were decontextual­ ized to a degree and contextual information could not be re­ trieved to help explain how the process itself was constructed. Third, the studies were undertaken beginning in October of the

I

Texas Teacher Effecli\leness Project (TTEP) 1970-1973 Purpose: To identify those teaching behaviors that were associated with student achievement in a group of second and third grade teachers identified as stable in their effects on studenl achievement over a 3-4 year period.

Background Issues Leading to the Texas Teacher Effectiveness Studies 1969-1970

Year 1 : 31 teachers observed

Issue 1 : Large-scale studies of schooling minimized teachers' contributions to student achievement, but did not include systematic observation of teacher behaviors in the sludies (Coleman et al, 1966; Jencks et al., 1972),

Year 2: 28 teachers observed Instruments:

Responses/Actions Taken

Few studies existed that related teachers· classroom behaviors to student achievement outcomes. Issue 3:

Issue 4: Methodological impediments to systematic study ol effective teaching had to be overcome (e.g.. lack of stability of teacher behavior, Rosenshine, 1970).

Teacher interviews and questionnaires

Research team responded to the need identified from past studies: the need to observe teaching systematicatly and to identify teacher effects.

Issue 2:

Curriculum development focused more on what to teach rather than how to teach. Teacher effects appeared less important than curricula to which students were exposed (Walker & Shaffarzick, 1974)

Category system using event sampling (expanded version of Brophy & Good Dyadic Observation System)

�

Questions tor study were Identified. New methods of observation were designed to overcome some of the methodological problems which led to the lack of replication in previous studies and to the limited knowledge base.

Materials and methods observation checklist Observer ratings and checklists

>-------,

Student achievement measures in mathematics and language Observer ratings of target students

-

I I

I

I

Junior High Classroom Organization Study (JHCOS): Extension of cos 1978-1979

Classroom Managemen! Improvement Study: Experimental CMIS 1980-1981

Arkansas Classroom Management Training Study (C�HS): Experimental 1982-1983

Purpose: To extend TTEP to the junior high school level

Purpose: To explore how teachers organized and managed their classroom from the first day of school

Subjects: 29 math teachers 31 English teachers

Purpose: To discover how teachers in secondary school organized and managed classes al the beginning of the year.

Subjects: 28 teachers in self-contained third grade classes

Purpose: To evaluate the extent to which materials and teacher workshops helped teachers to establish and maintain good classroom learning environments

Purpose: (a) To test the exportability of research findings to teachers in a different region ol the country: (b) to determine if training and observalion !unctions could be handled by school personnel.

Instruments: Expanded category coding system based on event-sampling system in TTEP

Instruments: Detailed narrative records describing all events relating to organizing classrooms. Counts of student engagement (% of students on-lask, off-lask. wailing, etc.)

Teacher interviews and questionnaires Observer ralings of teachers

Interviews with teachers aboul their start of school planning, rules for behavior, difficult or disruptive students, rationales for organization of classrooms, etc.

Observer descriptions of general classroom format and activities Records of lime spent in various formats

Observer ratings of lesson management, student behavior. class climate, etc .. after each observation and at end of year

Student ratings of teachers Experimental study of first grade reading groups 1974-1975 Purpose: To test find·

'--- ings of TTEP and effecliveness of 22 principies dealing with teacher managemenl of small group instruclion in first grade.

Subjects: 27 first grade teachers in self-contained classrooms 1 O treatment observed; 7 treatment unobserved; 1 O control observed Instruments: Category system tor coding interactions in small groups. Development informed by Blank (1973) and investigators' training in early childhood development, as well as findings from TTEP. Ineluded categories for coding content and methods. Teacher interviews and questionnaires Student achievement data in reading

Fi

I I

Descriptive-Correlational: Classroom Organization Study (COS) 1977-1978

Texas Junior High Study: Extension of Texas Teacher Effectiveness Proteet 1974-1975

�

Time logs of time spent in various class activities Student achievement measures in reading & math

Subjects: 26 English teachers 25 math teachers

-

Instruments: Detailed narrative records describing all events relating to organization of classrooms at beginning of the year. Interviews with teachers about their start of school planning, rules for behavior, rationales tor organization of classrooms, etc. Counts of student engagement (% of students on-task, off-task, waiting, etc.) Observer ratings of lesson management, student behavior. class climate, etc., after each observation and at the end of the year Time logs of time spent in various class activities Student achievement data Student ratings of teachers Elementary School Pilot Study (ESP$): Experimental (1979) Purpose: To test strategies for effective classroom management found in the

cos.

Subjects: 12 beginning teachers received the following: 4 manual + consultation; 4 manual only; 4 control no manual lnslruments: Detailed narrative records (see COS) Counts of student engagement Observer ratings after each class period (See COS) Interviews with teachers about usefulness of each part of the manual

Subjects: 41 teachers in grades 1 - 6 in 14 schools were observed 23 teachers received workshops and manuals 18 controls (9 received delayed workshop and manual) Instruments: Detailed narrative records (See COS, ESPS, JHOCS) Counts of student engagement: (on-task academic: on-task procedural: probably ontask academic or procedural; off-task (sanetioned or unsanctioned. dead time) Observer ratings of classroom climate, lesson management, student appropriate or inappropriate behavior, etc. after each lesson and at end of year Time logs of classroom time use and types ol activities Teacher interviews al usefulness of workshops and manuals.

Subjects: 16 math.and English teachers 7-9 grades. 8 trained-observed 8 control-observed Instruments: Audiotapes of classroom lessons and teacher-stu• den! lalk. Counts of student engagement (See COS, JHCOS, ESPS, CMTS) Observer ratings of instruclion, pupil behavior. classroom climate alter each observation and at end of year Student achievement measures

l

Exploring Models of Management: 1983- 1984 Purpose: (a) To explore management approaches of effective and less-effective managers, all of whom were trained in classroom management; (b) to explore the relationship of management of classroom environments and lesson/academic organization and management using data from Arkansas Classroom Management Training Study. Utilize data from the Arkansas Classroom Management Training Study. Descriptive system for coding lesson content and teacher and student talk is developed.

OBSERVATION AS INQUIRY AND METHOD school year. This sampling procedure did not enable the re­ searchers to explore how teachers established and maintained effective management and organization patterns and structures. In addition, with the exception of Smith and Geoffrey (1968) and Tikunoff, Ward, and Dasho (1978), little information ex­ isted on how teachers began school. These factors led to a mod­ ification of data collection procedures and to a change in obser­ vation systems. These changes are discussed again in Loop 2. LOOP 2: DECISION MAKING AND SHIFTING LENSES IN AN INQUIRY CYCLE

Loop 2 also begins with the Texas Teacher Effectiveness Study (TTEP). In this way Loop 1 and Loop 2 overlap. Loop 2 con­ sists of two patterns: (a) a set oflinked descriptive-correlational studies that explored effectiveness in general terms at two differ­ ent levels of schooling (elementary and secondary), and (b) a descriptive-correlational-experimental set that explored a more focused question, a question identified in Pattern 1 and in Loop 1 . Pattern 1, the descriptive-correlational studies that made up the Texas Teacher Effectiveness studies, showed the interac­ tive-reactive nature of the studies in this program of research. The Texas Junior High School Study (TJHSS) (Evertson, An­ derson, Anderson, & Brophy, 1980; Evertson, Anderson, & Brophy, 1978) was informed by the elementary TTEP. First, the observational systems were modified to fit the setting and to eliminate the problems identified in the elementary study. For example, within the first week, additional data collection was needed. Some observers were turning in coded sheets with few or no recordings in the teacher-student public interaction sec­ tions of the system. One of the major reasons for the lack of codes in these sections was that much of the coding system pro­ vided for recording only public or private verbal interaction between teachers and students. The classes in which some of the observers were recording information were ones in which there was limited verbal interaction and larger amounts of silent or seatwork. Therefore, in order to ascertain what was occurring when the teacher and students were not interacting verbally, all observers were asked to record brief narrative descriptions. This qualitative information was used to index what was occur­ ring in classrooms where verbal interaction was limited. Also the descriptions provided a context, some academic content in­ formation, and some idea of the flow of events. Second, two classes were observed for each teacher to explore the issue of teacher stability of effective practices. Third, student outcome measures were expanded. Two types were collected: student achievement measures (pre and post) and student rat­ ings of their teachers. Student achievement was measured with a content-referenced test designed to align with the curriculum in the district. Student scores on the California Achievement Test were used as pretest measures. Estimates of student learn­ ing gain were then related to classroom process measures and teachers' classroom behaviors. Once again, management and organization factors correlated with student achievement. As in the case of the studies in Loop 1, the category system did not permit exploration of the process of management and organization nor did the sampling proce­ dures permit observation of the onset of these processes. These

193

limitations led to a further modification of instrumentation and collection procedures in Pattern 2, the set of descriptive-corre­ lational-experimental studies. Pattern 2, the Classroom Organization Study (COS), the first in the descriptive-correlational-experimental loop, was a more focused, in-depth look at management and organization proce­ dures. This study was designed to capture events starting on the first day of school, therefore maximizing the opportunity to de­ tect and explore the procedures teachers used to estabiish and maintain the organization and management of classrooms. A shift also occurred in the ways in which data were collected. Instead of designing a category system that counted behaviors, a series of procedures were adopted: (a) narrative recorcis were made; (b) observer ratings focusing on lesson management, stu­ dent behavior, class climate, and so forth, were ootained; and (c) time spent in various class activities was recorded. From this study, a series of principles for effective management and organ­ ization of classrooms was obtained (C:mmer, Evenson, & An­ derson, 1980; Evertson, Anderson, & Clements, 1980). These principles formed the basis for the pilot experimental field study and, ultimately, the Classroom Management Improvement Study (CMIS, 1 980- 198 1 ; see Evertson, Emmer, Sanford, & Clements, 1983). The Classroom Organization Study also led to a second de­ scriptive-correlational study at the junior high school level. This study, like the Texas Junior High School Study (TJHSS), was designed to provide information about how effective pro­ cesses were realized in a setting with older stucients. The junior high COS (JHCOS), therefore, was an extension of the COS and the TJHSS. The findings from this study, along with find­ ings from the pilot experimental field study, formed the basis for the Classroom Management Improvement Study (CMIS). The interrelatedness of the studies shows how the type of fo­ cusing cycle suggested in the Corsaro (1985) description of a research strategy can be realized over a series of studies. Rather than occurring within a single study a8 in the case of an ethno­ graphic study, the decisions and modifications occurred across studies. Each new study builds on the ones before and informs the ones that follow. In this way this cycle is interactive and reactive in nature. The descriptive-correlational-experimental loop does not end with the Classroom Management Improvement Study, however. In 1982- 1983, Evertson, in cooperation with the Ar­ kansas State Department of Education, conducted an action research study and tested the Texas training model in Arkansas. This study explored the question of exportation of a model de­ veloped in one region to another region. It used a modified ver­ sion of the training model. The fact that the state and local districts provicied the resources, rather than a funding agency, influenced both how the study was undertaken and the nature of the data collected. Training of observers was limited; obser­ vers used rating forms similar to those in the COS and CMIS and counts of student engagement. Observers took notes but did not use detailed narratives. Live audiotapes replaced narra­ tive records. This study involved a cooperative effort between state and local educators and Evertson, the consuitant who trained ob­ servers and workshop leaders. This study, therefore, is a collab­ orative effort between researcher and school districts; it is an

194

CAROLYN M. EVERTSON AND JUDITH L. GREEN

action research study, a study undertaken in situ and adapted to the needs and conditions of the local groups requesting the training.

LOOP 3: A QUALITATIVE-QUANTITATIVE MERGER

The Arkansas Classroom Management Training Study (ACMTS) is both the end of a cycle and the beginning of a new loop. This third loop is just underway with the study entitled Exploring Models of Management (1983-1984). This study, funded by NIE, is a collaborative effort between the Arkansas State Department of Education and a team of researchers (Evertson and Green). In this study, the focus on management models narrows even further. The lens shifts from a training model derived from a descriptive-correlational-experimental loop to an exploration of the nature of the models of manage­ ment and organization of environment and content. To explore the models used by effective and less effective managers identi­ fied in the training study, an in-depth secondary analysis using a sociolinguistic ethnographic perspective (Green & Bloome, 1983) and a descriptive analysis system adapted from the work of Green (Green & Harker, 1982; Green & Wallat, 1981b) is being used. This study involves a secondary analysis of data and a shift in analytic approach. It merges qualitative and quantitative analyses. It also merges qualitative and quantita­ tive with micro- and macro-levels of analysis.

Summary of the Decision-Making Process in a Descriptive- Cor­ relational-Experimental Loop. As discussed above, in a system­ atic program of research over an extended period of time, decisions made in one study can be used to modify and inform later studies; that is, they can be used to determine a new focus, what is to be collected, and how it is to be collected, as well as when data are to be collected. In other words, decisions and the resulting research cycle become interactive-reactive (cf. Hymes, 1978). In this way decisions become principled and theoretically driven. The use of such decision-making strategies can lead to changes in ways of observing the segment of reality under study. Table 6.9 shows the shift in instrumentation used in the Evertson et al. program of research. As in the case of the Cor­ saro approach, multiple collection procedures were used within and across studies; data collection procedures provided conver­ gent information within and across studies; and data collection procedures were modified, added, and deleted when the ques­ tion and situation required. Finally, issues of entry and intent influenced the types of information collected and available for analysis. The inquiry cycle described above, therefore, is both linear and generative; that is, it contains both preset, a priori aspects of design and interactive-reactive types of decision points. As the program of research developed, the approach used moved from the decontextualized (exclusive) end of the continuum pre­ sented in Figure 6.3 to the more context inclusive end of the continuum; it also moved from a preset format in decision mak­ ing to one in which questions are generated throughout the study as described in Figure 6.4.

Example 2: Merging Perspectives in Collaborative Re­ search The second example illustrates how differing research programs and questions can be merged to form a unique and continuing program of research. Figure 6.7 graphically presents this pro­ gram. As indicated in Figure 6.7, prior to 1979, Weinstein and Marshall each had their own programs of research. Weinstein was concerned with investigating the effect of teacher expecta­ tions and began studying teachers' differential treatment of stu­ dents and students' perceptions of such treatment. This early work led to the development of a student mediation model of the processes by which patterns of teacher differential treatment result in differences in student achievement. This model sug­ gests that students perceive, interpret, and act on information contained in teacher cues about expected achievement (Wein­ stein, in press). As a part of this research, Weinstein developed a student questionnaire to explore student perceptions of teacher treatment, the Teacher Treatment Inventory (Weinstein & Middlestadt, 1979). This work provided the background for the NIE study of Student Perceptions of Differential Teacher Treatment in 1979. In the precollaborative stage, Marshall was involved in developing ways of exploring the nature of classroom structure and functioning across open and traditional classrooms and in identifying the characteristics of open classrooms (1972-1978). During this phase, Marshall developed the Dimensional Occurrence Scale (DOS), a low-inference, broad-range instru­ ment, designed to capture a wide range of classroom structures and ways of functioning (e.g., grouping practices, time spent on various activities, types of materials available and in use, the types of strategies teachers used, etc.). This instrument included ratings and a sign system as the primary ways of capturing the similarities and differences in structuring and functioning (Mar­ shall, 1976a; Marshall et al., 1977). Along with the DOS, Mar­ shall used a categorical system, a revision of the Reciprocal Category System (Ober et al., 197 1), modified to capture differ­ ent contexts or types of language use (Marshall & Green, 1978). Marshall also used other coding systems designed to explore student engagement and teacher movement paths (Marshall, 1976b). In other words, Marshall was involved in studying open and traditional classrooms using multiple perspectives. In 1979, Marshall joined Weinstein for the data collection and analysis stage of the NIE study of Student Perceptions of Differential Teacher Treatment. One contribution to this pro­ ject was Marshall's knowledge of classroom structure and the literature on open-traditional education. A major hypothesis for this study was that students would perceive less differential teacher treatment in open than in traditional classrooms. This hypothesis was not supported, although the study did show large classroom differences in the amount of differential treat­ ment perceived by students (Weinstein et al., 1982). Moreover teacher expectations were found to contribute more to the pre­ diction of achievement in classrooms in which students per­ ceived high degrees of differential teacher treatment as opposed to classrooms where students perceived low amounts of differ­ ential teacher treatment (Brattesani, Weinstein, & Marshall, 1984).

195

OBSERVATION AS INQUIRY AND METHOD

Table 6.9. N ature of Data Collected Throughout Inquiry Cycle Category System

Ratings

Checklist/Sign System

Texas Teacher Effectiveness Study (TTEP) 1 970-1973

Expanded version of Brophy-Good Dyadic lnteration System

Observer ratings of teacher behaviors/ characteristics/ class climate, clarity, enthusiasm. etc.

Observer checklists of teacher methods, materials, time use, lesson presentation, feedback, etc.

Texas Junior High School Study (TJHSS) 1 974- 1 975)

Expanded version of TTEP observation system with added categories for student questions/ comments

(See above)

Observer records of classroom time spent in various formats

First-Grade Reading Group Study (FGRGS) 1 974- 1 975

A specially developed category system capturing methods and materials, along with teacher student interaction and treatment implementation

Narrative

Interviews

Technological

Outcome Measures

Teacher interviews about methods, materials, use of rewards, beliefs, etc.

Brief observer descriptions of class activities, format, content being presented

(See above)

Student achievement on California Achievement Tests used as a covariate with specially designed end-of-year test Student attitude measure

Teacher interviews about reading methods and use of 22 principles

Student achievement in reading

Classroom Organization Study (COS) 1 977-1 978

Observer ratings of lesson presentation, pupil behavior, teacher reaction to misbehavior, class climate, etc.

Counts of student levels of task engagement (e.g. on-task, academic; on-task, procedural; offtask, sanctioned or unsanctioned; offtask)

Detailed narratives of events related to managing and organizing classrooms from start of school

Teacher interviews about management practices, beliefs about student control, etc.

Student achievement in reading and math

Junior High Classroom Organization Study (JHCOS) 1 989-1 979

(See above)

(See above)

(See above)

(See above)

(See TJHSS)

Elementary School Pilot Study (ESP) 1 978-1 979

(See above)

(See above)

(See above)

Teacher interviews about their use of the manual and opinions about utility

Classroom Management Improvement Study (CMIS) 1 980- 1 981

(See above)

(See above)

(See above)

(See above)

Arkansas Classroom Management Training Study (ACMTS) 1 982- 1 983

(See above)

(See above)

Exploration of Management Models 198'4

Observer notes to describe classroom context, student task engagement, and time

Audiotapes of lesson content, teacher-student talk

Student performance on SRA used as a covariate with end-of-year content referenced tests

{uses data from ACMTS) Primarily secondary analysis

The findings of the 1979-1980 NIE study led to a refinement and expansion of the research questions in the next set of studies. One of the major questions generated was : What was the nature of classroom context in which students' perceptions of high and low amounts of differential teacher treatment were embedded, which might then contribute to students' self-expec­ tations? To answer this question, an observational study was designed to be conducted in a subset of 12 of the 30 classrooms (in the NIMH and NIE 1980-1984 studies). These classrooms were selected from the extremes of student-perceived high- and low-differential-treatment classrooms. A model of classroom

factors influencing the development of student self-evaluations was postulated (Marshall & Weinstein, 1984). This model was developed based on the earlier work of both researchers, the NIE study (1979-1980), new research findings on public-pri­ vate feedback (Bossert, 1979; Blumenfeld et al., 1979), and re­ search on single and multiple abilities of students (Cohen, 1979; Rosenholtz, 1979, in press). This model of classroom factors affecting students' self-eval­ uations and a refined student mediation model served as a framework for planning and framing the next set of studies (the NIMH and NIE 1980-1983 studies). These models influenced

r- - - - - - - - - - - - - - - - - - - - - - - - - - - -7

I. PreCollaborative Period

Dissertation study (Weinstein, 1 976)

I

B. Prior research on teacher expectations (Rosenthal & Jacobson, 1 968; Brophy & Good, 1 970b. 1974)

r

I I I I Weinstein's Student Mediation Model: Students perceive, interpret, act on differential teacher treatment, an internal process model (Weinstein, 1981, in press).

I

Student Perceptions of Differential Teacher Treatment 1919-1 950 Study Funded by NIE; Comes from the student mediation model Hypothesis that students would perceive less dif­ ferential treatment in "open" classrooms was not supported ; but large class differences in the amount of perceived difResearch question: What ferential treatment were was the nature of found (Weinstein, Marshall, classroom contexts surBrattesani, & Middlestadt, - rou nding student percep1 982). tions of high and low amounts of differential Teacher expectations conteacher treatment? tributed more to the pre­ diction of achievement in high, rather than low, pre­ ceived differential in classroom treatment (Brat­ tesani, Weinstein, & Mar­ shall. 1 984)

Development of student questionnaire (Teacher Treatment Inventory; Weinstein & Middlestadt, 1 979)

Investigator II: A. Literature review on classroom structure, open/traditional (Marshall. 1972, 1981) B. Model of Dimensions of Classroom Structure and Functioning, 1973-1976 Development of Dimensional Occurrence Scale (DOS) 1 973-1 976 (Marshall ----et al., 1977; Marshall & Green, 1978; Marshall, 1 976 a. 1 976b)

Research on public/private feedback (Bossert, 1979; Blumenfeld, Hamilton, Wessels, & Falkner, 1 979)

-

-

---

I I

I I I I

Background factors con­ tributing to the theoretical perspective Investigator I: A. Investigator l's clinical experience with under­ achieving children

11. Collaborative Period

I I I FRAMING & PLANNING

-

-

-

-

- -----

----

Model of classroom factors influencing development of student self-evaluations: 1 . task structure 2. grouping practice 3. feedback, information, and evaluation about ability 4. motivational strategies r--+-- 5. locus of responsibility 6. quality of relationships (Marshall & Weinstein, 1 984) Research questions: What may allow students to perceive differential treatment and act on this to the detriment of low achievers? Why don't students in some classes report dif­ ferential treatment?

I

r-

-------------,r-------

Research on single/multiple abilities (Cohen, 1979; _______ Rosenholtz, 1 979)

Ir

-

----�

_____J Research on multilevel analyses of classroom data (Burstein, 1 980; Martin et al., 1 980)

- -�

II. DATA COLLECTION

A. Observation Sequence Classroom Dimension Scale (CDScale)

Weinstein 1 980-1 983 NIE and NIMH Studies

(a) Focused Field Notes Used on line in class. Run­ ning account of events in classroom. Verbatim ac­ counts of teacher state• ments (excluding content) Type 1 : Literal narratives Type 2: Impressions & interpretations Focus of System Aspects of classroom rele­ vant for communication of achievement expectations. 1. Structure of tasks, sub­ ject matter, materials 2. Grouping practices 3. Locus of responsibility in learning 4. Feedback and evalua­ tion 5. Motivation 6. Quality of teacher-stu­ dent interaction 7. Statements of expecta­ tions. attributions

Year 1. Instrument development. Item content mod­ ified to reflect model For­ mat for: 1. More ecologically valid time periods 2. Level of i nteraction (e.g .. teacher to student in context of whole class, group, or individ­ ual) more consistent with level of data analy­ sis (Marshall, Weinstein, Sharp, & Brattesani, 1 982; Weinstein, in press; Weinstein & Mar­ shall, 1 984) Year 2. Data collection Fall sample: 30 classrooms-10 each in Grades 1 , 3, and 5 Spring subsample for observational study: 1 2 classrooms: 2 high-, 2 low-differential treat­ ment in Grades 1, 3, and 5. Year 3. Data analysis Note: Other hypotheses beyond the observational study of classroom factors were considered, including the influence of student self-concept, expectation level, race, sex, grade level, and parents (Marshall et al., 1 982; Weinstein, in press; Weinstein & Mar­ shall, 1 984)

(b) Observational Scale (CDScale/ Used on line in class for: Part I. Coding overview of general structure of learning environment Used following observa­ tion for: Part II. Tallying exact number of instances of behaviors observed and recorded in field notes Part Ill. Rating aspects of climate B. Teacher Interviews For nonobservable events. Captures deci­ sions on grouping, cur­ riculum sequences, eval­ uations of students. etc. C. Summary Impressions by Observer D. Student Interviews By researcher other than observer about their per­ ceptions of teacher's and mother's feelings about their schoolwork

Fig. 6.7. Collaborative research cycle.

OBSERVATION AS INQUIRY AND METHOD decisions about the nature of observations needed (e.g., high­ and low-reading groups as well as whole-class observations). In addition three other factors contributed to the development of the observation system and to the research design : (a) the loss of information resulting from the use of categorical systems (Marshall and Green, 1978), (b) the need to use more ecologi­ cally valid observation periods (event rather than time sam­ pling), and (c) the work by Burstein (1980) and Martin et al., ( 1980) on multilevel analyses of classroom data. An observation system was needed that would permit the broad-based explora­ tion of the characteristics of classrooms where students per­ ceived differing amounts of differential teacher treatment; therefore, given the refined questions and the new model, the Marshall instrument, the Dimension Occurrence Scale (Mar­ shall, 1976a; Marshall et al., 1977), was modified to meet the requirements of the new studies. Both quantitative and qualita­ tive information was collected with the modified observation system (renamed the Classroom Dimension System). Qualita­ tive data were collected in on-line narrative records. Quantita­ tive data were collected in three sections of the Classroom Dimension Scale (CDScale). As indicated in Figure 6.7 under "Data Collection," Part I of the CDScale was used on-line; that is, data were recorded on coding forms while the observer watched the classroom activities. This part provided an over­ view of the general structure of tasks, groupings, and evaluation procedures that create the context for learning during the ob­ servation period. This information provides a general picture of (a) whether students were working individually, in groups, or together, (b) where the teacher was working, (c) the subject­ matter content, (d) the amount of choice that students had, and (e) the predominant types of teacher evaluation. This section consists of a nominal scale, recordings of amount of time, di­ chotomous variables (occurs - does not occur), counts of the exact number of cases, and a hierarchical system in which items can be coded for each level of grouping. This section, therefore, uses multiple types of data collection in order to obtain a wide view of the context on a general level. Part II of the CDScale focuses on teachers in interaction with students or groups of students. The information obtained from this section includes : (a) types of tasks, (b) motivation strate­ gies, (c) responsibilities, (d) evaluation and feedback, and (e) quality of the relationships. The items in this section of the CDScale are coded from narrative records, rather than live. The observer makes a narrative record (field notes) that includes the context in which the interaction occurred, the specific behaviors that occurred, and the general flow of the interaction between teacher and student. The observer records the individual stu­ dent with whom the teacher interacts and whether the interac­ tion occurs with (a) the class as a whole, (b) an individual with others around, (c) a group, (d) the individual within a group, or (e) the individual alone. The tone of the teacher's interaction is also recorded (e.g. matter-of-fact, warm). In addition, the ob­ server also makes a second type of field note to indicate impres­ sions and interpretations of events. Field notes are typed immediately after the observation period using a set format to permit ease of retrieval of teacher statements and interactions. After the observation period, the observer applies the Part II of the CDScale (on teachers' interactions) to the field notes. Using this combined procedure has several advantages over coding

197

interactions on line alone or coding only after an observation period. These are: (a) an observer can count the exact number of times each behavior occurs from field records; (b) an obser­ ver can reflect on the context and consider whether the behav­ iors actually fit the definition as well as the nuances of behaviors not fully recorded; and (c) this approach allows ac­ curate retrieval of instances of occurrence and discussion with other observers when checking reliability. Part III of the CDScale is a high-inference rating scale. The observer rates the frequency and intensity of warmth and irrita­ tion conveyed to the class, the individual, and the group. In addition to the Classroom Dimension System, student measures developed in the early Weinstein work (Teacher Treatment Inventory, Weinstein & Middlestadt, 1979), and those added in the NIE 1979 study (measures of student and teacher expectations) were modified to meet the needs and re­ fined hypotheses of the present study. These changes included an expansion from Grade 4 to Grades 1, 3, and 5, as well as an exploration of the influence of student self-concept and level of expectation (for achievement) on (a) student perceptions of differential teacher treatment, (b) student perceptions of their teacher's expectations for them, and (c) on year-end achievement. To ascertain a more complete picture of the classroom from other viewpoints, teachers were interviewed concerning the less observable aspects of the classroom (e.g., decisions about grouping, curriculum, evaluation) and students were inter­ viewed concerning their perceptions of their teacher's (and mother's) attitudes and behaviors regarding the quality of their schoolwork. The discussion above demonstrates how the work of both Weinstein and Marshall has combined to create a new ap­ proach that is an extension of the individual researcher's previ­ ous work. In addition, this work continues to develop the general framework begun by Weinstein. The question of factors that influence student learning and the effect of differential teacher treatment continue to be the thread that is consistent across studies. The process reflected in the development of the current program of research suggests that this program is inter­ active-reactive in the same sense as the work by Evertson and her colleagues. One study informs other studies ; the ways in which observations are made are informed by previous work; and the needs of the current study (e.g., the type of setting, the age of the students, the questions under study, etc.) lead to the modification of past data collection procedures as well as the general design of the study. The Weinstein work began with a questionnaire approach and added observation with the addition of Marshall's work. The observation system was systematically modified to include more and more contextual information. Marshall & Weinstein ( 1984) point out that the contextual information from the nar­ rative records permits both exploration of variation and poten­ tial explanations for the quantitative findings. In addition, as Marshall and Weinstein (1982) suggest, the use of narratives permits retrieval of contextual information that can be used in obtaining interrater reliability and in resolving differences in perceptions of events. This type of record appears to be impor­ tant when videotape records are not available, not cost effective, or not permitted by the population under study. This

198

CAROLYN M. EVERTSON AND JUDITH L. GREEN

multiple-instrument and multiple-perspective approach per­ mits retrieval of a wide range of information.

Example 3 : A Mode! of Triangulation to Collect Multiple Perspectives The third study represents a systematic effort to obtain different perspectives of the same process, discourse processes (e.g., ques­ tion use in different settings: classroom, play group, home). In contrast to the presentation of the previous examples, the dis­ cussion in this section begins with the presentation of the trian­ gulation study and then moves to a discussion of the historical roots of the various components of the study. The design of the study by Morine-Dershimer and Tenenberg (1981) is unique in that it was actually a series of preplanned contrasts and interre­ lated analyses of a systematically collected data set. The study involved collecting three different perspectives on each type of phenomenon: (a) teacher-student interaction in language arts lessons in which discussion was a means of instruction, (b) com­ munication in home situations of a select set of students (fami­ lies reflecting different ethnic groups), and (c) interactions in play groups outside the classroom. Figure 6.8 presents a de­ scription of this triangulation approach (Adelman & Walker, 1975). Space does not permit more than a cursory exploration of each study. Therefore, rather than focus on the studies per se, the discussion will focus on the model on a general level. The general framework for this study was drawn from work on sociolinguistic research. The specific question focused on (a) the nature of participants' perceptions of classroom discourse; (b) the interface between and among the perceptions of different participants; (c) the relationship of students' perceptions and performance with achievement and participation in classroom events (e.g., discussion in language arts lessons); and (d) the continuity of perceptions of language functions at home, in school, and during play. One of the outcomes of this work is the identification of what is required of students in classrooms in terms of communicative participation and understandings, or, in Morine-Dershimer's words, in what it takes to be a "literate" pupil (Morine-Dershimer, 1981). The contrasts in Triangle 1 permit exploration of the same phenomena (36 language arts lessons: 6 for each of 6 teachers in Grades 2, 3 and 4) from different sociolinguistic analytic per­ spectives: speech act analysis (Ramirez, 1979; cf. Smith & Coulthard, 1975), structural analysis of questions and cycle sequences (Tenenberg, 1981, in press; cf. Johnson, 1979), and analysis of language dimensions (Morine-Dershimer, Tenen­ berg, & Shuy, 1981; Tenenberg, Morine-Dershimer, & Shuy, 1981; Morine-Dershimer & Tenenberg, 1981). This point of tri­ angulation is labeled "Alternative Sociolinguistic Descriptions of Classroom Discourse." The findings from this set of triangu­ lations provided both convergent and complementary informa­ tion about the nature of classroom discourse, with a special emphasis on questions (cf. Johnson, 1979; Mehan, 1979a). Taken together the information provides a more holistic picture of the nature of teacher-student interactions. This set of anal­ yses identified teachers' use of differential questioning styles that were related to differences in student achievement. Triangle 2, "Alternative Perceptions of Classroom Dis­ course," focused on perceptions of classroom discourse by

teacher, student, and researcher. This triangle shares a common analysis with the previous triangle. As indicated in Figure 6.8, information was obtained using a variety of collection proce­ dures. Videotape records were played back to teachers and pupils on the day they were taken and viewers were asked to respond to what they remembered hearing. Students and teachers were also asked to generate sentences which might be said by or to the pupil to "get someone's attention" or "get someone to do something." A sentence completion task on "rules" of discourse, constructed on the basis of pupil response to an open-ended question about "how people talk in your classroom" was also completed. Participants were asked to or­ ganize a set of three-by-five cards obtained from "what do you hear" into groups of cards that "belonged together because people were saying the same kind of thing." And finally, partici­ pants were asked to study a set of teacher questions asked in the lesson (also collected was information about teacher praise and pupil responses); they were then asked to explain who said these things, to whom, for what reason. This information was explored for convergence of perspective as well as for contrast points and relationships to achievement and participation. Triangle 3, " Pupil Perceptions of Discourse in Alternative Settings" also explored multiple perceptions of discourse; how­ ever, in this triangle, the discourse explored was that which oc­ curred in alternative settings (family conversations of a selected set of families, classroom discourse, and play groups discourse). Once again Triangles 2 and 3 shared a common analysis; in this instance, analysis of classroom discourse and perceptions of pupils of this discourse constituted the common analysis point. The tasks in this triangulation were similar to those in Tri­ angle 2. These tasks included (a) sentence completion, (b) gen­ erating sentences, (c) a "reporting-what-was-heard" task, and (d) organizing the cards on what was heard and what belonged together. This set of analyses also permitted exploration of con­ vergence of perceptions and points of contrast in perceptions of discourse used in the three different settings (home discourse, classroom discourse, and the discourse of play). The final triangle contrasted alternative perceptions of dis­ course in play settings. This triangle, "Alternative Perceptions of Discourse in Play Settings," shares a common analysis point with Triangle 3. In Triangle 3, perceptions of classmates of dis­ course in a play setting were contrasted with the perceptions of a group of early childhood educators (n = 10), and finally, with pupil participants' perspectives. The decision making in this study was preplanned and the contrasts were developed on a systematic and principled basis. These contrasts guided the decision making about the data col­ lection procedures to be used. Observations were made of three types: videotapes (a) of a set of language arts lessons in which teachers taught a series of self-selected lessons, (b) of play groups, and (c) of family interaction. The limit set by the re­ searcher was that the language arts lessons could not contain seatwork; rather the lessons had to have some discussion so that the discourse could be explored. Stimulated recall was used in conjunction with interviews of students and teachers. These participants were shown the tapes and asked a series of ques­ tions after viewing the tapes (e.g., What do you remember hear­ ing?). The comments were recorded by the researcher on 3 x 5 cards. The information on the cards became a source of data

, _,,

Pupil Participants as Observers of Classroom Discourse

Sociolinguist as Observer (Speech Act Analysis)

Sociolinguist as Observer (Analysis of Structural Sequence of Question Cycles)

Videotapes were viewed & analyzed by three different sociolinguistic systems. Comparisons gen­ erated among these and with teach· er and pupil data.

Schoolmates as Observers of Family Conversations

ALTERNATIVE SOCIOLINGUISTIC DESCRIPTIONS OF CLASSROOM DISCOURSE

Sociolinguist as Observer (Analysis of Language ''Dimensions'')

ALTERNATIVE PERCEPTIONS OF CLASSROOM DISCOURSE

Videotapes were played back to pupils, teachers, and sociolinguist. Each completed tasks: 1. Completing sentences on "rules" of discourse 2. Generating sentences used for gaining attention 3. Reporting what was heard after videotape playback 4. Organizing "what you heard" on cards that "belonged to­ gether" 5. Studying questions ask­ ed in the lesson and explaining who said these to whom and why.

videotapes of family conversations and children's play groups Tasks used were similar to Triangle 1 to compare pupil perceptions of classroom language to language in less formal settings.

PUPIL PERCEPTIONS OF DISCOURSE IN ALTEA­ NATIVE SETTINGS

Classmates as Observers of Play Group Conversations

Teacher Participants as Observers of Classroom Discourse

ALTERNATIVE PER­ CEPTIONS OF DIS­ COURSE IN PLAY SETTINGS

Videotapes of child play were observed by 10 early childhood specialists. Perceptions were compared with the play group participants and class­ mates to discover more about the language of lessons.

E8irly Childhood Specialists as Observers of Play Group Conversations

Pupil Participants as Observers of Play Group Conver­ sations

PRIMARY DATA: Six videotaped language arts lessons in each of six classrooms (Grades 2, 3, & 4) between September & January Videotapes of family conversations and children's play group conversations OUTCOME MEASURES: Pre and post reading achievement scores

Fig. 6.8. Use of triangulation to study classroom discourse : Study design and i nstru mentation. Adapted from Morine-Ders h i mer (1 981 ).

r

200

CAROLYN M. EVERTSON AND JUDITH L. GREEN

(see Triangles 2 and 3). The participants were asked to organize the cards to reflect what belonged together. Finally, partici­ pants were given questions obtained from the lessons and asked to specify who said these to whom and why. This information, collected from multiple sources, provided the basis for triangu­ lation of findings. Triangulation, therefore, is a means of explor­ ing convergence of perception and analysis approaches. In addition, this approach permits exploration of differences in perception and/or description obtained from each perspective. This research highlights continuities and discontinuities in per­ ceptions of various groups and various research approaches. It also identifies differential behaviors and their relationship to achievement as well as student perceptions of these differential behaviors. It is an approach that provides a broad picture of events and permits exploration of factors that can influence stu­ dent performance and achievement. In this way, this study is related to the Weinstein and Marshall program in Example 2. The discussion above focused on a single study with multiple components, but as argued previously, to fully understand a study, previous work by the researcher must be considered. Therefore, to understand why Morine-Dershimer used this par­ ticular set of collection procedures, and the specific design, the research history of this project must be considered. In the re­ mainder of this section, an exploration of earlier work that serves as a grounding for the present study will be considered. The Historical Groundingfor the Triangulation Study. All work in classrooms undertaken by Morine-Dershimer has been de­ signed to help teachers observe themselves and what they do 1 965. Dissertation: A - 1 969. Discovery Modes-A Criterion Model of Inductive for Teaching Pro­ Discovery for In­ cess (Morine, 1 969) structional Theory (Morine, 1 965) Explored implica­ tions of different kinds of subject mat­ ter for instructional p rocedures, ways to have students try to think. Premise: Process by which knowledge was developed is an i mportant variable to consider in trying to teach the knowl­ edge.

with children. The study discussed above is a logical step in a consistent program concerned with teachers' and students' thinking and the nature of instruction. Figure 6.9 provides a graphic representation of previous work. As indicated in Figure 6.9, the studies in this past program were linked. The link was both methodological and substantive. The studies focused on building understandings of instruction and in exploring the understandings of participants in the in­ structional setting. What is not evident are the other aspects or premises of this body of work. The work on perception was informed and based on the work of Piaget and Taha. The pro­ cedures used considered the students' conceptions and logical ways of thinking. Piaget (Inhelder & Piaget, 1958) showed that the use of indirect procedures could lead to an understanding of children's conceptions of phenomena. Taba's ( 1967) work on concept formation suggested developmental issues to be con­ sidered and systematic ways to use children in instruction as well as research. This work and the resulting procedures were tied to the theory of instruction being developed by Morine­ Dershimer (Morine, 1965). The cyclic nature of the question was derived from the work of Bellack et al. ( 1966). This work explored question-re­ sponse-reaction cycles from the perspective of the outside ob­ server. The Bellack work suggested that a focus on question cycles would be productive. The discussion above demonstrates how the procedures and substantive aspects of the triangulation study were related to previous work. This study, therefore, was partially theoretically driven by the theory of instruction being developed by

1 971 . Discovering New Dimensions i n the Teaching Process (Morine, Spauld­ ing, & Greenberg, 1 97 1 ; Morine & Morine, 1 973) Focused on classroom observa­ tion. Development of procedu res for hav­ ing teachers observe their own behavior. Focused on question cycles, particularly types of teacher reactions to pupil re­ sponses.

1 975. Interaction Analysis in the Classroom : Alterna­ tive Applications (Morine, 1 975)

1 975. Teacher and Pupil Perceptions of Classroom Interac­ tion (Morine & Val­ lance, 1 975)

Proposed alternative u n its of interactive analyses and dis­ plays of data to serve alternative purposes.

Short segments of interaction in les­ sons taught by dif­ ferent instructional methods, i n unfamil­ iar classrooms, were shown to pupils and teachers. Asked, "What did you see happening?" Com­ pared pupil and teacher observa­ tions.

Raised the ques­ tions: 1 . If students re­ ported what they ob­ served, what would the students' units be-how would they differ from those of sociolinguists? 2. Would alternative approaches to de­ scription of classroom language provide complemen­ tary find ings?

Fig. 6.9. H istorical perspective of work leading to the triangu lation study.

Raised the q ues­ tions: 1 . What differences would pupils ob­ serve in noninstruc­ tional settings (e.g., home, play)? 2. How would pupil and teacher observa­ tions compare when focused on their own classrooms?

OBSERVATION AS INQUIRY AND METHOD

Morine-Dershimer and partially by related work on concept development and formation.The procedures were also concep­ tually influenced by this theoretical frame. One final aspect of the match between the research approach and the theory in which it is grounded should be brought out. Morine-Dershimer built this program on the notion that a par­ ticular perspective on instruction grows out of the discipline in which it is grounded. In her case, the discipline was language. Therefore, the way in which she chose to look at language had to be related to the kind of knowledge within the discipline. Since she was concerned with language in use, she grounded her work and the framework for the study in sociolinguistics. Build­ ing on the work of Stubbs (1976) she developed the planned contrasts to explore how language worked in use and what par­ ticipants' perspectives were of this use. The triangulation study, therefore, was constructed within a particular theoretical framework. The framework served as a guide to all aspects of the study-the design of the study; the selection, modification, and development of instruments and procedures; and the analysis and interpretation of findings.

Example 4: An Ethnographic Program Example 4 focuses on the ethnographic research program of Cook-Gumperz, Gumperz, and Simons (1981) and Cook-Gum­ perz (in press). With this example, the discussion comes full cir­ cle and returns to the interactive-reactive cycle (Hymes, 1978) described by Corsaro (1985) presented at the beginning of this section on observation as inquiry. In contrast to Examples 1 through 3 in which the interactive-reactive aspect of the re­ search program occurred across studies, the interactive­ reactive research cycle occurs within this study and is a central part of the research cycle. That is, the researcher enters with a general question that leads to the use of specific instruments, schedules of data collection, interviews, and so forth. As the researcher becomes more familiar with the setting and begins to understand the way the group under study functions, the re­ searcher begins to focus on specific aspects of the ethnographic process. Foci are selected because they are recurrent phe­ nomena or events or they highlight some specific area of inter­ est (e.g., miscommunication, literacy, etc.). The cycle continues to be interactive-reactive: the questions are refined and new ones generated; new schedules are constructed; instruments are added or modified; interviews are added to obtain verification and validation, and so on. This cycle continues throughout the study (e.g., Corsaro, 1985; Heath, 1982; Pelto & Pelto, 1977; Spindler, 1982; for a synthesis of criteria and a discussion of issues related to ethnographic research in reading and other educational processes, see Green & Bloome, 1983). The studies by Cook-Gumperz, Gumperz & Simons (1981) reflect this interactive-reactive cycle. The entire cycle is pre­ sented in Figure 6.10. While the cycle appears linear in this graphic representation, the parts interact; that is, as one moves from left to right, a history develops. Each individual part (col­ umn) has a history; that is, it is informed by what preceded it. Decisions, therefore, at any point are grounded in prior deci­ sions and in the types of questions under study at any given point in the study. As indicated in the Corsaro (1985) quote presented earlier, the decisions are principled and part of the

201

demographic information about the study. The interrelation­ ship of the parts and the decision-making process will be high­ lighted in the discussion that follows. The historical context of this study is highlighted in Column 1, "Background of the Study." While this sequence of events frames this research, each of the researchers on this team had extensive experience on research in educational and other cul­ tural settings, such as home, community (e.g., Cook-Gumperz & Corsaro, 1977; Cook-Gumperz & Gumperz, 1976; Gumperz, 1982a, 1982b; Simons & Murphy 1981). The merger of these perspectives led to the exploration of classroom processes and communicative factors that influenced assessment of student ability and student performance. The multiple perspectives are reflected in Column 2, the "Mental Frame-Grid: Assumptions guiding/driving the collection, analysis, and interpretation of data." The perspective of this research team is reflected in the following representative questions for this study, The School/ Home Ethnography Project: 1. What are the continuities and discontinuities between classroom, playground, and home communication stra­ tegies? 2. How does what is said and done in the classroom influence the child's view of the learning process or goals of educa­ tion? 3. How does the mismatch in discourse understanding be­ tween home experience and that in the classroom lead to miscommunication? These questions indicate that the concern of this team was focused on the communicative environment and communica­ tive processes of school and their relationship to home and community. The approach that this team took is generally re­ ferred to as ethnography of communication (Gumperz & Hymes, 1972). This approach to ethnography is both topic cen­ tered (Hymes, 1978) and hypothesis oriented (Sanday, 1979); that is, the project selected a focus on the classroom and used discourse theory, developmental theory, and reading theory, among other areas to guide interpretation, collection, and anal­ ysis. This study, therefore, like the studies presented earlier in this section, is part of a program of research in which prior work is used to inform the current study. While this study has a specific conceptual and theoretical framework, this framework serves as a guide; it does not deter­ mine in advance what will be observed, collected, or counted. The determination of these factors comes from extended obser­ vation in the setting. The data collection phase took place over a 1-year period. Observation of recurrent phenomena and the expectations for participation led to the identification of a vari­ ety of loci for more specific observation. As indicated in the column entitled "Participation Observation," the observer en­ gaged in a variety of roles: participant observer, observer parti­ cipant, and aide. These roles allowed the researcher to obtain a variety of perspectives (e.g., insider and outsider) on the every­ day life and events of the classroom. By observing the teacher and being a teacher's aide, the observer was able to capture different aspects of the teacher's theory of pedagogy, her ration­ ale for organizing and managing events, and her plans and planning behavior. By focusing on the everyday events of the day from before school to after the children left, the observer

BACKGROUND DF THE STUDY 1965-:1977

s

1965-after the first year of Head S!art, the U.S. Office of Ed­ ucation convened a small confer­ ence of anthro­ pologist;, lin­ guists. psycholo­ gists, and sociolo­ gists to suggest priorities tor research on chiIdren·s language and its relationship to school success. Gumperz participated (Caz• den et al., 1972). 1966- Fottow-up confer• ence was held. The outcome of this conference was published in Functions of Lan­ guage in the Classroom (Caz­ den et al.. 1972). Gumperz and Hernandez­ Chavez published "'Bilingualism, Bi• dialectalism, and Classroom In­ teraction" in this volume. 1974-The National lnsli­ tute ol Education convened a group of panels of re­ searchers to set the research agenda !or re­ search on teach­ ing. Gumperz par­ ticipated in Panel 5, ··reaching as a Linguistic Pro­ cess in a Culturar Setting·· (NIE, 1974). 1977-The NIE funded a set of proposals focused on the nature of "Teach­ ing as a Linguistic Process in a Cul­ tural Setting" (NIE, 1977) which included a study by Cook-Gum­ perz, Gumperz. & Simons (1978). the one over­ viewed in this section.

Fig. 6.10.

STAGE I: FRAMING AND PLANNING THE PROJECT COMPONENTS

Mental Frame-Grid: As­ sumption$ guiding/driv­ ing the colleclion, analy­ sis, and interpretation of data Assumptions are derived from theoretical and re­ search literature, which includes work on • discourse processes • conversational analysis • ethnography of communication . • classroom organ1zati6n • teaching learning processes ___ _ • adult child interac• lions • child language • cross-cultural communication • evaluation of performance • socialization • reading acquisition and development • metalinguistic awareness • sociolinguistics • cognitive development

Participant Observation Single observer assigned to each classroom. Ob­ server assumes three roles: 1. Participant observer: participates in events and observes during participationrecords information after event 2. Observer participant: primarily observesparticipates only if approached by students for help 3 Aide: acts as aide for teacher, helps stu• dents . . Each role provides a d11lerent view of events. This approach allows ob­ servers to assume an insider"s view at times and an outsider's view at other times. Ongoing in­ volvement provides time for informal interviewing, capturing developmental aspect of evenls, and es• tablishing a shared per­ spective of events with teacher and students.

STAGE II. DATA CDLLECTIDN-GENERAL PARTICIPANT OBSERVATION

Participant Observation I Teacher Planning Organization Participant Observer (PO) works with teacher before school year begins. before class at breaks, and after school as aide. PO is not trained teacher, is naive, and can ask questions on a ··real need to know" basis; this talk is ""talk for doing·· job as aide. Approach is used to observe: 1. teacher plans and planning behaviors 2. teacher organizational behaviors and prac­ tices 3. teacher theory of ped­ agogy

Explicit Definition of Be­ haviors Teacher definition of ac· lions available through question and answer sequences belween PO and teacher in PO's role of aide Implicit Oefimt1on of Be­ haviors Inferred from practice and from directions given to PO as aide. Can be made explicitly during informal interviewing as part of aide's role Implicit Definition of Be­ haviors

STAGE Ill. TOPIC­ CENTERED PARTICIPANT OBSERVATION

STAGE IV· INFERENCING/ HYPOTHESIS GENERATING

STAGE V NATURAL EXPERIMENTS

Participant Observation Ill: Observe for contrastive situations/signs ol dis­ crepancy

Observe discourse strat­ egies used within the context

Plan and execute natural experiments that permit contrast of observed phenomena with similar phenomena in controlled or contrastive seltings

t Look for instances of differential learni�g 2. Look for events of day that reflect problem. Best site is event with high incidence of problems (e.g .. mis­ communication) 3. Look for atypical hap­ penings within typical events 4. Begin to predict type ol event that will occur. not specific event

Participant Observation II: Classroom Processes Practices Observer participant of total day to ascertain: 1. formal segments of Inferred from sequences day of behavior. from actions 2. ways in which teacher of teacher and students fram�s activi---working with each ties/events others' behaviors and 3. orchestration of observation of contextuevents alization cues. Participant observation of events to help stu­ dents and to ascertain 1. segments of events 2. conflict/contrast points 3. expectations for be• havior Basis of segmentation: contextualization cues and participation st�uc· tures. observable behav­ iors.

Participant Observation IV: Observe and videotape select events Observe participation structures and obtain rights and obligations for participation in events Observe verbal signals and conventions or for­ mulaic/ritualistic uses of language Observe target individu­ als selected during earli· er stages and who permit observation of contras­ tive behaviors

Observe indications of evaluation of student performance (verbal and nonverbal) Observe students and identify differential per­ formance and treatment within and across set­ tings Observe recurrent events and begin to predict oc­ currence of types of events

I Science laboratory Lawrence Hall of Sci­ ence E)(plore whether the difference in partici­ pation selfing and structure produce differences in per­ formance II. Pear stories (Chafe): Explore narrative production (oral & written) 1n control sit­ uation with naturally occurring narratives in classroom (e.g .. sharing time) IH Storytelling task: Fur­ ther explore students narrative abilities and differences in narrative style among groups of stu­ dents of different lan­ guage traditions IV. Referential com­ munication task/phonemic per­ ception task: Explore students' ability to use decontextualized language and con­ trast to reading-skill performance V. Home data collec­ tion: Collect data on narrative events in home. Work with parents to select events, have parents tape events {no PO), suggest events.

Design of Cook-Gumperz, Gumperz, and Simon's School/Home Ethnography Project and Green's Framing and Planning Components. Part 2 of figure (Stages

from "Research on Teaching as a Linguistic Process: A State of the Art" by permission.

I-IV)

J. Green (1983b). Copyright 1983 by the American Educational Research Association. Reprinted by

OBSERVATION AS INQUIRY AND METHOD

was able to obtain a picture of the segments of the day (tempo­ ral arrangement); of the ways in which the teacher framed, es­ tablished, checked, maintained, suspended, and reestablished, activities, norms and so forth within and across days; and the ways in which teacher and students orchestrated and nego­ tiated daily life. The observer had a "real need to know" in that she was a member of this social group and had roles to play in different situations. This general observation phase led to the selection of a more focused perspective within the general perspective. That is, as the participant observer continued to collect, analyze, and in­ terpret data, she and the other members of the research team, acting as a debriefing and analysis team, began to (a) identify recurrent patterns and phenomena; (b) generate questions for further study and hypotheses to test in later events; and (c) select new loci for observation. The latter loci would be cap­ tured on videotape as well as in field notes. Prior to this, in­ formation and descriptions were primarily obtained from field notes and observer records of various kinds. The videotape per­ mits in-depth exploration of a preselected type of event over­ time. This type of record permits exploration of communicative processes involved in the event. Given the general questions listed above, the use of videotape is an important data collec­ tion procedure for it permits talk that is abstract in nature to be "frozen." That is, once talk has occurred it is not possible to remember or reconstruct how it was delivered. People fre­ quently know what they meant, but rarely can retrieve how they said it. The frozen record, therefore, permits in-depth anal­ ysis of what talk occurred, where, when, to whom, in what ways, and for what purpose. It also permits examination of multiple perspectives (e.g., teacher, student, group, parent) and the ways in which factors such as frame of reference, linguistic abilities, expectations, and other contextualization factors (e.g., Cook­ Gumperz, 1976; Corsaro, 1981) influence participation and communicative competence (Hymes, 1974). The cycle described here begins with general participant ob­ servation (from passive to fully active). While these different types of observation continue throughout the project, a second type of data collection has been added, the collection of video­ tape records of selected phenomena and events. The decision about collection stems from the earJier observations and the nature of the general questions under study. As the research team gains greater and greater knowledge of the setting and social life in this setting, the questions being asked are refined and other questions are identified. Each question leads to a series of decisions about who and what to observe, how, when, where and for what purpose. One decision influences other de­ cisions. New observation schedules within the general schedule of the school day are constructed; instruments are added; and procedures are refined, modified, and added to permit the re­ searcher to obtain adequate information. Each decision is prin­ cipled (Heath, 1982). Analysis is ongoing as is collection and interpretation. The participants' perspectives are obtained from informal interviews (e.g., as teacher aide) and from observing spontaneous comments about the events and phenomena as they occur. Specific collection periods co-occur with the more general collection procedures. The cycle did not stop with the selection of specific events or phenomena to explore. One of the primary recurrent patterns

203

observed had to do with the nature and function of various discourse processes (language in use). The research team decid­ ed that one process, narrative performance, would be selected as the unit of analysis. This unit was selected because it was readily identifiable and because by selecting this unit as the lo­ cus of observation the contexts in which it occurred could be allowed to vary. Therefore, the question of functional equiva­ lence of context (Erickson & Shultz, 1981; Florio & Shultz, 1979) was not a problem for this research team. In other words, narrative was the stable unit and all other aspects were permit­ ted to vary; the way in which factors of context influenced nar­ rative performance could then be explored (e.g., the way in which the teacher accepted or sanctioned storytelling during group discussion or sharing time). The selection of narrative as the unit of observation also permitted exploration of differences in language use and functioning at home, in school, and on the playground. The selection of narrative as the unit of observation also led to the inclusion of a set of natural experiments and experimen­ tal studies within the general ethnography. This set of substud­ ies is presented in the last column of Figure 6.10. Each of these experiments was added so that different aspects of narrative performance could be explored to develop a more holistic pic­ ture of the students' narrative ability and knowledge. This in­ formation could then be contrasted with performance in everyday contexts of home and school. The ways in which teacher-student interactions supported, constrained, and/or overlooked these abilities could also be explored, as could the consequences for students of such actions. For example, find­ ings in this project showed a difference in teacher interactions with high- and low-reading-group students (Collins, 1983), as well as differences in positive and negative sanctions of stu­ dents' "sharing time" narratives (Michaels & Cook-Gumperz, 1979; Michaels, 1981). Narrative performance was shown to vary with contextual demands, with audience, and with task; narrative was also shown to be culturally patterned, and such patterns, when they did not match the expected pattern of the teacher, often led to negative assessment of ability and to sanc­ tions of student performance (Cook-Gumperz, in press).

Conclusions and Beginnings These representative research studies illustrate how multiple perspective approaches have evolved over the last decade. They show how decision making is both interactive and reactive. That is, changes in studies, selection of locus of observation, selection of settings, refinement of questions, and generation of new questions come from prior work, observed needs, unex­ plained problems or recurrent patterns within and across studies (e.g., management and organization as a locus of observation in the Evertson et al. studies; narrative as the locus of observation in the Cook-Gumperz, Gumperz, & Simons (1981) study). Each of the studies described in this section is part of a program of research. Each has a history. This history, as shown above, must be understood in order to understand the reason for the study under consideration, the procedures se­ lected, the locus of observation, and so forth. Another way to think about the relationship of one study to previous ones with­ in a program of research is that the individual study is part of a

204

CAROLYN M. EVERTSON AND JUIJITH L. GREEN

I

PERSONAL THEORY PHASE Identify the problem goal

/

What questions are pressing to me? Are they meaningful? Specify issues, questions, phenomena that bother, intrigue or make me curious.

-

I

What can I tap from others' experiences that might help? How have they been dealt with?

\[This section forms the intuitive rationale] PRACTICAL AND THEORETICAL RATIONALE Delimit specifically THE GROUNDING

PRACTICE

PAST WORK

THEORETICAL FRAME

What do I know about the topic from my own experience, work?

What other questions have been asked that might relate to mine?

What is the groundwork underlying my question?

What is required in state and local curriculum guides? What are "real time" constraints (e.g., curricula, schedules, etc.)?

What research exists to build on?

Identify related theories.

What topics relate to my topic? (Identify scope.)

[Contrast these three components] REDEFINED THEORETICAL FRAME [Obtained by considering:]

Define assumptions about the subject that lead to questions. Explicitly state assumptions about phenomena and about present knowledge. Identify own assumptions in relation to the above set.

Determine whether I have hypotheses or questions to guide the observation. Is the purpose to test hypotheses or to generate questions within the study? A PLACE TO BEGIN DESIGN

Define questions' possible starting places (locus of observation) Define terms; get descriptors; delineate State questions and rationale clearly so that others can understand CLASSROOM REALITY

Fig. 6.11. A framework to guide decision making in observation.

larger process, even though on a general level the current study can stand alone. However, to understand what is being de­ veloped, the study must be considered in relationship to pre­ vious ones and to the related studies used to build the frame­ work of the study and the evolving program. For example, the need to explain findings that were counter to those predicted led to the construction of projects that used a variety of qualita­ tive and quantitative collection and analysis procedures. The reasons for the changes in methodology and questions would

(continued)

not be evident without an understanding of the past history of a study. The studies also demonstrated that qualitative and quantita­ tive approaches are complementary. Each serves a different purpose. The use of a specific approach depends on the purpose of the study, the questions asked, and the setting in which the observations are occurring. These studies also demonstrate that any one study captures only a slice of reality. In addition, they demonstrate that different collection tools and procedures

205

OBSERVATION AS INQUIRY AND METHOD Fig. 6.1 1 .

I

(continued) How can I conceptually identify factors that are important to the topic (e.g., teacher, students, time, materials, etc.)?

Define aspects of topic to be studied (e.g., content delivery, social processes) Identify components of the topic.

1

!

Define grade level and subject matter of interest. What is the sampling time frame (e.g., daily, weekly, yearly)?

I

�

Consider the perspectives or point of view to be used (e.g., teacher, student, researcher, a combination, etc.) Consider the levels of context. How micro or macro to look.

[Consider past research and how questions from past bear on present study]

�

I

/

[Rationale is now supported by theory and constrained by practice and personal assumptions]

l

�

/

l

STATE OPERATIVE FORMS OF QUESTIONS

HOW TO OBSERVE Define strategies with which to answer the question(s).

What is the total set of approaches from which to choose? Be constructive. Build on past work. Provide alternatives. How do you know the alternative is better, not just different? How do I show predictive power is better, and not just different?

What is important to collect?

What consultants are available?

Do I have a way to analyze what I am collecting?

What consultants will I need?

!

[At each step, consider the trade-offs]

CONCLUSIONS

What are the limitations of the study? What can (cannot) be said?

DEFINE THE NEXT STUDY TO BE DONE AFTER COMPLETION OF PRESENT STUDY

produce different and often complementary pictures of the observed phenomena. As background for these studies, a description of the general nature of observation was presented in the first section of the chapter, which dealt with observation as method. A wide vari­ ety of factors involved in engaging in observational research were presented: • • • • • •

observation as a means of representing reality issues of selectivity observation as a contextualized process systems for recording and storing observational data units of observation and aggregation of data issues in sampling

• representative sources of error; limits on certainty in obser­ vation • toward criteria for adequate observation These factors identified the issues and defined the nature of the process of observation from a generic perspective. The two sections, when considered together, demonstrate what is involved in using observation as a systematic approach to exploring events, processes, and phenomena that occur in schools and other educational settings. In the remainder of this section, the question of how to begin will be considered. The framework that follows is meant to serve as a guide and will be presented in several formats. The formats of the framework are the product of three processes: (a) the interactions between the

206

CAROLYN M. EVERTSON AND JUDITH L. GREEN

authors of this chapter, who represent differing research ap­ proaches suggested some of the issues to be presented; (b) the review of research involved in preparing this chapter, which also suggested some of the issues ; and (c) other issues raised in research seminars on observing classroom processes at the Uni­ versity of Delaware and Ohio State University. The general in­ formation from Process 1 and 2 was presented in the previous section; therefore, the information and framework that define ways of considering where to begin and how to proceed, and that resulted from discussions with junior researchers, will now be considered. Once this framework has been considered, a summary framework will be presented. The first framework was the result of a two-part process. Part 1 involved generating a list of steps to be considered when try­ ing to design a study using observation. The participants of an observational research seminar at the University of Delaware, who had little previous experience with observation as an ap­ proach to studying teaching and learning processes (Pat Le­ fevre, Deborah Sardo, Deborah Smith, and Timothy Smith), generated the list presented in Table 6.10. The steps in Table 6.10 are not presented in a specific order; they are presented as they were developed over time. In Part 2, the students took these steps, ordered them, and added to them to design a framework that reflected the types of decisions involved in de­ signing an observational study. Therefore, rather than present­ ing them in a fixed order, readers are invited to order these steps for themselves and to construct their own framework. These questions demonstrate the types of decisions that need to be considered. As the seminar participants began to consider each of these statements and questions and to organize them, the relationships between and among these topics became clearer. Figure 6. 1 1 shows how one student conceptualized the relationships between and among factors. This representation was selected because it demonstrates how a framework can be developed to approach the question of observation systemati­ cally as an inquiry process. The junior researcher ordered the questions and topics above within six general topics. These topics move from a general identification of the question and what is in the field through a series of steps that help select the locus and make systematic decisions about what to observe, to the issue of how to observe. Observation methods are selected after decisions are made about whom to observe, what to observe, when, where, and for what purpose. Once these decisions are made on either a gen­ eral or a specific level, how the observations will be made must be determined. This cycle is not rigid; rather it provides a set of steps to guide decision making. The cycle can be used both be­ fore and throughout the study. As indicated above, while the steps are viewed linearly, the components interact with each other both within and across steps. The diagram in Figure 6. 1 1 illustrates the variety of decisions that must be made. This framework suggests that the grounding of a study is not simple. It is mediated by personal theory and assumptions, theory and assumptions of those in the situations (practice issues), and theory and assumptions about the phenomena and process obtained from the literature on past work. In other words, the development of a framework for a particular study or the identification of the grounding of a par­ ticular study requires a form of triangulation. In addition, the

Table 6.10. Steps to be Considered i n Conducting an O bservational Research Study Define q uestion : What q uestions are pressi ng to m e 1 Are they meaningfu l ? Define terms. Defi ne assu m ptions about the subject to be observed. Locate past research and consider how this work bears on the q ues­ tion. Take a constructive perspective. Go beyond criticism to consider alternatives to overcome problems of past work or issues not considered. Ask : how is the present or p roposed approach bet­ ter? Think about how to show predictive power of the alterna­ tive approach. Define grade level and subject matter of interest. Define aspects of the topic (e.g., content, del ivery, social aspects) that are to be studied. What d i mension of the q u estion will be looked at in what time frame (e.g., weekly, daily, yearly)? Defi ne strategies with which to answer q u estions. Conside r : 1 . Which have been used previously! 2. Which are available for use1 3. What is im portant to collect! 4. How is data to be analyzed ? 5. Is special trai n i ng needed to use the approach or strategy? Defi ne next study to be done after this one. Consider the program of research. What are the l i m itations of the study1 What statements can or cannot be made 1 Specify the issues, q u estions, phenomena that bother, i ntrigue, and/ or excite curiosity. Identify the whole. Del i m it the topic specifically. Consider the perspectives or points of view from which to consider the q u estion (e.g., student, teacher, observer, combination, etc.). How can teacher, hour, materials, students, and so forth, which are i m portant to the topic, be conceptually related 1 Consider the levels of context (e.g., m icro or macro). Defi ne terms, and get descri ptors. Clearly defi ne and delineate topic. State q u estions and rationale clearly so that others can know what the q u estion is. Identify related theories. What topics relate to the specific topic? Identify the scope of informa­ tion needed to d e l i mit the topic fully. Explicitly state assu m ptions about the phenomena, about the knowl­ edge we have. What is the groundwork underlying my q uestion ? What other q u estions might relate t o the q uestion? Determine whether hypotheses exist to be tested or the pu rpose of the study is to generate q uestions to be tested with i n and across studies. What do I know about the topic? What are my own experiences! Work1 What is req ui red in state/local curricu l u m gu ides ? What are " real t i m e " constraints (e.g., curricula, schedules, ac­ cess, etc.) 1 What can be d rawn from others' experiences that might help (e.g., other teachers, friends, etc.) 1 At each step, what are the trade-offs for the decisions made1

framework suggests that as one makes different decisions the framework is modified, concepts become clarified, terms are de­ fined, the locus of observation and focus of the study are delin­ eated, and the limitations are identified. The selection of observation methods is made in light of the framework. These decision-making processes may be set a priori or can be used to guide actions throughout the study. While the sample frameworks above appear linear, the process is often reflexive and recursive. Therefore, one final

207

OBSERVATION AS INQUIRY AND METHOD

Where do I begin? What do I observe?

What is available? What do we know?

Develop framework

Gain access to situation Consider levels of access required (e.g., teacher, students, administration, school board, research committees, etc.)

What approaches are available?

Identify and refine questions and hypotheses

topic

Data searches

Theory/knowledge from literature

Select locus/lens of observation

Practical theory from members of situation/group

Select setting

Select methods

Select analysis procedures Pilot-observe in situ

Identify related

Personal theory­ assumptions and beliefs

Decide whom 11nd what to observe, when, where, and for what purpose.

Decide on researcher's role(s) (e.g., participant observation, observ­ er participant, etc.)

Consider criteria for adequate observation, nature of evidence, etc.

Other .

Select appropriate design (e.g., ethnographic, reflexive, experimental structure with observation as means of data collection, etc.-theory testing, theory generating, grounded theory, etc.)

Other . .

Fig. 6.12. A framework to guide decision making in observa tion: A summary.

framework will be presented. This framework is a synthesis of the previous work and the two main sections of this chapter and, as mentioned before, may be entered at different points. However, the different cells contained in this framework need to be considered when planning a systematic observational study and a program of research based on observation as the inquiry method. The framework in Figure 6.12 is a general one. The authors recognize that there are different approaches to research and that each investigator will engage in· observation in a different way. That is, the psychologist may observe one child in great depth or a small group of children to explore the nature of cog­ nitive development. In contrast, the anthropologist might study a social group over time. In contrast to both of these ap­ proaches, the educational psychologist might want to observe a large number of classrooms in order to develop a normative model of how teachers manage classrooms. Each type of study may use similar observational methods; however, they will not use them in the same way for the same purpose. The framework is a place to begin; it is not a fixed guide as indicated by the

empty cells. We invite you to add to this framework, to modify it, and to restructure it. It is meant to be a heuristic device to help the person interested in using observation design a system­ atic study or program of research.

REFERENCES Adelman, C. (198 1). Uttering, muttering, collecting, using, and reporting talk for social educational research. London : Frank McIntyre. Adelman, C., & Walker, R. (1975). Developing pictures for other frames: Action research and case study. In G. Chanan & S. Dela­ mont (Eds.), Frontiers of classroom research. East Windsor, Berk­ shire, England: National Foundation for Educational Research. Amidon, E., & Hough, J. (Eds.). ( 1967). Interaction analysis: Theory, research, and application. Reading, MA: Addison-Wesley. Anderson, L. (1981). Short-term student responses to instruction. Ele­ mentary School Journal, 82, 97- 108. Anderson, L., Evertsen, C., & Brophy, J. (1979). An experimental study of effective teaching in first grade reading groups. Elementary School Journal, 79, 193-223. Au, K. (1980). Participation structures in a reading lesson with Hawaiian children. Anthopology and Education Quarterly, 11(2), 9 1 - 1 1 5.

208

CAROLYN M. EVERTSON AND JUDITH L. GREEN

Bales, R., & Strodtbeck, R. (1967). Phases in group problem-solving. In E. Amidon & J. Hough (Eds.) Interaction analysis: Theory, research, and application. Reading, MA: Addison-Wesley. Barker, R. G. (1963). The stream of behavior. New York: Appleton­ Century-Crofts. Barker, R. G. (1968). Ecological psychology: Concepts and methods for studying the environment ofhuman behavior. Stanford, CA: Stanford University Press. Barker, R. G., & Wright, H. (1955). Midwest and its children. New York : Harper & Row. Barnes, D., Britton, J., & Rosen, H. (1971). Language, the learner, and the school (rev. ed.). Harmondsworth, Middlesex, England : Penguin. Barnes, D., & Todd, F. (1977). Communication and learning in small groups. London: Routledge & Kegan Paul. Barr, R. (in press). Classroom interaction and curricular content. In D. Bloome (Ed.), Literacy, language, and schooling. Norwood, NJ: Ablex. Barr, R., & Dreeban, R. (1983). How schools work. Chicago : University of Chicago Press. Becker, H. (1970). Sociological work. Chicago : Aldine. Bellack, A., Hyman, R., Smith, F., & Kliebard, H. (1966). The language ofthe classroom. New York: Columbia University Press. Berliner, D. (1976). Impediments to the study of teacher effectiveness. Journal of Teacher Education, 27(1), 5-13. Berliner, D. (1978). Clinical studies of classroom teaching and learning. Paper presented at the annual meeting of the American Educational Research Association, Toronto. Berliner, D., & Koehler, V. (1983). Introduction [Special issue: Re­ search on teaching]. Elementary School Journal, 83(4), 261-263. Birdwhistell, R. (1970). Kinesics and context: Essays on body motion communication. Philadelphia: University of Pennsylvania Press. Blank, M. (1973). Teaching learning in the preschool. Columbus, OH: Charles E. Merrill. Bloome, D. (1981). An ethnographic approach to the study of reading activities among black junior high school students. Unpublished doctoral dissertation, Kent State University, Kent, OH. Bloome, D. (1983). Classroom reading instruction : A socio-communi­ cative analysis of time-on-task. Thirty-second yearbook of the National Reading Conference. Rochester, NY: National Reading Conference. Bloome, D., & Green, J. (1982). The social contexts of reading: A mul­ tidisciplinary perspective. In B. Hutson (Ed.), Advances in reading/ language research (Vol. !). Greenwich, CT: JAI Press, Inc. Bloome, D., & Green, J. (1984). Directions in the sociolinguistic study of reading. In D. Pearson, R. Barr, M. Kami), & P. Mosenthal (Eds.), Handbook of reading research. New York : Longman. Blumenfeld, P. C., Hamilton, V. S., Wessels, K., & Falkner, D. (1979). Teaching responsibility to first graders. Theory into Practice, 18, 171-180. Blumer, H. (1969). Symbolic interactionism: Perspective and method. Englewood Cliffs, NJ: Prentice-Hall. Blurton-Jones, N. (1972). In N. Blurton-Jones (Ed.), Ethological studies of child behavior. London: Cambridge University Press. Borman, K. (Ed.). (1982). The social life ofchildren in a changing society. Hillsdale, NJ: Lawrence Erlbaum Associates. Bossert, S. (1979). Tasks and social relationships in classrooms. Cam­ bridge: University Press. Bracht, G., & Glass, G. (1968). The external validity of experiments. American Educational Research Journal, 5(4), 437-474. Brannigan, C. R., & Humphries, D. H. (1972). Human non-verbal be­ havior: A means of communication. In N. Blurton-Jones (Ed.), Eth­ ological studies of child behavior. Cambridge: Cambridge University Press. Brattesani, K. A., Weinstein, R. S., & Marshall, H. H. (1984). Student perceptions of differential teacher treatment as moderators of teacher expectation effects. Journal of Educational Psychology, 76(2), 236-247. Brophy, J. (1973). Stability of teacher effectiveness. American Educa­ tional Research Journal, 10, 245-252. Brophy, J. (1979). Teacher praise: A functional analysis. Review ofEdu­ cational Research. 51. 5-32.

Brophy, J. (1983). If only it were true: A response to Greer. Educational Researcher, 12( 1 ), 10- I 3. Brophy, J., & Evertson, C. (1974). Process-product correlations in the Texas Teacher Effectiveness Project. (ERIC Document Reproduc­ tion Service No. ED 09 1 394) Brophy, J., & Evertson, C. (1976). Learning from teaching: A develop­ mental perspective. Boston: Allyn & Bacon. Brophy, J., & Evertson, C. (1978). Context variables in teaching. Educa­ tional Psychologist, 12, 310-3 16. Brophy, J., & Good, T. (1970a). The Brophy - Good Dyadic Interac­ tion System. In A. Simon & E. Boyer (Eds.), Mirrors for behavior: An anthology of observation instruments: 1970 supplement, Vol. A.

Philadelphia: Research for Better Schools. Brophy, J., & Good, T. (1 970b). Teachers' communication of differen­ tial expectations for children's classroom performance: Some behavioral data. Journal of Educational Psychology, 61, 365-374. Brophy, J., & Good, T. (1974). Teacher-student relationships: Causes and consequences. New York: Holt, Rinehart & Winston. Brunswik, E. (1956). Perception and the representative design ofpsycho­ logical experiments. Berkeley: University of California Press. Burstein, L. (1980). The analysis of multilevel data in educational re­ search and evaluation. In D. Berliner (Ed.), Review of research in education (Vol. 8). Washington, DC: American Educational Re­ search Association. Carey, R., Harste, J., & Smith, S. (1981). Contextual constraints and discourse processes: A replication study. Reading Research Quarterly, 16(2), 201-212. Cazden, C. (1976). How knowledge about language helps the classroom teacher, or does it? A personal account. The Urban Review, 9, 74-90. Cazden, C. (1983). Contexts for literacy: In the mind and in the class­ room. Journal of Reading Behavior, 14(4), 413-427. Cazden, C., John, V., & Hymes, D. (1972). Functions of language in the classroom. New York: Columbia University, Teachers College Press. Chafe, W. (1980). The pear stories. Norwood, NJ : Ablex. Cochran-Smith, M. (1984). The making of a reader. Norwood, NJ: Ablex. Cohen, E. (1979). Status equalization in the desegregated school. Paper presented at the annual meeting of the American Educational Re­ search Assocation, San Francisco. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educa­ tional and Psychological Measurement, 20, 37-46. Cole, M., Griffin, P., & Newman, D. (1979). They're all the same in their own way (Mid-quarter report, NIE No. G-78-01 59). Washington, DC: National Institute of Education. Coleman, J., et al. (1966). Equality of educational opportunity. Wash­ ington, DC: U.S. Government Printing Office. Collins, J. (1983). Linguistic perspectives on minority education : Dis­ course analysis and early literacy. Unpublished doctoral disserta­ tion, University of California, Berkeley. Collins, J. (in press). Using cohesion analysis to understand access to knowledge. In D. Bloome (Ed.), Literacy, language, and schooling. Norwood, NJ: Ablex. Cook-Gumperz, J. (1981). Persuasive talk: The social organization of children's talk. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings. Norwood, NJ: Ablex. Cook-Gumperz, J. (Ed.). (in press). The Social Construction ofLiteracy. New York: Cambridge University Press. Cook-Gumperz, J., & Corsaro, W. (1977). Social-ecological constraints on children's communicative competence. In J. Gumperz & J. Cook-Gumperz (Eds.), Papers on language and context (Working Paper No. 46). Berkeley : Language Behavior Research Laboratory. University of California. Cook-Gumperz, J., & Gumperz, J. (1976). Context in children's speech. In J. Gumperz & J. Cook-Gumperz (Eds.), Papers on language and context (Working Paper No. 46). Berkeley: Language Behavior Re­ search Laboratory. University of California. Cook-Gumperz, J., Gumperz, J., & Simons, H. (1978). School- home ethnography project (Proposal). Washington, DC: National Institute of Education. Cook-Gumperz, J., Gumperz, J., & Simons, H. (1979). Language at school and at home: Theory, methods, and preliminary findings

OBSERVATION AS INQUIRY AND METHOD

(Mid-quarter report, NIE No. G-78-0082). Washington, DC: National Institute of Education. Cook-Gumperz, J., Gumperz, J., & Simons, H. (1 981). School- home ethnography project (Final report, NIE No. G-78-0082). Wash­ ington, DC: National Institute of Education. Cooper, C., Ayres-Lopez, S., & Marquis, A. (1981). Children's discourse cooperative and didactic interaction : Developmental patterns of ef­ fective learning (Final report, NIE No. G-78-0098). Washington, DC: National Institute of Education. Cooper, R. (1968). How can we measure the roles which a bilingual's languages play in his everyday behavior? Proceedings of the Interna­ tional Seminar on the Measurement and Description of Bilingualism. Ottawa: Canadian Commission for UNESCO. Corsaro, W. (1981). Entering the child's world: Research strategies for field entry and data collection. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings. Norwood, NJ: Ablex. Corsaro, W. (1985). Friendship and peer culture in the early years. Norwood, NJ: Ablex. Delamont, S., & Hamilton, D. (1976). Classroom research: A critique and a new approach. In M. Stubbs & S. Delamont (Eds.), Explora­ tions in classroom observation. Chichester, England: John Wiley & Sons. Denham, C., & Lieberman, A. (Eds.). (1980). Time to learn. Wash­ ington, DC: National Institute of Education. Denzin, N. (1 977). The research act. Chicago : Aldine. DeStefano, J., & Pepinsky, H. (1981). The learning of discourse rules of culturally different children in first grade literacy instruction (Final report, NIE No. G-79-0032). Washington, DC: National Institute of Education. Doyle, W. (1977). Paradigms for research on teacher effectiveness. Review of Research in Education, 15, 163- 1 98. Duncan, S., & Fiske, D. (1977). Face-to-face interaction: Research, methods, and theory. Hillsdale, NJ: Lawrence Erlbaum Associates. Duncan, S., & Niederehe, G. (1974). On signalling that it's your turn to speak. Journal of Experimental Social Psychology, 10, 234-247. Dunkin, M., & Biddle, B. (1974). The study of teaching. New York: Holt, Rinehart & Winston. Eder, D. (1982). Differences in communicative styles across ability groups In L. Cherry Wilkinson (Ed.), Communicating in the class­ room. New York: Academic Press. Edwards. A., & Furlong, V. (1978). The language of teaching : Meaning in classroom interaction. London : Heinemann. Elliott, J. (1976). Developing hypotheses about classrooms from teachers' practical constructs. Grand Forks: North Dakota Study Group. Emmer, E., Evertson, C., & Anderson, L. (1 980). Effective classroom management at the beginning of the school year. Elementary School Journal, 80(5), 219-231 . Emmer, E., & Peck, R. (1973). Dimensions of classroom behavior. Journal of Educational Psychology, 64, 223-240. . ... Erickson, F. (1977). Some approaches to inquiry in school-community ethnography. Anthopology and Education Quarterly, 8(2), 58-69. Erickson, F. (1979). On standards of descriptive validity in studies of classroom activity (Occasional Paper No. 16). East Lansing: Michi­ gan State University, Institute for Research on Teaching. Erickson, F. (1982). Classroom discourse as improvisation: Relation­ ships between academic task structure and social participation structure in lessons. In L. Cherry Wilkinson (Ed.), Communicating in the classroom. New York: Academic Press. Erickson, F., Cazden, C., Carrasco, R., & Guzman, A. (1978- 1981). Social and cultural organization of interaction in classrooms of bilingual children (Mid-quarter report, NIE No. G-78-0099). Washington, DC : National Institute of Education. Erickson, F., & Schultz, J. (1981). When is a context? Some issues and methods in the analysis of social competence. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings. Norwood, NJ: Ablex. Erickson, F., & Wilson, J. (1 982). Sights and sounds of life in schools: A resource guide tofilm and videotape for research and education. East Lansing: Michigan State University, Institute for Research on Teaching. Evertson, C. (1984). Taking a broad look: Quantitative analyses of

209

classroom management training. Paper presented to the American Educational Research Association, New Orleans. Evertson, C., Anderson, C., Anderson, L., & Brophy, J. (1 980). Relation­ ships between classroom behaviors and student outcomes in junior high mathematics and English classes. American Educational Research Journal, 1 7(1), 43-60. Evertson, C., Anderson, L., & Brophy, J. (1978). The Texas Junior High School Study: Final report ofprocess-outcome relationships. Austin, TX: The University of Texas Research and Development Center for Teacher Education. (ERIC Document Reproduction Service No. ED 1 73 744) Evertson, C., Anderson, L., & Clements, B. (1 980). Elementary school classroom organization study: Methodology and instrumentation (Rep. No. 6002). Austin, TX: University of Texas Research and De­ velopment Center for Teacher Education. (ERIC Document Repro­ duction Service No. ED 205 486). Evertson, C., & Emmer, E. (1982). Effective management at the begin­ ning of the school year in junior high classes. Journal ofEducational Psychology, 74, 485-498. Evertson, C., Emmer, E., & Clements, B. (1 980). Report ofmethodology, rationale, and instrumentation of the Junior High School Classroom Organization Study. (Rept. # 6100) Austin, TX: The University of Texas Research and Development Center for Teacher Education (ERIC Document Reproduction Service No. ED 189 076). Evertson, C., Emmer, E., Sanford, J., & Clements, B. (1983). Improving classroom management: An experiment in elementary school class­ rooms. Elementary School Journal, 84, 173- 1 88. Evertson, C., & Holley, F. (1 981). Classroom observation. In J. Millman (Ed.), Handbook of teacher evaluation. Beverly Hills: Sage Publications. Evertson, C., & Veldman, D. (1981). Changes over time in process mea­ sures of classroom behavior. Journal of Educational Psychology, 73(2), 1 56-163. Fassnacht, G. (1982). Theory and practice of observing behavior. Lon­ don : Academic Press. Fisher, C., Berliner, D., Filby, N., Marliave, R., Cahen, L., & Dishaw, M. (1980). Teaching behaviors, academic learning time, and student achievement: An overview. In C. Denham & A. Lieberman (Eds.), Time to learn. Washington, DC: National Institute of Education. Fisher, C., Berliner, D., Filby, N., Marliave, R., Cahen, L., Dishaw, M., & Moore, J. (1978). Teaching and learning in elementary schools: A summary of the Beginning Teacher Evaluation Study. San Francisco, CA : Far West laboratory. Flanders, N. (1 967). Estimating reliability. In E. Amidon & J. Hough (Eds.), Interaction analysis: Theory, research, and application. Reading, MA: Addison-Wesley. Flanders, N. (1970). Analyzing teaching behavior. Reading, MA: Addison-Wesley. Florio, S., & Clark, C. (1982). The functions of writing in an elementary classroom. Research in the teaching of English, 16(2), 1 15-130. Florio, S., & Schultz, J. (1979). Social competence at home and at school. Theory into Practice, 18(4), 234-243. Frederiksen, C. (1975). Representing logical and semantic structure of knowledge acquired from discourse. Cognitive Psychology, 7, 371 -458. Frederiksen, C. (1 981). Inference in pre-school children's conversations: A cognitive perspective. In J. Green & C. Wallat (Eds.), Ethno­ graphy and language in educational settings. Norwood, NJ: Ablex. Frick, T., & Semmel, M. (1978). Observer agreement and reliabilities of classroom observational measures. Review of Educational Research, 48(1), 1 57-184. Gage, N. L. (Ed.). (1963a). Handbook of research on teaching. Chicago: Rand McNally. Gage, N. L. (1963b). Paradigms for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally. Galloway, C. (1971). Analysis of theories and research in nonverbal communication. Journal of the Association for the Study of Percep­ tion, 6(2), 1 -21. Galloway, C. (1 984). Nonverbal and teacher- student relationships: An intercultural perspective. Manuscript submitted for publication. Garnica, 0., & King, M. (Eds.). (1979). Language, children, and society. Elmsford, NY: Pergamon.

210

CAROLYN M. EVERTSON AND JUDITH L. GREEN

Genishi, C. (1983a). Observational methods for early childhood educa­ tion. In B. Spodek (Ed.), Handbook of research in early childhood education. New York: Free Press. Genishi, C. (1983b). Studying classroom verbal interaction. Paper pre­ sented to the National Conference for Research in English session at the annual meeting of the American Educational Research Associa­ tion, Montreal. Genishi, C., & Di Paolo, M. (1982). Leaming through argument in a preschool. In L. Cherry Wilkinson (Ed.). Communicating in the classroom. New York : Academic Press. Gilmore, P. (in press). Sulking, stepping, and tracking: The effects of attitude assessment on access to literacy. In D. Bloome (Ed.), Literacy, language, and schooling. Norwood, NJ: Ablex. Gilmore, P., & Glatthorn, A. (Eds.). ( 1982). Children in and out of school: Ethnography and education. Washington, DC: Center for Applied Linguistics. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory. Chicago : Aldine. Good, T. (1983). Uses and misuses of classroom observation. In W. Duckett (Ed.), Observation and the evaluation of teaching. Bloom­ ington, IN: Phi Delta Kappa. Good, T., Biddle, B., & Brophy, J. ( 1975). Teachers make a difference. New York : Holt, Rinehart & Winston. Good, T., Cooper, H., & Blakey, S. ( 1980). Classroom interaction as a function of teacher expectations, student sex, and time of year. Journal of Educational Psychology, 72, 378-385. Gordon, E. (Ed.). ( 1983). Review of research in education (Vol. 10). Washington, DC: American Educational Research Association. Gordon, I., & Jester, R. (1973). Techniques of observing teaching in early childhood and outcomes of particular procedures. In R. M. W. Travers (Ed.), Second handbook of research on teaching. Chicago: Rand McNally. Green, J. (1983a). Exploring classroom discourse: Linguistic perspec­ tives on teaching-learning processes. Educational Psychologist, 18(3), 180- 199. Green, J. (1983b). Research on teaching as a linguistic process: A state of the art. In E. Gordon (Ed.), Review of research in education (Vol. 10). Washington, DC: American Educational Research Association. Green, J., & Bloome, D. (1983). Ethnography and reading: Issues, ap­ proaches, criteria, and findings. Thirty-second-yearbook of the National Reading Conference. Rochester NY: National Reading Conference. Green, J., & Harker, J. (1982). Gaining access to learning: Conversa­ tional, social, and cognitive demands of group participation. In L. Cherry Wilkinson (Ed.), Communicating in the classrooms. New York: Academic Press. Green, J., Harker, J., & Walla!, C. (Eds.). (in press). Multiple perspec­ tives analysis in classroom discourse. Norwood, NJ: Ablex. Green, J., & Wallat, C. (Eds.). ( 1979). What is an instructional context? An exploratory analysis of conversational shifts over time. In 0. Garnica & M. King (Eds.), Language, children, and society. New York: Pergamon. Green, J., & Wallat, C. (Eds.). (1981a). Ethnography and language in educational settings, Norwood, NJ: Ablex. Green, J., & Wallat, C. (198 1 b). Mapping instructional conversations. In J. Green & C. Wallat (Eds.), Ethnography and language in educa­ tional settings. Norwood, NJ: Ablex. Green, J., & Weade, R. (1984). Taking a closer look: Qualitative explor­ ations of task management. Paper presented at the meeting of the American Educational Research Association, New Orleans. Griffin, P. (1977). How and when does reading occur in the classroom? Theory into Practice, 16(5), 376-383. Gump, P. (1969). Intra-setting analysis: The third grade classroom as a special but instructive case. In E. Willems & H. Raush (Eds.), Natur­ alistic viewpoints in psychological research. New York: Holt, Rinehart & Winston. Gumperz, J. ( 1982a). Discourse strategies. Cambridge: Cambridge University Press. Gumperz, J. ( 1982b). Language and social identity. Cambridge: Cam­ bridge University Press. Gumperz, J., Cook-Gumperz, J., & Simons, H. ( 1978). School/home eth-

nography project. Proposal to the National Institute of Education.

Unpublished manuscript, University of California, Berkeley. Gumperz, J., & Hernandez-Chavez, E. ( 1972). Bilingualism, bidialect­ alism, and classroom interaction. In C. Cazden, V. John, & D. Hymes (Eds.), Functions of language in the classroom. New York: Columbia University, Teachers College Press. Gumperz, J., & Hymes, D. ( 1972). Directions in sociolinguistics. New York : Holt, Rinehart & Winston. Hall, E. T. (1966). The hidden dimension. New York: Doubleday. Hamilton, D. (n.d.). Some contrasting assumptions about survey analysis and case study research. Unpublished manuscript, University of Glasgow, Department of Education. Hamilton, S. (1983). The social side of schooling: Ecological studies of classrooms and schools. Elementary School Journal, 83(4), 3 13-334. Hammersley, M. ( 1974). The organisation of pupil participation. Socio­ logical Review, 22, 355-368. Hansen, J. F. (1979). Sociocultural perspectives on human learning: An introduction to educational anthropology. Englewood Cliffs, NJ: Pre­ ntice-Hall. Harker, J. (in press). Contrasting the content of two story reading lessons: A propositional analysis. In J. Green, J. Harker, & C. Wallat (Eds.), Multiple perspectives analysis of classroom discourse. Norwood, NJ: Ablex. Harper, R. G., Wiens, A. N., & Matarazzo, J. D. (1978). Nonverbal com­ munication : The state of the art. New York : John Wiley. Heap, J. (1980a). Description in ethnomethodology. Human Studies, 3, 87-106. Heap, J. ( 1980b). What counts as reading: Limits to certainty in assess­ ment. Curriculum Inquiry, 10(3), 265-292. Heap, J. ( 1982). Understanding classroom events: A critique of Durkin, with an alternative. Journal of Reading Behavior, 14(4), 391 -41 1 . Heap, J . (1983). On task in discourse: Getting the right pronunciation. Paper presented at the annual meeting of the American Educational Research Association, Montreal. Heath, S. B. ( 1982). Ethnography in education: Defining the essentials. In P. Gilmore & A. Glatthorn (Eds.), Children in and out of school. Washington, DC: Center for Applied Linguistics. Heath, S. B. (1983). Ways with wnrds: Language, life, and work in com­ munitieiand classrooms. Cambridge: Cambridge University Press. Herbert, J., & Attridge, C. ( 1975). A guide for developers and users of observation systems and manuals. American Educational Research Journal, 12(1), 1 -20. Hough, J. B. (Ed.). ( 1980a). Computer processing of data/or the study of instruction. Columbus, OH: The Ohio State University. Hough, J. B. (Ed.). (1980b). Concepts and categories for the study of instruction. Columbus, OH: The Ohio State University. Hough, J. B. (Ed.). (1980c). Data displays and their interpretation for the study of instruction. Columbus, OH: The Ohio State University. Hymes, D. (1974). Foundations in sociolinguistics: An ethnographic ap­ proach. Philadelphia: University of Pennsylvania Press. Hymes, D. (1977). Critique. Anthropology and Education Quarterly. 8(2), 9 1 -93. Hymes, D. (1978). What is ethnography? (Sociolinguistic Working Paper No. 45). Austin, TX: Southwest Educational Development Laboratory. Hymes, D. (1981). Ethnographic monitoring project, (Final report, NIE No. G-78-0038). Washington, DC: National Institute of Education. lnhelder, B., & Piaget, J. (1958). The growth of logical thinking from childhood to adolescence, New York: Basic Books. Jencks, C., et al. (1972). Inequality: A reassessment ofthe effect offamily and schooling in America. New York: Basic Books. Johnson, M. C. ( 1979). Discussion dynamics. Rowley, MA: Newbury House. Johnson, S., & Bolstad, 0. (1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In L. Hammerlynk, L. Handy, & E. Mash (Eds.), Behavior change. Cham­ paign, IL: Research Press. Kimmel, A. (Ed.). (1981). Ethics of human subject research. San Fran­ cisco : Jossey-Bass. Knapp, M. L. ( 1978). Nonverbal communication in human interaction. New York: Holt, Rinehart & Winston.

• OBSERVATION AS INQUIRY AND METHOD

Koehler, V. (1978). Classroom process research: Present and future. Journal of Classroom Interaction, 13(2) 3-10. Koehler, V. (in press). Issues in secondary analysis [tentative title]. In J. Green, J. Harker, & C. Walla! (Eds.), Multiple perspectives analysis of classroom discourse. Norwood, NJ: Ablex. Kugle, C. (1978). A potential source of bias in classroom observation systems: Coder drift. Paper presented at the annual meeting of the American Educational Research Association, Toronto. (ERIC Document Reproduction Service No. ED 1 59 203) Langer, I., Schultz, V., & Thun, F. ( 1974). Messung komplexer Merk­ male in Psynchologie and Piidagogik. Ratingverfahren. Munich: Reinhardt. Le Compte, M., & Goetz, J. (1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52(1), 3 1 -60. Light, R. J. (1971). Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychological Bulletin, 76(5), 365-377. Little, J. W. (198 1). School success and staff development : The role of staff development in urban desegregated schools (Final report, NIE Contract No. 400-79-0049). Boulder, CO: Center for Action Re­ search. Little, J. W. (1982). Norms of collegiality and experimentation: Work­ place conditions of school success. American Educational Research Journal, 19(3), 325-340. Lorenz, K. (1981). The foundations of ethology: The principal ideas and discoveries in animal behavior. New York: Touchstone Books. Lundgren, U. (1977). Model analysis of pedagogical processes. Stock­ holm: Stockholm Institute of Education, Department of Educa­ tional Research. Marshall, H. H. (1972). Criteria for an open classroom. Young children, 28, 1 3-20. Marshall, H. H. (1976a). Dimensional Occurrence Scale: Manual. Berkeley: University of California, School of Education. Marshall, H. H. ( 1976b). Dimensions of classroom structure and func­ tioning project: Final report. Berkeley: University of California. Marshall, H. H. (1981). Open classrooms: Has the term outlived its usefulness? Review of Educational Research, 51, 1 8 1- 192. Marshall, H. H., & Green, J. (1978). Context variables and purpose in the use of verbal interaction. Journal of Classroom Interaction, 14(2), 24-29. Marshall, H. H., Hartsough, C., Green, J., & Lawrence, M. (1977). Stability of classroom variables as measured by a broad range observational system. Journal of Educational Research, 70(6), 304-3 1 1. Marshall, H. H., & Weinstein, R. (1982). Classroom Dimensions Obser­ vation System : Manual. Berkeley: University of California, Psy­ chology Department. Marshall, H. H., & Weinstein, R. (1984). Classrooms where students per­ ceive high and low amounts of differential teacher treatment. Paper presented at the annual meeting of the American Educational Re­ search Association, New Orleans. Marshall, H. H., & Weinstein, R. (1984). Classroom factors affecting student self evaluation: An interactional model. Review of Educa­ tional Research, 54(3), 301-325. Marshall, H. H., Weinstein, R. S., Sharp, L., & Brattesani, K. A. (1982). Students' descriptions of the ecology of the school environment for high and low achievers. Paper presented at the meeting of the

American Educational Research Association, New York. Marston, P., & Zimmerer, L. (1979). The effect of using more than one type of teacher observation system in the same study. (Report No. 5070). Austin: The University of Texas Research and Development Center for Teacher Education. Martin, J., Veldman, D., & Anderson, L. (1980). Within-class relation­ ships between student achievement and teacher behaviors. Ameri­ can Educational Research Journal, 1 7(4), 479-490. May, L. (1981). Spaulding school: Attention and styles of interaction. In D. Hymes (Ed.), Ethnographic monitoring project (Final report, NIE No. G-78-0038). Washington, DC: National Institute of Education. McCall, G., & Simmons, J. ( 1969). Issues in participant observation. Reading, MA: Addison-Wesley.

211

McCutcheon, G. ( 1981). On the interpretation of classroom observa­ tions. Educational Researcher, 10(5), 5-10. McDermott, R. (1976). Kids make sense: An ethnographic account of interactional management of success and failure in one first-grade classroom. Unpublished doctoral dissertation, Stanford University,

Stanford, CA. McDermott, R., & Hood, L. (1982). Institutionalized psychology and the ethnography of schooling. In P. Gilmore & A. Glatthorn (Eds.), Children in and out of school. Washington DC: Center for Applied Linguistics. McGaw, B., Wardrop, J., & Bunda, M. ( 1972). Classroom observation schemes: Where are the errors? American Educational Research Journal, 9(1), 1 3-27. Mead, M. (1975). Blackberry winter: My early years. New York : Pocket Books. Medley, D., & Mitzel, H. (1963). Measuring classroom behavior by sys­ tematic observation. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago : Rand McNally. Medley, D., Soar, R., & Coker, H. ( 1984). Measurement based evaluation of teacher performance. New York: Longman. Mehan, H. ( 1979a). Learning lessons. Cambridge, MA: Harvard Uni­ versity Press. Mehan, H. (1979b). What time is it, Denise? Asking known information questions in classroom discourse. Theory into Practice, 18(4), 285-294. Merritt, M. (1982). Distributing and directing attention in primary classrooms. In L. Cherry Wilkinson (Ed.), Communicating in class­ rooms. New York: Academic Press. Merritt, M., & Humphrey, F. (1981). Service-like events during the indi­ vidual work time and their contribution to the nature of communica­ tion in primary classrooms (NIE No. G-78-01 59). Washington, DC:

National Institute of Education. Michaels, S. (1981). Sharing time: Children's narrative styles and differ­ ential access to literacy. Language in Society, 10(3), 423-442. Michaels, S., & Cook-Gumperz, J. ( 1979). A study of sharing time with first grade students: Discourse narratives in the classroom. In Pro­ ceedings of the Berkeley Linguistic Society, 5, 87-103. Miller, J. (1974). The nature of living systems: An overview. In Inter­ disciplinary aspects of general systems theory. Proceedings of the third annual meeting of the Middle Atlanzic Regional Division. Col­

lege Park, MD: Society for General Systems Research. Mishler, E. (1984). The discourse of medicine. Norwood, NJ: Ablex. Moos, R. (1976). The human context: Environmental determinants of behavior. New York: John Wiley. Morine, G. (1965). A model of inductive discovery for instructional theory. Unpublished doctoral dissertation, Columbia University, Teachers College. Morine, G. (1969). Discovery modes: A criterion for teaching. Theory into Practice, 8(1 ), 25-30. Morine, G. (1975). Interaction analysis in the classroom: Alternative applications. In R. Weinberg & F. Wobds (Eds.), Observation of pupils and teachers in mainstream and special education settings: Alternative strategies. Minneapolis: University of Minnesota Press. Morine, G., Spaulding, R., & Greenberg, S. (1971). Discovering new di­ mensions in the teaching process. Toronto: Intext. Morine, G., & Vallance, E. (1975). Teacher and pupil perceptions of classroom interaction. Beginning Teacher Evaluation Study (Phase

B). San Francisco: Far West Laboratory. Morine, H., & Morine, G. (1973). Discovery: A challenge to teachers. Englewood Cliffs, NJ : Prentice-Hall. Morine-Dershimer, G. ( 1981). The literate pupil, The J. Richard Street Lecture Series. Syracuse, NY: Syracuse University Press. Morine-Dershimer, G. ( 1982). Pupil perceptions of teacher praise. Ele­ mentary School Journal, 82(5), 421-434. Morine-Dershimer, G. (in press-a). Comparing systems: How do we know? In J. Green, J. Harker, & C. Walla! (Eds.), Multiple perspec­ tive analysis ofclassroom discourse. Norwood, NJ: Ablex. Morine-Dershimer, G. (in press-b). Talking, listening, and learning in elementary school classrooms. New York: Longman. Morine-Dershimer, G., & Tenenberg, M. (1981). Participant per­ spectives of classroom discourse (Executive summary, NIE No. G-78-01_61). Washington, DC: National Institute of Education.

212

CAROLYN M. EVERTSON AND JUDITH L. GREEN

Morine-Dershimer, G., Tenenberg, M., & Shuy, R. (1981). Why do you ask? (Final report, Part 2, NIE No. G-78-0162). Washington, DC: National Institute of Education. Namuddu, C. (1982). The structure of classroom communication in some biology lessons in selected secondary schools in Kenya. Proposal to the International Development Research Center, Ottawa. National Institute of Education. (1974a). Reports ofpanels 1-10 of the National Conference on Studies in Teaching. Washington, DC: U.S. Office of Education. National Institute of Education. (1974b). Teaching as a linguistic pro­ cess in a cultural context (Panel 5 report). Washington, DC: U.S. Office of Education. Noblit, G. W. (1982). Not seeing the forest for the trees: The failure of synthesis/or the desegregation ethnographies. Paper presented at the meeting of the American Educational Research Association, New York. Ober, R., Bentley, E., & Miller, E. (1971). Systematic observation of teaching: An interaction analysis-instructional strategy approach.

Englewood Cliffs, NJ: Prentice-Hall. Ochs, E. (1979). Transcription as theory. In E. Ochs and B. Schieffelin (Eds.), Developmental pragmatics. New York: Academic Press. Ogbu, J. (1981). Cultural ecological contexts of classroom interactions. Paper presented at the annual meeting of the American Educational Research Association, Los Angeles. Osgood, C. E., Suci, G. T., & Tannenbaum, P. H. (1957). The measure­ ment of meaning. Urbana-Champaign: University of Illinois Press. Pelto, P., & Pelto, G. (1977). Anthropological research : The structure of inquiry. New York: Harcourt Brace. Philips, S. (1 972). Participant structures and communicative compe­ tence: Warm Springs children in community and classroom. In C. Cazden, V. John, & D. Hymes (Eds.), Functions of language in the classroom. New York: Columbia University, Teachers College Press. Philips, S. (1982). The invisible culture: Communication in classroom and community on the Warm Springs Indian Reservation. New York: Longman. Pinnell, G. S. ( 1977). Language in primary classrooms. Theory into Practice, 14(5), 3 1 8-327. Popkewitz, T., Tabachnick, B., & Zeichner, K. ( 1979). Dulling the senses: Research in teacher education. Journal of Teacher Education, 30(5), 52-60. Popper, K. (1963). Conjectures and refutations. London : Routledge & Kegan Paul. Power, C. ( 1977). A critical review of science classroom instruction stu­ dies. Studies in Science Education. 4. 1-30. Ramirez, A. (1979). Discourse patterns during composition lessons: The sequence of speech acts. Paper presented at the annual meeting of the American Educational Research Association, San Francisco. Ramirez, A. (in press). Analyzing speech acts. In J. Green, J. Harker, & C. Wallat (Eds.), Multiple perspective analysis of classroom dis­ course. Norwood, NJ: Ablex. Reiss, A. (1971). Systematic observation of natural social phenomena. In H. Costner (Ed.), Sociological methodology. San Francisco: Jossey-Bass. Remmers, H. H. ( 1963). Rating methods in research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally. Rentel, V. (in press). The development of cohesive harmony: A second­ ary analysis. In J. Green, J. Harker, & C. Walla! (Eds.), Multiple perspectives analysis of classroom discourse. Norwood, NJ: Ablex. Research on teaching [Special issue]. ( 1983a). Educational Psycholo­ gist, 18(3).

Research on teaching [Special issue]. ( 1983b). Elementary School Journal, 83(4).

Richards, M., & Bernal, J. ( 1972). In N. Blurton-Jones (Ed.), Ethological studies of child behavior (pp. 175- 197). London : Cambridge University Press. Rosenholtz, S. R. (1979). Modifying the effects of academic status: The multiple abilities classroom. Paper presented at the meeting of the American Educational Association, San Francisco. Rosenholtz, S. R. (in press). Treating problems of academic status. In J.

Berger & M. Zelditch. Jr. (Eds.), Studies in expectation states theory: Pure and applied. San Francisco: Jossey-Bass. Rosenshine, B. ( 1970). The stability of teacher effects upon student achievement. Review ofEducational Research, 40, 647-662. Rosenshine, B., & Furst, N. (1973). The use of direct observation to study teaching. In R. M. W. Travers (Ed.), Second handbook of re­ search on teaching. Chicago : Rand McNally. Rosenthal, R., & Jacobson, L. ( 1968). Pygmalion in the classroom: Teacher expectations and pupils' intellectual development. New York: Holt, Rinehart & Winston. Sacks, H., Schegloff, E., & Jefferson, G. (1974). A simplest systematics for the organization of turntaking for conversation. Language, 50(4), Part I), 696- 735. Sanday, P. (1976). Emerging methodological developments for research design, data collection, and data analysis in anthropology and edu­ cation. In C. Calhoun & F. Ianni (Eds.), The anthropological study ofeducation. The Hague: Mouton. Sanday, P. ( 1979). The ethnographic paradigm(s). Administrative Science Quarterly, 24(4), 527-538. Sanders, D. (1981). Educational inquiry as developmental research. Educational Researcher, 10(3), 8-1 3. Schatzman, L., & Strauss, A. (1973). Field research : Strategies for a natural sociology. Englewood Cliffs, NJ: Prentice-Hall. Scheflen, A. E. (1 972). Body language and social order: Communication as behavioral control. Englewood Cliffs, NJ: Prentice-Hall. Scollon, R., & Scollon, S. (1984). Cooking it up and boiling it down: Abstracts of Athabaskan children's story retellings. In D. Tannen (Ed.), Coherence in spoken and written language. Norwood, NJ: Ablex. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19, 321-325. Scribner, S., & Cole, M. ( 198 1). The psychology of literacy. Cambridge, MA: Harvard University Press. Sevigny, M. (1981). Triangulated inquiry-A methodology for the analysis of classroom interaction. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings. Norwood, NJ: Ablex. Shavelson, R. (1983). Review of research on teachers' pedagogical judg­ ments, plans, and decisions. Elementary School Journal, 83(4), 392-413. Shavelson, R., & Dempsey-Atwood, N. ( 1976). Generalizability of mea­ sures of teaching behavior. Review of Educational Research, 46(4), 553-6 1 1 . Shaver, J . ( 1983). The verification o f independent variables i n teaching methods research. Educational Researcher, 12(8), 3-9. Shulman, L. (1981). Disciplines of inquiry in education: An overview. Educational Researcher, 10(6), 5-12. Shuy, R. (in press). Identifying dimensions of classroom language. In J. Green, J. Harker, & Wallat (Eds.), Multiple perspective analysis of classroom discourse. Norwood, NJ: Ablex. Shuy, R., & Griffin, P. (1981). What do they do at school any day: Studying functional language. In W. P. Dickson (Ed.), Children's oral communication skills. New York: Academic Press. Simon, A., & Boyer, E. G. (Eds.). (1970a). Mirrors for behavior: An anthology of classroom observation instruments (Vols. 7-14 and Summary). Philadelphia: Research for Better Schools. (ERIC Docu­ ment Reproduction Service No. ED 03 1 613). Simon, A., & Boyer, E. G. (Eds.). (1970b). Mirrors for Behavior: An anthology of observation instruments continued, 1970 Supplement.

Vols. A & B. Philadelphia: Research for Better Schools. Inc. Simons, H., & Murphy, S. (1981). Spoken language strategies and read­ ing acquisition. School- home ethnography project (NIE No. G-780082). Washington, DC: National Institute of Education. Sinclair, J. M., & Coulthard, R. M. (1975). Toward an analysis of dis­ course: The English used by teachers and pupils. London: Oxford University Press. Smith, L., & Geoffrey, W. ( 1968). Complexities of an urban classroom : An analysis toward a general theory of teaching. New York: Holt, Rinehart & Winston. Snow, R. (1974). Representative and quasi-representative designs for research on teaching. Review of Educational Research, 44(3), 265-29 1.

OBSERVATION AS INQUIRY AND METHOD Spencer-Hall, D. (1981 ). Looking behind the teacher's back. Elementary School Journal, 81, 281 -289. Spindler, G. (1 982). Doing the ethnography of schooling: Educational anthropology in action. New York: Holt, Rinehart & Winston. Spradley, J. (1980). Participant observation. New York: Holt, Rinehart & Winston. Stallings, J. (1 977). Learning to look : A handbook on classroom observa­ tion and teaching models. Belmont, CA : Wadsworth. Stallings, J. (1983). The Stallings observation system. Unpublished man­ uscript, Vanderbilt University, Peabody Center for Effective Teach­ ing, Nashville, TN. Stoffan-Roth, M. (1981). Conversational access strategies in instruc­ tional contexts. Unpublished doctoral dissertation, Kent State Un­ iversity, Kent, OH. Stubbs, M. (1976). Language, schools and classrooms. London : Methuen. Stubbs, M. (1983). Discourse analysis; The sociolinguistic analysis of natural language. Chicago : University of Chicago Press. Stubbs, M., & Delamont, S. (Eds.). (1 976). Exp/orations in classroom observation. Chichester, England: John Wiley. Stubbs, M., Robinson, B., & Twite, S. (1979). Observing classroom language (Block 5, PE232). Stony Stratford, Milton Keynes, England: Open University Press. Szwed, J. (1977). The ethnography of literacy. Paper presented to the National Institute of Education Conference on Writing, Los Angeles. Taba, H. (1967). Teacher's handbook for elementary social studies. Reading, MA : Addison-Wesley. Tannen, D. (1979). What's in a frame? Surface evidence for underlying expectations. In R. Freedle (Ed.), Advances in discourse processing (Vol. 2). Norwood, NJ: Ablex. Taylor, D. (1983). Family literacy: Young children learning to read and write. Exeter, NH : Heinemann. Tenenberg, M. (1981). Cycling and recycling questions: The " when" of talking in classrooms. Paper presented at the annual meeting of the . American Educational Research Association, Los Angeles. Tenenberg, M. (in press). Diagramming question cycle sequences. In J. Green, J. Harker, & C. Wallat (Eds.), Multiple perspective analysis of classroom discourse. Norwood, NJ : Ablex. Tenenberg, M., Morine-Dershimer, G., & Shuy, R. (1981). What did anybody say? (Final report, Part I, NIE No. G-78-0162). Wash­ ington, DC: National Institute of Education. Tikunoff, W., Ward, B., & Dasha, S. (1978). Volume A : Three Case Studies (Report No. 78-7). San Francisco: Far West Laboratory. Travers, R. M. W. (Ed.). ( 1973). Second handbook of research on teach­ ing. Chicago: Rand McNally. Trueba, H. T., Guthrie, G. T., & Au, K. (Eds.). (198 1). Culture and the bilingual classroom: Studies in classroom ethnography. Rowley, MA: Newbury House. Veldman, D., & Brophy, J. (1974). Measuring teacher effects on pupil achievement. Journal of Educational Psychology, 66, 319-324. Walker, D., & Shaffarzick, J. (1974). Comparing curricula. Review of Educational Research, 44, 83- 1 1 1.

213

Wallat, C., & Green, J. (1979). Social rules and communicative contexts in kindergarten. Theory into Practice, 18(4), 275-284. Wallat, C., & Green, J. (1982). Construction of social norms. In K. Bor­ man (Ed.), Social life of children in a changed society. Hillsdale, NJ: Lawrence Erlbaum Associates. Wallat, C., Green, J., Conlin, S., & Haramis, M. (1981). Issues related to action research in the classroom: The teacher and researcher as a team. In J. Green & C. Wallat (Eds.), Ethnography and language in educational settings. Norwood, NJ : Ablex. Weinstein, R. S. (1976). Reading group membership in the first grade: Teacher behaviors and pupil experience over time. Journal of Edu­ cational Psychology, 68, 103- 1 1 6. Weinstein, R. S. (1981). Student perspectives on "achievement" in var­ ied classroom environments. In P. Blumenfeld (Chair), Student per­ spectives and the study of the classroom. Symposium presented at the meeting of the American Educational Research Association, Los Angeles. Weinstein, R. S. (1983). Student perceptions of schooling. Elementary School Journal, 83(4), 287-312. Weinstein, R. S. (in press). Student mediation of classroom expectancy effects. In J. B. Dusek (Ed.), Teacher expectancies. Hillsdale, NJ: Lawrence Erlbaum Associates. Weinstein, R. S., & Marshall, H. (1984). Ecology of students' achieve­ ment expectations: Final report. Berkeley: University of California, Psychology Department. Weinstein, R. S., Marshall, H. H., Brattesani, K. A., & Middlestadt, S. E. (1982). Student perceptions of differential teacher treatment in open and traditional classrooms. Journal of Educational Psychology, 74, 678-692. Weinstein, R. S., & Middlestadt, S. E. ( 1979). Student perceptions of teacher interactions with male high and low achievers. Journal of Educational Psychology, 71, 421 -31. Whiting, B. (1963). Six cultures: Studies of child rearing. New York : John Wiley. Wilkinson, L. (Ed.). ( 1982). Communicating in the classroom. New York: Academic Press. Wilkinson, L., & Calculator, S. (1982). Effective speakers: Student's use of language to request and obtain information and action in the classroom. In L. Cherry Wilkinson (Ed.), Communicating in the classroom, New York: Academic Press. Woods, P., & Hammersley, M. (1977). School experience: Exp/orations in the sociology of schooling. London: Croom Helm. Woolfolk, A., & Brooks, D. (1983). Nonverbal communication in teaching. In E. Gordon (Ed.), Review of research in education, (Vol. 10). Washington, DC: American Educational Research Association. Wright, H. (1960). Observational child study. In P. Mussen (Ed.), Handbook of research methods in child development. New York: John Wiley. Yinger, R., & Clark, C. (1981). Reflective journal writing: Theory and practice (Occasional Paper No. 50). East Lansing:, Michigan State University, Institute for Research on Teaching.

7.

Sy n t h eses of Resea rc h o n Teac h i n g H e rbert J . Wal berg

University of Illinois at Chicago

Research synthesis makes the 1980s an extraordinary time in the history of research on teaching. The seminal contributions of Bloom (1976), Gage (1978), Glass (1977), and Light and Smith (1971) served to provide the major substantive and meth­ odological breakthroughs in consolidating findings of educational research. Glass, McGaw, and Smith (1981); Hedges, Giaconia, and Gage (1981); Hedges and Olkin (1981, 1983); Kulik, Kulik, and Cohen (1980), and Light and Pillemer (1982) have since provid­ ed excellent examples and insights on the synthesis of research on teaching and related educational factors (see Walberg, 1984, for a compilation and analysis). They and others are demon­ strating the consistency of educational effects and are helping to put teaching on a more sound scientific basis. Before turning to an overview of this chapter that shows how this is being accom­ plished, several working definitions are worth considering: Primary research, for the purpose of this chapter, reports an original analysis of raw qualitative or quantitative data. It may include a literature review, theoretical framework, and explicit hypotheses. Secondary analysis is the reanalysis of data by the original or, more often, other investigators for the same purpose for which they were collected or a different'one. Reviews are narrative commentaries about primary and sec­ ondary research. They usually summarize and evaluate studies one by one, but rarely quantitatively analyze them. Reviews often contain theoretical insights and practical recommend­ ations, but rarely assess rival theories by their parsimonious accounting of empirical regularities. Research synthesis explicitly applies scientific techniques and standards to the evaluation and summary of research; it not only statistically summarizes effects across studies but also pro­ vides detailed descriptions of replicable searches of literature, selection of studies, metrics of study effects, statistical proce-

eVries, D., & Slavin, R. (1978). Teams-games-tournaments (TGT): Review of ten classroom experiments. Journal of Research and Development in Education, 12, 28-38. Dweck, C., & Goetz, T. (1978). Attributions and learned helplessness. In J. Harvey, W. Ickes, & R. Kidd (Eds.), New directions in attribution research. Hillsdale, NJ: Erlbaum. Dweck, C., & Reppucci, N. (1973). Learned helplessness and reinforce­ ment responsibility in children. Journal of Personality and Social Psychology, 25, 109- 1 1 6. Edwards, K., DeVries, D., & Snyder, J. (1972). Games and teams: A winning combination. Simulation and Games, 3, 247-269. Gage, N. L. (Ed.). (1963). Handbook of research on teaching. Chicago: Rand McNally. Geffner, R. (1978). The effects of interdependent learning on self-esteem, interethnic relations, and interethnic attitudes of elementary school children: A field experiment. Unpublished doctoral dissertation,

University of California, Santa Cruz. Gersten, R., & Carnine, D. (1983, April). The later effects of Direct Instruction Follow Through: Preliminary findings. Paper presented at the American Educational Research Association, Montreal. Good, T. (1981). Teacher expectations and student perceptions: A decade of research. Educational Leadership, 38, 415-422. Good, T., & Dembo, M. (1973). Teacher expectations: Self-report data. School Review, 81, 247-253. Goodrich, R. L., & St. Pierre, R. G. (1 980, February). Opportunities for studying later effects of Follow Through: Executive summary (Con­ tract No. HEW-300-78-0443, Examination of Alternatives for Follow Through Experimentation). Cambridge, MA: Abt Asso­ ciates. Gordon, I. J. (1969). Early childhood stimulation through parent educa­ tion (Final report to the Children's Bureau, Social and Rehabilita­ tion Service, Dept. of Health, Education, and Welfare). Gainesville: University of Florida. Gray, S. W., Ramsey, B. K., & Klaus, R. A. ( 1982). From 3 to 20: The Early Training Project. Baltimore: University Park Press. Groff, P. ( 1974). Some criticisms of mastery learning. Today's Education, 63, 88-91. Halstead, J. S. (1982). Annual report of the Follow Through program (Dept. of Education Grant No. G007501 868). Richmond, VA: Richmond Public Schools. Hamblin, R., Hathaway, C., & Wodarski, J. (197 1). Group contingen­ cies, peer tutoring, and accelerating academic achievement. In E.

Ramp and W. Hopkins (Eds.), A new direction for education: Behavior analysis (pp. 41 -53). Lawrence: University of Kansas, Department of Human Development. Horton, L. ( 1979). Mastery learning: Sound in theory, but . . . Educa­ tional Leadership, 37, 154- 1 56. House, E., Glass, G., McLean, L., & Walker, D. (1977). No simple answer: Critique of the " Follow Through" evaluation. Urbana: University of Illinois, Center for Instructional Research and Curriculum Evaluation. Jaynes, J. ( 1975). Hello, teacher . . . Contemporary Psychology, 20, 629-631. Johnson, D., & Johnson, R. (1974). Instructional goal structure: Co­ operative, competitive, or individualistic. Review of Educational Research, 44, 213-240. Johnson, D., & Johnson, R. (1975). Learning together and alone. Englewood Cliffs, NJ: Prentice-Hall. Johnson, D., Johnson, R., & Scott, L. (1978). The effects of cooperative and individualized instruction on student attitudes and achieve­ ment. Journal of Social Psychology, 104, 207-216. Keller, F. (1968). Goodbye, teacher . . . Journal of Applied Behavior Analysis, 1, 79-89. Kim, Y., Cho, G., Park, J., & Park, M. (1974). An application of a new instructional model (Res. Rep. No. 8). Seoul: Korean Educational Development Institute. Kulik, J., Kulik, C., & Cohen, P. (1979). A Meta-analysis of outcome studies of Keller's personalized system of instruction. American Psychologist, 34, l l0-1 1 3. Laughlin, P. (1978). Ability and group problem solving. Journal of Research and Development in Education, 12, 1 14-120. Lazar, I., Darlington, R., Murray, H., Royce, J., & Snipper, A. ( 1982). The lasting effects of early education: A report from the Consortium for Longitudinal Studies. Monographs of the Societyfor Research in Child Development, 47 (2-3, Serial No. 195). Lazar, I., Hubell, V. R., Murray, H., Rasche, M., & Royce, J. (1977). The persistence of preschool effects (Publication No. [OHDS] 78-301 30). Washington, DC: Department of Health, Education and Welfare. Lee, Y., Kim, C., Kim, H., Park, B., Yoo, H., Chang S., & Kim, S. (1971). Interaction improvement studies on the mastery learning project

(Final Report on Mastery Learning Program, April-November 1971). Seoul: Seoul National University, Educational Research Center. Levine, J. ( 1983). Social comparison and education. In J. Levine & M. Wang (Eds.), Teacher and student perceptions: Implication for learn­ ing. Hillsdale, NJ: Erlbaum. Lloyd, K., & Knutzen, N. ( 1969). A self-paced programmed undergrad­ uate course in the experimental analysis of behavior. Journal of Applied Behavioral Analysis, 2, 125-133. Lucker, G., Rosenfield, D., Sikes, J., & Aronson, E. (1976). Performance in the interdependent classroom: A field study. American Educa­ tional Research Journal, 13, 1 1 5-123. Maccoby, E. E., & Zellner, M. (1970). Experiments in primary education: Aspects of Project Follow Through. New York: Harcourt Brace. Madden, N., & Slavin, R. (1983). Effects of cooperative learning on the social acceptance of mainstreamed academically handicapped stu­ dents. Journal of Special Education, 1 7, 171- 1 82. Meyer, L. (1984). Longterm academic effects of Direct Instruction Follow Through. Elementary School Journal. 1984, 4, 380-394. Meyer, L., Gersten, R., & Gutkin, J. (1984). Direct Instruction: A Project Follow Through success story. Elementary School Journal. 1984, 2, 241 -252: Miller, L., & Bizzell, R. ( 1983a). Long-term effects of four preschool programs: Ninth and tenth grade results. Louisville, KY: University of Louisville. Miller, L., & Bizzell, R. (1983b). Long-term effects of four preschool programs: Sixth, seventh, and eighth grades. Child Development, 54, 727-741. National Commission on Excellence in Education. (1983). "A Nation at risk: The imperative for educational reform." Washington, DC: National Commission. Okey, J. R. (1974). Altering teacher and pupil behavior with mastery teaching. School Science and Mathematics, 74, 530-535. Okey, J. (1975, August). Development of mastery teaching materials

RESEARCH ON EARLY CHILDHOOD AND ELEMENTARY SCHOOL TEACHING PROGRAMS

(Final Evaluation Rep., USOE G-74-2990). Bloomington: Indiana University. Olmsted, P. P. (1977). The relationship of program participation and parental teaching behavior with children's standardized achievement measures in two program sites. Unpublished doctoral dissertation,

University of Florida. Olmsted, P. P., & Rubin, R. I. ( 1982, September). Assistance to local Follow Through programs (Annual Rep., Dept. of Education, Grant No. G00-770-1 69 1). Washington, DC: U.S. Dept. of Education, Follow Through Branch. Pavlov, I. P., (1927). Conditioned reflexes (G. V. Anrep, Trans.). London: Oxford University Press. Rubin, R., Olmsted, P., & Kelly, J. (1981). Comprehensive model for child services. Children and Youth Services Review, 3, 175- 192. Rubin, R., Olmsted, P., Szegda, M., Wetherby, M., & Williams, D. (1983). Long-term effects of parent education Follow Through pro­ gram participation. Paper presented at the American Educational Research Association annual meeting in Montreal, Canada, April, 1983. Sameroff, A. J. (1979, March). Theoretical and empirical issues in the operationalization of transactional research. Paper presented at the biennial meeting of the Society for Research in Child Development, San Francisco. Schweinhart, L., & Weikart, D. ( 1980). Young children grow up: The effects of the Perry Preschool Program on youths through age 15

(Monograph No. 7). Ypsilanti, MI: High/Scope Educational Re­ search Foundation. Sharan, S., & Sharan, Y. (1976). Small-group teaching. Englewood Cliffs, NJ: Education Technology Publications. Sharan, S. (1980). Cooperative learning in small groups: Recent meth­ ods and effects on achievement, attitudes, and ethnic relations. Review of Educational Research, 50, 241-271. Skinner, B. F. (1968). Technology of teaching. New York: Appleton-Century-Crofts. Slavin, R. ( 1977). Classroom reward structure: An analytical and practical review. Review of Educational Research, 4 7, 633-650. Slavin, R. (1978a). Student teams and achievement divisions. Journal of Research and Development in Education, 12, 39-49. Slavin, R. (1978b). Student teams and comparison among equals: Effects on academic performance and student attitudes. Journal of Educational Psychology, 70, 532-538. Slavin, R. (1980a). Effects of student teams and peer tutoring on academic achievement and time on task. Journal of Experimental Education, 48, 252-257. Slavin, R. (1980b). Using student team learning (rev. ed.). Baltimore: Johns Hopkins University, Center for Social Organization of Schools. Slavin, R. (1983). Cooperative learning. New York: Longman. Slavin, R. (1984). Students motivating students to excel: Cooperative incentives, cooperative tasks, and student achievement. Elementary School Journal. 1984, 85, 53-64. Slavin, R., & Karweit, N. (1981). Cognitive and affective outcomes ofan intensive student team learning experience. Journal of Experimental Education, 50, 29-35. Slavin, R., Leavey, M., & Madden, N. (1982). Effects of student teams

753

and individualized instruction on student mathematics achievement, attitudes, and behaviors. Paper presented at the annual convention

of the American Educational Research Association, New York. Smith, J. (1981). Philosophical considerations ofmastery learning theory: An empirical study. Paper presented at the annual meeting of the American Educational Research Association, Los Angeles. Spady, W. (1981). Outcome-based instructional management: A sociologi­ cal perspective. Unpublished manuscript, American Association of School Administrators, Arlington, VA. Stallings, J. (1975, December). Implementations and child effects of teaching practices in Follow Through classrooms. Monograph of the Society for Research in Child Development, 40, 7-8. Stallings, J. (1977). Learning to look: A handbook on classroom observa­ tion and teaching models. Belmont, CA: Wadsworth. Stallings, J., & Kaskowitz, D. (1974). Follow Through classroom obser­ vation evaluation, 1972-73. Menlo Park, CA: SRI International. Stallings, J., Robbins, P., & Wolfe, P. (1984). A staff development program to increase student learning time and achievement. Napa, CA: Unpublished student program. Stebbins, L., St. Pierre, R. G., Proper, E. C., Anderson, R. B., & Cerva, R. R. ( 1977). Education as experimentation: A planned variation model (Vol. 4A-D). Cambridge, MA: Abt Associates. Stodolsky, S. S. (1972). Defining treatment and outcomes in early childhood education. In H. J. Walberg & A. T. Kopan (Eds.), Rethinking urban education. San Francisco: Jossey-Bass. Terman, L. W., & Merrill, M. A. (1960). Stanford-Binet Intelligence Scale. Cambridge, MA: Houghton Mifflin. Tiegs, E. W., & Clark, W. W. (1963) Manual: California Achievement Test, complete battery. Monterey Park : California Test Bureau (McGraw-Hill). Tiegs, E. W., & Clark, W. W. ( 1970) Test coordinator's handbook: California Achievement Tests. Monterey, California: California Test Bureau (McGraw-Hill). Travers, R. M. W. (Ed.). ( 1973). Second handbook of research on teaching. Chicago: Rand McNally. Vapova, J., & Royce, J. (1978, March). Comparison of the long-term effects of infants and preschool programs on academic performance.

Paper presented at the annual meeting of the American Educational Research Association, Toronto. Webb, N. (1980a). Group process: The key to learning in groups. New Directions for Methodology of Social and Behavioral Science, 6,

77-87. Webb, N. (1980b). A process-outcome analysis of learning in groups and individual settings. Educational Psychologist, 15, 69-83. Webb, N. (1982). Peer interaction and learning in cooperative small groups. Journal of Educational Psychology, 74, 642-655. Weikart, D., Rogers, L., Adcock, C., McClelland, D. (1971). The cognitively oriented curriculum: A framework for preschool teachers.

Urbana: University of Illinois - NAEYC. Weiner, B. (1979). A theory of motivation for some classroom experi­ ences. Journal of Educational Psychology, 71, 3-25. Ziegler, S. (1981). The effectiveness of cooperative learning teams for increasing cross-ethnic friendship: Additional evidence. Human Or­ ganization, 40, 264-268.

26.

.

Resea rc h o n Teac h i n g 1 n H i g h e r E d u cat i o n M i chael

J.

Dunkin The University of Sydney, Australia

with the assistance of

J e n n ife r Barnes The University of Sydney, Australia

A Conceptual Framework

During the decade following publication of the Second Hand­ book of Research on Teaching (Travers, 1973), several notable developments occurred in research on teaching in higher educa­ tion. Innovative approaches to structuring learning experiences for students, such as computer-based teaching and the Keller Plan, generated enough research for several reviews to be written. Developments in techniques of reviewing the research and synthesizing its findings, described by Walberg (this vol­ ume), were applied to higher education. Increased interest was shown in observational research of classroom processes. Con­ troversial issues surrounding student evaluation of instruction stimulated much research activity and research on attempts to implement knowledge about teaching in higher education in order to improve it through faculty development programs was conducted. This chapter has been designed to present and discuss the preceding developments and to suggest their impli­ cations for future research and practice in higher education. In subsequent sections, a conceptual framework for integrat­ ing the research is suggested, and evidence pertaining to the various types of relationships involving teaching variables in higher education is summarized. The section on Research on Teaching Methods is concerned mainly with methods effective­ ness experiments. The section on Research on Facets of Teach­ ing Behavior is organized in terms of qualities of teaching behavior observed in studies of the full range of relationships involving teaching variables. Later parts of the chapter focus upon research in the evaluation and improvement of teaching in higher education. The chapter concludes with a discussion of issues arising from research reviewed in earlier sections.

According to Gage (1963), there are three main questions involved in research on teaching. They are: How do teachers behave? Why do they behave as they do? and What are the effects of their behavior? Information about three types of variables is required to answer those questions. First, there is teaching behavior, or the processes of teaching; the set of variables involved in all three questions and most central to research on teaching. Second, there are the causes or determin­ ants of those processes. Finally, there are variables indicating the effects, or consequences, of teaching processes. Teaching processes are behaviors engaged in for the purposes of promoting learning in others. They can be thought of in global terms as teaching methods, such as lecturing, leading discussions, and demonstrating. They can also be thought of as more specific teaching behaviors, such as statements of fact, particular types of questions, or categories of reactions to student behaviors. Some teaching methods involve face-to-face contact between teachers and students more than others. Lec­ turing, for example, is usually performed live by the teacher, but sometimes recorded lectures are presented, and some methods, for example, the Keller Plan, are designed to minimize such contact. Whether the teacher is present or absent on those occasions when learning by students is being guided, teaching processes are observable and are subject to analysis in terms of specific categories of teaching behavior. Furthermore, teaching need not always be performed by those chiefly employed for that purpose. One teaching method is peer teaching, whereby

The author thanks reviewers James Kulik (University of Michigan) and Wilbert McKeachie (University of Michigan). 754

755

RESEARCH ON TEACHING IN HIGHER EDUCATION students attempt to promote the learning of each other, and the Keller Plan includes senior students as proctors to assist more junior students. Again, the behavior engaged in by students in such teaching capacities is analyzable as teaching behavior. The answer to Gage's first question lies in both the teaching methods and the specific categories of teaching behavior em­ ployed by those who attempt to promote the learning of others by exerting influence upon the particular types of experiences of those others. Determinants of teaching behavior consist of attributes of the teachers, the students, or the contexts in which they behave. Teachers perform any one set of processes in association with other sets of processes and the connection between the two sets may be causal. Teachers also behave as they do because of who they are. They possess characteristics that may predispose them to behave in some ways more than in others. Included, too, are their formal learning backgrounds, their education as teachers and researchers, and even early experiences quite remote from their positions as teachers. Teachers behave as they do partly because of their students. Characteristics of students, some relating to formal criteria of selection into formal educational contexts, such as previous academic performance, but also other characteristics, such as student sex and ethnicity, may affect the way teachers behave. Most immediately, however, the way students present them­ selves, their behavior in contact with teachers, influences the latter. Student attentiveness, completion of tasks set by the teacher, and evaluations of the teaching they receive must all be assumed to be potential determinants of the teaching processes experienced. Teachers and students come together in contexts that them­ selves contain the power to influence teaching and learning. Contextual factors such as the size of the institution, its resources, and whether it is public or private are relevant here. Contextual variations can range from type of teaching space and size of class to type of discipline taught, to the society and culture in which the particular institution is located. Different areas of knowledge, curricula, and programs may also be expected to have their emphases reflected in teaching processes. Inquiry-oriented learning materials should result in classroom events more consistent with inquiry than didacticism. Teaching in medical schools might be expected to differ from teaching in engineering, and teaching in advanced courses from teaching in introductory courses. The effects of teaching in higher education, as at other levels of education, are most commonly researched in terms of academic achievements of students. Comparative studies of different teaching methods are the best examples of achieve­ ment criteria being used in research on teaching in higher education. Other criteria of effectiveness are, however, often employed. They include students' evaluations or ratings, time taken for students to complete courses, withdrawal or comple­ tion rates, and students' attitudes toward the course. In summary, the ways in which teachers behave in higher education classrooms have been conceptualized in terms of both global teaching methods and specific categories of teach­ ing behavior. It is these processes of teaching that are central to research on teaching. Sometimes research on teaching in higher education is concerned with explaining variations in the occur-

rence of these processes. Processes themselves are researched as influences upon other processes, but more often explanations of variations in them are sought in the background characteristics of teachers themselves and their students. Characteristics of teachers and students when used as possible sources of explana­ tion of variation in process variables are referred to as presage variables, that is, as the bases for predictions or forecasts of classroom processes. Features of the substantive, physical, and institutional environments in which teachers and students come together are also employed as possible influences upon process variables in higher education. When used in that way, they are referred to in this chapter as context variables. While teaching variables are classified as process variables and determinants of variations in teaching are called process, presage, or context variables, the effects of teaching processes when considered at a point in time are either other process variables or such outcomes as student attitudes and achieve­ ments. More commonly, however, the effects of teaching pro­ cesses are sought not in subsequent processes but in the products taken away by the students in terms of academic or professional achievements and attitudes. Research in teaching in higher education has been more concerned with the effects of processes upon products than with any other of the possible issues involving teaching behavior. The pursuit of Gage's three questions for research on teach­ ing (Gage, 1963) leads in the higher education domain, as well as in others, to an elaboration of the issues and questions in terms of four classes of variables: process, presage, context, and product. These four types of variables are useful in describing research on teaching as follows: Gage's Questions

Types of Variable and Relationships

How do teachers behave?

Process occurrence

Why do they behave as they do?

Process-Process relationships Presage-Process relationships Context-Process relationships

What are the effects of their behavior?

Process-Process relationships Process-Product relationships

It should be noted that other possible relationships such as those between presage and product variables are not included here. This is because they do not qualify as research on teaching. To do that, they must include process variables. In the rest of this chapter the preceding conceptual framework will be applied to research on teaching in higher education.

Research on Teaching Methods This section is divided into two parts. The first part presents the conclusions of reviewers who have synthesized research on teaching methods that rely for their effectiveness mainly on interpersonal communication between teachers and students. In this part, therefore, research on lectures versus discussion, team teaching, and peer teaching is reviewed. The second part focuses on teaching methods that rely upon highly structured learning materials, usually, but not always,

756

MICHAEL J. DUNKIN

involving the use of hardware, such as tape recorders, projec­ tors, or computers, and designed to facilitate individualized learning. One of the most rapidly expanding areas of higher education is distance education. Research on this topic is not reviewed in this chapter because it is much more than a teaching method and because most of the methods that are reviewed in the chapter are available for use in distance education. Readers interested in research in distance education will find works by Daniel, Stroud, and Thompson (1982), and Holmberg (1982) helpful.

Social Interaction Methods There have been several reviews of research on the relative benefits of lecturing and discussion methods in higher educa­ tion. McKeachie (1963) concluded his review of research com­ paring the two as follows: "When one is asked whether lecture is better than discussion, the appropriate counter would seem to be, For what goals?" (p. 1127). It seems that over the following 16 years, little reason for altering that conclusion emerged, for Kulik and Kulik (1979) indicated that a similar conclusion was warranted. They compared several reviews and wrote: These reviewers agree on some basic conclusions. First, they point out that teaching by discussion is neither more nor less effective than teaching by lecture when the criterion of effectiveness is learning of factual information. . . . The reviewers also agree that teaching by discussion is more effective than teaching by lecture for more ambitious cognitive objectives, such as developing problem-solving ability. . . . [Those who] reviewed research on the relative effectiveness of discussion versus lecturing in terms of changing attitudes. . . . found a third area of agreement [in favor of discussion]. . . . Where the reviewers disagree somewhat is in their conclusions about student satisfaction with teaching by lecture versus teaching by discussion. . . . (pp. 7 1-73)

Schustereit (1980) reviewed research on team teaching in higher education and accepted the definition by Johnson and Lobb (1959): "a group of two or more persons assigned to the same students at the same time for instructional purposes in a particular subject or combination of subjects" (p. 59). Schustereit divided the studies reviewed into two types, those comparing team-taught and "solitary-teacher-taught" classes, and those comparing three different teaching techniques includ­ ing team-teaching. Using "box-score" methods (see Walberg, this volume) of synthesizing the research, Schustereit found inconsistent results from the 10 studies reviewed and concluded as follows: ". . . While the reviewed studies gave a plurality of support to team-teaching, a generalization that team-teaching is a superior instructional technique would not be justified based on the various studies" (p. 88). Goldschmid and Goldschmid (1976) attempted to synthesize literature on peer teaching in higher education. After summariz­ ing sociopsychological, pedagogical, economic, and political arguments in favor of using peer relationships to enhance learning, the authors discussed five types of peer teaching: discussion groups, seminars, or tutorial groups, as led by

teaching assistants; the proctor model, as in the Keller Plan (Keller, 1968), in which senior students may assist individual students; student learning groups that are instructorless or self­ directed; the learning cell, or student dyad; and "parrainage," or senior students counselling entering students. Clearly, the var­ iety of procedures included in the grouping of those heteroge­ neous approaches made peer teaching too broad a concept for specific or confident conclusions to be reached. Moreover, only a small body of relevant research was found to exist. Goldsch­ mid and Goldschmid (1976) concluded: Despite its many advantages - including its low cost - peer teach­ ing is not a panacea for all instructional problems. In fact, it is best used together with other teaching and learning methods. It may be particularly relevant when one seeks to maximize the students' [sic] responsibility for his own learning and active participation in the learning process, and to enhance the development of skills for co­ operation and social interaction. (p. 29)

Research on other teaching methods relying upon social interaction, such as clinical teaching, is reviewed in other chapters (see Dinham & Stritter, this volume), except for a few studies of special relevance to the following section on facets of teaching behavior. The research just reviewed was predominantly process­ product research. It is difficult to obtain evidence regarding the actual processes in terms of which the prescriptively defined methods were implemented. Furthermore, questions such as the extent to which presage and context variables affected those processes and, indirectly, student outcomes, were rarely ad­ dressed. In all, it seems that university and college teachers can derive only very shaky guidance from research in choosing between lectures, discussion, team teaching, and peer teaching. At most, research might justify the choice of discussions rather than lectures where higher cognitive learnings and attitude change are the objectives. Within the samples of each of the preceding methods researched, there were instances that were highly effective and highly ineffective. Yet there has been surprisingly little research that has tried to discover the differences between those effective instances and the less effective ones. The section of this chapter on Research on Facets of Teaching Behavior discusses that approach to research.

Individualized Methods As mentioned at the beginning of this chapter, one of the most prominent developments in research on teaching in higher education in recent years has been the application of the techniques of meta-analysis in attempts to synthesize research. This is particularly true of research on methods of instruction associated with applications of technological developments to higher education. This part is concerned with meta-analysis connected with the Center for Research on Learning and Instruction at the University of Michigan. In these studies, the Michigan team synthesized research findings concerning audio­ tutorial instruction (Kulik, Kulik, & Cohen, 1979b), computer­ based teaching (Kulik, Kulik, & Cohen, 1980), Keller Plan instruction (Kulik et al., 1979a; Kulik, Jaksa, & Kulik, 1978),

RESEARCH ON TEACHING IN HIGHER EDUCATION

programed instruction (Kulik, Cohen, & Ebeling, 1980), and visual-based instruction (Cohen, Ebeling, & Kulik, 1981). The fact that a uniform set of procedures was used across the meta-analyses to identify and analyze studies makes it possible to attempt a synthesis of them here. There are also similarities among the instructional methods themselves that make treating them together reasonable. Apart from the use of technological developments, the methods generally share the attribute of being modularized in the sense that course content is divided into reasonably small, self-contained units in pursuit of well­ defined learning objectives. Most of the methods are attempts to provide individualization or personalization in higher educa­ tion. Third, most of the methods emphasize the provision of feedback to students on their performance on trials or tests that are taken frequently. Fourth, student activity is a common feature along with less emphasis upon the teacher in formal teaching situations, such as lectures. In these respects they form a relatively homogeneous set of teaching processes. The Michigan meta-analyses were attempts to answer ques­ tions mainly about process-product relationships. For exam­ ple, Does the innovative method prove to be more effective than conventional instruction in the typical comparative study? The influence of presage and context variables was also explored in questions such as: Do certain types of students benefit more than others from the innovative method? Under what condi­ tions is the innovative method most effective? PROCESS-PRODUCT RELATIONSHIPS ACROSS METHODS

The Michigan team synthesized findings concerning the relative effectiveness of the innovative methods in relation to the following main criteria: student achievement, both shorter and longer term, where possible; student satisfaction with the in­ struction and the course; withdrawal rates; where sufficient data were available, time used by students in taking the course; and correlations between student aptitude and student achieve­ ment. Several types of statistics were reported, including esti­ mated average percentage scores and percentile ranks in relation to achievement criteria, and estimated average scores on 5-point scales for satisfaction criteria. Traditional types of information regarding statistical significance were reported for most comparisons, but in all of the meta-analyses emphasis was given to effect size measures which sometimes were calculated according to procedures recommended by Glass (1976) and sometimes according to Cohen (1 969). Kulik, Kulik, and Cohen (1979b) found that four different methods of calculating effect size were highly intercorrelated (.99 ;;;, r ;;;, .83). " Box­ scores" of numbers of studies favoring innovative as against conventional methods were also reported. Table 26. 1 summarizes the results of the Michigan syntheses. It supplements Walberg's (this volume) Table 6.3. Table 26.1 indicates that for audio-tutorials, computer-based teaching, programmed instruction, and visual-based instruction, there was a small but statistically significant superiority in short-term achievement by students experiencing the innovative method over students in conventional courses. For students in Keller Plan courses, this superiority in achievement was considerably greater, as manifested in an effect of moderate size. Table 26.1

757

does not include information regarding longer term retention, but where such information was analyzed by the Michigan team, effect size was found to be at least as great and usually greater in the longer term in favor of the innovation. Differences in student satisfaction between innovative and conventional methods are shown in Table 26.1 to be small to negligible, with the exception, again, of the Keller Plan, in whose favor was another effect of moderate size. Differences in withdrawal rates also were shown to be negligible in all comparisons. In the few comparisons available for student time per week, statistically significant advantages were seen only for students receiving computer-based instruction. Because mastery is a common requirement among the inno­ vations researched, results were also reported for aptitude­ achievement correlations. Bloom's model (1968) suggests that under mastery requirements, in which students of both high and low initial ability have to attain the same standard of perfor­ mance to satisfy the criterion of success, the correlation between aptitude and achievement should be much smaller than in situations where mastery is not required. Table 26.1 indicates that this hypothesis was not supported in the research on innovations in higher education. Differences between the corre­ lation coefficients in question were small in every case. The research included in these meta-analyses was conducted in various contexts and under various design conditions, so that it was possible to investigate whether the effectiveness of a teaching method depended upon the methodological features of the research and the contexts of teaching. The Michigan team developed sets of categories for describing features of the studies. The list used in their meta-analysis of research on Keller Plan methods is presented in Table 26.2. Many of the categories were common across the several meta-analyses. One category not included for the Keller Plan meta-analysis, but included for the others, was whether or not the report was published. The only category to make a significant difference to achieve­ ment in the audio-tutorial research was, in fact, publication status. Studies published in journals were found to report achievement results more in favor of audio-tutorials than studies reported in dissertations and unpublished papers. Year of publication had a significant effect upon results in research on both programed instruction and visual-based instruction. Whether or not the research was conducted in a doctorate­ granting university made a difference in the meta-analysis of research on visual-based instruction. In three cases, attempts to control for instructor effects appeared to be influential. In research on computer-based teaching, Keller Plan, and visual-based instruction, student achievement results more favorable to the innovation were obtained if different instructors implemented the innovative and the conventional instruction than if the same instructor implemented both.

CONTEXTUAL EFFECTS

Interestingly enough, for only two of the innovative methods were substantial effects on student achievement found to de­ pend on variations in the instructional contexts themselves.

758

MICHAEL J. DUNKIN

Table 26.1 . Summary of Results of the Michigan M eta-Analyses of Research on Instructional Technology in H igher Education

Student Achievement N

Aud io-Tutorials Conventional instruction Significance level Mean effect size

42

Exam Scores

Percentile Rank

68.5% 66.9% p < .05 .20

58 50

Computer-Based teaching 54 60.6% Conventional instruction 57.6% p < .01 Sig'nificance level Mean effect size .25

60 50

Keller Plan instruction Conventional instruction Significance level Mean effect size

61

Programmed instruction Conventional instruction Significance level Mean effect size

56

Visual-Based instruction Conventional instruction Significance level Mean effect size

65

Note. A

N

Student Satisfaction (5-Point Scale) 3.56 3.30

6

N

22

13

70 50

67.1% 64.8% p < .05 .28

60 50

68.4% 66.9% p < .01 .15

56 50

11

4.1 9 3.40 p < .01 .46

4

3.41 3.49

27

- .06

1 3.9% 1 2.6%

8

4

2.25 3.50 p < .01

20.3% 1 9.7% N.S.

.06

10

1 3.1 % 1 3.2% N.S.

- .05

.41 .51

7

N.S.

.1 2

9 Approx. equal

.50 .50

N.S.

0

.10

9

.36 .39 .02

N.S.

- .1 0 3.45 3.48

26.9% 27.6%

Aptitude Achievement Correlation

N.S.

N.S.

N.S.

16

12

1 9% 1 7%

- .01

.24

73.6% 65.9% p < .0001 .49

N

0.06

.12

3.77 3.50

N

Time Taken (H rs/Week)

N.S.

N.S.

11

Withdrawal Rate

9

5

19

6 N.S.

.40 .48

N.S.

.09

16

.50 .45

N.S.

.06

dash indicates that the information was not available. "N.S." means not statistically significant at the .05 level.

Visual-based instruction that involved using videotape to pro­ vide student teachers with feedback on their teaching perfor­ mances was found to be much more effective than other uses of visual media. Keller Plan courses were more effective in mathe­ matics, engineering, and psychology than in the physical and life sciences and other social sciences. It seemed not to matter whether audio-tutorial techniques varied somewhat from those developed by Postlethwait (Postlethwait, Novak, & Murray, 1972), or whether computer-based teaching involved using the computer to manage student learning, tutor students, conduct simulation exercises, or involve students in programing for problem solving. No advantages of linear and branching pro­ grams over each other were apparent, and the use of still­ projection motion pictures, multimedia approaches, closed­ circuit TV, educational TV, and film or videotape for observing classrooms in teacher education seemed to be equally effective. PROCESS-PRODUCT RELATIONSHIPS WITHIN KELLER PLAN

Variations within applications of the Keller Plan proved to be opportune because they provided Kulik and his colleagues with opportunities to test the contributions of the various compon­ ent features of the method to its overall effectiveness (Kulik et al., 1978). The authors concluded on the basis of "box-scores" that requiring students to attain mastery on a unit before progressing leads to higher achievement without affecting with­ drawal rate or student satisfaction. The mastery requirement

emerged, therefore, as an important component of the Keller Plan. Implementations of Keller Plan principles also were found to vary in the size of the course units and in the number of quizzes students undertook. The few relevant studies on these issues contained various deficiencies in design, but were interpreted as a group to give support for smaller units and more frequent quizzing. Student retention of subject matter appeared to benefit from them, while satisfaction and study time seemed unaffected. Kulik and his colleagues also investigated various aspects of the student proctor's role in Keller Plan courses. It was pointed out that proctors performed three main functions: to score quizzes objectively, to provide immediate feedback on quiz performance to students, and to discuss course material with students. The research seemed to indicate that the amount of contact students had with proctors was unimportant, but that the timing of feedback was influential. Students' retention appeared adversely affected by delayed rather than immediate feedback. The timing of the feedback seemed to be more important than from whom it came, for students who graded their own quizzes were found to perform as well as students relying on proctors for feedback. In relation to the student self-pacing component of the Keller Plan, Kulik et al., (1978) concluded . . . self pacing in itself does not appear to make a significant difference in the level of student achievement. Self-pacing may contribute, however, to student morale and general satisfaction with

RESEARCH ON TEACHING IN HIGHER EDUCATION

Table 26.2. Categories for Describing Study Features and N umber of Studies in Each Category of M eta-Analysis of Keller Plan Research

Coding Category Subject matter Physical and life sciences Mathematics and engineering Psychology Other social sciences Course level I ntroductory Non introductory Type of school Major research universities Other research universities Doctorate-granting universities Comprehensive colleges Liberal arts colleges Community colleges and special in stitutions I nstructor Same Different Variations in control classes No PSI components Some characteristics of PSI

Number of Studies

10 11 18 6 32 15

9

6 7 17 6

3 22 22 43 5

Coding Category Historical effects Same semester Different semester Subject assignment Random Evidence of equivalence N o evidence of equivalence Variations in experimental classes Pure PSI Compromised PSI Authorship of examination Standardized exam Collaborated exam Single-instructordesigned exam

Number of Studies

45 3 10 17 21

41 7

9

759

The single most significant conclusion to be reached from research on innovatory teaching methods in higher education is that the Keller Plan is clearly superior to other methods with which it has been compared. Indeed, the Keller Plan has been so consistently found superior that it must rank as the method with the greatest research support in the history of research on teaching. The other innovatory methods also receive support from research when student achievement is the criterion. How­ ever, their benefits are relatively small although statistically significant. Another important contribution of research on the Keller Plan is that it has allowed evidence to be gathered on the relative contributions of the different components of the system. It seems that parsimony might be achieved through the use of mastery requirements, frequent quizzes, immediate feedback, and review units; but not proctor tutoring, self-pacing, explicit objectives, and occasional lecturing. Identification of the essen­ tials and nonessentials of the innovations, as in the review by Kulik et al., (1978), might ameliorate some of the difficulties found in implementing them. For the Keller Plan, at least, it appears that one of the most troublesome components for teachers and administrators, that is, self-pacing, may not be essential for deriving the benefits of the method.

11

17

Scoring bias Objective Blind essay Other essay Essay and objective

26 5 6 5

Weight toward final grade Same for both classes : either no weight or complete determination of grade Same for both classes : partial weight More weight for conventional exam

7 8 11

Note. PSI = personalized system of instruction. From "A Meta-Analysis of Outcome Studies of Keller's Personalized System of Instruction" by J. A. Kulik, C -L. C. Kulik, and P. A. Cohen, 1979, American Psychologist, 34, p. 315. Copyright 1979 by American Psychological Association.

a course. Course policies which limit self-pacing, especially those which provide positive incentives for student progress, have success­ fully reduced procrastination and lowered withdrawal rates in PSI courses. (p. 10) Regarding the remaining components of Keller Plan courses, explicit unit objectives, review units, and optional lectures for stimulation, the Michigan team found little evidence in favor of explicit objectives and none in support of the lectures. There was support for review units, but it came from only a small number of studies.

Research on Facets of Teaching Behavior The research reviewed in this section concerns aspects, quali­ ties, or facets of classroom behavior that might be found in any of the teaching methods described in the previous section. Rosenshine (1971 ), Dunkin and Biddle (1974), and Simon and Boyer (1974) delineated several facets of teaching behavior emerging from research on teaching. They included the follow­ ing: a cognitive facet concerned with intellectual properties, such as levels of thinking required to answer different types of questions; a socioemotional facet associated with displays of feelings, sanctioning behaviors, such as praise and criticism, and initiation and response; a substantive facet to which subject­ matter variations, such as the amount of content in a lecture, are relevant; and a communication facet involving properties of language use, such as clarity, fluency, and expressiveness. In part, these facets have been researched through the application of observational schedules to teaching as it occurs in naturalis­ tic teaching contexts. In some cases, however, the research has been experimental in that researchers have intervened to regu­ late the occurrence of the behaviors concerned. Most of the facets have had long histories of research at various levels of education. The main facets in terms of which the research reviewed in this section is organized are the cognitive and the socioemo­ tional. The communication facet is represented by research on clarity, but research on another category of behavior within that facet, expressiveness, is postponed until a later section. Research on the substantive facet, especially that on content coverage, is also postponed until a later section. Because research on expressiveness and content coverage is better known in higher education in association with research on student evaluations of teaching, they are both discussed under

760

MICHAEL J. DUNKIN

that heading. The section concludes with research on combina­ tions or syntheses of such facets as those just listed as they have been identified through the application of such statistical techniques as cluster analysis to data about specific categories of teaching behavior. Where the concerns and volume of studies and findings warrant, the discussion of research relevant to these facets is organized in terms of the frequency of occurrence of processes pertaining to the facets, and relationships involving those processes with other processes, with context variables, with characteristics of teachers and students (presage vari­ ables), and with outcomes of instruction (product variables).

Cognitive Operations in the Classroom Research on the cognitive processes in college level classrooms was a feature of the late 1970s and early 1980s. Attempts to classify teacher behavior in the cognitive domain have relied mainly on the concepts developed by Bloom and his colleagues (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956) and by Guilford (1956). Four such studies were reviewed by Wenk and Menges (1982). The Bloom taxonomy was developed in relation to state­ ments of educational objectives concerning knowledge and intellectual abilities and skills. The taxonomy was arranged in a hierarchy of six major classes of cognitive objectives, defined so that the intended behaviors in one class were likely to depend upon the intended behaviors included in lower levels of the hierarchy. The six major classes were Knowledge; Comprehen­ sion (Translation, Interpretation, Extrapolation); Application; Analysis; Synthesis; and Evaluation. Distinctions between lower- and higher-level classroom pro­ cesses have been used in relation to undergraduate education courses by Baumgart (1972, 1976); to college biology by Merlino (1977); in microbiology laboratory classes by Hegarty (1978, 1979); in science laboratory classes by Shymansky and Penick (1979), and Kyle, Penick, and Shymansky (1980); in literature, history, and humanities classes by Andrews (1980); and in clinical discussions in medical education by Foster (1981). Variations in the precise definitions of higher- and lower-level processes have existed across these studies. For example, Hegarty regarded Knowledge and the Translation subcategory of Comprehension as lower level, with the Inter­ pretation subcategory of Comprehension, together with Appli­ cation, Analysis, Synthesis, and Evaluation categories being referred to as "scientific processes." By contrast, Andrews included Memorization, Comprehension, and Application in his concept of lower-level processes. Shymansky and Penick (1979) and Kyle et al. (1980) distinguished between factual recall questions and extended thought questions. Barnes (1980) also used the Bloom taxonomy in her observa­ tions of classes across several disciplines. However, she did not dichotomize the categories into higher and lower levels. Smith (1977, 1980) used the taxonomy with classes in the humanities, social sciences, and natural sciences, but only with respect to students' reports of their out-of-class study behaviors and not in the observation of classroom processes. Guilford's (1956) three-dimensional model of the intellect was constructed on the idea that an intellectual ability involves

engaging a particular type of cogmuve operation, upon a particular type of content, in order to generate a particular type of product. Each combination of operation, content, and pro­ duct denoted a specific ability. It has been the operation facet of Guilford's model that classroom observers have found particu­ larly useful. Guilford posited five categories of operation: Cognition, Memory, Divergent Production, Convergent Pro­ duction, and Evaluation. In classroom applications of Guilford's categories, Cognition and Memory are usually combined to form a category similar to Bloom's Knowledge category. The Cognitive-Memory cate­ gory has then been rated as lower-level processing, with the remaining types constituting higher-level processing. Further­ more, the distinction between Convergent Thinking and Diver­ gent Thinking has allowed some researchers to investigate something akin to the creativity continuum (Dunkin & Biddle, 1974). Other systems for analyzing cognitive processes in class­ rooms that have been used in higher education are those devised by Bellack, Kliebard, Hyman, and Smith, (1966) and Smith (1971). The Bellack system is a complex one containing several facets, one of which was named the Substantive-Logi­ cal. Verbal behavior concerned with the substantive content of lessons was coded according to the type of logical process involved. The logical processes included were Defining, Inter­ preting, Fact Stating, Explaining, Opining, and Justifying. Aubrecht (1978) used these and other Bellack concepts in her study of classes in basic physics courses at a large midwestern university. The categories of Defining, Fact Stating, and Opin­ ing were regarded together as "lower" or "simpler thought processes," while Interpreting, Explaining, and Justifying were "higher" or "more complex processes " (p. 326). Powell (1974) used categories similar to the Bellack ones in his study of small group teaching methods across several disciplines. Smith's system (1971) was devised principally to observe "inquiry" as contrasted with "verification" or "investigative" behavior in science laboratory classes. Tamir (1977) used a modification of the system to compare university and high school classes. The findings of research using the Bloom, Guilford, Bellack, and Smith categories in observing cognitive qualities of teacher and student behavior in college and university classrooms have represented the various types suggested in the "Conceptual Framework" presented here. Below they are grouped in accor­ dance with those types as process occurrence, context-process relationships, presage-process relationships, process-process relationships, and process-product relationships. PROCESS OCCURRENCE

In all but one (Smith, 1977; 1980) of the studies in which process occurrence data were reported, the proportion of the observa­ tional units coded as lower level was greater than the propor­ tion coded as higher level (Baumgart, 1972; Barnes, 1980; Foster, 1981; Hegarty, 1978, 1979; Powell, 1974; Shymansky & Penick, 1979). The between-study mean for lower level was 60.63% (N = 7, S.D. = 27.27) and for higher level it was 26.88% (N = 7, S.D. = 16.17) with the rest being concerned with managerial topics. Smith's findings (1977, 1980), particu­ larly that greater use was made of higher than lower cognitive

RESEARCH ON TEACHING IN HIGHER EDUCATION

levels by students when participating in classroom interaction, are quite different from the findings of other studies, although they did reveal an emphasis on teachers' questions at the Cognitive-Memory level. Tamir (1977) observed teacher be­ havior in laboratory classes in chemistry, biology, physiology, and histology at the Hebrew University in Israel. He found, with data aggregated across the four disciplines, that verifica­ tion behaviors occurred twice as often as inquiry behaviors and concluded that the latter were given a low priority. To these findings concerning the occurrence of various types of cognitive processes should be added the findings of Aubrecht (1978) to be discussed later in this chapter. CONTEXT-PROCESS RELATIONSHIPS

Findings of this type concern differences between institutions, disciplines, and curricula within the one discipline, and types of leadership. Hegarty (1978) reported differences in cognitive levels between the two Australian universities in which she observed microbiology laboratory classes. She found quite large differences in the proportions of tutor talk to students' talk at the lower cognitive level (89% at one university and 67% at the other) and the higher cognitive level (12% and 33%) when the focus was on content. At one university there was also much more tutor-to-tutor talk at the higher cognitive level than at the other. These differences seemed to be consistent with differences between the laboratory manuals of the two institutions. When Hegarty (1978) did a detailed examination of the relationship between the cognitive emphases of the laboratory manuals and laboratory behavior in one of the universities, she found as follows: As the level of enquiry increased, there was a marked increase in the proportion of tutor's time spent on the development of substantive content, especially at low cognitive levels, and a concomitant marked increase in students' time spent listening to such talk. There was a slight trend towards a similar increase in the proportion of students' time spent on talk at low cognitive levels. Most impor­ tantly, there was a marked increase in time spent by students talking on scientific processes, most of which was accounted for by talk with a fellow student not a tutor, as target. (p. 50)

The results concerning tutor talk on scientific (higher-level) processes were not consistent. As the level of inquiry rose from exercises in which aims, materials, methods, and answers were all given, to those in which the aim was given but only part of the materials or part of the methods were given, with the answer left open, the percentage of time tutors talked on scientific processes increased slightly. However, when the level rose to exercises in which only the aims were given, the percentage of tutor talk on scientific processes decreased to below what it was for exercises of the lowest level of inquiry. During the more "open" exercises, there was a sharp increase in the amount of time tutors spent out of the classroom collecting and organizing materials in preparation areas and private laboratories. In her attempts to account for this finding, Hegarty considered evi­ dence from interviews with the tutors that they sought to perform the role of resource persons rather than challengers or promoters of inquiry behavior. She also considered the possibi­ lity that tutors felt threatened by not being able to act as authorities in relation to the more open exercises, and not

761

having been trained in the skills required for such exercises, sought security in management activities, especially those out­ side the classroom. Tamir (1977) also compared the relative occurrence of in­ quiry behavior in laboratory classes at the Hebrew University with their occurrence in high schools. He found that the ratio of inquiry to verification behaviors in high schools was more than twice that in the university. In the high schools inquiry behavior seemed to be elicited particularly during postlaboratory ses­ sions, which did not occur in the university classes observed. The difference was attributed by Tamir to the university's apparent adherence to"the traditional approach in which every measure is taken to ensure smooth and safe completion of the task by providing detailed instructions, by eliminating any difficulty, and by guarding the students against any 'mistakes' that may lead them to obtain unpredicted results" (Tamir, 1977, p. 315). Barnes (1980) found that differences across large and small, public and private colleges in the cognitive qualities of ques­ tions asked by professors were statistically significant (x2 = 27.09, df = 9, p < .01). The two large colleges were found to have a significantly higher proportion of convergent questions than the two smaller colleges. The large private college dis­ played the highest incidence of divergent questions, with the small public college having the lowest percentages of both divergent and evaluative questions. Both Barnes (1980) and Kyle et al., (1980) found question type to be independent of beginning and advanced course status, but dependent upon type of discipline. Barnes found that courses in the humanities/social science/arts disciplines had a significantly lower proportion of cognitive memory questions than courses in the mathematics/engineering/science group. Kyle et al., (1980) found differences among five science disci­ plines in the occurrence of both extended thought and factual recall questions. The occurrence of the former was so rare in all disciplines that the differences discovered have little meaning. Chemistry and physics tended to have higher incidences of factual recall questions than botany, geology, and zoology. Powell (1974) found that there was more student talk in discussion groups not led by tutors than in those that were led by tutors. However, there was little or no difference in the cognitive qualities of the discussion with or without the tutor. PRESAGE-PROCESS RELATIONSHIPS

Only two studies of those reviewed included these types of relationships. Aubrecht (1978) conducted a faculty develop­ ment experiment to be described more fully later in this chapter. Foster (1981) measured students' cognitive and affective char­ acteristics on entry to a clinical stage of a medical education program. She then explored the association between students' entry characteristics and their behavior in clinical discussion groups. Performance on the Medical College Admission Test (MCAT), the Watson-Glaser Critical Thinking Appraisal Test A (CTA-A), and the Flexibility scale of the California Psycho­ logical Inventory (CPI) accounted for significant but small proportions of the variance in students' classroom use of higher cognitive levels (4.5%, 2.9%, and 2.1%, respectively). In total, students' entry characteristics accounted for approximately

762

MICHAEL J. DUNKIN

10% of the variance in student use of higher cognitive levels in classroom discussion. PROCESS-PROCESS RELATIONSHIPS

Barnes (1980) investigated. several types of process-process relationships, but either found no significant relationships or significant but very small ones between professors' questions and students' cognitive levels. Andrews (1980) tested hypotheses involving questioning variables, drawn from both the Guilford and Bloom concepts of cognitive processes, in relation to student participation and "wait-time," the time elapsing between a teacher's question and a student response. He found that the mean number of student statements was significantly higher following divergent than following convergent questions. He also found that higher-level questions (those demanding analysis, synthesis, or evaluation) were followed by a larger mean number of student statements, but not longer wait-time, than low-level questions (those in­ volving memorization, comprehension, and application). These findings were obtained with the individual question as the unit of statistical analysis. He also developed a concept of "teaching style" that involved using the teaching assistant as the unit of analysis. A significant positive relationship was found between teacher use of divergent questions and the mean class number of student statements (Spearman r = .82, p < .03). The correlation between convergent questions and student state­ ments was sizeable, but, given a small number of teaching assistants, was not statistically significant (Spearman r = .60, p < .15). With the question used as the unit of analysis, Andrews examined the combined effects of the Guilford and Bloom categorizations. He found that high-level divergent questions were followed by significantly more student statements than low-level divergent questions. However, the same difference within convergent questions was not found. Andrews suggested that the single-correct-answer nature of convergent questions might inhibit the richness of response that higher-level ques­ tions otherwise elicit. Andrews also researched cognitive consistency in questions asked by teaching assistants. He noticed that sometimes, when more than one question was asked at a time, the questions varied in cognitive level, possibly creating confusion on the part of respondents. It was found that among multiple questions, those that contained logical inconsistencies were followed by significantly longer periods of wait-time, or "student thinking time," than those without inconsistencies. However, multiple consistent questions were followed by significantly more stu­ dent statements. When consistency was present in both multi­ ple and single questioning episodes, the former tended to be followed by more student statements, but shorter wait-time than the latter. Furthermore, multiple, divergent questions that were consistent were followed by significantly more student statements than multiple, inconsistent, divergent questions. Andrews' pursuit of questioning led him, thus, to the concep­ tualization of four attributes: divergence/convergence; higher­ leveljlow-level; single ("straightforward")/multiple; and consis­ tency ("structured")/inconsistency ("vague"). Various combin­ ations of these led further to the presentation of 11 different

types of questions. Three types, which Andrews grouped to­ gether as "Structured Divergent Questions," were found to be the most productive of student statements in response, together producing nearly three times as many as the other types together. Foster (1981) found significant positive correlations, each of about .30, between Knowledge, Comprehension, and Applica­ tion types of teachers' questions and students' responses at each of those levels. At the higher levels, the coefficient for the combined Analysis, Synthesis, and Evaluation categories was .43 (p < .01). Significant negative correlations were found between teachers' use of Knowledge questions and students' use of Application (r = - .36, p < .01), and between teachers' use of Application and students' use of combined Analysis, Synthe­ sis, and Evaluation (r = - .28, p < .01). In general, therefore, it seems that the cognitive quality of teachers' questions in higher education has a marked influence on student classroom behavior. Not only does the cognitive level of students' responses tend to reflect the demands placed upon them by their teachers, so that lower-level responses tend to be elicited by lower-level questions, but also the amount of student participation seems to be affected. Andrews' elabora­ tions in terms of the single versus multiple and the consistent versus inconsistent aspects of questioning behavior are certain­ ly worthy of further exploration in higher education contexts. PROCESS-PRODUCT RELATIONSHIPS

Merlino (1977) found that when teachers were grouped accord­ ing to the percentage of higher level questions asked during class sessions, significant gains in the development of students' understanding of key biological concepts were found to vary positively with that percentage. In addition, the greater the proportion of higher-level questions asked, the more positive were students' evaluations of the teaching they received. Smith (1977, 1980) used canonical correlation techniques to explore relationships between several classroom behavior variables and two student product variables. The product variables consisted of class mean standardized change scores on the Watson­ Glaser Test of Critical Thinking Ability and class reports of their use of various cognitive levels during out-of-class study, as measured with an instrument developed by Chickering (1972) based on the Bloom taxonomy. The set of six process variables consisted of student participation, divergent and evaluative questions, encouragement, and peer-to-peer interaction. Smith recognized disagreement among commentators on whether the unit of analysis in this type of research should be the student or the class and so decided to report results obtained when both units were used. Because Smith used the teacher as the sampling unit and did not investigate intraclass differences involving process variables, it would appear that the teacher is the more appropriate unit of analysis and so only results obtained on that basis are presented here. Unfortunately, the smallness of the sample of teachers (N = 12) was a severe limitation on the study. The canonical correlation coefficient between the set of six process variables listed here and the set of seven product variables was found to be .96 (p < .10). When the process variables were correlated with the class mean scores on the set

RESEARCH ON TEACHING IN HIGHER EDUCATION

of six study behaviors only, a perfect coefficient of 1.00 (p < .001) was obtained. These coefficients are so large as to be quite rare in research on teaching and should be accepted and interpreted with caution. When the percentage of divergent and evaluative questions combined was studied in relation to each of the seven product variables separately, only the correlation with the use of evaluation during study was statistically significant (r = .61, p < .05). The corresponding coefficient for divergent and evaluative student responses was also statistically significant (r = .57, p < .05). Neither of these two cognitive process variables was correlated substantially or significantly with class mean change in critical thinking scores or with any of the other five types of study behaviors. In general, and as will be seen in the section on Socioemo­ tional Qualities later in this chapter, Smith obtained stronger results for the noncognitive classroom processes he researched. He attributed the difference mainly to the narrow range of incidence of questioning behavior observed in the 12 classes. Given Smith's results for process occurrence, the maximum encounter with teachers' questions of the combined divergent and evaluative types could only have been an average of two or three per session. It is, then, surprising that any statistically significant, let alone sizeable, results were obtained with that variable. Foster (1981) found that presage variables represented by student medical achievement at the beginning of the year accounted for almost 57% of the variance in the product variable consisting of end-of-year scores on the NBME examin­ ation; that the presage variable, student MCAT scores, ac­ counted for 3%; and that the process variable, student higher cognitive level talk, accounted for less than 1%. When the Watson-Glaser form B (CTA-B) scores were used as the product variable, performance on the Watson-Glaser form A test accounted for about 24% of the variance. CPI Achievement via Conformance scores and student higher cognitive level talk accounted for 5% and less than 1%, respectively. However, zero-order correlation coefficients between students' use of higher cognitive processes and the NBME and CTA-B pro­ ducts were .30 and .36, respectively. Apparently, the partialling out of the variance in the multiple regression analyses greatly affected the measured contributions of the student higher-level process variable. Among her conclusions, Foster (1981) wrote that "the find­ ings would seem to suggest . .. that it is the better-prepared students who will perform better on tests and that they are incidentally the ones who talk more at the higher cognitive levels in class" (p. 839). She went on to caution against concluding that the value of clinical discussion groups is questionable, pointing out the limitations in the type of medical achievement tests used in the study and reporting that 89% of students considered the discussion groups to be valuable. The results of the three studies of process-product relation­ ships involving cognitive levels of classroom behavior suggest that such behaviors are positively related to achievement criteria. However, the contribution of cognitive level process variables to achievement products tended to be small in com­ parison with student presage variables and other process vari­ ables. Attempts to explain these small effects have suggested

763

that they might be due to design and measurement flaws in the research, such as insufficient variance in the process variables themselves, or inappropriate achievement tests. In summary, observational research on cognitive operations in higher education classrooms has been based on concepts of behavior used for many years in other contexts. Acceptance of dichotomies such as higher- and lower-level questions, and convergent and divergent thought, has been accompanied by inconsistencies among the precise definitions of those terms, so that the search for an accumulation of knowledge involving them is hazardous. Except for the work of Andrews (1980), there is little evidence of concern with refining concepts relevant to the cognitive facet of classroom behavior. As in other contexts of education, it seems that lower-level and convergent types of cognitive operations are predominant in higher education. Demands upon memorizing factual infor­ mation are more apparent than demands for the complex intellectual processing of information and conventional answers seem more the norm than new and unusual ones. The balance is not, however, impervious to influence, for it seems that the cognitive quality of classrooms may be dependent upon the types of instructional materials used. For example, inquiry-oriented laboratory exercises were found to be accom­ panied by larger proportions of higher cognitive categories in the verbal exchanges during laboratory classes (Hegarty, 1978). Furthermore, the evidence suggests that teachers have the power to vary the intellectual climate of classrooms by asking questions that stimulate desired types of responses by students. The matter is not uncomplicated, however, and attempts to manipulate the cognitive processes of classrooms sometimes have undesirable side effects. As Hegarty implied, teachers pressed into unfamiliar roles involving higher-level processes rather than lower-level ones might respond by escaping from the presence of their students, and as Andrews (1980) found, some attempts by teachers to vary the level of thought lead to vagueness, inconsistency, and confusion. Overcoming such problems is a challenge for faculty development programs such as those discussed later in this chapter. The findings concerning differences in cognitive processes according to type of institution and academic discipline are too few to be more than suggestive. Research that is designed to explicate the precise ways in which such variables as size or status of institution filter down to affect occurrences in class­ rooms is badly needed. It is to be expected that curriculum and staff development procedures and associated resources play important intermediary roles in that process. It is also an important possibility that differences in the content and struc­ ture of disciplines are manifested in the ways in which they are taught. Donald's work (1980) at McGill University in Canada is a good example of interest in this area. It has become commonplace for researchers to report that student entry characteristics, especially prior academic ability and achievement, are more strongly associated with subsequent achievements than processes exhibited in classrooms. Gage (1978) has argued that such findings should be expected and should not be used to denigrate the value of teaching, or research on process-product relationships, for such presage variables as prior achievement are themselves likely to repre­ sent the cumulative effects of teaching over many years,

764

MICHAEL J. DUNKIN

especially by the time students enter higher education. The few studies that have investigated relationships between cognitive processes and student achievement in higher education have produced inconsistent results and do not permit a conclusion to be reached. Similarly, relationships between cognitive processes and other product variables, such as students' study behaviors and evaluations of teaching, have only been explored in one or two studies and so conclusions there would also be grossly premature.

Socioemotional Qualities The most commonly used approach to the analysis of socio­ emotional qualities of teaching processes in higher education contexts has involved the application of the Flanders Interac­ tion Analysis Categories (FIAC; Amidon & Flanders, 1963) or a modification of them to analyze classroom behavior by both teachers and students. PROCESS OCCURRENCE

Smith (1977, 1980) applied a modification of FIAC to classes conducted by 12 faculty members in the humanities, social sciences, and natural sciences in a small liberal arts college in the United States. Adequate levels of intra- and interobserver agreement were attained over the four sessions observed for each teacher. Smith found that only about 4% of class time was spent on the teacher's praising, encouraging, and using stu­ dents' ideas and that student talk accounted for a total of 14% of class time. By implication, the incidence of teacher criticism was less than the incidence of teacher support. Barnes' (1980) study of teaching in four institutions found that 48% of sequences initiated by teachers' questions con­ tained students' responses, and that 11% contained teacher acceptance or use of students' ideas. Almost two thirds of the sequences in which there were students' responses contained no reaction from the teacher other than a resumption of lecturing. Foster (1981) found in clinical discussion groups in medicine that teacher support (encouragement and praise) occupied approximately 4% of class time, while teacher criticism occu­ pied less than 1% of that time. Students' responses and initia­ tions occupied about 10% and 8% of class time, respectively. Another 24% consisted of students' reports, with teachers' lecturing (46%) and teachers' questions (7%) occupying almost all of the balance. Baumgart (1972) used a system developed by Benne and Sheats (1948) for analyzing small group discussions into task functions and group building functions. He found that group building functions including showing solidarity, encouraging, harmonizing, and compromising comprised about 16% of task and group building functions combined, while disagreeing, showing tension, and showing antagonism accounted for a total of about 5%. Baumgart also found that in these discussions 35% of the talk was from the teachers, while 65% was by the students. Teachers did about 68% of the structuring and soliciting, but 13% and 28% of the responding and reacting, respectively. Students in this context had much more active roles than the students in New York high schools observed by Bellack and his colleagues (Bellack et al., 1966), and, appar-

ently, the students in the other contexts studied in the research reviewed in this section. Apart from Baumgart's findings, the trend from these few studies seems to be for affectivity to play a minor role in classrooms in higher education and for students to take rela­ tively passive parts. Baumgart's data were gathered in an institution established partly on the basis of a commitment to small group teaching in which perhaps two thirds of students' class time was spent. CONTEXT-PROCESS RELATIONSHIPS

Barnes (1980) investigated differences according to status of course (beginning versus advanced) and discipline (humanities versus math/science) in the frequency of occurrence of the five most common questioning patterns she identified. She found that advanced courses contained patterns including teacher acceptance or use of students' ideas almost twice as often as beginning courses, and that those patterns were three times as frequent in the humanities as in the math/science disciplines. Both Barnes (1980) and Hegarty (1979) found differences in socioemotional qualities between the institutions they com­ pared. Hegarty (1979) found in the microbiology laboratory classes she investigated that students talked 50% more at one university than at the other and that tutors at the former university talked only half as much as at the latter. At the first university tutors talked twice as much as each student, while at the second, tutors talked seven times as much as each student. At both universities, students almost never experienced or offered verbal acceptance or criticism during the laboratory classes. Barnes found that 51% of teachers' questioning sequences contained students' responses in the two large universities, while the corresponding figure for the two small universities was 41%. In particular, a higher percentage (21%) of those sequences in larger universities contained teacher acceptance and use of students' ideas than in smaller universities (9%). Barnes found only small differences according to whether the universities were public or private. In all, very little concern with contextual effects upon socio­ emotional aspects of classroom behavior is apparent in research in higher education. PRESAGE-PROCESS RELATIONSHIPS

There appears to have been a neglect of research on relation­ ships between college teachers' characteristics, training, and other background variables, on the one hand, and the systemat­ ically observed socioemotional qualities of their classroom behavior on the other. Indeed, the only presage-process rela­ tionship concerning socioemotional aspects of classrooms located in the present review involves students. Foster (1981) found that medical students' scores on the Flexibility seal'?_ of the California Psychological Inventory (CPI) were significantly related to individual student-initiated talk, but accounted for only about 4% of the variance in the latter. PROCESS-PROCESS RELATIONSHIPS

In a study stimulated more by an interest in a general classroom phenomenon than in the higher education context specifically,

RESEARCH ON TEACHING IN HIGHER EDUCATION

Klein (1971) conducted an experiment concerning student influence on teacher behavior. Her study involved as subjects 24 guest lecturers in education ranging from graduate teaching assistants to full professors in six universities in the United States. Regular undergraduate and graduate students applied the experimental treatments in the 24 classes. The students were given instructions as to when to behave normally, when to engage in positive behaviors, and when to engage in negative behaviors during predetermined periods during class times with the 24 teachers. Positive behaviors including smiling, looking at the teacher, and answering questions quickly and correctly. Negative behaviors included frowning, looking out the window, talking to other students, and disagreeing with a teacher's statement. Flanders Interaction Analysis Categories (FIAC) were applied to tape recordings of the class sessions along with live observational instruments of teacher nonverbal behavior and student behavior. Klein (1971) found that the teachers appeared to change their behavior in response to changes in student behavior. She concluded that teachers engaged more in directive and criticizing behavior during periods of negative student behavior than during periods of positive or natural student behavior, and that teachers used clarification more during positive student behavior than during negative student behavior. These findings suggested to Klein that if positive teacher behavior enhances student achievement, and if students can elicit positive teacher behavior, then "students may be encouraged to assume responsibility for their own behavior and purposely help their teachers behave more effectively" (p. 419). As reported earlier in this chapter under process-occurrence, Barnes (1980) found that teachers' questions elicited students' responses only about 50% of the time and that those responses were received with acceptance and/or use by the teacher much less often. It would be interesting to know whether some qualities of students' responses were more likely than others to win such support from teachers. On the basis of Klein's evidence, that would seem likely to the extent that students might even have the potential to manipulate teachers' reac­ tions! PROCESS-PRODUCT RELATIONSHIPS

Smith (1977) found quite large process-product correlations involving student participation, teacher encouragement, and student-to-student interaction. However, given a sample size of only 12, many of those coefficients were not statistically signifi­ cant. Both student participation and teacher encouragement correlated significantly with change in critical thinking (.63 and .62), with use of analysis (.55 and .56), and with use of synthesis (.56 and .55). Student-to-student interaction correlated .57, .52, and .53 with change in critical thinking, use of interpretation, and use of application, respectively. The three socioemotional variables together must have contributed in large proportion to the canonical correlations of .96 and 1.00 reported previously between the sets of process and product variables he researched. Again, the limitations of Smith's study suggest that caution is needed in accepting and interpreting these results. Foster's (1981) study of clinical discussion groups in medi­ cine found that the variables of classroom verbal behavior including student response and initiation, and teacher support

765

and criticism, made negligible contributions to variance in the cognitive outcome measures of medical achievement and post­ test critical thinking ability, compared with the contribution of students' entry performance and pretest critical thinking ability. It would indeed have been surprising if that had not been the case, given that both outcomes were measured with alternative forms of instruments used to measure cognitive attributes on entry and the two forms of each test were probably designed to maximize the correlation between scores obtained on them. Kounin (1970) reported a study of the phenomenon of "the ripple effect" arising from his informal observations of the effect on the audience of his reprimanding a student who was reading a newspaper during a lecture he was giving in a course in mental hygiene. As well as the target student ceasing to read the newspaper, Kounin (1970) noticed dramatic effects on the others: Side glances to others ceased, whispers stopped, eyes went from windows or the instructor to notebooks on the desks. The silence was heavy, as if the students were closing out the classroom and escaping to the safety of a notebook. I believe that if I had sneezed they would have written the sound in their notes. . . . Why were they so affected by an action of the instructor that wasn't even directed at them? (pp. 1-2)

In an experiment stimulated by the incident, two instructors, each teaching two classes in education (a total of four classes) administered a "threatening desist" in one of their classes and a "supportive desist" in the other, in each case to a "student­ stooge" who came very late by arrangement with the experi­ menter. Checks on whether the experimental manipulation "took" confirmed that students were well aware of the two different types of reprimand (p < .001). With data gathered by questionnaires administered before and after each lecture, Kounin found that students' ratings of the instructors' compe­ tence, likability, nonauthoritarianism, and fairness, and their own freedom to communicate about themselves to the instruc­ tor, tended to decrease in every class where threatening desists had been applied. The same was found for supportive desists except for one of the two classes on likability and the two classes on fairness where ratings tended to increase. He also found that decreases in ratings following threatening desists were significantly greater, or nearly so, than those following supportive desists for 8 of the 10 comparisons (2 desists x 5 qualities). However, the fact that students reported being surprised that an instructor would issue a reprimand to a latecomer, and that such behavior was not typical of either instructor, discouraged Kounin from attributing the differences in ratings to the desists themselves. Instead he recommended the "advisability of using teacher style variables that are within student expectations and that have some ecological prevalence" (Kounin, 1970, p. 7). In subsequent research on matters of discipline and group man­ agement in classrooms, Kounin sought contexts other than those in higher education and found strong support for his recommendations. It remains to be seen whether the recom­ mended observation of natural teaching contexts, and of style variables, rather than induced incidents in experimental de­ signs, gives greater promise of understanding such phenomena in higher education.

766

MICHAEL J. DUNKIN

The study by Cranton and Hillgartner (1981) also contri­ buted findings relevant to this section, but discussion of it is postponed until later in this chapter. In summary, research on the socioemotional facet of class­ room behavior in higher education, like that on t_he cognitive facet, has depended greatly upon categories of behavior deve­ loped with respect to other educational contexts much earlier. The only concepts of behavior that seem to have emerged in relation to this facet from observations commenced in higher education are "desist-techniques" and "the ripple effect" gener­ ated by Kounin and his colleagues in the 1950s. Findings regarding the incidence of such categories of teacher talk as praise, acceptance, lecturing, and questioning seem not to be generalizable across the contexts represented in the research reviewed here, where percentages of teacher talk were found to range from a low of 35% (Baumgart, 1972) to a high of 86% (Smith, 1977, 1980) and percentages of student initiation varied from a high of 32% (Baumgart, 1972) to a low of 8% (Foster, 1981). The most consistent finding was that teacher praise, encouragement, and acceptance accounted for less than 5% of total class time (Smith, 1977, 1980; Foster, 198 1). Perhaps one factor that needs to be taken into account in the search for generalizable findings is that teaching in higher education occurs in formats that place limits upon the occurrence of specific categories of behavior. One of the most common of these formats is the lecture, in relation to which it would be unusual to find little more than a token amount of verbal interaction involving such processes as praise, acceptance, and questioning. On the other hand, in the format of the small group discussion, it would be surprising if student talk did not account for a sizeable proportion of the time. It would seem that Smith's ( 1977, 1980) research was conducted within a predominantly lecture format. Baumgart (1972) and Foster (198 1), on the other hand, investigated formats that were explicitly small group discussions and in several respects ob­ tained quite different findings from Smith's. In addition to teaching formats, or methods, contextual factors seem to influence the occurrence of more specific classroom processes. If, as Barnes ( 1980) found, the incidence of teacher supportive behaviors is higher in advanced than in beginning courses, it could be because teachers and students know one another better in the former, and also because only less problematic students survive to advanced levels. These possibilities do not, however, explain why teacher supportive­ ness might be more frequent in the humanities than in the sciences (Barnes, 1980). Perhaps the explanation for that differ­ ence lies in the very basis for the distinction between those two, and the humanities do, after all, turn out to be more humanistic both in content and in personnel. That there are differences involving the socioemotional facet between institutions de­ pending on size (Barnes, 1980) or laboratory manuals (Hegarty, 1978) again suggests the need for research into the processes by which institutional characteristics are mediated to students. It comes as no surprise to learn that student behavior affects teacher behavior, but the idea that students might act in concert to manipulate teacher behavior is bound to startle even the most liberal college teachers. Yet Klein ( 1971) saw such poten­ tial in her findings. Barnes' less dramatic findings ( 1980) that teachers' questions often failed to evoke students' responses

leads to speculation about the qualities that might distinguish effective from ineffective questions. Andrews' research (1980), discussed previously, should prove valuable in that regard. Similarly, students' responses that win teacher acceptance are presumably different from those that do not. It would seem worthwhile to explore those differences together with data about teachers' criteria in distinguishing between the two in reacting positively or otherwise. On the basis of Smith's findings (1977, 1980) that student participation, studenHo-student interaction, and teacher en­ couragement correlated positively and quite strongly with both student growth in critical thinking ability and higher level study processes, college teachers might see value in increasing the occurrence of such processes in their classrooms. On the other hand, Foster (1981) found only trivial effects of variations in such processes. Limitations of both these studies have already been mentioned, so that a healthy skepticism is probably the most appropriate reaction to their results. Similarly, Kounin's evidence (1970) that students' ratings of the instructor were more negative after threatening desists than after supportive desists, while thoroughly plausible, was, in his view, flawed by the possibly confounding effect of students' expectations. That research, too, is in need of replication. Again, there have been too few attempts to investigate relationships between socioemotional aspects of college and university classrooms and product variables to permit more than the listing of individual findings given here. It is interesting that while at lower levels of education, research on socioemo­ tional qualities of classroom behavior has been voluminous, there has been only a handful of studies to review. It is as though the classrooms of higher education are not only inap­ propriate contexts in which to display such things as emotions, but that they are also inappropriate places in which to conduct research on emotions.

Clarity in Teaching One of the most common aspects of teaching included in evaluation instruments is clarity. Most often the rater has been expected to supply his or her own definition of clarity and the specific behavioral components of the concept have not been explained. Rosenshine and Furst ( 1973) reported that in all five studies they located using the variable, significant relationships were found with student achievement. They were particularly impressed with the robustness of the variable because different rating instruments, different types of raters, and different times of rating had been used in measuring it. During the 1970s, some progress was made in identifying the low-inference correlates of clarity and in exploring their relationships with student achievement. Land (1979) described five low-inference variables of teacher clarity which he named: vagueness terms; verbal mazes; specifi­ cation and emphasis; clear transitions; and unexplained addi­ tional content. Vagueness terms were initially reported as a process variable in research on teaching by Hiller, Fisher, and Kaess ( 1969), who conducted analyses of transcripts of high school lessons given in a lecture format on topics in social studies. These researchers took the view that uncertainty upon the part of the teacher about the subject matter being taught

RESEARCH ON TEACHING IN HIGHER EDUCATION

manifested itself in the use of vague terms and phrases, such as "things," "about," "some," "probably," and "may be." Hiller and his colleagues also developed a low inference measure of verbal fluency which was indicated by length of sentences, the appearance of commas in the transcript, and the use of "uhs," "ahs," and other hesitations. This last aspect of fluency was adopted by L. R. Smith (1977) and others in association with "verbal mazes," described as false starts, halts in speech, redundantly stated words, and tangles of words. Specification and emphasis were defined by Land (1979) as "the presence of an explanation of how a concept was an example of the concept definition" (p. 796). The same author defined clear transitions as "the presence of such transitional terms as 'now' and 'the last item "' when the teacher was indicating that one part of a lecture was ending and another part beginning (pp. 796-797). Additional, unexplained content was defined as "extra terms that were related to the lesson but were not essential to the main idea of the lesson" (p. 797). Land (1985) summarized research on these variables at the college level. He reviewed two process-product studies (Land & Smith, 1979a, 1979b) of mathematics classes at the college level which had experimentally manipulated the frequency of vague­ ness terms used by the teacher and had then sought effects on student achievement. In each case statistically significant (p < .05) or near significant (p < .07) negative effects on vagueness were found and the proportions of variance accounted for in unadjusted student achievement scores ranged from 2% to 8%. Land (1985) also reviewed process-product studies at the college level of the effects of combinations of all the clarity variables described previously. Three such studies (Denham & Land, 1981; Land, 1979, 1980) found significant effects in favor of clear expositions with clarity accounting for 20%, 8%, and 6% of the variance in student achievement in psychology. Land and Smith (1981) found no significant effects in college level social studies for vagueness terms and verbal mazes combined, but Land (1981) found a significant effect for that combination accounting for 6% of the variance in favor of clarity in college level mathematics. In both studies, large effects of the clarity combination were found on students' perceptions of teacher clarity. The process variable, clarity, accounted for 59% of the variance in the product variable, student perceptions of clarity, in Land's 1981 study and 32% in Land and Smith's study (1981). That students were so sensitive to variations in clarity is evidence that the experimental treatment "took." Land (1985) concluded his review of research on low­ inference measures of clarity as follows: On the basis of natural classroom studies, low-inference variables of clarity can be broadly divided into those that inhibit learning (e.g. vagueness terms), and those that facilitate learning (e.g. signaling transitions). More is known about two of these variables - vague­ ness terms and verbal mazes - than is known about other low­ inference clarity variables. Additional research in delineating other low-inference clarity variables and their effects (singularly and in combination) on student perception and achievement is needed. (p. 5410)

Land went on to point out the further need to apply the findings of such research in attempts to change teachers'

767

behavior and test the effectiveness of such change in enhancing student achievement. In a study using a different approach to measuring clarity Hines, Cruickshank, and Kennedy (1982) obtained observer ratings on a cluster of 29 different low-inference variables thought to comprise clarity in teaching. In the college level mathematics classes studied, variations in clarity were found to account for 52% of the variance in mean class achievement (p < .03). That a single variable of teaching behavior should account for so much variance in mean class achievement is unusual and perhaps should be accepted with caution. Clarity as observed on the 29 variables was also found to be strongly related to students' perceptions of teacher clarity. Those percep­ tions, in turn, accounted for 28% of the variance in mean class achievement and related strongly to student satisfaction. It seemed as though student perceptions of clarity in teaching mediated the effects of observed clarity upon both student satisfaction and achievement. As with research on teacher clarity in other settings, studies in the higher education context indicate that the clarity con­ struct has both predictive and concurrent validity in terms of a variety of product criteria and when operationalized in different ways. The role of student perceptions in the research has been interesting. On the one hand, evidence that students are sensi­ tive to variations in clarity establishes that experimental treat­ ments, where applied, did "take." On the other hand, evidence of a similar kind has led to an elaboration of theory, such that student perceptions have been seen as mediating variables between the objectively present variations in teacher clarity and other product variables of student achievement and student evaluations of the teaching. The role thus accorded to student perceptions of variations in teaching has important implica­ tions for research on student evaluations of teaching and is discussed at greater length later in this chapter. The effects of teachers' use of vagueness terms and verbal mazes upon student achievement have been consistently negative. The implications of these findings for the improvement of teaching are, however, in need of research. It is not yet established whether vagueness terms and the elements of verbal mazes are language impedi­ ments that can be eliminated through training in verbal expres­ sion or whether the problems are rooted in teacher lack of mastery of subject matter, requiring more academic develop­ ment. The study by Hiller (1971) suggested the latter, but there is a need for further study, preferably in higher education contexts. In the meantime, teachers in colleges and universities can be reasonably confident that students exposed to teaching that is high in clarity tend to achieve at a higher level and to evaluate that teaching more positively than students experien­ cing teaching that is low in clarity.

Syntheses of Teaching Behavior Research on teaching in higher education contains at least three approaches to the identification and description of ways in which separate categories and facets of teaching behavior cohere or are synthesized into patterns. Such syntheses, if seen to typify the behavior of some teachers over time, would seem to underlie concepts such as teaching styles and roles. One

768

MICHAEL J. DUNKIN

approach is the application of system-based observational techniques to yield quantitative descriptive data about varia­ tions in the occurrence of different categories and facets of classroom behavior and then to explore the extent to which they form clusters (Baumgart, 1976). Another approach de­ pends upon quantitative information about patterns of teach­ ing behavior as perceived by teachers themselves and conveyed through self-reports (Brown, Bakhtar, & Youngman, 1982). A third approach is the ethnographic one (Cooper, 1981a, 1981b; Cooper, Henry, Korzenny, & Yelon, undated; Cooper, Orban, Henry, & Townsend, undated; Yelon, Cooper, Henry, Kor­ zenny & Alexander, 1980). Only the first two approaches are represented in this subsection. The ethnographic approach as applied to higher education is still very much in its infancy and to date has yielded only a handful of case studies. Baumgart obtained audiotapes of SO-minute meetings in 29 small groups led by 20 different tutors in nine undergraduate courses in education at a university in Sydney, Australia. After transcribing the tapes, Baumgart analyzed the transcripts using category systems devised by Bellack et al., (1966), Benne and Sheats ( 1948), and a system for coding cognitive level derived from the work of Bloom et al., (1956), Gallagher and Aschner (1963), and Taha and Elzey ( 1964). Data on a final set of 37 classroom behavior variables were obtained. Two different methods of cluster analysis were applied and both revealed similar patterns of tutor or group leader behavior. In each case six "behaviourally differentiated tutor roles" (Baumgart, 1 976, p. 3 1 1) were identified as Reflexive Judge, Data Input, Stage Setter, Elaborator, Probe, and Cognitive Engineer. Baumgart found that students tended to engage in lower proportions of utterances concerned with facts and opinions and higher proportions of utterances concerned with processes including translation, interpretation, and generalization when tutors were performing more the role of reflexive judge. When the role of probe was emphasized, students also engaged in a higher proportion of processes involving explanation, analysis, and synthesis. The two roles, reflexive judge and probe, also tended to attract higher student ratings of the worth of the small group discussions, although the result for probing might have been an artifact of its overlap with the reflexive judge role. Baumgart's report indicated that his typology was useful in teacher development activities through modeling, role playing, and videotape feedback. Brown et al., (1982) focused their attention upon the styles of lecturing of 258 lecturers in two English universities in relation to subject areas, academic status, and years of experience. A 60item self-report inventory was administered and the responses factor analyzed to generate six scales, labeled as follows: Information Giving; Structured Lecture; Purposive Lecture; Visualized Lecture; Self-Doubt Lecture; and Presentation. Cluster analysis was used to yield five clusters of lectures, each having a distinctive pattern of lecturing behavior. These five patterns are described as Oral Lecturer, Exemplary Lecturer, Information Providers, Amorphous Lecturers, and Self-Doubters. Brown and his colleagues found a strong association among lecturing patterns and subject areas, with Oral Lecturers more common in the humanities and social sciences, Exemplaries more often found in biomedical sciences, and Information Providers and Amorphous Lecturers more frequent in science and engineering.

Length of experience was found to be unrelated to lecturing patterns, but a trend was noted for the Exemplary style to be more frequent among professors and for Information Providing and Amorphous styles to occur more among lecturers. These two attempts to provide empirically based typologies of teaching style are quite different from each other in spite of the approach to statistical analysis common to both. They investigated two different teaching formats, the small group discussion and the lecture. One used system-based observation­ al techniques to gather data about process variables, while the other used a self-report inventory yielding teachers' perceptions of themselves. Baumgart found evidence chiefly favoring the reflexive judge role, which students responded to with higher thought levels during the discussions themselves, and with more positive evaluations following them. Brown and his associates found further evidence of differ­ ences in classroom processes among disciplines, raising again the question of the connection between structures of knowledge and ways of teaching. Their finding a relationship between academic status and lecturing pattern is unique in the research reviewed in this chapter and evokes several plausible interpreta­ tions. Are some teaching patterns more conducive to promo­ tion than others? Does increased mastery of an academic discipline lead to the adoption of some teaching styles rather than others? Or, given the methodology of the study, do full professors merely see themselves differently from others, and if so, why? The studies by Baumgart and by Brown and his colleagues demonstrate the potential value of research into patterns of teaching behavior in higher education. They show that such patterns can be involved importantly in several different types of relationships ranging from context-process to process­ product associations. There is a need for much more research of this kind in higher education. It is especially valuable in that it provides knowledge of naturally occurring syntheses of teach­ ing behaviors that complements knowledge about the prescrip­ tively derived teaching methods reviewed earlier in this chapter.

Research on Evaluating and Improving Teaching The evaluation and improvement of instruction are closely linked in research in higher education, because evaluation has been seen as an instrument for enhancing the quality of teaching. In this section, research on student evaluations of teaching is reviewed and evidence relating to their credibility and usefulness in improving teaching is emphasized. A second main concern of this section is research on other approaches to the improvement of teaching, such as small grants schemes, faculty seminars and workshops, and microteaching.

Student Evaluations of Teaching Research in higher education in the 1970s gave particular emphasis to student evaluations of teaching. The literature on relationships between student ratings of instruction and student

RESEARCH ON TEACHING IN HIGHER EDUCATION

achievement was found to contain some hundreds of titles at the end of the decade, while comprehensive reviews of that research numbered at least a dozen (Cohen, 1981). Research concerned with other matters, including the effectiveness of feedback based on student ratings, was reviewed by Rotem and Glasman (1979), by Cohen (1980), and by Levinson-Rose and Menges (1981). Those reviews involved many additional stu­ dies. To qualify for inclusion in this chapter, research on student evaluations of teaching should include teaching process vari­ ables in its design. However, to invoke that criterion rigorously would be to exclude most of the research conducted in this important area. Instead, some research that does not satisfy the criterion is reviewed here because it illuminates issues related to research involving teaching process variables. Research on student evaluations that includes teaching process variables is important not just because it satisfies the definition of research on teaching advanced at the beginning of this chapter. It is important mainly because it is crucial to establish just what it is specifically that students respond to when they make judgments about the worth of the teaching they experience. Presumably, student evaluations are based, at least partially, on their perceptions of the teaching they receive and, presumably, those perceptions are accurate in that they reflect the actual processes engaged in by the teachers. If these presumptions are not justified, then it is difficult to see how student evaluations can be legitimately useful except in terms of intrinsic benefits for those making them, such as in venting their spleens or express­ ing their pleasure. It would be naive to expect that student evaluations would be impervious to influence also by students' characteristics and by the contexts in which they are taught. Psychological research over many decades has demonstrated clearly that personal attributes and general features of the environment do influence perceptions and judgments and there is no reason to expect that students are immune to these influences in their perceptions and evaluations of teaching. The usefulness of student evalu­ ations does not depend on their being free of such influences, so much as in the ability to take account of them. The several issues involved in the preceding discussion form the basis for the organization of this review of research on student evaluations of teaching in higher education. The first issues concern the relationships between teaching behaviors, on the one hand, and student perceptions and evaluations, on the other. These are process-product relationships in terms of the conceptual framework adopted in this chapter. In the course of discussing those relationships, research concerned with other types of relationships, such as between student characteristics and student evaluations (presage-product relationships) and environmental influences upon student evaluations (context­ product relationships) is relevant and is summarized. The latter part of the section reviews research on the effects of feedback from student evaluations upon the improvement of teaching.

PROCESS-PRODUCT RELATIONSHIPS

Some examples of research on the sensitivity of students to variations in classroom processes was reviewed in the previous section of this chapter. Included there was the study reported by

769

Kounin (1970) in which college students were found to be aware of experimental manipulations of threatening and non­ threatening desists by teachers and to evaluate the latter accordingly. Also, there were the studies by Land (1981), Land and Smith (1981), and Hines et al. (1982) in which college students' perceptions of variations in teacher clarity were found to be positively associated with actual variations in clarity exhibited in the classroom. The study by Hines and his col­ leagues went one step further and demonstrated a strong relationship between student perceptions and student satisfac­ tion. That study provided evidence along the whole chain of relationships stipulated earlier as necessary to establish the credibility of student evaluations. Variations in significant teaching processes were found related to student perceptions which were found related to student evaluations (satisfaction) and also to student achievement. The best known investigations of student evaluations in relation to teaching processes are studies of the " Dr. Fox effect" or "educational seduction" (Abrami, Leventhal, & Perry, 1982). The first of the Dr. Fox studies was conducted by Naftulin, Ware and Donnelly (1973) who had a professional actor assume the name, Dr. Myron L. Fox, and pose as a visiting professor. The actor proceeded to lecture to three groups of experienced educators in a manner that was highly expressive but low in substantive content. The experienced educators were reported as having expressed satisfaction with the amount they had learned from the lecture, thus suggesting that expressive behavior was at least as important as content coverage in determining audience reactions, and casting doubt on the validity of audience ratings as criteria of lecture effective­ ness. Kaplan (1974) and Frey (1978) criticized the study on several grounds including Jack of both internal and external validity. There followed a series of experiments in which lecturer expres­ siveness, content coverage, incentives to learn, and other vari­ ables were manipulated, and student ratings and learning were treated as dependent or outcome variables (Abrami, Dickens, Perry, & Leventhal, 1980; Abrami, Perry, & Leventhal, 1982; Marsh & Ware, 1982; Meier, 1977; Meier & Feldhusen, 1979; Perry, Abrami, & Leventhal, 1979; Perry, Abrami, Leventhal, & Check, 1979; Ramagli, 1979; Ramagli & Greenwood, 1980; Ware & Wiiliams, 1975; Williams & Ware, 1976; Williams & Ware, 1977). Attempts to review and interpret such studies have been made by several authors (Frey, 1978, 1979a, 1979b; Leventhal, 1979, 1980; Leventhal, Abrami, & Perry, 1979; Marsh, 1985; McKeachie, 1979; Ware & Williams, 1 979, 1980), culminating in a meta-analysis by Abrami, Leventhal, & Perry (1982). The last authors drew attention to disagreements among previous reviewers such that some concluded on the basis of the Dr. Fox studies that student ratings are invalid indicators of teacher effectiveness in enhancing student learning (e.g., Ware & Williams, 1975), while others concluded that expressiveness is an important aspect of teacher behavior which can be used to enhance student learning (e.g., Frey, 1978). Ware collaborated with Marsh (Marsh & Ware, 1982) in a reanalysis of the data from the Ware and Williams studies. A factor analysis of the original rating instrument indicated five factors or independent dimensions of the rating scores, and that instructor expressiveness seemed to affect only one factor, Instructor Enthusiasm, while content coverage seemed to affect

770

MICHAEL J. DUNKIN

the factor of Instructor Knowledge. Thus, it appeared that evaluations were, in fact, responsive specifically to the teaching variables manipulated, and that expressiveness was not disguis­ ing variations in content coverage, as the original results suggested. Abrami, Leventhal, and Perry (1982) investigated main and interaction effects involving expressiveness and content cov­ erage in relation to student ratings and achievement as criterion variables. Their results when student ratings were the criterion were as follows: In all [12] studies, the effect of expressiveness on summary or global ratings was significant and reasonably large (unweighted mean w 2 = .285) . . . The effect of content on ratings was inconsistent and generally much smaller (unweighted mean w2 = .046, weighted mean w2 = .038) than the expressiveness effect, about one sixth the size or less. Smaller still was the interaction effect of expressiveness and content on ratings (unweighted mean w2 = .016) .. . . (p. 452) In a study not conducted within the Dr. Fox tradition, and not included in the Abrami, Leventhal, and Perry (1982) meta­ analysis, Andersen and Withrow (1981) investigated the effects of variations in nonverbal expressiveness upon affective, beha­ vioral, and cognitive learnings of 299 undergraduate students in a lower-division speech communication course at a large east­ ern U.S. university. Videotaped lectures were used to present high, medium, and low levels of nonverbal expressiveness. Significant effects were found on affective criteria with varia­ tions in nonverbal expressiveness accounting for 9%, 3%, and 6%, respectively, of the variance in students' ratings of lecturer sociability, students' attitudes toward the lecture, and students' attitudes toward the videotape. No significant or sizeable effects were found on the likelihood of students' engaging in the suggested communication strategies or of attending another lecture. Another study of the validity of student evaluations as judged by the extent to which they relate to behavior actually engaged in by teachers in normal classroom conditions was conducted by Cranton and Hillgartner (1981), who explored this matter with 28 professors from several disciplines, using Shulman's (1975) modification of the FIAC system to observe teaching behavior. The professors were all involved in teacher improve­ ment activities. The authors found that three general areas of teacher behavior were more closely related than others to students' ratings. They summarized their results as follows: (1) When instructors spent time structuring classes and explaining relationships, students gave higher ratings on logical organization items. (2) When professors praised student behavior, asked ques­ tions and clarified or elaborated on student responses, ratings on the effectiveness of discussion leading were higher. (3) When instruc­ tor time was spent in discussions, praising student behavior, and silence (waiting for answers), students tended to rate the classroom atmosphere as being one which encourages learning. (p. 73) This type of information is not only evidence regarding an important aspect of the credibility of student ratings, but is also potentially useful to teachers wanting to know which aspects of their behavior are producing high or low student ratings.

Every one of the preceding studies found evidence that student evaluations were responsive to variations in teaching processes although the relationships were generally small to moderate in size and varied depending upon which teaching process was manipulated or observed and which attribute of the teaching was rated. Variations in the relationships give rise to the possibility that some of the teaching processes to which students are sensitive are unimportant while some to which they are insensitive are important. The Dr. Fox studies are also relevant to this issue which became one of the focal points of the meta-analysis by Abrami, Levanthal, and Perry (1982). As mentioned previously, those authors reported that student evaluations were much more responsive to variations in expres­ siveness than to variations in content coverage. The question then raised was: How important are expressiveness and content coverage for student achievement? When student achievement was the criterion, Abrami, Leven­ thal, and Perry (1982) obtained results which they reported as follows: In contrast to the substantial impact of expressiveness on ratings, the effect of expressiveness on achievement was insignificant in 5 of 10 analyses, on average accounting for only 4 percent of the achievement variance. The effect of content on achievement was significant in all studies except [one] .. . . On average, content accounted for a sizeable amount of achievement variance (un­ weighted mean w2 = .158). The Expressiveness x Content interac­ tion reached significance in only one study. ... where it accounted for less than 2 percent of the achievement variance. (p. 454) Andersen and Withrow (1981) also investigated the effects of variations in expressiveness upon student achievement in terms of two tests of cognitive learnings and found no significant effects. In all, the results of this study were consistent with the results of the Abrami, Leventhal and Perry (1982) meta­ analyses. Expressiveness was again found to be more effective in relation to students' rating of instruction than in relation to cognitive achievement. After concluding that expressiveness had been shown to have a sizeable impact on student ratings of instruction but not on achievement, while the reverse applied for content coverage, Abrami, Leventhal, and Perry (1982) discussed limitations arising from the ways in which those two process variables had been presented in the Dr. Fox experiments. In particular, it was pointed out that the levels of expressiveness and content coverage as manipulated could not be held to be representative of the variations in expressiveness and content coverage that would be found in field studies. Therefore, it was argued, findings of the differential effects of the experimental treatments involving those variables cannot be assumed to represent. the findings of effects that might be found in the field. The Dr. Fox studies were incapable, according to the authors, of telling much about the validity of student ratings as predictors of student achievement. The authors went on to advocate field research on the variations in occurrence of expressiveness, content coverage, and other characteristics. The latter type of research would require the development of observational instruments to permit the identification and measurement of expressiveness and content coverage as they occur in the field. It would be important that such instruments

RESEARCH ON TEACHING IN HIGHER EDUCATION

be developed on the basis of the definitions and examples of such concepts of teaching behavior as have been provided in the experimental studies. Definitions of expressiveness presented in the studies mentioned in this chapter contain qualities that vary considerably in the degree of inference that would be required by an observer trying to detect their presence in a lecture. Ware's definition (1974) was the only one in the Dr. Fox experiments that included the terms dynamism, emotional ap­ peal, seduction, and stimulation. Meier and Feldhusen (1979) alone referred to looking at notes, and smiles. Perry, Abrami, Leventhal, and Check (1979) made the only reference to eye contact. The most agreed upon ingredients of expressiveness were charisma, enthusiasm, friendliness, vocal inflection, hu­ mor, and physical movement, all of which are highly inferential concepts which probably subsume the more specific behaviors such as smiling, looking at notes, and making eye contact. Detailed analyses of the specific behaviors performed in differ­ ent sets of videotapes would require the development of an observational system which could then prove useful in research on expressiveness in naturalistic field settings. Content coverage, or "content density" as it was called by Meier and Feldhusen (1979, p. 341),was more specifically defined in the studies reviewed. In the Ware and Williams (1975) tapes, the high-content treatment contained 26 different teaching points. The medium-content and low-content treat­ ments contained 14 and 4 points, respectively. Length of treatment was kept uniform by the insertion of unrelated examples, meaningless utterances, and circular discussion. Si­ milar procedures were employed in the black-and-white and color videotapes prepared by Perry, Abrami, and Leventhal (1979). Variations in content, however, consist not only in the number of propositions, but also in levels of abstraction, degree of organization, and so on. It would seem important that these aspects also be investigated. The Ware and Williams (1975) videotaped lectures were on "The Biochemistry of Learning" and were prepared for an introductory course in psychology. The black-and-white video­ tapes used by Perry, Abrami, and Leventhal (1979) were on impression formation and were also directed toward students in an introductory course in psychology. The color videotapes used in the second experiment by Perry, Abrami, Leventhal, and Check (1979) were on sex roles and stereotyping and they, too, were prepared for use in an introductory course in psychol­ ogy. Clearly, before the relative influence of content coverage upon student outcomes is to be known, there is need for research in other disciplines than psychology. There still remains the matter of the effects of extraneous variables upon student evaluations. Commonly expressed views are that students do not have the mastery of subject matter required to evaluate the content of instruction, that students are biased by extraneous factors such as the sex and popularity of the instructor, and that students are more negative toward more demanding courses. Marsh (1985) found weak or mixed results for claims that course level, class size, nature of disci­ pline, instructor sex and rank, and student sex and personality influenced student ratings. He did find tendencies for classes receiving or expecting higher grades, having higher interest in the subject, and enrolled in elective rather than compulsory courses to give more positive ratings, but found ambiguities in

771

the interpretations that might be made of those results. For example, the association between expected or actual grade and ratings might have indicated either grading leniency or superior learning. Interestingly enough, evidence did confirm an associa­ tion between the workload or difficulty of a course and student ratings. However, it was a positive relationship, and so contra­ dicted the common expectation. Abrami, Leventhal, and Perry (1982) included in their meta­ analysis tests to see whether relationships between expressive­ ness and/or content coverage and student evaluations and achievement were affected by extraneous factors. They found that the effects of the following were minimal: monetary or experimental credit incentives, second exposure to the lecturer, opportunity for study, stated purposes of the evaluation, in­ structor reputation, student personality characteristics, and instructor grading standards. Other reviewers of the literature on student evaluations include Costin, Greenough, and Menges, (1971); Kulik and McKeachie (1975); McKeachie (1979); and Menges (1979); all of whom concluded that student evaluations were not unduly affected by extraneous elements. Finally, there is the question of the relationship between student evaluations of instruction and student achievement. Cohen's meta-analysis (1981) of research on relationships be­ tween student ratings of instruction and student achievement revealed mean correlations between student achievement and overall ratings of courses and teachers of .47 and .43, respecti­ vely. Although the correlation with student achievement tended to be higher if student ratings were obtained after students knew their grades in the course, Cohen concluded that "student ratings of instruction are a valid index of instructional effective­ ness" (p. 305). On balance, it seems from the research reviewed here that students are able to perceive variations in teaching processes, and that the latter, more than extraneous elements, affect student evaluations of teaching. There is also evidence that teaching processes vary in the size of their effects upon student evaluations. The possibility that the latter are unduly affected by unimportant teaching processes has not been adequately researched. Research on teaching in higher education provides little indication of which are important and unimportant teach­ ing processes. Finally, student evaluations have been found generally to have moderate effects upon student achievement. It now remains to be seen whether or not student evaluations can serve as a basis for the improvement of teaching. USEFULNESS OF STUDENT EVALUATIONS IN IMPROVING TEACHING

Student evaluations of instruction are useful to the extent that they assist in improving teaching, lead to better informed decisions about the careers of personnel, and help students to make suitable choices of courses and teachers. A fourth pur­ pose, to serve as a variable in research on teaching, is not so often recognized and is usually neglected as a criterion for evaluating the usefulness of student ratings. The earlier sections of this chapter contain references to studies in which student ratings were found useful for this fourth purpose. If feedback on student evaluations leads to the improvement of teaching, then presumably it does so by changing the

772

MICHAEL J. DUNKIN

behavior of the teachers concerned in ways that are consistent with criteria of teaching effectiveness. Thus, the question of the potential of student evaluations to improve teaching is analyz­ able into two separate questions: Do student evaluations bring about change in teaching processes? and Are the changes consistent with criteria of teaching effectiveness? To answer these questions, two types of studies would be required. First, there would be presage-process studies, in which feedback on student evaluations would be used as independent variables and teaching processes would be the dependent variables. Second, there would be process-product studies in which the processes shown to be changed by feedback on student evalua­ tions would be used as independent variables and criteria of teaching effectiveness, such as subsequent student evaluations, and student achievement, would be dependent variables. Re­ search on the usefulness of student evaluations has not been of those types, however, but has combined the two questions by omitting process variables in what are essentially presage­ product studies. That is, the research has been concerned mainly with providing teachers with feedback on student evaluations at one point in time, and then exploring their relationships with second measures of students' evaluations, or student achievement measures, obtained at some later point, such as the end of the course. The reviews by Rotem and Glasman (1979); Abrami, Leventhal, and Perry ( 1 979); Cohen ( 1980); and Levinson-Rose and Menges ( 198 1 ) were almost exclusively conducted upon research of the presage-product type. Rotem and Glasman (1 979) concluded " that feedback from student ratings . . . does not seem to be effective for the purpose of improving the performance of university teachers" (p. 507). Nevertheless, those reviewers did not recommend dispensing with feedback from students because they thought it might be rendered more effective in the future. Abrami, Leventhal, and Perry ( 1979) concluded that feedback from student ratings helped some instructors to improve their subsequent student ratings, but the effect was not reliable and was of unknown size. Cohen's meta-analysis (1980) led him to conclude that feedback from student ratings had made a " modest but significant contribution to the improvement of college teaching" (p. 336). Students' end-of-course ratings of teachers' skills and of pro­ vision of feedback to students were found to increase modera­ tely (average effect sizes .47 and .40, respectively) after teachers were informed of the results of midterm students' ratings. Students' overall ratings also increased moderately (effect size .38). All three increases were statistically significant. Students whose instructors received midterm feedback also tended to rate their own learning higher and to express more favorable attitudes toward the subject matter at the end of the course. Effects of midterm feedback were found to be much larger if it was augmented with consultation and the like. Levinson-Rose and Menges ( 1981) reached similar conclu­ sions about the importance of consultation in association with feedback on student evaluations. They also found that such feedback is likely to be more effective for teachers whose students' ratings are less favorable than their self-ratings. Such teachers were seen to be especially suitable for the investment of consultants' efforts. None of the studies reviewed by Levinson-Rose and Menges

(1981) provided direct evidence of changes in teaching in response to feedback from students, and only five used self­ reports concerning changes from teachers. Without exploration of intermediate steps in a theoretically derived process by which feedback from students might produce change that is improve­ ment, research on the utility of student evaluations for that purpose is incomplete. Research on the processes by which feedback results in changes in teaching which in turn affect subsequent student ratings and achievement is needed to en­ hance understanding of teaching and learning associations in higher education. In all, the evidence concerning the credibility of student evaluations of teaching has more often been based on associa­ tions with actual teaching processes than has the evidence of the usefulness of student evaluations in improving instruction. Indeed, there is very little research evidence on the usefulness of student evaluations in improving any of the processes thought to benefit from them. Coleman and McKeachie (198 1 ) found some evidence that student evaluations affect students' choices of courses, but whether they improve those choices and whether they affect and improve decisions made about teaching person­ nel are largely unresearched issues in higher education. Much more research into the classroom processes to which students respond in forming their evaluations is needed. Until, through such research, much more is known about student evaluations, their use as bases for decisions in education needs to be cautious. Some of the issues involving student evaluations are especially complex and, it seems, cannot be resolved solely on the basis of empirical research. For example, the nonempirical issues of whether students should control the curricula of higher education or appointments of faculty are probably independent of research on the validity of student evaluations and their usefulness in improving teaching.

Other Attempts to Improve Instruction Levinson-Rose and Menges (1981) reviewed several other ap­ proaches to the improvement of teaching in higher education. They were: grants for faculty development projects; workshops and seminars; microteaching and minicourses; and concept­ based training using protocols of teaching incidents. Grants to support development projects, commonly known as "small grants," were found in 58% of the 756 institutions surveyed by Centra ( 1978). These grants are usually awarded competitively, paralleling grants awarded in support of re­ search, and are seen as incentives promoting interest in teach­ ing. Apparently they are different from research grants at the evaluation and reporting stages because Levinson-Rose and Menges could find only one report of a study of the effectiveness of such grant schemes in improving teaching. Presumably, many reports on such projects do not include data on improv­ ing teaching. The evidence of that one study (Kozma, 1 978) was that even quite small grants can encourage increase in the use of instructional innovations by university teachers. Studies of the effectiveness of workshops and seminars in producing improvements in teaching have usually used gra­ duate teaching assistants as subjects and- have been directed

RESEARCH ON TEACHING IN HIGHER EDUCATION

toward either attitude change or skill development (Levinson­ Rose & Menges, 1981). Both Goldman (1978) and Graham (1971) found evidence consistent with the hypothesis that workshops produced desired affective changes in teachers. Where the focus of workshops and seminars was on skill development, Levinson-Rose and Menges (1981) concluded that 24 of 28 studies produced evidence supporting their effectiveness. Criteria of effectiveness employed in the studies included student ratings, ratings by trained observers, and student learning. The likelihood that evidence supported the intervention appeared unrelated to the type of criterion used. Although only one of the studies (Bray & Howard, 1980) had the design characteristics required for Levinson-Rose and Menges to have high confidence in the results, the research provides grounds for optimism that skill-oriented seminars not only influence teaching behavior in desired directions, but also through that influence, increase student learning and affect students' opinions of the teaching. The reviewers, in reaching similar conclusions, raised the issue of compulsory/voluntary participation in faculty development workshops and its impli­ cations for generalizability of research findings. Results of research conducted on interventions in which subjects are obliged to participate may not apply when participation in such .activities is voluntary. In addition to the research on workshops and seminars reviewed by Levinson-Rose and Menges (1981), there has been research on the effectiveness of whole courses or programs of learning activities, often organized in the workshop format. Among the best known of these is the research done by a team at Simon Fraser University in Vancouver (Martin, Marx, Hasell, & Ellis, 1978; Marx, Martin, Ellis, & Hasell, 1978; Marx, Ellis, & Martin, 1979). The Teaching Assistant Training Pro­ gram consisted of 15 hours of instruction for 3 hours on each of 5 evenings. Its objectives were to teach participants five major areas of educational content, skills, and attitudes, as follows: (a) the incorporation of principles of learning in the design of instruction; (b) the development and use of behavioral objec­ tives in instruction; (c) the measurement of learning; (d) effec­ tive discussion group or laboratory sessions; (e) ethical and moral dilemmas in teaching. Four groups of teaching assistants were involved in the program, while a fifth group became the control group. Inter­ view and questionnaire data were obtained from the partici­ pants as well as their students' perceptions, attitudes, ratings, and grades. The evaluation prompted the following conclusions among others, by the authors: (a) participants were perceived by their students to be better tutorial leaders and more willing to improve their teaching; (b) in terms of student achievement, participation for the 15 hours of the program was at least equivalent in instructional value to three semesters' teaching experience without training. Those who are responsible for organizing and conducting such training programs will find solace in the suggestion that their efforts are cost effective in comparison with the "school of hard knocks." Levinson-Rose and Menges (1981) located only three studies of microteaching in higher education (Johnson, 1977; Perlberg, Peri, Wienreb, Nitzan & Shimron, 1972; Perry, Leventhal, & Abrami, 1979). The first two found evidence that participation in microteaching programs produced desired changes in be-

773

havior, while the third found that instructors initially rated as highly effective had students who rated them higher and who achieved higher after the instructors participated in a modified microteaching program. No changes were observed for instruc­ tors initially rated as low-effective. More recently, Sharp (1981) reported that viewing videotaped models of lecturing skills increased short-term performance of those skills in a micro­ teaching context, and Brown (1980) found that teachers who participated in a microteaching program on skills in explaining viewed the experience favorably and were rated favorably by trained observers on changes in the quality of their explana­ tions. One approach to the improvement of teaching that Levin­ son-Rose and Menges (1981) regarded as particularly relevant was training in the development of concepts of teaching and learning appropriate to higher education. Concept-based train­ ing is not so much concerned with enhancing repertoires of teaching behavior as is microteaching. Rather, it aims to enhance teachers' powers of thought concerning teaching, thus increasing their ability to plan, hypothesize about, make deci­ sions about, and monitor their own teaching. The attempt by Taylor-Way (1981) at Cornell University to use the techniques of Interpersonal Process Recall (Kagan, 1973, 1975) is an example of this approach. In it teachers used microteaching contexts to study covert aspects of their teaching, with empha­ sis upon expression of thoughts and feelings about teaching, and the development of concepts and principles. Taylor-Way reported favorably upon the conceptual development and refinement observed in a case-study approach to the evaluation of the program. The study by Aubrecht (1978) also involved concept develop­ ment and included a case study approach, but it was based on a formal first stage in which teachers were introduced to Bellack's (Bellack et al., 1966) categories for observing classroom behav­ ior. Aubrecht found that teachers who were provided with feedback in the form of trained observer codings of their behavior and who received unconditional support from a consultant for changes suggested by themselves subsequently behaved differently from a control group in many of the pedagogical moves and substantive-logical operations per­ formed. In all, those responsible for faculty development in universit­ ies and colleges might derive encouragement from the research just reviewed. So far, there is no indication of a one best way to improve teaching in higher education. On the contrary, the small amount of research that has been conducted suggests that there is a variety of effective ways and, therefore, that combined approaches have a good chance of capitalizing on the strengths of each. Research to come might explore such combinations. In conclusion, it might be argued that research is needed that inquires into the attributes of teachers that affect their potential to benefit from teaching improvement efforts. Beliefs and attitudes tuward teaching and teacher education should be profitable avenues for investigation here if research such as that by Brown and Daines (1981) and Germ (1982) is any guide. Furthermore, Menges (1980) has suggested a theoretical model, a "theory of reasoned action" (p. 5) capable of accommodating such variables along with others such as feedback and teaching behavior.

774

MICHAEL J. DUNKIN

Implications for Future Research and Practice Research on teaching in higher education as reviewed in this chapter has revealed the following strengths and weaknesses: 1 . The innovatory methods of teaching discussed have in most cases been found slightly superior to conventional teaching methods, and in one case, the Keller Plan, considerably superior. Colleges and universities can be reasonably confi­ dent that careful implementation of these innovations can proceed with little risk of harm to students and with some promise of benefit. Cost-effectiveness information is how­ ever, lacking, except in the few cases where savings in student time have been found. 2. Recently applied techniques for synthesizing research find­ ings have proved invaluable in the preparation of this chapter. Not only have they contributed greatly to know­ ledge about cumulative findings of research; they have also led to the identification of gaps in research and to the development of category systems that have proved useful in describing and grouping studies. Those category systems will also prove useful in designing future research. Further­ more, meta-analysis has made possible the exploration of design and contextual influences upon research findings. In doing this, meta-analyses have had to rely on fortuitously occurring variations across studies. It is hoped that in the future such variations will more often be planned. 3. There has been an increase in concern with the nature of teaching behavior as revealed through the application of observational techniques in naturalistic settings. This ap­ proach promises to enhance knowledge and understanding of teaching in higher education contexts, whether innova­ tive or conventional. So far, concepts of behavior under­ lying those observations have been highly derivative of concepts used for many years at lower levels of education. Concepts that are especially applicable in higher education need to be developed and applied. There has been an almost total neglect of student academic learning behavior, both in and out of class sessions, in this research. Research at lower levels of education suggests that this is a profitable area of investigation. 4. There is now some evidence of the credibility of student evaluations of teaching in higher education in terms of their sensitivity to known variations in teaching processes. Much more research is needed to demonstrate ways in which student evaluations can be put to effective use in improving teaching. In particular, research on the effects of feedback from student ratings upon change in teaching processes is needed. 5. There is some cause for confidence that techniques for inducing desired changes in teaching behavior are effective. However, there has been little research on teaching skills in higher education, and so the efforts of faculty development agents are much in need of support from that quarter. The roles played by teachers' beliefs, values, and attitudes toward teaching and learning need to be explored in the context of teaching improvement efforts. 6. The dominant paradigm underlying research on teaching in higher education has been the process-product one. In

most cases, however, the process part has been assumed on the basis of prescriptive definitions, or rated by untrained observers, rather than documented through careful obser­ vation. The process-product paradigm is limiting in the types of knowledge about teaching and learning that can be generated within it. Research on teaching in higher educa­ tion might do well to explore alternative paradigms as presented in the chapter by Shulman in this volume. In particular, there would seem to be value in a paradigm of presage, context, process, and product variables elaborated so as to accommodate teacher thinking and valuing as well as student academic learning processes. Perhaps the most surprising aspect of research on teaching in higher education is that its contribution to the improvement of teaching has not been evaluated. Studies of dissemination and diffusion of research on teaching at this level are almost nonexistent and little is known of the extent to which faculty development agencies have been able to use such research in designing and implementing their programs. It is to be expected that the next decade will see an increase in concern with the usefulness of research not only in enhancing knowledge and understanding of teaching, but also in improving it.

REFERENCES Abrami, P. C., Dickens, W. J., Perry, R. P., & Leventhal, L. ( 1980). Do teacher standards for assigning grades affect student evaluations of instruction? Journal of Educational Psychology, 72, 107- 1 18. Abrami, P. C., Leventhal, L., & Perry, R. P. ( 1979). Can feedback from student ratings help improve college teaching? Paper presented at the meeting of the Fifth International Conference on Improving University Teaching, London. Abrami, P. C., Leventhal, L., & Perry, R. P. (1982). Educational seduction. Review of Educational Research, 52, 446-464. Abrami, P. C., Perry, R. P., & Leventhal, L. (1982). The relationship between student personality characteristics, teacher ratings, and student achievement. Journal of Educational Psychology, 74, 1 1 1-125. Amidon, E. J., & Flanders, N. (1963). The role of the teacher in the classroom. Minneapolis, MN: Amidon and Associates. Andersen, J. F., & Withrow, J. G. (1981). The impact of lecturer nonverbal expressiveness on improving mediated instruction. Com­ munication Education, 30, 342-353. Andrews, J. D. (1980). The verbal structure of teacher questions: Its impact on class discussion. POD Quarterly, 2, 1 30-163. Aubrecht, J. D. (1978). Teacher effectiveness: Self-determined change. American Journal of Physics, 46, 324-328. Barnes, C. P. (1980, April). Questioning: The untapped resource. Paper presented at the annual meeting of the American Educational Research Association, Boston. Baumgart, N. L. (1972). A study of verbal interaction in university tutorials. Unpublished doctoral dissertation, Macquarie University, Sydney, Australia. Baumgart, N. L. (1976). Verbal interaction in university tutorials. Higher Education, 5, 301 -317. Bellack, A. A., Kliebard, H. M., Hyman, R. T., & Smith, F. L. Jr. ( 1966). The language of the classroom. New York: Teachers College Press. Benne, K. D., Sheats, P. (1948). Functional roles of group members. Journal of Social Issues, 4, 42-47. Bloom, B. S. (1968). Learning for mastery. Evaluation Comment, 1(2). Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (Eds.). (1956). Taxonomy of educational objectives: The classification of education goals, handbook I: The cognitive domain.

New York: David McKay.

RESEARCH ON TEACHING IN HIGHER EDUCATION

Bray, J. H., & Howard, G. S. (1980). Methodological considerations in the evaluation of a teacher-training program. Journal ofEducational Psychology, 72, 62-70. Brown, G. A. (1980). Explaining: Studies from the higher education context. Final Report to the Social Science Research Council. University of Nottingham, England. Brown, G. A., Bakhtar, M., & Youngman, M. B. (1982). Toward a typology of lecturing styles. University of Nottingham, England. Brown, G. A., & Daines, J. M. (1981). Can explaining be learnt? Some lecturers' views. Higher Education, JO, 573-580. Centra, J. A. (1978). Faculty development in higher education. Teachers College Record, 80, 188-201. Chickering, A. (1972). Undergraduate academic experience. Journal of Educational Psychology, 63, 134- 143. Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press. Cohen, P. A. (1980). Effectiveness of student-rating feedback for improving college instruction: A meta-analysis of findings. Research in Higher Education, 13, 321-341. Cohen, P. A. (1981). Student ratings of instruction and student achieve­ ment: A meta-analysis of multisection validity studies. Review of Educational Research, 51, 281-309. Cohen, P. A., Ebeling, B. J., & Kulik, J. A. (1981). A meta-analysis of outcome studies of visual-based instruction. Educational Communi­ cation and Technology, 29, 26-36. Coleman, J., & McKeachie, W. J. (198 1). Effects of instructor course evaluations on student course selections. Journal of Educational Psychology, 73, 224-226. Cooper, C. R. (1981a). Different ways of being a teacher: An ethno­ graphic study of a college instructor's academic and social roles in the classroom. Journal of Classroom Interaction, 16, 27-37. Cooper, C. R. (198 1b). An ethnographic study of teaching style. Paper presented at the Midwest-Regional Conference on Qualitative Research in Education, Kent State University, Kent, Ohio. Cooper, C. R., Henry, R., Korzenny, S., & Yelon, S. (undated). Procedures for studying effective college instruction. Unpublished manuscript, Michigan State University, Leaming and Evaluation Service. Cooper, C. R., Orban, D., Henry, R., & Townsend, J. (undated). Teaching and storytelling : An ethnographic study of the instructional process in a college classroom. Unpublished manuscript, Michigan

State University, Learning and Evaluation Service. Costin, F., Greenough, W. T., & Menges, R. J. (1971). Student ratings of college teaching: Reliability, validity, and usefulness. Review of Educational Research, 41, 5 1 1 -535. Cranton, P. A., & Hillgartner, W. (198 1). The relationships between student ratings and instructor behavior: Implications for improving teaching. Canadian Journal of Higher Education, 11, 73-81. Daniel, J. S., Stroud, M. A., & Thompson, J. R. (Eds.). (1982). Learning at a distance: A world perspective. Edmonton: Athabasca University/ International Council for Correspondence Education. Denham, M. A., & Land, M. L. (198 1). Research brief- Effect of teacher verbal fluency and clarity on student achievement. Technical Teachers Journal of Education, 8, 227-229. Donald, J. G. (1980). Structures of knowledge and implications for teaching. Vancouver: Centre for the Improvement of Teaching, University of British Columbia. Dunkin, M. J., & Biddle, B. J. (1974). The study of teaching. New York: Holt, Rinehart & Winston. Foster, P. J. (1981). Clinical discussion groups: Verbal participation and outcomes. Journal of Medical Education, 56, 831-838. Frey, P. W. (1978). A two-dimensional analysis of student ratings of instruction. Research in Higher Education, 9, 60-91. Frey, P. W. (1979a). The Dr. Fox effect and its implications. Instruc­ tional Evaluation, 3, 1-5. Frey, P. W. (1979b). Dr. Fox revisited. Instructional Evaluation, 4, 6- 12. Gage, N. L. (Ed.). (1983). Handbook of research on teaching. Chicago: Rand McNally. Gage, N. L. (1978). The scientific "basis" of the art of teaching. New York: Teachers College Press. Gallagher, J. J., & Aschner, M. J. (1963). A preliminary report on

775

analyses of classroom interaction. Merrill-Palmer Quarterly of Behavior and Development, 9, 183-195. Genn, J. M. (1982, May). The receptivity of Australian university teachers towards academic staff development programs focusing on the teaching role. Paper presented at the annual meeting of the Higher Education Research and Development Society of Austral­ asia, Sydney, Australia. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8. Goldman, J. A. (1978). Effects of a faculty development workshop upon self-actualization. Education, 98, 254-258. Goldschmid, B., & Goldschmid, M. L. (1976). Peer teaching in higher education: A review. Higher Education, 5, 9-33. Graham, M. W. (1971). Development of an inservice program for geology teaching assistants to reduce role conflict and to improve skills. Unpublished doctoral dissertation, Ohio State University, Columbus, OH. Guilford, J. P. (1956). The structure of the intellect. Psychological Bulletin, 53, 267-293. Hegarty, E. H. (1978). Levels of scientific enquiry in university science laboratory classes: Implications for curriculum deliberations. Re­ search in Science Education, 8, 45-57. Hegarty, E. H. (1979). The role of laboratory work in teaching microbio­ logy at university level. Unpublished doctoral dissertation, Univer­ sity of New South Wales, Sydney, Australia. Hiller, J. H. (1971). Verbal response indicators of conceptual vagueness. America Educational Research Journal, 8, 1 5 1-161. Hiller, J. H., Fisher, G. A., & Kaess, W. (1969). A computer investiga­ tion of verbal characteristics of effective classroom lecturing. Ameri­ can Educational Research Journal, 6, 661-675. Hines, C. V., Cruickshank, D. R., & Kennedy, J. J. (1982, March). Measures ofteacher clarity and their relationships to student achieve­ ment and satisfaction. Paper presented at the annual meeting of the

American Educational Research Association, New York. Holmberg, B. (1982). Recent research into distance education. Hagen, F. R. Germany: Fern Universitat. Johnson, G. R. (1977). Enhancing community/junior college professors' questioning strategies and interaction with students. Community/ Junior College Research Quarterly, 2, 47-54. Johnson, R. H., & Lobb, M. D. (1959). Jefferson County, Colorado, completes three-year study of staffing, changing class size, program­ ming, and scheduling. National Association of Secondary School Principals Bulletin, 43, 57-78. Kagan, N. (1973). Can technology help us toward reliability in influenc­ ing human interaction? Educational Technology, 13, 44-51. Kagan, N. (1975). lnterpersonal Process Recall- a method for influenc­ ing human interaction. Michigan State University, Department of Counseling, Personal Services and Educational Psychology, East Lansing, MI. Kaplan, R. M. (1974). Reflections on the Dr. Fox paradigm. Journal of Medical Education, 49, 310-312. Keller, F. S. (1968). "Good-bye teacher . . . " Journal ofApplied Behavior Analysis, I, 79-89. Klein, S. S. (1971). Student influence on teacher behavior. American Educational Research Journal, 8, 403-421. Kounin, J. S. (1 970). Discipline and group management in classrooms. New York: Holt, Rinehart & Winston. Kozma, R. B. (1978). Faculty development and the adoption and diffusion of classroom innovations. Journal ofHigher Education, 49, 438-449. Kulik, J. A., Cohen, P. A., & Ebeling, B. J. (1980). Effectiveness of programmed instruction in higher education: A meta-analysis of findings. Educational Evaluation and Policy Analysis, 2, 5 1 -64. Kulik, J. A., Jaksa, P., & Kulik, C-L. C. (1978). Research on component features of Keller's Personalized System of Instruction. Journal of Personalized Instruction, 3, 2- 14. Kulik, J. A., & Kulik, C-L. C. (1979). College teaching. In P. L. Peterson & H. J. Walberg (Eds.), Research on teaching: Concepts, findings, and implications. (pp. 70-93) Berkeley, CA: McCutcheon. Kulik, J. A., Kulik, C-L. C., & Cohen, P. A. (1979a). A meta-analysis of outcome studies of Keller's Personalized System of Instruction. American Psychologist, 34, 307-318.

776

MICHAEL J. DUNKIN

Kulik, J. A., Kulik, C-L. C., & Cohen, P. A. ( 1979b). Research on audio­ tutorial instruction: A meta-analysis of comparative studies. Re­ search in Higher Education, 1 1, 321-341. Kulik, J. A., Kulik, C-L, C., & Cohen, P. A. (1980). Effectiveness of computer-based college teaching: A meta-analysis of findings. Re­ view of Educational Research, 50, 525-544. Kulik, J. A., & McKeachie, W. J. (1975). The evaluation of teachers in higher education. In F. N. Kerlinger (Ed.), Review of Research in Education, Vol. 3. Itasca, IL: F. E. Peacock. Kyle, W. C., Jr., Penick, J. E., & Shymansky, J. A. (1980). Assessing and analyzing behavior strategies of instructors in college science labor­ atories. Journal of Research in Science Teaching, 1 7, 1 3 1- 1 37. Land, M. L. (1979). Low-inference variables of teacher clarity: Effects on student concept learning. Journal of Educational Psychology, 71, 795-799. Land, M. L. ( 1980). Teacher clarity and cognitive level of questions: Effects on learning. Journal ofExperimental Education, 49, 48-5 1 . Land, M. L . (1981). Combined effects o f two teacher clarity variables on student achievement. Journal of Experimental Education, 50, 1 4- 17. Land, M. L. (1985). Vagueness and clarity in the classroom. In T. Husen & T. N. Postlethwaite (Eds.), International encyclopedia of educa­ tion : Research and studies. Oxford: Pergamon Press. Land, M. L., & Smith, L. R. (1979a). The effect of low inference teacher clarity inhibitors on student achievement. Journal of Teacher Educa­ tion, 30, 55-57. Land, M. L., & Smith, L. R. (1 979b). Effects of a teacher clarity variable on student achievement. Journal of Educational Research, 72, 196-197. Land, M. L., & Smith, L. R. (1981). College student ratings and teacher behavior: An experimental study. Journal ofSocial Studies Research, 5, 19-22. Leventhal, L. ( 1979). The Doctor Fox effect: An alternative interpreta­ tion. Instructional Evaluation, 4, 1 -6. Leventhal, L. (1980). Alternative interpretation of the Dr. Fox effect: Reply to Frey. Instructional Evaluation, 4, 13- 14. Leventhal, L., Abrami, P. C., & Perry, R. P. (1979, July). Educational seduction and the validity of student ratings - An alternative interpre­ tation. Paper presented at the Fifth International Conference on

Improving University Teaching, London, England. Levinson-Rose, J., & Menges, R. J. (1981). Improving college teaching: A critical review of research. Review of Educational Research, 51, 403-434. McKeachie, W. J. ( 1963). Research on teaching at the college and university level. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago: Rand McNally. McKeachie, W. J. ( 1979). Student ratings of faculty: A reprise. Academe, 65, 384-397. Marsh, H. W. (1985). Students as evaluators of teaching. In T. Husen & T. N. Postlethwaite (Eds.), International encyclopedia of education: Research and studies. Oxford: Pergamon Press. Marsh, H. W., & Ware, J. E. Jr. (1982). Effects of expressiveness, content coverage, and incentive on multidimensional student rating scales: New interpretations of the Dr. Fox effect. Journal of Educational Psychology, 74, 126-1 34. Martin, J., Marx, R. W., Hasell, J., & Ellis, J. F. (1978). Improving the instructional effectiveness of university teaching assistants: Report II. Canadian Journal of Education, 3, 1 3-26. Marx, R. W., Ellis, J. F., & Martin, J. (1979). The training of teaching assistants in Canadian universities: A survey and case study. The Canadian Journal of Higher Education, 9, 56-63. Marx, R. W., Martin, J., Ellis, J. F., & Hasell, J. (1978). Improving the instructional effectiveness of university teaching assistants: Report I. Canadian Journal of Education, 3, 1-12. Meier, R. S. (1977). Student ratings of instruction : Characteristics that influence evaluations of teachers. Unpublished doctoral dissertation, Purdue University, West Lafayette, IN. Meier, R. S., & Feldhusen, J. F. (1979). Another look at Dr. Fox: Effect of stated purpose for evaluation, lecturer expressiveness and density of lecture content on student ratings. Journal of Educational Psy­ chology, 71, 339-345. Menges, R. J. ( 1 979). Evaluating teaching effectiveness: What is the proper role for students? Liberal Education, 65, 356-370.

Menges, R. J. ( 1980). Incentives and motivation in the teaching-learning process: The role of teacher intentions. Paper presented at the 22nd International Congress of Psychology, Leipzig, Poland. Merlino, A. (1977). A comparison of the effectiveness of three levels of teacher questioning on the outcomes of instruction in a college biology course. (Doctoral dissertation, New York University, 1976). Dissertation Abstracts International, 37, 555 1-A. Naftulin, D. H., Ware, J. E. Jr., & Donnelly, F. A. (1973). The Doctor Fox lecture: A paradigm of education seduction. Journal of Medical Education, 48, 630-635. Perlberg, A., Peri, J. N., Weinreb, M., Nitzan, E., & Shimron, J. (1972). Microteaching and videotape recordings: A new approach to im­ proving teaching. Journal of Medical Education, 47, 43-50. Perry, R. P., Abrami, P. C., & Leventhal, L. ( 1979). Educational seduction: The effect of instructor expressiveness and lecture con­ tent on student ratings and achievement. Journal of Educational Psychology, 71, 107- 1 1 6. Perry, R. P., Abrami, P. C., Leventhal, L., & Check, J. (1 979). Instructor reputation: An expectancy relationship involving student ratings and achievement. Journal of Educational Psychology, 7I, 776-787. Perry, R. P., Leventhal, L., & Abrami, P. C. (1979). An observational learning procedure for improving university instruction. Paper pre­ sented at the meeting of the Fifth International Conference on Improving University Teaching, London, England. Postlethwaite, S. N., Novak, J., & Murray, H. T. Jr. (1972). 1he audio­ tutorial approach to learning. Minneapolis: Burgess Publishing Co. Powell, J. P. (1974). Small group teaching methods in higher education. Educational Research, 16, 1 67- 171. Ramagli, H. J. Jr. (1979). The Doctor Fox effe ct: A paired lecture comparison of lecturer expressiveness and lecture content. Unpubl­ ished doctoral dissertation, University of Florida. Gainesville, FL. Ramagli, H. J. Jr., & Greenwood, G. E. (1980, April). The Dr. Fox effect: A paired lecture comparison of lecturer expressiveness and lecture content. Paper presented at the annual meeting of the American

Educational Research Association, Boston. Rosenshine, B. (1971). Teaching behaviors and student achievement. Slough, Bucks (U.K.): National Foundation for Educational Re­ search. Rosenshine, B., & Furst, N. (1973). The use of direct observation to study teaching. In R. M. W. Travers (Ed.), Second handbook of research on teaching. Chicago: Rand McNally. Rotem, A., & Glasman, N. S. (1979). On the effectiveness of students' evaluative feedback to university instructors. Review of Educational Research, 49, 497- 5 1 1 . Schustereit, R . C . (1980). Team-teaching and academic achievement. Improving College and University Teaching, 28, 85-89. Sharp, G. (1981). Acquisition of lecturing skills by university teaching assistants: Some effects of interest, topic relevance, and viewing a model videotape. American Educational Research Journal, 18, 491 -502. Shulman, L. (1975). Development ofa category observation systemfor the analysis of video-taped class sessions. Montreal, Canada: McGill University, Centre for Learning and Development. Shymansky, J. A., & Penick, J. E. (1979). Use of systematic observations to improve college science laboratory instruction. Science Educa­ tion, 63, 1 95-203. Simon, A., & Boyer, E. G. (Eds.). ( 1974). Mirrors for behavior III: An anthology of observation instruments. Wyncote, PA: Communication Materials Center. Smith, D. G. (1977). College classroom interactions and critical think­ ing. Journal of Educational Psychology, 69, 1 80- 190. Smith, D. G. (1980, April). Instruction and outcomes in an undergraduate setting. Paper presented at the annual meeting of the American Educational Research Association, Boston. Smith, J. P. (197 1 ). The development of a classroom observation instrument relevant to the earth science curriculum project. Journal of Research in Science Teaching, 8, 23 1-235. Smith, L. R. (1977). Aspects of teacher discourse and student achieve­ ment in mathematics. Journal of Research in Mathematics Educa­ tion, 8, 195-204. Taha, H., & Elzey, F. F. (1964). Teaching strategies and thought processes. Teachers College Record, 65, 524-534.

RESEARCH ON TEACHING IN HIGHER EDUCATION

Tamir, P. (1977). How are the laboratories used? Journal of Research in Science Teaching, 14, 3 1 1 -3 16. Taylor-Way, D. G. (198 1, April). Adaptation of Interpersonal Process Recall and a theory of educating for the improvement of college instruction. Paper presented at the annual meeting of the American

Educational Research Association, Los Angeles. Travers, R. M. W. (Ed.). (1973). Second handbook of research on teaching. Chicago: Rand McNally. Ware, J. E. ( 1974). The Doctor Fox effect: A study of lecturer effective­ ness and ratings of instruction. Unpublished doctoral dissertation, Southern Illinois University, Carbondale, IL. Ware, J. E., & Williams, R. G. (1975). The Dr. Fox effect: A study of lecture effectiveness and ratings of instruction. Journal of Medical Education, 50, 149-1 56. Ware, J. E. Jr., & Williams, R. G. (1979). Seeing through the Doctor Fox effect: A response to Frey. Instructional Evaluation, 3, 6- 1 0.

777

Ware, J. E. Jr., & Williams, R. G. (1980). A reanalysis of the Doctor Fox experiments. Instructional Evaluation, 4, 1 5- 18. Wenk, V. A., & Menges, R. J. (1982). Classroom questions in postsecond­ ary education. Unpublished manuscript, Northwestern University, School of Education, Evanston, IL. Williams, R. G., & Ware, J. E. (1976). Validity of student ratings of instruction under different incentive conditions: A further study of the Dr. Fox effect. Journal of Educational Psychology, 68, 48-56. Williams, R. G., & Ware, J. E. (1977). An extended visit with Dr. Fox: Validity of student satisfaction with instruction ratings after re­ peated exposures to a lecturer. American Educational Research Journal, 14, 449-457. Yelon, S., Cooper, C., Henry, R., Korzenny, S., & Alexander, G. P. ( 1 980). Style of teaching : An ethnographic study. Unpublished manuscript, Michigan State University, Learning and Evaluation Service. East Lansing. ML

27. Researc h o n W ritte n Co m pos it i o n M a r l e n e Scardamal i a York Un iversity

Carl Bereiter

Ontario Institute for Studies in Education

Introduction

such magnitude as to provoke an educational crisis. Rather, it would seem that the source of the sudden dissatisfaction with writing competence is a rise in expectations. Increasing numbers of low-income and minority students have been entering higher education and seeking entry into middle-class occupations where facility with written language is expected. At the same time, with the trend in occupations toward processing. information rather than material, more and more people find themselves needing to communicate through written language and depending on written communication from others (Sawyer, 1977). This situation has created a new emphasis on readability and on the informational adequacy of texts, rather than on technical correctness (Black, 1982; Hirsch, 1977; Nystrand, 1979).

During the period following the publication of the Second Handbook of Research on Teaching, writing has risen from a relatively neglected school subject to an object of lively atten­ tion in both the popular and the academic media. Popular interest seems to have been set aflame by a Newsweek article (December 8, 1975) titled "Why Johnny Can't Write," a title that gives the flavor of much of the discussion that followed in the popular media. Given the short life expectancy of educa­ tional crises, it would not be appropriate in a handbook article to devote many paragraphs to analyzing the current writing crisis. There are, however, some points brought to light in the recent flurry of discussion that are important for establishing the educational significance of research to be discussed in this chapter. Of particular interest was the extremely wide range of people thought to be in need of writing improvement. Although the most conspicuous writing difficulties were to be found among speakers of nonstandard dialects (Shaughnessy, 1977a), univer­ sity students in general were found wanting (Lyons, 1976), as well as bureaucrats (Redish, in press) and business writers (Odell, 1980). It thus appears that the "writing problem" is not a matter of a minority for whom writing is especially proble­ matic but rather a matter of the majority. When the news media take up an instructional issue, they tend to focus on the question of whether things have gotten worse, a question that is usually unanswerable and often beside the point. Data from the National Assessment of Educational Progress (1975, 1980a, 1980b, 1980c) have, in fact, shown evidence of declines across the period 1969-1979, but not of

Response to the " Writing Crisis" One response to the public interest in writing was a dramatic increase in research activity. Whereas in 1974 there were no sessions at the American Educational Research Association's annual meeting presenting research on writing, by the time of the 1979 annual meeting there were 16 such sessions. There were, however, other more direct responses to the demand for writing improvement that also have relevance to research on teaching. Evaluation. Many educational jurisdictions in the United States and Canada called for the first time for systematic evaluation of writing abilities. Writing evaluation had long been recognized as a thorny problem, characterized by uncertainties both as to

The authors thank reviewers Robert de Beaugrande (University of Florida) and Martha King (Ohio State University). The preparation of this chapter was aided by grants from the Alfred P. Sloan Foundation and by the Ontario Ministry of Education, through its block grant to the Ontario Institute for Studies in Education. Marlene Scardamalia's affiliation has changed to the Ontario Institute for Studies in Education since the publication of this volume.

778

RESEARCH ON WRITTEN COMPOSITION the manner of obtaining data and the manner of scoring it (Diederich, 1964). The sudden need to do it anyway led to writing evaluation's becoming itself a major subject of research (Cooper & Odell, 1977). Document Design. Not all of the concern about writing im­ provement was focused on instruction. In 1978 U.S. President Carter issued an executive order (E012044) calling for govern­ ment regulations to be written in "plain English" that was understandable to the people who were expected to comply with the regulations. The Document Design Project, under­ taken by the American Institutes for Research, provided direct guidance to public and private agencies in producing more readable documents. With the involvement of researchers from Carnegie-Mellon University, the project also carried out a substantial program of research looking into the relation between what readers do to understand and what writers do to be understood. (See, for instance, "Revising Functional Docu­ ments: The Scenario Principle" [Flower, Hayes, & Swarts, 1980].) Instructional Improvement. The notion that something was wrong with writing instruction could hardly be said to have taken language arts educators by surprise. There was already ample evidence that most teachers from elementary school through university were ill prepared to teach writing (Morrison & Austin, 1977), that not much writing was done in schools (R. Applebee, 1966), and that much of the required writing gave little motivation or scope for the exercise of higher level composing abilities (Muller, 1967). It is probably safe to say, accordingly, that in the view of most language arts specialists the knowledge needed to improve writing instruction was presumed already to exist, so that the problems were mainly problems of implementation. The subtitle of a monograph by Donald Graves (1978), Let Them Write, succinctly expressed one view of how to do something straightaway about the condition of writing in the schools. The most comprehensive effort at instructional improvement was the Bay Area Writing Project, which soon broadened into the National Writing Project (Gray & Myers, 1978). Working through the medium of teacher development seminars, this project sought to increase teachers' own interest and compe­ tence in writing and to acquaint them with the best of available teaching activities. Thus tackling the known shortcomings of writing instruction in a direct and positive manner, the Bay Area Project received widespread endorsement (see, for in­ stance, Fadiman & Howard, 1979 ; Nelms, 1979).

New Focuses of Research At least nine new educationally relevant focuses of research on writing have emerged in the past decade. At present they are largely separate, conducted by only slightly overlapping groups of researchers and with little basis for relating findings from one area to another. Possibilities for synthesis exist, however, and will be touched on in the course of this chapter. 1. Early development of written symbolism. At the center of this research focus is Vygotsky's contention that the child's

779

discovery of written symbolism is a major step in the develop­ ment of thinking (Vygotsky, 1978, chapter 8). The principal interest has been in how children from the very beginning construct meaning - inventing spellings and other meaningful characters to convey meanings for which they have no conven­ tional symbols (Bissex, 1980; Clay, 1975; DeFord, 1980; Read, 1971). The practical outcome of this research has been the encouragement of writing at much earlier ages than had previously been thought feasible (Graves, 1983). This opens the possibility, currently under investigation by Graves and others, of unifying the acquisition of writing and reading skills. 2. Discourse analysis. Whereas earlier linguistic research on writing had focused on the sentence and its constituents (e.g., Hunt, 1965), recent research has tended to focus on principles that tie sentences to one another and to the context of action in which the language is embedded. Conceptual roots of the research are in systemic grammar, particularly the concept of cohesion (Halliday & Hasan, 1976), and in the theory of speech acts (Grice, 1975). Current research has been largely devoted to studying developmental trends in attainment of discourse com­ pete.nee (Bracewell, Frederiksen, & Frederiksen, 1982; Britton, Burgess, Martin, McLeod, & Rosen, 1975; King & Rentel, 1981; Mccutchen & Perfetti, 1982). 3. Story grammar. The early age and apparent universality of children's grasp of the structure of narratives has generated widespread research interest (Mandler & Johnson, 1977; Mandler, Scribner, Cole, & DeForest, 1980; Stein & Trabasso, 1982). A thorough airing of theoretical issues is given in Beaugrande (1982b) and accompanying responses by other scholars. Most of the research has dealt with story comprehen­ sion and recall, but there have been extensions into research on children's story production (Botvin & Sutton-Smith, 1977; Bracewell et al., 1982; King & Rentel 1981) and knowledge about stories (Bereiter & Scardamalia, 1982; Stein & Policas­ tro, 1984). 4. Basic writers. Whereas the preceding three focuses of research all deal with young children, another major recent focus has been on what have come to be called "basic writers," defined by Shaughnessy (1977b) as "adult beginners" - adult students displaying an extreme lack of development of writing ability. Research on basic writers has emphasized problems arising from discrepancies in uses and conventions between nonstandard spoken dialects and standard written English and other sociolinguistic factors that create barriers to college-level standards of written communication (Shaughnessy, 1977a; see also the Journal of Basic Writing, the first volume of which appeared in 1978). 5. The "new" rhetoric. One of the most interesting develop­ ments of the past decade has been the emergence of a new vein of rhetorical research oriented toward instruction (Warnock, 1976). With links to the classical idea of "invention," or the generation of novel material for compositions, this new vein of research has investigated teachable techniques for directing and elaborating thought about composition content (see Corbett, 1971; Kinneavy, 1980; Winterowd, 1975; Young, Becker, & Pike, 1970; survey by Young, 1976).

780

MARLENE SCARDAMALIA AND CARL BEREITER

6. Writing "apprehension." Psychometric research on anxiety toward writing has identified a reliably measurable trait that appears to be significantly implicated in how well people write and in how they feel about writing (Daly & Miller, 1975; Faigley, Daly, & Witte, 1981). 7. Classroom practices. Whereas earlier research on class­ room practices in the teaching of writing relied mainly on surveys (e.g., R. Applebee, 1966), a trend is now well established toward more in-depth analysis, using ethnographic and socio­ lingufstic methods (Florio & Clark, 1982; Pettigrew, Shaw, & Van Nostrand, 198 1 ; Staton, Shuy, & Kreeft, 1982). 8. "Response.1' Instructional research on writing has long been concerned with the effects of different kinds of marking or correction of student writing (reviewed by Knoblauch & Bran­ non, 1981). A more recent research interest, however, has been in other, more highly interactive kinds of response to student writing (Freedman, 1981). Paramount among these have been "conferencing" (Graves, 1978), a procedure using brief, individ­ ual consultations at various points during composition, and interactive journal writing, in which the teacher responds individually to students' personal journal entries (Staton, 1980). 9. The composing process. Methods and concepts of cognitive science have been brought to bear on the question of what goes on in the mind as people compose. This active area of research has included basic theoretical studies (e.g., Hayes & Flower, 1980a; Scardamalia, Bereiter, & Goelman, 1982), developmen­ tal studies (e.g., Burtis, Bereiter, Scardamalia, & Tetroe, 1983, analysis of writing in real time (Matsuhashi, 1982), and studies comparing experts and novices (e.g., Flower & Hayes, 1980a; Perl, 1979; Sommers, 1980). In addition to these nine new focuses of research, we should mention a potential focus - neuropsychological research re­ lated to writing (Emig, 1978). As Faigley (1982) points out, brain research currently serves primarily as a source of meta­ phors for people concerned with writing (see, however, Glassner, 1982).

Movements Toward Synthesis Older reviews of research on writing customarily lamented its neglect in comparison to other major uses of language (West, 1967). Although writing still lags far behind reading as an object of research, there is now enough research relevant to the teaching of writing and composition that it cannot be ade­ quately reviewed and synthesized in a chapter of this size. This is only partly a result of increased quantity, however. It is even more a result of the newness of the various research focuses previously described. There has not been time for the necessary cross-linkages to develop. As an example, the work on early development of written symbolism and the work on composing processes in older children and adults should obviously tie together into a coher­ ent picture of writing development (Gundlach, 1982). Yet the two bodies of research are at present unintegrated to the extent of being almost devoid of cross-reference. Integration will depend on evidence, currently lacking, as to what connection exists between the young child's discovery of written symboliza-

tion and the older child's development of competence in text composition. Two general approaches seem to be currently in contention as ways to bring the many strands of writing research together. One approach might be characterized as "contextual" (Mishler, 1979), centering on the meanings that writing has for people in different contexts (Britton et al., 1975; Graves, 198 1 ; King, 1978). The other is a cognitive science approach, drawing broadly on the contributions of various disciplines to this hybrid science (see Beaugrande, 1982a and Black, 1982 for overviews). At present the contextual approach is clearly ascendant among language arts educators (see any issue of Language Arts), and possesses a marked advantage when it comes to establishing links between theory, disciplined observation, and the everyday experiences of teachers. The cognitive science approach has its main advocates among researchers, for rea­ sons similar to those that have supported the "cognitive revolution" in many other areas dealt with in this volume. It provides pretty much the "only paradigm in town" for investi­ gating complex mental processes, which all sides agree are of central concern in writing. Moreover, it permits writing re­ searchers to take advantage of theoretical and methodological advances that have been made by cognitive scientists in such other areas as linguistics, psycholinguistics, developmental psy­ chology, and cognitive anthropology, and such special research focuses as problem solving, discourse processes, social cogni­ tion, metacognition, self-regulation, emotions, dreams, and consciousness. There is no immediate prospect of a cognitive science synthe­ sis of the nine strands of research previously mentioned. Some cognitive research is going on in each of them, however, and Beaugrande (1984) has made a major contribution to synthesiz­ ing diverse findings within a cognitive framework. Recent indications of a drawing together of previously unconnected areas are an examination of writers' emotional blocks in terms of composing strategies (Rose, 1980) and an effort to base a major international evaluation of writing achievement on cog­ nitive models (Purves & Takala, 1982). Perhaps the least assimilated focus of research is the one we have labeled "response," and it has been a major focus of those taking a contextual approach. In the present chapter we will try to establish some conceptual bridges between cognitive process research and types of student-teacher interaction in writing, but it seems that a real synthesis will require research that combines cognitive and ethnomethodological perspectives (cf. Wertsch, in press). What has been achieved so far is a fairly coherent description of mental processes that go on in writing and substantial progress toward understanding the cognitive changes involved as oral language competence gets reshaped into the ability to compose written texts. This chapter will be largely devoted to discussing progress on these fronts and its relevance to research on the teaching of writing.

Nature of the Composing Process Research into the composing process has been greatly aided by the procedure of having subjects think aloud while composing

RESEARCH ON WRITTEN COMPOSITION (Emig, 1971; Hayes & Flower, 1980a). Protocols thus obtained from skilled writers make it evident that, although there is obviously a structure to the composing process, it does not correspond to the traditional textbook picture of collecting material, organizing it, writing it out, and then revising it. Those procedures all occur, but not in such a linear manner (Hayes & Flower, 1980a; Sommers, 1980). It should be understood that thinking aloud protocols are not presumed to give a direct record of the mental processes of composing. Many significant mental events are not available to the writer for mention (Scardamalia et al., 1982), and thinking aloud itself has been found to introduce certain distortions (Black, Galambos, & Reiser, 1984). Thinking aloud protocols are better thought of simply as data (Ericsson & Simon, 1980), to be used in conjunction with other data to make inferences about the composing process. Bereiter & Scardamalia (1983b) discuss six different levels of inquiry into the composing process that need to be coordinated in theory development. Process description, such as may be based on protocol analysis, is one such level; others include text analysis, theory-guided experi­ mentation, and simulation (including simulation through inter­ vention with live writers). Although in the discussion that follows it will be impossible to indicate the nature of the evidence for every conclusion cited, we try to avoid stating conclusions that are based on evidence from only one level of inquiry. Several general models of the composing process have been put forth, each with the virtue of highlighting certain aspects of the process. Beaugrande (1984) offers a multilevel model that gives prominence to the different kinds of mental units that must be manipulated in composition, and we will appeal to that model in discussing mental representations of text. Augustine (1981) offers a model that highlights the rhetorical decisions writers must make. The model that gives the most explicit account of mental operations is that of Hayes and Flower (Flower & Hayes, 1981; Hayes & Flower, 1980a, 1980b), the gross structure of which is depicted in Figure 27.1. It is also the most widely cited model, and has tended to fix the vocabulary people use in talking about the composing process. According to this model, the main parts of the composing process are planning, translating, and reviewing. The heart of planning is generating ideas, most of which are ideas for what to write. These are culled and arranged to create a plan that controls the process of actual text production, which is called translating. Some of the generated ideas, however, are ideas for goals to be pursued, and these are stored for consultation throughout the composing process. These various subprocesses sound fairly obvious, of course. The strength of the model lies in its claim to account for the amazing diversity of mental events during composition on the basis of a relatively small number of such subprocesses. This is accomplished by a control structure that permits virtually any subprocess to incorporate any other subprocess. Thus, the whole planning process may be called up in the service of editing, or the reviewing process may be called up for purposes of arriving at an organizing decision. This property of the model, sometimes called "recursion," sets it apart from the step-by-step models of composition that have commonly been explicitly or implicitly endorsed in compo­ sition textbooks (Rose, 1981).

781

Figure 27. 1 depicts only the general structure of a model of the composing process. One of the claims made by Hayes and Flower (1980a) is that individual differences in composing strategies may be represented by different detailed models that all fit the general structure. In a later section we will discuss a model of an immature composing process which, even though it differs markedly from the model of mature composing con­ structed by Hayes and Flower, nevertheless fits the general structure and may be viewed as a radically reduced version of the mature model. Caccamise (in press) obtained protocol data on students generating composition ideas that led her to propose a different detailed model of the generating process from that originally proposed by Hayes and Flower (1980a), but one that still fits the general structure. The general structure of the model, thus, appears to do what it is supposed to, which is to serve as a frame for working out more detailed and possibly more controversial accounts of how the mind copes with writing tasks. The Hayes and Flower model is in the tradition of problem­ solving models pioneered by Newell and Simon (1972). Re­ cently Newell (1980) has argued that all human goal-oriented symbolic activity resembles problem solving in that it consists of a heuristic search through what Newell calls "problem spaces." A problem space contains a set of symbolic structures or knowledge states and a set of mental operations that can be applied to change one knowledge state into another. Heuristic search through the problem space consists of such mental operations being carried out under the constraints of "search control knowledge." What writing researchers typically call a "composing strategy," and the kind of thing that can be represented in the framework of the Hayes and Flower model, is what Newell calls search control knowledge. Describing a person's search control knowledge gives only a partial account of the person's writing competence. It remains to describe the symbolic structures available to the writer and to specify the operations on those structures that are within the writer's repertoire. Research on children's composing processes, to be discussed in the next section, suggests that an important key to understanding why they write the way they do is the kinds of mental representations of text that they have available to operate on. Figure 27.2 depicts a composing process model developed by Robert de Beaugrande (1984). It is based on a synthesis of experimental findings related to the kinds of symbolic struc­ tures operated on within the course of text production. As did Figure 27.1, Figure 27.2 gives only the gross structure of the model. Goals comprise a variety of symbolic structures - not only representations of the intended outcome of the writing effort but also representations of the reader, of text type, and style. An idea, in Beaugrande's model, is "a configuration of conceptual content that acts as a control center for building the text-world model (the total configuration of knowledge acti­ vated for processing the text . . . )" (Beaugrande, 1984, P. 109). Conceptual development refers to the generation and integration of specific items of content that constitute the detailed represen­ tation of the ideas. Beaugrande calls his a "parallel-stage interaction" model, indicating that the various processes of symbolic construc­ tion go on more or less simultaneously and that they are

782

MARLENE SCARDAMALIA AND CARL BEREITER

TASK ENVIRONMENT THE RHETORICAL PROBLEM

TEXT

Topic Audience Exigency

PRODUCED SO FAR

WRITING PROCESSES THE WRITER'S LONG-TERM MEMORY

"z I

Knowledge of Topic, Audience, and Writing Plans

I

.,,:

"

z

ORGANIZING GOAL SETTING

I

REVIEWING

TRANSLATING

PLANNING

I I

I

I

EVALUATING REVISING

I

l

MONITOR

I I

I

Fig. 27.1. Structure of a model of the composing process. From "Writing as Problem Solving" by J. R. Hayes and L. S. Flower, 1980, Visible Language, 14(4), pp. 388-399. Copyright 1980 by Visible Language. Reprinted by permission.

writer + look-back

. . . . . . .

EX�RE�SJ;ON: -: • : • : • :- :- : • : -: •

look-ahead�

PERCEPTION OF CURRENT TEXT

··············· .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ::....:·..:.:..:.·-r.;� •••• ••••••• -:-:--:.:-:.+·..:...:. : ..:. · : ::..c·

. . . . . . . . . . .. . . . .. .. .. . . . . . . • .

.

••• • ••• • • · • • X RESSION • • • • E P •

:....: : ·..:.:..:...:.. · ::..c· :

. . . . . . . . . •. . . . . . . . . . .

• •• •• ••• • •• : .": ." • • • • CONCEPTUAL • • • ••••••••• ·• •. •••••• • •• • • • • :-:- :- •• • • • • -: : • • • ••• • • • • . J.-t--:-�---:---:+-� • • • • MENT ELOP E V D • • • ••• • : • • • •

:I � · .- - - - - - - - - -

••• +

+

+

+

+ WM -+

-+

-+

-+

• CONCEPTUAL • • DEVELOPMENT

-+ •••

secondst---t--'1t---t---i--i---t--i--+--i---t---+--+-t---t---+--t--t--t--+--t---+--+--t-t---+--+--+----+-+--+---+--t 2 2 1 1

□...... . . .... srn

□

STSS

Fig. 27.2 A parallel-stage interaction model of text production. (LTM = long-term memory; STM = short-term memory; STSS = short-term sensory storage; WM = working memory.) From Text Production: Toward a Science of Composition (p. 106) by R. de Beaugrande, 1984, Norwood, NJ: Ablex. Copyright 1984 by Ablex. Reprinted by permission.

RESEARCH ON WRITTEN COMPOSITION

interpenetrable- that is, that what happens as a result of processing at one level may alter the knowledge states current at other levels. Broadly speaking, these premises are also embodied in the Hayes and Flower model. They imply that the skilled writer must have flexible access to a wide range of mental representations of actual and intended text and of conditions bearing on plans for the text, as well as a highly sophisticated control structure for coordinating operations on these many different kinds of knowledge states. A sense of the variety of levels at which text is mentally manipulated by mature writers may be gained from the follow­ ing fairly typical excerpt from a planning protocol taken from our files. The writer was planning a short essay on the question, "Should children choose what subjects they study in school?" Okay, reasons why .... I'm just wondering if values [referring here to a previously generated idea, in Beaugrande's terms] will fit under this initial one [referring to another idea], which means that this initial one is really the argument [shifting now to thinking of the item in terms of text type], and that argument would be supported from those different subsections [a plan, now, for achieving the unstated goal of convincing the reader].

The models of the composing process developed thus far are what Kosslyn (1980) calls "protomodels." They are precursors of theory, not instantiations of theories. Theory development itself - as one might expect from the newness of the field- is embryonic. As in a number of other areas of cognitive research, however, theoretically interesting questions are starting to emerge from the analysis of differences between experts and novices or children. What, in cognitive process terms, must be added to oral language competence in order to produce competence in writ­ ten composition? Do immature writers naturally possess the basic structure of the mature composing process, needing only more sophisticated knowledge in order to write like an expert, or does the control structure itself have to be acquired? Are all of the relevant kinds of mental representations available to them, or do the very prototypes of some of them have to be constructed? Are unskilled adult writers merely lacking in some of the knowledge experts have or have they gone down a different developmental route that has led them to a different system of text production? Questions such as these, which will be taken up in the next sections, point toward matters of both theoretical and practical significance.

Problems in Learning to Write Discussions of writing are usually organized according to outwardly visible parts of the process- gathering material, outlining, drafting text, revising, and so on. These divisions do not necessarily correspond to divisions in the underlying men­ tal process, however. For instance, it makes little psychological sense to treat changing a sentence after it is written down as a different process from changing it before it is written. Indeed, evidence to be cited later suggests that focusing on revision as if it were a special process may lead to wasted instructional effort. In the present section, accordingly, we shall abandon the conventional categories and organize the discussion around

783

what appear to be psychologically coherent aspects of the composing process. There has been considerable discussion of what fundamen­ tally distinguishes writing from oral communication (Olson, 1977; Rubin, 1980b). It is not simply the medium of expression. Many writers, past and present, have used dictation as a way of producing written text. A body of recent developmental re­ search supports the view that competence in written composi­ tion does not consist merely of special knowledge and skills added to those of oral language ability. Rather, it involves radical conversion "from a language production system depen­ dent at every level on inputs from a conversational partner to a system capable of functioning autonomously" (Bereiter & Scardamalia, 1982). In the following sections evidence will be examined concerning aspects of competence that appear to create special problems for the immature writer. In each case we will look first at what expert or mature writers do, then at what novices and children do, and finally at intervention studies that shed light on what is involved in moving students toward more expert-like competence.

Framing Written Discourse Much recent research has focused on formal text structures that are believed to guide the writer in constructing a text and a reader in comprehending it (Mandler & Johnson, 1977; Stein & Glenn, 1979; Thorndyke, 1977; van Dijk, 1980). Story gram­ mar, discourse schema, script, superstructure, and frame are terms that have been applied to such structures. We use the term framing here, to indicate the active process of creating an organizing structure for a composition (Bracewell et al., 1982). The premise shared by most investigators is that these struc­ tures are not merely characteristics that inevitably emerge in text but that they represent knowledge readers and writers possess (Mandler et al., 1980; Stein & Trabasso, 1982; but see Black & Wilensky, 1979). From this it would follow that a major requirement for competence in writing is learning the essential form of various literary types- narrative, exposition, argument, and the like. EXPERT COMPETENCE

If one accepts the premise that discourse schemata are a significant part of writers' competence, then it follows by definition that skilled writers have a well-developed repertoire of such schemata (Bereiter, 1980). What remains an empirical question is the kind of access writers have to their discourse knowledge. Is their discourse knowledge like geographical knowledge- functioning tacitly most of the time but available for conscious inspection and application when needed? Or is it like syntactical knowledge for people unschooled in gram­ mar- largely inaccessible, even though highly functional? There is not much evidence on this point, but what there is suggests that skillful adults can and frequently do made con­ scious use of abstract knowledge of discourse forms. Olson, Mack, and Duffy (1981) had university students think aloud while reading simple narrative and expository texts. They found readers of expository texts making predictions about the kind

784

MARLENE SCARDAMALIA AND CARL BEREITER

of information forthcoming in the text-for instance, predict­ ing that an example would follow, but not predicting what the content of the example would be. Examples are abstract entities, defined by their function in text; hence, predicting that an example will appear indicates explicitly thinking in terms of discourse schema knowledge. Thinking aloud protocols we have obtained from skilled writers also show numerous examples of explicit reference to discourse schema elements. In the following example, a writer who has been periodically reformulating her approach to a composition makes a third and final recapitulation of her plan: I would state my opinion, my point of view, taking in the fact that some people would not agree with it; and, taking note of this position, I would go through the disadvantages, which I think obviously can be overcome, and then move on to say how. After those three major points are out of the way I would conclude by talking about the benefits. Here we see that abstract notions such as"opinion,""position," and "major points" play an obvious function in helping the writer work out an overall structure for the text (Chafe, 1977). However, this is a rather extreme case of explicit use of discourse knowledge. In many other expert protocols discourse knowledge remains more in the background. It seems that a simple distinction between explicit and implicit or conscious and unconscious knowledge will not do. Skilled writers make aU kinds of use of their discourse knowledge, with varying degrees of consciousness and explicitness. STUDENT COMPETENCE

Most research on discourse schema knowledge in children has dealt with narrative. This research has yielded abundant evidence that implicit knowledge of narrative form guides children's comprehension of stories. After reviewing a number of such studies, Stein and Trabasso (1982) arrive at the forceful conclusion that the use of schematic knowledge is so powerful that listeners have little control over the types of retrieval strategies used during recall of narrative information. Even when listeners are instructed to reproduce texts in a verbatim form, they cannot do so when the text contains certain types of omissions or certain sequences of events. (p. 217) Story production evidently lags behind story comprehension, but by the end of the primary grades virtually all children are producing well-formed narrative compositions (Botvin & Sut­ ton-Smith, 1977; Bracewell et al., 1982; King & Rentel, 1981; Stein & Trabasso, 1982). Other standard literary genres, such as exposition and argument, have been much less extensively investigated, but in the several thousand compositions of these kinds that we have examined, we have noticed very few failures in children aged 10 years and up to conform to genre. Their discourse grammars may be simpler than those of experts, but when assigned an argument topic they produce something clearly recognizable as an argument and not a narrative or an exposition; likewise for the other genres. The deviations that do occur-usually producing narratives when some other genre

was called for - are more likely attributable to losing hold than to lack of the appropriate schema. Nevertheless, findings from the National Assessment of Edu­ cational Progress (1980a, 1980b, 1980c) indicate that students have serious weaknesses in meeting the more substantive requirements of written genres. On persuasive writing assign­ ments, about 31% of 13-year-olds and 22% of 17-year-olds were rated as failing to provide even minimal support for a position. On a task calling for making up a story about a picture, about a third of 17-year-olds were rated as failing to meet the fundamental demands of fiction "either because the plot is only barely outlined, the story rambles on without structure, the story is incomplete or the story is really several unconnected stories" (National Assessment, 1980a, p. 14). Such findings raise questions about the uses to which students are able to put their implicit discourse schema knowledge. Somewhat surprisingly, students from the age of 10 have been found to demonstrate ability to gain some conscious access to their discourse schema knowledge. Under questioning they can name discourse elements found in various genres (Bereiter & Scardamalia, 1982), and they can with some accuracy label such elements in their own and other texts (Scardamalia & Paris, 1985). Young students also appeal to schema knowledge when working on scrambled text problems, saying things like "It's about the planet . . . like telling the setting" (Scardamalia & Bereiter, 1984). Since it is unlikely that they were explicitly taught any of these things, it appears that they are able to draw deliberately on their implicit discourse schemata. The fact that children can gain access to their discourse schema knowledge through special questioning or special tasks does not, however, tell us anything about the uses they are able to make of this knowledge in composition. Thinking aloud protocols by school-age writers that we have analyzed show almost no evidence of the explicit use of abstract schema knowledge found in skilled adults. The difference here, however, could be in what young writers report rather than in what they do. Discovering the actual functions of discourse knowledge in children's composing requires experimental probing, which was the purpose of the intervention studies to which we now turn. INTERVENTIONS

In this and similar sections later, intervention studies are examined, not in terms of instructional effectiveness, but in terms of what they reveal about the nature of writing compe­ tence and its development. Instructional issues per se are treated separately in other portions of the chapter. One way to investigate children's use of discourse knowledge is to make elements of this knowledge more accessible and observe effects on composition. In one study (Paris, 1980) elementary school children were trained in recognizing ele­ ments of the argument genre (statement of belief, reason, reason on other side, example, etc.). The elements were as much as possible taught out of context and no practice or suggestion was given concerning the use of them in composition. Neverthe­ less the training showed some transfer to composition, resulting in increased presentation of reasons on both sides of an

RESEARCH ON WRITTEN COMPOSITION argument and a decline in the number of nonfunctional ele­ ments. These results suggest that children may be able to make some decisions in writing based on abstract structural considerations rather than on concrete considerations of content. A more direct test of this possibility was made by coaching children in a composing procedure of first choosing an abstract element and then producing a sentence of the chosen kind, then choosing a next element, and so on (study by Bereiter, Scardamalia, Anderson, & Smart, reported in Bereiter & Scardamalia, 1982). The most notable result here is that 12-year-olds were able to carry out the procedure and claimed it helped them think of things to say. This testimony was supported by a test of transfer to normal writing, which showed a general increase in the number and variety of discourse elements used. Apparently, then, children can use abstract specifications of discourse elements as probes for retrieving appropriate content from memory. This is a noteworthy capability, which will figure prominently in a subsequent description of a model of imma­ ture composing processes. In the preceding study, children made their own choices of which discourse elements to use where. A third experiment, by Bracewell and the present writers (reported in Bracewell, 1980), examined the effect of having such choices made by an expert, with the child still generating the actual content and language. Children first wrote argument essays. Then, in individual conferences, the experimenter labeled discourse elements and suggested the insertion of additional discourse elements that would make for a stronger or more sophisticated argument. Except for the increased elaboration of content, directly in­ duced by the experimenter, the essays showed no significant change. These studies leave much unexplained, but they do at least make it clear that there is more to competence in framing discourse than having an abstract schema in the mind that regulates what kind of element will go where. Such schemata are no doubt an important part of expert competence, clearly foreshadowed in the performance of young writers (Waters, 1980). But when a mature level of such competence is directly injected into the composing process of young writers, as in the Bracewell et al. experiment cited above, the effects are minimal. Over and above this regulative role, abstract discourse knowl­ edge in experts appears to serve a variety of other roles in formulating goals, making strategic decisions, and constructing high-level representations of content that can be manipulated effectively during text planning (Scardamalia & Paris, 1985).

Content Generation Thinking aloud protocols indicate that for experts and novices alike the greater part of effort in writing goes into generating content - thinking of what to say (Burtis et al., 1983 ; Hayes & Flower, 1980a). Mature writers, however, typically generate far more content than they will use or intend to use in their compositions, whereas for young writers finding enough con­ tent is frequently a problem and they cannot imagine discarding anything that would fit (Bereiter & Scardamalia, 1982). An obvious explanation is that young writers don't know enough

785

about the subjects they are asked to write on. While that is undoubtedly true (A. Applebee, 1981), young writers also have difficulty getting access to the knowledge they do have. Here the difference between composition and conversation is note­ worthy. In conversation there are continual cues from other speakers that aid retrieval of relevant content. Writers, unless they are in collaboration, are left to their own resources and thus have need of strategies for self-directed memory search. EXPERT COMPETENCE

Memory search is seldom directly revealed in protocols. Identi­ fying memory search strategies is accordingly a highly specula­ tive matter, resting on the general fit of hypothesis to data rather than on reconstruction of processes from individual protocols (cf. Caccamise, in press). Two processes that seem necessary to account for expert performance are metamemorial search and heuristic or goal-directed search. Metamemorial search is search aimed at determining the availability of information in memory rather than at retrieving specific information (Bereiter & Scardamalia, 1982). In proto­ col data we have collected it is suggested by statements about having much or little information on a topic, as in the follow­ ing: I think I'd probably stay away from [the issue of] testing be­ cause - I'm not really sure: I think there should be something but since I don't know enough, I'd rather stay away from it so it wouldn't weaken my argument.

As the example illustrates, this kind of search can play an important part in high-level planning. Heuristic search is a term taken from the problem-solving literature, where it stands in contrast to trial-and-error search (Hayes, 1981). It involves reducing the space of possibilities one must search through by taking advantage of partial knowledge of what it is one is looking for. Greeno (1978) observed that in composition problems generally (which include artistic creation of all sorts as well as theory construction) a large part of the problem-solving effort goes into elaborating constraints. Constraints function heuristically when they serve to direct memory search toward promising areas of content. In keeping with this analysis, Flower and Hayes (1981) found that 60% of the ideas generated by expert writers were not in response to the assigned topic or in response to preceding ideas but were in response to the writer's own elaboration of the rhetorical problem. Caccamise (in press) presented data on content generation which she found did not fit a model of heuristic search but which rather suggested that people retrieved ideas in an asso­ ciative (spreading activation) way and then edited out ones that did not fit task constraints - in other words, in a trial-and-error fashion. However, her task was one that called for exhaustive search of memory (listing all the ideas that would fit). This would tend to make heuristic search unprofitable. Indeed, the task would seem to force writers into the condition we find young writers often to be in, which is that of generating every suitable idea they can.

786

MARLENE SCARDAMALIA AND CARL BEREITER

STUDENT COMPETENCE

Young writers show a lack of effective metamemorial search strategies for surveying the information they have available. When students in Grades 4 and 6 were asked to name topics about which they knew relatively much and relatively little, they evidenced great difficulty deciding, and few could name as many as three topics of either kind (study by Scardamalia, Bereiter, & Woodruff, summarized in Bereiter & Scardamalia, 1982). Whereas Flower and Hayes (1981) found expert writers generating most content ideas in response to their own elabora­ tions of the rhetorical problem, they found novice adult writers to generate 70% of their ideas in response either to the topic assignment or to the last content item considered. These results suggest that novice writers do not make the same use that experts do of search heuristics based on problem analysis, and that accordingly they must rely more on trial-and-error search, using whatever cues are available in the environment to stimu­ late retrieval of content from memory. Much the same conclu­ sion was reached by Crowley (1977) from an analysis of writing diaries kept by college students. Differences between young and old novices appear to depend on the number of constraints in the writing assignment that they employ as cues or tests. McCutchen and Perfetti (1982) have run computer simulations of memory search models differing in the number of constraints honored. Holding the knowledge base constant, they found that a one-constraint model yielded content similar to that found in second grade expository writing, a two-constraint model yielded fourth grade content, and a three-constraint model yielded content like that found in Grades 6 and 8. INTERVENTIONS

Many of the modem educational practices used to improve student writing may be viewed as interventions that compen­ sate for students' lacks in metamemorial and heuristic search. One of the simplest practices is permitting students to write on topics about which they already have a strong drive to express themselves. Graves (1975) found this to significantly increase the amount written by primary grade children. This practice renders metamemorial and goal-directed memory search un­ necessary, by allowing the writer to make use of memory contents that have already, for one reason or other, been activated. "Prewriting" activities (Boiarsky, 1982) - be they confer­ ences, class discussions, films, or private exercises in reflection -have the effect of bringing a wealth of memorial content to the fore where it is available for selection and use. " Discovery" heuristics (Young et al., 1970) are prewriting procedures that typically involve nonspecific goal-directed search, as in seeking an answer to the question, "Are there different ways of inter­ preting X?" An interesting question, not yet systematically investigated to our knowledge, is to what extent these heuristics mirror the heuristics actually used by expert writers and to what extent they serve as substitutes, achieving somewhat the same effect but without the kind of problem analysis and goal setting carried out by experts.

In our experience with a large variety of intervention studies (Bereiter & Scardamalia, 1982), we find that no matter what the intent of the intervention, children will find a way to use the input as cues to aid memory retrieval and will enthusiastically report that the intervention helped them " think of what to say" (see also Scardamalia & Bereiter, in press-a, and Woodruff, Bereiter, & Scardamalia, 1981). Simply urging students to go on, after the point where they claimed to have written every­ thing they could think of, was found to double their production (Scardamalia et al., 1982). If one thinks of memory as arranged in hierarchical networks, then it appears that what students normally do is recall high-level nodes and then stop (cf. McCutchen & Perfetti, 1982), as they do when recalling text (Mandler & Johnson, 1977; Meyer, 1977). Evidently urging students to go on prompts them to work further down into subordinate nodes or to move to neighboring nodes. Another intervention that has been found to increase content generation significantly is to have children list-after they have been assigned a topic but before they begin writing - isolated words that they think they might use (Anderson, Bereiter, & Smart, summarized in Bereiter & Scardamalia, 1982). This procedure would seem even more directly to promote search through subordinate nodes. However, it is perhaps best viewed, not as a way of inducing metamemorial search but as a way of streamlining the trial-and-error procedure that young writers already use for searching through content. Content generation can also be greatly boosted by allowing young writers to dictate. It is evidently the speed of dictation rather than its freedom from mechanical demands that produces this effect, since the effect is largely lost if dictation is slowed down to the child's normal rate of writing (Scardamalia et al., 1982). This speed effect suggests that content generation by young writers is very much an associative process, depending on spreading activation (Caccamise, in press), rather than a goal-directed process of heuristic search. Although the evidence is not pre­ cisely comparable, dictation by skilled adults does not appear to have anything like the same effect on content generation (Gould, 1980). Are there any interventions, however, that actually cast light on the ability of students to undertake metamemorial or heuristic search? Studies that we are aware of provide tenuous evidence suggesting that, when the task calls for it, young writers can undertake both kinds of search to some extent. In as-yet unreported research by the present authors, evidence of metamemorial search began to appear in the planning proto­ cols of 12-year-old writers when they were given a general topic (such as "an interesting animal") and had to choose their own specific topic to write on. They frequently had difficulty taking inventory of the extent and appropriateness of their knowledge about various animals, and some students just plunged in and began generating content about the first animal that came to mind; but on the whole there was substantially more evidence of metamemorial search than when students were writing on an already specified topic. The most convincing evidence of heuristic search capabilities in young writers comes from studies cited in the preceding section showing that students can generate content to fit specified genre elements -reasons, examples, arguments on the other side, and so forth. Another task that shows evidence of

RESEARCH ON WRITTEN COMPOSITION

inducing heuristic search is the task of wntmg toward a prescribed ending sentence (McCutchen & Perfetti, 1982); research by Tetroe and others, reported in Bereiter & Scarda­ malia, 1982). There is also evidence that when children are made aware that an exhaustive presentation of content is required - as in writing instructions for playing a game (Kroll, 1978; Scardamalia, Bereiter, & McDonald, 1978)- then they search further down from memory nodes than they do in their spontaneous expository writing. A summary position compatible with available findings would be the following: Experts and novices alike generate content partly by heuristic search, guided by knowledge of what they are looking for, and partly by associative processes that bring content spontaneously to mind. Good writing undoubt­ edly requires both. Novice writers, however, appear to rely more on associative processes and to lack the executive controls that would enable them to undertake heuristic searches on their own initiative. Such an analysis serves to remind us that it is the discourse production system as a whole that generates an item of text, not a particular subcomponent of it. The ability to undertake heuristic search for content depends on ability to formulate goals and plans, and those abilities in turn depend on ability to carry out purposeful searches of memory.

Written Language Production The actual production of written text involves a host of problems that usually receive only passing attention from writing researchers. These include handwriting (or typing), spelling, punctuation, and "usage" in the various senses in which written expressions are expected to be more carefully chosen than spoken ones. In its own right, spelling has begun to receive attention both from linguists (e.g., Stubbs, 1980) and psychologists (see volume edited by Frith, 1980). At least one innovative approach to spelling instruction, based on mor­ phemes, has appeared (Dixon, 1976) and shown evidence of effectiveness (Maggs, McMillan, Patching, & Hawke, 1 981). Syntax has been a long-standing concern of writing researchers (Hunt, 1965), and the ubiquitous T-unit continues to appear as a dependent variable, even though it is frequently not relevant to the questions now being investigated. A writing researcher who has tried to incorporate a detailed treatment of both the language-related and the content-related aspects of writing into a theoretical account is Beaugrande (1984). A key notion linking the two aspects is "linearization," which refers to a whole range of processes involved in translat­ ing ideational material that is often hierarchically or spatially organized into a linear sequence of acts that result in a text that can be read linearly. Of interest in the present discussion will be the extent to which linearization poses recognizable problems in learning to write. Most investigators of the composing process have limited their concern with language production to two aspects: (a) its possible interference with other subprocesses or competition with them for mental resources (Perl, 1979; Scardamalia et al., 1982), and (b) the problem of establishing voluntary control over language production processes that normally run without conscious attention in conversation (Bracewell, 1980).

787

EXPERT COMPETENCE

With regard to the two concerns just mentioned, expert writers are assumed to enjoy the advantages of (a) having procedures such as spelling and punctuation largely automatized so that they leave mental capacity free to deal with higher level tasks; (b) having control structures that permit switching back and forth between higher and lower level demands, so that they do not have to be coped with simultaneously; and (c) possessing a repertoire of linguistic alternatives that can be put to strategic use in achieving rhetorical goals (Beaugrande, 1984). This last point is parallel to the one . 'made about discourse schema knowledge - that experts are distinguished not so much by having the knowledge as by having it available for planning. Evidence of automaticity comes from "slips of the pen" (Hotoph, 1980), which resemble those made in speaking and suggest that writers are planning ahead at the same time that they are putting words on paper. STUDENT COMPETENCE

The hypothesis that immature writers are handicapped by having to devote attention to writing mechanics holds, it appears, only for quite inexperienced writers. Clay (1975), studying beginning writers, observed midsentence switches in syntactic form which she attributed to difficulties in transcrip­ tion. King and Rentel (1981) found the dictated stories of first and second graders to be much more fully developed than their written stories; but at this early stage in literacy, it is not clear whether one is seeing interference from writing demands or simply an inability to put all one's thoughts into writing. By Grades 4 and 6, however, Scardamalia et al. (1982) found that a small advantage in quality of dictated over written texts was reversed when children were urged to keep going. As noted previously, the slow rate of written production impeded content generation, but it was found to have offsetting advantages in coherence. Primary grade children often subvocalize each word or letter while writing it (Simon, 1973), which suggests their attention is wholly taken up with transcription. But by Grade 4 they have begun to confine subvocalization to pauses during transcrip­ tion, they begin making the same kinds of "slips of the pen" experts make, and if stopped suddenly while writing they report about five words already definitely in mind in advance of the last word written (Scardamalia et al., 1982). All of this suggests that by the end of elementary school, even though the mechan­ ics of writing may still be quite poorly perfected, children have developed efficient executive controls for switching attention between mechanical and substantive concerns, so that interfer­ ence and cognitive overload are not serious problems (Bereiter & Scardamalia, 1983b; Scardamalia et al., 1982). What does this say about linearization as a problem? The issue is not clearly enough drawn for a definite answer, but informal observations suggest that converting nonlinear knowl­ edge to linear text may be more a problem for mature writers than for children. One must first reckon with the astonishingly short latencies for children in starting to write once a topic has been given - ranging from an average of 3 seconds for fifth graders writing on a simple story topic (Zbrodoff, 1984) to an average of 3 minutes for high school writing assignments (A.

788

MARLENE SCARDAMALIA AND CARL BEREITER

Applebee, 1981). Secondly, children seem to have trouble suppressing the production of linear text, as when asked to bramstorm or take notes (Bereiter & Scardamalia, 1982; Burtis et al., 1983). As we shall propose in a later section the overall composi�g str�tegy common in young writers a�pears to be one that 1s specifically geared to linear text generation, and it is � stra �eg� that permits them to ignore the deeper problems of lmeanzat10n. Del�berate control of language devices has mainly been . mvestlgated with respect to syntax. Kress and Bracewell (1981) found that when instructed to do so deliberately, high school students were unable to generate many of the grammatical errors that they produced spontaneously. Bracewell (1980; 1983) and Scardamalia (1981) present a variety of experimental results showing children's difficulties in exerting intentional control over syntax. In one study, children were shown different renditions of the same content, varying in amount of coordina­ tion, the extremes being the following: A. Ernie has a dog. Grover has a cat. G rover has a canary. G rover has a dog. B . Ernie has a dog ; but G rover has three different pets, a cat, a canary, and a dog.

Children of age nine and up reliably chose Version B over Version A, and could explain why. However, when given parallel content and asked to imitate either A or B, they produced sentences with intermediate amounts of coordination instead. It must be assumed that the children had the necessary syntactic structure for producing Version A, and so their failure to produce it implies inability to exert intentional control over their syntactic choices. Children's difficulties with intentional control of syntax fit nicely with developmental models of working memory capacity (Scardamalia, 1981). Although syntactic alternatives are important for manipula­ ting focus, Bock (1982) has proposed that in ordinary speech syntactic alternatives often serve the more mundane purpose of allowing the speaker to start talking before the whole sentence is planned. If this is true, then becoming able to make the strategic choices that skilled writers are assumed to make would require not only having a repertoire of alternatives but also learning to override the natural tendency to let syntax be determined by the order in which words come to mind. A language feature that has received special attention in recent studies of student writing is cohesion, the tying together of sentences in text by means such as conjunctions, pronouns, and definite articles (Halliday & Hasan, 1976). Developmental gains in use of cohesive devices are typically found (Bracewell et al., 1982; King & Rentel, 1981; McCutchen & Perfetti, 1982). McCutchen & Perfetti (1982) show evidence, however, that the cohesive ties found in children's texts reflect developmental differences in memory search strategies giving rise to text content, and that these search strategies in turn reflect differ­ ences in children's sensitivity to coherence constraints in the writing assignment or in the existing text. Thus, in an unusually penetrating way, McCutchen and Perfetti show the interdepen­ dence of different parts of the cognitive system in text composi­ tion.

INTERVENTIONS

!he dict�tion studies reported by Scardamalia et al. (1982) were

mtervent10n studies aimed at discovering the extent to which children's composing would become more expert-like if various low-level problems of production were eased. Although the greater speed of production made possible by dictation led to generating more text, there was nothing in the findings to suggest that quality of children's compositions was adversely affected by the mechanical difficulties of writing. Another intervention that may free writers from demands of written language production is simply to instruct them not to pay attention to mechanics, sentence formation, and the like. Teachers often advise students to ignore such problems in first drafts, believing that it will free them to attend more to content. A formal test of this conjecture was carried out by Glynn, _ Bntton, Muth, and Dogan (1982) with college students. Stu­ dents did, in fact, generate more ideas, the more low-level requirements they were instructed to ignore. These findings are thus consistent with those from dictation studies. Rather differ­ ent results were reported with 11-year-olds, however, in a 1981 master's thesis by Jeffrey Fisher (1981). When told to ignore mechanics and focus on content while revising their stories, children did produce fewer mechanical changes but not more content changes. The most content changes were produced by repeated revision with no instructions to focus on anything in particular. Although the two studies are not similar enough to draw confident conclusions, it may be that what we have in the divergence of results between adults and preadolescents is the factor of intentional control rearing its head. Fisher's results suggest that for young writers it took extra mental effort for them to ignore mechanics. With college students, on the other hand, intentional control over language processes may be sufficiently well developed that certain monitoring processes can be turned off at will, thus saving mental capacity for use on other parts of the composing task. We are not aware of intervention studies concerned with establishing intentional control over language choices. Sen­ tence-combining practice is evidently effective in increasing the syntactic options available to writers (O'Hare, 1973), but there is no indication that it leads to greater voluntary control of these options. Mellon's expressed intent (1969) in devising sentence combining was, in fact, that it should promote auto­ maticity. Bracewell (1980) found that children were able to exert more intentional control over syntax when the content to be expressed was given to them in tabular form than when it was given to them in sentence form. It is interesting in this connection that many of the discovery heuristics provided for students make use of matrices or other nonlinear formats, which may have the effect of breaking up automatic processes of linearization and forcing the student to give conscious thought to the construction of sentences and sentence se­ quences.

Goal Formulation and Planning Protocol analyses have served to dramatize the amount and variety of planning that goes on in composition (Flower & Hayes, 1980a). As the term is commonly used in cognitive

RESEARCH ON WRITTEN COMPOSITION

science, " planning" refers to working through a task at an abstract level before working through it concretely (Anderson, 1980; Newell, 1980). It is thus distinguishable from rehearsal, which refers to working through a task at approximately the same level of concreteness as will eventually be used. As we shall see, composition planning in children starts out more like rehearsal and gradually comes to be carried on at levels more remote from the level of text production. A level of special importance is that of goals and subgoals. A plan for any enterprise is, in fact, likely to consist mostly of subgoals rather than actions -"get lawnmower fixed," " patch tent," and so forth. While all writers presumably have goals of some kind, not all writers are able to construct networks of subgoals leading to their main goals. EXPERT COMPETENCE

Flower and Hayes (1981) show that expert writers very pur­ posefully plan by translating high-level goals into subgoals -for instance, converting the goal of " Make this part interest­ ing" into "So I'll start with a list of jobs they might consider." The result of this active construction of goals, however, is that subgoals tend to pile up, creating a potentially serious burden on working memory, which the skilled writer must accordingly have strategies for handling (Flower & Hayes, 1980b). Skillful planning is often "opportunistic" (Hayes-Roth & Hayes-Roth, 1979). The planner recognizes when the attain­ ment of one subgoal creates the opportunity for attaining another, and so chooses and arranges subgoals accordingly. Opportunities are often discovered in the course of writing rather than in advance, so that planning goes on throughout the composing process (Flower & Hayes, 1980a; Matsuhashi, 1982). The result is that even top-level goals may undergo change and thus have an emergent quality. Compositions often turn out in ways unanticipated by the writer (Murray, 1978), but this is not an indication of planlessness. Rather it is an indication of a very dynamic planning process that keeps adjusting decisions at every level in the light of decisions made at other levels (cf. Hayes-Roth & Hayes-Roth, 1979). STUDENT COMPETENCE

Graves (1975) sees the beginnings of children's composition planning in their use of drawing or acting things out as a preliminary to writing. Graves aptly calls this "rehearsal," however; for although it involves working through the task in a different form, it is done at a level at least as concrete as that of the eventual writing. When fourth graders are asked to plan a composition in advance, what they do is clearly rehearsal. The material they generate either orally or in notes closely resem­ bles the eventual product and may properly be regarded as a first draft rather than a plan (Burtis et al., 1983). Between the sixth and the eighth grade students' protocols and notes start to show evidence of working at a more abstract level than text production. Notes are more telegraphic, and sometimes refer to abstract entities like "my position" (Burtis et al., 1983). This suggests planning at the level of conceptual development or ideas in Beaugrande's model (see Figure 27.2). But there is little evidence of planning at the level of goals.

789

In 24 recently analyzed planning protocols from sixth graders we found only 1 protocol that contained as many as two references to goals; 6 contained one reference, and the remaining 17 contained no references to goals. Thus, even though these young students may have goals and occasionally allude to them, goals do not appear to enter explicitly into the planning process. Protocol data from adult novices show more references to goals, but goals still do not appear to be the highly functional symbolic entities that they are for experts, and there is an absence of formulation of subgoals (Flower & Hayes, 1981; Perl, 1979). The problem of flexible access comes to the fore here as it does with respect to discourse schema knowledge. In order for goals to be operated on in planning it is not enough that they exist as tacit knowledge that constrains behavior; they must be mentally represented in a way that permits them to be consulted, altered, and decomposed. Evidence about goal rep­ resentations is difficult to come by, but indirect evidence may be gleaned from recall protocols. Scardamalia and Paris (1985) found that when school-age writers were asked to state their main goal after writing they tended to restate the assignment or to summarize the main point they were trying to make (a representation at the level of ideas rather than goals), whereas graduate students were likely to give an elaborated goal state­ ment much like those observed by Flower and Hayes in composing protocols. When asked to recall details of text, such as how many times they used a certain word, graduate students sometimes recalled goals as an aid to reconstructing the surface text, whereas young writers worked directly with memory of the surface text. It is frequently pointed , out as a criticism of school writing tasks that they are too meaningless to elicit much goal-directed thought on the part of students (A. Applebee, 1981; Macrorie, 1976). This might help explain why students have not devel­ oped goal-related planning processes, but it does not serve as grounds for dismissing the evidence. Skilled writers, when given a task that they consider unfitting, may complain; but once they set about composing, they start constructing goals (Bereiter & Scardamalia, 1983a). It is important, we believe, to look on goal construction as a part of writing competence and not as an externally determined precondition of good writing. Can young writers carry out goal-related planning more successfully in narrative than in other genres? The fact that children can construct stories with plots would seem to stand as evidence that they can. Bartlett (1982) reported children by the middle school years to make use of foreshadowing statements of the "little did they know" variety, which would seem even more surely to evidence planning ahead to rhetorical targets. Not enough is known about the mental process of story construction to permit us to take these external indications as prima facie evidence, however. Children are known to have trouble with stories that occur in any but normal event order (Stein & Trabasso, 1982). The foreshadowed events in Bartlett's study were pictorially present, so that they did not have to be held in mind. Burtis (1983) compared thinking aloud protocols of young writers planning stories and arguments and found no greater evidence of high-level planning with narratives. One story feature that would require goal-directed planning is suspense, which depends on building slowly toward a crucial

790

MARLENE SCARDAMALIA AND CARL BEREITER

event. Bereiter & Scardamalia (1984b) reported a study in which students in Grades 3 through 7 tried to write suspenseful stories and then rewrote them to make them more suspenseful, after having been exposed to instruction and/or a model suspense story. Only 26 % of the original stories were rated as at least moderately suspenseful, and there was no increase in this percentage on revision. Although children are undoubtedly more successful in some respects with stories than with other literary forms (Hidi & Hildyard, 1983; McCutchen & Perfetti, 1982), evidence does not suggest that it is because they can carry out a higher level of planning in narrative. It is even possible that narrative, because it is so heavily constrained schematically, elicits a more concrete level of intellectual activi­ ty than other forms. Olson, Mack, and Duffy (1981) report that when reading narrative, readers tend to predict concrete con­ tent whereas with other genres they make more abstract predictions of type of content. INTERVENTIONS

An experimental intervention that has been shown to produce increased amounts of goal-directed planning is the ending sentence task, in which students are given a sentence that their composition must lead up to (Tetroe, Bereiter, & Scardamalia, 1981). Whether this task has instructional value is a matter currently under investigation. For the present, research using ending sentence tasks has served to indicate two things: (a) students' difficulties perhaps lie more in goal construction and representation than in strategies for pursuing goals. When goals are articulated and made salient for them, problem-solving processes are brought into play; (b) there is a definite informa­ tion-processing load associated with holding composition sub­ goals in mind. Tetroe, in thesis research summarized by Bereiter & Scardamalia (1984), varied the number of constraints im­ posed by ending sentences. For instance, the phrase "a crowd of angry people" was assumed to contain one more constraint than " a crowd of people." She found that the number of ending­ sentence constraints children could handle was predictable from independent measures of their working memory capacity. Scardamalia (1981) presents analyses of performance on other composition tasks that also show severe limits on the number of constraints or subgoals that young writers can cope with simultaneously. Interventions that have a significant effect on goal formula­ tion itself remain to be demonstrated. Reports of "conferenc­ ing" indicate that teachers may often perform the function of helping children articulate goals and convert global intentions into operative subgoals (Brandt, 1982; Graves, 1978). The goal­ formulating process in these cases, however, would appear to rely on executive processes in the mind of the teacher. The problem is how to induce such processes in the mind of the student. In the next section we report interventions that have been successful in increasing the amount of planning and the amount of reflective thought that goes on in planning.

Reprocessing Reprocessing refers to the notion that whatever is produced from an episode of text processing - be it text, notes, or

thoughts - can be used as input to a further cycle of processing that does not simply add to what was produced before but transforms it. Reprocessing thus spans everything from editing for mistakes to reformulating goals. Revision is a special case of reprocessing, applied to actual text. But as Murray (1978) explains, revision itself can encompass fundamentally different kinds of reprocessing. Murray distinguishes "internal revision," in which writers try "to discover and develop what they have to say" (p. 91), from "external revision," in which text is shaped toward its intended audience. According to Murray, revision begins with the reading of a completed first draft. Composing protocols, however, some­ times show a great deal of reprocessing of both the "internal" and the "external" type going on during the creation of a draft (see, for instance, the transcripts provided by Emig, 1971). Accordingly, the concept of reprocessing is a more suitable theoretical term than revision because it refers to what goes on mentally rather than being tied to differences in surface behav­ ior. Reprocessing is of interest not only because of its effect on the final composition but also because of its effect on the writer's knowledge. Discussions by writers are replete with testimonials to the effect that their understanding of what they are trying to say emerges in the course of writing rather than preceding it (Lowenthal, 1980: Murray, 1978; Odell, 1980). From an infor­ mation-processing viewpoint there is nothing romantic or implausible about these claims. If knowledge consists of propo­ sitions constructed by the knower, then it stands to reason that progressive reconstruction of propositions should generate new knowledge. This " epistemic" function of writing (Bereiter, 1980) has made it seem promising to use writing as a vehicle to promote learning in subject matter courses (Lehr, 1980). How­ ever, such uses of writing presuppose that the writer is engaged in active reprocessing at the level of concepts and central ideas, something that, as we shall see, cannot be taken for granted. EXPERT COMPETENCE

Although protocol data show instances of reconsidering, modi­ fying, and elaborating decisions at different levels, the main evidence about the importance of reprocessing in expert compe­ tence comes from testimonials like those referred to above. Steinberg (1980) enters the cautionary note that not all expert writing is a process of discovery. On the other hand, expert writers can often turn a run-of-the-mill writing chore into one that involves discovery. We have speculated that this is done by an assimilative process of reconstructing the goal so that it incorporates both the pragmatic goal or assignment and other goals of personal importance to the writer (Scardamalia & Bereiter, 1982). Skilled writers differ widely in overt behavior associated with reprocessing - in how many drafts they produce, in how much they fuss over the first draft, and so forth (Della-Piana, 1978; Emig, 1971; Wason, 1980). Hayes and Flower (1980a) indicate, however, that these large overt differences could be due to relatively minor "program" changes at the cognitive process level. An interesting untested hypothesis related to this is that skilled writers ought to be able to shift to a different overt behavior pattern with little difficulty and little effect on the eventual product.

RESEARCH ON WRITTEN COMPOSITION

STUDENT COMPETENCE

Revising by students has received a great deal of research attention, both because of its assumed importance and because students manifestly do so little of it (National Assessment of Educational Progress, 1977; Nold, 1981). Revising, even by college students, tends to be concentrated at the level of proofreading, with little revision of content (Crowley, 1 977; Faigley & Witte, 1981; Sommers, 1980). However, for students at a given level of expertise, more revision is not associated with better writing (Beach, 1979; Bridwell, 1 980; Faigley & Witte, 1981). Although paucity of revision suggests paucity of reprocess­ ing, there is evidence that certain kinds of reprocessing do go on in student writers. Momentary changes between the words students say they are about to write and the words they actually do write have been observed both at elementary (Scardamalia et al., 1982) and university levels (Beaugrande, 1984). Compar­ ing notes taken during planning and eventual texts, Burtis et al. (1983) found that by age 14 no notes were incorporated into text without major change. Even at age 10 about half the notes eventually incorporated into text underwent some major change- elaboration, reordering, combination, or addition. The kind of reprocessing that does not appear to go on in immature writers is that involving goals and main ideas (Perl, 1979; Sommers, 1980). This, of course, is serious, because it is reprocessing at these levels that is most likely to result in new knowledge. As we have noted previously, however, there is evidence that novices do not represent text explicitly at the level of goals and main ideas. Thus, to the extent that reprocessing consists of deliberate operation on such mental representations, novices cannot be expected to reprocess at high levels because they have not constructed the mental representations they need for reprocessing. INTERVENTIONS

The preceding analysis would suggest that direct efforts to boost revision would not be very successful, and this has often been found to be the case (e.g., Beach, 1979; Hansen, 1978). On the other hand, with certain kinds of external support, children even in the primary grades have been reported to make substantive revisions of their compositions (Calkins, 1979; Graves, 1978). Graves is quoted (Brandt, 1982) as saying that the task of the teacher in providing this support is to figure out "what the kid is about, what he has in mind," and help him resee and rethink it until it's clear, not by telling him what to say but by asking questions. (p. 57) This suggests that what the teacher is doing is providing the executive structure for reprocessing, and that accordingly what prevents students from carrying out successful revision by themselves is the lack of such an executive structure. A series of studies by members of the Toronto Writing Research group have explored this possibility through the use of interventions that provide carefully limited forms of execu­ tive support. (See Bereiter & Scardamalia, 1983b, for a general discussion of this research strategy, under the rubric of " simula­ tion by intervention.") The first study used a procedure in which students stopped after every sentence to execute a routine

791

of evaluating, diagnosing, choosing a tactic, and carrying out any revision decided on (Scardamalia & Bereiter, 1983b). The second study used a procedure more oriented to the whole text, which consisted of reading the text, placing markers where problems were noted, and then diagnosing the problems and selecting remedies (Scardamalia & Bereiter, in press-a). In both cases the procedures were simplified by providing a fixed set of evaluations or diagnoses to choose from. The outstanding result in both studies was the high quality of evaluations made by the students, as judged in relation to evaluations made by professionals. The procedures also elicited higher level revisions than those normally observed, a significant majority of which were rated as positive. The second study also showed evidence of transfer. A more extended classroom trial of these kinds of "procedural facilitations" has shown significant improvements in quality of the whole text as well as in quality of individual revisions (Cohen & Scardamalia, 1983). These results suggest that lack of an executive procedure for reprocessing is at least a contributory factor in young writers' problems with revision. Comments by the students themselves strongly supported this interpretation (Scardamalia & Bereiter, in press-b). Procedural facilitation of reprocessing has been extended to planning, through the use of cue cards that suggest reflective moves in the course of thinking aloud- " An impor­ tant point I haven't considered yet is . . . ," and so forth. This intervention has been found to increase rated reflectiveness of essays and to transfer to a significant increase in the number of reflective statements in unassisted planning protocols (Scarda­ malia & Bereiter, in press-a; Scardamalia, Bereiter & Steinbach, 1984). In all of these "simulation by intervention" studies the kind of reprocessing that has been observed has tended to be at what we would consider a local level - reconsidering a particular expression or point rather than one's whole approach or viewpoint. At least in the later studies, the importance of higher level reprocessing was discussed with students and they showed every indication of wanting to do it. That the majority of students had trouble doing so points again to the problem of mental representation - that without mental representations of goals and central ideas, reprocessing them is impossible. It is possible that some very general rise in the level of mental representation of problems is required, something comparable to the attainment of formal operational thought (Inhelder & Piaget, 1958). Even if this should be true, however, it would seem that written composition, carried out with as high a level of planning and reflection as children are capable of, should be an important way of fostering this overall development or representational ability.

Differences in Composing Strategies In the preceding sections a number of seemingly major differ­ ences were noted between the composing processes of experts and immature writers. These differences were found mainly in the kinds of mental representations writers have available to work on and in their ability to exert deliberate control over subprocesses. It is not difficult, on the basis of these differences, to explain why novices do not write as well as experts. What is

792

MARLENE SCARDAMALIA AND CARL BEREITER

difficult to account for is why young people write as well as they do - why, given any reasonable writing assignment, most stu­ dents can set to work immediately and produce a composition that makes sense, that is on topic, and that meets structural requirements of the appropriate genre. The general character of expert composing strategies has already been discussed in the section on the nature of the composing process. Hayes and Flower (1980a) have aptly characterized expert composing as a form of problem solving. It is heuristic search through a space consisting of mental repre­ sentations of possible text. It is, however, as Beaugrande's model (1984) emphasizes, problem solving carried on through parallel action in a number of different problem spaces, each characterized by a different kind or level of mental representa­ tion of possible text. We have proposed that immature writers manage to cope with the difficulties of composition by using a greatly simplified version of what might be considered the generating component of Hayes and Flower's model. A model of this strategy, called "knowledge telling" (Bereiter & Scardamalia, in press) is presented in its crudest form in Figure 27.3. The term "knowl­ edge telling" refers to the strategy's key simplifying feature, which consists of converting all writing tasks into tasks of telling what one knows about a topic. This model, like all other descriptions of cognitive strategies, is a hypothetical construction, not directly observable but test­ able according to its ability to account for known facts. Although this model was based on studies of school-age writers, it appears to fit many college and university students as well. A description by Perl (1979) of the planning activity of novice college writers will be found to correspond closely to the first four steps in the model. And the following description by Crowley (1977) serves as a virtual exegesis of the knowledge­ telling model : The students' model of the composing process, then, moves in a straight line from writing-as-remembering or writing-by-pattern through editing for mechanical errors. The students' writing process is strictly linear, with little or no recursive movement. Synthe­ sis - the composing - is either automatic, a spontaneous flow of memory generated by a writing idea, or generated by the imposition of an organizational pattern - an outline of main and subordinated ideas - which dictates the flow of prose. (p. 167)

The knowledge-telling model offers a way of accounting for the ability of young people to compose even though they have access to only a limited range of mental representations. It accounts for novice writers' occasional eloquence when they strike a topic that spontaneously elicits rich material from memory. Since knowledge telling is a component of an expert composing strategy, it is understandable how, with certain kinds of guidance that provide a more sophisticated executive structure for knowledge telling to work within, young writers can exhibit expert-like planning and reprocessing. But it also explains why the typical piece of unassisted writing by students should strike evaluators as lacking plan and purpose (Larson, 1971) and showing "an innocent lack of consideration for what their readers know and do not know, and for what they are or are not interested in" (Maimon, 1979, p. 364).

1. R EAD ASSI GN MENT.

"Write what you think: On the whole is TV a good or bad influence?"

I

2. LOCATE IDENT I FI ERS OF

a. TOPIC: "TV" "good or bad influence" b. G E N R E : "what you think" = argument

I

3. CONVERT I DENTI F I E R S I NTO MEMORY PROBES.

PROBE: What I think about TV shows I have seen

�

I

4. R ETR I EV E CONTENT F ROM MEMORY, USING PROBE.

CONTENT: I mpression of many bad, few good shows

5.

TEST CONTENT FOR FIT TO TOPIC AND GENRE.

FAI L

PASS 6. TRANS LATE CONTENT INTO LANGUAGE, R ET R I EVING ADDITIONAL CONTENT I F NEEDED FOR COH ERENCE.

LANGUAG E : " I think that TV should only be al lowed to be watched when a good creative or historical show comes on."

Fig. 27.3

A model of text generation accord i n g to the knowl­

edge-te l l i ng strategy.

Knowledge telling is capable of refinement in several ways to incorporate goals and reader-related considerations. Addition­ al test criteria may be inserted at Step 5 (Figure 27.3), so that items are rejected if they fail, for example, to meet a criterion of plausibility or interestingness. In this way young writers can achieve the kind of local-level reprocessing that was reported in the preceding section. There would seem to be a limit to the amount of refinement that can be achieved in this way, how­ ever. Caccamise's findings on idea generation (in press) indicate that as test criteria are added efficiency of memory retrieval declines and the burden on working memory increases. Thus one would expect the knowledge-telling strategy to break down when too many test criteria are added, and this could well be the cause of some student writing blocks (cf. Rose, 1980). Another refinement of the basic strategy is to switch it from producing text as output to producing notes, which may then be subjected to further tests that do not have to be kept in mind during content generation. Protocol studies show students

RESEARCH ON WRITTEN COMPOSITION

making this switch between Grades 4 and 10 (Burtis et al., 1983). None of these refinements, however, turn knowledge telling into a strategy that permits strategic pursuit of rhetorical goals or the reprocessing of goals and main ideas that results in growth of knowledge through writing. Thus, in some funda­ mental and extremely important respects, the knowledge-telling strategy appears to be a developmental dead end. In later sections we will be considering various approaches to instruc­ tion in light of their promise for inducing development beyond the knowledge-telling strategy. Two questions need to be raised about the generalizability of the knowledge-telling model. First, although it seems to de­ scribe the performance of many students, there are documented exceptions in the form of young writers who show expert-like goal formulation and reprocessing. There does not seem to be at present any way of determining whether these cases reflect the large unexplained variations in talent found in all the arts or whether they reflect life experiences that could make most children into accomplished writers. Second, although evidence supporting the knowledge-telling model comes from many sources, the kinds of writing episodes examined are the rela­ tively brief ones characteristic of school writing activities. Would different strategies appear in extended writing? Because extended writing by schoolchildren usually involves consider­ able teacher participation in the composing process (cf. Crow­ hurst, 1979), one cannot take performance in such situations as more than suggestive. One possibility worth investigating is that mental representations of goals and central ideas may take shape very slowly in the minds of immature writers and therefore may emerge only in extended writing. It has frequently been suggested that the kind of strategy represented in the knowledge-telling model is an adaptive response to the peculiar social context of school writing (A. Applebee, 1981; Perl, 1979; Shaughnessy, 1977b). Bereiter and Scardamalia (in press) itemize eleven common educational practices (some of them virtually unalterable) that encourage a knowledge-telling strategy. There would seem to be fertile ground here for combined cognitive and ethnomethodological research.

Instructional Issues Conceivably research on the composing process will lead in time to a coherent "cognitive approach" to writing instruction. Such an approach does not exist at present, however, and it is not necessarily a desirable eventuality if it means yet another approach to be put into competition with existing ones. Un­ doubtedly one hope motivating most people doing cognitive process research on writing is that it will lead to means of teaching expert composing processes to novices. That does not necessarily mean creating a rival method, however. Research on the composing process may also contribute to the teaching of writing by (a) improving existing approaches, (b) suggesting new or more clearly specified instructional goals, and (c) providing a generally more powerful conceptual basis for planning writing instruction. In the sections that follow, actual and potential contributions of these types will be discussed.

793

Contexts for School Writing "Contexts" has become almost as much a buzz word as "process" in current talk about writing instruction. It reflects the fact that writing occurs in many different situations and under a variety of conditions, all of which contribute, for good or ill, to the child's development as a writer. Consequently, an understanding of writing development requires attending to all these contexts in which writing occurs, examining the motives for writing, the task demands, and the social supports that prevail in them (Graves, 1981; Mosenthal, 1983). The main contexts for school writing that receive attention in the educa­ tional literature may be summarized as follows: WRITING AS A SELF-CONTAINED ACTIVITY

Here we refer to contexts in which writing is the focal activity and where it is carried out for its intrinsic satisfactions. Exam­ ples range from elaborately organized writing workshops (Crowhurst, 1979) to the perfunctory "On Friday afternoons we do creative writing." The type of writing commonly involved is " expressive" writing, which James Britton has defined as "writing that assumes an interest in the writer as well as in what he has to say about the world" (1982, p. 156). It spans the functions Florio and Clark (1982) classify as " writing to know oneself and others" and "writing to occupy free time." Among the kinds of expressive writing figuring in school activities are personal journal writing, personal experience narratives, and interpersonal exchanges, such as journal entries that form the basis for correspondence between teacher and student (Staton, 1980). Although data collected with a view to external validity are close to nonexistent, there seems little reason to doubt the abundance of case material indicating that, given a reasonably supportive context, most children will take readily to opportun­ ities for expressive writing (Calkins, 1983; Graves, 1983). There thus seems to be substantial merit in the current enthusiasm for expressive activity approaches to writing, especially as regards developing written language fluency and a sense of the personal satisfactions that can come from writing. Research on composing processes raises two questions about expressive writing activities, however. The first has to do with how much expressive writing calls upon the higher processes involved in expert writing. From a processing standpoint, expressive writing would seem to have the following character­ istics: (a) readily available content, so that heuristic search of memory is not required; (b) little need for intentional framing of the discourse, since content may be adequately presented in the form given to it in memory (Flower, 1979); and (c) little need for goal-related planning, since the goal of the activity is to a large extent realized through the very act of expression. All of this serves to explain why expressive writing should be easier for novices than other kinds of writing, but it also suggests that it may be limited in the kinds and levels of writing abilities it can be expected to foster. Thus there is reason, from an instruction­ al viewpoint, to regard expressive writing as a preliminary or bridge to other kinds of writing (Britton, 1982). The second question concerns the role of the teacher. In currently popular activity approaches to writing, the teacher

794

MARLENE SCARDAMALIA AND CARL BEREITER

serves as .a collaborator rather than an instructor (Dyson & Jensen, 1981). The teacher's collaborative function is likely to include taking over certain executive functions, such as direct­ ing goal formation and memory search, and serving as the source of high-level representations of the student's text. This could be an aid or an impediment to learning, depending on what students are able to internalize from the collaborative process (see subsequent section titled "The Problem of Inter­ na\ization"). WRITING AS A CONTRIBUTORY ACTIVITY

Here we refer to contexts in which writing is not the focal activity but where it is carried out as instrumental to or incidental to some other activity. In his study of writing in secondary schools, A. Applebee (1981) found that most student writing was not done as part of writing instruction but was incidental to other activities, such as taking tests or carrying out learning activities in various subjects. The " Writing Across the Curriculum" movement is dedicated to promoting and improv­ ing such contributory functions of writing (A. Applebee, 1977). Florio and Clark (1982) found that one of the main ways the elementary school teachers they studied worked writing into the curriculum was by devising ways to make it incidental to other activities - for instance, constructing posters as part of a safety campaign. The obvious advantage of contributory contexts is realism: As in most real-life contexts, writing is directed toward some pragmatic end rather than being done for its own sake. The principal limitation is in the range of purposes that writing can serve in school. A. Applebee ( 1981) found that most incidental writing in high school consisted of brief written responses executed for the purpose of displaying knowledge. Thus stu­ dents received little experience with the larger purposes and problems of text production (but, it would appear, received plenty of encouragement for knowledge telling). Although occasionally more red-blooded uses of writing are possible through community action projects, consumerism, and the like, it would seem that the most realistic prospect for improving practical uses of writing in the schools lies in discovering ways that writing can contribute to the existing purposes of subject matter courses. Current instructional re­ search that points in this direction deals with summarization as an aid to learning from texts (Brown, Campione, & Day, 1981; Christophersen, 198 1 ; Taylor, 1982). Lehr (1980) discusses a variety of other possibilities for integrating writing with class­ room instruction. WRITING AS EXERCISE OF SKILL

In this category we place all writing activities carried out for the explicit purpose of skill development. Included is actual compo­ sition writing, when the emphasis is placed on performance rather than on inner satisfaction or practical function, and also such ancillary activities as spelling drills and sentence-combin­ ing exercises, No sharp line divides writing as an activity from writing as exercise. Clark and Florio ( 1981) analyze an instance of something starting out as an informal activity (diary writing) and gradually becoming formalized into an exercise, and there are no doubt also instances of an exercise capturing student

interest and evolving into an intrinsically motivated activity. Nevertheless it seems important to distinguish the exercise context from other school writing contexts because of its distinctive uses and problems. Writing exercises account for the bulk of efforts to teach writing (A. Applebee, 1981). In recent years, however, exercise has come increasingly under attack as a fundamentally un­ sound way to teach writing. The grounds are that it is writing carried on without the normal sense of an audience and of communicative intent, with the result that the student fails to acquire a genuine sense of what it is to be a writer (Moffett, 1968; Smith, 1982) and develops a bland, depersonalized style that Macrorie (1976) calls " Engfish." It might also be specu­ lated that the exercise context encourages development of low processing-demand composing strategies such as knowledge telling. Not all writing exercise consists of composition assignments, however. Sentence-combining practice, which consists of as­ sembling simple sentences into complex ones, under the gui­ dance of rules or cues that ensure their well-formedness (Mellon, 1969; O'Hare, 1973), has yielded a preponderance of results showing gains in "syntactic maturity," with occasional indications of benefits for other aspects of writing ability and even for reading comprehension (see Stotsky, 1975, and White & Karl, 1980, for reviews). An exercise context that has been little used in writing is the game. Games provide a context for purposeful, rule-governed activity. Design of nontrivial games for writing depends on having a theory of the underlying processes, however, since it is these processes and not the objective conditions of writing that games have the potential to generate. It is perhaps for this reason that writing activities posing game-like challenges to rhetorical skill have only recently begun to appear (Scardama­ lia, Bereiter, & Fillion, 1981). WRITING INSTRUCTION

Compared to mathematics, for instance, where there is a great deal of explaining, demonstrating, and teaching of rules, very little direct instruction goes on in the teaching of writing. What there is is largely confined to teaching conventions - punctua­ tion rules, format conventions, and the like, and even then teachers appear to rely mainly on practice and correction (A. Applebee, 1981). Composition textbooks attempt to provide explicit principles, but the principles themselves are often of dubious validity and there are indications that students tend to take them too literally and to overextend them (Beaugrande, 1984; Harris, 1979; Rose, 1980, 1981). All of this has suggested to some that writing belongs to the category of things that can be "learned but not taught" (Elbow, 1973). The evidence that students tend to overapply textbook principles suggests, on the contrary, that expl!cit instru�tio_n can have a significant effect on how students wnte. The tnck 1s to make it have a positive rather than a detrimental effect. A major challenge for modern writing research is to disco�er teachable principles that are valid and that students can readily convert into procedural knowledge (Anderson, 1982). Progress in this direction will be discussed later under the headings of "strategy instruction" and " product-oriented instruction."

RESEARCH ON WRITTEN COMPOSITION

Instructional Goals Related to the Composing Process Traditionally, the goals of writing instruction have been ex­ pressed in terms of the qualities of good writing - clarity, organization, and the like - or in terms of abilities and apti­ tudes. One of the prospects of research on the composing process is that it might lead to the definition of instructional goals that can more readily be evaluated as to their attainability and more readily translated into instructional subgoals. From the earlier discussion of expert and novice strategies, three different instructional goals may be identified pertaining to development of composing strategies : (a) refining the knowl­ edge-telling strategy, (b) inducing goal-directed strategies, and (c) fostering the development of reflective planning. These goals, it will be recognized, are distinct. Attaining one goal is not necessarily a step toward attaining another. REFINING THE KNOWLEDGE-TELLING STRATEGY

The knowledge-telling strategy, as we have noted, is capable of considerable refinement by adding test criteria that enable the writer to reject inappropriate content or language and by adopting the practice of generating and arranging notes in advance of writing. In this way knowledge tellers can produce compositions meeting ordinary standards of style and content. Most traditional writing instruction seems to be aimed at this objective. That is, it is not concerned with altering the basic way students compose but rather with bringing their performance under the control of acceptable literary standards. Two serious limitations of the knowledge-telling strategy were pointed out, however : (a) Writing carried out through this strategy is not a very powerful tool either for achieving communicative aims or for developing one's understanding. (b) Refining knowledge telling through the addition of test criteria means decreased efficiency and increased burden on working memory as the number of test criteria increases. These practical limitations might mean that most people can never become very successful writers as long as they follow a knowledge-telling strategy. INDUCING GOAL-DIRECTED STRATEGIES

The principal weakness of a knowledge-telling strategy is that it does not contain provisions for explicit formulation and pursuit of goals, thus severely limiting · the writer's ability to realize intentions through writing. A common theme in calls for improving the conditions of writing in schools is that writing should become a more purposeful activity (A. Applebee, 198 1 ; Macrorie, 1976; Muller, 1 967). A reasonable assumption sup­ porting these views is that if students are writing in situations where they are trying to accomplish something definite through their writing and where they can find out whether they achieved it, they will naturally acquire goal-directed strategies. That, surely, is how most cognitive strategies are learned, and it serves to explain some of the highly effective writing skills that Odell and Goswami (1982) have observed in business people. A major difficulty, as noted in previous discussion, is to find school contexts that support writing goals other than knowl­ edge display and self-expression. In the absence of conditions that normally motivate writing to convince, to inform, and so

795

forth, educators may need to design games or problem tasks to create the necessary goal conditions. There is also, however, the problem that an overemphasis on pragmatic goals might lead to overly pragmatic composing strategies, oriented toward fixed goals and lacking the ability to harmonize personal and externally defined objectives (Scardamalia & Bereiter, 1982). FOSTERING REFLECTIVE PLANNING

Reflective planning refers to the reprocessing or progressive shaping of goals and ideas during composition (Scardamalia & Bereiter, in press-a). In a model proposed by Scardamalia et al. (1984), reflective processes in writing are attributed to interac­ tion between two problem spaces - a content space, in which problems of knowledge and belief are dealt with, and a rhetori­ cal space, in which problems of achieving goals of the compo­ sition are dealt with. An instructional experiment reported by Scardamalia et al. (1984) gave evidence that the two-way interaction between problem spaces could be fostered in ele­ mentary school children. Reflective planning is an instructional goal of such obvious importance that it warrants continued research into its teachability.

New Approaches to Writing Instruction A search through recent journals would indicate that methods of writing instruction receiving the most attention include conferencing (Graves, 1978), freewriting (Elbow, 1973), rhetoric of invention (Young, 1 976), and sentence combining (Mellon, 1969). In the present discussion, however, these will be treated as special cases of more general types of innovative approach to writing instruction. In the past, methods of writing instruction have tended to grow up piecemeal, connected to one another, if at all, only by broad philosophical premises. Research on composing processes has advanced far enough, however, that it is now possible to identify certain basic ways of trying to influence the composing process and thus to consider particular methods in terms of how they attempt to bring about such changes. The four basic approaches that will be considered here are strategy instruction (the most direct approach), procedural facilitation (a generic label for a variety of ways of helping students adopt more sophisticated composing strategies by providing external supports), product-oriented instruction (in­ struction that attempts to promote strategy development by providing students with clearer knowledge of goals to strive for in the written product), and inquiry learning (learning through guided experimentation and exploration).

Strategy Instruction Textbooks have begun to appear at the college level that present writing to students as a cognitive process and provide explicit instruction in cognitive strategies (Beaugrande, 1982c; Flower, 198 1). There are informal reports that college students respond well to strategy instruction but there does not yet appear to be evidence available about the effect of such instruction on students' actual composing strategies. Before­ and-after protocol analyses are obviously called for.

796

MARLENE SCARDAMALIA AND CARL BEREITER

Such studies, carried out by the Toronto Writing Research Group with younger students, suggest problems in trying to use direct teaching of composing strategies with school-age writers. In one study students were instructed briefly in what adults think about when planning a composition - audience, goals, what they know about the subject, organization, and problems. Some students were taught these as principles, some saw them demonstrated by a videotaped model thinking aloud, and some received both kinds of instruction. Ten-year-olds could not reliably identify what kind of planning the model was doing and showed no effect from any treatment; 14-year-olds could relia­ bly identify what the model was doing and showed signs of trying to imitate the model, but only at a very superficial level (Burtis et al., 1983). In a more extended instructional study with 12-year-olds (Scardamalia et al., 1984), before-and-after proto­ col analyses showed a significant increase in the number of planning statements judged to indicate reflective thought, but there was no systematic change in the distribution of kinds of planning carried out. The instruction provided in this latter study, moreover, included procedural facilitation as well as direct instruction, so that the results cannot be taken as a demonstration of effects of direct strategy instruction. The only other before-and-after research we are aware of on strategy instruction also involves procedural facilitation. This is research on teaching students heuristics of discovery (Odell, 1974; Young, 1976). Students were taught principles and proce­ dures for developing their thoughts on subjects in advance of writing, but were also provided with formal devices for organiz­ ing the discovery process. These studies, conducted with uni­ versity students, produced results similar to the study with 12-year-olds cited above, with gains being observed in the amount of reflective thought revealed in compositions. The basic problem in strategy instruction is how to convert declarative knowledge (knowledge about the composing pro­ cess) into procedural knowledge or know-how. In his theory of skill acquisition, Anderson (1982) proposes that this conversion depends on the learner's having already available a general­ purpose procedure that makes it possible to convert the declar­ ative propositions into operative subgoals. In the case of students who already have a goal-directed composing strategy, it would accordingly be expected that they could benefit from suggestions interpretable as subgoals - for instance, the sugges­ tion in Flower (1981) to define a "projected self' to be put across in the composition. But for students whose existing procedures were limited to knowledge telling, such suggestions would be expected to be ineffective because the students would have no way to convert them to subgoals that could be operative within the structure of their composing strategies.

Procedural Facilitation An alternative to direct teaching of composing strategies is to provide students with help in carrying out more sophisticated composing processes. In a sense, the traditional practice of teacher comments on student compositions has this aim, inso­ far as the teacher's comments are expected to help students take a critical view of their work, consider more sophisticated alternatives, and so forth. Such help may be called "substan-

tive" facilitation (Bereiter & Scardamalia, 1982). In effect, it involves the teacher as a collaborator. In recent times this collaborative role has come to be explicitly recognized as distinct from a direct instructional role (Dyson & Jensen, 1981). Even if the collaboration is limited to subtle probes and hints (e.g., Calkins, 1979), it still constitutes substantive facilitation to the extent that it involves the teacher's responding to what the student has said or intends to say. In contrast to substantive facilitation is "procedural" facilita­ tion (Bereiter & Scardamalia, 1982; Scardamalia & Bereiter, in press-b ). In procedural facilitation help is of a nonspecific sort, related to the student's cognitive processes, but not responsive to the actual substance of what the student is thinking or writing. Help consists of supports intended to enable students to carry out more complex composing processes by themselves. Procedural facilitation may be quite passive, such as that provided by a word processor, which simply makes it easier to execute certain mechanical parts of the composing process, thus presumably freeing the writer to devote more mental resources to higher level parts of the process (Daiute, 1982). A more active procedural facilitation is the provision of cue cards to prompt reflection during planning (Scardamalia & Bereiter, in press-a). Researchers in the rhetoric of invention have also made use of procedural facilitations, often in the form of matrices or other standard forms to be used for aiding heuristic search of memory and information sources (Young, 1976). Substantive and procedural facilitation are compatible and can occur in combination, but they present somewhat different instructional problems. With substantive facilitation the prob­ lem is to provide help that promotes learning rather than making learning unnecessary. In the light of modern concep­ tions of the composing process, which see it as going on at multiple levels and as extending from initial intentions to finished product, this problem appears as considerably more complex than it does when one views composition as restricted to what one does with pen in hand. Problems with procedural facilitation have to do with the unpredictable uses students make of extrinsic inputs. Examples have been noted in earlier sections, such as the tendency to exploit all inputs as cues to aid memory search. The problems with both kinds of facilitation may, however, be subsumed under the general problem of internalization, which will be discussed in a later section. Procedural facilitation appears to hold promise as a way of influencing the direction of attention during composing. We have seen evidence of transfer of facilitated procedures into unassisted composition in such areas as content generation (Bereiter & Scardamalia, 1982), attention to structural elements (Paris, 1980), critical evaluation of content, and diagnosis of text difficulties (study by Cattani, Scardamalia & Bereiter, reported in Scardamalia & Bereiter, in press-a). Such use of procedural facilitation presupposes, however, that the relevant mental representations are there to be attended to. It can do little good to direct students' attention to their goals for a composition if they are not consciously able to represent such goals. There is substantial evidence that procedural facilitation can increase the level of sophistication of the composing processes students carry out within the limits imposed by the kinds of mental representations they construct. It also seems reasonable to expect that it could foster development of higher

RESEARCH ON WRITTEN COMPOSITION

level mental representations of text; but this remains to be systematically investigated. CONFERENCING

Conferencing, which refers to any sort of one-to-one discussion between teacher and student during the evolution of a composi­ tion, can include both substantive and procedural facilitation. The teacher who says, "What do you think your next step should be in planning this essay?" is not responding substan­ tively to the student's work but is simply helping direct atten­ tion to a procedural decision. A combination of procedural facilitation of planning and substantive facilitation of mental representations of goals and central ideas could provide sup­ portive conditions for students to exercise more sophisticated composing strategies. The main drawback of conferencing currently seems to be the tendency of many teachers to take too much of the initiative, casting the student in the role of "receiver of information" (Jacob, 1982). In more expert hands, there is continual atten­ tion to the students' internalizing the planning, evaluating, and other processes that go on in the writing conference (Calkins, 1983). COMPUTER FACILITATION OF COMPOSING

Innovative work on the application of computers to writing is at present mainly focused on the "tool" use of computers - as word processors, automated editors, dictionaries, and encyclo­ pedias, and as components in communication networks (Anan­ dam, Eisel, & Kotler, 1980; Bruce & Rubin, in press; Frase, Macdonald, Gingrich, Keenan, & Collymore, 1981 ; Levin, Boruta, & Vasconcellos, 1983. For an overview, see Daiute (1983). Computers appear to have potential for fostering higher­ order abilities in composing, but software design is currently more determined by what computers can do than by what students need. Computers make it easy to move, delete, and add to text, thereby presumably fostering revision. But executing these external parts of the revision process is orders of magni­ tude easier than carrying out the nonobservable goal-construct­ ing, monitoring, and problem-solving activities that are part of the mature revising process. Although there is much talk at educational computing conferences about enhanced tools that will facilitate mental processes in writing, the prototype soft­ ware that has actually appeared consists mainly of additional information-manipulating aids for the writer who already has highly sophisticated composing strategies. Computers can, for the novice writer, take on a variety of procedural facilitative roles such as cueing, short-term memory support, and segmenting and sequencing parts of the compos­ ing task. Burns and Culp (1980) were able to computerize invention heuristics through question-asking programs. A "help" program that provided procedural prompts was enthu­ siastically received by junior high school students, although it had no observable short-term effect on their composing (Woodruff et al., 1 981). Whereas word processors eliminate only certain mechanical burdens of writing, it is possible for instructional purposes to have the computer selectively take over other parts of the composing process, so as to direct the

797

learner's efforts to particular problems. Story Maker, for in­ stance, is a program that supplies sentences, leaving the user to determine events (Rubin, 1980a), and EXPLORE leaves the user to make structural and stylistic decisions, the computer doing the rest of the work of producing an essay (Woodruff, Scardamalia, & Bereiter, 1982). To be useful as procedural facilitations, computer software must induce mental processes that are not only of a higher level than those students normally employ but that can also have an influence on students' normal composing processes. There is evidence that the Writer's Workbench (Frase et al., 1981) may meet this requirement. It contains tools for automated analysis of style in texts. Kiefer and Smith (1983) found that university students who used these tools showed transfer to stylistic revision under unassisted writing conditions, THE PROBLEM OF INTERNALIZATION

Procedural facilitation of no matter what variety is expected to have its instructional effect by providing external support for a procedure that the student will eventually run autonomously. Vygotsky (1978) called this movement from external depen­ dence to autonomy "internalization." The term is apt in its suggestion that the externally supported process must corre­ spond in a fundamental way to the intended internal process. Although it is not easily answered, the question of internali­ zability needs to be asked about all the kinds of facilitation we have been considering, substantive as well as procedural. Heur­ istic invention procedures, for instance, are often aided by matrix forms that the student fills in as a way of elaborating thoughts in an organized way. Although such devices may lead students to generate the kinds of ideas experts generate, it does not seem very plausible that what goes on in the mind of the expert bears much correspondence to the matrix format. Is the procedure, therefore, likely to be one that students will abandon as soon as external supports are removed? On first thought, conferencing would seem to be well designed for internaliza­ tion ; the thinking, carried out jointly at first, comes in time to be carried out in the mind of the student. But the form of the conference is dialogue, and there is no indication from research to suggest that the mature composing process has the form of an internal dialogue. A more readily internalizable form might be the "assisted monologue" (Scardamalia et al., 1984) where the talking is primarily done by the student, with the teacher inserting prompts rather than conversational turns. Sentence combining is perhaps the most surprising instruc­ tional intervention from the standpoint of internalization. Stu­ dents are provided with short sentences that often do not make sense in the order presented, but which, when combined accord­ ing to rule, deliver an intelligible message (Mellon, 1969). On the face of it, such a procedure bears no relation to how people normally compose sentences, since normally we have some prior idea of what we are trying to say. Yet the empirical evidence indicates that a syntactical process of some generality is internalized from sentence-combining practice (O'Hare, 1973). One can only surmise that, in spite of the evident dissimilarities, the contrived procedures used in sentence com­ bining correspond at some deep level to the cognitive process by which a complex sentence is synthesized (cf. Bock, 1982). At

798

MARLENE SCARDAMALIA AND CARL BEREITER

any rate, the sentence-combining story makes it clear that our current understanding of cognitive processes is not sufficient to enable us to answer questions of internalization by deductive argument. Serious research is needed to determine what stu­ dents internalize from what teachers have helped or induced them to do.

Product-Oriented Instruction Popular discourse on writing instruction has set up a somewhat artificial contrast between "product" and "process" ap­ proaches, the former concentrating on evaluating and upgrad­ ing the quality of what students write, the latter concentrating in one way or other on the composing process. As Perkins (in press) has argued, however, one of the main ways that cognitive processes develop in real life is through striving to produce an adequate product; and the more realistically learners are aware of what product characteristics they should be striving for and of how successful they are in achieving them, the more likely it is that the attendant cognitive processes will develop. Traditionally it has been the province of rhetoric to teach, through rules and examples, the characteristics of good written products- what constitutes a tightly knit paragraph, a con­ vincing argument, and so forth. Two kinds of current cognitive research show promise of contributing to this tradition. Both kinds, interestingly, involve connections between reading and writing. One is research on the effects of various literary devices on readers and the other is research on how students learn or fail to learn from literary models. Included in the first category are (a) studies that show how different organizations of content in text and different signals of that organization affect ability to recall the text (e.g., Meyer, 1977), (b) studies that show how use of a narrative format improves understanding of regulations (Flower et al., 1980), and (c) studies that show how manipulations of event order and narrative viewpoint generate effects such as surprise or sus­ pense in readers (Brewer, 1980; Brewer & Lichtenstein, 1981). Also of interest are trade-offs among effects, as when devices used to make a text more readable reduce its rememberability (Collins & Gentner, 1980). An obvious gain from research of this kind is that it should provide students with a basis for making more goal-related choices among literary alternatives. Furthermore, in order to study empirically the effects of literary devices, researchers are forced to define these devices explicitly (so that coders can use the definitions, for instance). This should have the result of producing descriptions of literary devices and strategies that are more accessible to students than many of the descriptions derived from traditional literary scholarship. We imagine, for instance, that Brewer and Lichtenstein's account of how sus­ pense is produced in narratives could be more readily adapted to practice by novices than more purely literary analyses. It is widely recognized that much of what students learn about writing must come from exposure to examples (Smith, 1982). The knowledge obtained from such reading is, of course, "product" knowledge; reading typically furnishes no clue to the process by which the literary work was brought into existence. At present it is largely a matter of assumption that obtaining knowledge of writing through reading requires some special

directing of attentfon, distinct from that required for ordinary reading comprehension. In current doctoral research at the Ontario Institute for Studies in Education, however, Nangar Sukumar has found measures of language awareness in reading to account for an additional 10 % of the variance in high school students' rated writing ability, over and above the 32 % ac­ counted for by reading comprehension and general intelligence test scores. Research on how people extract literary knowledge from examples has only recently begun, and only the most tentative results can be suggested. Elizabeth Church (Church & Bereiter, 1983) had high school students think aloud while reading different translations of a passage from the Divine Comedy. When told to pay attention to the style of the translation, students were able to do so for a short time, but then tended to lose hold of the instruction and lapse into attending only to content. This suggests that attending to style was not an orientation that students could simply switch on, but that it required conscious holding in mind of a constraint, and thus a risk of cognitive overload. When students were not told what to attend to, they attended only to content on first reading, but on subsequent readings a few began to attend to literary features. Most, once they had processed the content to their satisfaction, found nothing further to attend to. In one part of the study, students were asked to compare two translations and then to convert a passage from one translation into the style of the other. There was substantial correspon­ dence between the kinds of features noticed in reading and the kinds captured in imitation, which suggests that the thinking aloud procedure was picking up reading events that were relevant to writing. This study, and others conducted more recently by the present authors, demonstrate that students of all levels from Grade J up can extract some knowledge of literary features from examining model texts. However, the knowledge they most readily pick up is knowledge that could have easily been communicated through explicit statement. It remains for more extended investigations to find out what is required for students to pick up the more elusive qualities of style and form that teachers traditionally rely on literary models to communi­ cate.

Inquiry learning Writing instruction has no tradition of learning through guided inquiry and experimentation the way mathematics and science instruction do. However the inquiry approach to writing de­ scribed by Hillocks (1982) comes close. This approach involves students in work with a data source, such as concrete objects or texts. Problems of interpreting the data are handled concur­ rently with writing problems, such as how to report observa­ tions and defend conclusions about the data. Hillocks reports studies by himself and others that show strong learning effects. This inquiry approach has obvious application to the use of writing in content areas. Also, the approach may foster reflec­ tive processes by encouraging interaction between content and rhetorical problem spaces (cf. Scardamalia et al., 1984). A quite different sort of inquiry learning is represented by children's invention of their own writing systems. Descriptive accounts indicate that, given encouragement, children who do

RESEARCH ON WRITTEN COMPOSITION

not yet know how to read will begin producing documents containing letters or symbols that they interpret meaningfully. Through efforts to communicate with adults and peers, they gradually adopt consistent symbols, linear arrangements, ways of showing divisions between words, and eventually conven­ tions of spelling, punctuation, and the like (Bissex, 1 980; Clay, 1975). Educational effects of encouraging invented writing are just beginning to be tested. Obviously it enables children to start writing independently at an earlier age, and this benefit could be substantial. If there is some gain from the discovery process itself, it would seem likely to be in the form of metalinguistic awareness. Children should get an idea of writing as a commun­ icative medium, and understanding that writing conventions are conventions, and an appreciation of what such conventions are for (Graves, 1 983). Interestingly, a major payoff might come in reading rather than writing. According to the case studies of Read (1971) and Bissex (1980), children trying to work out their own spelling systems eventually settle on the idea of grapheme -phoneme correspondence. Thus, by inventing spelling they also invent phonics, which provides them with a tool for independent reading. Another possibility for learning by discovery is the compos­ ing process itself. Techniques such as thinking aloud and procedural facilitation make interpretable data on the compos­ ing process available not only to researchers but also to students. In a paper titled "Child as Coinvestigator" (Scarda­ malia & Bereiter, 1 983a), we detail a large number of proce­ dures that have proved workable for involving students in explorations of their cognitive processes during composition. We have favorable reports of older students' designing and conducting composing process studies of their own. Even students who dislike writing have been found to be interested in investigating their composing processes.

Prospect We sense a great deal of opt1m1sm among language arts educators that the conditions of learning to write will soon improve dramatically. Computer euphoria is one source of this optimism, but a better grounded source is the reports of what teachers have been able to do by way of creating a social climate supportive of written expression (e.g., Calkins, 1 980; Staton, 1980). One teacher we know has even managed to involve families in what she calls a "culture of writing" by keeping up interactive journals with the parents, the children serving as intermediaries and sometimes as interpreters, since many of the parents are not literate in English. There is no doubt that if a "culture of writing" can be established, learning to write should be greatly facilitated. The same cognitive changes would have to occur as have been discussed in this chapter, but the likelihood would become greater that they could occur through informal learning. It must be recognized, however, that what has actually been observed so far has been the often heroic efforts of a relative handful of dedicated teachers, and not a massive cultural shift supportive of literacy. Even a moderately optimistic forecast would have to allow that the teaching of writing will probably continue to take place

799

in a relatively uncongenial cultural environment - so that there is little prospect, for instance, that children will learn to write as naturally as they learn to talk (even if there is no biological impediment to their doing so). Accordingly, there is likely to be a continuing need for instructional solutions to writing prob­ lems. The hopeful prospect offered by research on composing processes is that certain processes that are not currently teach­ able may become so. Thus there is ground for optimism that the job of the writing teacher may take on a more progressive character and be less tied to the endless cycle of practice and response.

REFERENCES Anandam, K., Eisel, E., & Kotler, L. (1980). Effectiveness of a com­ puter-based feedback system for writing. Journal ofComputer-Based Instruction, 6(4), 125- 1 33. Anderson, J. R. (1980). Cognitive psychology and its implications. San Francisco, W. H. Freeman. Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369-406. Applebee, A. N. (1977). Eric/RCS Report: Writing across the curricu­ lum: The London projects. English Journal, 66(9), 81-85. Applebee, A. N. (1981). Writing in the secondary school: English and the content areas. Urbana, IL: National Council of Teachers of English. Applebee, R. K. (1966). National study of High School English pro­ grams: A record of English teaching today. English Journal, 55, 273-281. Augustine, D. (1981), Geometries and words: Linguistics and philoso­ phy: A model of the composing process. College English, 43(3), 221 -231 . Bartlett, E . (1982). Children's difficulties in establishing consistent voice and space/time dimensions in narrative text. Paper presented at the meeting of the American Educational Research Association, New York. Beach, R. (1979). The effects of between-draft teacher evaluation versus student self-evaluation on high school students' revising of rough drafts. Research on the Teaching of English, 13, 1 1 1- 1 19. Beaugrande, R. de. (1982a). Psychology and composition: Past, pre­ sent, future. In M. Nystrand (Ed.), What writers know: The lan­ guage, process, and structure of wrillen discourse (pp. 2 1 1-267). New York: Academic Press. Beaugrande, R. de. (1982b). The story of grammars and the grammar of stories. Journal of Pragmatics, 6, 383-422. Beaugrande, R. de. (1982c). Writing step by step. Gainsville: University of Florida, Office of Instructional Resources. Beaugrande, R. de. (1984). Text production: Toward a science of composition. Norwood, NJ : Ablex. Bereiter, C. (1980). Development in writing. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 73-93). Hills­ dale, NJ: Lawrence Erlbaum. Bereiter, C., & Scardamalia, M. (1982). From conversation to composi­ tion: The role of instruction in a developmental process. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 2, pp. 1 -64). Hillsdale, NJ: Lawrence Erlbaum. Bereiter, C., & Scardamalia, M. (1983a). Does learning to write have to be so difficult? In A. Freedman, I. Pringle, & J. Yalden (Eds.), Learning to write: First language, second language (pp. 20-33). London: Longman. Bereiter, C., & Scardamalia, M. (1983b). Levels of inquiry in writing research. In P. Mosenthal, S. Walmsley, & L. Tamor (Eds.), Research on writing : Principles and me/hods (pp. 3-25). New York: Longman. Bereiter, C., & Scardamalia, M. (1984a). Information processing de­ mand of text composition. In H. Mandi, N. Stein, & T. Trabasso (Eds.), Learning and comprehension of text (pp. 407-428) Hillsdale, NJ: Lawrence Erlbaum. Bereiter, C., & Scardamalia, M. (1984b). Learning about writing from reading. Wrillen Communication, 1(2), 163-188.

800

MARLENE SCARDAMALIA AND CARL BEREITER

Bereiter, C., & Scardamalia, M. (in press). Cognitive coping strategies and the problem of "inert knowledge." In S. S. Chipman, J. W. Segal, & R. Glaser (Eds.), Thinking and learning skills: Current research and open questions (Vol. 2). Hillsdale, NJ: Lawrence Erlbaum. Bissex, G. L. ( 1980). GYNS A T WRK: A child learns to write and read. Cambridge, MA: Harvard University Press. Black, J. B. (1982). Psycholinguistic processes in writing. In S. Rosen­ berg (Ed.), Handbook of applied psycholinguistics: Major thrust of research and theory (pp. 199-216). Hillsdale, NJ: Lawrence Erl­ baum. Black, J. B., Galambos, J. A., & Reiser, B. J. (1984). Coordinating discovery and verification research. In D. Kieras & M. Just (Eds.), New methods in reading comprehension. Hillsdale, NJ : Lawrence Erlbaum. Black, J. B. & Wilensky, R. (1979). An evaluation of story grammars. Cognitive Science, 3, 2 1 3-230. Bock, J. K. (1982). Toward a cognitive psychology of syntax: Informa­ tion processing contributions to sentence formulation. Psychologi­ cal Review, 89, 1 -47. Boiarsky, C. (1982). Prewriting is the essence of writing. English Journal, 71(4), 44-47. Botvin, G. J., & Sutton-Smith, B. (1977). The development of structural complexity in children's fantasy narratives. Developmental Psychol­ ogy, 13, 377-388. Bracewell, R. J. (1980). Writing as a cognitive activity. Visible Language, 14, 400-422. Bracewell, R. J. (1983). Investigating the control of writing skills. In P. Mosenthal, S. Walmsley, & L Tamor (Eds.), Research on writing : Principles and methods. (pp. 177-203). New York : Longman. Bracewell, R. J., Frederiksen, C. H., & Frederiksen, J. F. (1982). Cognitive processes in composing and comprehending discourse. Educational Psychologist, 17(3), 146- 164. Brandt, A. (1982). Writing readiness. Psychology Today, 16(3), 55- 59. Brewer, W. F. (1980). Literary theory, rhetoric, and stylistics: Implica­ tions for psychology. In R. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in reading comprehension (pp. 221-239). Hillsdale, NJ : Lawrence Erlbaum. Brewer, W. F., & Lichtenstein, E. H. (1981). Event schemas, story schemas, and story grammars. In A. D. Baddley & J. Long (Eds.), Attention and performance (Vol. 9, pp. 363-379). Hillsdale, NJ: Lawrence Erlbaum. Bridwell, L. S. ( 1980). Revising strategies in twelfth grade students: Transactional writing. Research in the Teaching of English, 14, 107 - 122. Britton, J. (1982). Spectator role and the beginnings of writing. In M. Nystrand (Ed.), What writers know: The language, process, and structure of written discourse (pp. 149- 169). New York: Academic Press, 149- 169. Britton, J., Burgess, T., Martin, N., McLeod, A., & Rosen, H. ( 1975). The development of writing abilities (1 1-18). London : Macmillan Education Ltd. Brown, A. L., Campione, J. C., & Dc:y, J. D. (1981). Learning to learn: On training students to learn fror,; texts. Educational Researcher, 10(2), 14-21. Bruce, B . C., & Rubin, A. D. (in press). What we're learning with QUILL. In M. L. Kami! & R. C. Leslie (Eds.), Perspectives on computers and instruction for reading and writing. Rochester, NY: National Reading Conference. Burns, H. L., & Culp, G. H. (1980). Stimulating invention in English composition through computer-assisted instruction. Educational Technology, 20(8), 5-10. Burtis, P. J. (1983). Planning in narrative and argument writing. Paper presented at the meeting of the American Educational Research Association, Montreal. Burtis, P. J., Bereiter, C., Scardamalia, M., & Tetroe, J. (1983). The development of planning in writing. In G. Wells & B. M. Kroll (Eds.), Explorations in the development of writing (pp. 1 53- 174). Chichester, England: John Wiley. Caccamise, D. J. (in press). Idea generation in writing. In A. Matsuhashi (Ed.), Writing in real time: Modelling production processes. New York: Longman.

Calkins, L. M. ( 1979). Andrea learns to make writing hard. Language Arts, 56, 569-576. Calkins, L. M. ( 1980). Research update: When children want to punctuate: Basic skills belong in context. Language Arts, 57(5), 567-573. Calkins, L. M. ( 1983). Lessonsfrom a child: On the teaching and learning of writing. Exeter, NH: Heineman Educational Books. Chafe, W. L. ( 1977). Creativity in verbalization and its implications for the nature of stored knowledge. In R. 0. Freedle (Ed.), Discourse production and comprehension (pp. 41 -55). Norwood, NJ: Ablex. Christophersen, S. L. ( 1 98 1 ). Effects of knowledge of semantic roles on summarizing written prose. Contemporary Educational Psychology, 6, 59-65. Church, E., & Bereiter, C. ( 1983). Reading for style. Language Arts, 60(4), 470-476. Clark, C. M., & Florio, S. (1981, August). Diary time : The life history of an occasion for writing (Research Series No. 106). East Lansing: Michigan State University, Institute for Research on Teaching. Resources in Education, 1982, 17(8), 137- 1 38. ERIC number ED 214 648. Clay, M. M. (1975). What did I write: Beginning writing behaviour. Auckland, New Zealand: Heinemann Educational Books. Cohen, E., & Scardamalia, M. (1 983). The effects of instructional intervention in the revision of essays by grade six children. Paper presented at the meeting of the American Educational Research Association, Montreal. Collins, A., & Gentner, D. (1980). A framework for a cognitive theory of writing. In L. W. Gregg & E. Steinberg (Eds.), Cognitive processes in writing : An interdisciplinary approach (pp. 5 1 -72). Hillsdale, NJ: Lawrence Erlbaum. Cooper, C. R., & Odell, L. (Eds.). (1977). Evaluating writing : Describing, measuring, judging. Urbana, IL: National Council of Teachers of English. Corbett, E. (1971). Classical rhetoric/or the modern student. New York : Oxford University Press. Crowhurst, M. (1979). The writing workshop: An experiment in peer response to writing. Language Arts, 56(7), 757-762. Crowley, S. ( 1977). Components of the composing process. College Composition and Communication, 28(2), 1 66- 169. Daiute, C. (1982, March/April). Word processing. Can it make even good writers better? Electronic Learning, pp. 29-3 1 . Daiute, C. A . ( 1983). The computer as stylus and audience. College Composition and Communication, 34, 1 34- 145. Daly, J. A., & Miller, M. D. (1975). Further studies on writing apprehension: SAT scores, success expectations, willingness to take advanced courses, and sex differences. Research in the Teaching of English, 9, 242-249. DeFord, D. E. (1980). Young children and their writing. Theory Into Practice, 19(3), 1 57-162. Della-Piana, G. M. ( 1978). Research strategies for the study of revision processes in writing poetry. In C. R. Cooper & L. Odell (Eds.), Research on composing: Points of departure (pp. 105-134). Urbana, IL: National Council of Teachers of English. Diederich, P. B. (1964). The measurement of skill in writing. School Review, 54, 584-592. Dixon, R. (1976). Morphographic spelling (Parts 1 and 2). Chicago : Science Research Associates. Dyson, A. H., & Jenson, J. M. (1981). " Real " writing in the elementary classroom. Momentum, 12(4), 1 1- 1 3. Elbow, P. ( 1973). Writing without teachers. London : Oxford University Press. Emig, J. (1971). The composing processes of twelfth graders (Research Report No. 1 3). Champaign, IL: National Council of Teachers of English. Emig, J. (1978). Hand, eye, brain : Some "basics" in the writing process. In C. R. Cooper & L. Odell (Eds.), Research ofcomposing : Points of departure (pp. 59-71). Urbana, IL: National Council of Teachers of English. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215-251. Fadiman, C., & Howard, J. (1979). Empty pages. New York: Fearon Pitman.

RESEARCH ON WRITTEN COMPOSITION

Faigley, L. (1982). Review of linguistics, stylistics, and the teaching of composition. College Composition and Communication, 33(1), 96-98. Faigley, L., Daly, J., & Witte, S. (1981). The role of writing apprehen­ sion in writing performance and competence. Journal of Educational Research, 75, 16-21. Faigley, L., & Witte, S. (1981). Analyzing revision. College Composition and Communication, 32, 400-414. Fisher, J. E. (1981). The interference of mechanical errors in the revision of compositions by grade six children. Unpublished master's thesis, University of Toronto, Toronto. Florio, S., & Clark, C. M. (1982). The functions of writing in an elementary classroom. Research in the Teaching of English, 16, 115-130. Flower, L. (1979, September). Writer-based prose: A cognitive basis for problems in writing. College English, 41(1), 19-37. Flower, L. (1981). Problem-solving strategies for writing. New York: Harcourt Brace. Flower, L., & Hayes, J. R. (1980a). The cognition of discovery: Defining a rhetorical problem. College Composition and Communication, 31(2), 21-32. Flower, L., & Hayes, J. R. (1980b). The dynamics of composing: Making plans and juggling constraints. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 31-50). Hills­ dale, NJ: Lawrence Erlbaum. Flower, L., & Hayes, J. R. (1981). The pregnant pause: An inquiry into the nature of planning. Research in the Teaching of English, 15, 229-244. Flower, L., Hayes, J. R., & Swarts, H. (1980). Revising functional documents: The scenario principle (Document DesignProject Tech. Rep. No.IO). Pittsburgh, PA: Carnegie-Mellon University. Frase, L.T., Macdonald, N. H., Gingrich, P. S., Keenan, S. A., & Collymore, J. L. (1981). Computer aids for text assessment and writing instruction. Performance and Instruction, 20,(9), 21-24. Freedman, S. W. (1981). Evaluation in the writing conference: An interactive process. In M. C. Hairston & C. L. Selfe (Eds.), Selected papers from the 1981 Texas Writing Research Conference. Austin: University of Texas. Frith, U. (Ed.). (1980). Cognitive processes in spelling. London: Aca­ demicPress. Glassner, B. (1982). Lateral specialization of the modes of composing: An EEG study. Unpublished doctoral dissertation, Rutgers Univer­ sity, New Brunswick, NJ. Glynn, S. M., Britton, B. K., Muth, K. D., & Dogan, N. (1982). Writing and revising persuasive documents: Cognitive demands. Journal of Educational Psychology, 74, 557-567. Gould, J. D. (1980). Experiments on composing letters: Some facts, some myths, and some observations. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 97-127). Hills­ dale, NJ: Lawrence Erlbaum. Graves, D. H. (1975). An examination of the writing processes of seven year old children. Research in the Teaching of English, 9, 227-241. Graves, D. H. (1978). Balance the basics: Let them write. New York: Ford Foundation. Graves, D. H. (1981). Research update: Writing research for the eighties: What is needed. Language Arts, 58,(2), 197-206. Graves, D. H. (1983). Writing: Teachers and children at work. Exeter, NH: Heinemann Educational Books. Gray, J., & Myers, M. (1978). The Bay Area WritingProject. Phi Delta Kappan, 59, 410-413. Greeno, J. G. (1978). Natures of problem-solving abilities. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (Vol. 5, pp. 239-270). Hillsdale, NJ: Lawrence Erlbaum. Grice, H.P. (1975). Logic and conversation. InP. Cole & J. J. Morgan (Eds.), Syntax and semantics: Vol. 3. Speech acts (pp. 41-58). New York: AcademicPress. Gundlach, R. A. (1982). Children as writers. The beginnings of learning to write. In M. Nystrand (Ed.), What writers know: The language, process, and structure of written discourse (pp. 129-147). New York: AcademicPress. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.

Hansen, B. (1978). Rewriting is a waste of time. College English, 39(8), 956-960. Harris, M. (1979). Staffroom exchange: Contradictory perception of rules for writing. College Composition and Communication, 30, 218-220. Hayes, J. R. (1981). The complete problem solver.Philadelphia: Franklin InstitutePress. Hayes, J. R., & Flower, L. S. (1980a). Identifying the organization of writing processes. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 3-30). Hillsdale, NJ: Lawrence Erlbaum. Hayes, J. R., & Flower, L. S. (1980b). Writing as problem solving. Visible Language, 14(4), 388-399. Hayes-Roth, B., & Hayes-Roth, F. (1979). A cognitive model of planning. Cognitive Science, 3, 275-310. Hidi, S., & Hildyard, A. (1983). The comparison of oral and written productions of two discourse types. Discourse Processes, 6, 91-105. Hillocks, G., Jr. (1982). Inquiry and the composing process. Theory and research. College English, 44, 659-673. Hirsch, E. D., Jr. (1977). The philosophy of composition. Chicago: University of ChicagoPress. Hotopf, N. (1980). Slips of the pen. In U. Frith (Ed.), Cognitive processes in spelling (pp. 287-307). London: Academic Press. Hunt, K. (1965). Grammatical structures written at three grade levels (Research Rep. No. 3). Champaign, IL: National Council of Teachers of English. Inhelder, B., & Piaget, J. (1958). The growth of logical thinking from childhood to adolescence. New York: Basic Books. Jacob, G.P. (1982). An ethnographic study of the writing conference: The degree of student involvement in the writing process. (Doctoral dissertation, Indiana University ofPennsylvania). Dissertation Ab­ stracts International., 43, 386A. (University Microfilms No. DA 8216050). Kiefer, K. E., & Smith, C. R. (1983). Textual analysis with computers; Tests of Bell Laboratories' computer software. Research in the Teaching of English, 17, 201-214. King, M. L. (1978). Research in composition: A need for theory. Research in the Teaching of English, 12, 193-202. King, M. L., & Rentel, V. M. (1981). Research update: Conveying meaning in written texts. Language Arts, 58(6), 721-728. Kinneavy, J. (1980). A theory of discourse. New York: Norton. Knoblauch, C. H., & Brannon, L. (1981). Teacher commentary on student writing: The state of the art. Freshman English News, 10(2), 1-4. Kosslyn, S. M. (1980). Image and mind. Cambridge, MA: Harvard UniversityPress. Kress, F., & Bracewell, R. J. (1981). Taught but not learned: Reasons for grammatical errors and implications for instruction. In I.Pringle and A. Freedman (Eds.), Teaching, writing, learning. Ottawa: Cana­ dian Council of Teachers of English. Kroll, B. M. (1978). Cognitive egocentrism and the problem of audience awareness in written discourse. Research in the Teaching of English, 12, 269-281. Larson, R. (1971). Toward a linear rhetoric of the essay. College Composition and Communication, 22, 140-145. Lehr, F. (1980). ERIC/RCS Report: Writing as learning in the content areas. English Journal, 69(8), 23-25. Levin, J. A., Boruta, M. J., & Vasconcellos, M. T. (1983). Microcom­ puter-based environments for writing: A writer's assistant. In A. C. Wilkinson (Ed.), Classroom computers and cognitive science (pp. 219-322). New York: Academic Press. Lowenthal, D. (1980). Mixing levels of revision. Visible Language, 14(4), 383-387. Lyons, G. (1976, September). The higher illiteracy. Harper's, pp. 33-40. Macrorie, K. (1976). Telling writing. Rochelle Park, NJ: Hayden. Maggs. A., McMillan, K.,Patching, W., & Hawke, H. (1981). Accelerating spelling skills with morphographs. Educational Psychology, 1(1), 49-56. Maimon, E. (1979). Talking to strar.gers. College Composition and Communication, 30, 364-36,. Mandler, J., & Johnson, N. (1977). Remembrance of things parsed: Story structure and rPcall. Cognitive Psychology, 9, 111-151.

802

MARLENE SCARDAMALIA AND CARL BEREITER

Mandler, J. M., Scribner, S., Cole, M., & DeForest, M. (1980). Cross­ cultural invariance in story recall. Child Development, 51, 19-26. Matsuhashi, A. (1982). Explorations in the real-time production of written discourse. In M. Nystrand (Ed.), What writers know: The language, process, and structure of writ/en discourse (pp. 269-290). New York: Academic Press. McCutchen, D., & Perfetti, C. A. (1982). Coherence and connectedness in the development of discourse production. Text, 2, 113-139. Mellon, J. C. (1969). Transformational sentence-combining: A method for enhancing the development of syntactic fluency in English compo­ sition (Research Rep. No. 10). Urbana, IL: National Council of Teachers of English. Meyer, B. J. F. (1977). What is remembered from prose: A function of passage structure. In R. 0. Freedle (Ed.), Discourse production and comprehension (Vol. 1, pp. 307-336). Norwood, NJ: Ablex. Mishler, E. G. (1979). Meaning in context: Is there any other kind? Harvard Educational Review, 49, 1-19. Moffett, J. (1968). Teaching the universe of discourse. Boston: Houghton Mifflin. Morrison, C., & Austin, M. C. (1977). The torchlighters revisited. Newark, DE: International Reading Association. Mosenthal, P. (1983). Defining classroom writing competence: A paradigmatic perspective. Review of Educational Research, 53, 217-251. Muller, H. J. (1967). The uses of English. New York: Holt, Rinehart & Winston. Murray, D. M. (1978). Internal revision: A process of discovery. In C. R. Cooper & L. Odell (Eds.), Research on composing (pp. 85-103). Urbana, IL: National Council of Teachers of English. National Assessment of Educational Progress. (1975). Writing mechan­ ics, 1969-1974: A capsule description of changes in writing mechanics (Report No. 05-W-0l ). Denver, CO: National Assessment of Educa­ tional Progress. National Assessment of Educational Progress. (1977). Write/rewrite: An assessment of revision skills; selected results from the second national assessment of writing. Washington, DC: U.S. Government Printing Office. (ERIC Document Reproduction Service No. ED 141 826) National Assessment of Educational Progress. (1980a). Writing achie­ vement, 1969-1979: Results from the third national writing assess­ ment (Vol. 1: 17-year-olds). Denver, CO: National Assessment of Educational Progress. (ERIC Document Reproduction Service No. ED 196 042) National Assessment of Educational Progress. (1980b). Writing achie­ vement, 1969-1979: Results from the third national writing assess­ ment (Vol. 2: 13-year-olds). Denver, CO: National Assessment of Educational Progress. (ERIC Document Reproduction Service No. ED 196 043) National Assessment of Educational Progress. (1980c). Writing achieve­ ment, 1969-1979: Results from the third national writing assessment (Vol. 3: 9-year-olds). Denver, CO: National Assessment of Educa­ tional Progress. (ERIC Document Reproduction Service No. ED 196 044) Nelms, B. F. (1979). Editorial: The writing projects: Toward a new professionalism. English Education, 10(3), 131-133. Newell, A. (1980). Reasoning, problem solving, and decision processes: The problem space as a fundamental category. In R. S. Nickerson (Ed.), Allention and performance (Vol. 8, pp. 693-718). Hillsdale, NJ: Lawrence Erlbaum. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall. Nold, E. W. (1981). Revising. In C. H. Frederiksen, & J. F. Dominic (Eds.), Writing: The nature, development, and teaching of writ/en communication (pp. 67-79). Hillsdale, NJ: Lawrence Erlbaum. Nystrand, M. (1979). Using readability research to investigate writing. Research in the Teaching of English, 13(3), 231-242. Odell, L. (1974). Measuring the effect of instruction in prewriting. Research in the Teaching of English, 8, 228-240. Odell, L. (1980). Business writing: Observations and implications for teaching composition. Theory Into Practice, 19(3), 225-232. Odell, L., & Goswami, D. (1982). Writing in a non-academic setting. Research in the Teaching of English, 16(3), 201-223.

O'Hare, F. (1973). Sentence combining: Improving student writing without formal grammar instruction (Research Rep. No. 15). Ur­ bana, IL: National Council of Teachers of English. Olson, D. R. (1977). From utterance to text: The bias of language in speech and writing. Harvard Educational Review, 47(3), 257-281. Olson, G. M., Mack, R. L., & Duffy, S. A. (1981). Cognitive aspects of genre. Poetics, JO, 283-315. Paris, P. ( 1980). Discourse schemata as knowledge and as regulators of text production. Unpublished master's thesis, York University, Downsview, Canada. Perkins, D. N. (in press). General cognitive skills: Why not? In S. S. Chipman, J. W. Segal, & R. Glaser (Eds.), Thinking and learning skills: Current research and open issues (Vol. 2). Hillsdale, NJ: Lawrence Erlbaum. Perl, S. (1979). The composing process of unskilled college writers. Research in the Teaching of English, 13, 317-336. Pettigrew, J., Shaw, R. A., & Van Nostrand, A. D. (1981). Collaborative analysis of writing instruction. Research in the Teaching of English, 15, 329-341. Purves, A. C., & Takala, S. (Eds.), (1982). An international perspective on the evaluation of written composition. Evaluation in Education, 5(3), 207-390. Read, C. (1971). Pre-school children's knowledge of English phonology. Harvard Educational Review, 41, 1-34. Redish, J. C. (in press). The language of the bureaucracy. In R. Bailey (Ed.), Literacy in the 1980's. Rose, M. (1980). Rigid rules, inflexible plans, and the stifling of language: A cognitivist analysis of writer's block. College Composi­ tion and Communication, 31(4), 389-400. Rose, M. (1981). Sophisticated, ineffective books-The dismantling of process in composition texts. College Composition and Communica­ tion, 32(1), 65-74. Rubin, A. (1980a). Making stories, making sense. Language Arts, 57, 285-298. Rubin, A. (1980b). A theoretical taxonomy of the differences between oral and written language. In R. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), Theoretical issues in reading comprehension (pp. 411-438). Hillsdale, NJ: Lawrence Erlbaum. Sawyer, T. M. (1977). Why speech will not totally replace writing. College Composition and Communication, 28(1), 43-48. Scardamalia, M. (1981). How children cope with the cognitive demands of writing. In C. F. Frederiksen, & J. F. Dominic (Eds.), Writing: The nature, development and teaching of wrillen communication (Vol. 2, pp. 81-103). Hillsdale, NJ: Lawrence Erlbaum. Scardamalia, M., & Bereiter, C. (1982). Assimilative processes in composition planning. Educational Psychologist, 17(3), 165-171. Scardamalia, M., & Bereiter, C. (1983a). Child as co-investigator: Helping children gain insight into their own mental processes. In S. Paris, G. Olson, & H. Stevenson (Eds.), Learning and motivation in the classroom (pp. 61-82). Hillsdale, NJ: Lawrence Erlbaum. Scardamalia, M., & Bereiter, C. (1983b). The development of evalua­ tive, diagnostic and remedial capabilities in children's composing. In M. Martlew (Ed.), The psychology of wrillen language: Develop­ mental and educational perspectives (pp. 67-95). London: John Wiley. Scardamalia, M., & Bereiter, C. (1984). Development of strategies in text processing. In H. Mandi, N. Stein, & T. Trabasso (Eds.), Learning and comprehension of text (pp. 379-406). Hillsdale, NJ: Lawrence Erlbaum. Scardamalia, M., & Bereiter, C. (in press-a). The development of dialectical processes in writing. In D. Olsen, N. Torrance, & A. Hildyard (Eds.), Literacy, language and learning: The nature and consequences of reading and writing. Cambridge: Cambridge Uni­ versity Press. Scardamalia, M., & Bereiter, C. (in press-b). Fostering the development of self-regulation in children's knowledge processing. In S. S. Chipman, J. W. Segal, & R. Glaser (Eds.), Thinking and learning skills: Current research and open questions (Vol. 2). Hillsdale, NJ: Lawrence Erlbaum. Scardamalia, M., Bereiter, C., & Fillion, B. (1981). Writing for results: A sourcebook of consequential composing activities. Toronto: OISE Press. (Also, La Salle, IL: Open Court)

RESEARCH ON WRITTEN COMPOSITION

Scardamalia, M., Bereiter, C., & Goelman, H. (1982). The role of production factors in writing ability. In M. Nystrand (Ed.), What writers know: The language, process, and structure of written dis­ course (pp. 173-210). New York: Academic Press. Scardamalia, M., Bereiter, C., & McDonald, J. D. S. (1978, August). Role-taking in written communication investigated by manipulat­ ing anticipatory knowledge. Resources in Education. (ERIC Docu­ ment Reproduction Service No. ED 151 792) Scardamalia, M., Bereiter, C., & Steinbach, R. (1984). Teachability of reflective processes in written composition. Cognitive Science, 8(2), 173-190. Scardamalia, M., & Paris, P. (1985). The function of explicit discourse knowledge in the development of text representations and compos­ ing strategies. Cognition and Instruction, 2(1), 1-39. Shaugnessy, M. P. (1977a). Errors and expectations: A guide for the teacher of basic writing. New York: Oxford University Press. Shaugnessy, M. P. (1977b). Some needed research on writing. College Composition and Communication, 28, 317-320. Simon, J. (1973). La Langue ecrite de l'enfant. Paris: Presses Universit­ aires de France. Smith, F. (1982). Writing and the writer. New York: Holt, Rinehart & Winston. Sommers, N. (1980). Revision strategies of student writers and exper­ ienced adult writers. College Composition and Communication, 31, 378-388. Staton, J. (1980). Writing and counseling: Using a dialogue journal. Language Arts, 57(5), 514-518. Staton, J., Shuy, R., & Kreeft, J. (1982). Analysis of dialogue journal writing as a communicative event (Final report, Vols. 1 & 2). Washington, DC: Center for Applied Linguistics. (ERIC Document Reproduction Service Nos. ED 214 196, ED 214 197). Stein, N. L., & Glenn, C. G. (1979). An analysis of story comprehension in elementary school children. In R. 0. Freedle (Ed.), New directions in discourse processing (Vol. 2, pp. 53-120). Norwood, NJ: Ablex. Stein, N., & Policastro, M. (1984). The concept of a story: A compar­ ison between children's and teachers' viewpoints. In H. Mandi, N. Stein, & T. Trabasso (Eds.), Learning and comprehension of text (pp. 113-155). Hillsdale, NJ: Lawrence Erlbaum. Stein, N. L., & Trabasso, T. (1982). What's in a story: An approach to comprehension and instruction. In R. Glaser (Ed.), Advances in instructional psychology (pp. 213-267). Hillsdale, NJ: Lawrence Erlbaum. Steinberg, E. R. (1980). A garden of opportunities and a thicket of dangers. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 155-167). Hillsdale, NJ: Lawrence Erl­ baum. Stotsky, S. L. (1975). Sentence combining as a curriculum activity: Its effect on written language development and reading comprehen­ sion. Research in the Teaching of English, 9(1), 30-71.

803

Stubbs, M. (1980). Language and literacy: The sociolinguistics of reading and writing. London; Routledge and Kegan Paul. Taylor, K. K. (1982). Can college students summarize? Journal of Reading, 26, 524-528. Tetroe, J., Bereiter, C., & Scardamalia, M. (1981). How to make a dent in the writing process. Paper presented at the meeting of the American Educational Research Association, Los Angeles. Thorndyke, P. W. (1977). Cognitive structures in comprehension and memory of narrative discourse. Cognitive Psychology, 9, 77-110. van Dijk, T. A. (1980). Macrostructures: An interdisciplinary study of global structures in discourse, interaction, and cognition. Hillsdale, NJ: Lawrence Erlbaum. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. (M. Cole, V. John-Steiner, S. Scribner, & E. Souberman, Eds. and Trans.). Cambridge, MA: Harvard University Press. Warnock, J. (1976, fall). New rhetoric and the grammar of pedagogy. Freshman English News, 5, 1-4, 12-22. Wason, P. C. (1980). Specific thoughts on the writing process. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 129-137). Hillsdale, NJ: Lawrence Erlbaum. Waters, H. S. (1980). "Class News": A single-subject longitudinal study of prose production and schema formation during childhood. Journal of Verbal Learning and Verbal Behaviour, 19, 152-167. Wertsch, J. (Ed.). (in press). Culture, communication, and cognition: Vygotskian perspectives. New York: Cambridge University Press. West, W. W. (1967). Written composition. Review of Educational Research, 37(2), 159-167. White, R. S., & Karl, H. (1980). Reading, writing, and sentence combining: The track record. Reading Improvement, 17, 226-232. Winterowd, W. R. (Ed.). (1975). Contemporary rhetoric. New York: Harcourt Brace. Woodruff, E., Bereiter, C., & Scardamalia, M. (1981). On the road to computer assisted compositions. Journal of Educational Technology Systems, 10(2), 133-148. Woodruff, E., Scardamalia, M., & Bereiter, C. (1982). Computers and the composing process: An examination of computer-writer inter­ action. In J. Lawlor (Ed.), Computers in composition instruction (p. 31-45). Los Alamitos, CA: SWRL Educational Research and Development. Young, R. E. (1976). Invention: A topographical survey. In G. Tate (Ed.), Teaching composition: Ten bibliographical essays. Fort Worth, TX: Christian University Press. Young, R. W., Becker, A. L., & Pike, K. E. (1970). Rhetoric: Discovery and change. New York: Harcourt Brace. Zbrodoff, N. J. (1984). Writing stories under time and length constraints. Unpublished doctoral dissertation, University of Toronto.

28. Research on Teaching Reading Robert Calfee Stanford University

Priscilla Drum

University of California-Santa Barbara

Introduction

lost in the turmoil of the 70s, and never moved beyond the initial rhetoric. The federal Right to Read office set up read­ ing centers for adults and distributed kits of "exemplary" materials; little else was accomplished, though see Carroll and Chall (1975), quoted in the next paragraph. The program was discontinued during the early years of the Reagan administra­ tion. The National Reading Council, also established in 1970 and designed to involve the private sector in the Right to Read effort, was closed in 1973. Before his death in 1971, Allen commissioned a report on the status of reading in the United States. The survey, prepared by John Carroll and Jeanne Chall (1975) for the National Acade­ my of Education, gave a comprehensive and thoughtful account of the national situation at the beginning of the decade covered in this chapter. They wrote under darkening clouds of national despair and turmoil, but their mood was one of hope and urgency:

What should the reader expect of a chapter entitled "Research on Teaching Reading"? Because varied perspectives might be taken, we thought it advisable to attempt an explicit answer to this question. First, our readers should know what we mean by research on teaching reading, including our definitions of reading and teaching. Second, the reader should be given a framework for organizing the research literature. Third, the reader should find an account of that literature. We will not attempt a compendium, but instead will highlight samples from the past decade of research on various topics. The goal is to fashion a guide for the informed professional and for appren­ tices trying to comprehend a new field. This introductory section will now continue with some contextual remarks. As experienced colleagues realize, contem­ porary reading research is not neutral, but reflects the social and political context in which it occurs.

Our national policy is that every child is expected to complete at least the twelfth grade; we ought then to expect every child to attain twelfth-grade literacy. . . . The simplest and, to us, most persuasive argument for literacy is that an individual cannot participate in modern society unless he can read, and by this we mean reading at a rather high level of literacy. (pp. 9-10)

The Place of Reading Instruction in American Education Research on reading instruction needs to be viewed against the purposes for reading in a particular place and time. In the 1960s optimism prevailed, with the promises of improved schooling an important consideration underlying this spirit. A dramatic event was the 1969 announcement of the Right to Read program by Commissioner of Education James E. Allen, Jr. Proposed at the beginning of the Nixon presidency, the pro­ gram might have been the primary focus of this chapter if its promise had been realized. As it turned out, Right to Read was

One feature of the Carroll and Chall volume deserves special notice. The report includes several case studies, brief accounts of fictional people for whom learning to read was difficult. Some individuals had to cope with difficult situations like poverty at home and in the neighborhood; some had personal problems. But the main point was that in every instance a

The authors thank reviewers Joseph R. Jenkins (University of Washington) and James R. Squire (Ginn Corporation).

804

RESEARCH ON TEACHING READING

teacher might have made a difference. Such was the tenor of the times. In the late 70s and early 80s, the Right to Read was replaced by the Demand for Excellence. The mood changed dramati­ cally: a concern for equity turned into an emphasis on stan­ dards. Minimum competency tests for high school graduation were mandated in many states and school districts. Remedial programs were provided to failing students, but the rate of high school dropouts increased for the first time in decades (U. S. Office of Education, 1 984) - dropout statistics are not altogeth­ er trustworthy, to be sure, and little is known about the relation between tighter standards, remedial high school programs, and school leaving. The shift in the societal context over the decade has been multifaceted. Emphasis on minimum standards in the basic skills (chiefly reading) shifted rapidly during the early 1980s to concern about excellence in education - increase in graduation requirements, lengthening of school days and years, and tough­ ening of testing. Funding remained stable or actually declined (in real dollars) in some states. Research and development activities dropped to a fraction of what had been planned 2 decades before. The institution of schooling has come under fire repeatedly in recent years. The National Commission on Excellence in Edu­ cation (1983) captured the prevailing concern in its oft-quoted charge that "the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people. " (p. 5). Copperman (1978) arrayed standardized test results showing that, after a brief period of hope during the post-Sputnik period (1957- 1965), the quality of public education in the United States declined markedly. He pointed to the inadequacy of literacy instruction in the nation's schools as the primary cause for the declines : Since the mid-1960's, academic performance and standards have shown a sharp and widespread decline. . . . Even in the more traditional classes, work demands and imposed standards have dropped considerably . . . high-school textbooks in most subject areas have been rewritten with a sharply reduced reading level, usually one or more years lower than the grade for which they are intended. . . . Federal money and intervention were supposed to solve our most pressing educational problems. Instead, they have proven to be a multi-billion dollar source of fraud and political interference. (pp. 1 5, 1 6, 21) Other causes for the decline, in his opinion, were the loss of authority by teachers and principals, the loss of respect for "structure," the replacement of basal textbooks by " open education," and a lowering of standards to accommodate less able students. Whatever the truth of such claims, the quotation captures the prevailing spirit of polemic and concern. As this chapter was written, a visitor to the schools would see efforts to respond to these challenges, most notably increased time in reading and mathematics. Teaching activities are de­ fined with greater precision, since direct instruction has been associated with higher test scores in several studies. Surveys indicate that most (perhaps 95% of) elementary teachers rely on . a basal series, and that they hew closely to the teacher's manual (Dixon, 1979). Student performance is monitored through

805

standardized tests; parents, the general public, teachers and administrators, and politicians give close attention to the findings. The results of these efforts are mixed. Reading performance in the primary grades shows improvements (NAEP, 198 1), and SAT scores are steadying, but achievement in the upper grades continues to drop, especially in the more demanding literacy tasks, and in writing and the so-called content areas (chiefly social studies and science). The preceding " snapshots" capture some aspects of reading instruction in the mid- 1980s. Our goal has been to sketch the societal and historical context within which the review is framed. The field of study, the methods of research, and the expected findings all reflect this context, which has seen major fluctuations in the past two decades.

Previous Handbooks The two previous handbooks (Gage, 1963 ; Travers, 1973) provide a historical perspective of a different kind. Russell and Fea (1 963) wrote during a " golden era" of research on reading instruction. The nation was optimistic, support for education was increasing, and hopes were high for a bright future. Russell and Fea's announced intention was to focus on research in classroom settings, with particular emphasis on methods and materials for instruction. They expressed less interest in ancill­ ary topics like psychology, physiology, sociology, and linguis­ tics. In fact, however, of the three topics in their review, two were predominantly psychological -perceptual processes in word recognition and conceptual processes in comprehension. The third topic, classroom management and organization, was not unique to reading instruction. These observations are not meant to criticize - the authors dealt with the literature avail­ able at that time. Ten years later, Della-Piana and Endo (1973) wrote at the peak of the "federal era." Research funds had increased sub­ stantially during the decade, and optimism remained high about the potential of public schooling. The stated goal of the review was to emphasize " sources that appear to be productive for methodological, theoretical or practical values in develop­ ing a body of generalizable knowledge and in improving reading instruction" (p. 883). They described the First Grade Reading Study (Bond & Dykstra, 1967) and Chall's (1967) Learning to Read: The Great Debate, both of which investigated variations in "the method of instruction." Should emphasis be placed on phonics or meaning? Should instruction go from part to whole or the reverse? The answer to these and other "methods" questions seemed to be "Yes" - many things help. The review also covered the newly emerging technologies. The computer had arrived, and with it new languages and concepts ; hence a section on product development. Once again, relatively little of the research examined classroom practices or the curric·..ilum. The studies were rooted in psychological theory and relied on laboratory analogues of reading. The "methods" experiments contrasted programmatic approaches (phonics, linguistic techniques, language experience, and so on), but actual practice was seldom observed. For whatever reason, the effects of variations in methods were generally small compared with the variance between teachers and students.

806

ROBERT CALFEE AND PRISCILLA DRUM

An Overview of the Chapter The substantive portion of the present chapter begins with a definition of reading that gives our perspective on the literate person. The definition provides a framework for the curricular domain by identifying four major components in skilled read­ ing: decoding, word meaning, the comprehension of sentences and paragraphs, and the comprehension of entire texts. Next comes a section on the acquisition of literacy which contrasts stage models with learning models. Finally come three sections on research findings: instruction in decoding, in word meaning, and in various forms of compre­ hension. The strategy in these sections will be to sketch each domain in broad strokes, and then select several studies to highlight methodologically and substantively. Our selections are constrained by the available literature, of course. We will concentrate on studies that inform the dimen­ sions of curriculum and instruction, and will emphasize exam­ ples from classroom practice where possible. Greater attention will be paid to issues and paradigms newly arrived on the scene - chief among these are advances in cognitive psychology and comprehension theory. On the other hand, as Barton ( 1963) noted more than 20 years ago, " It does appear . . . from an analysis of manuals, texts on reading instruction, introduc­ tions to readers, and similar advice to teachers over the past 1 50 years [and, one might add, from reviews of the research literature], that almost all of the issues raised in the past ten years were being raised long before there was anything such as educational research" (p. 249).

The Literate Person The aim of reading instruction is to help a person become literate. Literacy has at times meant little more than reading and writing one's name (Mathews, 1966), but modern society puts greater demands on the citizen. A literate person should be able to read almost anything in the native language, with understanding of the message or identification of the barriers to understanding. A literate person knows when a text is ambi­ guous, when the lexicon is unfamiliar, or when the conceptual framework for the text is unknown. How does this skill and knowledge come about, and of what does it consist? To address this question, the section will begin with a description of the curriculum domain of reading the English language. The syn­ opsis emphasizes three facets of the curriculum - the founda­ tion in oral language, the nature of English as a historical polyglot, and formal and informal styles of language usage. Next comes an analysis of the psychological processes in skilled reading. Heading the list are the basic cognitive re­ sources that underlie all thinking and behavior. Then follows a description of what goes on in the mind of a competent reader while scanning a text. These analyses provide a theoretical model of the reading process that will comprise the primary frame of reference for the remainder of the chapter.

The Curriculum of Reading A curriculum is a course of study - the word has its roots in the notion of a racetrack or an obstacle course. Literacy is deriva-

tive of language; reading is the mastery of written language. It is therefore appropriate to say something about the foundation in spoken language for the development of literacy. LANGUAGE

" By the age of three or four, virtually every child has learned a language. " You immediately and properly understand the reference to spoken language. The child's linguistic performance matures through adolescence and beyond, but the essential characteristics of adult language appear in the preschooler's speech and comprehension, including a well-developed phono­ logical system, a substantial store of morphemes and rules for adding to that store, a syntax that allows the child to parse and produce the strings of morphemes that relate ideas, and an understanding of the conventions for carrying on a conversa­ tion. On the other hand, the preschooler's language is rooted in pragmatics, in making things happen (Halliday, 1 975). The early forms of language serve the child's immediate needs in fairly economical fashion. By the fourth year, language and thought have begun to shape one another, and language has come to serve the needs of abstraction and problem solving. Children differ considerably in the extent and character of language usage, but virtually every 4-year-old has learned the most fundamental lesson about language -words stand for ideas ! Language is a complex and even mysterious phenomenon, but linguists agree on the major components. The partitioning into the categories of phonology, morphology, syntax, and some larger unit (discourse or the like) is widely accepted, and linguists are confident about examining these components independently of one another. Moreover, an individual may have problems in one element (e.g., a lisp) with no discernible influence on the other elements. THE ENGLISH LANGUAGE

History is the key to English (Nist, 1966; Balmuth, 1982). The complexities of the language spring from our variegated heri­ tage. Several waves of conquerors have swept across the British Isles during the past 2 millenia, each leaving an impression on the inhabitants and their speech. Some invasions were predo­ minantly physical (the Angles, the Saxons, the Danes, and the Normans), while others were intellectual (from the 17th century on, the French and the Italians; and afterwards most of the world). There was no institution comparable to the Academie Fran9aise to protect linguistic purity; instead, the culture seemed almost to invite linguistic intrusions. The foundation of English is Anglo-Saxon, a North German language. After their arrival during the 5 centuries after the birth of Christ until the Norman invasion, the Danes, Angles, and Saxons lived together in linguistic harmony. Over the centuries the language became simpler in its syntactic structure and more complex in its lexical system. The Battle of Hastings in 1066 marked an event that was to alter the English language greatly. The Normans, Norsemen who had lived for a time in France, spoke a variant of French, a Romance or Latinate language, which was designated after the conquest as the

807

RESEARCH ON TEACHING READING

official language of the land. As described by Nist (1966), the Normans won the battle, but they lost the war: The Norman King of England and his French nobility remained utterly indifferent to the English language until about 1200. During that time the royal court patronized French literature, not English. When the court and its aristocratic supporters did finally pay attention to the native language of the land they dominated, that language was no longer the . . . Teutonic and highly inflected Old English but the hybrid-becoming, Romance-importing, and inflec­ tion-dropping Middle English. (p. 107) By the middle of the 1300s Chaucer, Duns Scotus, and Roger Bacon wrote in a form recognizable today as English. The next major event equaled the Norman conquest in linguistic im­ pact - the Renaissance begun under Elizabeth I. It was the time of Shakespeare, Bacon, Milton, Donne, and Spenser; of Newton, Hooke, Halley; of the King James translation of the Bible; of the exploration of North America and the " rule of the waves" by the British Navy. The word had by then become the fundamental unit. The English language, opportunistic as ever, continued to beg, borrow, steal, and invent. Words flowed by the tens of thousands into English from the Continent and the Scholastic tradition. Thus the language acquired a word for every occasion; several, in fact, each differing subtly from the others in connotation and style (Barzun, 1977). Modern Eng­ lish had arrived, a polyglot of remarkable vitality, richness, and simplicity. WRITTEN ENGLISH

The Roman alphabet used to write English traces back to Egyptian pictographs and the Phoenician syllabary, and thence to the Greek alphabet (Gelb, 1952). The Greeks invented the alphabetic principle, each letter representing a single phoneme, more or less. Spoken Greek was fairly simple and consistent, and so the alphabetic principle was a natural fit. The particular insight of the Greeks was to develop symbols for the vowels as well as for the consonants. The Romans modified the Greek alphabet to suit the relatively simple demands of Latin. The subsequent adaptation of the Roman alphabet to Anglo-Saxon speech posed a greater challenge, which will be discussed in the section on decoding. In 1476, William Caxton introduced the printing press to England. This event, occurring just before the Elizabethan era, fixed the spelling of English during a time of turmoil in the language. As Nist ( 1 966) put it : These changes in the pronunciation of English took place during the worst possible time, when the spelling of the language jelled from 1550 to 1 650. Thus, for the most part, present-day English spells like a post-Chaucerian and pronounces like a post-Shakespearean. Whatever the correspondence between spelling and pronuncia­ tion before the "changes," the match thereafter was poorer.

Reading and Formal Language The graphic representation of speech is a remarkable invention, and analyses of literacy naturally begin with the printed aspects of reading. On the other hand, writing has influenced human

thought and communication in ways other than the conversion of speech into writing. These other influences are of fundamen­ tal importance in the reading curriculum. The contrast between spoken and written modes of expres­ sion has been emphasized by Olson (1977), who builds on previous work by Luria (1976), Vygotsky (1962), and Goody and Watt (1963), among others. The distinction has been most often expressed as the difference between utterance and text, that is, between speech and print. The characteristics of each style of expression are summarized in Table 28. 1 . A conversa­ tion between friends generally makes sense only in the situation in which the conversation takes place. In the language of text, little is left to chance; the writer eschews phrases like " y'know what I mean . . . really." Because it is so explicit, a book's message remains fairly constant regardless of where and when the book is read. To be sure, this constancy assumes that the reader knows the conventions used in the text; the reader must conform to the writer's idea of audience. Texts are relatively unchanging over time. The reader may forget a detail or change an interpretation, but at least the words are not lost in the air. Differences between writing and speech have been docu­ mented by both educators and behavioral scientists (Chafe & Danielwicz in press; Kroll, 1977; O'Donnell, 1974; Olson, 1977). The important contrast, however, is not between speech and print, but between natural and formal styles of thought and communication (Freedman & Calfee, 1984; Heath, 1983). Reading instruction does not simply teach the child a system for translating speech into print, but instills an understanding of the conventions for thinking and communicating in the modern world. Some scholars believe that the acquisition of the skills of literacy ensure that the individual has mastery of the formal style; for example, Goody ( 1977) says, "I see the acquisition of these [literate] means of communication as effectively trans­ forming the nature of cognitive processes" (p. 18). Others question whether transfer from reading to formal style is automatic. For instance, Wells (1979) states that "literacy can be associated with important facilitating effects, . . . but the [facilitation] will depend upon the uses to which literacy is habitually put, once it has been acquired" (p. 27). The evidence supports the doubters. Scribner and Cole (1978; also see Heath, 1983, for examples closer to home), from their studies of literacy among the Vai tribespeople of West Africa, challenge the assumption that abstract thinking is an automatic consequence of literacy. Instead, they find that the conditions of acquisition determine the conditions of transfer. The Vai learn their script in situations that relate to immediate, Table 28. 1 . Characteristics of N atural vs. Formal Language Natural Language (Utterance) H ig h ly I m p l icit ; I nteractive Context Bound U n i q u e ; Idiosyncratic; Personal Intuitive Sequential ; Descriptive

Formal Language (Text) Highly Explicit Context Free Repeatable ; Memory-Supported Logical ; Rational Expository ; " Context"

808

ROBERT CALFEE AND PRISCILLA DRUM

practical needs; they learn by doing, rather than from school­ ing. They practice on commonplace tasks - writing letters, recording contributions to a funeral feast, listing donations to a religious society. The consequences, according to Scribner and Cole, seem to be that the effects of literacy, and perhaps schooling as well, are restricted . . . to closely related practices. . . . If the educational objective is to foster analytic logical reasoning, that objective should guide the choice of the instructional program. (pp. 457, 460)

This point is central to the goals of the reading curriculum and instruction in reading. It is not enough for the citizens of a modern country to perform at minimum levels of competence in reading and writing. The deeper learning entailed in the acquisition of a formal style of language and thought, as described by Bruner (1966), is most likely achieved when instruction fosters the acquisition of broadly generalizable analytic problem-solving skills, where the primary task for the student is to discover the principles by which spoken English is converted into written form, and how to apply those principles to novel forms of expression and communication: Many . . . skills are taught in the subtle interaction of parent and child . . . as in the case of language learning, where the pedagogy is highly unselfconscious. It is probably true that most of the primitive skills of manipulating and looking and attending are also taught in this way. It is when the society goes beyond these relatively primitive techniques that the less spontaneous instruction of the school must be relied upon. At this point the culture necessarily comes to rely upon its formal education as a means of providing skills. And insofar as there has been any innovation in tools or tool­ using, . . . the educational system is the sole means of dissemina­ tion - the sole agent of evolution, if you will. (p. 26)

Cognitive Processes in Skilled Reading The previous section portrayed the reading curriculum in bold strokes. This section on the cognitive structure of the skilled reader begins with a general discussion of cognitive processes, and then applies these concepts to the domain of reading. The section builds on a variety of sources, many summarized in Calfee ( 198 1), but also see Crowder (1982) and Downing and Leong (1982) for additional background on the psychology of reading, along with Wittrock (1978) and Greeno ( 1980) on the cognitive movement in education. HUMAN INFORMATION PROCESSING

What is the mind of a skilled reader like? The answer to this question can be grounded in the psychology of human informa­ tion processing developed over the past quarter-century, nicely summarized by Simon (1978) as follows: A few basic characteristics of the human information-processing system shape its problem-solving efforts. Apart from its sensory organs, the system operates almost entirely serially, one process at a time, rather than in parallel fashion [many processes at a time]. This seriality is reflected in the narrowness of its momentary focus of attention. The elementary processes of the information-processing system are executed in tens or hundreds of milliseconds. The inputs

and outputs of these processes are held in a small short-term memory with a capacity of only a few (between, say, four and seven) familiar symbols or chunks. The system has access to an essentially unlimited long-term memory, but the time required to store a new chunk in that memory is on the order of seconds or tens of seconds. (p. 273)

From this perspective, the key to understanding how a person performs a skilled task like reading comes not from the mental "hardware," but from the " software," from the pack­ ages of knowledge stored in long-term memory that guide the performance of routinized activities. The ability to think is shaped by the limited capacity of the attentional system. The executive homunculus that " runs the mind" can focus on only a small number of distinctive entities, and these only one at a time. How do we manage thinking that is fast and complicated, which is clearly within our capability? The answer is that the competent person has internalized a large number of highly automated routines, so that he or she can carry out complex tasks "without thinking" while simultaneously attending to a small number of matters that do require focal attention. Automated routines, along with other knowledge, are stored in long-term memory, which is the repository of experience, facts, beliefs, skills, feelings, and so on. Human memory has a storage capacity that is, for all practical purposes, unlimited. Much of what we know requires nothing more than attending to an event. The details may be fuzzy, and we may be unable to recapture the memory unless provided a cue for recognition, but memory abides. Long-term memory is inherently inclined to abstract and organize. Life consists of a relatively small number of recurring events, and the mind captures these patterns of recurrence. The concept of abstract mental representations has been variously labeled by cognitive psychologists, but schema is probably most common (Rumelhart, 1980). A schema is defined as an outline or " frame" comprising the major elements and relations, along with a set of "slots" to be filled. The mind is not a static machine containing fixed outlines. As Estes (1980) points out it is alive : The human memory seems to be not at all like a storeroom, a library, or a computer core memory, a place where items of information are stored and kept until wanted, but rather presents a picture of a complex, dynamic system that at any given time can be made to deliver information concerning discrete events or items it has had experience with in the past. In fact, the human memory does not, in a literal sense, store anything; it simply changes as a function of experience. (p. 68)

How does the mind create a schema? Here are three answers to the question, answers that are not mutually exclusive. The first is a simple averaging mechanism; common features in a set of related experiences are remembered most clearly, thus form­ ing a "prototype. " This mechanism requires little in the way of focal attention; the individual may be quite aware of the nature of the abstraction, and the prototype may remain "nameless. " A second model links experience with a preexisting pattern; " registering for classes at the university is like going to a cafeteria," for example. While the person may be unaware of the link at times, on other occasions the analogy may depend on

RESEARCH ON TEACHING READING language and intention. Bolinger (1980) speaks elegantly of the latter possibility: This seeing of like and unlike, of putting together and classifying apart . . . is the mechanism through which reality is organized and the whole construct of language is built. . . . The world is a vast elaborated metaphor. . . . Nature does not tome to the child in ordered fashion, but the child is equipped to perceive parts of it, and is born with the intellectual capacity that surpasses all others: the ability to see resemblances. (p. 191)

The third source of mental structures is sufficiently different from the others to merit a unique label; we will designate it as a script. Schemata result from naturally occurring experiences and natural reactions to those experiences. No one teaches you the similarities among restaurant-like experiences; you figure them out on your own. In other instances, however, organizing structures come from rational analysis and formal instruction. The principles of Newtonian physics lead to conclusions that are counterintuitive and that seem to contradict everyday experience. Newton's derivation of the laws of physical motion was a brilliant insight, and the Three Laws of Motion stand as a prime example of a parsimonious framework for perceiving and interpreting a domain of experience - a framework of such importance that it has been placed in the curriculum, ensuring that successive generations will share this insight. Scripts and schemata probably comprise endpoints on a continuum. The two varieties of mental structures have differ­ ent characteristics, different strengths and weaknesses, and almost certainly they emerge through different sets of environ­ mental events (see subsequent discussion of learning by doing and by knowing).

809

The headings in Figure 28.1 resemble in some ways the top­ level categories in contemporary scope-and-sequence charts and basal reading series. The components also correspond to linguistic categories for analyzing spoken languages - phono­ logy, morphology, syntax, and various types of discourse structure. For better or worse, scope-and-sequence charts are not taken too seriously, probably just as well. In them one finds mixtures from different categories, so that word meaning objec­ tives may be listed under vocabulary, and so on. We will take the model quite seriously in this chapter, however, and will use it to organize our presentation of the research literature.

Learning to Read How does a child learn to speak? The answer is mysterious. Expose an infant to language, and he or she will learn to speak. No curriculum is needed; the skill is acquired without formal schooling. How does a child learn to read? Opinions differ, but one point is clear: " Exposure" to print does not guarantee that a child will benefit from the experience. What then is the explanation? At least four answers have been posed in response to this question: (a) Reading is acquired naturally, like learning to speak ; (b) reading is acquired through a series of stages; (c) reading is learned through the mastery of a set of specific skills; and (d) reading is learned by formal instruction in a new domain of knowledge. While these answers are not mutually exclusive, they are sufficiently distinctive to warrant separate coverage.

SEPARABLE PROCESSES IN READING

How should we describe the mind of a skilled reader - someone who has been taught to read English in a formal classroom setting? The student has acquired a rather complex domain of knowledge, which allows him or her to gain information from printed material. One approach is to characterize the domain by a set of relatively independent processes (Calfee, 1976; Frederiksen, 1980; Perfetti & Lesgold, 1979; R. J. Sternberg, 1977; S. Sternberg, 1969 ; and Simon, 1981, who refers to " nearly decomposable" components). While each component may have a complicated internal structure, the relation between any pair of components is simple. The reading process may appear highly interactive and complicated in operation, but the under­ lying structure is actually quite simple - such is the gist of the separable-process assumption. The key to understanding the reading process is the curricu­ lum. The "ideal" reader has acquired the set of components shown in Figure 28. 1. Independence of the components reflects neither physiology nor statistics, but rather the structure of knowledge used to perform the task. The components act as stable intermediate forms (Simon, 1981), as subparts of a larger system, with each part relatively unaffected by the other sub­ parts. A teacher can work on vocabulary apart from decoding; the reader can analyze the grammar of a sentence without necessarily engaging the overall text structure.

Acquisition as a Natural Process An extreme version of the naturalistic approach holds that exposure to reading materials is all that is required for acquisi­ tion (Goelman, Oherg, & Smith, 1984). The learning of oral language is viewed as a good model to follow. Tinkering with the natural state of affairs may lead to failure, because children will have to deal with concepts and structures that make little sense to them. Reading failure results from efforts to " teach" what children might more easily learn on their own, according to this position. A variant of the naturalistic position stresses the importance of providing reading materials and activities with which the child is already familiar. The Language Experience method (Allen & Allen, 1966) exemplifies this position. Applebee and Langer ( 1983) also focus on the language task confronting the student, but propose that the instructor provide a conceptual framework or " scaffolding" to aid the student. The level of support is critical. Compare " fill in the blank" worksheets with essay questions ; the first activity provides too much support and the second provides too little. Applebee and Langer recommend an inte_rmediate level of support, giving the student only as much guidance as is needed, where the discussion of a text, the explanation of a decoding principle, the definition of a word all fit a model determined by the instructor.

810

ROBERT CALFEE AND PRISCILLA DRUM

LANGUAGE DECODING HISTORY OF ENGLISH

Fig. 28.1 . Com ponents of the read ing p rocess

Stage Theory

STAGE 4. MULTIPLE VIEWPOINTS: HIGH SCHOOL

Chall (1983), a major figure in reading instruction for more than 3 decades, has proposed a six-stage model of reading acquisition that elegantly portrays this point of view. Chall likens the acquisition of reading to growth in spoken language or the development of cognition: "Like Piaget's cognitive stages, . . . reading stages have a definite structure and . . . generally [followJ a hierarchical progression" (p. 1 1 ). The sequential constraints are assumed to be critical; the student must move through one stage before tackling the next (pp. 55ft). Chall notes that passage from one stage to another depends on the student's interaction with the home and school environ­ ment (pp. 78ft), and she discusses in some detail the transitions from one stage to the next (pp. 40ft). Here are the stages: STAGE 0. PREREADING: BIRTH TO AGE 6

The numbering and the description of this stage are both unusual. It is the stage that is not a stage, yet it "covers a greater period of time and . . . a greater series of changes than any of the other stages"(p. 1 3). In this period fall the times of home, preschool, and kindergarten. While it is labeled the "preread­ ing" stage, what the student is taught about reading before first grade matters greatly (see references in Chall, pp. 1 3-14). STAGE I. INITIAL READING OR DECODING: AGES 6 TO 7

For Chall, the acquisition of letter-sound correspondences is critically important for the beginning reader (e.g., Chall, 1967); unsurprisingly, the first " real" step in her reading model emphasizes this domain. Chall does not equate decoding with reading, but does emphasize that beginning readers "have to engage, at least temporarily, in what appears to be less mature reading behavior - being glued to print" (p. 18). STAGE 2. FLUENCY: AGES 7 TO 8

With practice, the child no longer gives major attention to decoding, but begins to focus on the content. First must come the development of speed and "automatization" of decoding skills. "Stage 2 reading is not for gaining new information, but for confirming what is already known to the reader" (p. 18). STAGE 3. READING TO LEARN: AGES . . . ?

In this phase, students "start on the long course of reading to 'learn the new"' (p. 20). Reading becomes a tool for exploring subject matter areas and for handling daily affairs.

Texts at this level present the youngster with more than a single point of view. Comprehension is more demanding, because the student must hold one set of ideas in mind while acquiring another comparison or expansion. STAGE 5. RECONSTRUCTION: COLLEGE AND BEYOND

In this stage the student views from multiple perspectives - a "metareading" stage. The young adult has the knowledge and maturity to realize that text is not a God-given truth, but a collection of ideas to be examined and challenged. As an account of current practice, Chall's stages may describe progress toward literacy for many students. But the question remains - what is being described? Is it a developmental model, in which individual students proceed through these steps because of inherent characteristics of the human mind? Is it a learning model, in which the reading curriculum requires some things to be acquired before others? Is it a curriculum model, in which experts have constructed the optimal se­ quence? Chall's account does not focus on the developing child, but on the materials and the environmental conditions that sur­ round the child. In this sense, her model is not grounded in developmental concepts. Nor does she speak to the nature of learning. There is an implicit model of "practice"; with suffi­ cient exposure to materials, with instruction, encouragement and feedback, and with time, the child will acquire the skills and knowledge required for literacy. The psychology of learning is largely empirical. Chall seems to be describing a curriculum model. "Many practices in the schools seem to parallel the reading stages proposed here" (p. 65). She is attentive to the organizational characteristics of the school environment through which stu­ dents move, and she highlights several disjunctures - the entry to kindergarten, the jump to first grade, the changes from the early primary to the late elementary grades, and the shift to secondary school. Research and practice (for example, the fourth grade "slump," Chall, 1 983, p. 67) point to the impor­ tance of these transitions. Reading as the Mastery of Specific Skills Dominating contemporary design of reading materials, of texts for preservice training of teachers, and of reading tests is the notion that reading is a collection of precisely defined objectives or skills (Osborn, Wilson, & Anderson, 1985). The number and

RESEARCH ON TEACHING READING

character of the objectives varies with the publisher and the category. Decoding tends to be highly fractionated (e.g., If/, as in f, ff, gh, and ph: prefixes ex-, inter-, and super-; and suffixes -ist, -ible, and -able). Vocabulary objectives are generally fewer but more diffuse (e.g., antonyms, synonyms, multiple meanings, context clues). Comprehension objectives take shape as a collection of poorly defined terms (e.g., main idea, literal and inferential comprehension, fact versus opinion). Acquisition depends on teacher-led presentations and work­ sheet activities, following which students' performance is as­ sessed by curriculum-embedded tests. A low score leads to more practice on similar activities. The learning model is implicit and empirical - practice makes perfect (or at least permanent). A typical lesson which may span two or more reading sessions, contains a variety of objectives, usually covering all of the major categories but seldom with any clear relation to one another. The lesson generally begins with a review of several words from the text, oral reading of a story, interspersed questions about details and inferences from the text, after which the student may complete worksheets on /f/, word meanings from context, and map-reading skills. The goal is to cover the skills, which may number in the thousands. The incoherence of the collection is a poor match to the known characteristics of human mental processes.

Learning as a Formal Activity The last model to be presented combines cognitive psychology and curriculum theory. The ideas behind the proposal are more fully developed in Calfee (198 1 ), and will be only sketched here. The model begins with the assumption that the architecture of the mind is fundamentally simple. What makes the mind interesting and complex are its contents, the mental structures that underlie performance and knowledge. Some of those structures - for example, what we know about restaurants - occur through natural learning. Others - for example, what we know about sentence diagraming - result from formal in­ struction. At both ends of this continuum, the final product is a mental representation, a body of knowledge, shaped by the mind into a more or less coherent structure. The distinction between natural and formal learning does not entail a value judgment. Both kinds of learning are important in the development of competence, and the educated person relies on both types of learning more or less consciously to support each other. John Dewey (1902) who has spoken with wisdom on virtually every educational topic, describes this interplay between learning by experience and learning by schooling: The map is not a substitute for personal experience. The map does not take the place of the actual journey. The logically formulated material of a science or branch of learning, of a study, is no substitute for the having of individual experiences. . . . But the map, a summary, an arranged and orderly view of previous experience, serves as a guide to future experience. . . . Through the map every new traveler may get from his own journey the benefits of others' explorations without the waste of energy and loss of time involved in their wanderings. (pp. 20-21) The contrast between types of learning, summarized in Table 28.2, amounts to a theory of learning. The distinctions between

the two types entail significant variations both in content and conditions, both for learning and teaching. Learning-by-doing is relatively haphazard. Inefficient in the short run, it ensures practiced skills applicable to a wide range of settings. " Teach­ ing" in this approach is incidental, little more than supervision to protect the learner from harm. In learning-by-knowing, the content is critically important; it is the curriculum, grounded in the artifactual knowledge of the culture, in tools for thinking cherished because of their simpli­ city, their power, and their elegance. The criteria call to mind the separable-process model of reading presented earlier (Figure 28.1 ), which has coherence as the primary aim - a small number of simply related elements. And so our foundation for reviewing the literature on in­ struction in reading will be the content of the reading curricu­ lum. While instructional issues will be highlighted against this instructional backdrop, the perspective on instruction is not neutral. Just as reading is viewed as a formalism, grounded in a small number of fundamental components, so can teaching be construed. The components of teaching, as " unnatural" an act as reading (Gough & Hillinger, 1980), have been discussed elsewhere (Calfee & Shefelbine, 1981) and will only be sketched here: • A Conception of the Curriculum. " What is the nature of reading?" The teacher's answer should satisfy the criteria of simplicity and coherence. • A Conception of the Learner. The teacher's image of the nature of learning should transcend the subject matter, and should provide pragmatic guidance for instruction. • Development and Selection of Materials. A craftsman should be master of his or her tools - instructional materials are the teacher's work kit, and so the teacher should be master of these materials. • Assessment Procedures. Vital to instruction is measurement of instructional outcomes - a level of knowledge beyond the minutiae of "tests," but rooted in the basic concepts of valid evidence and interpretation.

Table 28.2. Major Disti nctions Between Learning Based on Doing and Learn i ng Based on Knowing Doing (Schema) Learning comes from repeated experiences Acq uisition takes much time and effort, with many m i stakes Repeated experiences eventually become organized through natural mental processes-coherence is the result of learning Little or no awareness of the content or process of acq uisition " What is learned " can serve as basis for gai n i ng knowledge

Knowing (Scri pt} Learning results from teaching Acq uisition may be virtually i nstantaneous-the " ah a ! " fee l i ng I nformation is presented in a carefu lly organized manner from the beg i n n i ng-coherence is the fou ndation for learning Awareness is often an essential part of acq uisition " What is learned " may facil itate development of skills based on practice

812

ROBERT CALFEE AND PRISCILLA DRUM

• Long-Range Management. The teacher has charge of a small business - grouping of students, allocation of resources (time, people, materials), scheduling of events, arrangement of the room - and all these matters require planning over days, weeks, months, even years. • Short-term Interaction. One child strikes another; a question is asked and no one responds; a student gives an unexpected answer. In these and a thousand other moment-to-moment situations, the teacher must decide immediately on a course of action. • Relation to School as an Organization. The teacher is part of a team, and the well-being of the school rests on the shoulders of the team as a whole. To reiterate the point made earlier, our purpose in listing these teaching components is to advise the reader of the framework guiding our analysis of the "teaching" portion of "research on teaching reading." The conception of the reading curriculum, the development and selection of materials, the instruction actually presented, and the assessment procedures, all merit attention when reviewing a research paper.

Decoding Perhaps no other topic m reading instruction provokes as much controversy as the place of phonics - should the begin­ ning reader concentrate on spelling patterns or on the meaning of the text? After a brief review of this continuing debate, we will provide a structural description of English letter-sound correspondence. This foundation gives a perspective on the several research areas covered in the second part of the section, which examines in turn the knowledge children bring to school about words and sounds, laboratory investigations of the acquisition of spelling patterns, the roles of fluency and of context in decoding, and some classroom studies of phonics instruction.

To Decode or Not to Decode Learning to Read : The Great Debate (Chall, 1967) argued the role and timing of decoding in reading acquisition (also see Mathews, 1966). Several issues were taken for granted in this debate, including the question: Decode or not decode? What is decoding? We will rely on a beguilingly simple definition: Decoding comprises the skills and knowledge by which a reader translates printed words into speech, that is, pronounces a written word. TODAY'S CURRICULUM

The decoding curriculum in most contemporary basal series takes the following shape: Divide the letter-sound correspon­ dences into specific objectives (e.g., f, ff, gh and ph make the sound /f/). Teach the child these objectives, often through seatwork. Avoid rules - they are untrustworthy (e.g., " When two vowels go walking, the first does the talking" ; translated, vowel digraphs in words like pain, beet, boat and fuel take the name of the first letter - but see exceptions like said, bread, broad, and language).

The centrality of decoding in the first stages of reading acquisition is often assumed. Some scholars seem to think that to read is to decode, that the child with decoding skills will easily transfer this skill and preexisting language to printed materials. The decoding curriculum, a skill-based, "bottom-up," drill­ and-practice program, is often the first encounter with formal reading instruction. Unsurprisingly, this state of affairs has drawn fire from educators opposing what they see as inhumane and mindless. They view meaning as the goal of reading, and see decoding as secondary. The child will discover the essentials of decoding if simply given opportunities to interact freely with print. Leave the mechanics of decoding for later, if needed. The debate has gone on for millenia, and shows no signs of slackening. Tchudi ( 1983) reviewed five books from the early 1980s that capture the intellectual side of the argument: • Flesch (198 1 ) in Why Johnny Still Can't Read repeats his arguments that phonics (decoding) is critical. (Copperman, 1978, also supports the Fleschian view). • Sharp (1982) argues with equal vigor that The "Real" Reason Why Johnny Still Can't Read is the inconsistency of English spelling. He recommends regularizing the letter -sound system. • Bettelheim and Zelan (1982) view mispronunciations as Freudian slips, best ignored. Expose the child to good literature (they have recommendations), help the child to find meaning, and let decoding take care of itself. Goodman (Golash, 1982) holds a similar view, though without the psychoanalytic overtones. Smith (1978), also of the Good­ man persuasion, surprisingly did not have a book for review. • Finally, Kohl ( 1982), in his book Basic Skills, stresses " using the language skillfully, . . . thinking for problem solving, thinking imaginatively, understanding fellow human beings, and knowing how to learn something for oneself' - cer­ tainly not the usual back-to-basics. On the empirical side, substantial evidence speaks to the advantages of early phonics. Chall ( 1967) reviewed many of these findings, to which can be added more recent findings from large-scale evaluation efforts (the First-Grade Reading Study, Bond & Dykstra, 1967; Planned Variation in Follow-Through, Stallings, 1975). Standardized tests show modest but consistent gains from direct instruction in decoding (Becker & Carnine, 1 980). The student who has not acquired decoding skill by the middle of elementary school will be hindered in further aca­ demic attainments. Studies comparing good and poor readers have focused on decoding or "word identification" skills as the critical dimension (Kleiman, 1982). Doehring, Trites, Patel, and Fiedorowicz (1981) assessed a cohort of 88 students with severe reading problems and found that "the greatest reading impair­ ment was in orally decoding nonsense words" (p. 1 90). In surveys, teachers emphasize the importance of decoding skill (e.g., Mason & Osborne, 1982), perhaps because they find this easier to describe than comprehension skill. Students also tend to identify decoding and oral reading fluency as the hallmark of the good reader (Johns, 1984). Small wonder that Liberman and Shankweiler ( 1979) state with confidence that "the child's fundamental task in learning to read is to construct a link

RESEARCH ON TEACHING READING

between the arbitrary signs of print and speech" (p. 1 10). Glushko (1981) puts it bluntly: "Dealing with nonsense is what reading and pronunciation are fundamentally all about" (p. 61).

813

solutions provide a start in decoding, but other layers emerge along the way. It is unfortunate that decoding instruction fades at about third or fourth grade, "just when the fun starts. "

The Decoding Curriculum THE ISSUES

What is one to believe? Opinions range widely, the facts do not give any position overwhelming support, and instructional programs and practices can be found for any of the prevailing motifs and various combinations thereof. Given some truth to each point of view, but also much to argue with, it seems appropriate to present our thoughts on some issues: • The questions are partly empirical, but not entirely. When proponents point to superior performance after phonics training, the evidence cannot be disregarded. On the other hand, empiricism has its limits. The advantage of phonics training is typically small, and many students continue to experience comprehension problems, especially in the later grades. Seldom does innovation spring from empirical stu­ dies of "what is," but rather when an innovator imagines "what might be. " • English spelling, complex on the surface, is not a simple alphabetic system, but is morphophonemic. The spelling­ sound patterns track "words" from the underlying layers of of the language (Venezky, 1970; Chomsky & Halle, 1968), and become clear only when one examines the historical evolution of the spoken language and parallel developments in writing. • Reliance on rote drill and practice as the primary vehicle for decoding instruction is probably a mistake. Rote instruction is slow in any event, but is especially problematic when "what is being learned" is complex, and when the instruc­ tional goal is transfered beyond the immediate domain of "what is taught." Kintsch (1979) remarks that "what hap­ pens in the later grades is not really a reading problem but a cognitive problem" (pp. 328-329). In fact, the intellectual demands in learning English spelling-sound relations are as significant as those in learning the structures of expository prose. The problem is aggravated by unsupervised seatwork, especially when students lack understanding of the purpose of a task (Anderson, 1982). • " What is learned" in decoding must include fluency, but it should also encompass understanding. The competent reader can not only pronounce a world like euchormonium but can also explain how he or she arrived at the results. • The acquisition of decoding skills and knowledge, while important in learning to read, need not be "the first thing taught. " It is probably a mistake to design an instructional program in which one component (e.g., decoding) becomes an unnecessary barrier to the acquisition of other com­ ponents (Resnick, 1979). The English spelling-sound system is, in short, a domain of knowledge. While challenging in its structure, it is by no means arbitrary. Solving this problem lies within the intellectual power of virtually every youngster, given appropriate condi­ tions. The problem does have an onionlike quality - the first

HISTORY AND ENGLISH SPELLING

As noted earlier, English writing reflects the history of the language (Bloomfield & Newmark, 1963 ; Gordon, 1972; Nist, 1966; Pei, 1968), and it is helpful to have an appreciation of how writing shaped herself to fit the history. Histories of the writing system are rare, and tend to be rather technical. Balmuth's (1982) book, designed specifically to treat the English writing system from a "phonics" perspective, provides useful informa­ tion (also see Chomsky & Halle, 1968; Fries, 1962 ; Gelb, 1963; Venezky, 1970; Wijk, 1966). Balmuth stops short of a " nuts and bolts" of instruction, and so we will sketch the major features of the English letter-sound system below. English spelling is fundamentally alphabetic, but this genera­ lization becomes clearer when the language is "carved" along its historical joints (Figure 28.2). Modern English contains words from virtually every language in the world, and hence a plethora of spelling patterns. Two of the contributors are especially important in reading instruction - Anglo-Saxon and the Romance languages (primarily French and Latin); Greek is significant for scholarship as well as some other familiar aspects of daily life (television, phonograph, microscope, dinosaur, and so on), but will not be covered because of limitation of space. LETTERS, SYLLABLES, AND WORDS

Decoding an unfamiliar word proceeds in three phases. Unless the word is quite short, the reader should first attempt to divide

GREEK

and others Special ized words, mostly in science, though some are common (television)

ROMANCE

Tech nical, " h igh-class" words ; used i n more formal settings, i ncluding books

ANGLO-SAXON

Commo n , everyday, down-to-earth words, used frequently in ordinary situations Fig. 28.2 The layers of the English language.

814

ROBERT CALFEE AND PRISCILLA DRUM

it into its basic morphemes, or meaning-bearing units. Pronunci­ ation and meaning are both easier with parts than with wholes. Compounding, in which two or more independent units are conjoined, is common in English - doghouse, uphold, and so on. Another pattern is affixation, with a root word modified by prefixes or suffixes - internationalization, thoughtfulness, and so on. Affixes are often bound morphemes, which exist only in combinations. The second step is to check for polysyllabic morphemes. If a morpheme has more than one vowel separated by consonants, then it is polysyllabic - napkin, inter-, poly-, and so on. Several strategies work about equally well for syllable division (Groff, 1 97 1 ; Ruddell, 1 974, p. 14). Finally, each morpheme or syllable is divided into letters or letter combinations, to each of which a " sound" or phonetic equivalent is assigned. The importance of the previous divisions now becomes apparent. Consider how you would pronounce nuthouse if divided nu-thouse rather than nut-house. Two caveats are in order : First, these stages describe the strategy of a skilled reader who has been taught these steps for attacking a new and difficult word. The skilled reader probably has direct access to frequently occuring words (Ehri & Roberts, 1979). Second, these steps are reversed in acquiring literacy. The young child is first taught the basic letter-sound correspon­ dences, then techniques for handling simple polysyllabic units, and then morphographic analysis. Each new step gives the reader access to a larger set of " long words. " THE ANGLO-SAXON CORE

English is grounded in the Anglo-Saxon heritage. Common­ place words, those most frequent in speech and print, come from this base. Familiarity should not lead to contempt, however; the Anglo-Saxon base is a complex structure in its own right. Frequently used words fall into two categories. In the first group are familiar terms comprising the stuff of everyday life, as described by Nist (1966): No matter whether a man is American, British, Canadian, Austra­ lian, New Zealander or South African, he still loves his mother,

father, brother, sister, wife, son and daughter; lifts his hand to his head, his cup to his mouth, his eye to heaven and his heart to God; hates his foes, likes his friends, kisses his kin and buries his dead; draws his breath, eats his bread, drinks his water, stands his watch, wipes his sweat, feels his sorrow, weeps his tears and sheds his blood; and all these things he thinks about and calls both good and bad. (p. 9)

The second category are function words - articles, preposi­ tions, conjunctions, some adverbs, and certain verbs. English grammar relies on sentence frames more than word endings: word order supported by function words. These terms, which have little substantive meaning beyond their functional role, tend toward unusual spellings - of, was, though, do,from, and so on. The irregularities arise primarily because these oldest words in the language retained their original spellings over centuries of changing pronunciation. Frequency of use led to constancy of spelling. The meaning of these words from the Anglo-Saxon core, the most frequent words in the language, are familiar to most children on entry to school. In many programs these are

the first words taught to the beginning reader, which means that students first encounter those words with the most irregu­ lar spellings. There is nonetheless order to Anglo-Saxon spelling patterns. Syllabic and morphographic patterns are relatively simple. Base morphemes generally contain one or two syllables - dog, cream, often, hundred, rabbit. Word compounds are common­ place - doughnut, greenhouse, peanut. Affixation also occurs. Suffixes mostly serve syntactic functions (teacher, funny, forgiv­ able). Prepositions are common as prefixes (undertake, input) and suffixes (blowup, knockdown). Both prepositions and base words in these combinations are familiar as stand-alone units. Anglo-Saxon letter-sound correspondences present some challenge. The phonology is complex, with more sounds than letters in the Roman alphabet. This mismatch was handled early in the development of English writing by (a) digraph spellings, with two letters representing a sound, and (b) markers, which cue alternate pronunciations of a spelling. The basic pattern is laid out in Table 28.3 (Venezky, 1970). Single-letter consonant spellings are virtually invariant - each letter stands for a single sound. Consonant blends are common and regular. Consonant diagraphs, combinations with h in the second position, handle sounds for which no letter exists. While consonants pose little difficulty to beginning readers, Anglo-Saxon vowels are another matter. Table 28.3 reveals that even here there is a structure. First, each single vowel spelling usually has one of two pronunciations, referred to as long and Table 28.3. Basic Structure of Letter-Sound Correspondences for Anglo-Saxon Words i n Engl ish CONSONANTS S i ngle Letter

Blended

Digraphs

Consistent and simple correspondences, easily learned:

Combi nations of single letter sounds :

Relatively few combi nations, each consistent :

b a t f i n p i ll

st a nd pr o ng spl i nt

ch atter sh are th eir, th i ng wh ere

VOWELS S i ngle Letter Long vs. Short mate Pete pin ing biter nodes cubed

mat pet pinning bitter nods cubbed

-r and -1 Affected

par her sir for

pare pal pall here _ _ _

Digraphs One sound : ai/ay maid , may ee meet oa boat oi/oy foi l , toy au/aw taut, law eu/ew feud, few Two sounds: ea breath breathe oo cook food roun d four ou /ow { cow snow

815

RESEARCH ON TEACHING READING

short, or free and checked. The contrast is fundamental for

decoding Anglo-Saxon vowels, and students are on their way to mastering the system when they have grasped this alternation. The most probable pronunciation of a single vowel letter is also marked in Anglo-Saxon spelling, as shown in Table 28.3. The marking often designated as the " final -e" rule, actually operates as follows: If a single vowel letter is followed by a consonant-vowel combina­ tion, it takes the long pronunciation; if the single vowel spelling is followed by a terminal consonant or by two or more consonants, it takes the short pronunciation.

Thus, the -e in mate, Pete, nodes and cubed is a vowel marker. At one time pronounced, it is "silent," or takes a schwa pronuncia­ tion (dozes). The doubled consonants in bitter, pinning and cubbed mark the short pronunciation of the vowel. Also shown under the single-letter vowel spellings are exam­ ples of a vowel plus -r and -/. These two consonants are actually semivowels, and fall between vowels and consonants in their pronunciation. Their spellings are probably best presented as combinations rather than blends. Finally are the vowel digraphs. These spellings, complex in their origins (see Balmuth, 1982), comprise a veritable melange, and are the most frequent source of complaints about the inconsistency of English spelling- though, through, rough, ought, bough, bound, hour - where is rhyme or reason? But Table 28.3 reveals order even here. One set of digraphs is consistently paired with a single sound. Digraphs in the second set take two distinctive sounds. In the latter instance, there is no pattern to the two sounds, nor are there any reliable markers for deciding between the two alternatives. Only three digraphs fall into this category, however, and the task of learning these correspondences should not be a major hurdle. Finally a handful of idiosyncratic pronunciations fall outside this structure - could, should, do, does, laugh, night, light, of, one, said, two, was, along with the set of -ough words. Some of these "weirdos " are frequent in print. The student encounters them often and will learn them by rote. Thus, to focus on the exceptions and ignore the general consistency of the Anglo­ Saxon base would seem a serious mistake. ROMANCE WORDS

Much of what the student learns from decoding Anglo-Saxon transfers to words of Romance origins. There are a few signifi­ cant modifications. Romance words entered English in two waves. First came the Norman invasion, at a time when for 200 years French was the official language of the land. The English accepted the French vocabulary, sometimes as a commonplace, but more often as a "fancy" alternative. French spelling had been regularized by the 10th century, and so remained relatively unchanged in crossing the Channel. English usage led to significant changes in pro­ nunciation, however, with consequences for the letter-sound correspondences. The second wave of Romance intrusions came during the English Renaissance, when scholars and scientists "borrowed" tens of thousands of Latin (and Greek) words. These had standard spellings kept constant by the printing press. Again,

the English pronunciation varied from the original, with impli­ cations for the spelling-sound relations. The most noticeable difference between Anglo-Saxon and Romance words is in length and apparent complexity. These are differences in morphology. Romance words tend to contain more morphemes, and include unfamiliar affixes - a-, e-, ex-, inter-, -ion, -ual, -ity, and so on. Transfer of word analysis strategies to Romance words is probably most likely when the student has a clearly articulated understanding of the strategy and is given some idea about the units. Three differences between Anglo-Saxon and Romance letter­ sound correspondences require note. First, long-short vowel alternations are still found, but the marking system is less consistent. Second, the stress patterns are complex. Romance words are far more likely to contain unstressed elements, in which the vowel becomes a schwa. These effects are seen in word pairs like sane/sanity, nation/national, audit/auditory. The third spelling-sound difference appears in suffixes of the general form consonant-vowel-vowel-consonant, as in -tion, -sion, -tial, and ·also -cious, -tious. The initial consonant­ consonant is pronounced /-sh/, the vowel digraph takes a schwa, and the final consonant is pronounced as usual. The cor­ respondence reflects a "laziness" in English pronunciation. Aside from these suffixes, vowel digraphs are less common than in Anglo-Saxon, and tend to be quite regular. Finally, the various languages do not operate in isolation. The declensions and conjugations of Anglo-Saxon English apply directly to words from all origins. Numerous combina­ tions blend elements from several sources. These combinations may be viewed as poor usage, but the vitality of the language suggests that purists will lose these battles. We can expect to hear that football is over-televised (with or without the hyphen), and we already know that many people are undernourished.

Research Topics in Decoding The preceding section attempted a broad view of two topics : (a) the current state of curriculum and instruction in decoding, and (b) a structural-historical review of English writing. The cov­ erage was normative rather than empirical. Insofar as the generalizations have merit, a tension exists between current practice and the structural foundations of English writing. The discussion of the writing system was intended to heighten that tension, on the grounds that research and practice may rest at present on too narrow a view of the system. The organization in the present section intentionally con­ founds two dimensions - (a) level of acquisition and (b) artifi­ cial versus natural settings for research. For each area, two questions merit attention: (a) Where do the studies fall on the continuum of beginning versus skilled reading? (b) Where do the studies fall on the continuum of laboratory investigation to naturalistic inquiry? ENTRY KNOWLEDGE

What do children know when they come to school that will help them in learning to decode English writing? How does this knowledge shape the instruction offered to students, and how does it influence progress during the primary grades? A serious

816

ROBERT CALFEE AND PRISCILLA DRUM

answer to these questions could take the entire chapter. We will hit some highlights. A few generalizations can be attempted without data. Chil­ dren come to school with a working command of a language. It may or may not match closely the language of the school. Children's capabilities include both reception and produc­ tion - they can understand and they can communicate. All four linguistic functions are operative - phonology, semantics, syn­ tax, and discourse. Children differ in their command of the language on entry to school (Heath, 1983). Some may lisp or have other speech impediments. Vocabulary extent and precision vary widely. Familiarity with formal language also covers a broad range. The only child of educated parents is probably more skilled at " grown-up" talk than the fifth child of a single parent. Children differ in their experience with books. Most pre­ schoolers in modern society have seen print - if only as a signal to their favorite fast-food chain. Some children have parents reading to them in the evenings. Some are fluent readers on kindergarten entry - certain cultures view such preparation as a parental duty. Some have "picked up reading" with little or no "teaching. " Differences at school entry predict response to instruction. An early tradition in educational research was the correlational study of reading readiness (Vellutino, 1979, surveys this work, and highlights the possibilities and limitations). From one perspective, the paradigm is self-serving. Some youngsters come to school better prepared than others. Those who are " ready" are given further reading instruction; those less well prepared are assigned to readiness training (generally not reading in­ struction). The predictions are then validated at the end of the first grade - the first group does much better than the second one. To be sure, it might be advisable to allocate more effort to those youngsters who are least "ready" (Calfee & Venezky, 1969). Intelligence is another gauge measuring differences between students. From his examination of correlational data, Jensen ( 1980, 1981) concluded that "the vast majority of poor readers [are poor] not because they lack decoding skills but because they are deficient in comprehension which, as measured by standard tests of reading comprehension, is largely a matter of g [generalized intelligence] " (1980, p. 325). Meta-analytic sur­ veys suggest that the correlation may be weaker than reported by Jensen (Hammill & McNutt, 1981 ). Stanovich, Cunn­ ingham, and Feeman (1984) found decoding skill (including phonological awareness; subsequent discussion) more strongly correlated with reading in the primary grades than was intelli­ gence; for older students, the pattern was more diffuse and generalized. Then there is the " letter name" literature. The preschooler's ability to recite the letter names, in English at least, is a strong predictor of reading achievement in the primary grades (Calfee, 1977; Venezk'y, 1 975); this correlation appears to be weakening with the advent of Head Start and " Sesame Street," and may depend on the culture (Nurses, 1979 ; Sulzby, 1983). Ehri ( 1983), in her review of this literature, highlighted instructional studies; students were taught letter names to determine any effect on reading acquisition. Little transfer was found, but flaws in the studies render any conclusion suspect.

This research is a vignette of sorts: Letter-name knowledge is correlated with success in learning to read. Why not teach letter names? Cause and effect are egregiously confused in this reasoning. Ehri (1983) concludes that "this is an empirical question to be answered with data, not logic" (p. 1 50). Sulzby ( 1983) argues that both data and logic are needed, and raises questions about the underlying causal mechanisms. A variation on the "entry" theme is the idea that many children enter school as readers through experience with printed materials in meaningful settings, from which they have discovered for themselves how to decode. As Smith (1976) has put it, "Children learn to read by reading." Evidence on this point is mostly anecdotal - "My child learned to read all by himself (or herself) ! " The evidence varies. If a child can compose original pieces, and if the spelling contains phonetic misspellings, then the youngster has prob­ ably acquired letter-sound principles (Chomsky, 1970; Read, 1975). In contrast, the preschooler who can read a book "with closed eyes" has learned something about literacy, but not necessarily about decoding. In a reasonably well-controlled study, Masonheimer, Drum, and Ehri ( 1984) showed that exposure to print did not neces­ sarily lead to an appreciation of English writing. Preschoolers ( 3-5 years old) were shown 10 popular logos (e.g., Mc­ Donalds's arches) ; children who could identify 8 of the 10 were tested on the saliency of the printed elements in the logos. As graphic and color cues were gradually eliminated until only the stylized writing remained, the children became less able to identify the logos. Performance was bimodal; 95% of the preschoolers could read none of the isolated words, while the remaining 5% could read virtually all of them. Masonheimer et al. (1984) conclude : Some authorities (Goodman & Goodman, 1979; Goodman & Altwerger, 1981; Harste, Burke, & Woodward, 1982) [hold that] reading begins during the preschool years when children become able to identify print frequently seen in their environment. . . . but environmental print does not contribute to reading acquisition [when] there is no " press" on the subject to look beyond the cues which are easiest to discern and most obvious (e.g., golden arches) and begin attending to alphabetic cues. (p. 258)

Most children need some guidance in figuring out the " writing game. " THE TECHNICAL VOCABULARY OF READING

Children differ on entry to school in knowledge of technical concepts and vocabulary in decoding. The concepts of word and sound are critical. Skilled readers take for granted several conventions variously labeled "language awareness" or "meta­ linguistic knowledge" (Downing & Valtin, 1984). With reading, says Ehri (1984), a person gains a new perspective on language and a new technical language for discussing spoken language: When children learn to read printed language, they become able to visualize what they are saying and hearing. When children learn to read clocks and calendars, they acquire a visual means of represent­ ing the passage of time. [When they] learn to read music, they become able to visualize what is sung or played on an instrument. . . . Acquisition of a spatial model . . . enables the possessor to hold

RESEARCH ON TEACHING READING

onto and keep track of phenomona which themselves leave no trace or have no permanence, [and] . . . imposes organization upon the phenomena by specifying units, subunits and interrelationships that might otherwise be difficult to detect or discriminate. (p. 1 19)

Writing and speaking differ, as noted earlier. In speech, "word" is poorly defined. Meaning-bearing elements or mor­ phemes are not the same as words. In print, a word is a letter string bounded by blank spaces or punctuation. In speech, " sounds " or phonemes seldom come to conscious awareness; indeed, psycholinguistic analyses of the phonetic stream show that the component phonemes overlay one another, so that isolating a single phoneme is unnatural if not impossible (Liberman, Cooper, Shankweiler, & Studdert-Kennedy, 1967). In speech, phonemes and morphemes come in streams that express thoughts, overlaid by prosodic and intonational cues. In print, this stream is chopped into chunks that are only partly meaningful, and the stress patterns and pauses so vital in speech are replaced by punctuation symbols, a weak substitute at best (Bolinger, 1975). What do children know of the technical language of writing when they come to school - sentences, words, sounds, conson­ ants, vowels, and so on? Indeed what is their concept of reading? The answer seems to be that most children know little, and what they know may be wrong. Reid (1966) found from interviews that prereaders lacked "any specific expectancies of what reading was going to be like, of what the activity consisted in, of the purpose and use of it. . . . [Many] were not even clear whether one 'read' the pictures or other 'marks' on the page" (pp. 58, 60). Some of these concepts are relatively easy to acquire (e.g., word), but others are much more difficult to grasp. Reviews by Downing and Valtin ( 1984) and Tunmer, Pratt, and Herriman (1984) provide detail on these generalizations. Valtin (1984) lists three metaconcepts in acquiring literacy : "(1) Understanding the nature of reading and writing, e.g., that written language represents . . . speech; (2) grasping the alpha­ betic principle of our script; and (3) conscious and deliberate mastery of language" (p. 256). Tunmer et al. (1984) organize metalevel facets according to standard linguistic categories - phonological awareness, knowledge of " the word," and lexical and pragmatic awareness. They also stress the impor­ tance of general awareness of the nature of literacy. They conclude that "children who have listened to printed text read by parents or siblings are more likely to benefit from reading instruction for two reasons; they are more familiar with the use of textlike language, and they are more apt to have a clearer concept of what reading is all about" (p. 1 53). Several studies show that, while prereaders know little about the " word" when they enter school, the concept is quickly acquired. Ehri ( 1975) studied " word consciousness" in pre­ schoolers, kindergartners, and first-graders, using a variety of tasks and materials. The first two groups were still confused about wordness at the end of the school year, especially functional words; first-graders did much better. It appears that experience with print helps the child grasp the conventions of print. Similar findings have been reported by Downing and Oliver (1973- 1974). Bowey, Tunmer, and Pratt (1981) suggest that previous investigations may have both over- and underestimated chil-

817

dren's knowledge. They note (as does Henderson, 1977) that it is one thing to perform tasks requiring a tacit concept of wordness, and another matter to articulate that knowledge. In three year-end experiments with preschoolers, first-graders, and second-graders (there is apparently no kindergarten in Australia), they systematically explored children's performance on various contrasts (words versus sounds, function versus content words, and words versus phrases). After an initial assessment, students received brief training with feedback, and were then retested. With more schooling, students had less need of training even on difficult tasks. Students whose performance was initially poor improved substantially, even with short-term training. The authors conclude that "the significant training effect . . . suggests that teachers may, with relatively little effort, succeed in teaching children the technical vocabulary of reading instruction" (p. 5 1 1). Sounds were more problematic. A substantial literature now exists on tasks variously labeled as phonetic awareness or segmentation. Reviews by Downing and Valtin (1984) and Tunmer et al. ( 1984) cover the topic, which has also been surveyed by Lewkowicz (1 980), Jorm and Share (1 983), and Routh and Fox ( 1984). Lewkowicz' analysis is especially com­ prehensive. Included under the rubric of phonetic awareness is a range of activities, from rhyming through isolation of sounds ("What is the first sound in /fish/?"), to Pig Latin. Although the tasks vary in complexity and clarity, two subsets seem particularly pertinent to decoding - segmentation and blending. Seg­ mentation requires the student to analyze the sounds in a word ; blending requires that the sounds be combined. Several generalizations can be drawn : (a) Phonetic aware­ ness on entry to school is correlated with later reading achieve­ ment, more strongly than virtually any other entering index except perhaps vocabulary knowledge; (b) young children have considerable difficulty in understanding what is intended on phonetic awareness tasks, unless they have already learned to read; and (c) under the right conditions training can be effective, though it is not easy to specify these conditions. Children do gain competence in segmenting and blending as they learn to read, but cause and effect are not clear - indeed, the relation may be symmetric. Some reading series, especially those with a strong phonics component, incorporate exercises in segmentation and blend­ ing. Experimental programs (Lindamood & Lindamood, 1969; Rosner, 1972; Venezky, Green, & Leslie, 1975 ; Wallach & Wallach, 1976; Williams, 1980) especially emphasize these areas. Some experimental studies have been conducted (Calfee, 1977; Lewkowicz, 1980, p. 693ft), but few are definitive. Train­ ing appears more effective and efficient when articulation is invoked, when pronunciation is slowed and overemphasized, when irrelevant aspects of the problem are minimized, and when learning is supported by mnemonics. Teachers are probably aware that children differ on this dimension; we know less about how teachers conceptualize these differences. Nor do we know about teachers' metalinguis­ tic knowledge, their concepts of the technical aspects _of the relations between sounds and print. In one survey, Pidgeon (1984) obtained data from "reception" (i.e., kindergarten and first grade) teachers in Great Britain. Among his conclusions:

818

ROBERT CALFEE AND PRISCILLA DRUM

Teachers mistakenly assumed that students had prior know­ ledge about the technical aspects of reading; they were gener­ ally more concerned with planning instructional activities than with assessment of student learning; learning relied more on associations than on understanding; and decoding was gener­ ally taught from sight to sound (see the printed word and pronounce it). These generalizations are troubling if true - the Pidgeon survey has serious limitations, but the pattern seems consistent with other observations (Duffy & Roehler, 1982). What is lacking in these studies is curricular specificity and a linkage to the instructional features of the teacher's manual. The upshot is that from the beginning of their experience in learning to read, many students appear confused about the fundamental nature of literacy, and especially about decoding. They may or may not be able to distinguish the elements that comprise words; they eventually gain some familiarity with letters and the notion that words are letter strings; many will never grasp what in the speech stream is to be associated with letters. For these students, reading is the learning of words as wholes, a strategy that eventually fails. Some years ago Vernon (1957) expressed a concern about reading disability that has a similar ring: The fundamental and basic characteristic of reading disability seems to be cognitive confusion [about the relations between successions of printed words and phonetic patterns] . The employment of reasoning is almost certainly involved in understanding the variable associations between printed and sounded letters . . . . these associa­ tions may be acquired through rote learning. But, even if this is possible with very simple letter-phoneme associations, the more complex associations and the correct application of the rules of spelling necessitate intelligent comprehension. (p. 82)

The point will be reiterated at the end of this section - decod­ ing, from the initial linkages between letters and sounds to the articulate awareness of the underlying morphophonemic struc­ ture, is a difficult intellectual challenge. To construe this task as the rote learning of piecemeal objectives may seriously mislead both students and teachers. " MINIATURE" DECODING STUDIES

Research in natural classroom settings is tough. It is under­ standable that investigators should turn to analogues. In minia­ ture decoding studies, the early stages of decoding acquisition are studied under conditions that allow careful control of relevant factors (e.g., Jeffreys & Samuels, 1967; Silberman, 1967). These studies generally entail training of individual students on a relatively small set of target words. The subjects may be children, but college students have also been used. This paradigm has its strengths and weaknesses; recent studies show advances over earlier efforts. For instance, Dahl ( 1979) factorially investigated three methods to improve the decoding skills of below-average second-graders. The subjects' tasks were to read passages aloud with fluency, to recognize words presented in isolation, and to answer several sets of comprehensive questions. The Hypothesis/Test method required the student to com­ bine spelling and contextual cues to generate a plausible pronunciation for an unknown word and to test out the

hypothesis. The method resembles the approach of the skilled reader, but slowed down. In Repeated/Readings, selections were read and reread aloud until the student was both accurate and fast. The Flashed/Word method amounted to rote training. The design included eight combinations of the presence or absence of these methods, with four students assigned to each condition. On most measures, the Hypothesis/Test treatment had a large positive effect, the Flashed/Word treatment had none, and Repeated/Reading fell in between. Analysis seemed to win out over rote learning. The researcher had predicted matches between treatments and measures - Hypothesis/Test should benefit doze and comprehension measures, Repeated/Reading the oral reading tasks, and Flashed/Words the word identifica­ tion tasks. The first two predictions held; the third did not. Virtually no interactions were significant. The study has promise as an experimental analogue, and the findings provide interesting leads. To be sure, instructional time may not have been adequately controlled, and the investigator failed to incorporate student entry factors in the design, thus weakening the power of the experiment and forestalling an examination of aptitude-treatment interactions. In a second example (Kibby, 1979), 144 first-graders were each taught 12 words in four sessions of 3 words each. The training factors were Method (phonics or sight-word), Practice (recognition or production), and Instruction (no correction, correction, and " mastery"). Materials were confounded with method ; Phonics words were at, bat, hat, mat, rat, sat, while the Sight-Word list included boy, children, farm, house, rabbit, wagon. A within-subject design was used ; each student was tested on all the training combinations. A variety of dependent measures was included in the battery. The results were somewhat complex, but several generaliza­ tions can be made. First, training performance was higher in the Sight-Word condition, sensible given the low interitem similar­ ity in this condition. Students learned the responses, but had difficulty pairing them with the appropriate stimuli. (We call this the " Pug" effect, after the difficulty experienced by a first­ grader in a phonics program; when uncertain he would regu­ larly say " Pug," the central character of the series.) Second, correction had only a slight positive effect on train­ ing and no transfer benefits; there were only six training trials. The "mastery" condition entailed exposure without feedback; if a student was already having trouble learning the list, further training had no benefit whatsoever. Finally, student perfor­ mance was lower during training under the Production condi­ tion (" say the word") than under the Recognition condition ("is it house or rabbit?"), but Production training paid off during the transfer tests, especially when these required production. This study represents a significant effort to investigate the early stages of decoding acquisition, but is also flawed in several ways. Certain problems may be arguable; using a within­ subjects design in a learning experiment always raises the possibility of transfer from one condition to another, though such transfer was not evident in this study. The amount of training was quite modest, a problem only if students fail to master the task, as in this study. Blanchard and McNinch (1980) note in critiquing another study that " it seems doubtful

RESEARCH ON TEACHING READING

that poor readers, as a result of a one-time, single-word decoding procedure . . . can compensate for a five-year differ­ ence in reading performance." (p. 560) More serious in both of the preceding studies is the absence of any model of the task - what was the curriculum, what pedagogy was intended, and what was the process of learning and transfer? Given the low levels of performance, one might expect the investigator to review the task for weaknesses in the instructional design. Instead, the purpose seemed limited em­ pirical investigation of the factor set. Empiricism has its limits. Finally, let us consider the study by Anderson, Mason, and Shirey ( 1984). Two observational studies are reported in the paper; we will focus on the first and more extensive of the two. The investigation is both extensive and complex, and might have been discussed elsewhere in the chapter; it is introduced here as a transition between "miniature" experimental and naturalistic studies. As the authors note, the investigation had the potential "to study the independent effects of variables that are correlated in nature, . . . to get a preliminary look at the causal dynamics in reading groups, . . . and to investigate the interplay of a large number of factors that converge at given moments to determine [performance]" (p. 10). The child's task was to learn 36 rather complex sentences (Anderson et al., 1984, p. 1 1): The hungry children were in the kitchen helping mother make donuts. The old shoes were put away in the back of the closet. Green blood ran out when the boy shot an arrow through the monster's head. The students, more than 250 third-graders, were taught in groups of 4. In the Meaning condition, the student predicted what might happen next after each sentence; oral miscues were ignored or corrected quickly and without comment. In the Accuracy condition, miscues were corrected and the student had to reread the sentence to a criterion of accuracy and fluency. Other factors in the design included the makeup of the group (homogeneous or heterogeneous), active or passive turn taking, reading ability, reading fluency, and the readability and interestingness of the sentences. The measures included sen­ tence recall (given " old shoes" as a clue, the student recalled the rest of the sentence) and decoding performance (16 hard words from the sentences). The data were subjected to multiple regression analyses - more than 30 main effects and interactions were assessed. Attention, consonant with the experimenters' purposes, focused on sentence recall. In general, the Meaning treatment had a positive effect on recall, especially for students at higher levels of reading achievement. Interesting sentences were better recalled than dull ones. Difficult sentences were better recalled in the Accuracy condition - students had to read these sentences several times, and so were better practiced on them. In homoge­ neous high-ability groups receiving the Accuracy treatment, recall was especially poor- they read each sentence quickly, and then went on to the next. Finally, fluency interacted with the treatment on the decoding measure - the Accuracy condi­ tion helped students who lacked fluency in decoding.

819

This investigation illustrates the costs and benefits of at­ tempting a controlled study under conditions that approximate the regular classroom. Several of the findings are quite informa­ tive, but multicollinearity in the predictors led to some ambi­ guities of interpretation. In this investigation, as in Dahl's, interactions moderated certain main effects, but the structure of the findings was relatively simple. Two limitations deserve note. First, the strong emphasis on meaning led the investigators to give less attention to the assessment of other outcomes. Students tend to learn what they are taught. If they are taught to focus on meaning, then they will typically do better on a test of meaning than if their attention is directed to the surface aspects of the material. Beginning readers need to understand what they read, but fluent and accurate decoding is also important. The task for instructional design is the creation of programs that meet both of these needs. The second limitation is related to the adequacy of the instructional treatments. Students had opportunities for prac­ tice, but relatively little instruction. This limitation may explain the rather substantial aptitude-treatment interaction - the abler students gained more from the Meaning treatment. Op­ portunities for practice help when the student already has some understanding. As in the Kibby experiment, the present study would have been more informative if grounded in a theoretical framework of the curriculum and the pedagogy of the task. The "miniature" studies reviewed above are clearly no longer small. In their design, in the student sample, in the tasks and measures of performance, they approach the complexity and reach of naturalistic studies, while attempting the precise con­ trol of laboratory studies. We can expect to see increasing payoff from these efforts if present trends continue. WHAT IS LEARNED FROM DECODING INSTRUCTION?

Studies of word recognition have a long history in experimental psychology (Smith & Spoehr, 1974; Venezky, 1977). Recent years have seen a new development in efforts to relate word recognition processes to instructional experiences. The idea, often implicit, is that no single model determines word recogni­ tion, but that differences between and within subjects will reflect knowledge, skills, and strategies acquired during the elemen­ tary grades. Two major research questions have emerged in this domain. First, when a subject "reads" a word, does the translation entail direct access to a visual pattern, or is it moderated by a phonological letter-to-sound activity, or do both processes operate? Second, in those situations in which translation is phonological, does the coding depend on spelling-sound rules or on analogies to familiar words? The role of instruction in establishing codes should be especially marked in English writing. If students have not learned the spelling-sound patterns (perhaps because they were not taught), then one would not expect to see evidence of phonological codes. For these "Chinese" readers (Baron & Strawson, 1976; Baron, Treiman, Wilf, & Kellman, 1980), pronunciation speed should depend only on the number of times the child has previously encountered the word. The reasoning behind these hypotheses is heuristic rather than

820

ROBERT CALFEE AND PRISCILLA DRUM

formal, but the description does fit research findings for poor readers (e.g., Carnine, Carnine, & Gersten, 1984; Lesgold & Curtis, 1981, pp. 340ff). A clearer picture awaits more precise studies of the relation between what has been taught and what appears to have been learned. Research on visual versus phonological codes in word recog­ nition has been reviewed by McCusker, Hillinger, and Bias (1981), and by Jorm and Share (1983). McCusker et al. (1981), from studies of skilled adult readers, conclude that "the data point to a dual access model in which high-frequency words enjoy high-speed access via a visually based representation, whereas low-frequency words are accessed using a slower, phonological recoding process" (p. 217). As a teacher might put it, some words are learned by sight (those seen often in print), while others (less frequent) are handled by word attack strate­ gies. Students are taught both approaches, and it is reassuring to see research findings consistent with instructional practice. To be sure, these results hold for the schools' success stories (college students). This area continues to attract attention; articles by Underwood and Thwaites (1982), Taft (1984), and Rossmeisl and Theios (1982) reflect the sophistication with which researchers now determine the codes used by skilled readers in high-speed tasks. Jorm and Share (1983), viewing the research from a develop­ mental and instructional perspective, propose that "although phonological recoding may play a minor role in skilled adult reading, it plays a critical role in helping the child become a skilled reader" (p. 103). Their concern is primarily on the role of phonological recoding in reading acquisition, and so they give less attention to studies of visual coding in young readers. They contrast two theoretical positions: Stage theory (readers begin by using phonological recoding, and with practice resort to the faster access available through direct visual codes); and Com­ parator theory (a "race-horse" model in which visual and phonological codes compete with one another; the winner depends on experience and situational factors). Stage theory comes in two versions, the probabilistic version described above, and a strong version, easily rejected, in which readers move irreversibly from one stage to the next. In presenting these models and reviewing the literature, Jorm and Share pay little attention to instructional practices. In English-speaking countries, one finds two or three dominant approaches to teaching word recognition. Present practice is typically rather eclectic; the child begins with a set of sight words learned by rote and useful in round-robin reading. Phonics instruction follows, with emphasis on word attack strategies during the midelementary grades but also stress on fluent oral reading. In the late elementary grades and onward, silent reading (fostering visual coding?) predominates. One can find variations. Some programs stress phonics from the begin­ ning, whereas others emphasize sight recognition and "reading for meaning." The words from these latter programs typically provide the student little basis for discovering letter-sound generalizations. Finally, individual teachers may do much to reconfigure any given program. Several studies indicate that inadequate phonological recod­ ing may be a hallmark of the poor reader (Jorm & Share, 1983, p. 133ff). To test this skill, the subject is asked to read a synthetic or nonsense word, one that conforms to the underly-

ing structure of English spelling but has no assigned meaning. The usual finding, mentioned later in the section, is that poor readers are at a special disadvantage with words that they have never seen before. A few studies have directly addressed the relation between how words are coded in a person's memory and the decoding instruction and other relevant experiences of the person. Each investigation has its limitations, but the overall pattern makes sense. Nelson (1974), for instance, studied the oral reading perfor­ mance of kindergartners taught by a phonics or sight-word basal series. A factor analysis of the test battery yielded two factors, one of which seemed to be a decoding index and the other a sight-word index. These factor scores were significantly different for students in the two basal programs, and in the predicted directions. (Barr, 1972, in a study to be described later, reports a similar pattern). Alegria, Pignot, and Morais (1982) assessed the phonetic segmentation skills of teachers taught in a phonics or a sight-word program. The segmentation task was quite difficult, requiring the student to reverse the sounds in a word (so that TOE became OAT). Students in the phonics program were four times more successful than those in the sight-word program. Finally, Treiman and Baron (1983), investigating the effects of phonemic awareness training on reading acquisition, also found training experiences reflected in the students' coding. Several assessment and training studies reveal students using both visual and phonological codes, depending on their level of experience and the conditions of assessment. Waters, Seidenberg, and Bruck ( 1 984) illustrate one approach. They note that English, like most modern written languages, relies on the alphabetic principle (hence the tendency for readers to rely on phonological coding when in need), but parallel systems exist (e.g., signs like $, %, # , and "weirdos" in English spelling), so that some words must be learned as visual codes. In their study, spellings of the test words were either regular (best, dust), exceptional (give, bowl), or "strange" (once, view); word fre­ quency was also controlled. The subject's tasks were to pro­ nounce each word as quickly as possible, and (in a separate block of trials) to identify each stimulus as word or nonword. Age (fourth grade, fifth grade, and adult) and reading level (high or low) were also controlled. Our chief interest is in the pronunciation task. Familiar words, no matter how spelled, were pronounced quickly by the better readers (500 milliseconds for adults, 750 milliseconds for fourth-graders). Regularly spelled unfamiliar words were also quickly pronounced. Unfamiliar words, especially those with strange spellings, took an additional 50-100 milliseconds. Low­ ability fourth-graders performed differently in several respects. First, they are much slower in general; familiar words, which they handled most quickly and without regard to spelling pattern, required 1200 milliseconds or more. Unfamiliar words took longer, even if regularly spelled (1500 milliseconds), and especially if "strange" (2500 milliseconds). Waters et al. interpret the findings as showing "greater involvement of phonological information in the early stages of reading, . . . [but] fast decoders treat a larger class of items as 'high-frequency' [i.e., familiar] items" (pp. 302ft). These conclu­ sions are somewhat compromised by limitations in the stimulus words (all were short, Anglo-Saxon, and single syllable, and the

RESEARCH ON TEACHING READING classification of spelling patterns was rather crude), and by the presence of a floor effect for the better reade�s. �onetheless, the _ �t dual findings do support the previous generahzat10n abo coding systems, depending on the reader's charactenst1cs, the _ spelling patterns, and the task. The article contains a substan­ tial number of references to earlier studies along the same Imes. Assuming that a student has been taught spelling patterns, what is acquired-rules or analogies? One might expect the answer to this question to be placed in the context of a careful analysis of what is taught. As noted earlier, patterns more than rules are emphasized in teaching. Glushko (1981) proposed that readers rely solely on analogy: "As letters in a word are identified, an entire neighborhood of words that share ortho­ graphic features is activated in memory, and the pronunciation emerges through the coordination and synthesis of many partially activated phonological representations" (p. 62). He presents findings from three "mini-experiments" to support this conclusion. The"analogy" model has been criticized - Jorm and Share (1983) point out that the beginning reader does not have too many patterns to activate, and point to conflicting findings by Andrews (1982); Barber and Millar (1982) note flaws in Glushko's stimulus words. On the other hand, the issues raised by Glushko seem of fundamental importance-it is perhaps unfortunate that the question is posed as "What do readers do?" rather than "How can we help readers learn what they need to do?" Certainly analogy is a powerful approach for learning, and for training as well (Silberman, 1967; also see Cunningham, 1975-1976, for a demonstration of the effective­ ness of the analogical approach in a "miniature" decoding study). The same can be said about rules, generalizations, and regularities. Indeed, the contrast is not either-or; analogy and rule are better construed as endpoints on a continuum. The task for assessing and training decoding would seem to be establish­ ment of a clear and workable structure for describing the regularities, followed by empirical investigation of the condi­ tions under which students can be taught the essentials of that structure. FLUENCY IN DECODING

The competent reader decodes with accuracy, but with fluency as well. Perfetti and Lesgold (1979) have reviewed this area, and summarize several studies by Perfetti and his colleagues de­ limiting various conditions under which speed does or does not indicate competence. Rapid response is sometimes only a sign of impulsivity (Kagan, Rosman, Day, Albert, & Phillips, 1964), but it is more often a marker of competence. Curtis (1980) tested elementary students on a variety of reading skills, and found that by fifth grade any speeded task differentiated the more from the less able readers (also see Biemiller, 1977-1978. Speed is an end in its own right, but it also is an indicator of other advantages. LaBerge and Samuels (1974) were among the first in recent times to point out the benefits of "automaticity" in decoding. Their model of the reading process incorporated several stages between the translation of letters and the final extraction of meaning; as decoding became an automatic response, intermediate stages could be bypassed, relieving de­ mands on human attention. When decoding requires attention,

821

then something else (presumably comprehension) must "'.ait. Perfetti and Lesgold (1979; also see Liberman & Shankwe1ler, 1979) have identified "several components �n reading t�at, when not fully developed, could increase [sic] the workmg memory bottleneck ... : access to long-term memory, speed an� automation of decoding, and efficiency of readmg strategies (p. 59). . . Reducing demands on short-term memory 1s certamly one advantage of fluency. Another benefit is the ease in understand­ ing text when the rate is adequate (a related issue is the loss of prosodic and suprasegmental information when words are uttered one at a time; slow readers do not "read with mean­ ing"). Five syllables per second is the rate of normal speech; 5 to 10 units can be retained for 10 to 15 seconds without loss. Under normal conditions the "listener" can easily organize a sentence and transfer the gist to long-term memory. When the rate is greatly reduced, problems arise: "The computation of sentence structure must take place within a certain limited period. If a sentence is uttered too slowly-say one word every 5 seconds-its structure collapses, . .. [and what remains] is a string of words" (McNeill, 1968). Poor readers are slow readers; what can be done to alleviate this problem? One answer is to "practice, practice, practice." Huey (1968) put it more elegantly: "Repetition progressively frees the mind from attention to details, makes facile the total act, shortens the time, and reduces the extent to which con­ sciousness must concern itself with the process" (p. 104). Jenkins and Pany (1981) review several studies on this issue. The general finding seems to be that practice in pronouncing a list of words speeds performance on that list, but with little transfer to other words and few benefits for comprehension. Most of these studies used a relatively short list of words, practice did not include instruction, and training was relatively short. Fleisher, Jenkins, and Pany (1979), for example, trained fourth- and fifth-graders on speeded pronunciation of a list of 75 words.In the first experiment, words were presented on flash cards one at a time; in the second study the words were embedded in phrases. After training, students read passages containing the words, and both reading speed and comprehen­ sion were assessed. The training was effective. Poor readers without training read words in isolation at about half the rate of good readers; with training, this difference became negligible. The training did not totally eliminate differences; trained poor readers improved, but still performed more slowly than skilled readers when reading passages containing the words. The training had no effect on comprehension performance -none whatsoever. Stu­ dents in this situation learned only what they were taught. Blanchard (1980), on the other hand, found that providing flash-card training to extremely poor readers did have a positive effects on comprehension-extent of need may also matter. Samuels (1979; also Carver & Hoffman, 1981; C. Chomsky, 1978) proposed the method of "repeated readings" to promote fluency. The student (or perhaps another skilled reader) at­ tempts a first reading, works on any problems, and reads it again until the result is "smooth." Dahl (1979) evaluated this method in her study, and found benefits on all measures of

822

ROBERT CALFEE AND PRISCILLA DRUM

decoding speed and accuracy (errors were cut in half, and speed increased by 50%). Improvements in comprehension were smaller. Finally, what happens to reading speed during silent read­ ing? In particular,are oral and silent reading similar operations in good and poor readers? Juel and Holmes (1981) illustrate how information-processing techniques can be used to address this question (they also review the earlier literature on the topic). Second- and fifth-graders of high and low reading achievement read 64 sentences in which both word and sen­ tence factors were systematically varied (for words, the factors included spelling-sound regularity, number of syllables, fre­ quency, and abstractness; for sentences, the reversibility of subject and predicate nouns). Word factors had substantial effects on oral reading, especially for the poor readers, who were much slower than the good readers (7.4 versus 3.3 seconds/sentence). The effects were less noticeable in silent reading, and the differences between poor and good readers were smaller (4.8 versus 3.2 seconds/ sentence). Comprehension, as measured by a picture-recogni­ tion task, did not differ between good and poor readers, nor between oral and silent reading. The authors note that "the added attention expended on achieving oral pronunciations [does not result in] less time spent on sentence comprehension. Rather, it appears simply to increase poor readers' overall processing time" (p. 560). The purpose of oral reading exercises is an issue worth further examination. Fluency may correlate with other perfor­ mance indicators, but cause and effect are unclear. As in the case of letter names, the direct instructional approach misses the point. Training the student to "read words fast" may succeed in some purposes but fail in others. Recommendations promoting rereading and greater reliance on silent reading, however, would appear to warrant more careful investigation. DECODING AND CONTEXT

Several studies have found that words are read aloud more quickly in the context of a sentence or passage than in isolation (e.g., Fleisher et al., 1979; Stanovich et al., 1984). Advocates of reading for meaning reinforce the point, noting that syntactic and semantic clues allow the reader to play a "psycholinguistic guessing game" (Goodman, 1970) rather than struggle with the nonsense of print. How are decoding and comprehension related? One finds strong differences of opinion, and individual positions are not always clear. Perfetti and Lesgold (1979), for instance, state: Skilled reading can be partly understood as a set of interrelated component processes, ... [such that] a gain in one subskill allows gains in other subskills, that an insufficiently developed subskill may limit the apparent inadequacy of other subskills, and that the processes underlying the skills are difficult to study in isolation. (p. 57) This statement is then qualified: "The component processes are isolable in principle although interrelated in practice, ... [and] the situation . . . is one of structural independence but functional interdependence" (p. 58). Perfetti and Lesgold rely on analogy to a hi-fi music system: In operation (i.e., when a

reader is reading or a hi-fi playing) processes are likely to be complexly interrelated. Nonetheless, it is still possible to iden­ tify the distinctive functional components that underlie observable performance. And so what does happen to decoding when words are read aloud in context? What is the character of the interaction? Perfetti and Roth (1981; Perfetti, Goldman, & Hogaboam, 1979) discuss several investigations of this question. The sub­ jects' task was to read (or listen to or study) part of a passage, then pronounce as quickly as possible a word displayed on a visual display. The factors included context (present or not), word frequency and length, and reading achievement of the subjects (all in third to fifth grade)." A major result was that less skilled readers benefitted from story context at least to the same extent as did skilled readers. . .. [In particular], less skilled readers showed very large facilitation for long low-frequency words" (pp.273,275). The good readers could decode all of the words very quickly (about 500 milliseconds/word), and a floor effect may exist: "Skilled readers are already so good at word identification that context is of little consequence" (Perfetti et al., 1979, p. 281). The second experiment performed by Perfetti et al. showed that for less skilled readers the text had to be highly supportive for facilitation to occur. If the target word was not readily predictable from the context, then poor readers were not helped by context. Moreover, "subjects who were faster at word identification tended to be more accurate at predicting the next word in a story. Skilled readers thus appear to have at least two advantages. They are more efficient at context-free verbal coding and more able to use context in anticipating words" (p. 283). The question remains - do these results indicate complex interactions between processes, or do they reflect confounded variations in students' instructional experiences? That is, if the student has difficulty in becoming a fluent decoder, then the chances may be slim that he or she will be taught comprehen­ sion strategies-thus the "double whammy." Juel (1980; also see Juel, 1983, for a related study) also investigated the effects of context on decoding performance. She laid out the opposing theories. One position holds that skilled readers are predominantly "text-driven"; they use cues from the printed text (whether through phonological or visual cues is irrelevant). Poor readers are "concept-driven"; they rely less on print and more on their sense of contextual meaning. The contrasting position is that skilled readers make extensive use of context, turning to the printed material only when their predictions fail, whereas poor readers rely mainly on the text. Second- and third-graders, either high or low in reading achievement, were presented words varying in spelling-sound regularity, number of syllables, and frequency, in either a strong or weak sentence context, or in isolation. The dependent measure was the percentage of correct pronunciations. When the task was to pronounce isolated words, performance was strongly influenced by all factors in the design. The differences were much smaller when supporting context was provided. This pattern was most marked for low-ability students; high ability students did not require contextual support. Moreover, low­ frequency, hard-to-decode words were difficult for all groups, quite apart from context. In Stanovich et al.(1984) context was

RESEARCH ON TEACHING READING

provided by coherent and random paragraphs. The subjects, _ first-graders, were tested in the late fall and spnng. Coherent context aided both skilled and unskilled readers, but the benefits were much greater for the former and during the spring rather than fall testing. The contrast between these two sets of findings may reflect the relative skill of students and the difficulty of the words. The preceding studies give some idea of findings from re­ search on the effects of context on decoding. Several other investigations could be added to this list: Ehri has been active in this area, finding in one study that contextual training of function words led to a better grasp of syntactic/semantic identities (Ehri & Wilce, 1980), and in another that the same principle held for content words (Ehri & Roberts, 1979). Willows and Ryan (1981) explored the effect on oral reading of deleting words from the text or transforming text by a physical rotation (skilled readers were better at "filling the gaps"). Nemko (1984) found that training students to decode words in isolation or in context had differential transfer effects (if the goal is context-free decoding, train in isolation). Rash, Johnson, and Gleadow (1984), in a study immediately preceding Nemko's, came to the opposite conclusion (the study has serious limitations - students learned "Nicki's tooth fell out" and "Our television needs fixing" as sentences or as isolated words; not surprisingly, the two sentences were easier to learn than eight unrelated words). This expanding body of research raises questions as well as providing some answers. The questions motivating the research are only partly illuminated by the findings. Context helps the reader in decoding under some conditions, but not others. Both good and poor readers use context; poor readers have a greater need to rely on context, but good readers seem better able to use the surrounding materials if they have a need. These conclusions guide assessment more than training- a student's decoding skills are estimated with greater validity in isolation than in context. The role of context during instruction is more problematic. Few studies address this issue, and those that do provide little guidance. There is work to be done in this area, both theoretical and pragmatic. NATURALISTIC STUDIES OF DECODING

Perhaps because it is easier to "see" decoding instruction than other reading activities, several investigations of decoding under natural classroom conditions have been reported, some of which will be reported in this section. Several large-scale investigations are not discussed because they are covered elsewhere in the Handbook, or lack precise measures of decod­ ing (e.g., Berliner & Rosenshine, 1977; McDonald & Elias, 1975; Stallings, 1975). We begin with the study by Barr and Dreeben (1983), in many respects an exemplar. The data were collected by Barr in the early 1970s (Barr, 1972, 1973-1974, 1974-1975). The sam­ ple, designed to cover a range of instructional and organiza­ tional conditions, and a variety of student entry skills, included about 150 students from 15 first grade classes in six schools and three districts. Decoding was assessed in a variety of ways, the curriculum was carefully analyzed, and instruction was moni­ tored by classroom observations.

823

Barr and Dreeben present a detailed secondary analysis of the data. The major focus of the report was the social organiza­ tion of the classes. Students are generally not assigned at random to reading groups, and once a student is assigned to a group, several consequences ensue. One suspects that similar generalizations hold for other curriculum areas, but elementary reading may have some distinctive features in this regard. Our summary, however, will concentrate on curriculum and instruc­ tional features of the study. The report is especially revealing about the relationship between the content covered during instruction, the nature of that content, the manner in which instruction proceeds, and student performance on decoding and comprehension tests. Also noteworthy is the nature of these relations for children of differing entry levels in different classes. The data do not provide detail on teacher-student interactions, and so less can be said about the extent to which teachers converted allocated time into productive time. Differences in phonics coverage between classes was predict­ able from four factors - group phonics aptitude, difficulty of the phonics materials, time allocated to phonics instruction. and time spent in supervising learning. Teachers made signifi­ cant decisions- "We observed wide differences in new word and phonics coverage among groups sharing the same relative rank among classes" (Barr & Dreeben, 1983, p. 113)- and these decisions had consequences for student learning. The number of basal lessons covered by the teacher during the school year was a group-determined decision. Level and pace for each group of students was based primarily on the ease with which students could read aloud. Phonics coverage and pace were derivative on this decision, and were not directly related to group aptitude; "basal instruction is established, and the recommendations of the teachers' manual [and accompany­ ing worksheets] govern phonics coverage" (p. 124). Difficulty of the phonics materials was the strongest pre­ dictor of the amount covered, with group aptitude next in order; the two together account for almost 80% of the variabi­ lity in coverage. Designers of basal series treat the textual materials independently of phonics objectives, so that knowing a student is reading materials at a given grade level tells little about phonics tasks. If the phonics worksheets happen to be especially difficult, then teachers may adjust the pace. According to Barr and Dreeben (1983), time spent in supervi­ sion of phonics instruction by the teacher is a statistically significant predictor of coverage, and the descriptive statistics tell the tale quite dramatically: More phonics instruction [than basal instruction] occurs as seat­ work and supervised instruction is rare.... Teachers allocate more time to basal reading (43 minutes) and supervise a substantial portion of that time (33 minutes). By contrast, less time is spent on phonics each day (31 minutes) and considerably less of it is supervised (8 minutes). (p. 120)

Leinhardt, Zigmond and Cooley, 1981, in a detailed study of reading in special education classrooms, report similar findings: "The teachers in this study spent only 16 minutes per day in ... direct instructional contacts, and of that time, 14 minutes were used to give general reading instructions, 1 minute per day was

824

ROBERT CALFEE AND PRISCILLA DRUM

spent waiting while a student completed a reading task, and only one minute per day was used to explain or model correct elements in reading" - (p. 358. These are averages; the variabi­ lity was substantial.) Barr and Dreeben found that student performances reflected coverage: "Our evidence shows that pace is a major influence on learning; the more material children cover, the more they learn" (p. 130). Moreover, achievement was fairly specific to the substance covered. Thus, the number of sight words from the basal series pronounced by the student was strongly correlated with basal coverage (r = .93), and the number of phonics objectives mastered was correlated with phonics coverage (r = .62). The off-diagonal entries of this matrix were .50 or less. Standardized achievement tests were a closer match to basal (r = .75) than to phonics coverage (r = .57), unsurprising given the omnibus content of the former measures. Student aptitude on entry to school determined group as­ signment, which in turn determined coverage, which then determined standardized test performance, or so went the path analysis; "the effect of aptitude on general first grade achieve­ ment (beta = .15) is smaller than that attributable to both basal (beta = .45) and phonics (beta = .31) learning. . . . What the children have learned directly out of their basal reading materials has a greater impact on their general reading achieve­ ment than their aptitude, a finding that shows the preeminence of instructional events and their immediate outcomes over aptitude" (p. 143). Barr and Dreeben report that children assigned to different groups, and hence with differing entry levels, benefited differ­ ently from the instructional program. In their Table 6.2, Barr and Dreeben (1983) present a careful accounting of basal and phonics coverage for each class and group. The table requires detailed study, but even a cursory glance reveals enormous disparities between classes and groups, with more than an order of magnitude separating coverage by groups that were nomi­ nally the same (i.e., low, middle, or high). Nonetheless, and for whatever reasons, low-ability students were generally less able to transfer what they learned in one setting to other situations: In the low aptitude subsample [unlike the high aptitude group], phonics learning is not related to basal learning....Among these children, who are the least ready to read, the acquisition of phonics skills does not occur derivatively from basal reading or in conjunc­ tion with it, but more narrowly reflects the pace of phonics instruction. In short, the low aptitude children learn the phonics they are taught and do not pick it up as a by-product of more general reading. (p. 148)

This brief account barely scratches the surface of the Barr and Dreeben presentation. The study has limits, but nonethe­ less provides a model of what can be attained through a carefully planned research design within the environs of the regular classroom. A second example, the Reading Diary study, was undertaken in the mid-1970s to investigate the acquisition of decoding skills by first-graders. The first report appeared in Calfee and Piontkowski (1981; also see Piontkowski, 1981). Fifty students were selected from 10 first-grade classes in four schools, selected

to represent a variety of organizational contexts and student backgrounds. None of the students could read in September, but all were judged by the teacher as likely to read by midyear. Classrooms were observed, and students were measured on prereading skills, pronunciation ability, oral reading and com­ prehension, and standardized test performance. Emphasis was on the development of a multifaceted test of decoding ability, designed for comprehension and efficient assessment of growth during the school year. Calfee and Piontkowski report findings from the decoding test. The general pattern was one of rapidly expanding variabi­ lity between students. In October, most students could identify rhymes, match letters to initial sounds, and name the letters. Differences in performance on these tasks were negligible. By June, some children had not advanced much beyond the skills of October, while others could decode polysyllabic words with some fluency. Nor were the variations evenly distributed over schools and classes; student scores showed some classes far more effective in promoting student growth than others. Other analyses showed that level of decoding skill was at least partly independent of oral reading and comprehension (oral reading was assessed individually, not in the round-robin setting). Students who had mastered decoding were generally good at the reading and comprehension tasks, but other students could read a simple story even though their decoding skills were still relatively weak. Grouping practices play an important role in reading in­ struction, as noted earlier. Equally important is the teacher's response to the group. Allington (1980), demonstrating another approach to naturalistic investigation, made tape recordings of round-robin sessions in 20 classrooms, a high and low group for each teacher. The purpose was to determine teacher interrup­ tions during oral reading by high- and low-ability groups. The results showed that "teachers are more likely to inter­ rupt poor readers than good readers when an oral-reading error occurs, regardless of the semantic appropriateness of the error" (p. 375). Poor readers made more miscues than good readers (9.4 versus 5.8), and a larger proportion of these were semantically inappropriate (71 versus 53%). Teachers inter­ rupted readers in good groups on only 11% of the semantically appropriate errors; readers in low groups were interrupted 55% of the time. Interruptions on semantically inappropriate mis­ cues were 48 and 79% for good and poor readers, respectively. The overall result was that poor readers are interrupted fre­ quently. Allington also found that when poor readers were stopped, the teacher pronounced the word for the student or made a remark about the letter-sound cues; when good readers were stopped, corrective pronunciation was also likely, but the teacher tended to emphasize semantic cues. Direct teaching of decoding is relatively rare in basal instruc­ tion (some programs include a significant decoding component, but these are in the minority). Most children learn to "read" in the reading circle. Poor readers generally have trouble decod­ ing, and so the teacher may be right to guide the student by interruption and correction. On the other hand, it seems equally likely that the momentary exchange serves little pur­ pose other than to break the student's concentration on the meaning of the text, and perhaps to further convince the student that he or she lacks competence in dealing with text.

RESEARCH ON TEACHING READING

Decoding: In Summary Whether one views decoding as the first or most essential stage of becoming literate, it clearly is a critical aspect of reading acquisition. Research on this topic is vigorous, multifaceted, and evolving. The questions raised in the past decade represent new directions from the 10 years previous, and the next decade is likely to surprise us once more. We do find two major limitations in existing paradigms of decoding research. One is the reliance on correlational rather than experimental (i.e., interventional) approaches. Kintsch (1979) says: "If there is a moral [to the difficulty in drawing conclusions from existing research on reading], it is . . . that we continually get ourselves into trouble because [or when] we do not know how to interpret correlational data" (p. 328). We need to attempt more investigations that incorporate a signifi­ cant instructional component. The second limitation is the restricted view of the decoding curriculum, or to put it another way, the piecemeal selection of stimulus words and spelling patterns. The problem exemplifies Clark's (1973) "fixed-effect" fallacy-when the materials for a study have been selected to demonstrate a particular point, then what is the extent of generalization (Cronbach, Gieser, Nanda, & Rajaratnam, 1972)? But the question is even broader. The vision of English writing held by researchers and practitioners is remarkably unenlightened by either the history that generated the language or the linguistic principles that constrained this evolution. Future designs for selection of target words will undoubtedly include word frequency and regularity as factors. Our hope is that the accompanying prose might consider questions like "Why and how does frequency affect pronunciation skill?" and "What is meant by regularity?" (Venezky, 1976, pp. 22-23), with answers grounded in the underlying structure of the writing system. Finally, the emphasis on instruction and structure also has implications for assessment. In a sense, instruction uses assess­ ment as an anchor. One can teach, but in the absence of trustworthy evidence of what is being learned, the activity becomes pointless and undirected. Curtis and Glaser (1983), in their review of assessment practices in reading, note the impor­ tance of measuring fluency as well as accuracy. Implicit in this comment is the notion that actual pronunciation is probably the most trustworthy measure of decoding skills. Paper-and­ pencil tasks are less direct indices. We would also emphasize structure and understanding. The spelling patterns chosen for assessment should reflect the structural characteristics of the writing system, and should give information about the parts of the structure that have been mastered by the student. In addition, we would propose that understanding of the system should be a goal of instruction. Metacognitive research has provided insights into comprehen­ sion processes; there has been no parallel development in the area of decoding. By the end of elementary school, the student should be able to articulate a strategy for attacking novel words, a strategy that is sensitive to the morphophonemic character of English spelling, and that is grounded in the historical features of the language. Toward this end, it might be useful to determine teachers' knowledge of this domain. If

825

researchers, curriculum designers, and teachers cannot articu­ late the structural features of English spelling, then we should not be surprised if students fail to acquire this competence.

Vocabulary This section of the chapter focuses on word meaning in reading instruction. The discussion begins with an overview of the vocabulary curriculum and an account of vocabulary training as it typically occurs in the classroom. Next are segments on two topics that have been investigated rather extensively in recent years : readability, and the relation between vocabulary and comprehension. The section ends with a consideration of topics that warrant investigation in the future.

What Is the Vocabulary Curriculum ? While most reading teachers agree on the importance of vocabulary instruction, the goals are less clear. First let us attempt to simplify the question by eliminating one source of confusion - teaching vocabulary is teaching the meaning of a word, not how to pronounce it. The latter task falls under the rubric of decoding. What does it mean to have a word in one's vocabulary, to "know" a word? This question has baffled philosophers throughout recorded history; Green (1984) has argued that "for most words, the notion 'the meaning of the word x' simply does not make sense" (p. 1). Teachers cannot afford the luxury to wait for the outcome of philosphical debate - competence in reading requires that students have an extensive vocabulary and "know" a lot of words. The teacher's job is to help students achieve that goal. But again, what does it mean to know a word ? Answers to the question generally include the notion of labeling a more or less well-defined concept - we use words to point to things in the world (Wittgenstein, 1958). Function words, the syntactic "mortar" that binds together ideas, serve a different role, one better discussed under the rubric of sentence comprehension. Words also serve to label abstract concepts that may transcend any effort at illustration. Words are not neutral in this enterprise. Anyone who has played a word association game is aware that words come in structured sets. As Bransford and Nitsch (1978) point out: words are related to one another in addition to being associated with practical knowledge that gives them meaning. These relations are due in large measure, but not entirely, to the fact that practical knowledge is itself highly structured" (p. 281). Research in the last decade has converged on the nature of associative structure, and on the roles of experience and school­ ing on these structures (Clark & Clark, 1977; Anglin, 1977; Miller & Johnson-Laird, 1976). What it means to know a word is determined in part by how it is related to experiences and to other words. But a word may be known in varying degrees. "Knowing a word well involves knowledge of a whole series of networks in which the word functions" (Graves, 1984). It involves depth of meaning; precise usage; facile access (think of scrabble and crossword puzzle experts); the ability to articulate one's understanding; flexibility

826

ROBERT CALFEE AND PRISCILLA DRUM

in the application of the knowledge of a word; the appreciation of metaphor, analogy, word play; the ability to recognize a synonym, to define, to use a word expressively. Graves (1984) has organized school learning of vocabulary under five rubrics: (a) new labels for known concepts (the injunction "Withdraw your digit from your nasal orifice!" uses novel words for a command familiar to most children); (b) new labels for new concepts, a task in the upper elementary grades and upward in virtually all subject matter areas; (c) new meanings for known labels that may or may not entail new concepts - "force" is a known label, but the meaning in physics is new to most children; (d) enriching and expanding the meanings of words already known; and (e) moving words from receptive to expressive vocabularies (we all recognize far more words than we can use expressively). The full shape of the vocabulary curriculum remains to be determined. As the preceding comments show, _one goal of vocabulary instruction may be to increase the size of the student's vocabulary, but there is also the need to ingrain a respect for words and to instill a self-awareness of the adequacy of one's capability to determine a word's meaning in a particu­ lar context. WHAT DO PRESCHOOLERS KNOW ABOUT WORDS?

Children bring a lot of words to school with them. Estimates of "vocabulary size" vary by orders of magnitude (Anderson & Freebody, 1981; Nagy & Anderson, 1984; and references therein), with guesses ranging between 1,000 and 10,000 words, depending on the criterion for "knowing a word." Some students know more words than others, and some know words more deeply than others. Some students know more "school" words than others (see Bruce, Rubin, Starr, & Liebling, 1983). Finally, some students are better than others at explaining what a word means. Indicators of vocabulary knowledge are highly correlated with the construct of intelligence, which in turn is highly correlated with school success and with performance on read­ ing tests. What do these relationships mean? The possibilities are several: (a) Some students are inherently smarter than others; (b) some students have had a richer experience than others; (c) some students have been better trained than others to handle a particular assessment; (d) some combination of the three elements listed above is operating. Anderson and Free­ body (1981) point out that the first explanation, innate apti­ tude, implies an immutability of sorts; instruction may improve vocabulary for all students, but individual differences will remain. The other two hypotheses about individual differences are within the power of the school to change, in principle at least. "Access" or fluency of knowledge may be important (Mezynski, 1983), as may knowledge of words specific to particular text (the "instrumentalist" hypothesis). Whatever the explanation for individual differences, prior to school entry most youngsters have acquired language naturally, through trial and error, through repeated experience, without a great deal of reflection and examination about the nature of what has been learned. Metalinguistic and metacognitive awareness is limited (Tunmer, et al., 1984, and references therein). The preschooler's vocabulary reflects the particulars of

his or her environment and the accidents of experience. It is functional and concrete - " a hole is to dig," in Ruth Krauss 's words. Associations are direct and primitive. They may be based on rhyme (pig-dig), common features (cat-black) or category membership (cat-dog). Seldom are the structures highly developed, nor do systematic relations (super-, sub-, or co-ordination) appear to be part of preschool thinking. When such relations are present (some children have played "animal, vegetable, mineral"), they tend to be implicit. Many kinder­ gartners have begun to play with words, usually in the form of outrageous puns, but they are not inclined to reflect on their knowledge of"word meaning" - indeed, the concept of "know­ ing a word" is poorly developed (Al Issa, 1969; Anglin, 1977; Litowitz, 1977; Nelson, 1978). Building on What Already Exists. Preschoolers at all levels of entering ability possess language and demonstrated success at acquiring vocabulary. School is a place where students are exposed to words - to words familiar, to words unknown, in contexts that guide understanding, and in strange isolation. There has been little systematic study of "classroom" vocabul­ ary, but our impression from observations and examination of teachers' manuals is that teacher talk tends to be "familiar," that low-frequency words are rare in both student and teacher talk. Since preschoolers already have experience in acquiring vocabulary from context, there would seem to be value in promoting the intentionally rich use of language in the class­ room. The student is continually encountering new experiences in the classroom-new concepts, if you will. The teacher, by consciously "talking about" everything that happens, can pro­ vide opportunities to learn new concepts and the labels that identify them. It is equally important that the teacher encourage "student talk"; modeling is important, but the eventual aim is for the student to gain expressive control over vocabulary. Finally, students naturally rely on context as a substitute for precise word usage. They may have some idea of the most appropriate word to suit a given situation, but the "natural" thing when you cannot quickly find the right word is to say "You know what I mean," or "It's kinda like that," or in some other way to handle the situation by nonlinguistic means. The teacher, by example and by questioning, leads the student to more effective use of vocabulary, all of this in the course of other ongoing events and without "planning a lesson." Acquiring New Knowledge. Systematic vocabulary instruction

aims toward two goals. First is the acquisition of new concepts. As the student encounters the various disciplines and profes­ sions, new concepts abound and vocabulary becomes a critical concern. From journalism : "A story written in the inverted pyramid style will start with a hook in the lead, followed by details of diminishing importance so that the story can be cut from the tail. " From biology: "The chemical process called photosynthesis involves the reaction of a plant's chlorophyll with carbon dioxide from the air and sunlight to produce oxygen" (examples from Konopak, 1984). The student's task in these two passages may be generally construed as "learning vocabulary," but the demands are different in the two examples. In the journalism example, the

827

RESEARCH ON TEACHING READING

critical words are already known to most students, but under different meanings; in the biology example, both the critical words and the concepts are likely to be unknown. In both these examples, the aim is to discern relations among ideas and activities, and within that framework to acquire the proper labels. The student who learns by rote the definitions of the key words without understanding the underlying relations may pass a vocabulary test, but to little lasting advantage. More­ over, learning in such instances is unlikely to occur through trial and error. " Discovering" biological relations is improb­ able for most students; for journalism, the editor may decide to wait for students to learn from experience, but the quality of the school paper and the value of the experience for the students may both suffer. A second domain of new knowledge is the relatively small set of paradigmatic structures that account for the majority of semantic relations. The most primitive structure is the topic­ centered arrangement, often referred to as a "web" by teachers (Figure 28.3). Asked for associations to a known word, students can readily respond, starting with an immediate link, pursuing a thread of thought, a break while a new connection is sought, and thence onward. The organizing principle is a central hub with radiating spokes. A more sophisticated version clusters related associations, and may even name each set, the initial steps in imposing coherence on a domain. Matrices and hierarchies (also shown in Figure 28.3) are two other basic structures that alone or in combination undergird knowledge in the subject matters. These structures are frequent in content-area texts, but they also provide frameworks for organizing words, and familiarity with them should be an important goal of vocabulary instruction in the reading curri­ culum. Acquiring New Strategies for Vocabulary. Learning from experi­ ence and analyzing semantic structures are both important goals for vocabulary instruction, but schooling can also help the student sense when to stop and examine a word, and can provide good techniques for breaking a complex word into component parts, explicit tactics for analyzing contextual clues, and knowledge of when and how to use dictionaries and other references. These strategies represent a qualitatively different approach to the acquisition of vocabulary. They are marked by a reflectivity, by explicit analysis, and by thoughtful decisions about the techniques and resources most appropriately applied when the student is uncertain about how to interpret a word. The first step in this process is the development of a sensiti­ vity to such situations. Early in langua,ge acquisition parents often take time to label objects and events in their child's world. After entry to school, the youngster has fewer opportunities to pause during a discussion and ask "What does X mean?" The assumption seems to be that meaning will be made clear if necessary, and that understanding can otherwise be taken for granted. A goal of vocabulary training is to reinstitute an inquiring and critical attitude, and a sense of how to judge whether a word needs to be examined more carefully. Studies of word deletion using the doze procedure demonstrate that a third of the words may be deleted from some texts without significantly

Webbing

water animals whale } dolphin that are not fish

goldfish salmon } examples shark

aquarium where bowl l th0Y tank live water

FISH

rod & reel how to } net catch . fishing pole

gills scales } description fins

Weaving

TYPES OF DIAGRAMS FOR ORGANIZING INFOR MATION

Two-Dimensional Classification

Format

Example Classification No. of Days Holidays Season Weather Special Days

February 28 (29) Wash ington's Birthday Winter Cold, Rainy St Valentine's

May 31 Memorial Day Mother's Day Spring Sunny, Warm Dad's Birthday

September 30 Labor Day

Summer/Autumn Sunny, Warm Mom's Birthday

Hierarchical Classification Format

Example

Minerals

� Stones

Metals Platinum Silver Gold

Rare

Aluminum Copper Lead Iron

Common

Alloys

Bronze Steel Brass

Sapphire Emerald Diamond Ruby

Precious

Masonry

Limestone Granite Marble Slate

Fig. 28.3 Webbing, matrix, and hierarchical classification.

reducing understanding (Stratton & Nacke, 1974). A critical factor is the information value of a word context - under what conditions is it essential to know what a word means? This question has received relatively little attention by researchers. Even less attention has been given to students' awareness of information value, or to instruction in identifying the informa­ tiveness of a word in context. If a student has decided that a particular word contains critical information, what then? There would seem to be obvious benefit to techniques that can be applied without " leaving the text" - using context, or examin­ ing the morphological characteristics of the word. Decisions about the merits of these tactics (or perhaps more drastic steps, like consulting a dictionary) are essential elements in a strategic approach to vocabulary.

828

ROBERT CALFEE AND PRISCILLA DRUM

Beyond identification of word meaning sufficient for the comprehension of a passage, there is also the matter of expres­ sive control in speaking and writing. The aim is not simply increased knowledge of "big" or unfamiliar words, but im­ provements in precision and richness. Slater and Graves (1984) asked Martin Mann and Teresa Redd of Time magazine how they would make school texts more interesting and readable. The answer was to use "action verbs, contrasts, metaphors, colloquial expressions, familiar tone, word play, alliteration, vivid adjectives, participial phrases, appositives, and a few big words." Each of these items features vocabulary as a critical element.

often inadequate to clarify meaning. Other activities encoun­ tered from time to time, mainly through worksheets - diction­ ary practice, compound words, roots and affixes, occasionally a weblike association task - were sporadic and disconnected from the main thrust of the lesson. An exception was the program on morphographics or "word parts" in Distar (Engelmann, Haddox, Hanner, & Osborn, 1978); when these developers decide to incorporate an objective in the curriculum, training is implemented in a sustained and consistent fashion. Teachers use the basal series, and the patterns described above are readily observed in the classroom. As Schramm (1955) noted some time ago:

A REPRISE

The most influential group of men in this country in the making of school texts [may be] the educational editors of our large publish­ ing houses. Their only real competitors are the ideological leaders like Dewey and Thorndike whose concepts of what a textbook should be and how it should be made have been reflected in the efforts of hundreds of editors and authors. (p. 1 58)

The list of curriculum topics presented above is rather exten­ sive, but seems reasonable given the importance of vocabulary to success in other areas of schooling. Facility in handling words is likely to become a more valuable talent in the future (Sola Pool, 1983). Molloy's (1982) advice is to learn a new word every day, and you will be perceived as better educated, more important, more powerful, taller, and morally superior (not better dressed !). Be this as it may, certainly the school should give students a good start toward an improved vocabulary, and the skills needed to sustain growth on their own. The curricu­ lum now in place is rather more Il).odest in its aims.

Dewey and Thorndike have not been with us for a while, and only a few scholars have examined reading series in the past quarter-century. Some critiques have appeared recently (see Anderson, Osborn & Tierney, 1984, and references therein for work by the Center for the Study of Reading; Beck et al, 1980, for work at the Learning Research and Development Center). ADVICE IN TEXTBOOKS

What Happens in Typical Instruction ? In this section we discuss several features in present-day in­ structional practice. These issues form the backdrop for re­ search questions that have been pursued during the past decade. BASAL INSTRUCTION

The first point about current practice in vocabulary instruction is that there is not much of it. Beck, McKeown, Mccaslin, and Burke (1979) and Jenkins and Dixon (1983), after examining several basal reading series, found that direct instruction in word meaning varied from none at all to 200-500 words per year. These are small numbers to begin with, and many of the words selected for teaching were already known to most youngsters. Vocabulary is introduced in two ways by most existing basal series. In the first, a list of words is presented for instruction at the beginning of the lesson. These may be words judged as difficult to pronounce or occurring infrequently in print at that grade level. The words are generally not selected on the basis of mutual semantic relations or textual importance. Instruction typically proceeds by asking students to give a meaning or make up a sentence with the word. The glossary is consulted on occasion. Most series then follow the first pass by worksheet activities - the words are matched to synonyms, placed in a sentence frame, or the like. Beck et al. and Jenkins and Dixon also concluded that vocabulary instruction in the elementary grades lacked both "intensity and scope." The materials relied chiefly on rote practice and context as the basis for meaning. There were too few repetitions to ensure rote learning, and the context was

What advice is given to teachers about vocabulary instruction? One of the most popular texts is Johnson and Pearson (1984). One finds in it a large number of topics, a collage of meaning and decoding, student activities in great variety, all in all a great deal of material, a collection distinguished more by diversity than coherence. Johnson and Pearson do attempt some organi­ zation of the domain: Multimeaning Words

Classification Synonyms Antonyms Denotation/ Con notation Semantic maps Feature analysis Analogies Homophones

Polysemous words Homographs

Resources Dictionary Thesaurus Etymology

Some structure is better than none, but these lists are not altogether satisfying as a way to "carve the turkey." This text is typical of what is found in most textbooks on reading instruc­ tion, which in turn mirror what is found in teachers' manuals. VOCABULARY CONTROL

Students should not be expected to handle words with which they are unfamiliar; this is the principle underlying vocabulary control. Familiarity is measured by counting how often a word occurs in print. Control is imposed by limiting the words at each grade level (especially in the primary grades) to high­ frequency words. At the extreme, this reasoning leads to the advice that only a

829

RESEARCH ON TEACHING READING

handful of commonplace words need to be taught: "Since the 300 most common words constitute 75% and the 1,000 most common constitute 90% of the words that we write, it quickly becomes clear that the population of words whose mastery teachers should seek is much smaller than formerly imagined" (Clifford, 1978, p. 140). This suggestion ignores the information value of words; it is precisely the rare words that comprise the critical core differentiating one passage from another. The topic of readability will be examined in greater detail below. For the moment, we will simply observe that the basal materials available for reading instruction are typically con­ strained in vocabulary usage, especially in the primary grades. Moreover, teachers are commonly advised to determine the student's reading level (based on frequency counts), and then select textual materials matched to this level. The result is that beginning readers or those experiencing difficulty with phonics are exposed to a rather barren sample of words. PRACTICE IN CONTEXT

The most frequent advice from researchers in recent years emphasizes that meaning is largely gained from context. School children double their vocabulary between third and seventh grade (Jenkins & Dixon, 1983), and it is assumed that this gain occurs through multiple exposures in context. Since this ap­ proach seems to work, why not simply improve on it? "Repeti­ tions, opportunities to practice using the word, and review characterize formal vocabulary teaching" (Jenkins & Dixon, 1983, p. 5). "In general, there appear to be three important variables: (1) amount of practice given to the words, (2) breadth of training in the use of words, and (3) the degree to which active processing is encouraged" (Mezynski, 1983, p. 273). Most accounts of "learning a word's meaning from context" have lacked exactitude. Sternberg and Powell (1983; also Sternberg, Powell, & Kay, 1983; Jenkins & Dixon, 1983) have presented the most comprehensive conception to date of the role of context in word meaning. They first distinguish context clues from the conditions that mediate their use. Consider the sentence "At dawn, sol arose on the horizon and shone bright­ ly." Assume that the reader knows every word except sol. Clues in the surrounding text that shape the features of the unknown word include, according to Sternberg and Powell, dimensions of time, space, function description, value, cause and effect, class membership, and equivalence. These sources combine to paint a picture of the meaning of the unknown term. Mediating conditions are dimensions such as the number of occurrences of the unknown word, the variability of the con­ texts in which the word is embedded, the relevance and proximity of clues, the overall difficulty of the passage, the familiarity of the topic, the number of other unknown words, and so on. Jenkins and Dixon further classify "unknownness" in the form of a matrix: SIMPLE SYNONYM CONCEPT KNOWN

altercation

CONCEPT NOT KNOWN

arcane

(fight)

(obscu re)

NO SYNONYM odometer (the gadget that tells no. of m iles) legislature

These thoughts require refinement, but they appear to have value for analyzing how contexts support the growth of word meaning. Jenkins and Dixon note that research on context and word meaning have focused on the relative importance of various types of clues on the student's success in inferring word meanings. Equally important is how these factors influence the student's use of text to infer word meaning when the primary task is to comprehend the passage: "Readers can often get by without figuring out a word's meaning, and thus may not bother to try" (Jenkins & Dixon, 1983, p. 254). Konopak (1984) notes that textual context varies greatly from the primary through the secondary grades : in the primary grades, "vocabu­ lary development is mostly learner-based, relying on direct and indirect contextual settings to provide meaning. .. . the focus [then] changes from general vocabulary to content-specific words in the higher grades" (pp. 8, 9).

WORDS

Advice in a different style comes from a small book by Lodwig and Barrett (1981) e·ntitled Words, Words, Words. Designed for high schoolers and college students, the book begins, "This is a book about words and their meanings." The chapter titles differ from what one encounters in textbooks for teachers - Words;

Words and History; Word Creation ; Words and Their Mean­ ings; Changes in Word Meanings; The Development of the Dictionary. One gets a sense of the evolution of a communica­

tions technology, of the role of words as a tool of an increas­ ingly articulate human race. The approach to context learning differs considerably from the previous discussion : "A typical word does not have just a meaning; it has a cluster of meanings. . . . Each of these senses of the word is appropriately used in certain contexts" (p. 76). Or as Whorf (1956) put it, "The meaning of a word is less like a dollar bill with a fixed amount than like a blank check to be filled in as required" (p. 234). Lodwig and Barrett's advice is to look for restatements of the unknown word, for contrasts, for key words, or to attempt morphological analysis - they offer six pages of specific sugges­ tions in plain language with accompanying exercises. We mention this source because it represents an interesting contrast with the prevailing conceptions of vocabulary instruction, a contrast with implications for both research and practice.

Research Topics in Vocabulary Instruction Not too long ago it would have been relatively difficult to identify a significant body of research on vocabulary. The past decade, however, has seen something of a renaissance in this area. We have selected two broad categories for review : read­ ability and context effects, that is, the relation between vocabu­ lary and comprehension. Both focus on the content of instruction more than on teaching activities, but many of the findings appear to have instructional implications. A great deal of territory in this vital domain remains unexplored.

830

ROBERT CALFEE AND PRISCILLA DRUM

READABILITY

The foundations of readability research were described earlier under the rubric of vocabulary control. In the first years of school the child faces many challenges. Why not reduce one of the hurdles by selecting for initial instruction only already familiar words? As skill develops, less familiar words can be gradually introduced. How to measure word familiarity? Thorndike (1921a), in creating The Teacher's Word Book, relied on counts of printed words in school texts. Basal readers, when measured against these word counts, were found wanting - primers contained many words not in the 10,000 most frequent words (e.g., baggy, Bert, bluebird, and crawly, but also baff, bannock, bight, and Canute; Clifford, 1978, p. 143, after Thorndike, 1921b). Pub­ lishers began to respond to these findings, and today word counts dominate the design of basal reading systems. Readability research (Clifford, 1978; Klare, 1963, 1974-1975) grew up around the word count paradigm. Studies showed that texts that were easier to read were more quickly read and were better comprehended if they contained a preponderance of high-frequency words. Sentence length played a secondary role in the readability equation. Readability research is discussed here because vocabulary is the critical element in the equation. It is included in the chapter because of the importance of readability in determining the texts available for instruction, and in guiding teacher choices about the passages that are suitable for students of a given ability level. As the following selections reveal, the area remains active half a century after the appearance of the Word Book. Textbook publishers frequently have to revise original pas­ sages to conform to readability requirements - how do they manage this task? Davison and Kantor (1982) examined these rewriting tactics in four texts that were lowered to fit junior high students reading 2 or 3 years below grade level. Passage length was greatly reduced (two-thirds to one-quarter of the original), sentence length dropped from 20 + to 13 words per sentence, and clauses fell from three to fewer than two per sentence. Changes in lexical usage, according to Davison and Kantor, were equally dramatic: In general, a lexical item may be a candidate for change under two major circumstances: ifit is a proper name, technical, or specialized term with no obvious import to young readers; or if the item is vague or has fallen from current usage. [Thus] Hippocrates recom­ mended milk to his patients as a curative beverage [becomes] One of the most famous Greek doctors told his patients to drink milk to cure illness. (p.204)

This example and the principles espoused are not necessarily unreasonable, but the results can be unfortunate. Davison and Kantor point to a serious flaw: If deletion of topic sentences and splitting of longer complex sentences into shorter component sentences is done without a view of the entire text as connected discourse, then the adapted version may have fewer cues than the original for relating sentences and clauses. It might actually be harder to understand than the original. ... Readability formulas, in our opinion, fail to give any adequate characterization of readability, except in a purely statistical sense from which no particular conclusions can be drawn for creating readable texts. (p. 207)

The risks from formulaic application of word counts in the selection and design of reading passages were pointed out soon after the publication of the Word Book. Words at a common index of difficulty often present quite different conceptual demands; for instance, these pairs are at the same frequency in the original list: inside/err, ate/elate, dirty/deem, hers/withal (Clifford, 1978, p. 127, also reference therein). Davis (1944) found negligible correlations between word frequency and test performance, and concluded that teachers and textbook au­ thors who employed word counts to restrict vocabulary usage "should be aware that by so doing they are probably accom­ plishing very little toward controlling the actual 'difficulty' of the materials" (p. 174). Good writing always takes word choice into account, but only as one of several factors. To judge the range of outcomes possible within the constraints of the read­ ability formula and the creativity of writers, you might scan one or more basal texts for examples of Aesop's fables (they are reasonably popular), and compare basal versions with the originals. In another example of recent developments in this area, Goodman and Bird (1984) investigated intratext word counts. Readability research has generally employed aggregate word counts - measures averaged over hundreds of texts. Goodman and Bird were interested in the patterns of word frequencies within a text. Their empirical findings are probably less impor­ tant than the theoretical analyses. The major point is that, even though a word is quite rare in general, it may be relatively common in a particular text. The writer does not include a word in a text because it is frequent, but because the topic calls for the word. A rare word (along with its referents) may occur several times in a passage, in which case it is not rare in the context of the passage. A word occurring only once in a text is unlikely to be significant, whether it is common or rare in the aggregate.A common word may occur frequently in a text, but still pose a challenge to understanding because it takes on a different meaning (com­ mon words tend to have more meanings than infrequent words; compare draw with neurosurgeon). " Teachers concerned about vocabulary development would do better to focus on functional use of words and terms in the context of real passages than to resort to decontextualized lists. .. . A text, after all, is considera­ bly more than the sum of its words" (Goodman & Bird, 1984, p. 144). So what should be concluded from research on readability? On the one hand, it is probably easier for students to make their way through a text that has commonplace words and short sentences. They may not learn much from the experience, to be sure. On the other hand, a text filled with critical and unknown vocabulary can easily overwhelm the student (and even the teacher), as Thorndike (1934) points out: The worst, which often occurs in basic readers and supplementary reading, is to require children to read material which they cannot understand; or can understand, but do not enjoy; or cannot understand, and would not enjoy if they could understand it. . . . Books do not have to be bad literature; but the vocabulary and sentence structure must not thwart comprehension of what the book tells, and that must be something that the pupil cares to be told. (p. 19) The issues appear not to have changed greatly in 50 years.

RESEARCH ON TEACHING READING

VOCABULARY AND COMPREHENSION

Can a student understand a passage when some or many of the words are unknown? How can the student use context to gauge the meaning of unknown words? These questions have a long history. Hilliard ( 1926) exhorted that "training in vocabulary, with proper motivation, is a type of work well within the grasp of every teacher, and from this evidence seems to be one of the strategic points in building up comprehension" (p. 40; quoted in Clifford, 1978). Everyday experience suggests that we fre­ quently acquire word meanings from context, not through formal instruction, but as it were "in passing." Why discuss these topics under the rubric of vocabulary, especially given the emphasis on separability of the components of reading? In fact, most studies on this topic focus on the words, with less attention given to the specifics of the text, or to the relation of words to the text. Perhaps the next Handbook of Research on Teaching will be able to view these issues from the perspective of both word and text.

Does Studying Vocabulary Aid Comprehension? Mezynski ( 1 983) reviews eight studies of this question. All the investiga­ tions were "instructional" - students were specially trained on words thought to be problematic, and then were tested on comprehension of a test passage. The research, spanning 2 decades, varied widely in the nature and number of the stu­ dents, the amount of instruction, and the adequacy of the research design. Most students were in the upper elementary grades, however, and most of the programs lasted for several weeks; new words were introduced at the rate of about 10 per week. In "direct training" studies, the target words were from the passage, while in the "general transfer" studies training was on a large set of words, with the expectation of an overall advantage. Word knowledge and passage comprehension were measured by various combinations of recall and recognition tests. The typical findings were that vocabulary training improved performance on vocabulary tests, but had little effect on reading comprehension. As Mezynski (1983) notes, " It is difficult to imagine how a reader could comprehend text in which most of the words are unfamiliar" (p. 253). But that is seldom the situation. Rather, readers (and listeners) may occasionally encounter words that are unknown, but only some of these are critical to understanding the text. It also matters how compre­ hension is measured - multiple choice tests are less sensitive indicators of instructional interventions than are free-recall measures. The literature typifies the empirical tradition in educational research (Clifford, 1978). Lacking an overall framework, re­ searchers work on bits and pieces of a problem. Mezynski (1983, p. 258) points out a number of serious confoundings, and notes that, given the large number of uncontrolled procedural variations, the studies are difficult to compare. She also homes in on two critical issues that may explain why effects on comprehension were negligible: (a) vocabulary knowledge may be unnecessary for comprehension of some texts (e.g., narra­ tives), and (b) the instruction may have been inadequate to support comprehension (e.g., not enough training for automatic access, or rote training insufficient to transfer to the compre­ hension task).

831

Another perspective is gained by looking at the studies where there was an effect on comprehension. We will examine two of these, one employing "direct" training and the other " general" training. The " direct" study is reported by Kameenui, Carnine, and Freschi (1982) as a series of two experiments asking: (a) Do difficult words make text comprehensible? (also see Marks, Doctorow, & Wittrock, 1974); (b) Does adding redundant information improve comprehension? (c) Does learning the difficult words make the text more comprehensible? and (d) Does incorporating vocabulary training in passage review lead to improved performance? These questions certainly merit attention, although the way that they are operationalized in these experiments was not entirely satisfactory. The two experiments, both performed with upper elementary students, were virtually identical except for the passages. The texts were single paragraphs of less then 100 words. Six words were selected as targets for each passage. Training was short­ term and individualized. After instruction, students were asked multiple choice questions and given a free-recall test. In one condition, familiar synonyms were substituted for the target words. In other conditions, the "hard" words were in the text and students received (a) no training, (b) no training but redundant cues in the passage, (c) rote association training, or (d) " passage-integrated" training (the student was stopped during the reading and interrogated/trained on the difficult words). The experiment was carefully designed to yield the desired results - Kameenui et al. (1982) note that the materials were "contrived," and so they appear. Here is the text with the target words marked as Xs (unknown word meaning): Joe and Ann went to school in Portland. They were XXXXXXXXXXs. They saw each other often. They had lots of XXXXXXXXXXs. At the end of high school, Ann XXXXXXed Joe. Then Ann moved away. Joe stayed in Portland. He got a job as a XXXXXXX. One day Joe was working, and he saw Ann. Ann did not see Joe. Ann looked XXXXXXXXXXXX. She was being XXXXXXXXXXed.

The unknown words were quite dense (10% of the total passage, and perhaps half of the substantive words). You should decide for yourself whether you " understand" the passage. Below is the version with the infrequent words in place (as in the original), and with redundant information in parenthese�: Joe and Ann went to school in Portland. They were antagonists. They saw each other often. They had lots of altercations. (They just didn't get along very well.) At the end of high school, Ann maligned Joe. Then Ann moved away. Joe stayed in Portland. He got a job as a bailiff. One day Joe was working, and he saw Ann (talking to a policeman and answering questions). Ann did not see Joe. Ann looked apprehensive. She was being incarcerated. (p. 373-374)

The second passage was similarly designed, perhaps a bit more contrived. Some questions demanded little more than syntactic knowledge; for example, in the second text, the statement "Harry said Ace was pusillanimous" was followed by the question "Who did Harry think was pusillanimous?" The findings were that redundancy and vocabulary training did lead to better performance on questions where knowledge

832

ROBERT CALFEE AND PRISCILLA DRUM

of the difficult vocabulary was critical. The effects were substan­ tial and replicable. Viewed as a demonstration, the findings are provocative. They may be less a model for classroom practice, however, then an illustration of the persistence and ingenuity of educational researchers. A different sort of persistence appears in a study by McKeown, Beck, Omanson, & Perfetti (1983). In a previous study (Beck, Perfetti, & McKeown, 1982), children given inten­ sive and varied training on a list of 104 words over 5 months of schooling demonstrated their command of the words - they could define them, and were more adept in timed tasks. Performance on a story comprehension task did not distin­ guish trained from untrained students, however, except in one condition where students received substantial additional practice. The replication by McKeown et al. (1983) incorporated several methodological enhancements over the original design, primarily in the methods for assessing the effects on story comprehension. Again, training focused on a target set of 104 words divided into several structured subsets (e.g., rival, hermit, novice, and virtuoso are from the "people" cluster).· Training consisted of 75 half-hour lessons over 5 months. There were multiple opportunities for practice, interword relations were stressed, and students had to justify their answers about word meaning. The training clearly affected performance on the vocabulary tests. As exposures were increased from none to some (10-18 occurrences) and then to many (26-40 occurrences) the percent­ age of correct answers grew from 34 to 71 to 80% - still not perfect. The effects on story comprehension were statistically significant - recall of central ideas increased from 16 to 33 % , and multiple choice performance went from 44 to 66%. The density of target words in the test passages was quite high (about 30 per 300 running words). The replication by McKeown et al. (1983) is good news and bad news for advocates of the "vocabulary helps comprehen­ sion" position. On the positive side, it is encouraging to see evidence of effective instruction, and transfer to a new situation. On the negative side, the cost was substantial, and the magni­ tude of the effect relatively slight (two-thirds of the central ideas were not recalled by the students). Back to the positive side, prototypical recalls (p. 14) were convincing that the trained students had undergone a significant (and not merely statisti­ cal) change in their capability to explain what they had read in a clear and articulate fashion - target words were used in recall to good effect. It is possible, then, to find conditions in which knowledge of critical vocabulary is essential to comprehension of a passage. The training may influence basic understanding, but the stu­ dent's ability to express this understanding is also important (Duin & Graves, 1984). The research thus far has been demon­ strational, and lacks comprehensiveness. The character of the texts is either unknown or highly suspect. The relation of the critical vocabulary to the text is generally unspecified. While training has been quite extensive in some studies, instruction generally fell outside the regular reading activities, and so the practical applicability of the instructional techniques remains uncertain. With the exception of the Pittsburgh studies, there is little evidence of transfer. Students learn the target words, but

do not learn to learn words. All in all, this domain is best described as "promising," with much work remaining ahead. Does Studying a Passage Improve Vocabulary? It might seem self-evident that the person who reads a great deal will thereby increase his or her vocabulary. Given the relative impoverish­ ment of vocabulary instruction in present-day reading pro­ grams, several researchers have concluded that students' chief method for acquiring vocabulary in the elementary grades is by "default" through context. What are the mechanisms that drive contextual learning of word meaning? What features of a text promote the acquisition of unfamiliar words? How can students be assisted in determin­ ing that they need to reflect on a particular word in a passage? What strategies are most useful in taking advantage of context? These questions have been investigated in one form or another by several researchers during the past decade. The studies have generally relied on practice rather than instruction, and the emphasis has been on variation in worksheet content and layout more than strategic training. In one of the most frequently cited studies, Gipe (1979) varied treatment, grade, and student ability. The youngsters, third­ and fifth-graders, were trained on 12 words each week for 8 weeks. Students were pretested on a list of 150 words, and the 48 least well known were selected for training. During a given week, one of four treatments was in effect: Association (rote learning - "A barbarian is a cruel mean person"); Category (semantic association - "Add some words to the list of bad people, starting with mean, cruel, barbarian, robber"); Context (target words embedded in texts, with probe questions "Write what a barbarian might do at the dinner table"); and Dictionary ("Write the definition and use the word in a sentence"). Instruction consisted of worksheet exercises that lasted for 15 minutes a day; students worked on their own, with minimal teacher involvement. Assessment comprised two tests, one at the end of each week of training and a second test 1 week after the entire session. The results supported context training as the most effective of the four treatments, with the category scheme and dictionary exercises as least effective. The differences are not great - 48% correct for Context, compared with 34% for Dictionary. More notably, none of the treatments led to anything approaching mastery. At best, high-ability fifth-graders were correct on 9 out of 12 words from a list like barbarian, colossal, graphite, and wretched (the words are "eighth grade level" and hence not extraordinarily rare). The results may speak most directly to the ineffectiveness of worksheet-based methods of teaching voc,abulary. The differences between the treatments are con­ founded in several ways, and appear differentially appropriate to the criterion tasks, both of which required analysis of meaning in context. Jenkins, Stein, and Wysocki (1984) investigated several fac­ tors that "mediated" context effects - preexposure to a syn­ onym prior to reading the passage, number of paragraph exposures (0 to 10), and the word sets (three different versions). The measures included "Supply Definition" (write the meaning of an underlined word in a sentence), "Select Definition" (a multiple choice test), "Sentence Completion" (multiple choice), and a comprehension test using a new passage containing six

RESEARCH ON TEACHING READING

target words. The students, relatively high-ability fifth-graders, first reviewed pronunciation of the words, and then went through one of the treatment combinations using worksheet methods. The target words were rare for fifth-graders (e.g., acquiesce, loquacious, potent, reform(?), and virtuoso). Training continued for 10 days, on 0, 2, 6, or 10 of which students were asked to read paragraphs containing the target words. The factors all had substantial main effects, ability greatest, followed by preexposure, word set (a random factor), and then the number of exposures (or paragraphs). There was a substan­ tial jump from "0" to " 2 exposures," especiaily for the higher­ ability students, with further exposures leading to a linear increase in performance. interactions were generally negligible. Vocabulary learning occurred in the absence of systematic instruction, especially if preceded by a short " ciefinition" (a synonym), and more so for higher-abiiity students in an above­ average group. Ability level may be an important consideration in this study. These were able students, probably accustomed to learning on their own, and both the introduction of the words' pronunciation and the preexposure activities are likely to have been taken as clues about the purpose of the study. On the other hand, performance was substantially below maximum (60% correct for the " Supply Definition " task for the higher­ ability students after 10 sets of paragraphs). The significant effect of word sets raises an interesting question - it may be that the pas3ages were not all equally facilitative of vocabulary learning. What are the critical factors? Nagy, Herman, and Ander:.on, (1985) carried out a study similar in several respects to Jenkins et al., in which they varied the text genre (narrative versus exposi:ory). Surprisingly, they found little difference in the contextual support provided by the two genres. Both passages were difficult, there was only a single instance of each genre, and the in:ormation value of the words to the text was not determined. More questions were raised than answered, and one:: again the focus is on the mate,ials and worksheet practice. Carnine, Kameenui, and Coyle ( 1984) have aiso investigated mediating variables in vocabulary learning. The first 3tage of the study was descriptive ; no instructional treatments were administered. The form of context ciue an