Essential Statistics for Applied Linguistics [2012 ed.] 0230304818, 9780230304819

Table of contents :
Cover
Copyright
Contents
List of Tables
List of Figures
Preface: How to Use This Book
PART I THE BASICS
1. Types of Research
2. Systematicity in Statistics: Variables
3. Descriptive Statistics
4. Statistical Logic
5. Doing Statistics: From Theory to Practice
6. Common Statistics for Applied Linguistics
7. Conclusion
Notes
References
PART II HOW TO SPSS
1. How to Do Descriptive Statistics
2. How to Check Assumptions
3. How to Do a t-test
4. How to Do a One-way ANOVA
5. How to Do a Two-way ANOVA
6. How to Do a Correlation Analysis
7. How to Do a Simple Regression Analysis
8. How to Do a Chi-square Analysis
PART III SPSS PRACTICALS
1. Exploring SPSS and Entering Variables
2. Descriptive Statistics
3. Calculations Using SPSS
4. Inductive Statistics
5. Miscellaneous Assignments
6. Miscellaneous Assignments (Revision)
Index
Página em branco

Citation preview

ESSENTIAL STATISTICS FOR APPLIED LINGUISTICS

Also by the authors Kees de Bot, Wander Lowie and Marjolijn Verspoor, Second Language Acquisition: An Advanced Resource Book Marjolijn Verspoor, Kees de Bot and Wander Lowie (eds), A Dynamic Approach to Second Language Development: Methods and Techniques Monika Schmid and Wander Lowie (eds), Modeling Bilingualism: From Structure to Chaos

Essential Statistics for Applied Linguistics Wander Lowie University of Groningen, The Netherlands University of the Free State, South Africa

and

Bregtje Seton University of Groningen, The Netherlands

© Wander Lowie and Bregtje Seton 2013 All rights reserved. No reproduction, copy or transmission of this publication may be made without written permission. No portion of this publication may be reproduced, copied or transmitted save with written permission or in accordance with the provisions of the Copyright, Designs and Patents Act 1988, or under the terms of any licence permitting limited copying issued by the Copyright Licensing Agency, Saffron House, 6–10 Kirby Street, London EC1N 8TS. Any person who does any unauthorized act in relation to this publication may be liable to criminal prosecution and civil claims for damages. The authors have asserted their rights to be identified as the authors of this work in accordance with the Copyright, Designs and Patents Act 1988. First published 2013 by PALGRAVE MACMILLAN Palgrave Macmillan in the UK is an imprint of Macmillan Publishers Limited, registered in England, company number 785998, of Houndmills, Basingstoke, Hampshire RG21 6XS. Palgrave Macmillan in the US is a division of St Martin’s Press LLC, 175 Fifth Avenue, New York, NY 10010. Palgrave Macmillan is the global academic imprint of the above companies and has companies and representatives throughout the world. Palgrave® and Macmillan® are registered trademarks in the United States, the United Kingdom, Europe and other countries ISBN 978–0–230–30481–9 This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. Logging, pulping and manufacturing processes are expected to conform to the environmental regulations of the country of origin. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress. 10 9 8 7 6 5 4 3 2 1 22 21 20 19 18 17 16 15 14 13 Printed and bound in Great Britain by CPI Antony Rowe, Chippenham and Eastbourne

Contents

List of Tables

ix

List of Figures

xi

Preface

xiii PART I

1.

Types of Research 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10

2.

3.

4.

THE BASICS

Introduction Hypothesis generating vs hypothesis testing Description vs explanation Non-experimental vs experimental Process research vs product research Longitudinal vs cross-sectional Case studies vs group studies Qualitative vs quantitative In situ/naturalistic research vs laboratory research The approaches taken in this book

3 3 4 7 8 10 11 12 13 14 16

Systematicity in Statistics: Variables

17

2.1 2.2 2.3 2.4

17 17 18 19

Introduction Research design Why do we need statistics? Variables and operationalization

Descriptive Statistics

25

3.1 3.2 3.3 3.4

25 25 30 31

Introduction Describing datasets: means and dispersion A different view on variability Frequency distributions

Statistical Logic

39

4.1 4.2

39 39

Introduction The chance of making the wrong decision

vi

Contents 4.3 4.4 4.5 4.6 4.7

5.

6.

7.

Statistical decisions Degrees of freedom Parametric and non-parametric statistics Checking assumptions Alpha and beta

43 46 46 47 48

Doing Statistics: From Theory to Practice

51

5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8

51 52 52 54 55 55 56 57

Operationalization Forming hypotheses Selecting a sample Data collection Setting the level of significance Statistics Interpretation of the data Reliability of outcome

Common Statistics for Applied Linguistics

59

6.1 6.2 6.3 6.4 6.5

59 59 65 72 74

Introduction Comparing groups Assessing relations Analysing frequencies Doing statistics

Conclusion

77

7.1 7.2 7.3 7.4

77 77 79 80

Statistical dangers and limitations Validity and reliability Meaningful outcomes Statistics and the real world

Notes

81

References

83 PART II

HOW TO SPSS

1.

How to Do Descriptive Statistics

87

2.

How to Check Assumptions

91

3.

How to Do a t-test

95

4.

How to Do a One-way ANOVA

101

5.

How to Do a Two-way ANOVA

107

6.

How to Do a Correlation Analysis

113

Contents

vii

7.

How to Do a Simple Regression Analysis

117

8.

How to Do a Chi-square Analysis

121

PART III

SPSS PRACTICALS

1.

Exploring SPSS and Entering Variables

131

2.

Descriptive Statistics

133

3.

Calculations Using SPSS

137

4.

Inductive Statistics

143

5.

Miscellaneous Assignments

147

6.

Miscellaneous Assignments (Revision)

151

Index

155

This page intentionally left blank

List of Tables

I.1.1 I.2.1 I.3.1 I.3.2 I.3.3 I.4.1 I.4.2 I.5.1 I.5.2 I.6.1 I.6.2 I.6.3 I.6.4 I.6.5 I.6.6 I.6.7 I.6.8 I.6.9 I.6.10 I.6.11 I.6.12

Types of research Variable types Mean dispersion calculation Calculating the standard deviation Frequency distribution Error types Making decisions about H0 Example output from SPSS representing the descriptive statistics of a study Example output from SPSS representing the deductive statistics related to Table I.5.1 Choice of statistics for group means analyses Proficiency scores SPSS output descriptives SPSS output ANOVA SPSS output for a post hoc analysis Non-parametric equivalents for some of the parametric means analyses The sum of squares values in an ANOVA table SPSS output correlation analysis SPSS output with partial correlation SPSS output table of the regression coefficients SPSS output using cross-tabulation SPSS output using Chi-square test

16 22 28 29 32 42 43 56 57 60 60 60 61 62 64 65 67 69 71 73 73

This page intentionally left blank

List of Figures

I.3.1 I.3.2 I.3.3 I.3.4 I.3.5 I.3.6 I.3.7 I.3.8 I.4.1 I.6.1 I.6.2

Boxplot with explanation Min-Max variability analysis depicting the change over time of ‘don’t V’ negation A histogram A bar graph A line graph Frequency polygon representing the scores of a very large number of students on a vocabulary test The normal distribution with SDs and percentile scores Skewness and kurtosis Boxplot with early starters and late starters A scatterplot showing the relationship between two variables Regression plot SPSS with regression line through the data

27 31 33 33 34 35 35 37 40 66 70

This page intentionally left blank

Preface: How to Use This Book

This book provides a practical introduction to research methodology, focused on applied linguistics. It is the reflection of an introductory course to statistics and methodology that we have taught over the past seven years. Although the book could also be used for self-study, it was primarily designed to support a research methodology course and was intended to serve as a first-aid kit for students enrolling on an MA in applied linguistics with no or very little background in methodology and statistics. After studying the material described here, students will not have reached an expert level of statistics and methodology. Nevertheless, starting from scratch, this book will help students develop a sufficient working knowledge of statistics so that they will be able to understand the method sections in journal articles and to set up and analyse their own empirical investigations in the field of (applied) linguistics. Moreover, after working through this book, students will be able to extend their knowledge by searching for additional resources independently and adequately. The book takes an activity-based approach. This means that students will first be asked to consider a problem and to try to solve it with the knowledge they have, before the problem is worked out and discussed in terms of conventional methodology. It may be tempting to skip the activities, but that would reduce the learning effect. Therefore, we advise readers to consider seriously all the problems and activities in the book before reading on, even when the answers seem obvious or trivial. Only then can the maximum result of the approach be achieved. This book comes with a set of practical assignments to be used with SPSS. The accompanying website (http://www.palgrave.com/language/ lowieandseton) contains some demonstrations and How To sections using screen captures. These demonstrations are best used to get acquainted with newly introduced techniques, though to really acquire the skills and techniques the practical assignments are best done without making use of the demonstrations. The structure of the practical assignments is built up in such a way that users are first taken by the hand in a step-by-step procedure and then left to apply the newly developed skills on their own. Again, following this procedure will give the biggest chance of success. The How To demonstrations serve as a practical reference guide to refresh the memory on the procedures in SPSS whenever needed.

xiv

Preface: How to Use This Book

In our own research methodology course we always try to show that everyone can learn to understand and apply basic statistics. Over the past seven years, every single student who made a serious effort to acquire statistical knowledge and skills has managed to do so using this approach, even those who had initially considered themselves as ‘hopeless cases’ in this respect. So there is hope, but a serious effort is required. For students who really like the topic, we will offer opportunities to broaden their horizons with references to advanced statistics, so that this group will be sufficiently challenged. Throughout the book, we have included sections with a star in the margin denoting more challenging information or tasks. It may be obvious, but the data that are used as examples throughout the book, in the practical assignments and in the How To sections are mostly fictitious data to illustrate a point. The authors will not accept any claims based on these data. WANDER LOWIE BREGTJE SETON

PART I

The Basics

Chapter 1

Types of Research1

1.1 Introduction The field of applied linguistics is a large one and this means that applied linguists are interested in many issues. Here is a list of a variety of topics: Measuring the effectiveness of early bilingual education: how effective is an early start? The relation between characteristics of linguistic input and language development in early bilingual children. Assessment of problems of elderly migrants learning Swedish. The lag in development in language proficiency of migrant children or children in deprived areas where a local dialect is spoken. The storage of multiple languages in our mind: language selection and language separation: how do we keep our languages apart? The possibility of ‘blocking’ languages we know while we are listening or reading. The impact of learning a third language on skills in the first language (L1) and the second language (L2). The role of interaction in the language classroom: who’s talking, what is the input? The nature of the impact of information and computing technology on language learning. The question of how a threatened language can best be protected. The question of why prepositions in an L2 are so difficult to learn. The question of if and how a forgotten language can be reactivated. This list could be extended for pages. A quick look at the Current Contents list of journals in the arts and humanities, which shows the tables of contents of over 1,000 journals, will make clear that the scope is huge and that, even for journals focusing on second language development, the range of issues is breath-taking.

4

The Basics

Activity 1.1 ¾

¾ ¾

If you were to categorize the list of research topics in three or four categories according to the type of research required, how would you do it? On what grounds would you put a specific topic in a category? How would you label your categories? For four of these topics, briefly work out how you would go about investigating it.

There are many topics of research, but the range of types of research in applied linguistics is much more limited. In this chapter we will provide a systematic overview of different types of research: what are their relevant distinctions and how are they related? There will be no single optimal method for all research topics, since each topic can be worked out in different ways. For your understanding of the research literature, it may be useful to become acquainted with the major categories, so you know what to expect and are able to evaluate the use of a particular design. Also, some familiarity with research terminology and categorization will be helpful in finding relevant research on your own topic. For clarity’s sake, we will make use of contrasting pairs of types of research, but it should be stressed from the outset that these contrasts are actually the extreme ends on a continuum rather than distinct categories and that the categories are partly overlapping.

1.2

Hypothesis generating vs hypothesis testing

Testing and theory formation is a circular process. Theory must be based on empirical findings, and empirical studies can be used to test the theories formulated. One of the issues related to theory formation is to what extent a theory can be tested, that is to what extent the theory can be used to set up hypotheses that can be empirically tested. Quite often, though, this is not a matter of either–or, but a sequential process. Often theories have not been developed yet to the point that real testable hypotheses have been generated. A current example could be the relation between the use of hand gestures and L2 development. We are now only beginning to see the importance of this type of research, and we are still looking for theories that may help us explain why people gesture the way they do (see for instance de Bot and Gullberg, 2010). However, before we can test what explains cross-linguistic influence from L1 to L2 in gesturing, we first need to find out whether there are actually differences in gesturing between the two languages. Once we have established that, we can proceed to think about specific aspects of gesturing, such as the use of gestures with motion verbs.

Types of Research

5

In research reports, we often see phrases like ‘In this study, we test the hypothesis that ...’. However, the formulation of appropriate hypotheses is not always obvious. For instance, if someone claims to ‘test the hypothesis that when second language learning starts after puberty, a native level of proficiency cannot be attained’, then we may wonder what that actually means: is that true for every learner, no matter what? If only one single individual can be found who can achieve this, is the hypothesis then falsified? A hypothesis needs to be narrowed down as far as possible to show what will be tested and what outcomes count as support or counterevidence. Activity 1.2: Formulating a research hypothesis It is not easy to formulate a research hypothesis that is not too broad and not too narrow. The more specific the hypothesis is, the better the chance to test it successfully. The development of a research hypothesis typically goes in stages. A hypothesis like Elderly people forget their second languages easily. is not really a hypothesis, but rather a statement. Which elderly people? Compared to what other groups? Do younger people not forget their second languages? What does ‘easily’ mean here? So we need to narrow it down: Elderly people forget their second languages more quickly than middle-aged people. Still, this is rather broad and some concepts are not clear and we may have to specify the population. The definition of ‘elderly’ and ‘middle-aged’ may be somewhat vague, and in the description of the population it will have to be made clear what the age range is, though for the hypothesis this will do. But do we also want to include elderly people suffering from dementia or other diseases? And do we want to test every part of the language system? Maybe it is better to limit the study to syntax, morphology, lexicon or fluency. And do we want to look at all second languages? How about the level of education, which is likely to play a role? Narrowing the hypothesis down further could result in something like: Healthy elderly people forget words in their first second language more quickly than education matched middle-aged people. ‘More quickly’ is still a bit underdefined, but probably clear enough. Sometimes it may help to break the larger hypothesis down into a number of smaller ones in which more details can be provided. ¾

For three of the topics in Activity 1.1, formulate a clear and specific research hypothesis.

6

The Basics

The discussion on hypothesis testing inevitably leads to the problem of generalizability: When and how can we generalize our findings to more than the people or phenomena we have looked at? In the chapters on statistics, this will be one of the focal issues, because that is precisely what we need statistics for. Here we want to touch on some of the more general issues. Generalizability refers to the extent to which findings in a study can be generalized to a population that is larger than the samples tested. In most cases it is unimaginable that all individuals of a given group can be included in a study. No study of Chinese learners of English will include all those millions of people. What is typically done is that we draw a sample from that larger population. There are different methods for doing this. The most desirable approach is to have a so-called ‘representative’ sample, which means that all the variation in the larger population is represented in the sample tested. The ideal is hardly ever achieved, because it is very difficult to assess what makes a sample representative; we need to know all the traits that may be relevant and should be included in the sampling. No data will tell us exactly what the relevant traits are for drawing a sample from the large population of Chinese learners of English. The best we can do is to guess and use common sense (and all the relevant research there is, of course) to define the sample. This procedure represents the traditional research approach that we will discuss throughout the remainder of the book. However, although the majority of studies in applied linguistics have taken a traditional research approach in which the findings of representative samples are generalized to populations, this is not the only way of investigating second language development. A research perspective that is rapidly gaining ground in developmental psychology and applied linguistics is dynamic systems theory (DST) or complexity theory (CT). This perspective emphasizes the change of development over time and emphasizes the dynamic interaction of factors affecting the (language-) system over time. DST/CT is thus interested in the process of development rather than in the eventual learning outcomes. The focus on the process has important consequences for the research choices that are made. For instance, a DST/CT approach takes into account that developmental processes are complex processes in which characteristics of the individual learner interact with the environment. Every learner is unique, and her developmental path will be affected by the internal structure of her system and her interaction with the environment. Inevitably, there will be variation between individual learners. Individual patterns can tell us in detail how an individual developed and what factors may have played a role, but at the same time we need general tendencies, not only because of educational policy reasons, but also because we need information on the likelihood that a given factor has an impact on development, so that we can include it in the study of individual developmental patterns. This is basically a cycle in which we move from factors that seem to have an effect at the individual level to testing that

Types of Research

7

Activity 1.3 Order the items in Activity 1.1 according to the degree to which the topic could be easily investigated from a DST perspective focusing on individual developmental paths.

effect on a larger sample to obtain an estimate of its strength and then back to the individual level again to study the impact in more detail. An example could be the role of motivation: its effect may be suggested when we look at the learning process of an individual learner who indicates why he or she was motivated to invest time and energy in learning a language at some moments in time and not at all at other moments. To know more about the potential strength of such a factor, we may then do a study on a larger sample of similar learners. With that information, we can go back to individuals and see how the factors that appeared to be important in the larger sample affect the individual learner. The general pattern will be less detailed and typically will not give us information about change of motivation over time in the same way that an individual case study can. A comprehensive perspective on language development means balancing two ways of looking at research: on the one hand one may give more attention to individual developmental patterns, but on the other hand one may accept the relevance of general tendencies for various purposes. It may be true that for some learners there is no impact from the first language when learning the second, and the variation in cross-linguistic influences between individuals may be considerable; but what remains is that in general the first language does play a role in learning a second language. It is important to realize that these are complementary perspectives, each of which is equally relevant. The main focus in this book is on hypothesis testing and generalization, but we should not forget that this is only one side of the coin.

1.3

Description vs explanation

The discussion in the previous section is closely related to the distinction between description and explanation. Before we can explain anything, we first need a good description. For instance, before we can explain why interaction in the classroom is beneficial for second language development, we need to describe what goes on in classrooms: who is actually saying what to whom, how complex the language used is, whether what is said is also understood, and whether the language used is correct or full of errors. Most of the research we do in applied linguistics is descriptive. We describe processes of learning and teaching, naming different factors that play a role in these processes; we describe language policy programmes and their effectiveness; we describe the impact of learning environments on learning; and

8

The Basics

so on. Many studies investigate the effect of X on Y. A randomly chosen issue of one of the leading journals in our field, Studies in Second Language Acquisition, reveals the following contents. There is a study on the effect of ordering input and interaction on language proficiency (Gass and Torres, 2005), one on the effect of reading and writing on word knowledge (Webb, 2005), and one on the impact of negative feedback and learners’ responses to ESL question development (McDonough, 2005). All of these studies describe the effect of the manipulation of one variable on another one. But even if the relation between two variables is established (and it normally is, because journals do not usually publish null-results), then this is not the same as an explanation of that relation. The only conclusion that can be drawn is that the development of Y is influenced by X. To really account for the influence observed, there must always be a rigorous and detailed theory behind the research, which provides explanations for the phenomena. There is another sense of the concept of explanation, which is more statistical in nature. As we will see in Chapter 2, variation within individuals and groups vs variation between individuals and groups is the essence of statistical procedures. We try to explain variation in variable A by looking at the impact of the systematic and controlled variation of variable B. For instance, we look at the variation in the acquisition of new L2 words (variable A) by manipulating the methods of teaching (variable B). If the experiment works, the variation in variable A is reduced, because we have taken out the variation that is caused by variable B. Suppose we give two different methods to two different groups, a ‘strong’ group and a ‘weak’ group. If we look at the learning results of the two groups together, we will find some learners who improved a lot, some less, some not at all. If we look at the groups for methods 1 and 2 separately, we may find that for one method most learners will improve a lot, while for the other group learners improve only slightly or not at all. There is a great deal of variation in the two groups taken together because there will be good and bad learners within each group, while there is less variation in the groups separately because the good learners are in one group and the weaker learners are in the other group. Within the groups the learners are more similar and therefore there is less variation among the individuals in the group. In statistical terms this is referred to as ‘variance explained’, variance being a specific type of variation. The goal in experiments is to explain as much variation as possible, because that will tell us to what extent we can explain a given effect. However, this is not an explanation in the theoretical sense, but a description of the relation between variables.

1.4

Non-experimental vs experimental

Applied linguistics has positioned itself as part of the social sciences and distanced itself from the humanities by adopting research techniques and

Types of Research

9

Activity 1.4 One of the subdivisions we can make with regard to types of research is the one between experimental and non-experimental. ¾ ¾

Which of the topics in Activity 1.1 can best be investigated using experimental research and which with non-experimental research? Mention one disadvantage and one advantage of experimental research.

paradigms based on the science model. In this model, quantitative empirical research and controlled experiments are often considered the only way to make progress. The aim is to decompose complex processes in parts that can be studied and manipulated experimentally. Experiments and statistical manipulations provide ‘hard’ evidence as compared to the ‘soft’ evidence evolving from the more interpretative research that dominates in the humanities. If one looks at the bulk of research as reported in the most prestigious journals and books in applied linguistics, the experimental approach is dominant, though at the same time it is obvious that it is no longer seen as the only way to practise research. The choice of an experimental or a non-experimental approach largely depends on what a researcher wants to know. For a study of the organization of the bilingual lexicon or the perception of foreign accents by native speakers, experimental approaches may be the logical choice; the way human language processing takes place is not open to introspection and we can study this effectively through controlled experiments. Other aspects, such as non-instructed L2 development, we can study better through non-experimental techniques, such as observations and analyses of spontaneous speech. In studying Second Language Development (SLD), a wide range of experimental techniques have been used, ranging from grammaticality judgements to lexical decision tasks, and more recently different neuroimaging techniques have been used that provide insight into brain activity while processing language. A detailed discussion of various techniques is beyond the scope of this book, but a good overview of different experimental techniques that have been used for the study of SLD can be found in Mackey and Gass (2005) and thorough discussions of different brainimaging techniques have been provided by Brown and Hagoort (1999). Sometimes there is a choice between using an experimental technique and a non-experimental technique. The choice for one or the other is determined by the aim of the study. An interesting example is the study of L2 pragmatic competence. Hendriks (2002) studied the acquisition of requests by Dutch learners of English. Her aim was to study the impact of power relations, social distance and conversational setting on the use of politeness strategies in requests. She could have made recordings of requests in

10

The Basics

spontaneous conversations, but she wanted to study the systematic effects of each of these variables and their interactions. She would have needed a very large corpus of utterances to find sufficient examples of requests that differed in terms of power, social distance and setting. Therefore, she decided to use an experimental technique that tries to mimic real life interaction while allowing for systematic variation of variables. The technique used is the Discourse Completion Task. In this task a short description of a conversational setting is presented and the participant has to construct a sentence as a reaction. Here are two examples: The living room You were in your room upstairs doing your maths homework, but you were not able to do the sums. You need some help. You go down to the living room where your dad is watching a documentary on television. What do you say to your dad? The supermarket You are standing in line at the checkout with a shopping trolley full of groceries. You are late for an important meeting. There is one man in front of you. What do you say to the man in front of you? The use of such tasks allows for systematic variation of the variables, but it is of course not a natural setting or real conversation. This shows that the choice for an experimental or non-experimental approach is always a trade-off between controllability and ecological validity. Ideally, data from controlled experiments should be validated through a comparison with ‘real’ conversational data.

1.5

Process research vs product research

The distinction between process research and product research has to do with how change takes place and what the result of the change is. An example may help to explain what we mean by this. There is an abundance of research on different types of immersion education. Not only in a bilingual country like Canada but also in Europe, so-called CLIL programmes have shown that these approaches are very effective. CLIL means Content Based Instruction in North American terms or Content and Language Integrated Learning in European terms. A study by Huibregtse (2001; see also Admiraal et al., 2006) showed that Dutch secondary education students in a CLIL programme outperformed a matched control group with respect to the development of different aspects of English. This is typically a product study: there is an effect, but no indication of what might have caused the effect. To evaluate the causes of the effect, it would be necessary to have a detailed look at what goes on in CLIL classes, what the impact of such an approach is on

Types of Research

11

students’ attitudes to English and how the proficiency changes over time as a function of contact. That would then be the process part of that study. Of course, there are layers in both ‘products’ and ‘processes’. Going back to the CLIL study: there are products in terms of gains in proficiency on different levels (syntax, lexicon, reading, writing), but there can also be more long-term products, such as better jobs or more study abroad with the CLIL students as compared to the control students. There are layers in processes too: we referred to classroom processes, but ultimately we are interested in the impact of external resources on the individual student and how that leads to learning: what are the characteristics of the setting and how does the individual system react to that. Most studies in applied linguistics have been effect studies that focus on the products of language development, but recently the importance of process-oriented research has been emphasized. Investigating processes can be done with test-retest designs (‘repeated measurements’), to see how a certain intervention (like a teaching method) may have affected learning outcomes over time. In such designs there are usually two groups, a control group (that does not get the ‘treatment’) and an experimental group, and there are usually two or three test moments. However, to really make sense of the process of development, more measurements will have to be taken over a relatively long period of time, in what is referred to as ‘longitudinal’ research.

1.6

Longitudinal vs cross-sectional

Longitudinal research is research in which individuals’ development over time is studied. Most studies of children growing up bilingually are examples of longitudinal research: the child is typically video/audio-taped at regular intervals over a longer period of time, sometimes for more than three years, and transcripts of the recordings are analysed with respect to relevant aspects, such as mean length of utterance and lexical richness. But other types of development can also be studied longitudinally: Hansen et al. (2010) looked at the attrition of Korean and Japanese in returned missionaries, who typically acquired the foreign language up to a very high level, used it in their work as missionaries and after they returned hardly ever used that language. This study is unusual, because it is longitudinal with only two moments of measurement in ten years. In most longitudinal studies there are more moments of measurement with smaller time intervals. Such studies necessarily take a long time; even three-year data collection periods may be too short to cover a significant part of the developmental process. And funding agencies are not very keen on financing projects that take more than four or five years. Therefore, the number of longitudinal studies is small, but those projects (like the European Science Foundation study on untutored L2 development in different countries (see Klein and Perdue, 1992; Becker and Carroll, 1997)) have had a major impact on the field.

12

The Basics

Because of the time/money problem of longitudinal studies, many researchers use cross-sectional designs. In cross-sectional research, individuals in different phases of development are compared at one moment in time. For the study of the development of morphology in French as an L2, a researcher may compare first, third and fifth graders in secondary schools in Denmark. Rather than follow the same group of learners for four years as they progress from first to fifth grade, different groups in the three grades are compared at one moment in time. Both longitudinal designs and cross-sectional designs have their problems. In longitudinal studies the number of participants is generally very small because a large corpus of data on that one individual is gathered. Large numbers of participants would make both the data collection procedure and the processing and analysis of the data extremely demanding and time consuming. The small numbers mean that the findings may be highly idiosyncratic and difficult to generalize. As we have seen, this may not be a problem in studies that use the uniqueness of the individual’s development as the central issue, as is normally the case in DST/CT approaches to language development. Another problem of longitudinal studies is subject mortality, the dropping out of subjects in a study. With each new measurement, there is a risk of this happening, and the longer and more demanding the study, the higher the risk of drop out. An additional problem is that in such studies drop out is typically not random, but selective or biased: in a study on acquisition or attrition, subjects that do not perform well will be more likely to lose their motivation and drop out than more successful ones, leaving a biased sample that is even less generalizable. Cross-sectional designs can be problematic, because the assumption that the groups that are compared behave like one group tested longitudinally may not be true. Referring to the example above, there may be specific characteristics of different age groups, such as changes in the curriculum, natural disasters, changes in demographic trends and changes in school population that can make the three groups very different. One solution for this so-called cohort effect is to take more than one cohort; so rather than only testing groups 1, 3 and 5 in year x, one also tests the groups 1, 3 and 5 of the next year or cohort. If the findings for the two cohorts are similar, it is assumed that the groups do not behave atypically. Some studies try to get the best of two worlds by combining longitudinal and cross-sectional designs: in research on ageing such cross-sequential designs have been used frequently, and it was also used by Weltens (1989) and Grendel (1993) in their studies of the attrition of French as a foreign language in the Netherlands.

1.7

Case studies vs group studies

Most longitudinal studies will be case studies, while cross-sectional studies tend to be based on group data. Some studies use multiple cases, and

Types of Research

13

although in terms of numbers they may seem to be the same as a group, the approach is fundamentally different. In case studies, we typically find a holistic approach, which aims at trying to integrate as many aspects that are relevant for the individual case as possible. In group studies, it is similarity rather than difference that counts: a group is selected with specific characteristics – for example, Turkish undergraduate students doing a course on Academic Writing in English – while other differences are either ignored or controlled for by using background questionnaires and specific statistical techniques to cancel out such differences.

1.8

Qualitative vs quantitative

The discussion of qualitative versus quantitative studies would take another book to discuss in sufficient detail. It has been one of the main rifts in the social sciences including applied linguistics over recent decades. It looks as if the fiercest controversy is over now, but the different communities still view each other with distrust. For a long time a researcher had to be in one or the other community, but now it seems acceptable to take an elective stance and use more qualitative or more quantitative methods depending on the type of research question one wants to answer. Following Mackey and Gass (2005, p. 363), the two approaches can be defined as follows: Qualitative: Research in which the focus is on naturally occurring phenomena and data are primarily recorded in non-numerical form. Quantitative: Research in which variables are manipulated to test hypotheses and in which there is quantification of data and numerical analyses From these definitions it follows that the two approaches differ fundamentally in epistemological terms and with respect to the research methods used. Qualitative research is holistic, trying to integrate as many aspects that are relevant into a study. It is also by definition interpretative and therefore in the eyes of its opponents ‘soft’. In qualitative research a number of techniques are used, such as diaries of learners, interviews, observations and introspective methods such as think-aloud protocols (see ibid., ch. 6; Brown and Rodgers, 2002, chs 2–4 for discussions of various methods). One of the main problems is the lack of objectivity in those methods: in all methods, the researchers interpret what goes on, and some form of credibility can only be achieved through combinations of data (‘triangulation’) and the use of intersubjective methods, in which the interpretations of several ‘judges’ are tested for consistency. All of this may not satisfy the objections raised by hard-core quantitativists. For them only objective quantitative data are real ‘hard’ data; for example, there is little a researcher can change or interpret in the latencies in reaction-time experiments. The starting point

14

The Basics

Activity 1.5 ¾

¾ ¾

Which of the topics in Activity 1.1 can best be investigated using longitudinal research and which can best be investigated using cross-sectional research? Mention one advantage and one disadvantage of cross-sectional research. How can the cohort effect be avoided?

of quantitative research is that the entire world is one big mechanism and by taking it apart and studying its constituent parts we will in the end understand the whole machine. Exactly this position is criticized by qualitative researchers. A problem with the experimental and quantitative approach is that it is not always clear what participants in such experiments actually do. There is a substantial set of studies on the recognition of pseudo-homophones (like English ‘coin’ and French ‘coin’, which means ‘corner’ in English). The list of words to be recognized typically consists of many regular words with some of these pseudo-homophones interspersed. The researcher’s hope is that the participants will not notice these words and become aware of the fact that they are special, because that could have an effect on their strategies in processing. To what extent participants actually do notice the trick is often unclear. Participants in such experiments are typically psychology students who have to take part in many different types of experiment and who have accordingly become quite clever in detecting the trick.

1.9 In situ/naturalistic research vs laboratory research In situ or naturalistic research studies a phenomenon in its normal, natural setting and in normal, everyday tasks, while laboratory research both isolates a phenomenon from its normal setting and uses data that are an artefact of the procedures used. Laboratory research aims at finding ‘pure’ effects that are not tainted by the messiness of everyday life. In such studies either the grammaticality of sentences in isolation is tested through grammaticality judgements or the process of lexical access is studied using reaction-time experiments. Experimental laboratory research has reached extremely high standards, mainly through a very successful experimental psychology tradition in North America and Western Europe. Therefore, that type of research is held in high regard and is also used in applied linguistics. The counter-movement that advocates a more qualitative and naturalistic approach has long been marginalized and has created its own subculture and its own journals and societies. Their main argument is that

Types of Research

15

reductionist research has no ‘ecological validity’ in that is does not actually tell us what reality looks like. Researchers in the laboratory tradition have problems countering this argument, because their research does not always lead to the kind of deeper insight it is supposed to bring. The gap between the methods used and the reality it claims to inform us about has become so wide that even researchers themselves may have problems showing the relevance of what they do. To give an example, there is a large body of research on word recognition, mostly using the lexical decision paradigm in which participants are presented with letter strings on a computer screen and are asked to indicate as quickly as possible whether the letter string is a word in a given language or not. At the beginning of this field of research, it was assumed that word recognition data would inform us about the process of normal reading, but over time researchers working on word recognition have developed their own sets of questions that may be only marginally linked to the process of normal reading. Researchers using naturalistic data claim that their research is more ecologically valid because it focuses on the tasks in their normal setting. Up to a point this is probably true, but with the use of different introspective methods, they may also have crossed the line and adapted methods that may create their own type of data that are as far removed from reality as the reaction times and error rates of word recognition researchers. The validity of introspection has been questioned and the core of the problem is sublimely expressed in the title of Klein’s (1989) review of Kasper and Faerch on introspective methods in L2 research: ‘Introspection into What?’. Activity 1.6 Here is the summary of an article by Webb and Kagimoto (2011) from the journal Applied Linguistics: This study investigated the effects of three factors (the number of collocates per node word, the position of the node word, synonymy) on learning collocations. Japanese students studying English as a foreign language learned five sets of 12 target collocations. Each collocation was presented in a single glossed sentence. The number of collocates (6, 3, 1) varied per node word between three of the sets, the position of the node word (+1, −1) varied between two of the sets, and the semantic relationship between collocations (synonyms, non-synonyms) varied between two sets. Productive knowledge of collocation was measured in pre- and post-tests. The results showed that more collocations were learned as the number of collocates per node word increased, the position of the node word did not affect learning, and synonymy had a negative effect on learning. The implications for teaching and learning collocations are discussed in detail. ¾

What do you think are the main characteristics in terms of research types of the findings reported?

16

The Basics

Table I.1.1

Types of research

Hypothesis generating Description Non-experimental Process research Longitudinal Case studies Qualitative In situ research

Hypothesis testing Explanation Experimental Product research Cross-sectional Group studies Quantitative Laboratory research

We have presented various types of research by giving a number of more or less opposite characteristics of research types. In Table I.1.1 the list is repeated again, and a reflection on the characteristics on the left-hand side and on the right-hand side will make clear that the characteristics on either side are related and co-occur: quantitative research is often experimental, based on groups and laboratory research, while qualitative research is often based on case studies in a naturalistic setting aimed at description more than explanation. This is not to say that there is no mixing possible; many researchers nowadays are eclectic and take what suits them without caring too much about what different communities of researchers may say about this. This typology may help you to interpret research and to structure your own investigations.

1.10

The approaches taken in this book

We have elaborated on the necessity of looking at individuals on the one hand and of generalizing over larger groups on the other hand. Only with these complementary research methods will we be able to advance in science and bring different theories closer together. To go into all possible ways of analysing data will be too much for an introductory book on methodology and statistics. In order to do group studies, you will need to know how to advance in structuring your research and how to analyse the data you get. We often encounter (applied) linguists who think of an interesting study to carry out, but who forget to think about how to analyse the data – and then get stuck. Or some report statistics but have forgotten an important step which has caused their results to be unreliable. The aim of this book is to make you aware of how to construct a quantitative study in an analysable way and of the tests you can carry out to check your hypotheses. Although we focus on group studies, we will keep stressing that there is also the possibility of looking at individual data. If you want to learn more about how to analyse data from a (longitudinal) DST perspective, you can consult Verspoor et al. (2011).

Chapter 2

Systematicity in Statistics: Variables

2.1 Introduction An independent-samples t-test showed that the mean difference between the groups turned out to be significant at t (27) = 2.4, p < 0.05. Many students tend to skip the results sections of research reports, because they do not understand the meaning of sentences like the one exemplified above. The unfortunate consequence of this is that they will not be able to fully assess the merits and weaknesses of those studies. Moreover, they will not be able to report on their own studies in a conventional and appropriate way. This is not only unfortunate, but also unnecessary. Once the underlying principles are clear, understanding statistics is a piece of cake! The purpose of this book is for the reader to come to an understanding of some elementary statistics, rather than providing a full-fledged statistics course. We will demonstrate why it is necessary for many studies to apply statistics and which kind of statistic is normally associated with which kind of study. After studying this chapter, you will be able to understand most statistical reports in articles and apply some very basic statistics to your own investigations. We will do this by encouraging you to think about problems in second language research through a set of tasks. Wherever possible, we will ask you to try and find your own solutions to these problems before we explain how they are conventionally solved. Obviously, this learning-by-doing approach will only succeed if you work seriously on the tasks, rather than skipping to the solution right away.

2.2 Research design In the previous chapter, we discussed the relationship between theory and empirical observations for different types of research. In this chapter, we will move on to a more practical application of these observations: to doing and interpreting research. Since an understanding of research based on traditional theories is crucial for a full appreciation of the field of applied linguistics, we will focus on the more traditional methodologies and will

18

The Basics

only occasionally refer to approaches that are more appropriate for the investigation of non-linear development. One of the most important characteristics of any type of research is that it is carried out systematically and according to research conventions that are generally agreed on. The purpose of this chapter is to discuss the most relevant of these conventions and to outline the systematicity of empirical research.

2.3

Why do we need statistics?

Let us start by looking at a practical problem in the following activity. Activity 2.1 A researcher wants to know if the typological similarity of languages affects the acquisition of a second language. In a study, two groups of learners of English are compared. One group has German as their first language and one group has Spanish as their mother tongue. Learners in both groups take an English proficiency test (maximum score is 100) after a fixed number of years of studying English. The data and the mean score for each group are included in the two tables below, each representing a different hypothetical dataset. Dataset 1 Partici pants

L1

Dataset 2

English Score Mean

1 2 3 4 5 6 7 8 9 10

German German German German German German German German German German

25 26 42 46 48 30 36 52 42 39

11 12 13 14 15 16 17 18 19 20

Spanish Spanish Spanish Spanish Spanish Spanish Spanish Spanish Spanish Spanish

31 23 30 37 35 43 28 31 40 15

38.6

31.3

Partici pants

L1

English Score Mean

1 2 3 4 5 6 7 8 9 10

German German German German German German German German German German

35 36 41 46 48 44 36 52 42 39

11 12 13 14 15 16 17 18 19 20

Spanish Spanish Spanish Spanish Spanish Spanish Spanish Spanish Spanish Spanish

88 41 13 73 48 35 29 51 23 32

41.9

43.3

→

Systematicity in Statistics

¾

¾ ¾

19

What can you say about the difference between German and Spanish learners of English in the two different datasets? Can you say if there is a difference between German and Spanish learners? Explain your answer for each of the datasets. Are there any other observations you can make? Is there any additional information you would like to have to assess the value of this study?

The ‘correct’ answers to the questions in Activity 2.1 are not very relevant. The purpose of the exercise is to make you aware that it can be very difficult to make decisions based on empirical observations. It is clear that in dataset 1 the difference between the two groups is rather big, whereas the difference is smaller in dataset 2. Maybe you have noticed that not only the means of the two groups were different, but also the amount of dispersion. Maybe you have started making calculations to help you come to decisions. It will be obvious that without doing some calculations, it is impossible to say if there is a difference between German and Spanish learners regarding their proficiency in English, especially in doubtful cases like dataset 1. Now, wouldn’t it be great if we could feed these numbers into a computer program which would tell us that we can safely decide (beyond reasonable doubt) that there is a difference? That is exactly what we are doing in applying statistics to empirical data. We enter the data in for example SPSS, we select the right calculation for a particular problem, and the program will tell us which decision we can make (beyond reasonable doubt). The main issue for us is the selection of the right type of calculation. The ‘beyond reasonable doubt’ bit is another issue we will have to deal with. But that is basically all you need to know about statistics.

2.4

Variables and operationalization

Many a phenomenon that we want to investigate is not measurable in an obvious way. The first step that is necessary for doing systematic research is what is called operationalization. This means that the abstract phenomenon, or construct, we want to investigate is transferred into a measurable variable. For instance, if we want to investigate someone’s level of L2 proficiency at a certain moment in time, we need a way to express this. The level of proficiency could be operationalized as the number of vegetables that a person can mention in the L2 in two minutes (an example of the ‘verbal fluency test’). Alternatively, the level of proficiency can be expressed in terms of a TOEFL score or someone’s final school grade in that subject. This example immediately shows how controversial operationalization may be. The transfer of a construct into a variable is always the researcher’s own choice; and the validity of the outcomes of the investigation may strongly

20

The Basics

depend on it. If the variable resulting from the operationalization does not adequately represent the underlying construct, the entire study may be questionable. In many studies, the operationalization of constructs into variables is left implicit. It is therefore one of the first questions that critical readers of an empirical study should ask themselves. Activity 2.2 Provide two possible operationalizations of the following constructs: ¾ ¾ ¾ ¾ ¾

A person’s motivation (at one moment in time) to learn a particular foreign language. A person’s intelligence (at one moment in time). The love one person feels for another person (at one moment in time). A person’s pronunciation of English (at one moment in time). A person’s height (at one moment in time).

The answers to the questions in Activity 2.2 may vary wildly and may yield interesting discussions about the validity of empirical studies. However, the most important point appearing from this activity is that there is not one correct answer to these questions and that it is always the researcher’s responsibility to operationalize carefully the constructs into variables. Another important point is that all of the variables operationalized in Activity 2.2 take a synchronic perspective, which is manifest in the additional phrase ‘at one point in time’. It will be obvious that research designs for the investigation on the development of these factors over time may be considerably more complex and require different methods and techniques from the ones needed to investigate one time slice only. Before we carry on, there are two more points we need to discuss in relation to variables: the type of variables and their function in an empirical study. Answering the questions in Activity 2.3 below will make you aware of some of these issues. Activity 2.3 Applying statistics by definition involves doing calculations. Perform the following simple calculations: ¾ ¾ ¾

The height of three persons is 1.58 m, 1.64 m and 1.70 m. What is the average height of these people? A group consists of nine female students and seven male students. What is the average gender of this group? Three students participate in a bike race. At the end of the race, Julia finishes first, Hassan second and Nadia third. They decide to do a second race,

Systematicity in Statistics

¾

21

in which Nadia finishes first, Hassan second and Julia third. In a third race, Nadia is first, Hassan second and Julia third. Who is the overall winner? Four essays are graded using a system that has four scale points: A, B, C and D. If someone scored an A, two Bs and a D, what is that person’s average score?

Each of the four questions in Activity 2.3 includes a variable that represents an underlying construct. For our purpose, the underlying construct is not relevant this time, as we will concentrate on the variables themselves and the operations we are allowed to carry out with different types of variables. The calculation of the first one, a person’s height measured in metres, will not pose any problems. The average height can be calculated by the sum of the values, divided by the number of values: (1.58 + 1.64 + 1.70)/3. The answer is 1.64 m. The second question is more problematic. Why is it impossible to calculate the average sex of this group? The answer must be found in the scale that we use to measure the different variables. The scale used in the first question runs from 0 to ∞ and the scale points occur at regular intervals from each other. This type of scale is therefore called an interval scale or a ratio scale.1 Other examples of this scale are the number of vegetables a person can mention in an L2 in two minutes and the number of correct items in a test. The difference between ratio and interval data is that the ratio scale has a fixed zero. A good example of a scale that is interval but not ratio is temperature in Celsius or Fahrenheit, because the zero-point is quite random and zero does not mean ‘no temperature’ here. For the purpose of applied linguistics, the difference is not very important and we will therefore disregard the difference between the interval and ratio scale, and just refer to it as the ‘interval scale’. The implied scale for the second question is of a different nature. This scale has a limited number of scale points (or levels) that only serve as labels to distinguish categories. This scale type is therefore called a nominal scale. Clearly, mathematical calculations like averaging cannot be applied to nominal variables and the question cannot be answered. Another example of a nominal scale is a person’s nationality. Here too, the different nationalities are no more than labels to which mathematical operations like adding up and averaging cannot be applied. The third question relates to yet another scale that is referred to as an ordinal scale. Ordinal variables may have an infinite number of scale points, but the distance between these points does not occur at exactly the same intervals. Therefore, calculating averages is not possible for ordinal variables. To decide who was the fastest cyclist of the three, we would need the exact distance (for instance in time) between each of the students in each of the three races. As time can be expressed as an interval variable, the results of the three races can be added up.

22

The Basics

To answer the fourth question we would need to decide what the scale is on which the essays were graded. As the distances between the scale points A, B, C and D are not necessarily identical, this scale must be considered as an ordinal one and no averages can be calculated. Users of this type of scale may be tempted to transfer A, B, C and D to 1, 2, 3 and 4, so that seemingly an average can be calculated. However, this suggests an underlying interval scale, so that an essay that is rewarded a ‘4’ is exactly twice as good as one that is rewarded a ‘2’. For most types of essay grading this is not true and the transfer operation is not permitted. Similarly, essay grading on a scale of 1–10 must still be regarded as ordinal and, in theory, no averages can be calculated. Examples of the different variable types are given in Table I.2.1. Applied to research design, the distinction of variable types into nominal, ordinal and interval is extremely relevant, as the choice of operationalizing a construct as a particular type of variable will have important consequences for the calculations that can be performed in the analysis of the results of a study. Activity 2.4 Determine the scale type for each of the variables selected in Activity 2.2.

Another consideration with regard to variables is their function in an empirical study. This step is particularly relevant for quantitative, statistical studies with an experimental or quasi-experimental design. In these types of study, one or more independent variables is systematically changed to investigate the effect of the independent variable or variables to a dependent variable. An example would be a study that investigates the effect of instruction on vocabulary knowledge. The independent variable (the variable that is manipulated) in such a study would be ‘instruction’, which is a nominal variable with two levels: with and without instruction. The dependent variable (the variable that is measured) is vocabulary knowledge. This could be an interval variable that represents the outcome of a vocabulary test (for instance the number of correct items). In this way, we can measure the effect of instruction (independent variable) on vocabulary knowledge (dependent variable). If the vocabulary knowledge were

Table I.2.1

Variable types

Variable type

Characteristics

Example

Nominal Ordinal Interval Ratio

Alphanumeric labels only Rank order Numbers with fixed intervals Numbers with fixed intervals and absolute zero point

Nationality Essay grades Temperature in Celsius Weight in kilogrammes

Systematicity in Statistics

23

with instruction than without instruction, we could conclude that instruction is helpful. Obviously, this example is a gross oversimplification of the reality of such a study and only serves to illustrate the point.

Activity 2.5 A researcher wants to know if training positively promotes speed for runners. To investigate this, she sets up an experiment in which she times two different runners who complete a fixed distance after different training programmes. Runner A has trained twice as much as runner B. For the sake of the argument, we will assume that the two runners are otherwise identical. One of the variables in this experiment is the time (in seconds) it took the runners to cover the track distance. The other variable is the amount of training, represented by the two different runners. ¾

What is the function of each of these variables (dependent/independent) and what are the scales involved?

bigger It can thus be argued that the dependent variable (like the vocabulary knowledge in our example and the running time in Activity 2.5) changes as a result of the independent variable (like the vocabulary learning method or the amount of training). What is measured is always the dependent variable – what we want to investigate is the independent variable. In many experimental studies, more than one independent variable is included at the same time. In more complex designs, more than one dependent variable may be included. Sometimes, researchers decide explicitly not to include a variable in a study. This can be done by not changing the variable, or in other words keeping the variable constant. Such a variable is then referred to as a ‘control’ variable. Applied to the vocabulary study described above, the researcher may decide to include only female learners into the study. We can then say that sex is a control variable. The dependency relationship between variables is not relevant to all types of statistical studies (see Chapter 6).

Activity 2.6 Think of an experimental design in which: ¾

Age is an independent variable

AND ¾

Language proficiency is the dependent variable

AND ¾

The L1 background is a control variable.

24

The Basics

In this chapter we have introduced the basics of statistical terminology: operationalization and variables. Before you start designing a study, it is always good to think about the variables you want to use and to consider which is the independent and which is the dependent variable. After reading this chapter, we recommend doing the first SPSS practical (see Part III) or becoming familiar with any other statistical package that you will be using.

Chapter 3

Descriptive Statistics

3.1 Introduction Basically, two types of statistics can be distinguished: descriptive and inductive. Inductive statistics usually involves hypothesis testing, in which the results of a small sample are generalized to a larger population. Descriptive statistics are used to describe the characteristics of a particular dataset or to describe patterns of development in longitudinal analyses. The next section will focus on the most important notions of descriptive statistics that are commonly used in applied linguistics: mean, range, standard deviation (SD), z-scores and frequency distributions.

3.2

Describing datasets: means and dispersion

One of the most well-known types of descriptive statistic is the mean value. Providing the mean value of a dataset (like the exam grades of a group of students) immediately gives an insight into one of the main characteristics of that dataset. To calculate the mean value, add the values in the dataset and divide the sum by the number of items of that dataset. This operation is summarized by the following equation: X=

∑X N

(I.3.1)

where: X = mean Σ = sum (add up) X = items in the dataset N = number of items Activity 3.1 ¾ Calculate the mean values of the following datasets: 3, 4, 5, 6, 7, 8, 9 6, 6, 6, 6, 6, 6, 6 4, 4, 4, 6, 7, 7, 10 1, 1, 1, 4, 9, 12, 14

→

26

The Basics

A, A+, B, C–, D, D ¾ Describe in what way these datasets are similar and in what way they are different. ¾ What can you say about these kinds of calculations with regard to variable types discussed in Chapter 2 (refer to the last dataset in the list)?

The last dataset in Activity 3.1 clearly shows that it is not possible to perform means calculations for each and every possible dataset. As was explained in Chapter 1, means can only be calculated for interval data and not for ordinal or nominal data. Therefore, we will exclude this dataset from our further discussion. The answer to the first question in Activity 3.1 shows that all the datasets have the same mean value. This goes to show that although a mean value may reveal one important characteristic of a dataset, it certainly does not give the full picture. Two additional descriptive statistics relating to the central tendency are the mode and the median. The mode is the value that occurs most frequently. Some distributions have more than one mode (which are called bimodal, trimodal, etc.) and some have none. The median is the point in the dataset that is in the middle: half of the data are below this value and half of the data are above.

Activity 3.2 What are the modes, the medians and the ranges in Activity 3.1? Provide the mean, the mode, the median and the range of the following dataset: 1, 2, 2, 2, 3, 9, 9, 12 ¾ ¾

Looking at the mean, mode and median provides some useful information about the dataset, but it does not tell us exactly how similar (as in the second set) or different (as in the fourth set) all the items in the set are to each other. To find out more, we need details about the dispersion of the data. One of these is the range of a dataset, which is the difference between the highest and lowest values in the set. To calculate the range, take the highest value and subtract the lowest value. In Activity 3.1 the first dataset has no mode, as all values occur only once. In the second set the mode is 6, in the third it is 4, and in the fourth it is 1. The medians are 6, 6, 6 and 4 respectively. The ranges are 6 (9 – 3), 0 (6 – 6), 6 (10 – 4) and 13 (14 – 1) respectively.

Descriptive Statistics

27

One of the problems of the range is that it is rather strongly affected by extreme values. For instance of the dataset 1, 2, 2, 3, 4, 5, 5, 6, 7, 8, 29 the range will be 28, but that is not very telling for this dataset. What is therefore often done is to cut off the extreme values (25% of the values on either end) and then take the range (see Figure I.3.1). This is called the interquartile range. To calculate the interquartile range, you will have to divide the data into four equal parts. First determine the median, which will divide the data into two equal parts. Then take the median again, but now for each of these parts. The median of the lower half is called the lower quartile; the median of the upper part is called the upper quartile. The range between the lower quartile and the upper quartile is the interquartile range. In the example above, the median is 5, the lower quartile is 2, the upper quartile is 7 and so the interquartile range is (7 – 2) = 5. The quartiles divide the dataset into four equal parts: 1 2 2 3 4 5 5 6 7 8 29 The dispersion of a dataset can be graphically represented in a boxplot. A boxplot shows the median the extreme values and the interquartile range of dataset, and is illustrated in Figure I.3.1.

30.00

*

11

Outlier

25.00

20.00

15.00

10.00

5.00

Upper quartile Median

Interquartile range

Lower quartile 0.00 Variable X

Figure I.3.1

Boxplot with explanation

Notes: In this picture you can see a so-called boxplot with the different quartiles. You can also see that SPSS has taken the extreme value to be an outlier, instead of including it in the boxplot itself. The bottom tail of the boxplot shows the first quartile. The middle two parts are the third and fourth quartiles with the median in the middle. The top tail of the plot shows the fourth quartile.

28

The Basics

Activity 3.3 When we have a large set of data, we prefer to describe a dataset by saying what the ‘average dispersion’ is. For the following data, first calculate the mean of the whole set. Then calculate how far each separate value is away from the mean. ¾ Then calculate the mean of those distances. 3, 4, 5, 6, 7, 8, 9 6, 6, 6, 6, 6, 6, 6 ¾ ¾

Expressing how far on average each value in a distribution is away from the mean reveals the dispersion characteristics for a dataset at a single glance, as it expresses the mean dispersion for that set. In calculating the mean dispersion, we could follow the standard procedure for the calculation of means. For each individual value in the list, we determine how far it deviates from the mean, which is 6 in both datasets given in activity 3.3. For the first dataset, the mean dispersion can be calculated as is done in Table I.3.1.

Table I.3.1

Mean dispersion calculation

Value

Distance from mean (X − X)

3 4 5 6 7 8 9 Σ

3–6 4–6 5–6 6–6 7–6 8–6 9–6

Distance from mean

–3 –2 –1 0 1 2 3 0

The next step would be to divide the sum by the number of items. However, in this case, we have a problem because the sum is zero, which is caused by the positive and negative values being set off against each other. One way to solve this problem is to apply a common mathematical trick: we square all the values and get rid of the negative ones.1 We can do this as long as we undo the operation by taking the square root at the very end of the calculations. Table I.3.2 illustrates the squaring of each distance from the mean.

Descriptive Statistics Table I.3.2

Calculating the standard deviation

Value

Distance from mean (X − X)

3 4 5 6 7 8 9 Σ

3–6 4–6 5–6 6–6 7–6 8–6 9–6

Distance from mean

(Distance)2 2 (X − X)

–3 –2 –1 0 1 2 3 0

9 4 1 0 1 4 9 28

29

Notes: For each and every individual value in the dataset, we subtract the mean value (X − X ) and square the result. Then we take the sum (Σ) of the squares and then the square root of that.

28 = 4. This 7 number is referred to as the variance (s2) of a dataset. The final step is to take the square root of 4, to ‘undo’ the squaring we applied earlier: 4 = 2. So the average dispersion of the first dataset in Activity 3.3 is 2. This value is commonly referred to as the standard deviation (SD). The complete equation for the SD reads as follows and should now become transparent. In the equation, we divide the total by (N – 1) instead of N. The reason for this is related to the difference between a population and a sample, which will be explained in I.4.3, on page 45: The number of items in the list (N) is 7, so the result so far is

∑ (X − X )

2

SD =

(N −1)

(I.3.2)

Activity 3.4 ¾ Calculate the mean and the SD of the following datasets: 3, 4, 5, 6, 7, 8, 9 6, 6, 6, 6, 6, 6, 6 2, 5, 7, 8, 8, 9, 9, 15, 18

The SD represents the amount of dispersion for a dataset. Together with the mean, a clear picture of that dataset begins to emerge. So the SD provides information about how similar or different the individual values are in a dataset. Moreover, it can be used to provide information about how an

30

The Basics

individual item is related to the whole dataset: we can calculate how many SDs an individual score is away from the mean. The number of SDs for an individual item is called the z-score. Let us illustrate this with another example. Suppose a group of students has taken a 100-item language test. Now suppose the mean score for all the students is 60 (correct items) and the SD is 8. A person who has a score of 68 can then be said to have a score that is exactly 1 SD above the mean. That person’s z-score is therefore +1. The z-score of someone who scored 56 on that test is half a SD below the mean, which is –0.5, and a person who scored 80 has a z-score of 2.5 (80 – 60 = 20; 20/8 = 2.5). The advantage of the z-score is that it gives an immediate insight into how an individual score must be valued relative to the entire dataset.

Activity 3.5 In a 40-item language learning test the mean score is 20. Nadia participated in this test, and her score is 30. We know that Nadia’s z-score is +2. ¾ ¾ ¾

3.3

What is the link between the z-score and the SD? What is the SD in this test? If a person scored 15, what would be his or her z-score?

A different view on variability

In this chapter we have thus far concentrated on the mean value of a group and the extent to which individual members of that group deviate from that mean. The mean and the dispersion can give us an insight into the general tendencies for groups. Studies that investigate language development from a DST/CT perspective take a rather drastically different view on variability. In traditional means analysis, deviation is generally regarded as ‘noise’ to be avoided. If we want to make generalizations about groups, we’d want the groups to be as homogeneous as possible. DST/CT studies regard variability as containing useful information about the developmental process. In longitudinal analyses of individual learners we can often see that an increase in variability signals a change in the process. Therefore, in DST studies techniques are used to evaluate the change of variability over time. One of these techniques is the Min-Max graph, which illustrates the degree of variability by plotting the minimal observed value and the maximal observed value over time. Figure I.3.2 is an example of such an analysis, in which the use (number of instances per session) of a particular

Descriptive Statistics

31

Jorge 80 70 60 50 40 30 20 10 0 0

5

10

15

20

Figure I.3.2 Min-Max variability analysis depicting the change over time of ‘don’t V’ negation Notes: The dots represent the percentage of use of this type of negation in each of the recordings: the top line is the maximum value and the bottom line is the minimum value. Source: Verspoor et al. (2011, p.76).

type of negating construction (don’t V) occurs at different moments in time. By creating a moving window of maximum and minimum values, the amount of variability at different points in time becomes clear. Similarly, DST studies make use of means analysis in a different way. Instead of comparing the means of groups, they tend to look at how the mean value of a variable (for instance ‘average sentence length’) changes over time. The longitudinal developmental perspective in DST studies and the group studies we focus on in this book are largely complementary. The choice of perspective will depend on the research question that is addressed in a study.

3.4 Frequency distributions The mean and SD do not tell us exactly how often certain values occur and how they are distributed. For instance, a teacher may need to find how many students had how many items correct on an exam to be able to determine the grade. To find out, he or she can conduct a frequency tally. Tallies are recordings of the number of occurrences of a particular phenomenon. Table I.3.3 exemplifies this for a ten-item test. The tallies refer to the number of correct answers on the test.

32

The Basics Table I.3.3

Frequency distribution

Number of correct items

Number of students with that number of correct items

1 2 3 4 5 6 7 8 9 10

0 1 1 3 7 11 9 5 2 1

Activity 3.6 ¾

Use the blank figure below to draw a graph that represents the frequency distribution exemplified in Table I.3.3. The score in this distribution is the number of correct items. First put a dot for the occurrence of every score, and then use these dots to draw a line.

11 10

Number of occurrences

9 8 7 6 5 4 3 2 1 1

2

3

4

5 6 Score

7

8

9

10

Graphically, a frequency distribution can, for instance, be represented as a histogram, as in Figure I.3.3.

Descriptive Statistics

33

12

10

Frequency

8

6

4

2

0 0

Figure I.3.3

2

4 6 8 Number of correct items

10

12

A histogram

12 10

Count

8 6 4 2 0 2

Figure I.3.4

3

4 5 6 7 8 Number of correct items

9

10

A bar graph

Or it can be presented as a bar graph (Figure I.3.4) or as a line graph (Figure I.3.5).

34

The Basics 12 10

Count

8 6 4 2 0 2

Figure I.3.5

3

4 5 6 7 8 Number of correct items

9

10

A line graph

The different kinds of graphs help us to visualize how the data are dispersed. This brings us to an interesting phenomenon of frequency distributions that are based on natural data like human behaviour. If there are many data points, the frequency distribution will often result in the same bell-shaped line graph that is commonly referred to as the normal distribution. This phenomenon is used as a reference point for almost all of the statistics discussed in the remainder of this chapter. Figure I.3.6 provides an example of the normal distribution. Figure I.3.6 is a representation of the vocabulary scores of a large number of students. The vocabulary test in this example is distributed according to the normal distribution. The normal distribution has a number of typical characteristics. First of all, the mean, the mode and the median always coincide. In this example, they are all 60. In other words, the average is 60, the most frequently occurring score is 60 and, of the remainder of the scores, half will be below and half will be above 60. The extreme low and high values, on the left-hand side and on the right-hand side, occur much less frequently. This makes perfect sense because extreme values are always more exceptional than the middle values. In a frequency polygon of the normal distribution, predictable regularities go even further. Let us assume that in the example in Figure I.3.6, the SD is 9. Now we can instantly see that the vast majority of all the scores in this distribution fall

35

Number of students

Descriptive Statistics

42

53 60 69 Vocabulary score

78

Figure I.3.6 Frequency polygon representing the scores of a very large number of students on a vocabulary test

between –1 and +1 SDs (with scores between 51 and 69), much smaller numbers of scores occur in the adjacent sections (with scores between 42 and 51 or 69 and 78), and very limited number scores occur in the outlying sections that are above 2 SDs. In the normal distribution, the number of scores that occur in the sections related to the SD is always the same. This is expressed in percentile scores. The percentage of scores between the mean score and 1 SD away from that mean is always 34.4%. Between one and two SDs from the mean we always find 13.59% of all scores. Higher than 2 SD above the mean is only 2.14% and lower than 2 SD below the mean another 2.14% can be found. These observations are summarized in Figure I.3.7.

2.14%

13.59%

–2 SD

Figure I.3.7

33.4%

–1 SD

33.4%

X

13.59%

+1 SD

2.14%

+2 SD

The normal distribution with SDs and percentile scores

36

The Basics

Activity 3.7 One hundred people have taken a reading test. The test contains 11 items. The results are as follows: Score

¾ ¾

¾

Frequency

11 10 9 8 7 6 5 4 3 2 1

2 1 2 15 20 30 13 7 5 3 2

0

0

Calculate the mean and the mode. Draw a frequency polygon (line graph). Do you think the results of this reading test are approximately distributed according to the normal distribution? How many people are more than 2 SDs away from the mean? Is this a small or a large percentage? If someone else takes the same test, which score do you think would be the most likely score for this person to get?

In a perfect world we would always see a normal distribution in our data. However, since we test samples from the population, it could be that the distribution deviates from normality. To check this we can look at the shape it takes and we can turn to the values of skewness and kurtosis. Skewness says something about the symmetry of the distribution being more focused to the left or to the right of the curve. Kurtosis says something about the pointedness of the curve. When the values of skewness and kurtosis are zero, they represent a perfectly normal distribution. When they deviate from zero, they also deviate from normality. Examples of distributions when skewness and kurtosis are negative or positive are given in Figures I.3.8(a) to (d).

Descriptive Statistics 25

Frequency

20 15 10 5 0 0

5

(a)

10

15

20

Score

Figure I.3.8(a) Negatively skewed (tail to the left)

25

Frequency

20 15 10 5 0 0

5

Figure I.3.8(b)

10

15

20

Score

(b)

Positively skewed (tail to the right)

25

Frequency

20 15 10 5 0 0 (c)

5

10

15

Score

Figure I.3.8(c) Negative kurtosis (flat distribution)

20

37

38

The Basics 80

Frequency

60

40

20

0 0 (d)

Figure I.3.8(d)

5

10

15

20

Score

Positive kurtosis (pointy distribution)

Activity 3.8 Which of the following examples goes with the distributions illustrated in Figure I.3.8? ¾ ¾ ¾ ¾

The frequency of words in a language. The scores on a test that was too easy. The scores on a test that did not discriminate well amongst students. Giving the same test to all students in a language school (regardless of whether they are beginners, intermediate or advanced learners).

In this chapter we have introduced you to basic descriptive statistics and the concept of a normal distribution. Looking at the descriptives and the distribution of your data is an important step that many people tend to forget when they are analysing data. Before you carry out any statistical tests it is crucial to inspect your data visually and look at the descriptive statistics. This will give you information that statistical tests will not be able to provide you with. After reading this chapter, we suggest you to try Practical 2 in Part III of this book.

Chapter 4

Statistical Logic

4.1 Introduction In the previous chapter, we discussed descriptive statistics that can be used to get a first impression of the data. The next step would be to apply inductive statistics, which will help to evaluate the outcome of the data. To illustrate this, let us go back to the data in Activity 2.1. The descriptive statistics in these tables (the mean) could be extended with other descriptives, like the range and the standard deviation (SD) to get a good first impression. However, it will not provide us with the tools to estimate how certain we can be that the observed difference is not based on a coincidence. Moreover, since it is generally impossible to test entire populations, researchers normally select a small representative sample. Inductive statistics will help us in generalizing the findings from a sample to a larger population. However, there are some important assumptions underlying inductive statistics, which require testing by descriptive statistics. Most inductive statistical tests have been developed with the assumption that the data that are used to perform these tests follow a normal distribution. This means that when a sample shows data that are not normally distributed, we really need to be careful in applying statistical tests. Luckily, some clever statisticians have developed so-called non-parametric tests that can be used for data that deviates from a normally distributed pattern. In this chapter, we will introduce the logic behind inductive statistics and discuss the issue of generalization from samples to populations.

4.2

The chance of making the wrong decision

Suppose we want to compare two groups that participated in a vocabulary test. Let us assume these two groups represent two different starting ages for learning a second language – ‘early starters’ and ‘late starters’ – and that each group took the test after four years of L2 learning. Let’s say the mean score in the young group is 32 with an SD of 4.5 and the mean score in the old group is 38 with an SD of 3.2 (see Figure I.4.1(a)). If we disregard all other possible differences between the groups (which is hard to imagine) and assume that the starting age is the only independent variable involved, how

40

The Basics

can we evaluate the difference between these groups? Is a difference of 6 on this vocabulary scale large enough to say that there is a difference and that, in general, it is better to start late in learning a second language? And what if we had only two points difference (as in Figure I.4.1(b))? Would that also be proof that starting later is better? Or was it simply a coincidence that the older group in this sample scored better? In our example of vocabulary scores, this question may be very interesting, but being wrong about interpreting the results of our test would not be matter of life or death. In many cases, the results of tests or surveys have large consequences, and it is important that we can evaluate the differences between test scores and make sure that if we gave the same test to other similar groups we would get very similar results. The purpose of this section is to show how statistics can be helpful in determining how such differences between scores can be evaluated. 45

Score

40

35

30

25 Young

Old Group

Figure I.4.1(a) Boxplot with early starters (M = 32, SD = 4.5) and late starters (M = 38, SD = 3.2)

45

Score

40

35

30

25 Young

Old Group

Figure I.4.1(b) Boxplot with early starters (M = 34, SD = 4.5) and late starters (M = 36, SD = 3.2)

Statistical Logic

41

Activity 4.1 Return to your answer to Activity 3.5, a 40-item language-learning test with a mean score of 20 and an SD of 5. Let’s assume that the normal distribution applies to this test. ¾

¾

Draw a frequency polygon for this test and draw vertical lines at the mean and at 1 and 2 SDs from the mean in both directions (see Figure I.3.6 for an example). One person who participated in this test tells you she scored 34. Someone else tells you she scored 22. One of them is not telling the truth. Who would you believe? And why?

Activity 4.1 should help you see the usefulness of statistics. The best guess for the last question is probably to say that the person who scored 22 was telling the truth, while the other one was not. Using the normal distribution as a starting point, we know that the person who says she scored 34 would have had a score that is between 2 and 3 SDs from the mean and that her score would belong to less than 2.14% of all scores. The person who said she scored 22 would have had a score that is within 1 SD of the mean. As about 33% of all scores fall within this range, it is simply more likely that the person who scored 22 was right. It should be noted that it is certainly possible for someone to have scored 34, but if we have to make a decision based on these figures the chance of making the wrong decision is bigger if we decide that the person who says she scored 34 was right. In other words, doing statistics is all about calculating the chance of making the wrong decision and about comparing a limited dataset (like the two persons from the example) to a larger group of people. What are the chances of making the wrong decision? Logically, there are two types of errors possible when making a decision based on observations. Imagine a teacher has rather convincing evidence that a student has cheated, but he cannot be 100% certain. The immediate consequence of cheating is that the student will be excluded from further participation on a course. There are two possible correct decisions: the student did not cheat and is not excluded, or the student did cheat and is excluded. There are also two wrong decisions possible: the student did cheat but is not excluded, and the student did not cheat but is excluded. At least from the student’s point of view, the latter is a more serious error. In statistics the error of assuming that something is true which in reality is not true is commonly referred to as the alpha error. The other error type (assuming something is not true which in reality is true) is known as the beta error. The error types are summarized in Table I.4.1.

42

The Basics Table I.4.1

Error types

Decision made

Really cheated? Yes

No

Excluded

OK

α-error

Not excluded

β-error

OK

To avoid logical errors, one of the first steps of conducting a statistical study is to formulate specific research hypotheses about the relationship between the dependent and independent variables. When we apply this reasoning to the strongly simplified example about vocabulary acquisition by young starters and old starters, we can formulate three possible hypotheses: young starters do better, old starters do better, or there is no difference between the old and young starters. For each of these hypotheses, a decision table can be set up like the one in Table I.4.1. However, the main hypothesis that is tested is the null-hypothesis (H0), which states that there is no difference. The other two possible hypotheses are referred to as the alternative hypotheses (H1 and H2): the old starters perform better, or the young starters perform better. The reason for stating the null-hypothesis is that it is impossible to prove a hypothesis right, but it is possible to prove a hypothesis wrong; in other words, we can only reject a hypothesis. This principle of falsification is best illustrated with an example. Suppose we want to investigate whether all swans are white. If we formulated our hypothesis as ‘all swans are white’, we can accept this hypothesis only after making sure that all swans in the world are indeed white. However, we can easily reject this hypothesis as soon as we have seen one non-white swan. Therefore, hypotheses are always formulated in such a way that we are able to attempt to reject them, rather than accept them. In research terms, this means that we will normally test the rejectability of the nullhypothesis. In our vocabulary testing example, we would thus test the hypothesis that there is no difference between early and late starters of L2 learning. If we can reject this hypothesis beyond reasonable doubt, we implicitly accept the alternative hypothesis that there is a difference. The mean scores of the older and younger learners will then reveal which group performs better. Suppose one group does indeed score better than the other. How then can we be sure that this difference in score is not just by chance? In other words,

Statistical Logic Table I.4.2

43

Making decisions about H0

Decision made

Reality H0 false

H0 true

H0 rejected

OK

α-error

H0 accepted

β-error

OK

what do we mean by beyond reasonable doubt? This question is related to the alpha error that was introduced in the previous paragraph. The alpha error concerns the possibility that we incorrectly reject the null-hypothesis (and thereby implicitly incorrectly accept the alternative hypothesis). This is illustrated in Table I.4.2. To avoid the possibility that we reject the H0 incorrectly, we try to calculate the degree of chance that might have been involved in obtaining these scores. If we assume that our groups behave normally, with a normal distribution, then we can ascertain which scores are the least likely to have occurred. Recall the frequency polygon of the normal distribution (see Figure I.3.7). The small areas on either side of the mean beyond 2 SDs refer to a little over 2% of the scores. These scores are not that likely to occur and might have occurred by chance. Therefore, scores in this area are related to the conventionally accepted probability of an alpha-type error (abbreviated as p for probability), which is 2.5% on either side of the distribution, and 5% in total.

4.3 Statistical decisions In the social sciences, the generally accepted chance of incorrectly rejecting the null-hypothesis should be 5% or less (p < 0.05), but in cases of life or death, the chance of making an alpha error is usually set to 1% or less. It is important to realize that it is the researcher’s choice to set the alpha level and thereby define the term ‘beyond reasonable doubt’. The selected chance of incorrectly rejecting the null-hypothesis is closely related to the level of significance. A significant result is one that is acceptable within the scope of the statistical study. The level of significance expresses the researcher’s choice of the alpha error and must always be added to a research result. An expression like ‘a statistical analysis showed that the difference between the

44

The Basics

two groups was significant’ must always be followed by, for example, ‘at p < 0.05’. This should be interpreted as ‘I accept that there is a difference between the two groups and the chance that I make an error in assuming this is smaller than 5%’. Activity 4.2 A researcher has investigated the relationship between reading skills and listening comprehension. She reports the following: rxy = .56 (p < 0.001). ¾ ¾

What exactly is meant by ‘p < 0.001’? (Be as explicit as possible.) What is the implied null-hypothesis?

We have now seen that in statistics we can never say that a hypothesis is true; we can only express the chance of making the wrong decision. Conventionally, if that chance is 5 or 1% (depending on how serious it is to make the wrong decision), this is taken to be acceptable. In Figure I.3.7 (of the normal distribution) we saw that values between z = –2 and z = 2 make up about 96% of the scores of the entire population (95.72% to be precise). What z-scores do you think will belong to the 95% confidence interval? The answer should be ‘slightly less than 2’. To be precise, the z-scores –1.96 and 1.96 are the critical values related to the top 5% of the distribution (2.5% on either side). If the starting point is that the chance of making the wrong decision should be smaller than 1%, then the critical values (z-scores) for being 99% certain about a decision are –2.58 and 2.58. Now suppose we want to calculate the relationship between height and the age of children from birth to ten years old. The null-hypothesis would be: there is no relationship between age and height. One alternative hypothesis is that the older the children are, the taller they get, which sounds quite plausible. In this case, the relationship cannot really become negative: that is, it would be quite impossible to see children grow smaller as they get older. In cases like this, we would only have to consider one side of the spectrum, which is called one-tailed testing, and only one alternative hypothesis would be needed. In most cases, two-tailed testing is used because it is uncertain in which direction the results will go. Although in some extraordinary cases one-tailed testing is allowed, it is usually better to test two-tailed. This is because it becomes easier to reject your null-hypothesis in one-tailed testing, since you are disregarding one side of the distribution. When allowing a 5% chance of incorrectly rejecting the null-hypothesis, this is usually 2.5% on either side of the distribution, while with a one-tailed distribution, the 5% occurs only on one side. Therefore, to be on the safe side, two-tailed testing is preferred, unless one of the alternative hypotheses is theoretically impossible.

Statistical Logic

45

Activity 4.3 Think of two different null-hypotheses that test one aspect of second language acquisition. One of these should clearly be a ‘two-tailed’ problem (so there are two alternative hypotheses) and one should be a ‘one-tailed’ problem (so there is only one alternative hypothesis). ¾

For each, formulate the null-hypotheses and the alternative hypotheses as precisely as possible.

Activity 4.4 For the population Greek symbols are conventionally used, whereas Latin symbols are used for the sample. What do you think is the relation between: ¾ ¾ ¾

p and α? X and μ? SD (or s) and σ?

As we have described above, inductive statistics can help us in generalizing the findings from a sample to a larger population. For the statistical calculations, the difference between the sample and the population has some important consequences. The generalizability of the results found in the sample to the entire population will ‘automatically’ be taken into account in the calculations done by, for instance, SPSS. We have already seen one example of this: in Equation I.3.2 of the SD we used N – 1 for the number of participants in the group, rather than simply N. N – 1 can be seen to be a correction for the generalization (also see section I.4.4 below on degrees of freedom). To preclude confusion in whether we are talking about a sample or a population, the conventions are that Greek symbols are used for the population (σ for standard deviation, μ for mean, α for the chance of making the wrong decision), while Latin letters are used for the sample (S or SD for the standard deviation of the sample, M or X for the mean, and p for the chance of making the wrong decision). Activity 4.5 A statistic that you will regularly come across in SPSS output is the standard error of the mean (SE). This figure expresses how well the sample mean matches the population mean. It can be calculated by dividing the standard deviation of the sample (SD) by the square root of the sample size. ¾ ¾

What is the SE when the SD is 9 in a test with 9 participants? What is the SE when the SD is 10 in a test with 100 participants?

46

4.4

The Basics

Degrees of freedom

In reporting our statistical outcomes, we also report the degrees of freedom (Df). Df is a necessary correction that is needed for the calculation of the probability of the sample and is related to the number of ‘free choices’ that can be made. The principle can best be explained by a simple example. Suppose we have five numbers and a total that is given. Say the total is 100. We can then randomly choose four numbers, though the last number will always be fixed. If we choose 11, 41, 8 and 19, the fifth number must be 21. In a calculation of probability we then say that there are four (5 – 1) degrees of freedom (Df). For our purpose it will suffice to understand that Df is related to the total size of the sample. The bigger the sample size, the larger the Df, the more lenient we can be in rejecting the H0 and the more relevant is the value. When you are comparing different groups, the Df is determined by taking the total number of participants in the groups and subtracting the number of levels of the independent variable (which is the number of groups). If you have two groups and 20 participants in each group, the Df value would be (20 + 20) – 2 = 38. Activity 4.6 How many participants does each group consist of, when the numbers in each group are the same, for the following Dfs and number of groups: ¾ ¾

4.5

Df = 28, 2 groups? Df = 57, 3 groups?

Parametric and non-parametric statistics

As has been mentioned, inductive statistics can be divided into parametric and non-parametric tests. The choice between these types of statistics depends on the type of data that are analysed and on the question as to whether all assumptions of the tests have been fulfilled. For instance, when interval data are used and when the assumption of normality of the distribution is met, parametric statistics can be used, like a t-test, or an ANOVA. However, if we want to analyse ordinal or nominal data, or if the interval data in the sample does not prove to comply with the assumptions, we will have to use non-parametric tests. These tests are in general a bit stricter and take into account that the data is for example not normally distributed. Parametric tests are only allowed when the data: follow the normal distribution; are interval data;

Statistical Logic

47

of different groups have about the same amount of variance (homogeneity of variance). If your data does not match one of these points, you will need to apply non-parametric tests, which do not involve the calculation of means. These tests are generally less powerful and often more difficult to interpret. In the following section, we will elaborate on how to test whether normality and homogeneity of variance are not violated. Note that the assumptions we mention here are needed for the basic statistical tests, but that for some tests other assumptions play a role. Whenever this is the case, this will be pointed out in the discussions of these specific tests. An important additional assumption for all parametric statistics is that the relationship between variables is linear. For instance, in investigating the effect of motivation on language attrition, the assumption is that levels of motivation have a linear effect: equal amounts of motivation will have equal effects on attrition. Advocates of DST/CT approaches to language development have argued that relationships between variables are often non-linear and change over time. Applied to the example, this implies that at different moments in time the same amount of motivation may have a different effect on language attrition. This also means that the more variables are involved in non-linear relationships, the less predictable the development will be. Therefore, these researchers have argued that parametric statistics are of limited use for the evaluation of language development. Consequently, those who take a purely DST/CT approach to language development only make use of descriptive statistics and non-parametric statistics in addition to non-linear statistics and longitudinal time-series analyses. These analyses go well beyond the scope of this book, but it is important to realize that the choice of statistics is strongly related to the theoretical framework that is taken.

4.6 Checking assumptions One of the assumptions of parametric statistics is normality of the distribution. We have seen that we can draw a frequency plot to assess whether the curve approaches the bell-shape. We have also shown how the values of skewness and kurtosis can give a rough idea of the normality of the distribution. However, if you want to make a more objective decision on the normality of the data, you can run a separate statistical test to see if the normality hypothesis can be reliably assumed (for instance with 95% confidence). Such a test is the Kolmogorov-Smirnov test and the Shapiro-Wilk test. In Part II, How to 2, we will show how these tests can be done in SPSS. Another important assumption of parametric statistics is the homogeneity of variance, which refers to the similarity in distribution within the

48

The Basics

groups. The assumption is that the variance within each of the populations is the same. The variance is very similar to the SD, because it is the total sum of the deviation from the mean. Since the deviations from the mean are based on squared values (to neutralize the negative values) the notation for variance is s2 or σ2. For the application of parametric statistics the variance in all groups must be approximately the same. A rule of thumb to check whether different groups show similar variance is to compare the SDs. If one SD is more than twice as big as that for another group, then you know that the variance is not homogeneous. The t-test (the most commonly used test to evaluate differences in means between two groups) is especially very sensitive to violations of this assumption. Therefore, the SPSS output table for the t-test includes a test for homogeneity of variance: Levene’s test. If this test is significant, this means that the assumption is violated. Different from the violation of the normality of the distribution, this violation can usually be corrected for, so that in most cases parametric statistics can still be used. Levene’s test can also be done separately (see Part II, How to 3).

4.7

Alpha and beta

In section 4.2, we talked about setting our alpha to for example 5 or 1% and about the risk of getting an alpha error (or Type I error) or a beta error (or Type II error). The alpha error is important, because we would not want to incorrectly reject our null-hypothesis. This means the stricter we are, the less chance we have of getting an alpha error. The beta error is the chance that we incorrectly accept the null-hypothesis. Although this error may seem to be less serious, it is also an error we will want to avoid. Similar to why we have a convention for the alpha error (of a maximally 5% chance of making the wrong decision), the conventionally accepted beta error is 20%. This means that if there really is an effect (like the difference between the groups), we want to be 80% certain that we really find that effect in our sample. This number, 80%, is the desired power of an experiment: the power is 1 – β, since β is 0.20. The power of an experiment is logically related to the number of participants in a sample. Theoretically, if the entire population were tested, we would be 100% certain that we find an effect that really exists. The smaller the samples are, the weaker the power and the more difficult it is to demonstrate an existing effect. If you want to check if the power of your experiment is at least .8, you can use a free program called G*Power to calculate this (see the accompanying website for a link to this program: http://www.palgrave.com/language/ lowieandseton). The chance that we find an existing effect is not only dependent on the sample size, but also on the size of the effect. A big effect can be found with a limited sample size, but to demonstrate a small effect we will need really

Statistical Logic

49

large samples. The following numbers are suggested for this by Field (2005, p. 32) based on work by Cohen (1992). Using an alpha level of 0.05 and a beta level of .2, we would need 783 participants (per group in a means analysis) to detect a small effect size, 85 participants to detect a medium effect size and 28 participants to detect a large effect size. This means that to assess meaningfulness based on the power of an experiment, we will have to calculate the effect size of the experiment and we’ll have to be able to interpret the effect size. But what is a ‘small’ and what is a ‘big’ effect size? There are several statistical calculations that can be used for this purpose. We will discuss this when we dig more deeply into inductive statistics (in section 6.2).

In this chapter you have encountered many of the terms that are used in statistics. Understanding these terms will help to prepare you for the actual statistical tests that you will encounter in the next chapters. You can now do the assignments of Practical 3 that go with this chapter. In this practical we will give you a first introduction to one of the statistical tests that will be discused in the following chapters.

This page intentionally left blank

Chapter 5

Doing Statistics: From Theory to Practice

In this chapter we will go back to our example in Activity 2.1, in which we compared the two groups with L1 Spanish and L1 German, both of whom were learning English. There are several considerations we have to keep in mind before we can actually say that one group scores better than the other. To determine the significance of the outcome of a study, we have to consider the following seven steps:

5.1 Operationalization The first step is to operationalize the constructs under investigation into variables and thus to the scale for each variable (see section 2.4). At this stage it should also become clear how the different variables are related to one another. In other words, the function must be determined (as dependent, independent or control) for each of the variables in the research design. It is important to realize that a particular variable may serve as a dependent variable in one design, but as an independent variable in another. If a pharmaceutical researcher wants to investigate the effectiveness of a new antiageing medicine, he or she may regard the variable age to be a dependent variable and taking or not taking the medicine as the independent variable. However, if an applied linguist wants to investigate the effect of age on second language development, he or she may regard age as the independent variable and some proficiency measure as the dependent variable.

Activity 5.1 ¾ ¾ ¾ ¾

How would you operationalize the example with the German and Spanish learners of English? What is your independent and what is your dependent variable? Do you need a control variable? What scales are they measured with?

52

The Basics

5.2 Forming hypotheses The second step is to formulate the hypotheses that are tested. Even though the null-hypothesis remains implicit in many reports on statistical studies, it is crucial to understand that this is in fact the hypothesis that is tested. Apart from the null-hypothesis, the alternative hypotheses have to be made explicit at this stage. Another question is whether there is one or more alternative hypothesis that logically needs to be considered. For instance, if the relationship between the independent and the dependent variables can only take a single direction, this has relevant consequences for the considerations and calculations that are done at a later stage. An example of this would be a study investigating growth in relation to age in small children. It can be assumed that a person’s height can only increase with increasing age and never decrease. The hypothesis in this case only has to consider one side of the spectrum (one-tailed testing) and only one alternative hypothesis is therefore needed. In case of doubt, it is usually better to test two-tailed. Activity 5.2 ¾ ¾

5.3

What is the null-hypothesis and what are the alternative hypotheses in the example with the German and Spanish learners of English from Activity 2.1? Should you test one-tailed or two-tailed?

Selecting a sample

The third step is to select a sample that is representative for the population that you wish to investigate. For instance, in a study investigating the possible difference in language aptitude between men and women, it must be ensured that the participants selected represent the same population or subpopulation. If the women in this study are all university language students and the men are all professional mechanics, there is a fair chance that the outcome of this study is meaningless in spite of its possible significance. There are two ways of ensuring that the sample is representative. The first is to make sure that the sample is selected purely at random from the entire population. This means that the researcher finds men and women that do not belong to any subpopulation other than that determined by sex. Theoretically, this could be achieved by randomly picking men and women from a purely random list, like a telephone directory of all people on earth. This example will illustrate that, for such research questions, pure random sampling for large and general populations is very difficult to achieve. The second way of eliminating differences between participants is to focus on a very specific subpopulation (or stratum) by including all possibly

Doing Statistics

53

interfering factors as control conditions. Subsequently, a random selection can be made from this specific stratum. In the language aptitude example, this could mean that the study exclusively focuses on professional mechanics who are working at the same company, who have always lived in the same area, who went to the same school, who belong to the same socio-economic class, etc. The problem in this case may be that there are not enough female mechanics that live up to these criteria. Testing very specific populations has the disadvantage that the results cannot be generalized beyond that specific situation. In reality, random samples are normally taken from limited populations that can be assumed to be generally representative for the variables involved. However, the critical reader must be aware of this choice. The results of a language aptitude study among university students in northern England may say nothing about the distribution of the exact same variables among mechanics in the Indian province of Punjab. Strictly speaking, studies in which the sampling has not been done purely at random cannot be considered experimental studies. This type of study is then referred to as quasi-experimental. It will not be surprising that many studies in the field of applied linguistics belong to this group.

Activity 5.3 In many language studies, samples are taken from naturally occurring groups like a specific class at a specific school. ¾ ¾

What are the possible dangers of such a sampling strategy? How can you make sure your samples of German and Spanish learners of English are comparable?

For any study that uses representative sampling, it is crucial that the sample size is sufficiently large. But how can it be determined whether a sample size is sufficient? The required sample size is related to the effect size. The stronger the effect, the smaller the sample needed to demonstrate that effect in a statistical study. The strength of the effect in relation to the sample size is referred to as the power of a study. Although the experimenter has little control over the size of the effect, it may be possible to estimate this and to take a sample that is more than sufficient to demonstrate the effect. In language studies, the sample size is often confined by practical limitations. In these situations, a very small sample size of, say, 20 cases for each level might be the maximum number a researcher can practically use. This will often mean that small effects cannot be demonstrated in these studies (see section 4.7 for more details about this).

54

The Basics

5.4 Data collection The fourth step is to gather the data of the sample. In doing so, different kinds of dangers are lurking. Some of the dangers relate to the selection of the participants of the study, others relate to the method of elicitation that is used. We will briefly mention some of these dangers, but the list is far from exhaustive. Since it can be difficult to find participants for a study, it is tempting to work with volunteers only. The danger in this is that volunteers may well be the ones who tend to be very good at a particular skill. For instance, in testing pronunciation skills in a second language, it is not likely that very poor pronouncers will voluntarily participate. In more general terms, this danger is often labelled as self-selection. Volunteering is one example of this. Another example would be a study in which participants can choose which group they want to join. Although self-selection should be avoided, for some studies it may be the only way of carrying out the investigation. It is the researcher’s responsibility to take this into account when drawing conclusions about the study. The best studies from a methodological point of view are those in which neither the participants nor the people who conduct the study are familiar with the research hypotheses. Researchers may (subconsciously) influence the procedure due to the expectations they have (researcher expectancy) and participants may want to please the researcher and respond in a desirable way rather than objectively (subject expectancy). In gathering data, researchers have to take into account what has been referred to as the observer’s paradox. A researcher who wants to investigate the pronunciation of, say, /r/ in certain words could simply ask people to pronounce those words. However, these people may notice that the task is about the pronunciation of /r/ and (possibly subconsciously) adjust their pronunciation to a more desirable variety. In such a case, the presence of the observer will affect the observation. To avoid this, an elicitation method must be used that does not reveal the purpose of the study.

Activity 5.4 One way to investigate if the pronunciation of /r/ is related to socio-economic status would be to ask people from different social classes to read out some words containing /r/. However, asking people to pronounce words may affect their pronunciation. ¾

Think of a strategy for eliciting these data while avoiding the observer’s paradox.

Doing Statistics

5.5

55

Setting the level of significance

The fifth step in carrying out a statistical experiment is to set the level of significance (alpha). As discussed in 4.3 and 4.4, for language related studies this is usually set at 5%, meaning that we accept a chance of 5% that we incorrectly reject the H0. At this stage it should also be decided whether the testing should be done two-tailed or one-tailed. This is related to the number of alternative hypotheses that are required. If all alternative hypotheses except one can be disregarded on theoretical grounds, the results can concentrate on that H1 only and one-tailed testing can be applied. However, if there is even the smallest chance that an H2 is possible, two-tailed testing is required. In case of doubt it is therefore safe always to apply two-tailed testing. For example, in a vocabulary experiment in which one group of participants is given additional training and the other is not, we would want to determine whether the training has promoted learning. In such an experiment, it is highly unlikely that the group that did not receive additional training would perform better than the trained group. This could be an argument for applying one-tailed testing.

Activity 5.5 ¾ ¾

When you want to compare the German and Spanish learners of English from Activity 2.1, what does it mean to take an alpha of 5%? How will you know if there is a difference between the two groups?

5.6 Statistics The sixth step is where the statistics come in. Of course, the researcher will always first consider the descriptive statistics of the data – the mean scores of the dependent variable for the different levels of the independent variable, the range and standard deviation that belongs to these means, and other descriptives that are needed to provide a clear first impression of the data gathered. However, the most exciting part is to test if the null-hypothesis can be rejected beyond reasonable doubt. To do this, the researcher first has to select which statistical calculation (or statistic) has to be used. The choice of the most appropriate statistic depends on the number and type of variables as well as on the relationship between the variables. We will use the well-known statistic called the t-test to illustrate this. The t-test can be used only to evaluate the results of two groups (levels) representing one nominal independent variable on the scores of an interval dependent variable. An example would be an IQ test that is given to two different groups (like men and women) to find out which group has a higher score. In

56

The Basics

all other cases (more than two groups, no nominal independent, no interval dependent, etc.) a different statistic will have to be used. The t-test makes use of the following descriptives: the number of participants in each group, the mean scores of each group, and the standard deviations of each group. Although in our computer age it is very unlikely that anyone will ever have to calculate the value of t manually, we will give the calculation in order to provide more insight: t=

X Group1 − X Group2 SDGroup12 NGroup1

+

SDGroup2 2

(I.5.1)

NGroup2

The outcome of this calculation expresses the magnitude of the difference between the groups, taking into account the variation within the group (the SDs) and the group sizes (the Ns). Values of t range from minus infinity to infinity, but values between 0 and 5 (or –5 and 0) are most commonly reported. If there is no difference between the groups, X Group1 − X Group2 will be zero, so the closer the value of t approaches 0, the weaker the difference between the groups will be.1 Activity 5.6 ¾

5.7

Say in your own words what the equation for the t-test is doing. In your description, do not use technical terms (like ‘standard deviation’ or ‘variance’) but language that people without a background in statistics would also be able to understand.

Interpretation of the data

When the value of the statistic has been calculated, the next step is to interpret that value by determining the probability (the chance of incorrectly rejecting H0) associated with that value. The simplest way is to feed in the data into a computer program like SPSS and to read the probability from the output. An example of an SPSS output is represented in Tables I.5.1 and I.5.2. Table I.5.1 Example output from SPSS representing the descriptive statistics of a study Group Statistics group score

N

Mean

Std. Deviation

1

9

50.6667

14.06236

2

9

64.4444

12.20769

Doing Statistics

57

Table I.5.2 Example output from SPSS representing the deductive statistics related to Table I.5.1 Independent Samples Test t-test for Equality of Means t score

-2.220

df

Sig. (2-tailed) 16

.041

Note: Independent sample test carried out using t-test for equality of means.

Table I.5.1 reveals that two groups participated in a study, both of which contained nine participants. As the means and the SDs of the dependent variable are given, we may assume that the dependent is measured on an interval scale. At face value, the scores in Group 2 are higher than the scores in Group 1. The question is whether that difference is significant at p < 0.05. In other words, can the H0 (that there is no difference between the groups) be rejected within the default 95% certainty? The answer to this question can be found in Table I.5.2.2 The calculated value of t is –2.220. For this value, the chance of incorrectly rejecting the H0 that is related to these specific conditions is 0.041. As this value is smaller than the 0.05 that was set beforehand, the H0 can be rejected. However, had we selected an alpha level of 0.01, this result would not have been significant and the H0 would have to be accepted.

5.8

Reliability of outcome

When we look at the outcome of our statistics, we may also be interested in whether the experiment or exam we created was in fact a good one. An exam is reliable when students would perform the same if they had to take the exam again (test-retest reliability), but it is also reliable when students who are equal receive the same score. These two ideas of reliability are difficult to test, because you cannot really test students again and again on the same test and you can also never be sure how equal students are. There is another type of reliability that we can test. When we create a foreign language exam we expect our better language learners to outperform the less good students on every question. If the poor students do better on one of the questions than the better students, what does that tell us about that question? Clearly, it indicates that this might not be a reliable question for the test. In other words, there should be a strong (positive) correlation between the items in order to have a reliable exam. The easiest way to check this is to use a measure of split-half correlations. With this method we can divide the scores in two (for example odd and even questions or part 1 and part 2 of an exam) and check the

58

The Basics

correlation between them. The problem is that a difference is produced depending on how you make this division. Every time you make a new division, you will probably get a different correlation coefficient. A way to escape this is to use Cronbach’s Alpha. This method randomly divides up all the items to relate each score of each participant to the scores of the other participants. If you get a high outcome for this test (higher than .7), then this means that your items are highly correlated. You can also see which items are not representative for your exam or experiment. If one item is negatively correlated with the other items, then this means that the item is not a very reliable one.

In Chapter 5, we have given an overview of the steps that are taken in a study, from operationalising the variables and forming the hypotheses to analysing and interpreting the data. We also introduced you to the first statistical test. In the next chapter, we will elaborate on different statistical tests and the categories they belong to. Before moving on to the next chapter, we suggest doing Practical 4 in Part III of the book.

Chapter 6

Common Statistics for Applied Linguistics

6.1 Introduction In the preceding chapter, we illustrated statistical testing with the t-test. For all other statistics, the same principles hold and the same steps can be applied; the only differences are the scale and number of the variables involved. In this section, we have ordered the statistics according to three main types of studies. It should be noted that this is only one possible way of presenting these statistics and some of the types may overlap. For each of the parametric options we will also give you the non-parametric alternatives. Remember that for parametric tests you will always need to make sure that your data is normally distributed, shows homogeneity of variance, and is measured on an interval scale. If one or more of these assumptions are violated, you should opt for a non-parametric test (see section 4.5 for a brief explanation). As we stated in Chapter 4, all parametric statistics are based on the assumption of linear relationships between variables. For step-by-step guides to some of the most common statistics and detailed instructions on how to check assumptions, see Part II or the accompanying website (http://www.palgrave.com/language/lowieandseton).

6.2 Comparing groups The first type of study uses a research paradigm in which the mean scores of different groups are compared. The t-test example discussed in Chapter 5 belongs to this type. This type of study typically has one or more nominal independent variables (the grouping variables) and one or more interval dependent variables. The number of groups related to the same independent variable is referred to as the level of that variable. The analyses we will discuss go up to three levels for each variable. Table I.6.1 shows the relevant statistical test for the number of independent and dependent variables and the number of levels within the independent variable.

60

The Basics

Table I.6.1

Choice of statistics for group means analyses

Number of nominal independent variables

Number of levels (groups) for the nominal independent variable

Number of interval dependent variables (‘scores’)

1

2

1

1 1 2 N N

1 More than 2 Any Any Any

2 1 1 1 Any

Table I.6.2

Statistic to be used

Statistical expression

Independent samples t-test Paired-samples t-test One-way ANOVA Two-way ANOVA n-way ANOVA MANOVA

T T F F F F

Proficiency scores

Participant number

L1 background

1 2 3 4 5 6 7 8 9 10 11

Spanish Chinese Spanish Sutu Sutu Spanish Chinese Chinese Sutu Spanish Chinese

Table I.6.3

English proficiency score 45 56 38 62 43 36 66 72 46 34 66

SPSS output descriptives Descriptives

score N

Mean

Std. Deviation

Spanish

20

49.4000

5.62326

Chinese

20

60.5000

7.00751

Sutu

19

53.4211

9.41226

An example of a study that uses a nominal independent variable with more than two levels would be one in which an interval proficiency score is compared for learners with different L1s. The different L1s, say Spanish, Chinese and Sutu, would each form a different level of the variable L1. A fictitious (and not very representative) research result for this type is exemplified in Table I.6.2.

Common Statistics for Applied Linguistics

61

The null-hypothesis of this study would be that there is no difference between any of these groups. The descriptives for these scores are given in Table I.6.3. Activity 6.1 ¾ ¾

What are your first impressions about the group results summarized in Table I.6.3? Referring to the data in Table I.6.2, what are the dependent and the independent variable(s) and which statistic should be used to test the significance of the difference between these groups (consult Table I.6.1)?

At face value, the mean proficiency scores for the Chinese learners are the highest; the scores for the Spanish learners are the lowest. The Spanish learners are the most homogeneous (they have the smallest SD) and the Sutu learners show the largest differences within their group. The question we want to answer is whether the H0 can be rejected. The appropriate test according to Table I.6.1, the one-way analysis of variance (one-way ANOVA), is a test that calculates F, which represents the proportion of the variance between the groups to the variance within the groups.1 To reject the H0, we would obviously prefer a large difference between the groups, while the variance within the groups (as expressed by the SD) should be as small as possible: F=

Variance between groups Variance within groups

(I.6.1)

In Equation I.6.1 the value of F increases with increasing between-group differences, but decreases with increasing SDs within groups. So the greater the value of F, the more likely it is that we can reject H0. Whether or not we do actually reject it depends not only on the value of F, but also on the sample size (as expressed by Df) and on the level of significance that we have chosen. Running a one-way ANOVA in SPSS yields the result in Table I.6.4. Table I.6.4

SPSS output ANOVA ANOVA

score Sum of Squares

df

Mean Square

Between Groups

1262.212

2

631.106

Within Groups

3128.432

56

55.865

Total

4390.644

58

F 11.297

Sig. .000

62

The Basics

This table shows that even for a very strict level of significance of 0.001, the H0 can be rejected. Because the table only provides three decimal places, we don’t know the exact value of p, but we can say that the significance value is less than 0.001. Apparently, there is a difference between the three L1s for the English proficiency scores. But that is not the end of our analysis. We would want to know if the scores for each of the groups differ significantly from both of the other groups. To test this, it would be tempting to do three t-tests, one for each L1 pair. However, every time we run the test we are again allowing a 5% chance of making the wrong decision. If we ran several t-tests on the same sample, we would ‘capitalize on chance’ and the eventual level of significance would be more than 5%. Therefore, any program that can calculate the F for a one-way ANOVA also provides the opportunity to run a post hoc test. This test compares the difference between all the levels of the independent variable and corrects for capitalization of chance by applying a so-called Bonferroni correction. Table I.6.5 is the output of a post hoc analysis for the current example: This table shows that only the mean proficiency of the Spanish and Chinese learners differs significantly at p < 0.001 and that the mean proficiency of the Chinese and the Sutu learners differs significantly at p < 0.05. Table I.6.5

SPSS output for a post hoc analysis Multiple Comparisons

score Bonferroni (I) groups Spanish Chinese Sutu

(J) groups Chinese

Sig. .000

Sutu

.296

Spanish

.000

Sutu

.014

Spanish

.296

Chinese

.014

Notes: The Bonferroni correction is one of the most frequently used post hoc tests. It is a rather strict test, but it works well for small sample sizes. In the table, one group of learners (I) is compared to the other groups of learners (J).

Activity 6.2 Suppose we added an additional independent variable to the research design discussed in this section, for example sex, as suggested in the next paragraph. ¾ ¾ ¾

In such a case, could you still analyse the data using a one-way ANOVA? If not, which statistic should you choose? Use Table I.6.1 to help you make a decision.

Common Statistics for Applied Linguistics

63

We will not give examples for all other means analysis; the principles and interpretations of these are largely the same as those for the t-test and the one-way ANOVA. Step by step guides to running a t-test, a One-way ANOVA and a Two-way ANOVA are worked out in Part II. A final note on group designs concerns cases where more than one independent variable is included in a research design. A complicating but often interesting factor in these cases is that the independent variables may interact. For instance, if the Spanish, the Chinese and the Sutu groups are subdivided into male and female learners, the nominal variable sex is added to the analysis. It is possible that the effect of the L1 on scores is different for the male and female learners. In that case we speak of interaction of variables. In the output of statistical tests, the significance of interactions is always reported separately and explicitly.

More advanced means analysis When you start to get the hang of research, you will probably start to add more variables to your research designs. For example, you might want to look at the differences between learners from two age groups learning French as a second language, but you want to do so by comparing vocabulary scores to listening and reading skills. Or you may want to investigate the differences between these groups at different points in time using a vocabulary test. This means you would not only have two levels for your independent variable age, but also that you have more dependent variables. In the first example, vocabulary scores, listening skills and reading skills are independent of each other, although they might correlate (the more vocabulary knowledge, the better the listening and reading skills). This can be tested using a multivariate ANOVA (see Table I.6.1). In the second example, the different measures of vocabulary knowledge at different times cannot be called independent of each other, as they are repeatedly measuring the same thing. The latter should therefore be tested with a so-called repeated measures design, which treats the related dependent variables as different levels of one dependent variable.

Non-parametric alternatives Now, let’s consider the examples discussed above and suppose the proficiency scores of your Chinese, Spanish and Sutu groups were essay grades. In this case, you have ordinal data and you should not do a one-way ANOVA. If your data have violated the assumption of either normality or homogeneity of variance, or if your data is of an ordinal nature, you can do a Kruskal-Wallis test. This is a non-parametric test that uses the order of your data from the lowest to the highest values to create ranks. Table I.6.6 shows the most common non-parametric equivalents for some of the parametric tests that we deal with in Part II and Part III.

64

The Basics

Table I.6.6 analyses

Non-parametric equivalents for some of the parametric means

Parametric test

Non-parametric equivalent most commonly used

Independent samples t-test Paired samples t-test

Mann-Whitney U Wilcoxon

One-way ANOVA

Kruskal-Wallis H

One-way repeated measures ANOVA

Friedman

Effect size In Chapter 4, we discussed the relationship between the beta error, or chance of incorrectly accepting the H0, and the power of the experiment. We saw that the larger the sample size is, the bigger the chance is that we will find an effect that actually exists. We also saw that the power is related to the effect size. Big effects can be detected with relatively small sample sizes (about 28 participants per group if the beta error is set to 20%), but to detect small effects large samples are required. In other words, trying to find a small effect with a limited number of participants may be a waste of time, as it is like finding a needle in a haystack. Therefore, it is important for the evaluation of the meaningfulness of a result that the effect size is calculated. The effect size is expressed by r. The outcome of this statistic can be interpreted as a correlation coefficient (see 6.3), although the calculation itself is not the same. The effect size is related to the amount of variance that can be explained by the variables in our experiment. The explained variance is calculated by taking r2. This means that if r = 0.50, r2 = 0.502 = 0.25. So 25% of the variance is explained by the variables in our experiment. This is considered a large effect. Conventionally, the interpretation of effect size is as follows: r = 0.1 (r 2 = 1%) is considered a small effect; r = 0.30 (r 2 = 9%) is considered a medium effect; r = 0.50 (r 2 = 25%) is considered a large effect. For each statistic, this r for effect size can be calculated. For the t-test, it is relatively easy to calculate the effect size once the calculations of t have been carried out: r=

t2 t + df 2

(I.6.2)

So if t = 2 and Df = 36, then r = 0.32, which is a medium size effect.

Common Statistics for Applied Linguistics Table I.6.7

65

The sum of squares values in an ANOVA table

SSM

ANOVA

score Sum of Squares

df

Mean Square

Between Groups

3330.833

2

1665.417

Within Groups

9119.900

57

159.998

12450.733

59

Total

F 10.409

Sig. .000

SST

For the one-way ANOVA, the calculation of the effect size is calculated by the sum of squares of the model (SSM) and the sum of squares of the total (SST). These values are pointed out in Table I.6.7. To calculate the effect size of a one-way ANOVA, we use the following formula: r=

SSM SST

(I.6.3)

For more complicated means analyses the calculation of the effect size becomes more problematic, and this is a discussion that is more suitable for an advanced statistics book.

6.3 Assessing relations If we want to investigate the relationship between two interval variables, we cannot use a group analysis, for the simple reason that there are no groups. The relationship of interval variables is commonly tested with a ‘family’ of statistical techniques that is referred to as correlations. An example of a correlation study is a research design in which it is investigated whether the number of hours of language instruction in English as a second language is related to English language proficiency, if language proficiency is measured on an interval scale. It might be expected that someone’s proficiency in English increases with increasing duration of language instruction. The relationship between two interval variables is commonly plotted as in Figure I.6.1.

66

The Basics

Proficiency

60

40

20

0 0

Figure I.6.1

100

200 300 Hours of instruction

400

500

A scatterplot showing the relationship between two variables

Looking at this figure (a scatterplot), we can see that after a limited number of hours of instruction, the proficiency scores are rather low. We can also observe that after about 200 hours of instruction the scores no longer seem to increase further. Although this is a fictitious example, it is clear that a graph like this one can provide an important insight into the relationship of the two variables under examination. However, this cannot be the end of our statistical analysis, as we would want to test whether these two variables are significantly related. In correlation studies, the interpretation of the null-hypothesis is not about a difference but about a relation. The H0 that is tested in a correlation study is that there is no relationship between the two variables. The statistic to be used to test this is the correlation coefficient. If both variables are measured on an interval scale, the most appropriate statistic is Pearson r, or rxy. The value of rxy runs from –1 to 1. If rxy = 0, there is no relation. An rxy bigger than 0.5 or smaller than –0.5 signifies a moderately strong relationship. The more closely rxy approaches 1 or –1, the stronger the relationship is. A strong negative relationship means that if the values of one variable go up, the other one goes down. In Second Language Development research, negative correlations are sometimes reported for the starting age of L2 learning and the level of

Common Statistics for Applied Linguistics Table I.6.8

67

SPSS output correlation analysis Correlations Hours of instruction

Hours of instruction

Pearson Correlation

Proficiency 1

Sig. (2-tailed) N Proficiency

Pearson Correlation Sig. (2-tailed) N

.705** .000

28

28

.705**

1

.000 28

28

**. Correlation is significant at the 0.01 level (2-tailed).

L2 proficiency that is ultimately attained. The older a person is when he or she starts learning the language, the lower the eventual attainment in L2 proficiency tends to be. Besides the strength of a correlation, the output of a Pearson r analysis also reports on the significance of the correlation, that is, the chance of incorrectly rejecting the H0 that there is no relationship. In the example mentioned above, on the relationship between the hours of instruction and L2 English proficiency, the SPSS analysis yields the output shown in Table I.6.8. From this it follows that the two variables in our example show a rather strong correlation of 0.7. The H0 can safely be rejected, as the level of significance is smaller than 0.001 – the chance of incorrectly rejecting H0 is less than 1%. A common pitfall in correlation studies is to take a correlation as a causal relation. Although it is tempting to say that the increasing proficiency scores are caused by the number of hours of instruction, this is certainly not testified in a simple correlation study. The correlated variables are not in a dependency relation. Therefore, the distinction between dependent and independent variables is not relevant for correlation studies. To determine causality, advanced statistical methods are required, which are referred to as causal modelling. Clearly, this type of statistics goes beyond the scope of this book. Activity 6.3 Eight students have participated in a reading test and a listening comprehension test. Reading ability and listening comprehension are operationalized by the variables R and L respectively. Both variables are measured on an interval scale. The results have been summarized in the table below.

→

68

The Basics

Student 1 2 3 4 5 6 7 8

¾ ¾ ¾

R

L

20 40 60 80 100 120 140 160

65 69 73 77 80 84 89 95

What would be H0 if we want to test the relationship between reading and listening comprehension? Draw a plot of the results. A computer has calculated for these data that rxy = .996. What would be your first impression?

Partial correlations The previous example of the relationship between proficiency and the number of hours of language instruction tests the influence of language input. However, teachers may also want to know whether there is an influence of other factors involved. In this case, we may also want to take into account the amount of hours a student spends on his or her homework. If we also find a positive relationship between proficiency and the amount of hours of homework, how do we know whether both measures of total hours do not influence each other as well, and are therefore embedded in both correlation outcomes? In order to find out what influences the proficiency score more, we can run a partial correlation. This takes into account the variance caused by one variable and removes it from the outcome of the other variable. As you can see in Table I.6.9, the correlation of proficiency and time spent on homework becomes smaller when ‘hours of instruction’ is taken as a control variable. You can do the same with ‘time spent on homework’ as a control variable. Now we know that there is also a positive correlation between proficiency and time spent on homework without the influence of hours of instruction.

Common Statistics for Applied Linguistics Table I.6.9

69

SPSS output with partial correlation Correlations

Control Variables -none-

Proficiency

Proficiency

Time Spent homework

Hours of instruction

1.000

.809

.705

.000

.000

26

26

1.000

.565

Correlation Significance (2-tailed)

. 0

df Time Spent homework

Correlation

.809

Significance (2-tailed)

.000 26

0

26

.705

.565

1.000

Significance (2-tailed)

.000

.002

26

26

1.000

.701

df Hours of instruction

Proficiency

Correlation Significance (2-tailed) df

Time Spent homework

.002

Correlation

df Hours of instruction

.

.

0

.000 0

Correlation

.701

Significance (2-tailed)

.000

df

.

25

25 1.000 . 0

Regression When doing research, we are often interested in how significantly different or how significantly related two variables are. However, especially in applied linguistics, it is often worthwhile to know which variables contribute the most to the outcome. For example, a language teacher may ask you how much motivation or time spent doing homework plays a role in learning a language. To find out the answer to this question, you can do a different type of correlation analysis called regression analysis. An elaborate discussion of this type of analysis goes beyond the scope of this introductory book, but just in case you are interested in doing regression analyses, we will help you to get started. The idea of a regression analysis is that it can be used to predict the outcome on the basis of multiple variables. So all you do in a regression analysis is find out what are the most important predictor variables in your study and how much they contribute to the variance of your data. If we take the example of the correlation above, we can take the scatterplot and draw a regression line through the data points. A regression line will be the

70

The Basics

closest fitting linear line that can model and therefore predict your data. As you can see in Figure I.6.2, this regression line starts around the proficiency score (Y) of 25. This starting point is called the intercept (b0). The amount the line goes up per hour of instruction is called the slope (b1). So if we want to know the equation for this line, we take the slope (b1) multiplied by the number of hours of instruction and we add this to the intercept (b0). Of course, the students in this study deviate from the line, and the deviation of any particular person (X) from the regression line model is termed the error (ε). So in reality, we need to add the error of a particular person to the equation to get the real data points. The equation for the model then becomes: Outcome Yi = b0 + b1X1 + εi

(I.6.4) 2

R Linear = 0.497

Proficiency (Y)

60

40

SLOPE (b1)

INTERCEPT 20 (b0)

0 0

100

200

300

400

500

Hours of instruction (X)

Figure I.6.2

Regression plot SPSS with regression line through the data

Activity 6.4 Using Equation I.6.4, calculate what the expected score would be for the following people. Take into account the fact that you do not know the error. ¾ ¾

John had 200 hours of instruction. What is his proficiency score according to the model? Elena had 17 hours of instruction. What is her proficiency score according to the model?

→

Common Statistics for Applied Linguistics

71

¾

Belinda has a proficiency score of 50. How many hours of instruction should she have had according to the model?

¾

Can you think of a problem with these kinds of data? What would have happened to the regression line if you had only measured up to 200 instruction hours? What do you think would happen if we had had more data from people with more hours of instruction? Will the slope go up or down? Should the regression line be linear here?

¾ ¾ ¾ ¾

We can run a simple regression analysis to calculate the exact numbers for the intercept and the slope. Table I.6.10 shows that the constant or intercept is 26.06 and that the slope is 0.10. Table I.6.10 SPSS output table of the regression coefficients Coefficientsa Unstandardized Coefficients B

Model 1

(Constant)

Std. Error

26.058

4.206

.103

.020

Hours of instruction

Standardized Coefficients Beta

t

.705

Sig.

6.195

.000

5.068

.000

a. Dependent Variable: Proficiency

What we did in the simple regression analysis, can also be done with more variables in a multiple regression analysis, with which you can rule out the importance of certain variables. As you probably realized in the activity, your dataset can influence the model a lot depending on what you measure. You can choose to use different regression lines, for example a logarithmic model, to fit your data. The most important point is to make sure that you are aware of the choices you make. In section 7 of Part II, we have included a detailed step-by-step guide to help you do your own simple regression analyses.

Non-parametric alternatives So far, we have only discussed correlations for interval data. Correlations can also be calculated for ordinal data. This type of correlation is based on mean rank orders. The most commonly reported correlation statistic for ordinal data is the Spearman Rho (ρ). The interpretation of this is identical to that of Pearson r. When you have small sample sizes and if many scores are similar it may be better to report Kendall’s Tau (τ). Regression can also be done with ordinal or categorical data. In this case, it will be logistic regression.

72

The Basics

6.4 Analysing frequencies The last type of analysis that we will discuss here concerns that used for designs that have no interval variables at all. In those designs it is not possible to calculate means, as these require variables measured at an interval level. If a study wants to investigate the relationship between two ordinal variables, a Spearman correlation can be carried out. But in all other cases (for instance, nominal independent with ordinal dependent, nominal variables only) no statistics can be used that require regular calculations, and we will have to apply non-parametric statistics (see section 4.5). Although a wide range of non-parametric statistics is available, here we will focus on those cases where there are nominal variables only. As no calculations can be done with nominal variables, the only thing we can do is count the number of occurrences. Therefore, this type of analysis is commonly referred to as frequency analysis. The example we will use here is a sociolinguistic study in which a researcher wants to investigate the relationship between socio-economic status and the use of two alternative grammatical constructions for negation, ‘haven’t got’ and ‘don’t have’. The nominal variable ‘construction used’ has been operationalized as the number of times the participants used one of these constructions in an elicitation task. ‘Socio-economic status’ (SES) has been operationalized as the answer to a series of questions about income and education and is represented by two levels: ‘higher’ and ‘lower’. The null-hypothesis is that there is no difference in the use of the grammatical construction for the two levels of SES. The frequency data of two nominal variables are most clearly presented in a cross-tabulation. The cross-tabulation of the data from this particular study is shown in Table I.6.11. This table shows that the participants from the lower SES used ‘haven’t got’ 58 times and ‘don’t have’ 30 times. The participants with a higher SES used ‘haven’t got’ 68 times and ‘don’t have’ 62 times. At face value, it looks as if there is a difference in the use of this construction for the participants in the lower SES, but that the constructions are approximately levelled for the higher SES group. The question we would want to answer, however, is whether this difference is significant. What is the chance of incorrectly rejecting H0? The statistic that is most appropriate for this type of data is the Chisquare (or χ2) analysis. This calculates the number of occurrences in a particular cell relative to the margins of that cell, that is, the total number of instances of each construction and the total number of participants in each group. The value of the statistic, χ2, runs from 0 to infinity; the most common values reported are between 0 and 10. The (simplified) SPSS output of this Chi-square analysis is represented in Table I.6.12. This outcome shows that if we had chosen a level of significance of 0.05, we would be allowed to reject H0. As the H0 states that there is no difference in the use of ‘haven’t got’ and ‘don’t have’ between the two social classes, we can infer the reverse.

Common Statistics for Applied Linguistics Table I.6.11

73

SPSS output using cross-tabulation social class * Reply Crosstabulation

Count Reply

social class

haven't got

don't have

lower

58

30

88

higher

68

62

130

126

92

218

Total

Total

Table I.6.12 SPSS output using Chi-square test Chi-Square Tests Value Pearson Chi-Square

3.980

Asymp. Sig. (2-sided)

df 1

.046

The analysis in our example yields a result that can be interpreted without any problems. However, if we want to analyse the relationship between nominal variables that have more than two levels, the interpretation may be less obvious. If the χ2 analysis of a four-by-four cross-tabulation yields a significant result, the only conclusion we can draw is that the values in the cells are not equivalent. For frequency analyses the effect of the different levels cannot be determined by post hoc analyses. Activity 6.5 In this activity you will manually calculate the value of the Chi-square statistic. Normally you will use a computer program like SPSS to do these calculations for you (see How to 8 in Part II), but it is useful to do one yourself to gain a better insight into this statistic. At the university theatre a movie about Spanish culture is shown. Before the movie starts, 70 randomly selected people who enter the theatre are asked whether they are planning on visiting Spain in the next two years. Of the 70 people, 55 answer NO and 15 answer YES. The movie shows all the wonderful characteristics of the Spanish language and culture. After the film, 80 people are asked the same question, again randomly selected. This time 5 people answer NO and 75 people answer YES. The question the producers of the film are interested in is whether the film affects the number of prospective visitors to Spain. ¾ ¾

Draw a cross-tabulation similar to the one in Table I.6.11 for this study and fill in the observed value (FO) for each cell. Label the cells A and B for the top row and C and D for the bottom row.

→

74

The Basics

¾ ¾ ¾

Calculate the ‘marginal frequencies’ by adding up the totals of A and B; C and D; A and C; B and D; and the total frequency: A, B, C, D. What is your H0? Choose an alpha level.

The first step for calculating the appropriate statistic for this situation is to determine for each cell the ‘expected frequency’ (FE). This is expressed as the product of its margins (row x column), divided by the total frequency. So for cell A, the expected value is: (A + B)(A + C) TotalFreq

(I.6.5)

Now calculate the expected value for each cell. Are any of these expected values below 5? The next step is to calculate FE – FO for each cell. For some cells, this is bound to result in negative values. To neutralize these, square the outcome of each cell: (FE – FO)2. Finally, divide this value by the FE: (FE − FO )2 FE

(I.6.6)

Then add up this value for each cell (A, B, C and D):

∑

(FE − FO )2 FE

(I.6.7)

As no square root is taken for the resulting value, this remains a ‘squared’ value. The name of this statistic is therefore ‘Chi-square’ (χ2). ¾

Now calculate the Df. For the χ2 , this is defined as the (number of rows – 1) x (number of columns – 1).

6.5 Doing statistics In the previous sections we have discussed some of the most frequently used statistics in applied linguistics. The next step would be to put your newly acquired insights to the test. To this end, we have listed a few example studies in Activity 6.6. The assignment is to reflect on these studies in a systematic way. The very first thing to do in these cases is find out what is being tested (the construct) and how it is being tested (the variables). Drawing up a list of variables in a study and determining the function of each of these is a good way of systematically approaching general descriptions of studies. We then need to determine the scale of each of the variables. As we saw

Common Statistics for Applied Linguistics

75

in section 4.5, the scale of a variable determines which calculations can be carried out. For instance, no means analyses can be done for nominal variables. The next step is to find out which type of statistical test would be most appropriate for analysing the results. This is best done by first determining the family of statistics: is it a study that compares the means of two or more groups? What we are then dealing with is a means analysis. The most appropriate type of statistical test for means analyses can be found in Table I.6.1. Or is it the case that we are dealing with a study that investigates the relationship between two or more interval variables (related to the same participant)? If so, we are then dealing with a correlations study (see section 6.3). When a study only has nominal variables, it is not possible to calculate means or to run correlations analyses, so we will have to use non-parametric statistics, like counting frequencies of occurrence (see section 6.4). Then, to make sure that you know what you are testing, explicitly state the relevant hypotheses. This should not be limited to the null-hypothesis only, but should include the alternative hypotheses and the direction of the effect. So, for instance, when you are testing the difference between two groups, the H0 is that there is no difference, the H1 is that Group 1 has higher scores that Group 2 (Group 1 > Group 2) and H2 is that Group 2 has higher scores than Group 1 (Group 2 > Group 1). Some of the other decisions that will have to be made are the choice of the α-level and whether you want to test one-tailed or two-tailed. The default answers ‘α < 0.05’ and ‘two-tailed’ will be most frequent, but it is a good habit to be aware of the fact that this is the researcher’s decision. Finally, we would like you to anticipate possible pitfalls of the tests. Pitfalls could be found in the operationalization and the validity of the study (see section 7.2), but also in the required assumptions, like homogeneity of variance and normality of the distribution. Simply use your common sense in considering the pitfalls. Please note that these assignments are also used in Practical 5 in Part III, where you will be asked to run the related analyses, based on sample data, and evaluate the results. Activity 6.6 For each of the following imaginative studies, do the following: ¾ List the variables in the study. If relevant, say which ones are dependent and which are independent, and for each determine its type (nominal, ordinal, scale). In the case of independent variables, how many levels does each have? ¾ Identify the family of statistics (means, frequency or correlation) and indicate which statistical test would be the most appropriate for assessing the results of the study. ¾ Formulate a null-hypothesis (or null-hypotheses in the case of multiple tests) and the relevant alternative hypotheses.

→

76

¾ ¾ ¾

¾

¾

¾

¾

¾

¾

The Basics

Which α-level would you use and why, and would you test one-tailed or two-tailed (and why)? Comment on the possible methodological pitfalls of the test. 1. A researcher wants to investigate whether motivation affects the pronunciation of English by Dutch learners. To do this she makes tape recordings of 40 Dutch learners pronouncing English sentences. She then measures the difference in vowel length before voiced and voiceless obstruents (for example ‘tap’ vs ‘tab’). A questionnaire has determined that 20 of these students are highly motivated and that 20 others are not very motivated to pronounce English correctly. Tip: the dependent is the difference in vowel length between the two phonological contexts. 2. A researcher wants to find out whether the age at which one starts to learn a foreign language is related to language proficiency. To investigate this, she finds 20 Polish learners of French who had all been learning French for ten years. The starting age of these learners ranges from 1 to 20, in such a way that each starting age is included precisely once. All learners take a 50-item French proficiency test; the proficiency score is based on the number of correct items. 3. To investigate the effect of input on second language learning, 120 randomly selected Japanese learners of Hebrew are divided into two groups: one experimental group of 30 is isolated in a dark room and exposed to Hebrew television 24 hours a day (thereby achieving maximum exposure to Hebrew); one control group of 30 is not exposed to Hebrew. After two months, both groups are submitted to a 100-item Hebrew proficiency test; the proficiency score is based on the number of correct items. 4. The previous experiment concerning Japanese learners of Hebrew is done once more, but this time each of the groups is equally subdivided into three age groups: 11–30, 31–50 and 51–70. 5. A researcher is interested in the effects of social reinforcement on toddlers’ motor skills. In an experiment, 54 three-year-old children have to take marbles from a vase and put them into a box through a tiny hole. The number of marbles that are put into the box after four minutes is counted. The children were randomly attributed to two groups. In a ten-minute learning period preceding the experiment, the children in the first group were encouraged by smiles and words of praise. The children in the second group were not encouraged. 6. In which way would the previous experiment change if, in addition, the researcher wanted to find out if social reinforcement equally affects the boys and girls in the experiment? 7. To investigate the relation between active sports performance and stress a questionnaire is set up. The questionnaire determines whether the participants are active sportswomen and sportsmen (‘yes’ or ‘no’) and the degree of stress they experience in their daily lives (on a three-point scale).

Chapter 7

Conclusion

7.1

Statistical dangers and limitations

In the previous chapters, we have provided an introduction to essential statistics and we have given an overview of the most frequently used statistical analyses in applied linguistics. If you followed our suggestion, you have also experimented with carrying out both descriptive and inductive statistics in the practicals in Part III. Practical 6 has been added as a final practical that you can do after reading this last chapter in Part I. It will give you some more practice with the different statistical tests. Statistical studies can involve complex calculations that may have restrictions on their application or for which underlying assumptions have to be met. As both the calculations themselves and the assumptions are hidden from the user of computer programs like SPSS, it is unwise to apply them to a dataset and blindly interpret its output in terms of significance. An important purpose of our discussions has been to focus on the interpretation of statistical studies reported by other researchers. We hope that you are now able to assess the appropriateness of the statistics used. However, statistical studies cannot only go wrong at the level of the calculations themselves. In this chapter, we will briefly discuss two major requirements for statistical studies: validity and reliability. After that, we will point to some final practical issues, and we will conclude by considering the possible differences between the neatly measurable world of our studies and the real world. Perhaps the biggest danger in statistics is their blind application and the assumption that statistical significance is identical with the truth.

7.2

Validity and reliability

When we carry out a study, it is crucial that what we are measuring is the same as what we think we are measuring. This is generally referred to as the validity of a study. The most important type of validity, construct validity, is closely related to operationalization, and can be seen as the link between the statistical study and the real world. Some cases of operationalization are completely obvious, like that of the construct ‘height’ in terms of the number of centimetres measured from the top to the bottom of a person.

78

The Basics

The only point at which we may go wrong in measuring someone’s height concerns what we understand as the ‘top’ and the ‘bottom’. If a person wears high heels, we may not want to include the heels in our measurement. However obvious this may seem, this illustrates the fact that operationalization of constructs is based on choices made by the researcher and that it is important to be explicit about how constructs are operationalized and why they have been operationalized that way. The operationalization of abstract constructs like ‘language proficiency’ is less obvious and always involves choices on the part of the researcher. If the validity of a study is not warranted, the possible significant outcomes may be completely meaningless. The same holds for studies in which the samples are not representative. Here, too, the validity is at stake, because it is not clear that the outcome of the study is representative of a real-world situation. There is no simple way to test statistically the validity of a study. What is often done is to compare the outcome of the test to a well-established and standardized test. A new way to test language proficiency could for instance be compared to TOEFL scores or IELTS scores as a criterion. When the results from the new test deviate from the standardized test, we may have to conclude that what we have measured is not what we think we have measured. Reliability refers to the internal consistency of a study. Have all the necessary steps been taken and have all the assumptions been met? Do the variables included in the study test what we think they are testing? Contrary to validity, which is often a matter of argumentation, there are several ways to control or check the reliability of a study. Here we will use the split-half method as an example (as we did in section 5.7), but many more measures are available. The idea behind the split-half method is simple. Suppose we want to ensure that the test items in an experiment are consistently testing the variable as it has been operationalized. If this were the case, we would expect all the items in the test to refer to the same thing. To check this, we can split the test into two equal halves, for instance by taking the odd items of the test and the even items of the test and then running a correlation between these two. Such a correlation, referred to as a reliability coefficient must be high and significant. If the odd and the even items show a weak correlation of, say, 0.30, the reliability of the test is questionable. The relevant statistical test that is worth mentioning in this respect is Cronbach’s Alpha (see section 5.7). Simply put, what Cronbach’s Alpha represents is the average of all the possible split halves of a dataset. The outcome of this calculation can be interpreted as a correlation. Although the desired outcome depends on how critical we need to be, a frequently used value for an acceptable Cronbach’s Alpha is around .80. As we have seen in this section, the meaningfulness of a study is not only established by the statistical significance of the outcome. All steps have to be considered carefully, from operationalization to sampling (and the

Conclusion

79

extent to which a sample is representative). In addition, all assumptions will have to be met before statistics can be applied.

7.3 Meaningful outcomes After reading and doing everything up to now, you will have reached a sufficient understanding of statistics to read research reports and to approach critically what other researchers have done in terms of statistics. You will also have a rough idea of how to apply statistics to your own empirical work. However, at this stage we should sound a warning. We have discussed the basic principles of statistical studies, but we have not discussed the ins and outs of all statistical tests. It may therefore be wise to consult a statistician before you start analysing (or even conducting) your own empirical studies. If you want to take the risk of doing this on your own, there is one more issue you should know about. In the activities discussed, we have regularly asked when a result was statistically significant and whether you would consider the result meaningful. In most cases where we have illustrated research with very few cases, your answer about the meaningfulness of a study would correctly have been ‘no’. If we want to say something sensible about the difference between two groups, it will be obvious that we need more than about ten participants per sample group to represent truly the intended population. The number of participants needed to make an experiment meaningful depends on the beta error, the effect size and the power of an experiment, as we explained in section 4.7. There we saw that to demonstrate the existence of a small effect, the group size should be at least 780 participants. In applied linguistics this is rather exceptional. We can only hope that the effects we are looking for are large ones, so that we can make do with about 30 participants per group to make it meaningful. When you set up your own studies, it is important to realize that there is little use in taking samples of less that 20 participants per group when the aim is to demonstrate a difference between those groups. But even when the outcomes are significant, and are based on large samples that are normally distributed, the result is still not necessarily meaningful. In Activity 6.6 one of the imaginary studies discussed a group of learners that were ‘isolated in a dark room and exposed to Hebrew television 24 hours a day’. This was intentionally phrased as a rather obscure case of an unrealistic research, the outcomes of which, no matter how significant, may not be meaningful due to the setup of the study or the operationalization of the variables involved. Most real-life research flaws are less obvious, so it is essential that we are careful about these issues. In addition to knowledge about statistics, we need our common sense to assess the meaningfulness of a result.

80

7.4

The Basics

Statistics and the real world

When the decision is made to conduct a certain study in such a way that the data can be analysed according to a certain statistical tradition, then there should be an awareness of the written and unwritten rules that this tradition brings. Important to remember is that if there is a significant effect, then this does not directly ‘prove’ your hypothesis, it only ‘supports’ it. If there is no significant effect, then it should be clear that it is not acceptable to say that there is a difference despite the fact that there was no significant difference. Significance is not a gradient phenomenon, and we cannot speak of results that are ‘very significant’ or ‘nearly significant’. Significance is tested by comparing the results to hard criteria (the alpha level) and is purely binary. At the same time, when no significance is found it is rather preliminary to claim that this ‘proves’ that there is no difference at all. The absence of a significant effect could be due to many factors, for example power and/or sample size, or, even worse, a flaw in the study. The only time when we could even possibly begin to think about claiming to ‘prove’ something is after replicating the same outcome (once or maybe even more than once) and finding a significant effect each time. Nevertheless, even when all conditions of a statistical study have been met, the validity is ensured and the study is sufficiently reliable, the application of statistics is no more than one way of evaluating research data. Although it may be valuable, a statistical study is not the final answer to a research problem. There are many ways to analyse data, which makes the choice of which statistics to use a subjective one. A researcher needs to give good arguments for why he or she decides to use a specific statistical measure. Moreover, all the statistics discussed in this book are limited to the description of a synchronic situation. For the investigation of development, other techniques and statistics are required that all have their own limitations and drawbacks. One severe limitation of statistical studies in general is that they are strongly geared towards a generalization of human behaviour. This becomes obvious when we realize that the very basis of statistical argumentation is created by the normal distribution. However, when the emphasis on generalization becomes too strong, it may obscure the variation of individuals in a sample. In DST-based research it is precisely this variation that may reveal the true nature of language development. There is no way we can do without statistical methods if we want to make generalizations about human behaviour, but (parametric) statistics do not possess the magical power to provide a solution to all research questions.

Notes

1

Types of research

1.

This first chapter was written in collaboration with Kees de Bot.

2

Systematicity in statistics: variables

1.

In some statistical software, this is sometimes simply referred to as ‘scale’.

3 Descriptive statistics 1.

An alternative solution is to take the absolute values of the numbers, though the result is a different measure. We will only discuss the most commonly used way of calculating ‘average dispersion’.

5

Doing statistics: from theory to practice

1. 2.

For a step-by-step guide to doing the t-test, see Part II, section 4. This table is not exactly what the real SPSS output will look like, but is slightly simplified. For the full table, including a test for homogeneity of variance, see Table II.4.2.

6

Common statistics for applied linguistics

1.

Note that, like the t-test, an important assumption for ANOVAs is that all groups follow the normal distribution.

This page intentionally left blank

References

W. Admiraal, G. Westhoff and K. de Bot (2006) ‘Evaluation of Bilingual Secondary Education in the Netherlands: Students’ Language Proficiency in English’, Educational Research and Evaluation, 12(1), 75–93. A. Becker and M. Carroll (1997) The Acquisition of Spatial Relations in a Second Language (Amsterdam/Philadelphia: John Benjamins). K. de Bot and M. Gullberg (eds) (2010) Gestures in Language Development (Amsterdam/Philadelphia: John Benjamins). C.M. Brown and P. Hagoort (eds) (1999) The Neurocognition of Language (Oxford: Oxford University Press). J.D. Brown and T.S. Rodgers (2002) Doing Second Language Research (Oxford: Oxford University Press). J. Cohen (1992) ‘A Power Primer’, Psychological Bulletin, 112(1), 155–9. A. Field (2005) Discovering Statistics Using SPSS (London: SAGE Publications). S.M. Gass and M.J.A. Torres (2005) ‘Attention When?: An Investigation of the Ordering Effect of Input and Interaction’, Studies in Second Language Acquisition, 27(1), 1–31. M. Grendel (1993) ‘Verlies en Herstel van Lexicale Kennis’ [Attrition and Recovery of Lexical Knowledge], PhD dissertation, University of Nijmegen. L. Hansen, E.S. Kim and Y. Taura (2010) ‘L2 Vocabulary Loss and Relearning: The Difference a Decade Makes’, Paper Presented at the AAAL Annual Conference, Atlanta, 6–9 March. B. Hendriks (2002) More on Dutch-English ... Please? A Study of Request Performance by Dutch Native Speakers, English Native Speakers and Dutch Learners of English. (Nijmegen: Nijmegen University Press). I. Huibregtse (2001) ‘Effecten en Didactiek van Tweetalig Voortgezet Onderwijs in Nederland’ [Effects and Pedagogy of Secondary Bilingual Education in the Netherlands], PhD dissertation, University of Utrecht. W. Klein (1989) ‘Introspection into What? Review of C. Faerch and G. Kasper (eds), Introspection in Second Language Research 1987’, Contemporary Psychology: A Journal of Reviews, 34, 1119–20. W. Klein and C. Perdue (1992) Utterance Structure: Developing Grammars Again (Amsterdam/Philadelphia: John Benjamins). A. Mackey and S.M. Gass (2005) Second Language Research: Methodology and Design (Mahwah, NJ: Lawrence Erlbaum). K. McDonough (2005) ‘Identifying the Impact of Negative Feedback and Learners’ Responses on ESL Question Development’, Studies in Second Language Acquisition, 27(1), 79–103.

84

References

M. Verspoor and W.M. Lowie (2003) ‘Making Sense of Polysemous Words’, Language Learning, 56(3), 429–62. M. Verspoor, K. de Bot and W. Lowie (eds) (2011) A Dynamic Systems Approach to Second Language Development: Methods and Techniques (Amsterdam/ Philadelphia: John Benjamins). S. Webb (2005) ‘Receptive and Productive Vocabulary Learning: The Effects of Reading and Writing on Word Knowledge’, Studies in Second Language Acquisition, 27(1), 33–52. S. Webb and E. Kagimoto (2011) ‘Learning Collocations: Do the Number of Collocates, Position of the Node Word, and Synonymy Affect Learning?’, Applied Linguistics, 32(3), 259–76. B. Weltens (1989) The Attrition of French as a Foreign Language (Dordrecht/ Providence: Foris Publications).

PART II

How to SPSS

In this part, you will find step-by-step instructions for using a number of common statistics. Some of these how-to’s are presented as demonstration videos available from the accompanying website. The screenshots have been made using SPSS/PASW 18. Since SPSS brings out a new version each year, the pictures may be slightly different from the version you use.

This page intentionally left blank

How to 1

How to Do Descriptive Statistics

Before you carry out any inductive statistics, it is important that you visually inspect your data by making some plots. You can then calculate some descriptive statistics. SPSS has a few options to give you these descriptive statistics.

1.1

Analyze > Descriptives > Frequencies

When you choose Frequencies you will see a screen like the one shown in Figure II.1.1.

Figure II.1.1

Under Statistics and Charts, you will be able to choose different descriptive statistics and different options for charts. You can for example select the Mean, Median, Range and Standard Deviation. Once you have selected your descriptive statistics, click OK, and you will get an output, showing you the statistics that you asked for. Apart from this it will also give you a table with the frequencies of each score, as in Table II.1.1.

88

How to SPSS

Valid

Number of correct items

Frequency

Percent

Valid Percent

Cumulative Percent

2

1

2.5

2.5

2.5

3

1

2.5

2.5

5.0

4

3

7.5

7.5

12.5

5

7

17.5

17.5

30.0

6

11

27.5

27.5

57.5

7

9

22.5

22.5

80.0

8

5

12.5

12.5

92.5

9

2

5.0

5.0

97.5

10

1

2.5

2.5

100.0

40

100.0

100.0

Total

1.2

Table II.1.1

Analyze > Descriptive Statistics > Descriptives

Like the previous method, clicking on Descriptives will also give you different options to select from (see Figure II.1.2).

Figure II.1.2

If you select ‘Save standardized values as variables’ you will get z-scores for each individual case in your data sheet (see Figure II.1.3).

How to Do Descriptive Statistics

89

Figure II.1.3

This z-score can be convenient if you want to know how many SDs each participant is away from the mean.

1.3

Analyze > Descriptive Statistics > Explore

This option is especially useful when you have different groups and you want to find out mean scores and distributions of the specific groups. You can enter the independent variable ‘Group’ under Factor List. You will also get boxplots with this menu (see Figure II.1.4).

Figure II.1.4

90

How to SPSS

You can choose from Statistics and Plots or Both. Under Plots you can select Histograms to check the distribution. If you double click on the Histogram in the output, you can draw a distribution curve. The output will also give you the skewness and kurtosis values among the other descriptive statistics. With these and the histograms you can make an estimation of the normality of your data.

How to 2

How to Check Assumptions

Before you carry out inductive statistics, you need to look at the descriptive statistics and check the assumptions. Parametric tests are designed for: Interval data; Data that is normally distributed (especially for small sample sizes); Groups that show equality of variance.

2.1

Checking for interval data

Having interval data for the dependent variable is important for the design of your study and should be thought of before you proceed. If you give students essay questions and the ratings are therefore subjective grades, your data is not interval – so you should move to non-parametric statistics. Make sure you keep this in mind while designing your study.

2.2

Normal distribution and equality of variance

You may for example have two groups of language learners, a beginner group and an advanced group. They have proficiency scores on a scale from 1 to 100 on a multiple choice test. If you want to check the normal distribution of your data, go to Analyze > Descriptive Statistics > Explore. In the Dependent List you put your dependent variable, in this case the proficiency score. In Factor List, you put Group (beginners and advanced), as in Figure II.2.1.

Figure II.2.1

92

How to SPSS

When you click on Plots, you get another option screen (see Figure II.2.2). In this screen, you should tick ‘Normality plots with tests’ to test for normality. And you should tick ‘Untransformed’ to do a test for equality of variance. You can tick ‘Histograms’ for a visual presentation of your distribution.

Figure II.2.2

2.2.1 Normal distribution In your output file, you can start by looking at the values for skewness and kurtosis and how much they deviate from zero. Remember, skewness says something about the symmetry of the distribution being more focused to the left or to the right of the curve. Kurtosis says something about the pointedness of the curve. Next, you should look for the table as shown in Table II.2.1. You will get a significance value for the Kolmogorov-Smirnov test, and one for the Shapiro-Wilk. When these are > .05, you can assume your data is normally distributed. In general, it is best to use Shapiro-Wilk for sample sizes that are smaller than 50, and the Kolmogorov-Smirnov for larger sample sizes. However, when sample sizes are large, it is very easy to get a significant value for the Kolmogorov-Smirnov test. Especially with large sample sizes you should always check the histogram as well to see whether the distribution really deviates from normality. Table II.2.1

Tests of normality

Kolmogorov-Smirnova Group

Statistic

df

Shapiro-Wilk

Sig.

Statistic .951

15

.547

15

.906

.136

15

.200

Advanced

.120

15

.200*

*. This is a lower bound of the true significance.

Sig.

.973

Proficiency Score Beginners

a. Lilliefors Significance Correction

df

*

How to Check Assumptions

2.2.2

93

Equality of variance

A rule of thumb with equality of variance is that the largest SD of your groups should not be more than twice the smallest SD of your groups. So if you have two groups, and one group has an SD of 16, then the other group should not have an SD that is smaller than 8. Apart from this, you can perform a test to check the equality of variance. This test is the Levene’s Test. When you carry out this test, the null-hypothesis is that the groups do not differ in variance. In the output window, you look for the table that is similar to the one in Table II.2.2. Table II.2.2

Test of Homogeneity of Variance Levene Statistic

Proficiency Score

df1

df2

Sig.

Based on Mean

2.045

1

28

.164

Based on Median

1.814

1

28

.189

Based on Median and with adjusted df

1.814

1

22.618

.191

Based on trimmed mean

2.059

1

28

.162

When the significance value is above .05, you can accept your null-hypothesis, that is, it is safe to assume that the two (or more) groups are equal in variance.

This page intentionally left blank

How to 3

How to Do a t-test

3.1 Important assumptions and prerequisites for the t-test You can only do a t-test if you have an interval dependent and a nominal independent with two levels (‘groups’). The distribution in each group should be approximately normal. The group size must be the same in the two groups. If not, the variances in the two groups must be equal (test with Levene).

3.2 Types of t-tests In the Compare Means menu of the SPSS program you can find different types of t-tests (see Figure II.3.1).

Figure II.3.1

96

How to SPSS

3.2.1 The one-sample t-test This compares the mean score of a sample to a known value. Usually, the known value is a population mean. For example: You read in the newspaper that the average IQ in the country is 101. You want to find out whether university students have a higher average. You ask 25 university students to do an IQ test. Your data show that the average IQ of your university students’ sample is 119. Now you want to assess whether this finding is due to chance or whether university students are really more intelligent than the average person. You want to know whether they represent society or whether they are from a separate population.

3.2.2 The independent-samples t-test This is the test that you will use most often. Most of time, the data points from two different groups (like ‘male’ and ‘female’) are not in any way related.

3.2.3 The paired-samples t-test This can be used when the data points are related. For instance, when you take two measurements of the same people and then want to compare the means of the two measurements, you should select the ‘Paired-Samples t-test’. Note that with a slightly different organization of the data, you can always also do a correlation analysis if you can do a paired-samples t-test. There is a difference in interpretation though: a t-test tests the difference, whereas a correlation tests the similarity. So a significant t-test will yield a non-significant correlation and vice versa.

3.3 The independent-samples t-test: getting results First. In the SPSS menu, select the Independent Samples T Test. In the dialogue box that pops up (see Figure II.3.2), move the variables you want to test to the right boxes.

How to Do a t-test

97

Figure II.3.2

The Test Variable is the dependent variable (scores). The Grouping Variable is the independent variable. Second. You will have to tell SPSS which two groups you want to compare in this t-test. So click on Define Groups and fill in how your groups are identified in your data (see Figure II.3.3).

Figure II.3.3

This may be numeric (1, 2) or alphanumeric (‘a’, ‘b’; ‘boys’, ‘girls’; etc.). Make sure this matches exactly with the labels in the datasheet (which is case sensitive). Third. Now click on OK to run the analysis.

3.4 The independent-samples t-test: interpreting the output The SPSS output will give you two tables: the descriptives and the results of the t-test. The descriptives (Group Statistics, Table II.3.1) will show the independent variable and its levels, the number of participants (N) and the SD.

98

How to SPSS

In the descriptives, you will also see the ‘standard error of the mean’ (SEM or SE). This measure gives information about the error in generalizing the results from the sample to the population. Obviously, this error is smaller when the sample is larger, which is expressed by the formula: SEM =

SD N

(II.3.1) Table II.3.1

Group Statistics

each line gives the result of one level of the independent group score

N

Mean

Std. Deviation

Std. Error Mean

boys

20

32.10

4.553

1.018

girls

20

37.90

3.177

.710

N = number of cases in each group The standard error of the mean (SEM) expresses to what extent the sample is representative of the population. It’s the standard deviation of the sample divided by the square root of the sample size.

The next table (Table II.3.2) is the result of the independent samples t-test. Before anything else, you should consider Levene’s test for equality of variance. If this test is not significant, there is no problem with the equality of variance and you may read the results from the first row in the table. If Levene’s test is significant (< 0.05), then you should look at the bottom row of the output. The degrees of freedom for the t-test is based on the sample size for each group. For each group, take the sample size and subtract 1. So in this example Df = (20 – 1) + (20 – 1) = 38. This is indeed the Df reported by SPSS. The output gives you the value of t (which in itself is not very informative), the degrees of freedom and (most importantly) the level of significance (p). You may only test one-tailed if one of the alternative hypotheses can absolutely be ruled out. If you test one-tailed (which is rather exceptional), you may divide the significance by 2. When the significance value is for example 0.025 two-tailed, it will be 0.0125 one-tailed.

How to Do a t-test Table II.3.2

The 'variance' of a group is the sum of the distances of the individual items away from the mean. So sort of the SD, but not divided by N. (This is also referred to as the 'error'.) An assumption for the t-test is that the variances in the groups (say SDs) are the same. If not,we have to be stricter in rejecting H0. So before we can start considering the output of the t-test,we’ll have to check if the variances in the groups are the same. Mr. Levene has been so kind as to do this for us.

Independent samples test

The significance of Levene’s test is important. If the significance level of this test is smaller than .05, equal variances can NOT be assumed, and you will have to use the data in the row called ‘Equal variances NOT assumed’ So we’d rather have Levene’s test NOT significant! Then we can look at the top row

score Equal variances assumed Equal variances not assumed

The outcome of Levene’s test is expressed in F - this is not very relevant for our purpose.

t-test for Equality of Means

Sig. 2.969

The rest of the columns give details about the difference between the two groups. Not particularly relevant for our purpose

Independent Samples Test

Levene's Test for Equality of Variances

F

99

t .093

Sig. (2-tailed)

df

Mean Difference

Std. Error Difference

95% Confidence Interval of the Difference Lower Upper

-4.672

38

.000

-5.800

1.241

-8.313

-3.287

-4.672

33.959

.000

-5.800

1.241

-8.323

-3.277

DF (degrees of freedom) is related to the sample size

This is the most important outcome of the t-test - it expresses the chance of incorrectly rejecting H0

3.5 The independent-samples t-test: reporting on results The conventional way of reporting on the results of the t-test can best be seen in an example. The results in the following sentence relate to a study that tests the difference in intelligence between boys and girls. You can use this as a template (as suggested by Field, 2005, p. 303): ‘on average, the girls showed a higher level of intelligence (M = 37.9, SE = 0.71) than the boys (M = 32.1, SE = 1.02). This difference was significant (t(38) = –4.67; p < .001)’. Instead of the SE, the SD may also be reported. The figure in parentheses after ‘t’ refers to the degrees of freedom (Df). Instead of reporting p < .001, the exact value of p may be provided, but in this case we don’t know the exact value, because we only have three decimal places. If the p-value is for example .043, you may report p = .043. Alternatively, you can provide the descriptives using a table or a figure and provide the note: ‘an independent-samples t-test showed that this difference between the groups was significant (t = –4.67; Df = 38; p < .001)’. Do not include entire SPSS outputs in your results section apart from descriptives (tables) or (edited) figures. If the SPSS descriptives table contains a lot of redundant information, it is better to copy the relevant information to your own table. Conventionally, charts are included for significant results only. You may want to include any part of the output in an appendix if so desired.

100

How to SPSS

3.6 Non-parametric alternative for an independent-samples t-test If the groups do not meet the equality of variance, you can solve this by looking at the second line in the t-test table. However, when the data is not normally distributed (see section 2.2.1 on page 92), you can do a non-parametric test to compare your groups. The most common non-parametric alternative for an independent-samples t-test is the Mann-Whitney U test. You can find this test by going to Analyze > Nonparametric Tests > Legacy Dialogs > 2 Independent Samples (see Figure II.3.4).

Figure II.3.4

Here you can select the Mann-Whitney U Test. If you have small sample sizes (less than 25) it might be better to use the Kolmogorov-Smirnov Z test (not to be mistaken for the normality test). Instead of a t, you report the U or the Z for these tests, respectively.

How to 4

How to Do a One-way ANOVA

4.1 Important assumptions and prerequisites for the one-way ANOVA You can only do a one-way ANOVA if you have an interval dependent and a nominal independent with two or more levels (‘groups’). The distribution in each group should be approximately normal. The group size must be the same in the two groups. If not, the variances in all the groups must be equal.

4.2

The one-way ANOVA: getting results

First. In the SPSS menu, select Analyze > Compare Means > One-Way ANOVA. In the dialogue box that pops up (see Figure II.4.1), move the variables you want to test to the right-hand boxes.

Figure II.4.1

The Test Variable is the dependent variable (scores). The Factor is the independent variable (Groups). In this example, we used age groups (three levels). Second. In the Options submenu (see Figure II.4.2), you can select the statistics you need. To allow for the interpretation of the test, you will always need the Descriptives. In addition, you may want to test for the equality of variance (Levene).

102

How to SPSS

Figure II.4.2

4.3 The one-way ANOVA: interpreting the output 1 (main effect) Since we have asked for Descriptives, the first table (see Table II.4.1) in the Output lists the most relevant of the descriptives. It gives us the mean, score for each group, the SD, and the standard error (see p. 98). As you can see in the output, the value for each level of the independent has been labelled. The three age groups represent 11–30, 31–50 and 51–70. At face value, the score (representing proficiency) decreases with increasing age. Table II.4.1 Descriptives score 95% Confidence Interval for Mean Lower Bound

Upper Bound

Minimum

Maximum

11-30

20

75.60

12.258

2.741

69.86

81.34

56

95

31-50

20

66.35

10.153

2.270

61.60

71.10

50

88

51-70

20

57.35

15.055

3.366

50.30

64.40

28

85

Total

60

66.43

14.527

1.875

62.68

70.19

28

95

N

Mean

Std. Deviation

Std. Error

The next test we selected in the Options was Levene’s test for equality (‘homogeneity’) of variance. In this case, the outcome of the test is not significant (see Table II.4.2).

How to Do a One-way ANOVA Table II.4.2

103

Test of Homogeneity of Variance

score Levene Statistic

df1

df2

1.274

2

Sig. 57

.287

Had this test been significant, we should have calculated a corrected value for the one-way ANOVA. In that case, we should have selected an additional analysis in the Options menu, Welch’s F, and report on that instead of the one-way ANOVA. Since Levene’s test is not significant, we can proceed to the actual test. An ANOVA analysis (see Table II.4.3) compares the difference between the groups to the variance within the groups (see p. 61 in section 6.2). Therefore, these two values are always provided (as their ‘sum of squares’). The value of F is the Mean Square between the groups divided by the Mean Square within the groups. In our example, the result of that sum is 10.41. The most important bit of the output is the p-value, which can be found under Sig. (in this case 0.000, and so smaller than 0.001). It seems to be justified to reject the null-hypothesis that there is no difference between the groups. Table II.4.3

ANOVA

score Sum of Squares

df

Mean Square

Between Groups

3330.833

2

1665.417

Within Groups

9119.900

57

159.998

12450.733

59

Total

F 10.409

Sig. .000

This, however, does not answer all our questions. Since there are three groups involved, it is not sufficient to conclude that ‘the difference between the groups is significant’. After all, it may be case that the difference between groups 1 and 2 and the difference between groups 1 and 3 is significant, but that the difference between groups 2 and 3 are not. To further analyse the difference between the groups, we can run a post hoc analysis.

104

How to SPSS

4.4 The one-way ANOVA: interpreting the output 2 (differences between groups) First. We’ll do another analysis by returning to the main menu and selecting Analyze > Compare Means > One-Way ANOVA. In the screen that pops up (see Figure II.4.3), we’ll select the Post Hoc option.

Figure II.4.3

Second. In the Post Hoc menu (see Figure II.4.4) we can select many different statistics that carry out a post hoc analysis for us. Only real statisticians understand what the differences are, so we’ll go by the advice given by Andy Field (2005, p. 357): When you have equal sample sizes and homogeneity of variance is met, use REGWQ or Tukey HSD. If sample sizes are slightly different then use Gabriel’s procedure, but if samples sizes are very different use Hochberg’s GT2. If there is any doubt about the homogeneity of variance use the Games-Howell procedure.

Figure II.4.4

How to Do a One-way ANOVA

105

Since in our example the sample sizes are the same and the variances in the groups are homogeneous, we have selected ‘Tukey’. Third. Click on Continue and then on OK. Fourth. With Table II.4.4 we can compare each of the groups individually. It appears that the difference between age group 11–30 and 51–70 is significant, but not in any of the other groups. Table II.4.4

Multiple comparisons (score: Tukey HSD)

95% Confidence Interval Mean Difference (IJ)

(I) age group

(J) age group

Lower Bound

Upper Bound

age 11-30

age 31-50

9.250

4.000

.062

-.38

18.88

age 51-70

18.250*

4.000

.000

8.62

27.88

age 11-30

-9.250

4.000

.062

-18.88

.38

age 31-50

age 51-70

Std. Error

Sig.

age 51-70

9.000

4.000

.071

-.63

18.63

age 11-30

-18.250*

4.000

.000

-27.88

-8.62

age 31-50

-9.000

4.000

.071

-18.63

.63

*. The mean difference is significant at the 0.05 level.

4.5

The one-way ANOVA: plotting results

The Options menu also allows us to plot a graph of the results. In our example this yields Figure II.4.5 in the Output. 80

Mean of score

75 70 65 60 55 age 11-30

age 31-50 Age group

Figure II.4.5

age 51-70

106

4.6

How to SPSS

The one-way ANOVA: reporting on results

Use the following sentence as an example for your own reports: There was a significant effect of the levels of age group on proficiency scores, F (2, 57) = 10.41, p < 0.001. A post hoc analysis revealed that only the difference between the lowest (M = 75.6, SD = 12.3) and the highest age group (M = 57.4, SD = 15.1) was significant at p < 0.001. The middle age group (M = 66.4, SD = 10.2) did not differ significantly from any of the other groups. The figures following the F refer to the degrees of freedom Between Groups and Within Groups respectively, as displayed in the output. Do not include entire SPSS outputs in your results section apart from descriptives (tables) or (edited) figures. If the SPSS descriptives table contains a lot of redundant information, it’s better to copy the relevant information to your own table. Conventionally, charts are included for significant results only. You may want to include any part of the output in an appendix if so desired.

4.7 Non-parametric alternatives When the assumptions of normality or equality of variance are violated, we need to do a non-parametric test as an alternative to the one-way ANOVA. A good one is the Kruskal-Wallis H test. To run this in SPSS go to Analyze > Nonparametric Tests > Legacy Dialogs > K Independent Samples.

How to 5

How to Do a Two-way ANOVA

5.1 Important assumptions and prerequisites for ANOVA You can only do an ANOVA if you have an interval dependent and one or more nominal independent variables. An ANOVA can be run with any number of levels of the independent variable(s). In other words, if you can do a t-test, you could also do an ANOVA with the same data, but not the other way around. So the ANOVA can be seen as a (much) more flexible t-test. The distribution in each of the groups for all independent variables should be approximately normal. The variances in the groups (within the same independent variable) must be equal. Note: ANOVAs are very powerful analyses with many different possibilities and options. Here we have limited the discussion to the bare essentials of the analysis. If you need ANOVAs for serious studies, you are advised to consult a statistics book such as Field (2005), which spends more than 300 pages on different types of ANOVAs.

5.2

ANOVA: getting results

First. In the SPSS menu, go to Analyze > General Linear Model (GLM) > Univariate. Second. In the menu that pops up (see Figure II.5.1), you can fill in the variable you want to include in your analysis. Move your Score to the Dependent Variable box and move the dependent variables to the Fixed Factor box.

108

How to SPSS

Figure II.5.1

Third. If you click OK now, this would give you information about the significance of each of the independent variables and their interaction. However, usually you’d want to make use of one or more of the optional analyses. Below you can find them in order of relevance: The most important extra you probably want is some descriptive statistics. Select Options and tick the boxes you need (at least use Descriptive Statistics) as in Figure II.5.2. You can also tick Homogeneity tests, and you can choose to display the means for all factors and interactions.

Figure II.5.2

How to Do a Two-way ANOVA

109

For independent variables with more than two levels, you’ll need to select a post hoc analysis to assess the difference between each of the levels (see section II.4.4, p. 104) (see Figure II.5.3).

Figure II.5.3

For the interpretation of possible interactions, you may want to select Plots (see Figure II.5.4). Set up the way you want your plot to be displayed and then click Add.

Figure II.5.4

Fourth. Now you can run your analysis.

110

5.3

How to SPSS

ANOVA: interpreting the output

The SPSS output will give you the output you’ve requested. The most important of these are as follows. First, there are the descriptives (see Table II.5.1). Before looking at the significance, carefully study the descriptives. Do the differences between the groups go in the direction that you expected? Is the difference between the groups big or small? How big are the standard deviations? Table II.5.1

Descriptive statistics

Dependent Variable:score Mean

Std. Deviation

N

exposure

age group

exposure to Japanese

age 11-30

70.20

11.331

10

age 31-50

58.90

6.724

10

age 51-70

49.00

13.258

10

Total

59.37

13.639

30

age 11-30

81.00

11.136

10

age 31-50

73.80

7.005

10

age 51-70

65.70

12.157

10

Total

73.50

11.831

30

age 11-30

75.60

12.258

20

age 31-50

66.35

10.153

20

age 51-70

57.35

15.055

20

Total

66.43

14.527

60

no exposure

Total

The significance tables (see Table II.5.2). Here you will find which of the main effects turned out to be significant. What you can also find is whether any of the interactions are significant. In the example used here, this would imply that the difference between the exposure groups differs with age group. So, for instance, only at the low age group might an effect for exposure be found. In the example, the interaction (exposure * age) turns out not be significant.

How to Do a Two-way ANOVA Table II.5.2

111

Tests of between-subject effects

Dependent Variable:score Type III Sum of Squares

Source

df

Mean Square

F

Sig.

6418.533a

5

1283.707

11.492

.000

Intercept

264803.267

1

264803.267

2370.508

.000

exposure

2996.267

1

2996.267

26.822

.000

age

3330.833

2

1665.417

14.909

.000

91.433

2

45.717

.409

.666

Error

6032.200

54

111.707

Total

277254.000

60

12450.733

59

Corrected Model

exposure * age

Corrected Total

a. R Squared = .516 (Adjusted R Squared = .471)

Interactions are always difficult to interpret, especially when there are more than two independent variables. In those cases there may be three-way or four-way interactions. The best way to try and interpret an interaction is by plotting a picture like the one in Figure II.5.5. Figure II.5.5 exposure exposure to Japanese no exposure

Estimated Marginal Means

80

70

60

50

age 11–30

age 31–50 Age group

age 51–70

Clearly, there is no interaction in this case, as the lines for exposure and no exposure run almost parallel. A strong interaction would have occurred

112

How to SPSS

if the lines had crossed. For the interpretation of post hoc analyses, see section II.4.4.

5.4

The two-way ANOVA: reporting on results

The conventional way of reporting on the results of the ANOVA can best be seen in an example. The example worked out here is related to the data mentioned above (so check this with the output tables). Always include the F-ratio, the degrees of freedom of the ‘model’ and the residual degrees of freedom (‘error’): There was a significant effect of the levels of exposure on score, F (1, 54) = 26.82, p < 0.001. On average, the participants who had not been exposed to Japanese scored higher (M = 73.5; SD = 11.8) than the participants who had been exposed to Japanese (M = 59.4; SD = 13.6). [The part which reports on the descriptives can be left out if they have been reported in a table.] There was a significant effect of the levels of age group on score, F (2, 54) = 14.91, p < 0.001. There was a clear trend for the younger participants to score higher than the older participants. However, the oldest age group (M = 57.4; SD = 15.1) did not differ significantly from the middle age group (M = 66.4; SD = 10.2). The youngest group (M = 75.6; SD = 12.3) scored significantly higher than the other two age groups. [The part which reports on the descriptives can be left out if they have been reported in a table.] There was no significant interaction between levels of age group and levels of exposure, F < 1. You are advised not to include entire SPSS outputs in your results section apart from descriptives (tables) or (edited) figures. If the SPSS descriptives table contains a lot of redundant information, it’s better to copy the relevant information to your own table. Conventionally, charts are included for significant results only. You may want to include any part of the output in an appendix if so desired.

How to 6

How to Do a Correlation Analysis

An important assumption and prerequisite for the t-test is that you can only do a correlation analysis if you have interval or ordinal variables only.

6.1

Correlations: getting results

First. In the SPSS menu, go to Analyze > Correlate > Bivariate. Second. In the menu (see Figure II.6.1), move the variables you want to test to the right, select which statistical test you want to carry out (use Pearson for interval data or Spearman for ordinal data) and select whether you want to test one-tailed or two-tailed and then click on OK.

Figure II.6.1

6.2

Correlations: interpreting the output

The output of a correlation analysis is simple. In the table (see Table II.6.1), look at where the two variables you want to test cross (in the example this is ‘starting age’ and ‘French’ scores).

114

How to SPSS Table II.6.1

Correlations startingage

startingage

Pearson Correlation

French

1

.008

Sig. (2-tailed) N French

Pearson Correlation Sig. (2-tailed) N

-.572**

20

20

-.572**

1

.008 20

20

**. Correlation is significant at the 0.01 level (2-tailed).

The first number (–.0572) represents the strength of the correlation. The strongest possible negative correlation is –1 and the strongest possible positive correlation is +1. When this figure is 0, there is no correlation. The second figure (0.008) is the level of significance. In this example, the chance of incorrectly rejecting the H0 is 0.008. Note that the H0 in a correlation study is that there is no relationship between the two variables.

6.3 Plotting correlations First. In the Graphs menu, select Scatter/Dot (see Figure II.6.2).

Figure II.6.2

How to Do a Correlation Analysis

115

Second. Choose Simple scatter plot (see Figure II.6.3), move the variables to the axis you prefer and select OK.

Figure II.6.3

Third. The result for this example looks like the one shown in Figure II.6.4. 50

45

French

40

35

30

25

20 0

5

10 startingage

Figure II.6.4

15

20

116

How to SPSS

Fourth. By double clicking on the graph, you will enter the Chart Editor (see Figure II.6.5). Right click to add a ‘fit-line’ or trendline. Then close the Chart Editor.

Figure II.6.5

Fifth. If you want to copy the graph to Word, right click the graph and select Copy. In Word select Edit Æ Paste Special and then Bitmap. If you need high-quality graphics, the best option is to save your file as a pdf.

6.4

Correlations: reporting results

You can use the following sentence as a template for the report: There was a significant negative relationship between the starting age and French proficiency scores, r = –0.57; p < 0.01 (two-tailed). The younger the starting age, the higher the scores. You are advised not to include entire SPSS outputs in your results section apart from descriptives (tables) or (edited) figures. If the SPSS descriptives tables contain a lot of redundant information, it’s better to copy the relevant information to your own table. Conventionally, charts are included for significant results only. You may want to include any part of the output in an appendix if so desired.

How to 7

How to Do a Simple Regression Analysis

In addition to carrying out a correlation analysis on two interval variables to see whether they are related, we can ask ourselves whether we can predict the outcome of one variable by looking at the values of the other. Suppose you want to see how much a score on a language test can be predicted by the number of hours a student studied for the test. In this case the number of hours is the predictor variable and the score is the outcome variable (see section I.6.3). You can first make a scatter plot of your data as is explained in section II.6.3. What a regression analysis does is to come up with an equation for a fit-line in which the variance from the line is as small as possible. The starting point of this line is called the ‘intercept’. In this case the fit-line goes up, which means that the score on the test goes up when a student studies longer (in minutes). The fit-line is made out of an equation that takes the intercept (hence the starting point) and the amount the score goes up per minute, which is called the slope (see Figure II.7.1). R2 Linear = 0.173

40

Score

30

20

10

0 0

50 100 150 Number of Minutes Studied

Figure II.7.1

200

118

How to SPSS

To run a regression analysis, go to Analyze > Regression > Linear (see Figure II.7.2).

Figure II.7.2

Put the predictor variable(s) in the independent(s) box and the dependent variable in the dependent box (see Figure II.7.3). Click OK.

Figure II.7.3

How to Do a Simple Regression Analysis

119

In the output you will see three tables. The first table (Table II.7.1) gives the value of R, which is the same as the correlation coefficient. R gives the strength of the correlation between the predictor variable and the outcome variable. If we square this value of R, we get the amount of variance that is explained by our variable. In this case the R2 is .173, which means that the Amount of time studied in minutes explains 17.3 per cent of the variance in the data. Table II.7.1

Model summary

Model R .416a

1

Adjusted R Square

R Square .173

Std. Error of the Estimate

.134

7.396

a. Predictors: (Constant), Amount of Minutes Studied

In the ANOVA table (Table II.7.2) we can see that the p-value is .048 and so is smaller than .05. From this we can conclude that the amount of minutes a student has studied for the test is a good predictor for the score on the test. Table II.7.2 Model 1

Sum of Squares Regression

ANOVAb

df

Mean Square

240.737

1

240.737

Residual

1148.568

21

54.694

Total

1389.304

22

F

Sig.

4.402

.048a

a. Predictors: (Constant), Amount of Minutes Studied b. Dependent Variable: Score

In the last table (Table II.7.3) we will find the values for the intercept and the slope, with which we can write out the formula for our model. In this case the intercept is 17.11, so when a student does not study for a test at all, the outcome is likely to be 17.11. The value for the slope in this case is 0.062. This means that for every minute a student spends on studying for the test, the score will increase by 0.062 points. If a student studies for 100 minutes, the score will be 6.2 points higher than 17.11. The equation for this model then becomes: Outcome Yi = b0 + b1Xi + εi; Outcome Yi = 17.11 + 0.062Xi + εi.

120

How to SPSS Table II.7.3

Coefficientsa

Model Unstandardized Coefficients B 1

(Constant) Amount of Minutes Studied

Std. Error

17.108

2.727

.062

.029

Standardized Coefficients Beta .416

t

Sig.

6.274

.000

2.098

.048

a. Dependent Variable: Score

For a simple regression analysis with only one predictor variable, this is all you need to know. As soon as you start to add more variables, the interpretation becomes more complicated. On the other hand, it also becomes more interesting, because you might be able to explain more of the variance in the data.

How to 8

How to Do a Chi-square Analysis

8.1 Important assumptions and prerequisites for the Chi-square test You must have nominal variables only. Each case must occur in only one cell of the cross-table. None of the expected frequencies in the table can be lower than 5 in a 2x2 table. In a larger table, the frequencies must be at least 1, and no more than 20% of the cells are allowed to be less than 5. Below we’ll describe how you can find the expected frequencies.

8.2

The chi-square test: entering the data

There are two ways in which data can be entered in SPSS for a chi-square analysis. First, if the data are organized for each individual case, there’s no problem and you can simply run the analyses as described below. An example of this type of data organization would be: Participant – Variable 1 – Variable 2 1 NO YES 2 NO NO 3 YES NO Etc. Second, if you only have the total frequencies for each of the cells in the contingency table, you will have to enter the data using the weight cases procedure. Suppose we investigated the colour preference of men and women using a questionnaire that only allowed the participants to choose between red, yellow and blue. The contingency table may look something like Table II.8.1. Table II.8.1 Colour Red Yellow Blue

Female Male 28 24 23 23 29 42

122

How to SPSS

With this second way, we’ll create an SPSS data file in which we enter these data as in Figure II.8.1.

Figure II.8.1

Then we’ll go to Data Æ weight cases (see Figure II.8.2).

Figure II.8.2

We will weight cases by Frequency. From now on SPSS will understand that the frequencies represent individual cases. Note that you do not have to do this if the data is organized for each individual case.

8.3

The chi-square test: checking the assumptions

First. Before anything else, we must check if the assumptions for the chisquare analysis have been met. For this purpose we need to find out what the expected values are of each cell. We can do this using Crosstabs in the Descriptives menu (Analyze > Descriptive Statistics > Crosstabs). Enter the variables as illustrated in Figure II.8.3 (which variable ends up in the rows and which in the columns doesn’t make a difference).

How to Do a Chi-square Analysis

123

Figure II.8.3

Second. Click on Cells and tick the Expected box (see Figure II.8.4)

Figure II.8.4

Third. Look at the output and check whether the conditions have been met (Table II.8.2).

124

How to SPSS Table II.8.2

ColourPref* Gender Crosstabulation

Expected Count Gender Female ColourPref

Total

Male

Total

Blue

33.6

37.4

71.0

Red

24.6

27.4

52.0

Yellow

21.8

24.2

46.0

80.0

89.0

169.0

In our case, none of the Expected values is smaller than 5, so there’s no reason to worry and we can continue to the next step.

8.4

The chi-square test: getting results

First. The simplest way to analyse the frequency data is by using Crosstabs in the Descriptives menu (Analyze > Descriptive Statistics > Crosstabs). We have already entered the variables in the first step of section 8.3. Second. If you select Statistics, you can enter which statistical test you want to be carried out (see Figure II.8.5). Select at least the Chi-square. You may also want to select Lambda, as this will give you information on the direction of the effect (which variable is likely to be dependent on the other one).

Figure II.8.5

How to Do a Chi-square Analysis

125

Third. Then in the Cell submenu, tick the boxes that have been selected in Figure II.8.6, select continue and then OK.

Figure II.8.6

8.5

The chi-square test: interpreting the output

The first table in the output summarizes the number of cases. The second table (Table II.8.3) gives us the descriptives we asked for. It is quite relevant that we have asked for percentages, since we are now able to compare the relative frequency in each cell, from the Gender point of view and from the ColourPref point of view.

126

How to SPSS Table II.8.3

ColourPref* Gender Crosstabulation Gender Female

ColourPref

Blue

Red

Yellow

Total

Count

Male

Total

29

42

71

% within ColourPref

40.8%

59.2%

100.0%

% within Gender

36.3%

47.2%

42.0%

% of Total

17.2%

24.9%

42.0%

28

24

52

% within ColourPref

53.8%

46.2%

100.0%

% within Gender

35.0%

27.0%

30.8%

% of Total

16.6%

14.2%

30.8%

23

23

46

% within ColourPref

50.0%

50.0%

100.0%

% within Gender

28.8%

25.8%

27.2%

% of Total

13.6%

13.6%

27.2%

Count

Count

Count % within ColourPref % within Gender % of Total

80

89

169

47.3%

52.7%

100.0%

100.0%

100.0%

100.0%

47.3%

52.7%

100.0%

The next table (Table II.8.4) gives the significance of the chi-square test. Look at the top row for the interpretation of the results. Note: the degrees of freedom in a chi-square analysis is not related to the number of participants, but to the number of cells. Clearly, the effect in the current study did not reach significance. Since this value is not significant, there is no use in considering further analyses, like the Lambda we asked for. Table II.8.4

Chi-Square Tests

Value

Asymp. Sig. (2-sided)

df

2.215a

2

.330

Likelihood Ratio

2.222

2

.329

N of Valid Cases

169

Pearson Chi-Square

a. 0 cells (.0%) have expected count less than 5. The minimum expected count is 21.78.

If the result had been significant, the table below the chi-square tests in the output would have shown the Lambda. If Lambda has a value of 0 it does not predict the outcome of the other variable well, whereas when it is 1 it does predict it.

How to Do a Chi-square Analysis

8.6

127

The chi-square test: reporting on the results

Conventionally, the results of the chi-square is reported on as follows: ‘a chi-square analysis revealed that the association between gender and colour preference was not significant χ2 (2) = 2.22, p = 0.33’. In the case of significant results, don’t forget to report on the direction of the association. In your results, always include the contingency table (observed values, without the margins). If you like charts, you can tick ‘display clustered bar charts’ in the main menu of the Crosstabs and copy that to your output. Conventionally, charts are included for significant results only (see Figure II.8.7). Gender Female Male

50

Count

40 30 20 10 0 Blue

Red ColourPref

Figure II.8.7

Yellow

Bar chart

This page intentionally left blank

PART III

SPSS Practicals

This part contains six practicals to practice SPSS skills. The Practicals are related to Chapters 2–6. The instructions are given with reference to SPSS/ PASW 18. Since SPSS publishes annual updates of the program, the pictures shown may slightly deviate from those in your version.

This page intentionally left blank

Practical 1

Exploring SPSS and Entering Variables

This practical relates to Chapter 2 of Part I. In this practical you will become familiar with the statistical program SPSS. You will practise defining variables and entering data in SPPS. 1. START SPSS/PASW (Statistical Package for the Social Sciences/Predictive Analytics Software) a. The examples we give have been done in SPSS/PASW 18, but if you are working with an earlier or later version of SPSS you should not have too much difficulty with following the steps. b. When the program asks you what you want to do, click cancel. 2. EXPLORING THE PROGRAM a. What you see now is the Data View of the SPSS Data Editor. The columns on the screen are the variables; the rows are the cases (for example participants). b. Now switch to the Variable View by clicking the Variable View tab in the lefthand lower corner of your screen. In this view you can see the characteristics of the variables defined. Since you have not yet entered any data, everything is empty. c. Go back to Data View by clicking the Data View tab next to the Variable View tab. d. We will discuss the items in the Menu Bar later on. The only menu items you will need for this practical and the next are File (to open files, save files, etc.) and Analyze (to analyse your data). 3. DEFINING VARIABLES a. The first step in working with SPSS is to define the variables. Later on you can enter data for each variable defined. To practise working with SPSS we will create four variables: i. Subject number, nominal; ii. Age, interval (or scale); iii. Sex, nominal; iv. Proficiency score, interval (or scale).

132

SPSS Practicals

b. Enter these variables using the Variable View tab (as in 2b). For all the variables, fill in all fields as follows: c. The variable name is a brief name by which you can recognize the variable. In older editions of SPSS, the maximum length of the name is eight characters. d. The type refers to the type of characters used for the data to be entered later on. If you’re only going to use numbers, select numeric (and define the number of digits and the number of decimals). If you’re going to use letters, select string. This is related to, but not the same as, the variables scale (nominal, ordinal or interval): interval variables are always numeric, but nominal variables may also be numeric. e. The width is the number of characters you’ll maximally need for the data. This is the same as the width defined in the type. Normally this does not need changing. f. The same holds for the decimals. Obviously, for ‘string’ variables, the number of decimals is automatically set to 0 and cannot be changed. g. The label is the way in which the variable will appear in the output of your analyses. There are no length restrictions here, but it is best to keep it brief and clear. If the name is completely clear, you don’t need a label. h. If you use numbers for nominal variables (‘numeric’), you can enter the labels for the levels under values. So you can fill in what the numbers of the nominal mean. In this case you want to add the following labels to your levels: 1 = female, 2 = male. i. The way SPSS treats missing values can influence the calculations. For this practical, you can select the default no missing values. This means that any missing values that may occur will be treated as completely coincidental. j. Columns defines the width of the column as it is displayed in Data View. The default of eight is usually sufficient. If not you can always change it in Data View, so you don’t need to change it here. k. Align also defines how the data are displayed in Data View. You don’t need to change this. l. The Measure is very important. Here you must define what the scale is for the variable. Nominal and ordinal are simply the same; for interval data select Scale. This is just a matter of naming conventions. 4. SAVING THE SPSS DATA SHEET a. SPSS data sheets have the extension ‘.sav’. Save your file as SPSS-Prac1[your initials].sav. 5. ENTERING DATA IN SPSS a. You can find the data on the website in a Microsoft Word file. You can enter the data manually or use Copy and Paste from the document (only copy the numbers). 6. SAVING THE SPSS DATA SHEET a. Save your file again. It is a good habit to save your work regularly using SPSS. Especially older versions of the program tend to crash regularly.

Practical 2

Descriptive Statistics

This practical relates to Chapter 3 of Part I. In this practical you will become familiar with some more functions of SPSS. You will use the data that you entered in the previous practical and do some first analyses involving descriptive statistics.

Part A 1. CREATE A WORD FILE FOR YOUR ANSWERS a. Start a new Word file and save it as SPSS-Prac2[your initials].doc. Keep the Word file open to answer questions that will be asked later in this practical. Although figures are commonly copied from SPSS and pasted in Word, it is not customary to copy and paste tables from the SPSS output in research papers. Instead, report on the findings in your Word document and, if necessary, make your own table. 2. OPEN YOUR .SAV FILE FROM YOUR LAST SESSION (PRACTICAL 1) 3. FIRST CALCULATIONS: DESCRIPTIVE STATISTICS a. Use Analyze > Descriptive Statistics > Frequencies to find the frequency of occurrence of each age. Select the variable you want to analyse and move it to the right. Tick Display frequency tables. Then click OK. b. The results of the analyses done by SPSS will be displayed in a separate window, the Output File, displayed in the SPSS Viewer. Everything you do with SPSS in one session will end up in the same output file. Just like other file types, you can edit and save output files and start a new output file as well. c. Which age occurs most often? Answer this question in your Word file, but DO NOT cut and paste the table from SPSS to Word. Save your word file. 4. SAVING THE SPSS OUTPUT FILE a. SPSS output files have the extension .spv (or .spo in some older versions). Save your output file as Prac2a[your initials].spv. 5. MORE DESCRIPTIVES: EXPLORING THE FREQUENCIES MENU a. Once again, select Frequencies in the Descriptive Statistics menu. Now click on Statistics in the Frequencies selection menu.

134

SPSS Practicals

b. Using this option, find out what is the mean, the median, the mode, the range and the SD of the proficiency score in your data. Report on your findings in the Word file. Please keep in mind that the international standard is to use a dot (and not a comma) to report decimals (for example 0.5) and that you do not always have to report all decimals (use your common sense). 6. MORE DESCRIPTIVES: USING THE DESCRIPTIVES MENU a. Similar analyses can be done using different menu items in SPSS. Using Analyze > Descriptive Statistics > Descriptives, find out what is the minimum age, the maximum age, the mean age and the SD. Tip: use Options in the Descriptives menu to select the descriptive statistics you want. Report on your findings in the word file (again, don’t copy tables from SPSS). 7. MORE DESCRIPTIVES: DO IT YOURSELF a. What is the most frequently occurring proficiency score? Report your findings in the Word file. 8. MORE DESCRIPTIVES: COMPARING GROUPS a. Using Analyze > Descriptive statistics > Explore, find out the mean proficiency scores and the SDs for the male and the female subjects. Don’t include age in this analysis. Consider before you start: what is the dependent and what is the independent variable? In SPSS an independent variable is often called a ‘factor’. Only select the (descriptive) statistics, not plots. You don’t have to select any Options of Statistics. b. Which group has higher proficiency scores, the male or the female participants? c. Which group scored more homogeneously? d. Report your findings in the Word file and save your files. 9. CREATING A FIGURE a. Repeat step 8, but now produce a plot. If you click on Plots you can select a ‘stem-and-leaf’ plot and a histogram. In this case tick only Stem-and-leaf and click OK. Apart from the stem-and-leaf figure which is in the top part of the output, you will also get a boxplot, which is in the bottom of the output. Copy the boxplot to your Word file. After copying, right-click on the figure and select Caption. Enter a clear and explanatory caption to your figure. b. Compare your boxplot with the descriptive statistics. Can you find out how the boxplot is built up and what the different points of it signify? Explain this in your Word file (hint: there are four quartiles). c. Save your files.

Descriptive Statistics

135

Part B 1. Enter the following four datasets in SPSS and provide the mean, the mode, the median, the range and the SD. Hand in your data file and your output file in the Assignment in Nestor, using the same naming conventions as in Practical 2a. a. 3, 4, 5, 6, 7, 8, 9; b. 6, 6, 6, 6, 6, 6, 6; c. 4, 4, 4, 6, 7, 7, 10; d. 1, 1, 1, 4, 9, 12, 14. HINT: you can enter the datasets next to each other in one file.

The data to this part can be found on the website. The file is called prac2C_ optional.sav. When you open this file, you will see a sample of data from 100 university students who chose to study French. The data represents the motivation to learn French and the score on a French proficiency test. 1. OPEN THE .SAV FILE 2. DESCRIPTIVE STATISTICS a. Find out the mean, median, mode and SD for the group of students as a whole and then for the different motivation groups. Make a table in your Word file. b. Make a boxplot. Judging from the boxplot, do you think the different groups will differ from one another? Report on your findings. 3. CHECKING THE NORMAL DISTRIBUTION a. Now go to Analyze > Descriptive Statistics > Explore, leave out motivation as a factor, tick Both in Display, and select the Plots option. To check the distribution, select Histogram. Click OK. b. In the second table of your output, labelled Descriptives, look for the values of skewness and kurtosis. Do they deviate from zero? c. Now scroll down a bit to Histogram. If you double click on this, a new window opens in which you can customize your histogram. Right-click with the mouse and select Show distribution curve. By looking at the histogram, does the data seem to follow a normal distribution? d. Usually, you want to know whether your different groups are normally distributed by themselves. Carry out step 3a–c for the different motivation groups.

This page intentionally left blank

Practical 3

Calculations Using SPSS

This practical relates to Chapter 4 of Part I. In this practical you will review some descriptive statistics and you will learn to get a first impression about the normality of a distribution. Finally, you will do some first ‘real’ statistics.

Part A 1. In the file Prac3a-data.xls, on the website, you will find the results of the English Phonetics I exam of a student cohort in the English Department in Groningen, the Netherlands. The scores are specified per question (Q1, Q2, etc.), and we are going to assume that they are measured on an interval scale. Since these are real results, we have replaced the names with numbers for discretion. The questions below all refer to this file. Create one Word file containing all your answers in which you copy graphs from SPSS. Note: please do not copy output tables from SPSS to your Word file; just answer the questions based on the output. a. Importing data from Excel: Save the Excel file from Nestor (e.g. to the hard disk of your computer). Start up SPSS. Select File > Open > Data in SPSS. In the field Files of type, select Excel (*.xls). Make sure you ‘read the variable names’ while importing into SPSS, so that they are included in the header row. Import the range A1:V131, as this is the part of the Worksheet that contains data. b. Descriptives and graphs for groups: In Practical 2, you calculated mean values etc. for different groups. Check how to do this if you have forgotten. The students were in different groups (1A, 1B, etc.). Calculate the mean, maximum, minimum, range and SD per group, using the TOTAL score. Which group performed best? And which group performed most homogeneously? Which teacher (A or B) performed best? Using Explore, create a Box plot of the scores (based on TOTAL) for both teachers. Copy the boxplot to your Word file and create a clear caption that explains the contents of the figure.

138

SPSS Practicals

c. Checking for normality: Create a frequency polygon of the grades using the Frequencies function: opt for a Histogram with a normal curve line in the Chart menu. Do these results approximately follow the normal distribution (at face value by looking at the histogram)? Copy and paste the histogram to your Word file to illustrate your answer. As always, create a caption explaining the contents of the figure. The resulting graph should look something like the one shown in Figure III.3.1. 40

Mean = 5.27 Std. dev. = 1.702 N = 130

Frequency

20

15

10

0 0

2

4

6

8

10

GRADE

Notes: Mean = 5.27; SD = 1.702; N = 130.

Figure III.3.1 d. Using z-scores: Calculate and report on the z-score of the grade of the following students: 11, 33, 44 and 55. Hint: SPSS has a built-in facility to create z-scores as a new variable. In the Descriptives menu, opt for Save standardized values as variables. The z-scores will now be added to the data sheet (not the output) as a new column. 2. SAVE YOUR DATA FILE (.sav): YOU’LL NEED IT LATER ON 3. INDUCTIVE STATISTICS a. So far you have only done the descriptive statistics. What is your first impression about the difference between the groups of the two teachers?

Calculations Using SPSS

139

b. We will now go through a first example of hypothesis testing. The question we want to find out is whether there is a significant difference between the two teachers. We are going to find out if your impressions in 3a are correct. What is the null-hypothesis belonging to the research question? c. The statistical test we will carry out to test the null-hypothesis is the t-test. We will discuss this test later on in more detail; this is just to give you some first hands-on experience. You can find the t-test in the Analyze menu under Compare means. That, after all, is what we want to do; we want to compare the means of the TOTAL SCORES yielded by both these teachers. The test you want to use is the independent samples t-test, because the two samples we took (from the two teachers) are not related (there are different students in each group). Which variable is the grouping variable (that is, the independent variable)? Define the two groups by entering the exact ‘names’ as you find them in the column that represents this variable. d. Run the t-test. The t-test compares the scores of the two groups so that you can see whether there is a significant difference between them, that is whether the null-hypothesis should be rejected or accepted. Your output file will contain some information that is currently redundant (the first two columns in the output table, on Levene’s test). Only look at the results of the t-test: the most important information can be found in the column ‘Sig. (2-tailed)’: this is the chance of incorrectly rejecting the null-hypothesis (the alpha error), which looks as shown in Table III.3.1. What is the chance of incorrectly rejecting the null-hypothesis? Table III.3.1

Independent samples test t-test for Equality of Means t

TOTAL

Equal variances assumed

4.266

df 118

Sig. (2-tailed) .000

e. What is the conclusion you can draw with regard to the research question in 3a? Is the difference between the two groups significant? Is it safe to reject H0? What is the chance of erring in our decision to reject H0 (the alpha error)?

Part B 4. The file Prac3b_data.xls is an Excel file that contains the results of a vocabulary test (interval scores) for participants from two different motivation levels. The data come from an experiment in which motivation was a nominal independent variable and vocabulary score was an interval dependent variable. Using all the tools and knowledge you have used so far, determine whether there is a (significant) effect of motivation on the vocabulary scores and report on it in your Word file.

140

SPSS Practicals

1. CHECKING FOR NORMALITY One of the assumptions of the t-test is that the data are distributed according to the normal distribution. You could of course simply run Explore and look at the skewness and the kurtosis. If they are very close to 0, or at least between –1 and 1, you can assume that the distribution is approximately normal. However, some of you may want to know what the chance is of going wrong in assuming the normal distribution. This can be established by applying the KolmogorovSmirnov test or the Shapiro-Wilk test. For sample sizes smaller than 50 it is better to use Shapiro-Wilk. When sample sizes are larger, it is better to use Kolmogorov-Smirnov. Note: if you have more than one group, you’ll have to put in the groups as a factor. It’s important that the distribution of each group is normal, rather than the distribution of the scores of the groups taken together. In this assignment, we will go back to the extra-challenging section at the end of Practical 2. If you don’t have the .sav file anymore you can open prac2C_ optional.sav from the website. a. Go to Analyze > Descriptive Statistics > Explore, put in Motivation as a factor, tick Both in Display, and select the Plots option. To do a normality test, check Normality plots with tests and Histogram. Click OK. b. Above your histogram in the Output file should be a table that says Tests of Normality (see Table III.3.2). Because we want to know the distribution of the different groups, and since these groups are smaller than 50, we should look at the Shapiro-Wilk results. The null-hypothesis is that the distribution is not different from the normal distribution. If the significance value is above .05, then you can assume that the data are normally distributed. In this case, do the data show a normal distribution? (Remember always to use these tests with caution and always make a histogram as well. Especially with large sample sizes, these two tests will more easily give a p-value smaller than .05.) Table III.3.2 Tests of normality This is the significance of the test. NB: The H0 of this test is that the distribution does not deviate from the normal distribution. So when this test is NOT significant, the distribution can be considered ‘normal’.

This is the value of the test (something like the value of t in a t-test) Tests of Normality Kolmogorov-Smirnova Motivation Proficiency

Statistic

df

Sig.

Shapiro-Wilk Statistic

df

Sig.

Highly unmotivated

.106

20

.200*

.963

20

.605

Somewhat unmotivated

.139

24

.200*

.962

24

.486

Neutral

.082

27

.200*

.979

27

.849

Somewhat motivated

.100

17

.200*

.977

17

.927

Highly motivated

.183

12

.200*

.920

12

.289

a. Lilliefors Significance Correction *. This is a lower bound of the true significance.

the df is the same as the N in a test

Calculations Using SPSS

141

2. CHECKING FOR EQUALITY OF VARIANCE Another assumption is that there is equality of variance between the groups. This means that the SD in one group should not be drastically different from that in another group. A good rule of thumb is usually to make sure that the largest SD is not more than twice as big as the smallest one. a. Again, go to Analyze > Descriptive Statistics > Explore, leave in motivation as a factor, and click on the Plots option. Deselect Normality plots with tests and Histogram. Under Boxplots select none. Under Spread vs Level with Levene Test, select Untransformed. Click OK. b. First look at the different SDs of the different groups. Which is the largest SD? And which is the smallest? Do you think these groups are equal in their variance? c. In your output file, you will also see a table labelled ‘Test of Homogeneity of Variance’. The null-hypothesis here would be that the groups are similar in variance. If the significance value is higher than .05, you can assume that the assumption of equality of variance is met. In this case, are your groups equal in variance?

This page intentionally left blank

Practical 4

Inductive Statistics

This practical relates to Chapter 5 of Part I. In this practical we will take the next step in applying inductive statistics. You will do a simple means analysis and a correlation analysis. You will also learn how you should report the results of these statistical calculations. This practical contains an optional advanced assignment on correlation for reliability. Create a Word document containing all your answers. Only cut and paste from the SPSS output when you’re explicitly asked to do so. So type your answers in the Word file, rather than copying them from the .spv file. Use the usual naming conventions for your files: Prac4[your initials].doc/sav/ spv. 1. APPLYING THE t-TEST A researcher wants to find out whether boys or girls are more intelligent. Eleven girls and eight boys (randomly selected) participated in an experiment in which scores were in the range 1–20 (interval). The data can be found in Table III.4.1. Table III.4.1 Girls

Boys

17 16 14 19 18 17 16 15 16 15

16 15 13 19 15 14 13 12

19

a. What are the dependent and independent variables? b. What kinds of measures (nominal, ordinal or interval/scale) are used for the variables?

144 c. d. e. f.

g.

h.

i. j.

k.

SPSS Practicals How many levels does the independent variable have? Formulate a statistical hypothesis. Select an alpha level suitable for this study. As is explained in Chapter 6, the statistical test you want to use to analyse the result is the t-test. Look at Table I.6.1 and check whether you can confirm this. Enter the data in SPSS (creating two variables, refer to 1a above). Hint: carefully consider this step – the two columns (Girls and Boys) in the data are not necessarily the variable columns in SPSS. Remember that the columns in SPSS represent variables, not levels of variables. Provide the following descriptive statistics for both groups: means, range, minimum, maximum, SDs. To calculate these, use Analyze > Descriptive Statistics > Explore and enter the dependent and the independent variable (often called a ‘factor’ in SPSS). What are your first impressions about the difference between the boys and the girls? Create a boxplot to visualize the results. Hint: use Explore. Include the plot in your Word file. Hint: select the plot, use the right mouse button to select Copy and in Word use Paste Special and select ‘Bitmap’ or ‘Picture’. Using SPSS, test the statistical significance of this experiment using the following steps: First go to Compare Means in the Analyze menu in SPSS to obtain the descriptive statistics. In this menu you can find several items related to means analyses. The top one Means, gives the most important descriptives (this is very similar to Explore). Find out which group has a distribution that most resembles the normal distribution. Just give your first impressions here. Select the Independent samples t-test from the Compare Means menu. Why do you have to use this test rather than the one sample t-test or the paired samples t-test? In this test, the dependent is called the ‘test variable’ and the independent the ‘grouping variable’. Define the levels of the independent by selecting Define Groups. Run the test. Carefully study the SPSS output. Taking Levene’s test into account, what is the value of t? Which degrees of freedom are applied to this test? What is the level of significance of these samples? Compare this to the alpha level you set in 2e above. Can you reject H0? Reporting on results of statistical studies has to be done according to fixed conventions. The rule is never to copy and paste tables from SPSS but to report in writing. Here’s how you should do it, as suggested by Field (2005, p. 303), applied to our test. Fill in the missing details; our comments are in square brackets:

On average, the ... [fill in boys or girls] showed a higher level of intelligence (M = ... , SE = ... ) than the ... (M = ... , SE = ... ). This

Inductive Statistics

145

difference was ... [fill in ‘significant’ or ‘not significant’] t (... [fill in Df]) = ... [fill in the value of t], p ... [fill in < 0.05 or > 0.05 or whichever α you’ve selected]. l. What can you say about the meaningfulness of this outcome? m. Is there any additional information you’d like to have about this study? 2. CONSIDER THE FOLLOWING DATA a. Eight students have participated in a reading test and a listening comprehension test. Reading ability and listening comprehension are operationalized by the variables R and L respectively. Both variables are measured on an interval scale. The results have been summarized in Table III.4.2. Enter the data in SPSS. Table III.4.2 Student 1 2 3 4 5 6 7 8

R

L

20 40 60 80 100 120 140 160

65 69 73 77 80 84 89 95

b. What would be H0 if we want to test the relationship between reading and listening comprehension? c. Make a plot of the results. Hint: use Graphs > Scatter/Dot and define the xand y-axes. Include the plot in your Word file. d. At face value, do you think reading and listening, as plotted in the graph, are related? e. We want to know if we can conclude that reading skills and listening comprehension are significantly related. To determine this, you will have to calculate a Pearson r (or rxy). To do this, go to Analyze > Correlate > Bivariate and enter the two variables you want to correlate. Make sure the computer calculates the Pearson correlation for a two-tailed test. What is the value of rxy? Is this a strong correlation? What is the chance of incorrectly rejecting your H0? What do you decide? f. Write a sentence that you could include in the Results section of a study reporting on the outcome of your test. It will be something like this: A correlation analysis showed that reading skills and listening skills were ... [significantly or not significantly] related (r = ... , p ... [fill in < 0.05 or > 0.05 or whichever α you’ve selected]).

146

SPSS Practicals

In the extra-challenging section at the end of Chapter 5 in Part I, we briefly discussed reliability and the fact that Cronbach’s Alpha was a good measure to check for reliability of a test. The teachers in the data in Practical 3 Part A are interested in the reliability of their exam. They have decided to use Cronbach’s Alpha to check this. a. Open the data for Prac3A to check the reliability of a 40-item vocabulary test. b. Decide whether the test is reliable by going to Analyze > Scale > Reliability Analysis. Put all the Questions in the Items (and not the Total and the Grade), and choose Alpha next to Model. Click OK. The Output will give you a correlation coefficient. Now put A and B in the Items and choose Split-Half next to Model. Click OK. The Output will give you the value for Cronbach’s Alpha. Do you think this is a reliable test? c. Now we will check the individual items. Click on Statistics. Tick Inter-Item Correlations and Descriptives for Scale if item deleted. Click OK. The output will give you the correlations between items and all of the Cronbach’s Alpha values without a particular item. With the deletion of which item do you get the highest reliability?

Practical 5

Miscellaneous Assignments

This practical relates to Chapter 6 of Part I.

5.1

Weight and height

A researcher wants to find out if there is a relationship between weight and height. She collects data from six subjects. (Hint: if you have problems with the dots, depending on the settings of your computer (Regional and Language settings), you may have to change the decimal sign in the data into commas. The data can be found in Table III.5.1 Table III.5.1 Weight

Height

40 50 40 70 80 90

1.40 1.50 1.60 1.70 1.80 1.90

a. List the variables in the study. If relevant, say which variables are dependent and which are independent. b. What kind of measures (nominal, ordinal, interval) are used for the variables? c. Formulate the relevant statistical hypothesis. d. Is the relation linear (plot the data in a simple graph)? e. Which alpha level would you use and why? f. Would you test one-tailed or two-tailed (and why)? g. Which statistic could be used (consult the text in section 6.3)? h. Apply this statistic using SPSS. Can you reject H0? i. What can you say about the meaningfulness of this outcome? j. Report on the results of this study in the way it is conventionally done in research papers (see section II.6.4 on p. 116).

148

SPSS Practicals

5.2 Writing scores An English Department wants to determine the effect of different types of instruction on writing skills, namely no instruction, explanation in lectures only, and guided writing (GW). Thirty students are randomly assigned to the programs. The scores in Table III.5.2 are the results of a writing test, measured on an interval scale (0–100). Table III.5.2 No_instr

Lectures

GW

34 58 56 47 35 31 55 65 61 27

65 54 43 57 65 49 74 79 54 65

68 87 94 69 81 75 94 78 63 78

a. List the variables in the study. If relevant, say which variables are dependent and which are independent. b. What kind of measures (nominal, ordinal, interval) are used for the variables? c. In the case of independent variables, how many levels does each independent variable have? d. Formulate the statistical hypotheses (null-hypothesis and alternative hypotheses). e. Which statistic could be used? f. Using SPSS, provide the following descriptive statistics for each group: means, range, SDs. g. Using SPSS, test the statistical significance of this experiment. Can you reject H0? h. What can you say about the meaningfulness of this outcome? i. Report on the results of this study in the way it is conventionally done in research papers.

Now suppose we want to find out whether the effect of different teachers would also influence the result. In this case we are adding an extra variable to the existing dataset. Answer questions a–e (above) for this situation.

Miscellaneous Assignments

149

5.3 Crosstabs A sociolinguistic researcher wants to find out if there is a relation between the use of haven’t got versus don’t have and social class (two levels). Social class is determined by asking people what they do for a living. The use of haven’t got or don’t have is determined by asking subjects to rephrase sentences (for instance: Jim is jobless). a. List the variables included in this study. b. For each variable, say what its function is (dependent, independent, etc.) and its type (nominal, ordinal, interval). c. How would you formulate H0 and H1? d. Which statistic could be used? e. Choose your alpha level. f. Using the data file provided (Prac5_data.xls), run the SPSS analysis. g. Can you reject the null-hypothesis? h. Report on the results of this study in the way it is conventionally done in research papers.

5.4 Mixed statistics Choose at least three of the problems that you encountered in Activity 6.6 in Chapter 6 (you’re welcome to do them all) and work these out in detail. Provide the independent and dependent variables(s) and the scale of each variable. Then indicate which statistic would be most suitable to assess the results (first determine the nature of the problem and pick a family of statistics: means, frequency or correlation; then choose the most appropriate statistic). Sample data for all of these imaginary studies can be found on the accompanying website. Using these data, you can run the analyses you selected and draw your conclusions. Include the following points in all your answers: List the variables in the study: if relevant, say which variables are dependent and which are independent. For each of the variables determine its type (nominal, ordinal, scale). In the case of independent variables, how many levels does each independent variable have? Identify the family of statistics(means, frequency or correlation) then choose the most appropriate statistical test. Formulate the relevant statistical hypotheses. Which alpha level would you use and why?

150

SPSS Practicals

Would you test one-tailed or two-tailed (and why)? Run the analyses using the data on the website and then answer the following questions: ● What is the value of the statistic? ● What is the significance level? ● What is your decision regarding H0? Report on the results of this study in the way it is conventionally done in research papers. If necessary, you can illustrate this sentence using one or more figures or tables with descriptives.

Here is another assignment, which was not mentioned in Activity 6.6. A researcher wants to know whether proficiency in the second language can be used to predict successfulness in a third language. Students who have taken six years of French and who are currently enrolled in a Spanish language program are participating. Their average scores for French in their final year of studying are taken as the measure for L2 Proficiency (L2Prof), and they are asked to do an exam for Spanish. a. b. c. d.

The scores are in the file Prac5_4.doc on the website. Enter these into SPSS. Are the variables normally distributed? Make a scatter plot of the variables with a fit-line. What do you notice? Now go to Analyze > Regression > Linear. Put L3Score in Dependent and L2Prof in Independent. Click OK. What are the values of the intercept and the slope? Does the proficiency in the second language significantly predict the success of the third language? e. How would you report these results?

Practical 6

Miscellaneous Assignments (Revision)

Answer the following questions about the research designs sketched below: Explain how the constructs are operationalized (list the variables). For each of the variables, mention its function in the experiment (dependent, independent) and its scale (nominal, ordinal, interval). Formulate the relevant research hypotheses (H0 and H1/H 2). Say whether you would test one-tailed or two-tailed. State which statistic you would use to analyse the results and explain why. Comment on the possible methodological pitfalls and the validity.

After this, carry out the relevant analyses using SPSS and report on the studies in the conventional way. If possible, also report on the assumptions, such as homogeneity of variance or normality of distribution. Also report on effect size when possible (see Equation I.6.2) and illustrate your answer with tables (descriptives, but not copied from SPSS) and graphs. Provide figures to illustrate your data when this is relevant.

6.1

L2 syntax development

As part of an ongoing study on the nature of L2 input, two researchers want to examine the role of conversation in L2 syntax development. They assume that the more interaction takes places within the classroom, the better the learners are able to construct well-formed sentences. In order to test this, they conduct a small-scale study in which thirteen 17-year-old pupils at a secondary school in one and the same English class are closely monitored. For a period of six weeks the pupils are asked to keep a diary in which they report how often and for how long they speak English in the classroom. At the end of the six-week period all pupils take an oral exam in which their sentence constructions are assessed. The grades of the exam are compared to the total time spent on conversation (added up in minutes). What is the relation between conversation and L2 syntax development?

152

SPSS Practicals

Use the data in Table III.6.1 to carry out your analysis: Table III.6.1 Pupil 1 2 3 4 5 6 7 8 9 10 11 12 13

6.2

Minutes

Grade

32 76 89 41 17 47 62 81 93 56 68 71 26

4 5 9 6 4 7 7 8 8 6 8 8 6

Spanish pronunciation proficiency

A Spanish language researcher wants to measure the influence of spending time in a Spanish speaking country on the pronunciation proficiency of Chinese students of Spanish after four years of studying. During the course of their studies, the students can choose to spend either one semester abroad or not to go abroad. At the end of their fourth year all students take a pronunciation test, which they can either pass or fail. Does an international exchange make a difference to an individual’s pronunciation proficiency? Use the data in Prac6_2.doc on the website to carry out your analysis (refer to How to 8 – pp. 121–7) for information about how to enter these data into SPSS for this type of statistic).

6.3

Vocabulary learning experiment

In a vocabulary learning experiment, two methods of learning words were compared: the ‘CORE’ method and the ‘PERI’ method. Fifty-eight Dutch learners of English were randomly assigned to these two methods. All learners were given the same 100 English sentences (‘S2’), containing a word they did not know. They first had to guess the meaning of the word and were then given the Dutch translation equivalent. The two groups were each given a different additional sentence to learn the words. In the ‘CORE’ method group, new words were learnt by referring to the core meaning of the word. The core meaning is the most prevailing literal meaning of a word. For instance, the figurative meaning of the word nugget (see Table III.6.2) was learned with reference to a small lump of gold. In the ‘PERI’

Miscellaneous Assignments (Revision)

153

method group, new words were learned by referring to a different figurative (peripheral) meaning. Table III.6.2 nugget

S1 S3

S2

Example of the different sentences as used in the experiment

His father originally sent him solid golden nuggets. They came up with the nugget that he had been involved in dubious business speculations. The new LSS does that with a choice of V6 engines and with a body, interior and suspension that make the car a true nugget in today’s rushing stream of fancy cars.

Goudklompje (nugget of gold) interessante informatie (interesting information) Juweeltje (Jewel)

Interval scores were recorded for each participant at three moments in time: the guessing of the word before the translation equivalent was given (G); the short-term retention score ten minutes after the item had been studied (ST); and the long-term retention score one week after the initial experiment (LT). The researchers wanted to know if the CORE method led to more correct short-term memory of the words than the PERI method and if the CORE method led to better long-term retention than the PERI method. Hint: although this type of repeated testing would normally be analysed using a so-called Repeated Measures ANOVA design, this goes beyond the scope of this introductory book. In this case, another way to deal with this problem of testing the difference between the long-term and short-term memory is by subtracting the ST scores from the LT scores. The data for this assignment can be found in the file Prac6_3data.sav on the website. See Verspoor and Lowie (2003).

6.4

Vocabulary scores and instruction

In the Netherlands, children usually start with English classes in grade 5. Most children by then have already had some exposure to English through the media. A researcher wanted to test a group of fifth-graders at the beginning of the year (pretest) and test them again at the end of the year to see how much vocabulary they learn in a year (post-test). A group of 42 children participated in this experiment. Both the pretest and the post-test consisted of 20 similar multiple choice questions about the translation of nouns from English to Dutch. For each correct answer, the children could get 1 point. Hint: the group of fifth-graders takes two tests. Therefore, these scores are related. You can find the data in Prac6_4data.sav on the website.

This page intentionally left blank

Index

A Alpha error (α) 41–3, 48, 55 Alternative hypothesis 42, 43, 52 ANOVA 63 Assumptions 39, 46–8, 91–3 B Bar graph 33 Beta error (β) 41–2, 48–9 Bivariate correlation see Correlation Bonferroni correction 62 Boxplot 27, 134 C Case studies 12–13 Causal modelling 67 Chi-square (χ2) 72–4, 121–7 Cohort effect 12 Complexity theory 6 Construct 19–21 Construct validity 77–8 Contingency table see Cross-tabulation Correlation 65–9, 71, 113–116, 145 Cronbach’s alpha 58, 78, 146 Cross-sectional research see Research Cross-tabulation 72–3, 121 D Data collection 54 Data view 131 Degrees of freedom (Df) 46, 98 Description 7–8 Descriptive research see Research Descriptive statistics 25, 55, 87–90, 133–5 Discourse completion task 10 Dispersion 26–8 Distribution see Frequency

DST or DST/CT see Dynamic Systems Theory Dynamic Systems Theory 6, 12, 30, 47, 80 E Effect 8 size 48, 53, 64–5 Equality of variance see Variance, homogeneity Error 70 Expected value 74, 121, 122–4 Experimental research see Research Explore 89 F F 60–2, 103 Factor see Variables, dependent Falsification (principle of) 42–3 Frequency 31–4, 87 distribution 31–2 analysis 72 polygon 35 tally 31 G Gabriel’s procedure 104 Games-Howel procedure 104 Generalization 6, 12, 39, 45 G-power 47 Grouping variable see Variables, independent H Histogram 33, 138 Hochberg’s GT2 104 Homogeneity of variance see Variance

156

Index

Hypothesis 5 formulation 5, 52 I In situ research see Research Inductive statistics 25, 39, 45, 143 Interaction 63, 110–11 Intercept 70, 117 Interquartile range 27 Intervention 11 Introspection 15 K Kendall’s Tau 71 Kolmogorov-Smirnov test 47, 92, 140 Kolmogorov-Smirnov Z-test 100 Kruskal-Wallis H-test 63, 106 Kurtosis 36–8, 47, 92, 140 L Lambda 124, 126 Level of significance see Significance Levels 21, 59 Levene’s test 48, 93, 98–9, 139, 141 Line graph 34 Longitudinal research see Research Lower quartile 27 M Main effect 110 Mann-Whitney U test 64, 100 Mean 25 Meaningfulness 49, 79 Means analyses 60 Median 26, 27 Min-Max graph 30–1 Mode 26 Mortality (subject mortality) 12 Multiple regression analysis 71 Multivariate ANOVA 63 N Non-parametric statistics 39, 46, 59, 63–4, 71, 100, 106 Normal distribution 34–5, 41, 47, 92, 138, 140 Null-hypothesis (H0) 42–3, 52, 55, 66

O Observer’s paradox 54 One-tailed 44, 52, 55, 98 One-way ANOVA 60–3, 101–106 Operationalization 19–24, 51, 78 Output file 133 P Parametric statistics 46–8, 59 Partial correlation 68–9 Pearson r 66, 113 Percentile 35 Population 6, 25, 39, 45, 52 Posthoc analysis 62, 103–105 Power 48, 53 Predictor variable see under Variables p-value 45, 103 Q Qualitative research see Research Quantitative research see Research R Range 26–7 Regression 69–71, 117–20 Regression line 70 RegWQ 104 Reliability 57–8, 78 Repeated measurements 11 Repeated measures 63 Representative sample see Sample Research cross-sectional 11–2 descriptive 7 experimental 8–10, 14, 53 in situ 14–5 longitudinal 11–2, 31 process 10–1 product 10–1 qualitative 13–4 quantitative 13–4 quasi-experimental 53 Researcher expectancy 54 S Sample 6, 25, 39, 45, 52–3 random 52–3 representative 6, 7, 52–3 stratified 52

Index Sample size (N) 46, 48, 53 Scale 21–2 Scatterplot 66, 114–116 SD see Standard deviation 30 Second-language development 9 Self-selection 54 SEM see Standard error of the mean Shapiro-Wilk test 47, 92, 140 Significance 43–4, 51, 55, 57, 80 level of 54 Skewness 36–8, 47, 92, 140 negatively skewed 37 positively skewed 37 SLD see Second-language development Slope 70, 117 Spearman’s Rho 71, 113 Split-half correlation 57–8, 78 Standard deviation (SD) 29, 45 Standard rrror of the mean (SE or SEM) 45, 98 Statistic 55 Stratum see Sample, stratified Subject expectancy 54 Subject mortality 12 T Test variable see Variables, dependent Test-retest reliability 57 t-test 48, 55–7, 60, 95–100, 139 one-sample 96 independent samples 96–9, 143 paired samples 96

Tukey HSD 104–5 Two-tailed 44, 52, 55 Two-way ANOVA 60, 107–12 Type I error see Alpha error Type II error see Beta error U Upper quartile 27 V Validity 10, 15, 19–20, 77–8 Variable view 131 Variables 19–24, 51 control 23 dependent 22, 97 independent 22, 97 interval 21 nominal 21 ordinal 21–2 predictor 69, 117–8 ratio 21 Variance (s2) 8, 29, 64 homogeneity of 46–8, 93, 141 Variation 8 Volunteer effect 54 W Weight cases 121–2 Welch’s F 103 Z z-score 30

157

This page intentionally left blank