Quantitative Methods for Second Language Research_ A n Language Assessment)

Citation preview

Quantitative Methods for Second Language Research

Quantitative Methods for Second Language Research introduces approaches to and techniques for quantitative data analysis in second language research, with a primary focus on second language learning and assessment research. It takes a conceptual, problem-solving approach by emphasizing the understanding of statistical theory and its application to research problems while paying less attention to the mathematical side of statistical analysis. The text discusses a range of common statistical analysis techniques, presented and illustrated through applications of the IBM Statistical Package for Social Sciences (SPSS) program. These include tools for descriptive analysis (e.g., means and percentages) as well as inferential analysis (e.g., correlational analysis, t-tests, and analysis of variance [ANOVA]). The text provides conceptual explanations of quantitative methods through the use of examples, cases, and published studies in the field. In addition, a companion website to the book hosts slides, review exercises, and answer keys for each chapter as well as SPSS files. Practical and lucid, this book is the ideal resource for data analysis for graduate students and researchers in applied linguistics. Carsten Roever is Associate Professor in Applied Linguistics in the School of Languages and Linguistics at the University of Melbourne, Australia. Aek Phakiti is Associate Professor in TESOL in the Sydney School of Education and Social Work at the University of Sydney, Australia.

Quantitative Methods for Second Language Research A Problem-Solving Approach

Carsten Roever and Aek Phakiti

BUTUH LENGKAP HUB

[email protected]

First published 2018 by Routledge 711 Third Avenue, New York, NY 10017 and by Routledge 2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN Routledge is an imprint of the Taylor & Francis Group, an informa business © 2018 Taylor & Francis The right of Carsten Roever and Aek Phakiti to be identified as authors of this work has been asserted by them in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Every effort has been made to contact copyright-holders. Please advise the publisher of any errors or omissions, and these will be corrected in subsequent editions. Library of Congress Cataloging-in-Publication Data A catalog record for this book has been requested ISBN: 978-0-415-81401-0 (hbk) ISBN: 978-0-415-81402-7 (pbk) ISBN: 978-0-203-06765-9 (ebk) Typeset in Bembo by Apex CoVantage, LLC Visit the Companion Website: www.routledge.com/cw/roever

Contents

List of Illustrations Foreword Preface Acknowledgments 1 Quantification 2 Introduction to SPSS 3 Descriptive Statistics 4 Descriptive Statistics in SPSS 5 Correlational Analysis 6 Basics of Inferential Statistics 7 T-Tests 8 Mann-Whitney U and Wilcoxon Signed-Rank Tests 9 One-Way Analysis of Variance (ANOVA) 10 Analysis of Covariance (ANCOVA) 11 Repeated-Measures ANOVA 12 Two-Way Mixed-Design ANOVA

13 Chi-Square Test 14 Multiple Regression 15 Reliability Analysis Epilogue References Key Research Terms in Quantitative Methods Index

Illustrations

Figures 2.1 New SPSS spreadsheet 2.2 SPSS Variable View 2.3 Type Column 2.4 Variable Type dialog 2.5 Label Column 2.6 Creating student and score variables for the Data View 2.7 Adding variables named ‘placement’ and ‘campus’ 2.8 The SPSS spreadsheet in Data View mode 2.9 Accessing Case Summaries in the SPSS menus 2.10 Summarize Cases dialog 2.11 SPSS output based on the variables set in the Summarize Cases dialog 2.12 SPSS menu to open and import data 2.13 SPSS dialog to open a data file in SPSS 2.14 Illustrated example of an Excel data file to be imported into SPSS 2.15 SPSS dialog when opening an Excel data source 2.16 The personal factor questionnaire on demographic information 2.17 SPSS spreadsheet that shows the demographic data of Phakiti et al. (2013) 2.18 The questionnaires and types of scales and descriptors in Phakiti et al. (2013) 2.19 SPSS spreadsheet that shows questionnaire items of Phakiti et al. (2013) 3.1 A pie chart based on gender 3.2 A pie chart based on a 10-point score range

3.3 A bar chart based on a 10-point score range 3.4 An example of questionnaire items using a Likert-type scale 3.5 The positively skewed distribution of length of residence 3.6 The negatively skewed distribution of speech act scores 3.7 The low skewed distribution of implicature scores 4.1 Ch4TEP.sav (Data View) 4.2 Ch4TEP.sav (Variable View) 4.3 Defining gender in the Value Labels dialog 4.4 Defining selfrate (self-rating of proficiency) in the Value Labels dialog 4.5 Defining missing values 4.6 SPSS menu for computing descriptive statistics 4.7 Frequencies dialog 4.8 Frequencies: Statistics dialog 4.9 Frequencies: Charts dialog 4.10 A histogram of the self- rating of profi ciency variable with a normal curve 4.11 SPSS Descriptives options 4.12 SPSS graphical options 4.13 SPSS bar option 4.14 SPSS pie option 4.15 SPSS histogram option 4.16 The histogram for the total score variable 5.1 A scatterplot displaying the values of two variables with a perfect positive correlation of 1 5.2 A scatterplot displaying the values of two variables with a correlation coefficient of 0.90 5.3 A scatterplot displaying the values of two variables with a correlation coefficient of 0.33 5.4 A scatterplot displaying the values of two variables with a perfect negative correlation coefficient of –1 5.5 A scatterplot displaying the values of two variables with a low correlation coefficient of 0.06 5.6 SPSS output displaying the Pearson product moment correlation between

two subsections of a grammar test 5.7 A view of Ch5correlation.sav 5.8 SPSS graphs menu with Scatter/Dot option 5.9 Simple scatterplot options 5.10 A scatterplot displaying the values of the listening and grammar scores 5.11 Adding the fit line in a scatterplot 5.12 A scatterplot displaying the values of the listening and grammar scores with a line of best fit added 5.13 SPSS Bivariate Correlations dialog 6.1 A normally distributed data set 7.1 Accessing the SPSS menu to perform the independent-samples t7.2 SPSS dialog for the independent-samples t7.3 Lee Becker’s effect size calculators 7.4 Accessing the SPSS menu to perform the paired-samples t7.5 Paired-Samples T Test dialog 8.1 SPSS menu to perform the Mann-Whitney U test 8.2 SPSS dialog to perform the Mann-Whitney U test 8.3 SPSS menu to perform the Wilcoxon Signed-rank test 8.4 SPSS dialog to perform the Wilcoxon Signed-rank test 9.1 SPSS menu to launch a one-way ANOVA 9.2 Univariate dialog for a one-way ANOVA 9.3 Options for post hoc tests 9.4 Options dialog for ANOVA 9.5 SPSS menu to launch the Kruskal-Wallis test 9.6 Setup for the Kruskal-Wallis test 9.7 Variable entry for the Kruskal-Wallis test 9.8 Analysis settings for the Kruskal-Wallis test 9.9 Kruskal-Wallis test results 9.10 Model Viewer window for the Kruskal-Wallis test 9.11 Viewing pairwise comparisons 9.12 Pairwise comparisons in the Kruskal-Wallis test 10.1 Accessing the SPSS menu to launch the Compute Variable dialog 10.2 Compute Variable dialog

Minitab software 14 missing values, assigning 47–8, 47–8 mode 37–9 moderator variables 135–9 multiple regression: ANOVA result and 210–11, 211, 216, 216; assumptions of 205; collinearity and 205, 212, 212, 217, 217; describing 203–5, 204; descriptive statistics in 209, 209; hierarchical regression and 203, 204; model coefficient outputs and 211–12, 211–12, 216–17, 216–17; overview 218; sample size in 205; in second language research 200; simple 200–3, 201; in SPSS program 206–12, 206–12 multivariate analysis of variance (MANOVA) 118, 160, 161 Multivariate Tests 164 negative correlations 62–6, 66, 68 negatively skewed distribution 41, 42 negative ranks 114 nominal data 7–8, 8, 39 nominal variables, assigning value to 44–7, 45–7 nonparametric tests: determining use of 106; Mann-Whitney U test 106–11, 107–11; overview 116; in second language research 106; Wilcoxon Signed-ranked test 111–16, 112–15 non-SPSS method for chi-square test 195–8, 196–8 normal distribution 66, 85, 85 normal distribution measures 40–3, 41–2 null hypothesis 89, 89, 183, 185 one-dimensional chi-square test 182–5, 183 one-way analysis of variance see analysis of variance (ANOVA) ordinal data 5–7, 5–6, 39–40 ordinal-ordinal relationships 68 outcome variable 119–20 outliers 36–7, 36, 106 paired-samples t-tests 93–5, 95, 102–4, 102–4 pairwise comparisons 133, 133, 164, 164 430

parameters 81 parametric statistic 66 partial eta squared 121, 157–8 partialing out covariate 139 Pearson: correlation analysis 43; correlation coefficient 121, 182, 204; Product Moment 66, 70–9, 71, 77, 230; Pearson’s r 66–8 percentage of agreement 229 performance rating 234–8 phi coefficients 68, 184; Phi value 184, 187, 195 pie charts: in descriptive statistics 33–5, 34; in descriptive statistics in SPSS program 56, 56; SPSS program instructions for 56, 56 platykurtic distribution 43 point-biserial correlation 69 populations 81–3 positive correlations 60, 62–6, 64, 68 positively skewed distribution 41, 41 positive ranks 114 post hoc tests 119–21, 126, 127, 140, 150, 156–7, 178–9, 178 predictor variable 201, 205, 209–10, 209 pre-post studies 154–6, 154–5 probability 83–4 PSPP software 14 purposive sampling 82 p-value 84, 88–90, 93, 100, 120 qualitative data coding 234–8 quantification: categorical data in 7–8, 8; constructs in 2; data in 2; describing 2–3; descriptive statistics in 28; at group level 28–30, 29–30; hypotheses in 2; inferential statistics and 81–2; interval data in 3–4, 4; issues in 2–3; measurement scales in 3–8; nominal data in 7–8, 8; ordinal data in 5–7, 5–6; overview 1, 13; ratio data in 3–4, 4; sample study 12–13; in second language research 1, 3; topics in second language research and 11–12; transforming data in real-life context 8–11, 9–11

431

random assignment 82 random sampling 82 ranks statistics 114, 115 rater reliability: Cohen’s kappa and 234–8, 234–8; correlation coefficients and 229–30; describing 227–8; inter-coder reliability and 228–30; inter-rater reliability and 228–30; intraclass correlation coefficient 238–9, 239; intra-rater reliability measures and 228; percentage of agreement and 229; Spearman-Brown coefficient and 230–4, 231–3 ratio data 3–4, 4 regression see multiple regression; simple regression reliability 219–20 reliability analysis: Cronbach’s alpha in 90; estimates and, factors affecting 244–5; instrument reliability versus research validity and 245; internal consistency measures and 219–27, 222–7; overview 245; rater reliability and 227–43, 228–43; reliability and 219–20; reliability coefficient and 220; in second language research 219; standard error of measurement and 243–4 reliability coefficient 220 reliability estimates, factors affecting 244–5 repeated-measures ANOVA: assumptions of 156–7; between-subjects contrasts and 151, 151, 163; describing 154; descriptive statistics in 155, 161, 162; effect size for 157–8; Mauchly’s Test of Sphericity and 161–2, 162; overview 164–5; paired-samples t-test and 154; post hoc tests and 157; in second language research 154–5, 154–5; sphericity and 156, 161–2; in SPSS program 158–64, 158–64; statistical significance and 157; within-subjects factors/contrasts and 162, 163, 163 repeated measures t-tests 93 R software 14 R-squared 210 R-value 210 samples 81–3 sample size 84–8, 88, 205 SAS (Statistical Analysis Software) 14 scatterplots: correlational analysis and 63, 64–7; in simple regression 201, 201; SPSS program instructions for 73, 74–7 432

Scheffé post hoc test 119–20, 126, 127, 140, 178, 178 screening data 15 selective sampling 82 sequential regression see hierarchical regression setting the alpha level 89–90 Sidak post hoc test 150, 156 significance level 83–4, 84 significance, statistical 2–3, 83–4, 86–7, 89–90, 140, 156–7 simple regression 200–3, 201 skewness statistics 40–3, 41–2 Spearman-Brown coefficient 230–4, 231–3 Spearman-Brown prophecy formula 221, 230 Spearman correlation 70–9, 72, 78 Spearman’s rho 67–8, 71, 182, 230 sphericity 156, 161–2, 175, 175 split-half reliability 221 spreadsheet, creating in SPSS 16–20, 16–19 SPSS program (IBM): analysis of covariance in 140–53, 141–52; analysis of variance in 122–7, 122–7; application of, in real study 24–7, 25–6; background information 14; bar graphs in 54–5, 55; case selection in 142, 143–53, 143–51; case summaries in, generating 20–2, 20, 21; chi-square test in 190–5, 191–5; Cohen’s d in 97; Cohen’s kappa in 235–8, 236–8; computing descriptive statistics in 48–54, 49–52; Cronbach’s alpha in 223–7, 223–7; data file in, saving and naming 22; data in, preparing 14–15; describing 14; descriptive statistics option in 52–4, 53–4; diagrams in 54–8, 55–8; Excel spreadsheet and, importing data from 15, 22–4, 22–4; frequency option in 48–52, 49–52, 54; graphs in 54–8, 55–8; hierarchical regression in 213–18, 213–18; importing data from Excel and 22–4, 22–4; independent-samples t-tests in 97–102, 98–101; intraclass correlation in 240–3, 240–3; Kendall’s tau in 68; Kruskal-Wallis test in 128–34, 129–33; Mann-Whitney U test in 108–11, 108–10; missing values in, assigning 47–8, 47–8; multiple regression in 206–12, 206–12; notes on, important 15–16; overview 14, 27; paired-samples t-tests in 102–4, 102–4; Pearson Product Moment in 78, 78, 79; pie charts in 56, 56; repeated-measures ANOVA in 158–64, 158–64; scatterplots in 73, 74–7; in second language research 14; Spearman-Brown coefficient in 231–4, 231–3; Spearman correlation in 76, 79, 78; 433

Spearman’s rho and 68; spreadsheet in, creating 16–20, 16–19; standard deviation in 38; statistical significance and 86; Test of Between-Subjects Effect 163; Tests of WithinSubjects Contrast 163; two-way mixed design ANOVA in 170–80, 170–80; value labels in, assigning 44–7, 45–7; variables in, computing 136–7, 136–7; Wilcoxon Signed-rank test in 112–16, 112–15; see also descriptive statistics in SPSS program standard deviation (SD) 38–9, 118, 243 standard error of measurement (SEM) 243–4 Statistical Package for Social Sciences program see SPSS program (IBM) statistical significance 2–3, 83–4, 86–7, 89–90, 140, 156–7 stratified random sampling 82 Tamhane T2 post hoc test 120, 126, 140, 178, 178 Test of Between-Subjects Effects 163 Test of English as a Foreign Language (TOEFL) 4, 58, 61–2, 83 Test of English for International Communication (TOEIC) 4 Test of English Pragmatics (TEP) 30, 126, 140, 206, 223 test item discrimination 68 test-retest reliability 221 theories 2 transforming data in real-life context 8–11, 9–11 t-tests: assumptions of 96; Cohen’s d in 88, 96–7; dependent 93; effect size for 96–7, 104; equal variances assumption and 96; independent-samples 93–4, 93, 96–102, 98–101, 106, 117, 121, 138; Levene’s test and 96; overview 104–5; paired-samples 93–5, 95, 102–4, 102–4; repeated measures 93; in second language research 92–3; steps for using 97 t-value 93, 103 two-dimensional chi-square test 185–9, 185–7 two-way analysis of variance 117 two-way mixed-design ANOVA: between-subjects factors/contrasts and 174, 174, 176, 176; descriptive statistics in 168, 168, 174, 175, 177, 177; Levene’s test and 175, 176, 178–9; Mauchly’s Test of Sphericity and 175, 175; overview 180–1; pairwise comparisons and 177, 177, 179; post hoc tests and 178–9, 178; pretest-posttest control-group design and 166, 167; results, written 180; in second language research 166–9, 167–9; in SPSS program 170–80, 170–80; univariate tests and 177, 177; within-subjects factors/contrasts and 166–7, 434

174–5, 174, 176, 178 type I error 90 type II error 90 univariate analysis of variance see analysis of variance (ANOVA) U-value 107 variables: confounding 135–9; dependent 7, 119–20; excluded 217, 218; factor 119–20; grouping 119–20, 128; independent 7–8, 119–20, 128; intervening 135–9; moderator 135–9; nominal, assigning values to 44–7, 45–7; outcome 119–20; predictor 201, 205, 209–10, 209; in quantification 2; SPSS program and computing 136–7, 136–7 VassarStats website 196–7, 196, 198 Wiseheart’s calculator 104 within-subjects factors/contrasts 162, 163, 163, 166–7, 174–5, 174, 176, 178 Yates correction 187, 194, 197 Z-value 107, 110–11, 114–15, 115

435