Conducting Quantitative Research In Education 9811391319, 9789811391316

Table of contents :
Contents......Page 5
1.1 What This Book Will Do......Page 9
1.2 The Importance of Engaging with Quantitative Research Methods and Analysis......Page 10
1.3 The Neglected Non-parametric Analysis......Page 11
1.4 Format and Suggested Engagement......Page 12
References......Page 13
2.2.1 The Research Process......Page 15
2.2.2 When Are Quantitative Methods Appropriate?......Page 18
2.2.3 What Types of Quantitative Measures Are Available to Me?......Page 19
2.2.4 In the Event of Writing Your Own Survey......Page 20
2.3.1 What Could My Setting and Sample Look Like?......Page 22
2.3.2 Dissemination Methods for Surveys and Other Quantitative Methods......Page 24
2.4.1 Pure Quantitative Research Designs......Page 26
2.4.2 Mixed Methods Research Designs......Page 29
References......Page 31
3.1 Age Appropriate Tools......Page 32
3.2 Satisficing......Page 34
3.2.3 No One Will Know What You Said......Page 35
3.3 The Importance of Piloting......Page 36
3.4.1 You Don’t Know What You Don’t Know......Page 37
3.4.2 You Want Your Data to Actually Be Reliable......Page 38
3.5 Ethics......Page 39
3.6.1 The Cover Letter......Page 41
3.6.2 Other Considerations......Page 44
References......Page 45
4.1 Data Types......Page 46
4.2 Samples......Page 48
Reference......Page 52
Chapter 5: Data Preparation......Page 53
5.1 Data Entry in SPSS......Page 55
5.2 Knowing Your Data......Page 57
5.3 Recoding and Recomputing Data......Page 65
Chapter 6: Analysis: Difference Between Groups......Page 71
6.1 Mann-Whitney U......Page 72
6.1.1 Assumptions......Page 73
6.1.2 Procedure......Page 76
6.1.4 Checking the Assumptions......Page 77
6.1.5 Descriptive Statistics......Page 79
6.2 Kruskal-Wallis......Page 81
6.2.1 Assumptions......Page 82
6.2.2 Procedure......Page 84
6.2.3 Output......Page 85
6.2.4 Checking the Assumptions......Page 89
6.3 Chi-square......Page 91
6.3.1 Assumptions......Page 92
6.3.2 Procedure......Page 93
6.3.4 Checking the Assumptions......Page 95
6.3.5 Report......Page 96
6.4.1 Assumptions......Page 97
6.4.2 Procedure......Page 99
6.4.3 Output......Page 100
6.4.5 Report......Page 101
6.5.1 Assumptions......Page 102
6.5.2 Procedure......Page 104
6.5.3 Output......Page 105
6.5.4 Checking the Assumptions......Page 107
6.6.1 Assumptions......Page 108
6.6.2 Procedure......Page 110
6.6.3 Output......Page 111
6.6.4 Checking the Assumptions......Page 112
6.6.5 Report......Page 116
7.1 Spearman’s Rho......Page 117
7.1.1 Assumptions......Page 118
7.1.2 Procedure......Page 119
7.1.3 Output......Page 120
7.1.4 Checking the Assumptions......Page 121
7.2 Kendall’s Tau......Page 126
7.2.1 Assumptions......Page 127
7.2.2 Procedure......Page 128
7.2.3 Output......Page 129
7.2.4 Checking the Assumptions......Page 130
7.3 Cramer’s V......Page 132
7.3.1 Assumptions......Page 133
7.3.2 Procedure......Page 134
7.3.3 Output......Page 136
7.3.4 Checking the Assumptions......Page 137
7.3.5 Report......Page 138
Chapter 8: Analysis: Regression......Page 139
8.1.1 Assumptions......Page 142
8.1.2 Procedure......Page 146
8.1.3 Output......Page 149
8.1.4 Checking the Assumptions......Page 150
8.2 Binomial Regression......Page 158
8.2.1 Assumptions......Page 159
8.2.2 Procedure......Page 161
8.2.3 Output......Page 163
8.2.4 Checking the Assumptions......Page 166
8.3 Multinomial Regression......Page 169
8.3.1 Assumptions......Page 170
8.3.2 Procedure......Page 171
8.3.3 Output......Page 172
8.3.4 Checking the Assumptions......Page 177
8.3.5 Report......Page 180
Chapter 9: Write Up and Research Translation......Page 182
9.1 The Plain English Report for Schools......Page 183
9.2 The Journal Article......Page 184
9.2.1 Good Fit......Page 185
9.2.2 Reputable......Page 186
9.3 Unique Challenges in Publishing Quantitative and Mixed-Methods Articles in education......Page 187
9.4 Thesis by Publication......Page 189
9.5 Plain English Dissemination......Page 191
9.6 Altmetrics and (Social) Media-Supported Dissemination......Page 194
9.7 The Conference......Page 195
References......Page 196
Chapter 10: Conclusion and Further Reading......Page 198
10.1.2 Readings About Experimental Methods Design......Page 200
10.1.5 Readings About Survey Design......Page 201
10.1.8 Readings About Reporting Educational Research......Page 202
Index......Page 204

Citation preview

Saiyidi Mat Roni  Margaret Kristin Merga  Julia Elizabeth  Morris 

Conducting Quantitative Research in Education

Conducting Quantitative Research in Education

Saiyidi Mat Roni • Margaret Kristin Merga Julia Elizabeth Morris

Conducting Quantitative Research in Education

Saiyidi Mat Roni School of Business and Law Edith Cowan University Joondalup, WA, Australia

Margaret Kristin Merga School of Education Edith Cowan University Perth, Australia

Julia Elizabeth Morris School of Education Edith Cowan University Mount Lawley, WA, Australia

ISBN 978-981-13-9131-6    ISBN 978-981-13-9132-3 (eBook) https://doi.org/10.1007/978-981-13-9132-3 © Springer Nature Singapore Pte Ltd. 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Contents

1 Introduction����������������������������������������������������������������������������������������������    1 1.1 What This Book Will Do������������������������������������������������������������������    1 1.2 The Importance of Engaging with Quantitative Research Methods and Analysis ����������������������������������������������������������������������    2 1.3 The Neglected Non-parametric Analysis������������������������������������������    3 1.4 Format and Suggested Engagement��������������������������������������������������    4 References��������������������������������������������������������������������������������������������������    5 2 Getting Started: What, Where, Why������������������������������������������������������    7 2.1 Introduction��������������������������������������������������������������������������������������    7 2.2 What Are Quantitative Methods All About and When Should I Use Them?��������������������������������������������������������������������������������������    7 2.2.1 The Research Process ����������������������������������������������������������    7 2.2.2 When Are Quantitative Methods Appropriate?��������������������   10 2.2.3 What Types of Quantitative Measures Are Available to Me?������������������������������������������������������������   11 2.2.4 In the Event of Writing Your Own Survey����������������������������   12 2.3 Where Can I Collect Quantitative Data and How Do I Go About It? ����������������������������������������������������������������������������   14 2.3.1 What Could My Setting and Sample Look Like?����������������   14 2.3.2 Dissemination Methods for Surveys and Other Quantitative Methods������������������������������������������������������������   16 2.3.3 How Many Times Should I Collect Data?����������������������������   18 2.4 Why Use Quantitative Methods? A Few Examples��������������������������   18 2.4.1 Pure Quantitative Research Designs ������������������������������������   18 2.4.2 Mixed Methods Research Designs����������������������������������������   21 2.5 Final Comments��������������������������������������������������������������������������������   23 References��������������������������������������������������������������������������������������������������   23 3 Conducting Research with Children and Students������������������������������   25 3.1 Age Appropriate Tools����������������������������������������������������������������������   25 3.2 Satisficing������������������������������������������������������������������������������������������   27 v

vi

Contents

3.2.1 This Is Not a Test������������������������������������������������������������������   28 3.2.2 I Want to Know What You Really Think, Not What You Think I Want You to Say������������������������������������������������   28 3.2.3 No One Will Know What You Said��������������������������������������   28 3.2.4 I Value Your Opinion������������������������������������������������������������   29 3.3 The Importance of Piloting ��������������������������������������������������������������   29 3.3.1 Pre-pilot��������������������������������������������������������������������������������   30 3.3.2 Pilot��������������������������������������������������������������������������������������   30 3.4 What Am I Looking for in a Pilot? ��������������������������������������������������   30 3.4.1 You Don’t Know What You Don’t Know������������������������������   30 3.4.2 You Want Your Data to Actually Be Reliable ����������������������   31 3.4.3 Your Reviewers Will Appreciate It ��������������������������������������   32 3.5 Ethics������������������������������������������������������������������������������������������������   32 3.6 Understanding and Engaging Schools����������������������������������������������   34 3.6.1 The Cover Letter ������������������������������������������������������������������   34 3.6.2 Other Considerations������������������������������������������������������������   37 3.7 Final Comment���������������������������������������������������������������������������������   38 References��������������������������������������������������������������������������������������������������   38 4 Data Types and Samples��������������������������������������������������������������������������   39 4.1 Data Types����������������������������������������������������������������������������������������   39 4.2 Samples ��������������������������������������������������������������������������������������������   41 Reference ��������������������������������������������������������������������������������������������������   45 5 Data Preparation��������������������������������������������������������������������������������������   47 5.1 Data Entry in SPSS ��������������������������������������������������������������������������   49 5.2 Knowing Your Data��������������������������������������������������������������������������   51 5.3 Recoding and Recomputing Data ����������������������������������������������������   59 6 Analysis: Difference Between Groups����������������������������������������������������   65 6.1 Mann-Whitney U������������������������������������������������������������������������������   66 6.1.1 Assumptions��������������������������������������������������������������������������   67 6.1.2 Procedure������������������������������������������������������������������������������   70 6.1.3 Output ����������������������������������������������������������������������������������   71 6.1.4 Checking the Assumptions����������������������������������������������������   71 6.1.5 Descriptive Statistics������������������������������������������������������������   73 6.1.6 Report������������������������������������������������������������������������������������   75 6.2 Kruskal-Wallis����������������������������������������������������������������������������������   75 6.2.1 Assumptions��������������������������������������������������������������������������   76 6.2.2 Procedure������������������������������������������������������������������������������   78 6.2.3 Output ����������������������������������������������������������������������������������   79 6.2.4 Checking the Assumptions����������������������������������������������������   83 6.2.5 Report������������������������������������������������������������������������������������   85 6.3 Chi-square ����������������������������������������������������������������������������������������   85 6.3.1 Assumptions��������������������������������������������������������������������������   86 6.3.2 Procedure������������������������������������������������������������������������������   87

Contents

vii

6.3.3 Output ����������������������������������������������������������������������������������   89 6.3.4 Checking the Assumptions����������������������������������������������������   89 6.3.5 Report������������������������������������������������������������������������������������   90 6.4 McNemar test������������������������������������������������������������������������������������   91 6.4.1 Assumptions��������������������������������������������������������������������������   91 6.4.2 Procedure������������������������������������������������������������������������������   93 6.4.3 Output ����������������������������������������������������������������������������������   94 6.4.4 Checking the assumptions����������������������������������������������������   95 6.4.5 Report������������������������������������������������������������������������������������   95 6.5 Cochran’s Q test��������������������������������������������������������������������������������   96 6.5.1 Assumptions��������������������������������������������������������������������������   96 6.5.2 Procedure������������������������������������������������������������������������������   98 6.5.3 Output ����������������������������������������������������������������������������������   99 6.5.4 Checking the Assumptions����������������������������������������������������  101 6.5.5 Report������������������������������������������������������������������������������������  102 6.6 Wilcoxon Signed-Rank ��������������������������������������������������������������������  102 6.6.1 Assumptions��������������������������������������������������������������������������  102 6.6.2 Procedure������������������������������������������������������������������������������  104 6.6.3 Output ����������������������������������������������������������������������������������  105 6.6.4 Checking the Assumptions����������������������������������������������������  106 6.6.5 Report������������������������������������������������������������������������������������  110 7 Analysis: Correlation������������������������������������������������������������������������������  111 7.1 Spearman’s Rho��������������������������������������������������������������������������������  111 7.1.1 Assumptions��������������������������������������������������������������������������  112 7.1.2 Procedure������������������������������������������������������������������������������  113 7.1.3 Output ����������������������������������������������������������������������������������  114 7.1.4 Checking the Assumptions����������������������������������������������������  115 7.1.5 Report������������������������������������������������������������������������������������  120 7.2 Kendall’s Tau������������������������������������������������������������������������������������  120 7.2.1 Assumptions��������������������������������������������������������������������������  121 7.2.2 Procedure������������������������������������������������������������������������������  122 7.2.3 Output ����������������������������������������������������������������������������������  123 7.2.4 Checking the Assumptions����������������������������������������������������  124 7.2.5 Report������������������������������������������������������������������������������������  126 7.3 Cramer’s V����������������������������������������������������������������������������������������  126 7.3.1 Assumptions��������������������������������������������������������������������������  127 7.3.2 Procedure������������������������������������������������������������������������������  128 7.3.3 Output ����������������������������������������������������������������������������������  130 7.3.4 Checking the Assumptions����������������������������������������������������  131 7.3.5 Report������������������������������������������������������������������������������������  132 8 Analysis: Regression��������������������������������������������������������������������������������  133 8.1 Simple Regression����������������������������������������������������������������������������  136 8.1.1 Assumptions��������������������������������������������������������������������������  136 8.1.2 Procedure������������������������������������������������������������������������������  140

viii

Contents

8.1.3 Output ����������������������������������������������������������������������������������  143 8.1.4 Checking the Assumptions����������������������������������������������������  144 8.1.5 Report������������������������������������������������������������������������������������  152 8.2 Binomial Regression������������������������������������������������������������������������  152 8.2.1 Assumptions��������������������������������������������������������������������������  153 8.2.2 Procedure������������������������������������������������������������������������������  155 8.2.3 Output ����������������������������������������������������������������������������������  157 8.2.4 Checking the Assumptions����������������������������������������������������  160 8.2.5 Report������������������������������������������������������������������������������������  163 8.3 Multinomial Regression��������������������������������������������������������������������  163 8.3.1 Assumptions��������������������������������������������������������������������������  164 8.3.2 Procedure������������������������������������������������������������������������������  165 8.3.3 Output ����������������������������������������������������������������������������������  166 8.3.4 Checking the Assumptions����������������������������������������������������  171 8.3.5 Report������������������������������������������������������������������������������������  174 9 Write Up and Research Translation������������������������������������������������������  177 9.1 The Plain English Report for Schools����������������������������������������������  178 9.2 The Journal Article����������������������������������������������������������������������������  179 9.2.1 Good Fit��������������������������������������������������������������������������������  180 9.2.2 Reputable������������������������������������������������������������������������������  181 9.2.3 Fast����������������������������������������������������������������������������������������  182 9.3 Unique Challenges in Publishing Quantitative and Mixed-Methods Articles in education����������������������������������������  182 9.4 Thesis by Publication������������������������������������������������������������������������  184 9.4.1 But Is This Right for Me? ����������������������������������������������������  186 9.5 Plain English Dissemination ������������������������������������������������������������  186 9.6 Altmetrics and (Social) Media-Supported Dissemination����������������  189 9.7 The Conference��������������������������������������������������������������������������������  190 9.8 Final Comment���������������������������������������������������������������������������������  191 References��������������������������������������������������������������������������������������������������  191 10 Conclusion and Further Reading ����������������������������������������������������������  193 10.1 Further Reading������������������������������������������������������������������������������  195 10.1.1 Readings About Mixed Methods Design����������������������������  195 10.1.2 Readings About Experimental Methods Design ����������������  195 10.1.3 Readings About Sample Size, Power and Effect size��������������������������������������������������������������������  196 10.1.4 Readings About Ethical Issues in Education and Social Science Research����������������������������������������������  196 10.1.5 Readings About Survey Design������������������������������������������  196 10.1.6 Readings About Validity and Reliability����������������������������  197 10.1.7 Readings About Quantitative Analyses������������������������������  197 10.1.8 Readings About Reporting Educational Research��������������  197 Index������������������������������������������������������������������������������������������������������������������  199

Chapter 1

Introduction

Welcome to Conducting Quantitative Research in Education! This book has been written to help support higher degree by research students, early career academics and life-long researchers who are looking to increase their capacity to both choose and use quantitative data collection and analysis in educational research.

1.1  What This Book Will Do By the end of this book, you should be able to design and undertake a research project which is entirely or partly reliant on quantitative research methods. You will have an understanding of some of the unique contextual factors involved in conducting quantitative research in educational settings, and you will be able to prepare your data for a broad range of analyses. We will walk you step-by-step through each of these stages, so even those of you who are absolutely new to quantitative design and data analysis (and slightly freaked out by the weird symbols and things) should be able to apply the ideas that we cover herein to your own research plans with a degree of confidence. Each of these analytical chapters will be self-contained, beginning with its own overview of a statistical procedure, a survey instrument, research question, hypothesis, SPSS procedures and output. Unlike many statistical support books, we consistently use a similar set of variables to illustrate the analyses in this book. We do this so you may focus your attention on understanding each analysis, as opposed to deciphering the purpose of different variables in different research designs. This approach also allows you to see multiple analyses conducted on the same variables, which helps to minimise confirmation bias where researchers unknowingly bias toward asserting their hypotheses while disregarding other possible explanations. In this book we use reading frequency as a dependent variable in multiple chapters. In doing so, you can see how reading frequency is influenced by gender, encouragement to read and other factors, and how we conducted analyses to justify conclusions. © Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_1

1

2

1 Introduction

We also include examples of how the results from each analysis could be reported, based on APA style, as this is most typically used in education. In addition, we provide advice about how to write up your work for publication, and highlight additional readings that you can draw upon if you find yourself wishing to delve more deeply into non-parametric statistics. You should be able to use this book to support educational research with samples derived from pre-school, primary, secondary and tertiary education contexts.

1.2  T  he Importance of Engaging with Quantitative Research Methods and Analysis Quantitative data analysis occupies an odd place in educational research. While there are many exceptional and well-known researchers working in this paradigm in education, and it is prevalent in some sub-fields such as educational psychology, we’ve noticed that research involving quantitative analysis is less common in other sub-fields. Some supervisors in education may not feel comfortable supporting students wishing to work with this approach due to lack of experience and knowledge with this method. We have even detected distinct nose-wrinkling at the mention of quantitative data analysis in some educational research contexts. Perhaps this is because the research questions we seek to explore, and the theories that underpin our research in education, are often better suited to a qualitative method. It could also be due to the tension between adherents of qualitative and quantitative approaches, which situates them as approaches in opposition. However, researchers in educational and social sciences are increasingly moving toward accepting a “legitimate complementarity of paradigms” (Salomon 1991, p. 10), with both paradigms seen as important and valuable when used in a way that is responsive to both the intended inquiry and context. For the contemporary researcher, we need to get over this unnecessary dichotomy. It is useful to be knowledgeable across the broadest range of methods possible, including quantitative methods, so that we can understand and interpret the literature that we review in our research areas. Even if the body of your current work uses qualitative methods, understanding of quantitative methods adds the option of going down this path for parallel or follow-up data collection on current research projects, enabling the interrogation of a research concern through multiple phases and the generalisation of exploratory research (Creswell and Plano Clark 2011). Knowledge of this area can also help us to support and/or supervise higher degree by research students wishing to use this approach. Having at least a basic understanding of how to employ a range of methods can enable you to take a creative and novel position on a research area. For example, where you have already done a great deal of qualitative research, you may now seek to create a survey tool based on some of the emerging themes to test their generalisabilty. There are a range of advantages to using a quantitative approach. Quantitative approaches are sometimes related to higher impact, both in terms of publication and translation, perhaps due to their broader capacity for generalisability. While many

1.3 The Neglected Non-parametric Analysis

3

areas that fall within the educational research umbrella are informed by theories and methods that align more readily with qualitative research, in general, high impact journals in education may favour quantitative data. For example, an analysis of papers in literacy journals found that quantitative research designs were far more prevalent (42%) that qualitative designs (6%) (Parsons et al. 2016). Likewise, at a US conference in 2017 Margaret discussed the possibility of submitting a paper based on qualitative findings into a leading education and technology journal. She was told that she would need to make a strong argument justifying this methodological choice, both in her cover page and article, for this approach to even be considered. There are some research contexts and outlets where qualitative data are not well-regarded. In a climate where adherence to a qualitative or quantitative approach can be somewhat intractable, many academics have a far stronger skill base in one area, and a potential deficit in the other. In writing this book, we are not suggesting that methods involving quantitative data analysis are superior to those involving qualitative research analysis. Indeed, all three of us have published articles using both methods, and mixed methods. Rather, we seek to promote the idea that methods need to be responsive to the research inquiry. Personally, we often like to use a mixed-methods approach as it tends to suit our research approach, questions and intent. This approach enables us to explore our areas of interest in both breadth and depth, and make the most of a very valuable resource: children’s time. We fall into the category of the “pragmatist”, as we argue “against a false dichotomy between the qualitative and quantitative research paradigms and advocate for the efficient use of both approaches” (Cameron 2009, p. 140), however only when appropriate. For example, Saiyidi and Margaret have used a single-stage mixed methods approach to paint a picture of a neglected demographic, the avid non-fiction book reader (Merga and Mat Roni 2018). We included quantitative survey items to enable generalisation of typical demographic characteristics in relation to reading volume and frequency and library usage. However, we needed to take a qualitative approach to explore respondents’ motivation to read and barriers to reading so that responses would not be limited by what we could conceive, and so that new understandings in this area could be generated. Using both approaches enabled a more holistic understanding in our area of inquiry, and it also enabled us to make the most of the rare opportunity of collecting data from a large sample, as the International Study of Avid Book Readers sought to capture data from over a 1000 readers in over 80 countries.

1.3  The Neglected Non-parametric Analysis Non-parametric statistics are commonly used in educational research, as the types of research samples we use often lead to data that do not fit a normal distribution, also known as a bell curve. This book is needed as while there are many statistics books out there, very few focus specifically on non-parametric data analysis, with non-parametric analysis the veritable wallflower at the statistics party. This

4

1 Introduction

comparative exclusion or underrepresentation can lead to a false impression of poor worth, whereas in reality, non–parametric methods offer many advantages (e.g., Moses 1952), and they can be particularly appropriate for the research samples in education. It also ignores the fact that parametric approaches are simply not always appropriate due to the nature of the sample being studied, and the constraints on data collection (e.g., AMC 2013). While positioned as less powerful and somehow inferior, nonparametric tests can be very useful where the research can only be designed to accommodate data structure that are ordinal, or scale data that violate a normality assumption, which is required for parametric tests. Non-parametric data are a staple of educational research, and as such, it is essential that educational researchers learn how to work with these data with confidence and rigour. They are also surprisingly easy to conduct once you learn how to prepare your data and conduct the analysis.

1.4  Format and Suggested Engagement We use SPSS to illustrate the data analysis procedures. While the latest version of SPSS (v25) changes the colour of diagrams of the default outputs, these are cosmetic changes with no apparent effect on the substance of the result. For example, in the e-Book version of this text, histograms are now in blue colour instead of greenish brown which had been the default colour for more than 10 years. Paper book versions are in grey-scale. Below is the new default output.

And here is the old default output (the one that we use).

References

5

You will probably find this book most useful if you try to use it, not just read it, so we challenge you to consider what uses you might make of it as you move through the book, applying your new knowledge to your own interests as soon as possible. Doing educational research can be quite lonely, which is one of the reasons we enjoy collaborating with like-minded individuals who we can learn from, learn with and support. We suggest that it might be fun to work through this book with a friend, research collaborator or a group of learners, so that you can nut out the ideas it contains together. We hope that you enjoy your foray into a deeper understanding of quantitative research.

References Analytical Methods Committee (AMC). (2013). An introduction to non-parametric statistics. Analytical Methods, 5(20), 5373–5374. Cameron, R. (2009). A sequential mixed model research design: Design, analytical and display issues. International Journal of Multiple Research Approaches, 3(2), 140–152. Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Los Angeles: Sage. Merga, M. K., & Mat Roni, S. (2018). Characteristics, preferences and motivation of avid non-­ fiction readers. Collection and Curation, 37(2), 50–59.

6

1 Introduction

Moses, L. E. (1952). Non-parametric statistics for psychological research. Psychological Bulletin, 49(2), 122–143. Parsons, S.  A., Gallagher, M.  A., & George Mason University Content Analysis Team. (2016). A content analysis of nine literacy journals, 2009–2014. Journal of Literacy Research, 48(4), 476–502. Salomon, G. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10–18.

Chapter 2

Getting Started: What, Where, Why

2.1  Introduction It is important to consider the many shades of grey when planning for research in the educational setting. In education there are many factors that impact on research outcomes: the classroom setting, teacher, students, past educational experiences, family experiences and values, and school culture (to name a few). All of these variables need to be considered when planning and conducting research. So, where do you start in planning to conduct quantitative research? This chapter will provide a basic introduction into planning for educational research using quantitative methods, from thinking about what quantitative methods are about, considerations for where it is appropriate to use quantitative methods and why they may be useful in achieving your research aims.

2.2  W  hat Are Quantitative Methods All About and When Should I Use Them? In order to answer this question, it is necessary to have some background knowledge on the research process, as this will determine if quantitative methods are appropriate for your research aims. It is also important to understand a bit about quantitative methods and how they can be used.

2.2.1  The Research Process Generally, the research process starts with a question. This question may relate to an observation that the researcher wants to understand or explain, or it may be that the researcher wants to apply other research in a new context, or adapt and/or extend © Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_2

7

8 Fig. 2.1  An overview of the research process

2  Getting Started: What, Where, Why

Identify the research topic

Review the literature

Identify research aims

Collect data

Analyse data

Report findings

earlier research to meet their own needs. This curiosity begins the research process, outlined below in Fig. 2.1. The chosen research question or topic generally seeks to address a ‘problem’ in practice or policy. The research problem can be affecting education on a small ­student or classroom scale, such as a researcher wanting to understand why some students in a class prefer practical tasks over written tasks; or it may be broader scale, such as how and why national educational policies are developed. Part of identifying the research topic is justifying the importance of the research. How will conducting the research improve education? Will it help teachers to support individual student’s needs? Will it change teacher practice? Will it change educational practice in a particular context? Will it help to change policy? Consider the outcomes of the research and how it can be translated into practice (note: considerations on research translation can be found in Chap. 9). In considering the purpose of the research, it is useful to review existing literature on the research topic. There is no point continuing with research if the answer is already out there! However, reviewing the literature may help you to identify what has and hasn’t been done in your topic. Perhaps you are interested in student selfefficacy and there has been a lot of research on undergraduate tertiary courses but not postgraduate courses as one of Julia’s doctoral students encountered (Norris et al. 2018), or your topic has been investigated in a different subject area to yours so you can research the transference of methods or instruments to new contexts (Morris et al. 2017). Reviewing the literature helps to situate your research within the global research community. As part of your review you will locate any published material that is relevant to your research topic, and summarise key ideas that are useful to your research. As the amount of published literature can be significant, it is important to be selective in your review: find experts on the topic who may have generated

2.2  What Are Quantitative Methods All About and When Should I Use Them?

9

important theories, find peer-reviewed journals where published materials have been reviewed for quality, and critique research to ensure it meets your needs. It is quite common for researchers to amend or revise their research topic at this stage, based on a deeper understanding of research that has already been conducted. Once the literature has been reviewed, it is appropriate to identify specific research aims. This involves taking the broader research topic or questions and refining them into research questions and hypotheses. Research questions differ from the general question or topic of research in that they are specific. These are the questions that the data are designed to answer. For example, a researcher may ask ‘What types of tasks do my students prefer?’ This question could lead the researcher to have students practise different task types and to rate their enjoyment and preference of each task. Research hypotheses outline an assumption that the research is designed to test. For example, a researcher may hypothesise that their students will prefer inquiry-­ based tasks. This hypothesis can be written in many forms; however, most researchers will write the null hypothesis and the alternative. Students will not prefer inquiry-based tasks. Null hypothesis (H0): Alternative hypothesis (HA or H1): Students will prefer inquiry-based tasks. Once research questions and/or hypotheses have been written it is necessary to determine the variables that need to be measured in order to answer them. There are three main types of variables that need to be considered: (i) Independent variable: A variable that causes something to happen. It is not uncommon to measure more than one independent variable in trying to establish a cause that is statistically significant. (ii) Dependent variable: A variable that is affected by the cause. This effect is what we are trying to measure and remains consistent throughout the study. Again, there may be more than one outcome or effect that you are trying to measure. (iii) Control variables: Variables that are not independent or dependent, but that may influence the research study. In a pure experiment design (which we will talk about later in this chapter) these are variables that we try to minimise so they have a limited effect on the independent and dependent variables. In a quasi-experimental design (or other designs) it may not be possible to control these variables, but it is important to acknowledge the effect they may have on the research outcomes. In educational research control variables could include the school culture, family values and experiences, students’ past educational experiences, and their personal interests. These factors are beyond your control as a researcher, but shape the lived experiences of your research participants. Once you have determined the variables in the research you need to collect data on these variables. Collecting data involves recruiting participants that meet your needs (i.e., addressing sampling criteria), then collecting information that answers your questions and sorting information for analysis. Collecting data is an exciting process, and in quantitative processes, generally involves administering surveys or compiling existing numerical information until there are enough data to reach

10

2  Getting Started: What, Where, Why

power. Achieving power is having a sample big enough for a statistical test to “detect an effect of a particular size” (Field 2013, p. 1015). It is a common method in determining sample size due to its robustness. Creswell (2014) describes the general process, as initially outlined by Lipsey in 1990: (i) Identify the level of significance required (p 1 for scale data.

Number of x?

x data type?

Statistical test

1

Ordinal

Spearman’s rho or Kendall’s tau

Scale

Pearson’s r

Nominal

Cramer’s V

Reference

45

Are you comparing two samples or groups with x (predictor) and y (outcome)?

y data type? Ordinal

Nominal

Independent

Related

Independent

Chi-Square

Pair-sample t-test*

t-test*

Mann-Whitney U

McNemar

Independent

Wilcoxon sign rank

Related

Related

Statistical test

Kruskal-Wallis (>2 groups)

Are the groups related or independent?

Scale

*Parametric test. Not covered in this book.

Reference Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G∗Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/brm.41.4.1149.

Chapter 5

Data Preparation

Once you have decided which test best suits your data and your analytic needs, it’s time to open up SPSS and prepare the data for testing. We will now give some advice about how to read your data in SPSS. SPSS has three major windows. These are the main window where you can click between data and variable views (.sav – for your dataset), an output window which, as it says, display output of selected command (.spv), and a less popular syntax window (.sps – a file containing a set of instructions used to run a test). When you start SPSS, the main and output windows are normally displayed. The syntax window only appears when you select it, or when you hit the Paste button in many dialog boxes.

© Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_5

47

48

5  Data Preparation

Whilst the main interest in many data analyses is the data view, the variable view is also useful. The variable view allows you to describe the data in a meaningful way. For example, when you type in 1, 2, 3, and so on as values for a given variable, you may want to define in the variable view what these numbers mean. The next section of this chapter explores the variable view in detail.

Syntax is the last window, which is less frequently used. One of the many advantages of syntax is that it can be separately saved (just like the output) and shared. As such, it can be easily used to run the same analyses on another set of data (as long as the other dataset has a similar structure, e.g., the variable name to be analysed).

5.1 Data Entry in SPSS

49

Once you are familiar with the windows in SPSS, you can now start entering your data. Ideally, you should start your data entry with variable view to describe your variables.

5.1  Data Entry in SPSS As mentioned in the previous section, you should start entering the data with descriptions in the variable view of the main window. Many online survey tools, such as Qualtrics and Survey Monkey, enable you to directly convert your survey findings into a .sav file which can then be uploaded into SPSS. However, it is also possible to manually enter quantitative data. When you do so, you should specify the name of the variable, type, label, values, and measure (i.e. variable type). Even when importing data, it is important to check the variable view as some meta data may not be correctly imported from the survey tool download (e.g., the variable type labels are often scale when you import the  data, even if your variable is dichotomous!).

The variable name can be typed in directly in the Name field. The same goes with Label. As for the Values field, you will need to enter one value at a time. For example, variable Q4.ReadFreq in the diagram has four values ranging from 1 to 4 (Never, Sometimes, Often and Every Day). In order to indicate which value represents what, you will need click the triple-dot icon in the Value field and enter each of the four values, as per below.

50

5  Data Preparation

If you choose to specify missing values, you can click the triple dots in the Missing column of each variable. The missing values are usually a result of nonavailable data or response. For example, some respondents in your sample choose not to answer certain questions in the survey. You can code this missing value with a number. Note that the number you use should be beyond your instrument’s scale. For instance, if you use a five-point Likert’s scale ranging from 1 to 5, the identification for missing value has to be anything below or above the scale range. In the example in the following diagram, we use 99 to identify a missing value. Note that we prefer to use 99 because our demographic data also include age. Although we could use 7 as a code for a missing value, this may be confused with age 7 in our demographic data. Of course, 99 would not be appropriate if we were surveying elderly Australians. While some researchers choose to leave out the missing value code, we do suggest that this should be recorded. This is because the recoding of a missing value could potentially be a study on its own. You can analyse the missing values to see if there is any peculiar pattern. A systematic missing values in the data can suggest an issue with the instrument that makes the respondents reluctant to respond. If you are at the pilot stage, you could rephrase or remove the question. If you are already collecting data you will keep consistent omissions in mind for future revisions and iterations of your survey tool, and you will include this a potential limitation in your study. You can even initiate a study to find out why the respondents choose not to answer that particular question(s) in your survey instrument.

5.2 Knowing Your Data

51

5.2  Knowing Your Data As with many things in life, if we want quality results, we need to input quality at the outset. Unfortunately, even when we are able to simply import a .sav file into SPSS that appears ready to go, we need to take the time to look closely at the data set, and where required, prepare it for analysis. As such, before any analysis is done, it is always a good practice to look closely at the data. This process is to eyeball data input which may have been wrongly keyed in, or have not been keyed in at all. A data point which sits beyond the expected range or is missing could be a result of error in data entry. If the error is genuine, this has to be corrected prior to your analytics to ensure that your statistical results are correct. For example, in reporting your demographic data of your respondents from school-aged children, you are expecting age to fall within 7–12 years, hence your mean (average) age is somewhere in that region. The average age is affected if there is an error at the data entry stage, for instance, when someone keys in 77 for age instead of 7. This value can skew the average age, making your summary of the demographic data incorrect. You can quickly check for missing values and errors in the data by using the Explore function in SPSS to find the maximum and minimum values for each variable. This will help you to discover if the data is within an expected range, or if there are missing or incorrect values. Another advantage of this Explore function is the two mean values that it produces. These are (overall) mean and 5% trimmed mean. The former is the average value of the variable and the latter is a recalculated mean with the bottom 2.5% and the top 2.5% of the recorded values left out (hence the word “trimmed”). We can compare these two values to see if there is a considerable difference. When you find a large difference, there is a strong indication of outliers present in your dataset. An outlier is a data point that is significantly far from the rest. This is normally more than

52

5  Data Preparation

3 standard deviations1 away from the mean. The value of 77 which is entered for age in the prior example could be a genuine value and not a data entry error. This could be a result of the survey instrument going to someone of that age. This of course is an outlier that should prompt you to remove the whole response from your dataset. The following diagrams show you how to explore your data and interpret the output.

1  Standard deviation (s.d.) is a measure of a dispersion. In a simple term, s.d. measures how large your data spreads away from the mean.

5.2 Knowing Your Data

53

Once the options are completed, hit OK on the Explore main dialog. You will then be presented with an output window.

54

5  Data Preparation

5.2 Knowing Your Data

55

56

5  Data Preparation

5.2 Knowing Your Data

57

58

5  Data Preparation

Fig. 5.1  A survey instrument measuring attitudes toward reading story books

In many cases of influential outliers, we could treat them by either trimming the dataset, i.e., remove the outliers from the dataset; or by using the Winsor method, which brings the outlier value to the nearest data point. The treatment of outliers is advisable, especially when the difference between 5% trimmed mean and mean is large. When you decide to treat the outliers either by trim or Winsor method, you should start with the furthest values. This is because once the furthest value is treated, and later you run the Explore function, the mean, the 5% trimmed mean, and standard deviation will be recomputed. This will give a different sets of results. Consequently, the next furthest value from the previous Explore run may no longer be flagged as a potential outlier. The data preparation also involves a step to check for monotone responses. These are responses which have no variations, such as when a respondent answers 5 for all questions on a five-point scale. Let’s have a look at a sample of survey questions in Fig. 5.1. There are four questions on attitudes toward reading measuring like, enjoyment, and easy dimensions. Question 4 which also measures like is specifically designed to control for acquiescence (familiarity) where the tone of the question is in reverse compared to question 1. For a genuine response, option 5 in question 4 should only be selected if option 1 is selected in response to question 1 because these two questions ask the same dimension  - like. However, if a respondent answers 5 for question 1 and 4, or for all questions in this block, the authenticity of the responses is very much questionable. In other words, we may want to disqualify this response set. An easy way to check for a monotone is to use Microsoft Excel. You can export or copy and paste the data from SPSS into an Excel file, and use an Excel variance function for each row to scan for responses (cases) with zero or negligible variance.

5.3 Recoding and Recomputing Data

59

In the following example, we illustrate how monotone responses are screened using Excel. In this dataset, there are 11 questions two of which are demographic (AGE and GENDER). All other questions tap into complexity domain (four questions), behaviour (two questions), and intention (three questions). These questions use a seven-point scale.

5.3  Recoding and Recomputing Data SPSS also allows data to be recoded or recomputed. In a recoding process, you can change (recode) the original value of a response with another value which conveys more meaning to the context of your study. For example, certain survey instruments include one question which is phrased in a negative tone, while the rest are in a positive manner to measure a single domain. The responses for the negatively-phrased question will have to be reverse coded so that it is consistent with other questions in the instrument. Have a look at these questions which responses are measured on a five-point Likert scale with 1 as strongly disagree and 5 strongly agree. Q1. I like reading books. Q2. I enjoy reading books. Q3. I do not feel good about reading books.

60

5  Data Preparation

These three questions attempt to tap into three dimensions of attitude toward reading – like, enjoy, and feel. The respondents are asked to rate between 1 and 5 whether they agree with the statement. While Q1 and Q2 are positively worded, Q3 is intentionally phrased in a negative manner. This is to provide a psychological break from a series of monotonous questions, making the respondent more alert. Given that all three questions are on a similar 1–5 scale (strongly disagree to strongly agree), the responses on Q3 will be so different from the first two. If we use all three questions as a composite variable, whether we total all scores or use the average (mean) of all three, the resulting score will be inaccurate unless we recode. This is because the responses for Q3 are on the opposite of Q1 and Q2. Responses for Q3 have to be reverse-coded, which means, all responses of 1 are to be recoded as 5, 2 as 4, and so on. This will make Q3 responses consistent with Q1 and Q2 where a higher value (4 or 5) will indicate a more positive response. As for recomputing, SPPS allows us to calculate a new variable based on the existing variables. For the three questions above, the variable that the study wants to measure is attitude toward reading. Attitude, however, is a latent (unobserved) variable which is derived from directly measured dimensions. In this case, those three questions tap into three separate dimensions of attitude. In order to arrive at one single variable representing attitude, we can total or average the score for all three questions (this is done after the reverse-coding process for Q3). This single variable can subsequently be used for later analyses, e.g., to investigate a correlation between attitude toward reading and performance in exam. Without this compute function, we may end up using either one of the three questions to use as a proxy for attitude. Let’s look at another example. In this study, respondents are asked to indicate whether they have any access to Kindle/iPad, mobile phone, and computer. These devices make up to three categories of access to electronic readers. The initial codes for access to devices are 1 for yes and 2 for no. They are also asked to rate their reading frequency (1 to 5 scale, with 1 being the least and 5 as the most frequent). When we want to run a correlation test between reading frequency and the number of devices accessible to the respondent, the three questions on access to devices will have to be summarised as a single question (i.e., variable). In this case, we use sum function in SPSS. Given that the initial codes for access to devices are 1 for yes and 2 for no access, we will arrive at an artificial total number. This is because 2 does not indicate the respondents have two accesses to Kindle/iPads, computers, or mobile phones. Rather, it represents no access. We therefore have to recode the these responses as 2 or 0 (shown in the diagrams below).

5.3 Recoding and Recomputing Data

61

62

5  Data Preparation

Once the recoding is done, we can now calculate how many devices the respondents have access to. We run Compute Variable command under Transform for this purpose.

5.3 Recoding and Recomputing Data

63

64

5  Data Preparation

In this chapter, you have been introduced to some preliminary steps you need to do before running the statistical analyses that can give you deeper insight into your data. Again, it is always a good idea to run the initial Explore function to get an overall view of your data. Where appropriate, you need to recode or transform the data in order to make sure the data is prepared and suitable for further analyses.

Chapter 6

Analysis: Difference Between Groups

A test of differences between groups is one of the most commonly used statistical procedures. For instance, we observe that student academic performance at one school may be better than at another. We can confirm whether this observation is true by using Mann-Whitney U to test if there is a significant difference in the student performance at these two schools. If the result is statistically significant, we can determine which school is currently outperforming the other. This category of test is relatively simple, yet it has profound effects on many other subsequent analyses. For example, when we distribute our survey instrument to collect data on attitudes toward reading, and we only receive a response rate of 40%, there is a possibility that those who choose not to respond may have different attitude toward reading when compared to the 40% who return the survey. If our response rate was 100%, the result of our tests on attitude could be different. In this case, the data with the absence of 60% responses may have an inherit non-response bias (NRB). We therefore need to test if NRB could be of concern in our study. In this situation, we would compare early and late responses (a proxy for non-response) to determine a possible presence of NRB. Usually, we could split the dataset into two groups chronologically; we could take all responses received in the first wave as early responses, and those that are received after a reminder is sent as late responses. We then compare the variables of interest from these two groups. Unless the test shows a statistically non-significant difference, the results of subsequent analyses are questionable if the NRB is not addressed. Other uses of test of difference between groups are in situations where: i. we want to see if there is any change in learning outcomes of the same respondents before and after an experiment for an experimental research design. This is called a related-sample test. ii. we want to compare if there is any change in learning outcomes of those who were given a treatment (e.g., new method of teaching) and those who were not. This is referred as an independent-sample test.

© Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_6

65

66

6  Analysis: Difference Between Groups

iii. we want to see if there is any difference between specific subgroups (e.g. male and female, early readers and adults, Australians and Malaysians). • Notes: i. Check the nature of your samples. If the same people are measured more than one time, e.g., before and after an experiment or treatment, this is called related-samples. Hence, different tests should be used. ii. If the respondents in the study are measured once, e.g., one group who has an access to ebook (let’s call this treatment group) and the other with only access to physical book (let’s call this our control group), then these are called independent samples. For this research design, the statistical tests are different to related-sample tests. Similarly, when we compare males and females, as the respondents are measured once, these are also referred as independent samples. iii. If we have more than two (independent samples) groups to compare, we normally begin with Kruskal-Wallis to see if there is a statistical difference between at least two groups. Then, we run multiple pair-wise comparisons, perhaps using Mann-Whitney U, to investigate which group(s) are different from each other. This subsequent tests are referred as post-hoc (a Latin word meaning “after this”). The p-values for the multiple pair-wise comparisons have to be adjusted. We normally call it Bonferroni correction or Bonferroni adjusted alpha. What this means is that the initial p-value (e.g., p = .05 for the initial Kruskal-Wallis test), has to be divided with how many subsequent pair-wise comparison tests made. The reason to use the Bonferroni correction is to reduce the likelihood of making a conclusion that there is a statistical difference between the groups, while in fact there is none (i.e., false positive). This is what we call a Type I error – finding something which actually is not there (statistically speaking). In short, if we do not apply the Bonferroni correction, which means reducing the p-value thresholds, the statistically significant result is probably due to chance rather than emanating from an actual phenomenon.

6.1  Mann-Whitney U Mann-Whitney U (MW), introduced by Mann and Whitney in 1947, is used to compare differences between two groups (e.g. private and public schools). For example, we want to see if girls visit libraries more often than boys. The dependent variable in this example is the number of library visitations. The dataset is split into boys and girls. Consequently, the respondents are only measured once, as the boys’ library visits cannot be the girls’ visits. The result, therefore, should point to whether our hypothesis that girls visit the libraries more frequently than boys, is true or otherwise, on a statistical ground.

6.1 Mann-Whitney U

67

The MW test compares the difference between two groups of an observed phenomenon. It does not indicate that one variable causes the other or that there is a correlation between the independent (grouping) variable (e.g. girls/boys) and the dependent (outcome) variable (e.g. library visitations). Similarly, if we use Mann-­ Whitney to investigate students’ proficiency in a subject offered through an online course and a face-to-face classroom (grouping variable), and we find that the proficiency score (outcome variable) of the online group is statistically better than the face-to-face, we should not make an over-arching conclusion that online delivery of the course can improve student performance. MW only provides indicative improvement of one group compared to the other, but it does not say the online course causes the improvement. As a researcher, we should offer other possible explanations for the improved score of the online group.

6.1.1  Assumptions i. There are only two groups to be compared. ii. The two groups are mutually exclusive, which means the respondents are only measured once. This is what we termed as independence of observations. iii. The dependent variable is ordinal (e.g., library visit, which is measured as less frequent, frequent, and very frequent), or scale (e.g., temperature which is measured in degrees Celsius or Fahrenheit). iv. The data distribution for both groups should look similar (i.e., same shape and spread). If the data distribution for both groups have the same shape and spread, we can use MW to compare the median. If, however, the data distribution shapes differ, we should use mean rank to interpret the MW result. The latter is almost always the case with real life data. Assumptions i, ii, and iii are in the research design. Only assumption iv can be observed using SPSS. Example 6.1 A study was conducted among young children to see if girls visit libraries more frequent than boys. A survey of 320 school-aged children was made where 302 useable returned survey sets were found to be useful for analyses.

68

6  Analysis: Difference Between Groups

6.1 Mann-Whitney U

69

70

6.1.2  Procedure

6  Analysis: Difference Between Groups

6.1 Mann-Whitney U

6.1.3  Output

6.1.4  Checking the Assumptions Note that only assumption iv (data distribution shape) is relevant to SPSS.

71

72

6  Analysis: Difference Between Groups

6.1 Mann-Whitney U

73

6.1.5  Descriptive Statistics Although the main analysis is to find the difference in the library visit frequency between girls and boys, it is also a good practice to have a quick look at descriptive side of the sample. In some occasions, descriptive statistics can point to possibilities that can explain the result. In this example, we can see that the percentage of girls who visit the library is comparatively higher. The statistical test however, suggests that this is not necessarily the case. We illustrate this phenomenon in the subsequent section.

74

6  Analysis: Difference Between Groups

6.2 Kruskal-Wallis

75

6.1.6  Report

6.2  Kruskal-Wallis Kruskal-Wallis (KW) test is a non-parametric equivalence to analysis of variance (ANOVA), which is a well-known parametric test tool. These tests are used to investigate statistical differences where there are more than two independent groups to compare. KW is also good to use when the dataset violates assumptions for a parametric test, for example, the normal distribution assumption that is required for ANOVA test. Similar to MW, where two groups are compared, a statistical significant KW test does not imply there is a correlation between variables, nor a causation. The test only finds differences among the groups. For example, we want to compare student academic performance from three schools, each in a different geographic context (metropolitan, rural, remote). If KW analysis shows there is a significant difference,

76

6  Analysis: Difference Between Groups

this does not mean the student performance is correlated with geographic context. The test simply says there is at least one school statistically differs from the other. KW test is usually used in conjunction with MW. When KW results show significant difference, pair-wise comparison with adjusted alpha (also known as Bonferroni correction) is used as post-hoc analysis to determine which group(s) differ. In the student performance and school location example, we need to run three pair-wise comparisons through MW test if the KW test is statistically significant. The pair-­ wise should be designed as follows: i. Metropolitan vs rural ii. Metropolitan vs remote iii. Rural vs remote This post-hoc test allows use to check which of the three schools is different in terms of their student academic performance. Let’s assume the MW tests are as follows: The result in Table 6.1 indicates only student performance from the school in the remote area is significantly different (p < .001 for pairs ii and iii). Pair i does not differ significantly, suggesting that the student performance from the schools in these two areas are similar. We should note here that the difference is not caused by the location. Rather, this could be attributed by the level of access to learning materials, as metropolitan and rural schools may typically have more libraries. A range of additional factors could also contribute, so we cannot claim causation based on this result alone.

6.2.1  Assumptions i. There are more than two groups to compare. ii. The two groups are mutually exclusive, which means the respondents are only measured once. This is what we term as independence of observations. iii. The dependent variable is ordinal (e.g. library visit which is measured as less frequent, frequent, and very frequent), or scale (e.g. temperature which is measured in degrees Celsius or Fahrenheit). iv. The data distribution for both groups should look similar (i.e. same shape and spread). If the data distribution for both groups have the same shape and spread, we can use Kruskal-Wallis to compare the median. If, however, the data

Table 6.1  Post-hoc analysis with Mann-Whitney U

Pair i. Metropolitan-rural. ii. Metropolitan-remote. iii. Rural-remote.

MW result p > .05 p < .001 p < .001

6.2 Kruskal-Wallis

77

d­ istribution shapes differ we should use mean rank to interpret the KruskalWallis result. The latter is almost always the case with real life data. Example 6.2 A survey was conducted among parents to investigate the influence of socio-­ economic factors on reading frequency among young children. A total of 184 useable responses were found to be suitable for analyses.

78

6.2.2  Procedure

6  Analysis: Difference Between Groups

6.2 Kruskal-Wallis

79

6.2.3  Output

Assuming that KW test shows significant difference, p < .05, the following post-hoc pair-wise comparison with MW can be carried out. i. Low vs Middle ii. Low vs High iii. Middle vs High

80

6  Analysis: Difference Between Groups

6.2 Kruskal-Wallis

81

82

6  Analysis: Difference Between Groups

6.2 Kruskal-Wallis

6.2.4  Checking the Assumptions

83

84

6  Analysis: Difference Between Groups

6.3 Chi-square

85

6.2.5  Report

6.3  Chi-square MW is suitable to compare groups when your data are at least ordinal. As we previously mentioned, ordinal means the values in your dataset convey an order. For example, observations with 1, 2, 3, and so on indicate an order of values, which means 2 is larger than 1. However, in a case where the numerical data is representative of an identification of a group rather than value, the chi-square test (χ2) is more appropriate. Let’s look at this example: you want to know if children’s preference of book genre is related to gender. You identify girls as 1 and boys as 2 for gender, and 1 for fiction and 2 for non-fiction. The type of books and the gender are both categorical variables, where 1s and 2s only identify the groups but have no orderly value. For

86

6  Analysis: Difference Between Groups

this, we have to use Pearson’s chi-square test of contingencies, which is available under crosstab in SPSS, to investigate if the preference of book genre is contingent to gender.

6.3.1  Assumptions i. Each participant is measured only once. In other words, the groups are independent of each other. If your groups are related, as you measure the same people twice, McNemar test is more appropriate, or Cochran’s Q in the case when measurements of the same individuals are taken more than twice. ii. The variable of main interest is nominal (i.e. categorical). iii. No more than 20% of expected frequencies in each cell is lower than five. For example, if you have two groups of girl and boy (gender), and the other variable is book type (e.g., fiction and non-fiction), there are four cells in this study design (2 x 2). All four cells should not have less than five cases. We explain this further in Sect. 6.3.4. Example 6.3 A survey was conducted among young children to investigate if preference for book genre is related to gender. A total of 165 useable responses were found to be suitable for analysis. As the group surveyed are young children who may struggle with the terms “fiction” and “non-fiction”, fiction is referred to as “story books”, and non-­ fiction is referred to as “books about information and facts”.

6.3 Chi-square

6.3.2  Procedure

87

88

6  Analysis: Difference Between Groups

6.3 Chi-square

89

6.3.3  Output

6.3.4  Checking the Assumptions Assumption i and ii are built into the study design. Only assumption iii can be checked through SPSS. The Chi-Square Tests table in the output indicates whether the expected frequencies is less than five.

90

6.3.5  Report

6  Analysis: Difference Between Groups

6.4 McNemar test

91

6.4  McNemar test McNemar test is used in situations where you have two related samples. In other words, when you measure a respondent at two intervals. For example, you organise a workshop on statistical analysis for a group of new research students, and you want to know if the workshop is effective in reducing the students’ anxiety level when it comes to data analysis for their research project. For this purpose, you send out a survey instrument to all registered participants at the start of the workshop and at the conclusion of program. The results can be analysed to see if their anxiety level is different. In situations where a related sample is required, it is always good practice to ensure that the instrument is designed in a way that the first and the second measurement can be tracked to the same respondent. One of the simplest ways is to ask for the last few digits of the respondent’s mobile number to be written on both instruments. This approach allows the researchers to determine which of the second responses correspond to which earlier data, while at the same time preserving the anonymity of the respondents.

6.4.1  Assumptions i. Your outcome (dependent) variable is dichotomous or binary. For example, student performance is measured in pass/fail, and anxiety is measured as yes/no. ii. The same individuals are tracked at two different measurement points, e.g., before and after a treatment. Example 6.4 A series of workshops on data analysis was organised for new research students. One of the objectives of the workshops was to address the students’ anxiety when it comes to using statistical procedures to analyse their research data. For each workshop run, the students were given two separate surveys, one at the start of the workshop, and another a few weeks after the workshop.

92

6  Analysis: Difference Between Groups

6.4 McNemar test

6.4.2  Procedure

93

94

6.4.3  Output

6  Analysis: Difference Between Groups

6.4 McNemar test

95

6.4.4  Checking the assumptions The two assumptions for McNemar test are addressed through the design of the measurement instruments and the way the data are collected.

6.4.5  Report

96

6  Analysis: Difference Between Groups

6.5  Cochran’s Q test Cochran’s Q is used when you have more than two related samples with a binary (dichotomous, i.e. yes/no, pass/fail) outcome variable. More than two related samples mean the same respondents are measured more than twice (e.g. before, during, and after an experiment). For example, if you want to find out if the introduction of a new teaching approach has an influence on student anxiety (measured as high/ low) in learning, you could design the study to collect the data on the anxiety before the introduction, during the roll out, and a few months after the intervention ends. This design will give a better result as it tracks the progression of the anxiety level of the same subjects over a period of time. Note that, in this study design, the participants should be assigned with a unique identification to track their responses at the three different measurement points. This is normally referred as a longitudinal study. In many cases, this study design is not possible due to time and resourcing constraints, in which case some researchers opt for a cross-sectional survey, that is, they may measure the anxiety level after the introduction of the new teaching method, and compare this with a control group which is not influenced by or introduced to the method/treatment. For this scenario, the two groups (treatment and control) are considered independent samples. In this situation, we need to forgo Cochran’s Q and use chi-square test of contingency (Sect. 6.3) instead.

6.5.1  Assumptions There are two assumptions in Cochran’s Q, both of which have to be addressed at the design stage of a study. i. Your outcome (dependent) variable is dichotomous or binary. For example, student performance is measured in pass/fail, and anxiety is measured as high/low. ii. There are more than two related-sample groups. This is where the same individuals are tracked at three or more different measurement points, e.g. before, during, and after a treatment. Example 6.5 Students were reported to have high anxiety in the classroom while studying a particular unit, which can negatively affect their learning experience. A new approach to teaching was piloted and its effectiveness was monitored. The students were given three separate surveys, one before the rollout of the new teaching approach, one during and a final survey few weeks after the conclusion of the semester.

6.5 Cochran’s Q test

97

98

6.5.2  Procedure

6  Analysis: Difference Between Groups

6.5 Cochran’s Q test

99

6.5.3  Output

As the test shows there is a significant difference, we need to run post hoc multiple pair-wise comparison tests to find out at which level the anxiety actually differs. We use McNemar procedure for these related-sample tests.

100

6  Analysis: Difference Between Groups

6.5 Cochran’s Q test

101

6.5.4  Checking the Assumptions There are two assumptions in Cochran’s Q, and these are addressed in the study design. The first assumption is the outcome variable has to have only two options. In this example, we have low anxiety and high anxiety which are coded as 1 (low anxiety) and 2 (high anxiety). The respondents are also tracked at all stages of data collection through a unique identification (the three digits of their mobile phone number) to ensure that the related sample assumption is met.

102

6  Analysis: Difference Between Groups

6.5.5  Report

6.6  Wilcoxon Signed-Rank The Wilcoxon sign rank test is similar to McNemar test. Both methods test a variable for differences between related groups, which means the respondents are measured twice. However, the McNemar test is suitable when your variable is dichotomous (i.e. a variable which only has two outcomes, such as yes/no). When your variable is ordinal (e.g. reading frequency on a scale of 1 to 5), and the samples are related (e.g. the same individual is measured twice), you should consider using Wilcoxon.

6.6.1  Assumptions i. Your outcome (dependent) variable has to be at least ordinal. This means the values carry an order or a rank. For example, reading frequency is measured on a 1 to 5 scale with 1 being least frequent and 5 as being most frequent.

6.6 Wilcoxon Signed-Rank

103

ii. The same respondents are measured at two points of measurement. For instance, each respondent provides responses before and after treatment or intervention. iii. The difference between pre- and post-intervention scores are roughly symmetrically distributed, which approximately resembles a bell shape. We use Explore function in SPSS to plot a histogram to visually inspect if this assumption holds. A histogram is like a bar chart. We illustrate this in Sect. 6.6.4. Example 6.6 There is concern among parents and teachers that young children read less frequently. Several local libraries ran a campaign through a series of programs to encourage reading in young people. Prior to the intervention (i.e. the campaign), visitors to the libraries were invited to participate in the study. Several weeks after the campaign ended, the participants were contacted and requested to indicate their recreational reading frequency.

104

6.6.2  Procedure

6  Analysis: Difference Between Groups

6.6 Wilcoxon Signed-Rank

6.6.3  Output

105

106

6  Analysis: Difference Between Groups

6.6.4  Checking the Assumptions Assumption i and ii are built into the study design. Only assumption iii, score difference distribution, needs closer inspection using SPSS. There are two stages to check whether this assumption holds – first, calculate the difference between pre and post scores; and second, run Explore in SPSS to plot a histogram of the difference scores distribution. If the distribution is approximately symmetrical, the result from Wilcoxon test is said to be reliable. The following steps illustrate how to create a new variable – the difference score. After the new variable is created, we can plot the distribution.

6.6 Wilcoxon Signed-Rank

107

Once the new variable (difference scores) is created in the data view, we can run Explore to plot a histogram to visually inspect the distribution.

108

6  Analysis: Difference Between Groups

6.6 Wilcoxon Signed-Rank

109

110

6.6.5  Report

6  Analysis: Difference Between Groups

Chapter 7

Analysis: Correlation

Correlation indicates how two variables move together, and how strongly one variable moves in relation to other variable. For instance, a store owner observes that sales of soccer balls and sunglasses over a given period tend to increase and decrease together. When a correlation test is run and it is found that the correlation is statistically significant and strong, we can conclude that both items move together. This finding allows the store owner to manage the inventory level of sunglasses based on the trend in the sale of balls, or vice versa. Note that a correlation does not imply a causation. It is an indicative movement of two variables. In the case of the sales of balls and sunglasses, these values correlate, but the correlation does not mean that any increase in the sales of ball causes similar increase in the sales of sunglasses. The sales figures of these items happen to move in the same direction as each other. It does not indicate that people need sunglasses because they want to play soccer. Perhaps the cause for the increase in the sales of these two items is good weather allowing more people to play soccer and requiring the use of sunglasses to protect their eyes. After all, it could be disastrous to play soccer wearing a pair of sunglasses. In this section, we introduce two non-parametric tests of correlation. These are Spearman’s rho and Kendall’s tau.

7.1  Spearman’s Rho Spearman’s rho measures an association of two variables, both of which are ordinal, for example, student in year 1, year 2, and so on. As previously explained, ordinal data are a type of data for which value has an orderly rank. If your variables are both continuous (i.e. interval or ratio, such as time and weight), and they are normally distributed, you may want to use Pearson correlation instead. Pearson correlation is a parametric equivalent of Spearman’s rho.

© Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_7

111

112

7  Analysis: Correlation

7.1.1  Assumptions i. The variables are at least ordinal. In other words, Spearman’s rho can only be used if your data is ratio, interval, or ordinal. The test is not suitable for nominal (categorical) data. The hierarchy of data starting from the highest order is ratio, followed by interval, ordinal, and lastly nominal. We outline the types of data in Sect. 4.1. ii. There is a monotonic relationship between the variables. Put simply, if you have variable x and y, these variables should move in a straight line (also called l­ inear). You can use a simple scatter plot to check if this is true for your data. A U-shape or an inverted U-shape relationship is not monotonic. Example 7.1 A study was conducted to see if access to devices with eBook reading capability is associated with general reading frequency in young children. A survey of 800 school-aged children was made and analysed.

7.1.2  Procedure

114

7.1.3  Output

7  Analysis: Correlation

7.1  Spearman’s Rho

115

7.1.4  Checking the Assumptions In our case, only assumption ii, that the variables moves in one direction, can be checked with SPSS. We can check this assumption using a scatter plot. A scatter plot is a diagram that plots data points of one variable against the other on an x-y plane. Assumption ii holds when the plotted of data points of these two variables roughly resemble an upward or downward trend but not both at the same time. We illustrates these trends in Figs. 7.1, 7.2, 7.3, and 7.4. Assumption i is related to the variable type, which is incorporated into the research design and research instrument design.

116

7  Analysis: Correlation

7.1  Spearman’s Rho

117

118

Fig. 7.1  An inverted u-shape scatter plot.

Fig. 7.2  A u-shape scatter plot

7  Analysis: Correlation

7.1  Spearman’s Rho

Fig. 7.3  Both variables move upward

Fig. 7.4  As x increases, y decreases

119

120

7  Analysis: Correlation

7.1.5  Report

7.2  Kendall’s Tau Kendall’s tau is similar to Spearman’s rho. It is a nonparametric equivalent to Pearson’s correlation as it also tests for a bivariate correlation (i.e. a correlation between two variables). Kendall’s tau is particularly useful when the data do not meet assumption(s) required by a parametric test, such as the assumption of normal distribution. As previously explained, normally distributed data has a distribution histogram that resembles a bell shape. You can also test for normality assumption using Shapiro-Wilk or Kolmogorov-Smirnov test under Explore function in SPSS. We discuss these methods in Sect. 5.2. As in other tests for correlation, a statistically significant Kendall’s tau does not convey a causation. One cannot assert or prove that x causes y with a correlation test. Similarly, the significance and strength of correlation from the Kendall’s tau procedure only suggests the tendency of variable x and y to move together either in a similar or opposite direction.

7.2  Kendall’s Tau

121

7.2.1  Assumptions i. The variables are at least ordinal. This means you can only use this test for ratio, interval, and ordinal data, but not for nominal variables. ii. There is a monotonic relationship between the variables. Put simply, x-y move in a linear pattern (i.e. a straight line). You can use a simple scatter plot to check if this is true for your data. A U-shape or an inverted U-shape relationship in not monotonic. See section Figs. 7.1, 7.2, 7.3, and 7.4 to find out what the scatter plots look like and how to interpret the pattern of your data. Example 7.2 A study was conducted with young children to see if age is associated with general reading frequency. A  total of 800 school-aged children (8 to 12 years old) was surveyed.

122

7.2.2  Procedure

7  Analysis: Correlation

7.2  Kendall’s Tau

7.2.3  Output

123

124

7.2.4  Checking the Assumptions

7  Analysis: Correlation

7.2  Kendall’s Tau

125

126

7  Analysis: Correlation

7.2.5  Report

7.3  Cramer’s V Spearman and Kendall tests are suitable when your data are ordinal, interval, or ratio (see our discussion on data types in Sect. 4.1). However, if your variables of interest are categorical (nominal), you need Cramer’s V to test for correlation. This is similar to Pearson’s chi-square contingencies test which is discussed in Sect. 6.3, but the Cramer test gives you an index of strength of association between the variables. Cramer’s V not only tests for a significant correlation between the variables, it also reveals how strong the correlation is. A value closer to 1.0 (such as .90) is considered a strong correlation, while a value closer to 0 suggests a weak association. While this sounds a straightforward explanation, the interpretation is slightly different from that of Spearman’s and Kendall’s. This is because the categorical data that we analyse using Cramer’s V do not have an order of magnitude. The values in the data are used to identify the category. For example, we assign 1 and 2 to represent students from public and private schools respectively; and we also use 1 and 2 to identify their reading materials preference, either fiction or non-fiction. The numbers represent the groups not magnitude. Therefore, a significant correlation means the students from private schools (value = 2) prefer non-fiction reading materials (value = 2), and vice versa.

7.3 Cramer’s V

127

7.3.1  Assumptions i. The two variables are categorical. ii. The observations are independent of each other. In other words, each respondent is measured only once. Example 7.3 A survey was conducted among young children to investigate if type of books is related to library visit pattern. A total of 902 useable responses were found to be suitable for analysis.

128

7.3.2  Procedure

7  Analysis: Correlation

7.3 Cramer’s V

129

130

7.3.3  Output

7  Analysis: Correlation

7.3 Cramer’s V

131

7.3.4  Checking the Assumptions There are two assumptions in Cramer’s V, and these are addressed by the design of the study.

132

7.3.5  Report

7  Analysis: Correlation

Chapter 8

Analysis: Regression

Regression is used to explain variations in an outcome variable, y, accounted by an explanatory (predictor) variable, x. In other words, how much can one phenomenon be explained by a given variable or a set of variables. Say for example, we hypothesise that access to reading materials (x) can increase reading frequency (y) among young people. Using regression, we can analyse how much variability in reading frequency is “shaped” by the level of access to reading materials. Generally, regression is (almost) always used to predict an outcome given one or more predictors. While correlation allows us to examine how two variables move together, regression presents an opportunity to anticipate which predictor variables significantly affect the outcome, and how strong each predictor exerts its influence on the outcome variable. For example, reading frequency in young children might be modelled as being partly influenced by parental encouragement (x1), school support (x2), and availability of reading materials (x3). In this situation, the outcome variable, y, is reading frequency, and the predictor variables are the influential factors, x1, x2, and x3. If the model in the example above is to be written in a standard regression equation, this can be expressed as below. Although the Greek notations may look intimidating, don’t panic, they are there to represent numbers. Each of these notations is explained right after the equation. In addition, we include Fig. 8.1 to illustrate this further. y = a + b1 x1 + b 2 x2 + b 3 x3 + e

Where,

y = outcome variable, reading frequency. α = intercept, y – value when all predictors are zero. Simply put, this is a theoretical value which does not necessarily exist in the real world. But this value has to be there in the equation for the regression line to be properly drawn.

© Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_8

133

134

All dots are actual (observed) values

8  Analysis: Regression

Residual, ε Regression equation

Regression line, also known as best-fit line. Intercept (constant), α

Fig. 8.1  Simple regression

β1 = regression coefficient for parental encouragement, x1. In its simplest form, this is the slope of the regression line for predictor, x1. β2 = regression coefficient for school support, x2. This is the slope of the regression line for predictor, x2. β3 = regression coefficient for reading material availability, x3. This is the slope of the regression line for predictor, x3. ε = error term. This is the error of approximation or prediction. It is not uncommon to have some data points which lie above or below the regression line. The difference between the actual data points and those predicted by the regression line is the error. Note that this is a multiple regression model (or equation). This is because we have multiple predictors in the model. In a simple regression model in which there is only one predictor, a typical model can be expressed as: y =a + bx +e

Where,

y = outcome variable. α = intercept, y – value when all predictors are zero. β = regression coefficient for predictor variable, x; the slope of the regression line for predictor, x. x = predictor variable. ε = error term. Using the simple regression, we can visualise the simple regression model as in Fig. 8.1. In this scenario, we want to investigate the effect that the level of reading encouragement (x) has on reading frequency (y) among young people. Reading

8  Analysis: Regression

135

encouragement and reading frequency are measured using 5-point scales through a survey. The data points are plotted using a simple scatter plot with encouragement on x-axis and reading frequency on y-axis. We later ran a simple regression that results in the equation y  =  1  +  .63x (we explain this in detail below the figure). In this equation we know that the intercept, α = 1, and the coefficient, β = .63. Based on this result, a regression line is later drawn on the plane as shown. This is the best-fit line with a slope of .63 (i.e., the coefficient). What this line represents is the predicted values of y for a given value of x. Let’s break this down to a set of simple chunks. We are given this regression equation: y = 1 + .63 x

Where,

y = reading frequency measured on a 5-point scale where 1 means does not read and 5 means read daily; x = reading encouragement received, which is measured as how many people encourage the respondent to read. Therefore, if one respondent reports there are two people who encourage him or her to read (x), his or her reading frequency (y) is predicted to be:



y = 1 + .63 ( 2 ) y = 1 + 1.26 y = 2.26

The value of 2.26 is the predicted value on the regression line drawn in Fig. 8.1. However, if we look closely at the figure, we find that y value is 3 when x is 2. The difference between the actual (also called observed) value and the predicted value on the regression line is known as error or residual, ε. Now let’s assume we calculate all the differences (errors, residual, ε), square each of them, and later sum them, we will arrive at a total called sum of squares. Using the same data plot in Fig. 8.1, we draw another line and repeat the process to arrive at the sum of squares for each line, we can then compare which sum of square is the smallest. This process is a method that gives rise to one notable aspect of the chart in Fig. 8.1 – the best-fit line does not run through most of the data points. This is because regression, hence the best fit-line, minimises the square differences (ε) of the observed data and its corresponding predicted value on the regression line. Another characteristic of regression is that it allows researchers to estimate the extent to which variations in y can be explained by predictor x. In the reading frequency example, we can determine the extent to which variations in reading are caused by the level of encouragement received. This explanatory variation is termed as R2 and can be illustrated in a simple Fig. 8.2. R2 is the area where the predictor variable, x overlaps with the outcome variable, y.

136

8  Analysis: Regression

Fig. 8.2  Coefficient of determination, R2 Reading frequency, y R2

Encouragement, x

In this case, we estimate R2 is about .14, which means 14% of the variations in reading frequency can be explained by encouragement. Hence, 86% of the variations in reading frequency can be explained by other predictors which are not in the regression model. Looking at this example, we can clearly see that the higher the R2, the better the regression model, because there is a larger overlapping area between reading frequency and the explanatory variable. As a general rule, R2 of .19 is usually considered as weak, .33 as average, and .67 as substantial. As R2 is also considered as the goodness of fit of a given regression model, there is a tendency to include many predictors in the model despite some of these predictors have weak theoretical underpinnings. R2 tends to increase as the number of predictors in the model increases. Therefore, it is always a good idea to check and report adjusted R2 together with the normal R2. Adjusted R2 controls a spurious increase in R2 with penalties on variables that add little or no statistical value in the model. In short, adjusted R2 gets smaller as the number of non-significant predictors are added into the model. A huge difference between adjusted R2 and R2 should warrant a further investigation of the efficacy of the model and the predictors.

8.1  Simple Regression A simple regression only has one predictor (explanatory) variable, x and a single outcome variable, y.

8.1.1  Assumptions i. There is a linear relationship between variables. ii. The variables are normally distributed. Kolmogorov-Smirnov and Shapiro-Wilk can be used to check for normality. A significant Kolmogorov-Smirnov or Shapiro-­Wilk (p < .05) is an indication of non-normal data distribution.

8.1  Simple Regression

137

Alternatively, visual inspection using Q-Q plot can also be used to determine if the data are approximately normally distributed. The data are approximately normally distributed if the data plots cluster tightly along the diagonal line. Both visual inspection and statistical test approaches are discussed in Sect. 5.2. iii. Homoscedasticity. Residual (error) is assumed to be the same across predicted values. If you look at Fig. 8.1, the distance, indicated as ε, between the actual (observed) data point and the line has be relatively similar across the regression line for all values on the x-axis. This can be checked with a scatter plot as explained on page 151 (a range of plot types are also shown on p. 20). iv. Normality of residual. Residuals are the differences between predicted and the observed values. We illustrate this in Fig. 8.1. This assumption can be visually inspected using Normal P-P plot of regression standardised residuals (also known as errors) in SPSS. v. Outliers. Outliers are data points that are more than three standard deviations away from the mean. Treat the outlier if the data point is influential, i.e., significantly changes the regression slope. We explained how to treat an outlier on pages 51–58. vi. In the case of multiple regression (more than one predictors in the model), check for multicollinearity. Multicollinearity is when at least two predictors in the model correlate highly with each other. High predictor-predictor correlation (r > .85) results in an unstable regression model – in short, the result is questionable. This means we cannot determine with a great certainty which predictor causes the outcome. For example, we want to investigate students’ academic performance (outcome variable, y) by taking into account their family income (as predictor, x1), and payment for private tuition (as predictor, x2). We initially find that the correlation between x1 and x2 is high and significant (e.g., xs  =  .82, p = .05). In this situation, we will not be able to confidently say the family income (x1), and cost of private tuition (x2) improve the student performance, although both predictors are statistically significant in the regression model. In this instance, income and tuition cost are very much likely move together as more income a family has, more they can spend on private tuition. Another way we can say about multicollinearity is, it is like entering a classroom where everyone is talking. We can hear the students but we can hardly determine who says what. In order to check if multicollinearity is an issue in our regression model, we can look at Tolerance and VIF (variance inflation factor). Tolerance is a measure of influence of one predictor variable (i.e., independent variable) has on other predictors. A reciprocal of tolerance is VIF. The higher VIF value the more influence one predictor has on the others. In short, the stronger the correlations between predictors are. Ideally, VIF should not exceed 3.30, although VIF not exceeding 5.0 is acceptable is also acceptable, especially in research which is at exploratory stage. Example 8.1 In this example, we want to investigate how attitude towards reading, x, influences reading frequency, y, in children. Here is a sample of a survey instrument.

138

8  Analysis: Regression

8.1  Simple Regression

139

140

8.1.2  Procedure

8  Analysis: Regression

8.1  Simple Regression

141

142

8  Analysis: Regression

8.1  Simple Regression

8.1.3  Output

143

8.1.4  Checking the Assumptions Assumption i in this simple linear regression is linearity. The predictor variable, x, is assumed to move in a linear fashion with the outcome variable, y. We can test this assumption using a scatter plot in SPSS.

8.1  Simple Regression

145

Assumption ii requires that the variables be normally distributed. This can be checked using a statistical test (Kolmogorov-Smirnov or Shapiro-Wilk) or by using visual inspection of a Q-Q plot. The following steps describe how these checks can be done.

146

8  Analysis: Regression

8.1  Simple Regression

147

Assumption vi assumes there is no influential outlier in the dataset. While the Q-Q plot provides an indication of normality, it also points to possible potential outliers in the data. We can examine the data in detail using the box plot below.

148

8  Analysis: Regression

In this example, there are a few data points for the variable Attitude which can be regarded as potential outliers (see the box plot result below). However, since these data points are not too far away from the expected values based on the Q-Q plot (Attitude) and the fact that they are not flagged with asterisk (∗), we choose to include the data points (i.e., cases number 182, 173, 191, 120, 103, and 180) in the analysis, rather than excluding them. In a more conservative analysis, these data points are winsorised or trimmed (as previously described on page 58). A typical 90% Winsor means that all data points below 5th percentile are moved to the fifth percentile, and all data points above 95th percentile are brought to the 95th percentile. What this means is that the farthest and lowest values are brought to the next nearest lowest, and the upper-most values are brought to the next highest. Trim on the other hand, simply means we exclude the case from the analysis.

8.1  Simple Regression

149

Assumptions iii, iv, and v require the residuals to have homoscedasticity, linear, and normally distributed. The following output from the regression procedures can be used to check for these assumptions.

150

8  Analysis: Regression

8.1  Simple Regression

151

152

8  Analysis: Regression

8.1.5  Report

8.2  Binomial Regression A binomial regression is also known as a binomial logistic regression, or simply logistic regression. This type of regression is used when researchers want to predict an outcome where the outcome variable is dichotomous (e.g. yes/no, or m ­ ale/

8.2  Binomial Regression

153

female). In other words, the researchers want to estimate a likelihood of an observation falls into one of the two categories.

8.2.1  Assumptions i. The dependent variable has to be dichotomous, i.e., there are only two possible outcomes (e.g., yes/no, girls/boys). ii. There has to be an independence of observations. In a simple terms, the respondents are measured only once. iii. If your predictor variable is continuous (i.e., scale type), there has to be a linear relationship between this predictor and the logit transformation of the outcome variable. Generally, this is done through a Box-Tidwell test that examines the interaction term of the predictor and the log odds. A non-significant, p > .05 indicates that this assumption holds. Don’t worry if this jargon sounds complex. The logit transformation and the Box-Tidwell are illustrated in Sect. 8.2.4. Example 8.2 A survey was conducted to see how encouragement to read influences young people’s attitudes towards reading. A total of 200 responses were received and analysed.

154

8  Analysis: Regression

8.2  Binomial Regression

8.2.2  Procedure

155

156

8  Analysis: Regression

8.2  Binomial Regression

8.2.3  Output

157

158

8  Analysis: Regression

8.2  Binomial Regression

159

160

8  Analysis: Regression

8.2.4  Checking the Assumptions The first two assumptions for binary logistic regression are covered in the design of a study. SPSS is used to check for assumption iii, linearity when there is a continuous variable. In our example, the predictor variable is a continuous variable. Therefore, we need to check assumption iii holds in our regression model. There are two steps to check for this assumption. The first step is to compute logit value of the predictor (Encouragement). The second step is to check if the interaction between the Encouragement and its logit value is non-significant.

8.2  Binomial Regression

161

162

8  Analysis: Regression

8.3  Multinomial Regression

163

8.2.5  Report

8.3  Multinomial Regression When the outcome variable, y, is a categorical type and has more than two possible responses, we have to use multinomial regression for prediction. This is similar to binomial logistic regression, where y only has two possible outcomes.

164

8  Analysis: Regression

8.3.1  Assumptions i. The dependent variable has to be nominal (i.e. categorical). For example, types of school – public, private, or international. ii. There has to be an independence of observations. The respondents are measured only once. iii. If your predictor variable is continuous (i.e. scale type), there has to be a linear relationship between this predictor and the logit transformation of the outcome variable. Typically, this is done through a Box-Tidwell approach where the continuous predictor, and the interaction with its logit are included into the model. A non-significant, p > .05 indicates this assumption holds (see Sect. 8.2.4). iv. Outliers. An outlier is an influential data point which can distort the reliability of a regression result. Normally, a data point which sits beyond three standard deviations away from the mean is considered an outlier. Example 8.3 A survey was conducted to investigate if gender has an influence on library visitation times. The study also wants to know if the pattern in gender influence is consistent across all groups with different reading frequency. A total of 277 out of 302 responses received were valid and analysed. 25 responses had missing data and were excluded from the analysis.

8.3  Multinomial Regression

8.3.2 

Procedure

165

166

8.3.3  Output

8  Analysis: Regression

8.3  Multinomial Regression

167

168

8  Analysis: Regression

8.3  Multinomial Regression

169

170

8  Analysis: Regression

8.3  Multinomial Regression

171

8.3.4  Checking the Assumptions The first two assumptions for multinomial logistic regression are covered in the design of a study. SPSS is used to check for assumption iii, linearity when there is a continuous variable. In this example, assuming that reading frequency (ReadFreq.General) is a continuous predictor variable, we then need to check if assumption iii holds. There are two steps to check for this assumption. Firstly, compute logit value of the predictor; and secondly, check if the interaction between the predictor and its logit value is non-significant when we include this in the model.

172

8  Analysis: Regression

8.3  Multinomial Regression

173

174

8.3.5  Report

8  Analysis: Regression

8.3  Multinomial Regression

175

Chapter 9

Write Up and Research Translation

Now you know how to run and report statistical analyses, you will be keen to share your findings with the world at large. Very few researchers in contemporary research environments conduct research with the sole aim of producing a thesis; most of us want our findings to get out into schools, homes and communities and lead to change. An academic thesis is not always the most effective vehicle for achieving this! While educational researchers may have the best intentions toward research translation (also termed knowledge mobilisation), Cherney et al.’s (2012) review of the literature in this field leads them to conclude that in education, “academic research rarely has a policy impact and often fails to meet the needs of policy-­ makers and practitioners”, with “this disjunction… partly seen as originating in communication problems between policymakers, practitioners and academic researchers, drawing on the argument that they live in different worlds with differing languages, values and professional rewards” (p. 23). Communicating our findings in an early, broad and frequent manner can lead to the changes that we desire, establish our professional reputations, and help to build partnerships with key stakeholders in our fields. While we may be hesitant to share our findings before we have the chance to replicate our study a few hundred times or so, Levin (2011) argues for the value of timely translation: Research has the potential to continue to increase the effectiveness of education systems as more is learned about desirable and undesirable practices. The knowledge emerging from research is not always correct, and is subject to revision as time goes on but it still, in our view, provides both good grounds for many practices and, just as importantly, can be a counterbalance to the emphasis on practitioner knowledge or conventional wisdom, both of which are regularly found later, based on systematic inquiry, to be incorrect or even harmful. (p. 16).

To this end, while we’ve shown you how you can report quantitative outputs in APA style in our chapters, we also wanted to also provide additional tips for writing up results and disseminating your findings so that researchers, educators, parents, students and the community can easily access and use them. Once we started © Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_9

177

178

9  Write Up and Research Translation

writing this chapter, we realised that we could write a whole book on this alone, so we hope that you use it as a starting point for further inquiry in this area. In our discussion here, we’ll leave the traditional thesis alone, as it is still the most likely pathway for student researchers to transmit their findings, and it’s likely that your supervisors will be well-situated to help you to negotiate that journey. We will also avoid offering a rigid model of any text type that offers a prescribed format and style; though we will provide suggestions, it is imperative that these be taken as recommendations only, and adapted to your unique circumstances (Kamler and Thomson 2008). Instead, we focus on the following: • • • • • •

Plain English report for schools. journal articles. the thesis by publication. other Plain English research dissemination, such as through The Conversation. media interviews. conference presentations.

9.1  The Plain English Report for Schools As a researcher, you probably promised your schools or respondents a Plain English report, and now it is time to deliver. This is usually the very first thing we do once we’ve read through the data and performed preliminary analysis. We are not providing all of our final findings, just noteworthy trends in the preliminary data, and their implications. Schools wishing to keep abreast of the results of the study can subscribe to a mailing list. We also typically request that the report not be published to protect our intellectual property. Firstly, you may need to get your head around the Plain English part if you have been working in academia for a while. This may no longer come naturally to you, as you may have spent the last couple of years trying to extend your capacity to embrace polysyllabic words! You now need to communicate in easy to understand, highly accessible language, and you will be communicating with busy school administrators who may switch off at the first hint of jargon. This desire to communicate clearly and effectively should also inform your process of selection of content for the report, as well as the language you use to communicate your ideas. Even though you may be deeply committed to your theoretical framework, very few schools will be at all interested in it. When providing advice about good writing, famous author Elmore Leonard suggests that we “try to leave out the part that readers tend to skip. Think of what you skip reading a novel: thick paragraphs of prose you can see have too many words in them” (Leonard 2010). We believe that this advice is definitely transferrable to the Plain English report, where we should leave out the bits that schools will ignore, and keep our paragraphs brief and succinct. If it simply hurts too much to fail to celebrate your epistemological

9.2  The Journal Article

179

and theoretical choices, you can always refer your audience to your future publications should they be interested. Here’s what we typically include in a Plain English report for schools, unless we’re given a proforma to follow by the funder or school association: i. As aforementioned, an opportunity for schools to subscribe to updates (e.g. publication alerts, etc.). ii. A brief outline of the study (again, in brief) that should include why and how this study was conducted, what information was collected and who participated. iii. Key trends in the preliminary findings, which does not need to include complex explanations around analytical techniques. iv. Implications for schools: news that they can use and share in the school newsletter or parent information night. v. Any references used (we suggest that it is better to not be overly reliant on other sources; your audience wants to know about your project and your findings, after all). You should try to keep the report as brief as possible, though pictures (that do not identify your respondents), graphs and diagrams that can help you to keep things simple and communicate your ideas visually are a good idea (as long as you pitch them at a non-academic audience). You may also wish to include an executive summary if you find that you’ve gotten carried away with your explanation. It helps to look at a few models before embarking on writing a Plain English Report—the public service typically produces these types of reports, though they won’t necessarily be research-based, so numerous free samples can be found online that can help you to grade your language and finalise the structure that you wish to use. The previously mentioned SMOG readability tool can also assist in supporting you to pitch your language at an appropriate level.

9.2  The Journal Article Publishing papers in peer-reviewed journals is essential for sharing your findings with the research community; while “traditional motivations to publish emanate from scholarly, scientific and ethical philosophies regarding the importance of disseminating knowledge” (McGrail et al. 2006, p. 19), the impetus to publish is further enhanced by contemporary university requirements. For academics, it is also related to our workload requirements, promotional opportunities and research funding success, as well as contributing to our institutional performance (Kamler 2008). We are judged by our scholarly contribution as typically evidenced through peer-­ reviewed journal volume and quality, amongst other measures. In addition, the peer review journal writing process can play an integral role in supporting researchers “to improve and become more effective researchers and communicators” (Merga

180

9  Write Up and Research Translation

et al. 2018). For higher degree by research students, publishing during candidature can offer a range of benefits, which we will address is further detail in the thesis by publication section of this chapter. When we write a journal article, we seek to craft a paper that communicates key findings. The paper will align with style guidelines, as well as the norms of expression and academic voice common to our field. It may be a good idea to select a target journal at the outset so that voice, style and format of the article can be tailored to the explicit author guidelines and implicit expectations of a specific journal. However, the sheer choice of outlets can be overwhelming (depending on your field), particularly when we know that there is a high chance of rejection. The possibility of rejection can be more marked in education than some other disciplines; for example, a 2013 study found that while the acceptance rate in the discipline of education was typically higher than that of business, it was far lower than in health (Sugimoto et al. 2013). In addition to jeopardising our ability to share our findings, rejection can also have a detrimental effect on our confidence and self-concept (e.g. Horn 2016), so we avoid it at all costs. This is why it can be contended that writing journal articles is a kind of like high-stakes gambling in which we invest time, money and confidence in the hope of success. While success is reflective of our skill and knowledge, it also arguably involves an element of luck. Good outlet choice is important, to optimise our odds. We may typically try to place our papers in quality peer-reviewed journals where we can contribute to an existing conversation, which is often (but not always) indicted by the journal that we most frequently cite in the references of our paper. However, we provide a brief overview of some of the relevant considerations below for those new to publishing in this space.

9.2.1  Good Fit Questions: Does your article look/read like a typical article in this journal? Does the article conform with the author guidelines of the journal? Is the look and voice similar to other articles published by this journal? Does this article cite literature from this journal? Does it fall within the scope of the journal? What is the acceptance rate for this journal? The most important thing to do before selecting a journal is to read numerous recent papers from that journal. They need to be recent, as editorial board changes can lead to sweeping cultural changes in a journal, and what was a strong bet in the 1980s or 1990s may no longer hold sway. If your paper looks and feels like the papers in the journal, you may have a good fit. It’s also important to try to achieve a degree of subjectivity in determining the fit. If you know the data that your paper reports on is not actually that novel and you’re submitting to a journal with a >20% acceptance rate, it is probably not a good fit.

9.2  The Journal Article

181

9.2.2  Reputable Questions: Have you heard of this journal before? Is the journal valued by researchers in your field? How does the article rank in terms of impact? Does the article meet minimum impact requirements for your School or Faculty? Is the journal highly visible in your field? Is the journal valued by educators in your field? Do established researchers in your field publish with this journal? Determining impact in relation to journal reputation in educational research is highly contentious. Education journals lack impact when compared with other disciplines, such as medical science journals. For instance, the highest ranked journal on the bibliographic database SCImago (which is often used for impact benchmarking in education) is CA- A Cancer Journal for Clinicians, with a SCImago Journal Rank (SJR) of 39.285. In contrast, the highest ranked journal in the field of Education as ranked on SCImago is currently the American Educational Research Journal, with an SJR of just 4.216 (as on September 27th, 2017), so we are talking about a mouse standing next to an elephant in terms of relative impact presence. In addition, some sub-fields within education are just typically very low impact. Not all quality journals are indexed by bibliographic databases, and North American and European journals are typical privileged in these databases (Sugimoto et al. 2013). If Education journals are filtered by country, it is clear to see the absolute dominance of the US, with 9 of the current top 10 journals in education US located (with the other one in the UK). To put this further into perspective, the top Australian journal, Research in Science Education, is at number 86 on the whole list. As journal impact factor calculations have a number of significant shortcomings which make them somewhat unreliable as an overall measure of reputability and quality, it is recommended that we avoid viewing them as “the holy grail of quality assessment” (Ha et al. 2006, p. 915). There’s also the point that ranking does not remain static. SCImago also ranks journals by Quartile (Q), with Q1 being the highest ranking. You might choose to publish in a Q3 journal because it is most likely to be read by educational practitioners in your field, and the following year it may level up to Q2, leading to a surge in your impact. The reverse can also happen. This is why we take impact into account, but do not get hung up on it. It is more useful to speak with respected researchers in your specific sub-field to get a more long-term perspective on the reputation of a journal. That said, in some contexts you will have to pay close attention to these measures. Not all Schools or Faculties in universities have a minimum standard, though others do. For example, one university requires that in order to “count” as a quality publication (i.e. toward your workload), a journal article needs to be published in a minimum Q2 journal as rated on SCIimago, and of course, preferably in a Q1 journal. Other factors also mess with our benchmarking. We consulted ERA journal rankings until they were accepted as defunct and SCImago became the accepted ranking system, and no doubt in a few years they will be replaced by another model. Sadly, a number of journals that were ranked A on the ERA system are Q2 or lower

182

9  Write Up and Research Translation

on the newly accepted SCImago measure, which makes us feel terrible if we’ve gone to extensive efforts to have papers accepted in the old ‘A’ level journals. These efforts now mean very little in terms of impact. We know how the evil queen in Snow White feels when we are suddenly no longer the fairest one of all, thanks to SCImago.

9.2.3  Fast Questions: What is the typical length of time to journal publication? (i.e., How long is the peer review process typically? How long is the period from paper acceptance to online/paper publication?) This is simple: the faster your ideas are published, the faster they can move and be used. Author guidelines rarely include an estimated time to publication, and while approaching colleagues who have published with that journal can give you a ballpark figure, there can be considerable variation due to a range of factors. Research from other fields suggests that the nature of the results being reported can significantly impact on publication times (e.g., Hopewell et  al. 2007). While we could not find any research confirming the applicability of this research to the education context, we feel that there is likely some degree of currency to a preferencing of papers with more compelling data. We tend to typically favour journals that have previously moved our papers through peer-review and publication in a timely manner.

9.3  U  nique Challenges in Publishing Quantitative and Mixed-Methods Articles in education Even if you have many qualitative articles published in peer-reviewed journals, you may need to revisit your strategy for journal selection when shifting to a mixed or quantitative method. Knight and Steinbach (2008) suggest that method needs to be taken into account when choosing a journal to approach with a manuscript, recommending that the following questions be asked: i. Is the manuscript in harmony with the journal’s quantitative or qualitative research bias, if any? ii. Has the journal published articles using the manuscript’s methodology before? iii. If the manuscript reports insignificant results or if it is concerned with methodological issues, has the journal published papers of this same type in the past? (p. 73) If you submit a paper showcasing your new quantitative data analysis skills to a journal that has a qualitative bias, it is unlikely to end well for you. Firstly, it may

9.3  Unique Challenges in Publishing Quantitative and Mixed-Methods Articles…

183

be hard for the editors to find someone to review your paper. Secondly, you might get a desk reject as your methods fall outside the typical scope of the journal. Thirdly, if it goes out for review, you may be really surprised by what you get in return. While peer review reports are typically extremely useful and generous missives, and even wholly negative reviews can do much to support our growth as writers and researchers, a report on a quantitative paper by someone who is clearly offended by the quantitative paradigm may not be very helpful. However, just because a journal tends to have a bias, does not mean that you should necessarily rule it out of consideration. As we mentioned in the introduction, some journals in education strongly privilege quantitative methodology, however, these are not always the journals that are discussing what is relevant to you. As such, sometimes we have submitted papers to journals that less typically publish quantitative methods, and this has led to some unusual occurrences. For instance, on one paper reporting on quantitative findings, a peer reviewer asked about the meaning of the large N before the total number in the sample. On another quantitative paper, we were asked to explicitly explain correlational analysis in the paper so that the audience could understand what was meant by the term. It was easy to accommodate these unanticipated requests, and to learn to be flexible in order to meet the needs of the audience. It is also really important that if you are used to writing qualitative papers, that you understand the style differences between the two fields. One common difference is in relation to authorial voice, such as the use of the personal pronoun, common in qualitative fields but still frowned upon in the quantitative space. It’s also a good idea to clearly explain your methodological choices if you have used mixed methods: for example, you might “discuss how qualitative and quantitative methods can serve the dual purpose of confirmation and elaboration of results” (Creswell and Tashakkori 2007, p. 109). However, publishing mixed methods can be very tricky, so we’re going to provide further explanation around this area. In our experience, publishing mixed-methods papers holds the biggest challenge as it is still very much seen as a new approach (Creswell and Plano Clark 2011) that is “still developing and will do so for years to come” (Tashakkori and Creswell 2007, p.  4). We have had a number of frustrating encounters, usually involving instances where the editor has sought to accommodate the methodological nature of the paper by sending it to one reviewer in the quantitative camp, and the other reviewer in the qualitative camp. As explained by Leech et al. (2011), “the problem lies in the fact that readers with a qualitative orientation and readers with a quantitative orientation have different ideas about what components of a (mixed) research article is interesting and relevant” (p. 10). Unsurprisingly, this leads to one review suggesting that we truncate or quantify the qualitative component of the research, and the other suggesting that the real richness of the paper lies in the qualitative findings. While it is easy to feel discouraged when this happens, we prefer to see ourselves as pioneers of mixed methods, so we don’t give up. When revising the paper, we don’t accept changes that don’t make sense in relation to mixed methods, carefully justifying this refusal with academic references.

184

9  Write Up and Research Translation

To deal with the mixed methods in sufficient descriptive depth and detail takes a lot of words, so it is not advisable to target a journal with a stringent and minimal word limit. Sometimes when papers blow out substantially, you have to consider cutting a paper in two, and publish the qual followed by the quant (or the reverse); we’d like to stress that we don’t recommend doing this, but if you are in a subfield where all of the prominent journals have tiny word limits that simply do not accommodate mixed-methods, you may not have much of a choice. On the bright side, learning to squeeze mixed methods into an acceptable word count can help to improve your capacity to communicate your research story in a succinct manner. The success of a mixed methods paper is often, but not always, contingent on the degree to which the methods are cohesively integrated, and perhaps more importantly, the degree to which mixed methods are supported by the journal. The problem with getting mixed methods papers published isn’t always related to the methodology, so it is important to remain responsive to reviewer feedback, as long as it is relevant.

9.4  Thesis by Publication As we’ve mentioned previously, quality peer-reviewed publications are a foundation to future success in most dimensions of academic life, allowing an academic to both survive and thrive in their work role. Johnson (2012) situates publication as integral to a successful research career: If you do not publish your research outcomes no one will ever know of its existence. Producing publications is not easy and it is not in fact research but it is essential to your research effort, as future grants, promotion, and other job opportunities will depend upon the substantial high-quality research outputs documented in your CV. (p. 46)

Higher degree by research students are increasingly expected to publish during candidature, sharing their research findings in a timely manner, while establishing themselves as experts in their respective fields. This can be particularly important if they are hoping to secure a post-doctoral fellowship and to work in academia after completion, as academic positions are secured by a minority of graduates. Completing a thesis by publication is one way of optimising research output in a way that may seem relatively economical—instead of writing up your findings in papers post-thesis submission, once the data are older and ideally you should be moving on to the next research project, you are sharing your findings while they are fresh. In addition, when brought together with some structural components to ensure that they tell a story, they can have equal, if not greater depth and significance to a thesis. We define the thesis by publication as “a collection of research papers, preferably published in well-regarded, peer-reviewed journals, as well as binding materials, such as an introductory chapter and/or discussion section, which bring together the ideas explored in the papers into a cohesive whole” (Merga 2015, p. 291). A recent

9.4  Thesis by Publication

185

analysis of thesis by publication (TBP) documents in the Humanities and Social Sciences (including education) found that “journal articles and conference proceedings are the favoured text modes for inclusion in TBPs” at present (Mason and Merga 2018, p. 151). The first question asked is usually “How many papers do I need?”; this can vary widely, with the aforementioned research project finding that these theses ranged “from one to 12 publications”. with “an average of 4.5 publications per thesis, and a mode of 4 publications” (Mason and Merga 2018, p. 146). As such, we would argue that there is not yet a typical thesis by publication in education, as it is still in the early stages in our field, but that four or five papers seems to be a relatively accepted norm, though it is important to be responsive to your university’s policy. It is not the number that counts, anyway; it is how the papers are used, and the strength of the contribution they make to establishing your overall argument. The second question is often “What should the thesis look like?”. Possible shapes based on analysis of extant theses are illustrated in a recent paper, which found 11 recurring structural choices, which can help to guide doctoral candidates to make the best choice for their project (Mason and Merga 2018). In a previous paper in this area, Margaret (Merga 2015) outlined six of numerous reasons to undertake the thesis by publication. We’ve already touched on how the thesis by publication shows responsiveness to contemporary academic culture, but we outline the following five reasons: i. Finding authorial voice(s): Not only do you learn to write at a level suitable for peer-reviewed publication, you also learn to develop your own authorial voice, which you can adapt to be responsive to unique journal requirements. ii. Research translation: Schools and the media (a powerful dissemination tool that we explore further subsequently) seem more likely to engage with a journal article than a doctoral thesis. As such, it may be easier for your research to translate in this thesis format. iii. Critical feedback: Instead of having critical feedback from just a supervisory panel of two or three, you also get (typically) high quality feedback from peer reviewers, which can really help you to shape your work. This feedback also has the added advantage of being anonymous and it is not dulled by repeated exposure to the research as supervisory critique can sometimes be (particularly by the end of the research journey). iv. Becoming a researcher: You don’t have to wait until you submit your thesis to make a name for yourself in your field. You can use your papers as a basis for networking, and as writing for journals is a core part of the academic profession, whereas writing theses is arguably not, you can focus your writing improvement in a relevant area. v. Unpopular ideas: If your thesis is a bit controversial, or challenges existing ideas, the critical feedback you receive on your papers can help you to find a niche, and to present your findings in such a way as to counter opposition.

186

9  Write Up and Research Translation

9.4.1  But Is This Right for Me? If you want to do your thesis by publication, but you are not sure that it is right for you and your project, checking against the following criteria will help, though each are not individually essential. However, if you don’t have any of the following in your favour, it might not be the best idea for you. i. A number of research questions to focus on: This enables breadth and depth so that you can put out a range of papers that do not have much overlap. ii. An experienced, understanding and supportive supervisory team: They will be able to support you and may be willing to co-author at least your early papers while you are developing your skills, though you should always be the first and principle author on your papers where at all possible. They also need to be aware of the heightened time constraints that you are working under so that they can review and contribute to your papers in a timely manner (if applicable). iii. An understanding and supportive School or Faculty: They will be able to support your training to help you to meet your objectives. iv. Strong writing skills: You will need to be writing early and often to get your papers out in a timely manner. v. Excellent time management: To get your papers out, you will need to plan carefully and strategically. There will not be time for procrastination or confusion. vi. Confidence, resilience and an autonomous working habit: You will need to get on with it and not need to be micromanaged, otherwise you will become too time intensive for your supervisors. You also need confidence to cope with rejection which can be a lot more frequent in this thesis approach. If you are a student currently considering this mode, it is good to commit early, so we encourage you to read the institution policy on this kind of thesis, and discuss the possibilities with your supervisory team.

9.5  Plain English Dissemination We also share our research in Plain English dissemination, by writing Plain English articles that are not for peer-review, but rather seek to share our peer-reviewed findings published elsewhere in a condensed and engaging format. To this end, we have put a number of pieces in The Conversation, as we’ve found that it is a great way to generate discussion around our research findings. Saiyidi and Margaret had a piece in The Conversation that we are able to share here, due to the fact that The Conversation operates under a Creative Commons license. This license is very convenient for news outlets, as the pieces we submit to The Conversation are often picked up and published verbatim elsewhere, with attribution to The Conversation as the original source. This license also means that we can include it in full here.

9.5  Plain English Dissemination

187

As you can see below, we’ve taken a rather dense article that reports on complex statistical analysis, and pulled out what we imagine the average intelligent person might find most interesting. We’ve abandoned jargon and tried to adopt a persuasive voice. We say very little about method and nothing about informing theory. It is also contrary to our more typical academic writing style in its brevity, use of generalising terms, and perhaps most challenging for an academic writer, its use of one sentence paragraphing. The trick to writing these 800 word missives is to only include what you would tell a smart person from outside your area if they asked why your most recent findings were important. The key is to focus on what is important for the audience of Plain English pieces, not what we think is the most interesting thing from our academic perspectives. Sometimes the two align, but often, they really don’t. Children Prefer to Read Books on Paper Rather than Screens (From The Conversation, Merga and Mat Roni 2017) There is a common perception that children are more likely to read if it is on a device such as an iPad or Kindles. But new research shows1 that this is not necessarily the case. In a study of children in Year 4 and 6, those who had regular access to devices with eReading capability (such as Kindles, iPads and mobile phones) did not tend to use their devices for reading – and this was the case even when they were daily book readers. Research also found that the more devices a child had access to, the less they read in general. It suggests that providing children with eReading devices can actually inhibit their reading, and that paper books are often still preferred by young people. These findings match previous research2 which looked at how teenagers prefer to read. This research found that while some students enjoyed reading books on devices, the majority of students with access to these technologies did not use them regularly for this purpose. Importantly, the most avid book readers did not frequently read books on screens. Why Do We Think Children Prefer to Read on Screens? There is a popular assumption that young people prefer to read on screens. This was mainly driven by education writer Marc Prensky who in 2001 coined the term “digital natives”.3 This term characterises young people as having high digital literacy and a uniform preference for screen-based reading. But young people do not have a uniform set of skills,4 and the contention that screens are preferred is not backed up5 by research.

 https://authors.elsevier.com/a/1UgVS1HucdAJy9  http://search.informit.com.au/documentSummary;dn=381519903117510;res=IELHSS 3  http://www.emeraldinsight.com/doi/pdfplus/10.1108/10748120110424816 4  http://ro.uow.edu.au/cgi/viewcontent.cgi?article=2465&context=edupapers 5  http://www.mdpi.com/2304-6775/3/4/237/pdf 1 2

188

9  Write Up and Research Translation

Despite this, the myth has already had an impact on book resourcing decisions6 at school and public libraries, both in Australia and in the US, with some libraries choosing to remove all paper books in response to a perceived greater preference for eBooks. But by doing this, libraries are actually limiting young people’s access to their preferred reading mode, which in turn could have a detrimental impact on how often they choose to read. Young people are gaining increasing access to devices through school-promoted programs, and parents face aggressive marketing to stay abreast of educational technologies at home. Schools are motivated to increase device use, with Information and Communication Technology being marked as a general capability7 to be demonstrated across every subject area in the Australian Curriculum. The drivers toward screen-based recreational book reading are strong, but they are not well-founded. Why Are Students More Likely to Prefer Paper Books? Reading on devices through an application leaves more room to be distracted,8 allowing the user to switch between applications. For students who already experience difficulty with attention, the immediate rewards of playing a game may easily outweigh the potentially longer-term benefits of reading. Digital literacy could also be an issue. In order to use a device to read books, children need to know how to use their devices for the purpose of reading books. They need to know how to access free reading material legally through applications such as Overdrive9 or websites such as Project Gutenburg.10 Tips for Encouraging Your Child to Read Research shows that reading books is a more effective way to both improve and retain literacy skills,11 as opposed to simply reading other types of text. Yet international research suggests that young people are reading fewer and fewer books.12 While equipping children with devices that have eReading capability is unlikely to encourage them to read, there are a number of strategies, supported by research, that can help encourage children to pick up a book. These include:

 http://search.informit.com.au/documentSummary;dn=381519903117510;res=IELHSS  http://www.australiancurriculum.edu.au/generalcapabilities/information-and-communicationtechnology-capability/introduction/introduction 8  https://www.amazon.com/Shallows-What-Internet-Doing-Brains/dp/0393339750 9  https://www.overdrive.com/ 10  http://www.gutenberg.org/ 11  http://www.sciencedirect.com/science/article/pii/S1041608013000642 12  https://www.oecd.org/pisa/pisaproducts/pisainfocus/48624701.pdf 6 7

9.6  Altmetrics and (Social) Media-Supported Dissemination

189

i. Be seen to enjoy reading. This study13 found that a number of students did not know if their literacy teachers actually liked reading. Teachers who were keen readers inspired some students to read more often and take an interest in a broader range of books. ii. Create (and regularly access) reading-friendly spaces14 at home and at school. Loud noises, poor lighting and numerous distractions will not help provide an enjoyable reading experience, and are likely to lead to frustration. iii. Encourage regular silent reading15 of books at school and at home. Giving children time to read at school not only encourages a routine of reading, but it also may be the only opportunity a child has to read self-selected books for pleasure. iv. Teachers16 and parents17 should talk about books, sharing ideas and recommendations. v. Continue to encourage your child and students to read for pleasure. While we know that children tend to become disengaged with books over time, in some cases this can be due to withdrawal of encouragement18 once children can read on their own. This leads children to falsely assume that reading is no longer important for them. Yet reading remains important for both children and adults to build and retain literacy skills. vi. Find out what your child enjoys reading, and support their access19 to books at school and at home. We hope that you find this a useful and concrete example to support your forays into Plain English dissemination of your findings. To illustrate the power of The Conversation as a knowledge translation vehicle, as at February 22 2019, 230,701 people had read this article from a wide variety of countries worldwide. The article had been tweeted 1334 times and shared on Facebook 94,598 times. A large range of media outlets published it, including in the USA and Asia. These shares in turn enhance our altmetrics, which we will explain further below.

9.6  Altmetrics and (Social) Media-Supported Dissemination The most important reason to garner media coverage of your work is that it enables the target audience in education, whether they be parents, principals, teachers, students, policy-makers or other stakeholders or practitioners, to be aware of your  http://onlinelibrary.wiley.com/doi/10.1111/eie.12126/abstract  http://www.tandfonline.com/doi/abs/10.1080/01930826.2016.1185854 15  http://onlinelibrary.wiley.com/doi/10.1111/eie.12026/full 16  http://journals.sagepub.com/doi/abs/10.1177/0004944114565115 17  http://onlinelibrary.wiley.com/doi/10.1111/eie.12043/full 18  http://onlinelibrary.wiley.com/doi/10.1111/eie.12043/abstract 19  http://onlinelibrary.wiley.com/doi/10.1111/eie.12071/full 13 14

190

9  Write Up and Research Translation

research. Until your research reaches your target audience it cannot lead to change in the knowledge base or practice, so increasing awareness of your research is absolutely paramount for most researchers. Getting your research into the media often also enables its rapid spread beyond your local community context into national and international conversations. However, it can also be good for your career in terms of increasing your impact. We’ve talked about the use of bibliographic databases such as SCImago for determining research impact, however, this is not the only way that research impact is quantified. One of the newer measures involves altmetrics. As explained by Featherstone (2014), altmetrics can be used to measure research impact drawing on text sources outside traditional scholarly publishing, and “altmetrics analyze tweets, blogs, presentations, news articles, comments, or any social commentary about a diverse group of scholarly activities that are captured on the web” (p. 60). Assuming that you are affiliated with a research institution, the best way to attract media interest is through your media and communications (M&C) centre. Typically, this would mean that once you have a quality research paper in press and about to be released, you would craft a press release with the support of a media staff member. On release of the research, the press release will also be released, sharing your findings and promoting your new article. The other advantage of always going through M&C is that if your research garners a lot of interest, it will not be humanly possible to do all of the interviews requested. You will need M&C to act as your agent, and to cut off further media once you have become overwhelmed, as you need to also have time to do your other work. You will also often get media attention through The Conversation and other Plain English pieces. We strongly suggest also asking M&C for some media training before putting out a press release or doing a Plain English piece that you think might be widely shared. Questions without notice is the norm for interviews in the media, and this can feel somewhat out-of-control for a researcher. Training can build your skills and your confidence so that you don’t spend hours feeling depressed over not representing your research in the best possible light.

9.7  The Conference Last but not least, we suggest that you get thee to a research conference. Choosing a conference in education can be challenging, as going to a conference is a financial investment, and we have found that many conferences in education are typically more researcher or practitioner oriented. Once you decide if you would rather pitch your research in this instance to fellow researchers or your end users, the practitioners, this can help you to choose your conference and shape your approach. We typically try to visit both types of conferences where possible within budgetary and time constraints. When you are new to using quantitative methods, the idea of presenting your findings in front of a potentially critical audience may seem daunting. What if they

References

191

ask you something about an aspect of quantitative analysis that you are unfamiliar with? We like to tell our students and early career researchers that this won’t happen to them, but this can be a lie. Exactly this thing happened to an early career researcher that Margaret spoke with at the American Educational Research Association (AERA) in Texas, 2017, and she was still shaking from the experience some time later. While she was unlucky to be targeted, it did raise the importance of being prepared. While you can learn as much as possible about presenting quantitative findings, no one knows everything in this area. You need to be prepared to thank someone for their interesting question and let them know that you will look further into it, and take their email, if you are challenged on the spot by someone keen to show the breadth of their knowledge. As such, don’t be afraid, just be prepared.

9.8  Final Comment Don’t keep them to yourself; get your research findings out there. It is no longer enough to just produce a traditional thesis, or a paper—knowledge mobility requires a more strategic level of communicative engagement. We hope that the ideas that we briefly raised in this chapter can support your communication of your findings so that they lead to real change in your field.

References Cherney, A., Povey, J., Head, B., Boreham, P., & Ferguson, M. (2012). What influences the utilisation of educational research by policy-makers and practitioners?: The perspectives of academic educational researchers. International Journal of Educational Research, 56, 23–34. Creswell, J. W., & Plano, C. V. L. (2011). Designing and conducting mixed methods research. Los Angeles: Sage. Creswell, J. W., & Tashakkori, A. (2007). Developing publishable mixed methods manuscripts. Journal of Mixed Methods Research, 1(2), 107–111. Featherstone, R. (2014). Scholarly tweets: Measuring research impact via altmetrics. Journal of the Canadian Health Libraries Association/Journal De l'Association Des Bibliothèques De La Santé Du Canada, 35(2), 60–63. Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals-Academy of Medicine Singapore, 35(12), 911. Hopewell, S., Clarke, M. J., Stewart, L., & Tierney, J. (2007). Time to publication for results of clinical trials. The Cochrane Library. Horn, S.  A. (2016). The social and psychological costs of peer review: Stress and coping with manuscript rejection. Journal of Management Inquiry, 25(1), 11–26. Johnson, A. M. (2012). Charting a course for a successful research career (2nd ed.). Amsterdam: Elsevier. Kamler, B. (2008). Rethinking doctoral publication practices: Writing from and beyond the thesis. Studies in Higher Education, 33(3), 283–294. Kamler, B., & Thomson, P. (2008). The failure of dissertation advice books: Toward alternative pedagogies for doctoral writing. Educational Researcher, 37(8), 507–514.

192

9  Write Up and Research Translation

Knight, L. V., & Steinbach, T. A. (2008). Selecting an appropriate publication outlet: A comprehensive model of journal selection criteria for researchers in a broad range of academic disciplines. International Journal of Doctoral Studies, 3, 59–79. Leech, N.  L., Onwuegbuzie, A.  J., & Combs, J.  P. (2011). Writing publishable mixed research articles: Guidelines for emerging scholars in the health sciences and beyond. International Journal of Multiple Research Approaches, 5(1), 7–24. Leonard, E. (2010). Elmore Leonard’s rules for writers. The Guardian. Retrieved from https:// www.theguardian.com/books/2010/feb/24/elmore-leonard-rulesfor-writers Levin, B. (2011). Mobilising research knowledge in education. London Review of Education, 9(1), 15–26. Mason, S. & Merga, M. K. (2018). Integrating publications in the social science doctoral thesis by publication. Higher Education Research & Development. https://www.tandfonline.com/doi/ful l/10.1080/07294360.2018.1498461 McGrail, M. R., Rickard, C. M., & Jones, R. (2006). Publish or perish: A systematic review of interventions to increase academic publication rates. Higher Education Research & Development, 25(1), 19–35. Merga, M. (2015). Thesis by publication in education: An autoethnographic perspective for educational researchers. Issues in Educational Research, 25(3), 291–308. Merga, M.  K. & Mat Roni, S. (2017). Children prefer to read books on paper rather than screens. The Conversation. Retrieved from https://theconversation.com/ children-prefer-to-read-books-on-paper-rather-than-screens-74171#comment_1234533 Merga, M.  K., Mason, S. & Morris, J.  (2018). Early Career experiences of navigating journal article publication: Lessons learnt using an autoethnographic approach. Learned Publishing. https://onlinelibrary.wiley.com/doi/epdf/10.1002/leap.1192 Sugimoto, C.  R., Larivière, V., Ni, C., & Cronin, B. (2013). Journal acceptance rates: A cross-­ disciplinary analysis of variability and relationships with journal measures. Journal of Informetrics, 7(4), 897–906. Tashakkori, A., & Creswell, J.  W. (2007). The new era of mixed methods. Journal of Mixed Methods Research, 1, 3–7.

Chapter 10

Conclusion and Further Reading

We set out to write this book for like-minded people – those who were interested in using statistics in educational research and who needed a helping hand in doing so. In many educational sub-fields there is a tendency to preference qualitative methods (well, speaking as a visual arts education researcher Julia knows most of the studies she reads are qualitative in nature), but each methodology has its merits and it is important to select a method that suits the research aims, even if it can be a bit daunting at first (and yes, imagine us as fairly stereotypical visual arts and English teachers/researchers who thought it would be a good challenge to learn statistics – naturally, we were terrified). If you read the book cover-to-cover (not our intention) or read through the contents page (far more likely), you will have noticed that we intentionally introduced you to some general considerations about quantitative methods and working with children and students to give you some background information before launching into the data preparation and analysis. We also concluded with reporting your findings (which you will have also seen for each specific non-parametric test as well as an overall report in Chapter Nine). We used this format as we wanted this book to be useful at each stage of your quantitative project – from planning, to collection and analysis, as well as interpretation and reporting. As educators ourselves, we chose to focus on educational research in this book, partly because it meant we could provide examples of research from our own experiences and also because we understand that it is most helpful to learn from seeing how research in your field is interpreted and reported (we know how challenging it can be to see many examples of statistics in the sciences and try to think about their application in the education context). The aim of this book was to introduce you to non-parametric statistics, the ‘veritable wallflower at the statistics party’. As social science researchers, it is important to note that participants’ responses to quantitative instruments are shaped by their past experience, values and attitudes; as such, there are times when your data just won’t fit the bell curve. This is when it becomes important to know something about the forgotten less-popular sibling of parametric statistics (and hopefully you do by © Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3_10

193

194

10  Conclusion and Further Reading

the time you get to this chapter). This book has explored non-parametric tests that you are most likely to encounter, particularly in exploring statistical differences between groups (Chapter Six) as well as correlations between variables (Chapter Seven) and the more challenging regression tests (Chapter Eight). In the complex setting of educational research, it is common to find us needing non-parametric tests to deal with the data we collect, and to allow us to report on these data in a way that is valid and useful to improving educational practice. Below is a brief checklist for your quantitative research project. You may like to take a look at it as a guide (it is by no means comprehensive ... that would take a second book!): i. Have you defined a research topic? Do you have support from the literature to identify reasons to undertake your research based on what has (or hasn't) been done before? ii. Have you clearly defined your research questions/hypotheses? Are they written in a way that can be answered through quantitative research? iii. Have you received ethical clearance? Keep in mind that you may need institutional approval as well as approval from school sectors (for example, the Department of Education in your state or territory). iv. Have you identified your intended sample and required sample size? Do you have a sampling frame (criteria) for participation? v. Have you identified the quantitative instrument/s you will use? Are the instructions and questions age-appropriate and clear? vi. Have you piloted the survey and received feedback that allows for improvement? vii. Have you considered how you are administering the survey to your participants, and how to get the data into the appropriate format for analysis? viii. Once the data are ready for analysis, how are you going to 'clean' them to account for non-responses and to add value labelling (if required)? ix. Have you checked for the validity and reliability of any scales in your instrument? x. Have you identified the statistical tests you need to run? Are they descriptive or inferential in nature? xi. What kind of output do you need? Can you write the findings in sentence form, or do you need graphs and tables? This impacts on reporting. xii. In your reporting, have you provided information on your sample, recruitment, methods of instrument administration and quantitative analysis? Have you described any limitations of your research?

10.1  Further Reading

195

10.1  Further Reading As non-parametric statistics and quantitative methods become more familiar to you, it is likely that you will want to find other sources of information to extend your knowledge. Below is a list of additional readings organised by topic as a starting point.

10.1.1  Readings About Mixed Methods Design Castro, F. G., Kellison, J. G., Boyd, S. J., & Kopak, A. (2010). A methodology for conducting integrative mixed methods research and data analyses. Journal of Mixed Methods Research, 4(4), 342–360. Creswell, J. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks: Sage. Creswell, J., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research. Los Angeles: Sage. Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2(3), 270–283. Edmonds, W. A., & Kennedy, T. D. (2016). An applied guide to research designs: Quantitative, qualitative, and mixed methods. Thousand Oaks: Sage. Howe, K. R. (2012). Mixed methods, triangulation, and causal explanation. Journal of Mixed Methods Research, 6(2), 89–96. Leech, N. L., Dellinger, A. B., Brannagan, K. B., & Tanaka, H. (2010). Evaluating mixed research studies: A mixed methods approach. Journal of Mixed Methods Research, 4(1), 17–31. Morse, J. M., Niehaus, L., Wolfe, R. R., & Wilkins, S. (2006). The role of the theoretical drive in maintaining validity in mixed-method research. Qualitative Research in Psychology, 3(4), 279–291. doi:10.1177/1478088706070837. Wheeldon, J. (2010). Mapping mixed methods research: Methods, measures, and meaning. Journal of Mixed Methods Research, 4(2), 87–102.

10.1.2  Readings About Experimental Methods Design Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). New York: Routledge. Creswell, J. (2014). Educational research: Planning, conducting and evaluating quantitative and qualitative research (4th ed.). Essex: Pearson. Mertens, D. M. (2015). Research and evaluation in education and psychology (4th ed.). Thousand Oaks: Sage Publications.

196

10  Conclusion and Further Reading

10.1.3  Readings About Sample Size, Power and Effect size Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283–292. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). New York: Routledge. Lipsey, M.  W. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park: Sage. Rouquette, A., & Falissard, B. (2011). Sample size requirements for the internal validation of psychiatric scales. International Journal of Methods in Psychiatric Research, 20(4), 235–249. doi:10.1002/mpr.352.

10.1.4  R  eadings About Ethical Issues in Education and Social Science Research Creswell, J. (2014). Educational research: Planning, conducting and evaluating quantitative and qualitative research (4th ed.). Essex: Pearson. Christians, C. G. (2000). Ethics and politics in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 133–155). Thousand Oaks: Sage. Head, G. (2018). Ethics in educational research: Review boards, ethical issues and researcher development. European Educational Research Journal, 1474904118796315. Heath, S., Brooks, R., Cleaver, E., & Ireland, E. (2009). Researching young people’s lives. Thousand Oaks: Sage. National Health and Medical Research Council. (2007). National Statement on Ethical Conduct in Research Involving Humans. Canberra: AusInfo. Roberts, L. D., & Allen, P. J. (2015). Exploring ethical issues associated with using online surveys in educational research. Educational Research and Evaluation, 21(2), 95–108. Punch, K. F. (2014). An introduction to social research: Quantitative and qualitative approaches (3rd ed.). London: Sage. Wellington, J. (2015). Educational research: Contemporary issues and practical approaches. London: Bloomsbury.

10.1.5  Readings About Survey Design Ary, D., Cheser Jacobs, L., Sorenson, C., & Walker, D. A. (2018). Introduction to research in education (9th ed.). Belmont: Cengage Learning.

10.1  Further Reading

197

De Vaus, D. A. (2002). Surveys in social research (5th ed.). Crows Nest: Allen & Unwin. Nardi, P. M. (2016). Doing survey research: A guide to quantitative methods (3rd ed.). New York: Routledge. Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A. (2010). Mixed research as a tool for developing quantitative instruments. Journal of Mixed Methods Research, 4(1), 56–78.

10.1.6  Readings About Validity and Reliability Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). New York: Routledge. Drost, E. A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38(1), 105–124. Morrell, P. D., & Carroll, J. B. (2010). Conducting educational research: A primer for teachers and administrators. Rotterdam: Sense Publishers. Wilson, E. (Ed.). (2017). School-based research: A guide for education students. London: Sage.

10.1.7  Readings About Quantitative Analyses Field, A. (2013). Discovering statistics using IBM SPSS statistics (4th ed.). London: Sage. Muijs, D. (2011). Doing quantitative research in education with SPSS (2nd ed.). London: Sage. Teo, T. (2013). Handbook of quantitative methods for educational research. Rotterdam: Sense Publishers. Wilson, E. (Ed.). (2017). School-based research: A guide for education students. London: Sage.

10.1.8  Readings About Reporting Educational Research Bell, J. (2010). Doing your research project (5 ed.). Berkshire: Open University Press. Fallon, M. (2016). Writing up quantitative research in the social and behavioral sciences. Rotterdam: Sense Publishers. McKenney, S., & Reeves, T. C. (2018). Conducting educational design research. London: Routledge.

198

10  Conclusion and Further Reading

Norris, J. M., Plonsky, L., Ross, S. J., & Schoonen, R. (2015). Guidelines for reporting quantitative methods and results in primary research. Language Learning, 65(2), 470–476. doi:10.1111/lang.12104 Taylor, J., Furtak, E., Kowalski, S., Martinez, A., Slavin, R., Stuhlsatz, M., & Wilson, C. (2016). Emergent themes from recent research syntheses in science education and their implications for research design, replication, and reporting practices. Journal of Research in Science Teaching, 53(8), 1216–1231.

Index

A Age-appropriate tools, 25–27 Altmetrics, 189–190 Analysis of variance (ANOVA), 75 Asymp. Sig., 71, 105–106 Authorial voice, 183, 185 B Bias, 13, 16, 17, 182, 183 Binary, 91, 96, 160 Binomial, 41, 152–163 Bonferroni, 66, 76 Boxplot, 58 C Challenges, 5, 16, 33, 34, 182–185, 191, 193 Chi-square, 85–91, 96, 126 Cochran, 86, 96–102 Coefficient, 134–136 Collecting data, 3, 9, 14, 18, 33, 50 Concrete reference points, 27 Conference, 3, 178, 185, 190, 191 Confirmation bias, 1 Construct, 11, 12, 21 Correlation, 19, 20, 39–42, 60, 67, 75, 111–131, 133, 137, 194

Correlational designs, 19 Covariate, 155, 165, 173 Cover letter, 34–37 Cramer, 40, 126–132 D Data types categorical, 12, 40, 41, 112, 126 interval, 39–42, 91, 112 nominal, 39, 40, 126 ordinal, 39, 40, 126 ratio, 39, 40, 42, 126 scale, 40, 41 Dependent variable, 1, 9, 11, 19, 66, 67, 76, 153, 164 Descriptive designs, 21 Designing surveys, 27 Dichotomous, 40, 41, 49, 91, 96, 102, 152, 153 Double questions, 13, 27 E Effect size, 10, 196 ERA, 181 Ethics, 28, 32–34 Exact Sig., 105 Experimental designs, 18, 19

© Springer Nature Singapore Pte Ltd. 2020 S. Mat Roni et al., Conducting Quantitative Research in Education, https://doi.org/10.1007/978-981-13-9132-3

199

200 G Generalisability, 2, 21 Goodness of fit, 136, 180 GPower, 42 H Histogram, 4, 42, 43, 103, 106, 107, 120 Homoscedasticity, 137, 149 Hypotheses, 1, 9, 10, 18, 66, 133, 194 I Independent, 9, 26, 41, 66, 67, 75, 86, 96, 127 Independent variable, 9, 11, 19, 137 J Journal articles, 12, 178–182, 185 K Kendall, 39, 40, 42, 111, 120–126 Knowledge mobilisation, 177, 191 Kolomogorov-Smirnov, 42 Kruskal-Wallis (KW), 66, 75–85 Kurtosis, 84 L Language, 12, 13, 27, 177–179 Latent, 60 Likert’s scale, 41, 50, 59 Linearity, 144, 160, 171 M Mann-Whitney U, 41, 65–76 McNemar, 41, 86, 91–96, 99, 102 Mean mean rank, 40, 67, 77 trimmed mean, 51, 58 Median, 19, 39, 67, 76 Missing value missing at random, 14, 26 missing completely at random, 14 missing not at random, 13, 14, 26 Mixed-methods, 3, 10, 21, 22, 25, 31, 182–184, 195 Monotone, 58, 59 Multicollinearity, 137 Multiple regression, 134, 137

N Nagelkerke, 158, 163, 167, 174 Non-parametric, 2–4, 39, 41, 42, 75, 111, 120, 193–195 Non-response bias (NRB), 65 Normal non-normal, 136 normality, 4, 120, 136, 137, 147 O Outcome variable, 67, 96, 101, 133–137, 144, 152, 153, 163, 164 Outliers, 51, 52, 58, 137, 147, 148, 164 P Pair-sample t-test, 43 Pearson, 42, 86, 111, 120, 126 Piloting, 11, 13, 18, 27–32, 36 Plain English, 29, 34, 178–179, 186–190 Post-hoc, 66, 76, 79, 99 Predictors, 133–137, 144, 153, 160, 164, 171 Priming, 27 Q Q-Q plot, 42, 43, 137, 145, 147, 148 Quality, 9, 26, 31, 32, 34, 40, 51, 179–181, 184, 185, 190 Quartile, 181 R R2, 135, 136 Recode, 59, 60, 64 Regression binomial, 152–163 linear, 144 logistic, 152, 160, 163, 171 multinomial, 163–171, 174 Reliability, 11, 12, 18, 21, 25, 164, 194, 197 Research questions, 1, 2, 8–10, 12, 18, 186, 194 Research translation, 8, 10, 191 Residual, 135, 137, 149 S Sample sampling strategies, 10, 16 size, 194

Index

201

Satisficing, 13, 26–29, 31 Scatter plots, 42, 112, 115, 118, 121, 135, 137, 144 School relationships, 37 SCImago, 181, 182, 190 Shapiro-Wilk, 42, 44, 120, 136, 145 Social media, 16 Spearman, 39, 40, 111–120, 126 Standard deviation, 52, 58, 137, 164 Statistical power, 42 Sum of squares, 135 Survey design, 13, 196–197 Syntax, 47, 48

Transformation log transformation, 147, 158 T-test independent, 48 paired samples, 48

T Thesis by publication (TBP), 178, 180, 184–186 Tolerance, 137

W Wilcoxon, 102–107 Winsor, 58, 148 Write up, 2, 177–191

V Validity, 11–13, 16, 32, 194, 197 Variables, 1, 7, 40, 47, 65, 111, 133, 194 Variance, 58 Variance inflation factor (VIF), 137