Critical Statistics: Seeing Beyond the Headlines [1 ed.] 1137609796, 9781137609793

Table of contents :
Contents
LIST OF FIGURES AND TABLES
Figures
Tables
Abbreviations
Preface
TOUR OF THE BOOK
Headlines
Boxes
Summaries
Terminology used in this chapter
‘Seeing Beyond the Headlines ’toolboxes
Exercises
Italic, Bold and Underline
GUIDE TO THE WEBSITE
The Critical Statistics website
ACKNOWLEDGEMENTS
1: 99% of Statistics are made up
On bullshit
The world runs on numbers
Sometimes contraception doesn’t work
Statistics in the fake news era
Don’t be part of the problem
Statistics for social science students
Numbers and your degree
Numbers and your career
2: Where do NumbersCome From?
The wires
Making the news
#nofilter
Where’s the harm?
A lie can run around the world before the truth can get its boots on
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. WHICH ORGANISATION OR PERSON PRODUCED THE STATISTIC?
2. IS THE ORGANISATION/PERSON LIKELY TO HAVE AN AGENDA?
3. DOES THE JOURNALIST CHALLENGE THE STATISTIC?
Exercises for Chapter 2
Exercise 1: Follow the press-release
Exercise 2: Zombie statistics
3: SAMPLES, SAMPLES EVERYWHERE …
It’s samples all the way down …
A lot of Australians don’t believe in climate change
1.9 unemployed people for every vacancy
Canadians, despite being Canadian, still sometimes kill each other
Swords and dragons: not just for geeks any more
Samples almost all the way down …
Size matters
Brexit errors
Percentage points
Margins of error: the maths bit
Low fidelity
Numbers other than percentages
Bad samples
Spotting biased samples
Self-selected samples
Scientific surveys
Non-response bias
Sampling beyond surveys
The magic of sampling
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. HOW BIG IS THE SAMPLE?
2. WHAT IS THE RISK OF SAMPLING BIAS?
3. WHAT IS THE RISK OF NON-RESPONSE BIAS?
Example
Exercises for Chapter 3
Exercise 1: Is it a good sample?
Exercise 2: Surveying the headlines
4: MEASURE FOR MEASURE
The dark side of immigration in open, generous Sweden
Define your terms
Two is the loneliest number
How to lie with definitions
Counting is hard
Is racism a thing of the past?
Asking the right questions
Tracking down the source of the ‘160,000’ figure
Validity and reliability
There’s no such thing as a perfect statistic
Summary
Terminology used in this chapter
Seeing beyond the headlines
Exercises for Chapter 4
Exercise 1
Exercise 2
5: What Does it Mean to be Average?
Average man
The mean doesn’t always mean what you think it means
Why doesn’t everyone know this already?
The median: The mean’s under-appreciated brother
What is a ‘distribution’?
Averages are not real
The mode
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. WHAT TYPE OF AVERAGE IS BEING USED?
2. ARE THERE LIKELY TO BE LARGE OUTLIERS?
3. IS THE AUTHOR ‘ESSENTIALISING’?
Example
Exercises for Chapter 5
Exercise 1
Exercise 2
6: Fraction of a Man
There are two kinds of data in the world
What’s the point of percentages?
Per (out of) cent (a hundred)
Do we need a White Lives Matter?
Does gun violence mean the USA is deadlier than a warzone?
Percentages – backwards and forwards
Risky business
All about that base
Likert scales
Statistics aren’t real
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. IS A COMPARISON BEING MADE IN TERMS OF RAW COUNTS WHEN PERCENTAGES (OR ANOTHER PROPORTION) WOULD BE MORE APPROPRIATE?
2. IS THE RIGHT PERCENTAGE BEING USED?
3. IF THE CLAIM IS ABOUT AN INCREASED OR DECREASED RISK, HAS THE BASELINE RISK BEEN PROVIDED?
Examples
Exercises for Chapter 6
Exercise 1
Exercise 2
7 Cause and Efect
Kill or cure
CSI: Causation
The habits of highly successful people
The drugs don’t work
Postscript – fish brain
‘Have smartphones destroyed a generation?’
Establishing causation is really hard
Establishing causation is not impossible
Natural experiments
Eliminating alternative explanations using statistics
The rush to infer causation
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. IS THIS A CAUSAL CLAIM?
2. WHY MIGHT X AND Y BE ASSOCIATED?
3. WHICH EXPLANATIONS HAVE BEEN RULED OUT? WHICH EXPLANATIONSREMAIN?
Example
Exercises for Chapter 7
Exercise 1
Exercise 2
8 Bad Graphics
Electioneering
Charts as a collection of ‘visual metaphors’
A brief history lesson
Bad charts: A spotter’s guide
Bar charts
What are they used for?
Bar chart or column chart?
What makes a bar chart misleading?
Pie charts
What are they used for?
What makes a pie chart misleading?
Line charts
What are they used for
What makes a line chart misleading?
Scatter plots
What are they used for?
What makes a scatter plot misleading?
Nonsense graphs
Why have one pie when you can have six?
Obama’s legacy in nine unreadable charts
The worst graph in the world
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. WHAT TYPE OF CHART AM I LOOKING AT?
2. WHICH VISUAL ELEMENTS ARE CARRYING INFORMATION?
3. WHAT QUANTITY DOES EACH VISUAL ELEMENT REPRESENT, AND HOW?
4. DOES THE CHART ACCURATELY CONVEY THE STORY OF THE DATA?
Examples
Exercises for Chapter 8
Exercise 1a
Exercise 1b
Exercise 1c
Exercise 2
9: Context is Everything
‘Is that a big number?’
Four questions
‘There is no epidemic of police killing black people … ’
Is the author trying to say that the number is big, or that it is small?
What contextual information does the author include?
What potentially relevant information does the author exclude?
Does the excluded information make the number seem bigger or smaller?
Emotive statistics
American carnage
Is the story trying to say that the number is big, or that it is small?
What contextual information does the author include?
What potentially relevant information does the author exclude?
Does the excluded information make the number seem bigger or smaller?
The lens of history
Competing contexts: What is the Queen of England worth?
What contextual information does the author include?
What potentially relevant information does the author exclude?
Does the excluded information make the number seem bigger or smaller?
Camera tricks
Summary
Seeing beyond the headlines
Exercises for Chapter 9
Exercise 1
Exercise 2
10: Do it Yourself
The gender pay gap
The data
How big is the gender pay gap overall?
Equal pay for equal work?
Fun with definitions
Is the gender pay gap caused by sexism?
It’s not about sexism
It is about sexism
The verdict
Writing up the results
IMRaD
How to lie with true statistics
Cherry-picking
How to use statistics to tell the truth
Summary
Terminology used in this chapter
Seeing beyond the headlines
1. HAVE YOU BEEN HONEST WITH YOURSELF ABOUT THE RESULT YOU WANT?
2. HAVE YOUR METHODOLOGICAL DECISIONS BEEN SELF-SERVING?
3. HAVE YOU TRIED TO PROVE YOURSELF WRONG?
Example
Exercises for Chapter 10
Seeing beyond the headlines
1. IS THE NUMBER AN ESTIMATE BASED ON A SAMPLE?
2. WHAT IS BEING MEASURED?
3. IS THE NUMBER AN AVERAGE?
4. IS THE NUMBER A RAW COUNT?
5. IS THE NUMBER A PERCENTAGE?
6. IS THE NUMBER A RISK?
7. IS A CAUSAL CLAIM BEING MADE?
8. ARE NUMBERS BEING PRESENTED IN THE FORM OF A GRAPH?
9. IN WHAT CONTEXT HAS THE NUMBER BEEN PLACED?
10. IS THIS A NUMBER YOU HAVE PRODUCED YOURSELF?
NOTES
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Index

Citation preview

‘Never be lied to by statistics again: this book will teach you everything you need to know to combat dodgy data, spot shoddy stats, and to start constructing your own robust and reliable statistics.’ – Jude Towers, Lancaster University, UK ‘This is precisely the type of book needed to empower the public to disentangle the valid from the invalid in the information age.’ – Jennie E. Brand, UCLA, USA ‘This book is a must for those taking introductory statistics courses. It provides real-world every day examples of the use of statistics in everyday life.’ – John Jerrim, UCL, UK ‘70% of married women have cheated on their partners’ ‘32,000 people in the US die from gun violence every year’ The number is the story – but what if the story isn’t true? Statistics can inform and enlighten – but they can also mislead and deceive. This accessible and entertaining new textbook will provide you with the skills you need to understand the barrage of numbers we encounter in our everyday lives and studies. Almost all the statistics in the news, on social media or in scientific reports are based on just a few core concepts, including measurement (ensuring we count the right thing), causation (determining whether one thing causes another) and sampling (using just a few people to understand a whole population). By explaining these concepts in plain language, without complex mathematics, this book prepares you to meet the statistical world head on and to begin your own quantitative research projects. Key features include: • Abundant real-world examples demonstrating the ubiquity of numbers, where they come from and how to understand them. • ‘Seeing Beyond the Headlines’: A step-by-step guide to applying each chapter’s concepts to real statistics. • A wealth of online resources including more examples and exercises, links to key datasets and a portal to share your own examples of bad statistics which can be found on www. macmillanihe.com/devries-critical-statistics. Ideal for students facing statistical research for the first time, or for anyone interested in understanding more about the numbers in the news, Critical Statistics helps you see beyond the headlines and behind the numbers. Robert de Vries is a lecturer in Quantative Sociology at the University of Kent, UK, where his innovative Critical Thinking course introduces first-year social scientists to the mysteries of statistics. His research focuses on social and economic inequality.

Cover image by Andrew Davis

‘Never be lied to by statistics again: this book will teach you everything you need to know to combat dodgy data, spot shoddy stats, and to start c­ onstructing your own robust and reliable statistics.’ —Jude ­Towers, Lancaster University, UK

‘This book provides a straightforward and timely tutorial in how to make informed decisions regarding which statistics are to be trusted. This is precisely the type of book needed to empower the public to disentangle the valid from the invalid in the information age.’ —Jennie E. Brand, UCLA, USA

‘This book is a must for those taking introductory statistics courses. Rather than being a dry, technical textbook, it provides real-world every day examples of the use of statistics in everyday life.’ —John Jerrim, U ­ niversity College London, UK

‘This book provides a welcome complement to the wide range of “how to do ­statistics” books that are available. It takes students in measured steps, ­providing useful exercises along the way.’ —Gillian Whitehouse, University of Queensland, Australia

‘Critical Statistics provides an accessible and entertaining tour through the ways that statistics can be used to mislead us. It’s a thorough introduction for people who shudder at the thought of data, but people who see themselves as experts will learn something from this too.’ —Mark Taylor, University of Sheffield, UK

‘This is a highly readable introduction to the ways numbers are manufactured and misrepresented in today’s society, teaching the importance of thinking ­critically about statistics and showing how to do better in our own learning and research. In the age of fake news, this is essential reading for all students of the social sciences.’ —Richard Harris FAcSS, University of Bristol, UK

‘This is the perfect statistics book in an era in which it is so difficult to navigate the numbers and data we are exposed to in our everyday life. It helps the reader – anyone, from students to more expert readers – understand how difficult it is to interpret and utilize statistics in the news, and it teaches how to make better use of the incredible amount of data available today.’ —Maria Sironi, University College, London, UK

‘This is a most impressive teaching resource. De Vries adopts an e­mbedded approach to introduce students to statistical reasoning, guaranteed to increase student engagement with quantitative methods and to encourage a ­much-needed critical eye to quantitative evidence.’ —Stella Chatzitheochari, University of Warwick, UK

‘Darrell Huff’s How to Lie with Statistics is rightly a classic but after over 60 years is inevitably showing its age. Robert de Vries “seeing beyond the ­headlines” is a much needed Huff for our times. It takes the reader skilfully through the ­essentials of statistics by taking apart the way in which numbers are routinely misused by the media. The result is not only an engaging ­introduction to ­statistics but an essential guide in the “post truth” era of “alternative facts” and “fake news”.’ —John MacInnes, University of Edinburgh, UK

Critical Statistics: Seeing Beyond the Headlines Robert de Vries

© Robert de Vries, under exclusive licence to Springer Nature Limited 2019 All rights reserved. No reproduction, copy or transmission of this publication may be made without written permission. No portion of this publication may be reproduced, copied or transmitted save with written permission or in accordance with the provisions of the Copyright, Designs and Patents Act 1988, or under the terms of any licence permitting limited copying issued by the Copyright Licensing Agency, Saffron House, 6–10 Kirby Street, London EC1N 8TS. Any person who does any unauthorized act in relation to this publication may be liable to criminal prosecution and civil claims for damages. The author has asserted his right to be identified as the author of this work in accordance with the Copyright, Designs and Patents Act 1988. First published 2019 by RED GLOBE PRESS Red Globe Press in the UK is an imprint of Springer Nature Limited, registered in ­England, company number 785998, of 4 Crinan Street, London N1 9XW. Red Globe Press® is a registered trademark in the United States, the United Kingdom, Europe and other countries. ISBN 978–1–137–60980–9 hardback ISBN 978–1–137–60979–3 paperback This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. Logging, pulping and manufacturing processes are expected to conform to the environmental regulations of the country of origin. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress.

Don’t take anyone’s word for it. — Motto of the Royal Society (roughly translated)

Contents

List of Figures and Tables

ix

Abbreviations xi Preface xii Tour of the Book

xiv

Guide to the Website

xv

Acknowledgements xvii

1.

2.

99% of statistics are made up

1

The world runs on numbers Statistics in the fake news era Don’t be part of the problem

6 9 12

Where do numbers come from?

17

Making the news 20 #nofilter 23 Where’s the harm? 27 A lie can run around the world before the truth can get its boots on 32

3.

4.

Samples, samples everywhere …

38

It’s samples all the way down … Size matters Bad samples Spotting biased samples The magic of sampling

39 44 54 57 62

Measure for measure

68

The dark side of immigration in open, generous Sweden Define your terms How to lie with definitions Counting is hard Asking the right questions There’s no such thing as a perfect statistic

68 71 74 77 80 86

vii

Contents

5.

6.

7.

8.

What does it mean to be average?

91

Average man The mean doesn’t always mean what you think it means Why doesn’t everyone know this already? The median: The mean’s under-appreciated brother Averages are not real

91 94 97 99 105

Fraction of a man

111

There are two kinds of data in the world What’s the point of percentages? Percentages – backwards and forwards Risky business Statistics aren’t real

111 113 118 120 124

Cause and effect

131

Kill or cure CSI: Causation The drugs don’t work ‘Have smartphones destroyed a generation?’ Establishing causation is not impossible The rush to infer causation

133 135 139 143 148 150

Bad graphics

156

Electioneering 156 Charts as a collection of ‘visual metaphors’ 158 Bad charts: A spotter’s guide 161 Nonsense graphs 182

9.

Context is everything

192

‘Is that a big number?’ Four questions Camera tricks

194 194 206

10. Do it yourself The gender pay gap Writing up the results How to use statistics to tell the truth

210 211 224 230

Notes237 Index246

viii

LIST OF FIGURES AND TABLES

Figures 3.1 Bar chart visualising the results of the IFOP infidelity survey 52 3.2 IFOP infidelity bar chart, with error bars added 53 4.1 Screenshot captured from @SenatorBaldwin Twitter feed 75 5.1 Histogram showing the distribution of Instagram followers in your group 95 5.2 Histogram showing the distribution of Instagram followers in your group, plus Anthony 97 5.3 Histogram of a hypothetical distribution of weights 103 5.4 Histograms of possible distributions of 100m times 104 5.5 Histogram of hypothetical IQ distribution among Bingons and Cromulans 106 8.1 Photograph of James Barber’s election leaflet 156 8.2 My corrected visualisation of the figures provided in the Liberal Democrat leaflet 158 8.3 William Playfair’s bar chart of imports and exports in Scotland 159 8.4 Florence Nightingale’s polar area diagram of the causes of death of British soldiers in the Crimean war 160 8.5 Box office receipts of Transformers movies and best picture winners 162 8.6 Screen capture of Fox Business bar chart 163 8.7 Screen capture of Gizmodo’s comparison of iPad battery sizes 164 8.8 Pie chart showing the gender breakdown of movie protagonists 166 8.9 Pie chart showing the gender breakdown of movie protagonists, now three-dimensionalised 167 8.10 Pie chart of support for 2012 Republican presidential candidates 168 8.11 Line chart showing global temperatures by year 169 8.12 Screenshot captured from @NRO Twitter feed 171 8.13 Line graph showing hypothetical changes in weight over time 172 8.14 Line graph showing changes in Planned Parenthood procedures between 2006 and 2013 173 8.15 Bar chart of homicide rates by country 176 8.16 Bar chart of average homicide rates for high (Gini index ≥34) and low (Gini index