Calling BS: Data reasoning during an infodemic

Banner image: Shutterstock/ShutterLibrary

Like Comment

This article is part of a series in which OECD experts and thought leaders  from around the world and all parts of society  address the COVID-19 crisis, discussing and developing solutions now and for the future. It aims to foster the fruitful exchange of expertise and perspectives across fields to help us rise to this critical challenge. Opinions expressed do not necessarily represent the views of the OECD.

To keep updated on all of the OECD's work supporting the fight against COVID-19, visit our Digital Content Hub.


By Jevin West and Carl Bergstrom

The University of Washington, where we both teach, conferred 8,853 undergraduate degrees in the 2017-2018 academic year. A mere 20 of these degrees were awarded in the field of statistics. Indeed, only a small fraction of our students receive any formal training in statistics whatsoever—and of those who do, the vast majority take only a class or two in the area. When looking at the general population, the percentage is even smaller. According to the National Center for Education Statistics, only about 11% of United States high school students completed a statistics course in 2009 [1]. World Statistics Day offers an opportunity to reflect on how we can improve data literacy in a “big data” world, not just for university students, but for everyone.   

In 2017, we developed a new type of course, designed to teach data reasoning to students in STEM and non-technical majors alike. The class, “Calling Bullshit: Data Reasoning in a Digital World” has been wildly popular on our campus, and versions are now being taught at universities around the globe. Our core mission was well summarised by John Alexander Smith over a hundred years ago. Speaking to the incoming class at Oxford, he described the sole purpose of education as teaching students how “to detect when a man is talking rot”. 

More on the Forum Network: Fighting Disinformation: A key pillar of the COVID-19 recovery by Anthony Gooch, Director of Public Affairs and Communications, OECD

We discuss misinformation and disinformation, but “Calling Bullshit” is not a class solely about fake news or a nation in an epistemic crisis. Our focus is teaching students not to be fooled by the air of legitimacy that is conferred upon quantitative information. Numbers, graphs, statistical analyses—these all seem objective, precise, scientific and incontrovertible. As a result, many do not question claims that employ sophisticated mathematics or statistics. This free pass is often unwarranted, and our goal is to teach our students not to issue it.

In the hundred-plus years since Smith’s address, higher education has excelled at teaching the mechanics of STEM disciplines. Our graduates can write code, work in the lab and perform all manner of calculations. But we have been less effective at teaching our students to think critically about what they are doing, let alone why they are doing it in the first place. In addition to the mechanics of analysing data, we should be teaching students how to reason about the statistical information they will inevitably confront on the job. We need to focus students’ attention on detecting the problems that will frequently arise in their own data analyses: data censoring, instances of Berkson’s paradox, observation selection effects and so forth. 

On our COVID-19 Hub: Transparency, communication and trust: The role of public communication in responding to the wave of disinformation about the new Coronavirus

To help students and the public develop these skills, we adopted the lessons from our classroom into a popular book: Calling Bullshit: The Art of the Skepticism in a Data-Driven World. Last January, when we sent the final proofs to our publisher, we still inhabited a world of restaurants, travel and live music. The book was released this August in a world that had been utterly transformed by COVID-19. Such massive change might have rendered the effort obsolete on arrival, but the opposite occurred. The pandemic and associated “infodemic” powerfully underscored the importance of our message. Today, social and traditional media are filled with graphs, tables and statistics about COVID-19 case counts, death rates, test sensitivities and reinfection risks. With the science around the pandemic so highly politicised, data are weaponised for persuasive effect as often as they are leveraged to advance our understanding of the novel coronavirus. Sorting through it all is difficult even for public health professionals. How can members of the public distinguish between reliable science and propaganda? 

We hope that our book provides some of the necessary skills. After finishing our chapter about causality and confounding, a reader might question the Forbes article [2] about how bald men face a higher risk of severe COVID symptoms. A peer-reviewed paper in the Journal of the American Academy of Dermatology [3] laid out the evidence and proposed an elaborate endocrine explanation. Our readers might have spotted the glaring problem that forced Forbes to issue a subsequent correction: “The study did not control for age, which is a risk factor for hair loss and severe COVID-19”. 

Calling BS on data is not just about looking smart in front of your friends: it is an act of social responsibility.

After reading our chapter on selection bias, readers might see through a story such as the one in the Washington Examiner [4] about “mask mouth”—the idea that wearing a mask for COVID causes dental problems. New York City dentists observed that the patients coming into their office in April had more severe dental problems than they were used to seeing. From this they inferred that masks must be the cause of the problem. But there is a far simpler explanation. During the peak of a respiratory pandemic, no one goes to the dentist for a routine cleaning or minor filling. People only see the dentist if they are in urgent need of immediate care. Of course the patients who arrived during the pandemic had more severe dental issues: everyone else stayed home. 

Calling BS on data is not just about looking smart in front of your friends: it is an act of social responsibility. As much as we would like to see our university issuing more statistics degrees, we are confident students in all majors can learn to spot – and refute – a large fraction of the quantitative misinformation that is out there. Common sense, clear thinking, and a little practice will get you a long way.

Related topics

Tackling COVID-19 Post-truth Future of Education & Skills

Whether you agree, disagree or have another point of view, join the Forum Network for free using your email or social media accounts and tell us what's happening where you are. Your comments are what make the network the unique space it is, connecting citizens, experts and policy makers in open and respectful debate.

Jevin West & Carl Bergstrom

Co-authors, Calling Bullshit: The Art of Scepticism in a Data-Driven World

Jevin West is an Associate Professor in the Information School at the University of Washington. He is the co-founder of the DataLab, the Director of the new Center for an Informed Public at UW, and co-author of the book, “Calling Bullshit: The Art of Skepticism in a Data-Driven World”. His research and teaching focus on the impact of technology on science and society, with a focus on slowing the spread of misinformation. ------ Carl Bergstrom is a professor of Biology at the University of Washington and co-author of the book Calling Bullshit: The Art of Skepticism in a Data-Driven World. His research focuses on the flow of information through biological and social systems, from the process of natural selection to the spread of misinformation across social media.

No comments yet.