Calling BS: Data reasoning during an infodemic

Banner image: Shutterstock/ShutterLibrary
Calling BS: Data reasoning during an infodemic
Like

This article is part of a series in which OECD experts and thought leaders  from around the world and all parts of society  address the COVID-19 crisis, discussing and developing solutions now and for the future. It aims to foster the fruitful exchange of expertise and perspectives across fields to help us rise to this critical challenge. Opinions expressed do not necessarily represent the views of the OECD.

To keep updated on all of the OECD's work supporting the fight against COVID-19, visit our Digital Content Hub.


By Jevin West and Carl Bergstrom

The University of Washington, where we both teach, conferred 8,853 undergraduate degrees in the 2017-2018 academic year. A mere 20 of these degrees were awarded in the field of statistics. Indeed, only a small fraction of our students receive any formal training in statistics whatsoever—and of those who do, the vast majority take only a class or two in the area. When looking at the general population, the percentage is even smaller. According to the National Center for Education Statistics, only about 11% of United States high school students completed a statistics course in 2009 [1]. World Statistics Day offers an opportunity to reflect on how we can improve data literacy in a “big data” world, not just for university students, but for everyone.   

In 2017, we developed a new type of course, designed to teach data reasoning to students in STEM and non-technical majors alike. The class, “Calling Bullshit: Data Reasoning in a Digital World” has been wildly popular on our campus, and versions are now being taught at universities around the globe. Our core mission was well summarised by John Alexander Smith over a hundred years ago. Speaking to the incoming class at Oxford, he described the sole purpose of education as teaching students how “to detect when a man is talking rot”. 

More on the Forum Network: Fighting Disinformation: A key pillar of the COVID-19 recovery by Anthony Gooch, Director of Public Affairs and Communications, OECD

We discuss misinformation and disinformation, but “Calling Bullshit” is not a class solely about fake news or a nation in an epistemic crisis. Our focus is teaching students not to be fooled by the air of legitimacy that is conferred upon quantitative information. Numbers, graphs, statistical analyses—these all seem objective, precise, scientific and incontrovertible. As a result, many do not question claims that employ sophisticated mathematics or statistics. This free pass is often unwarranted, and our goal is to teach our students not to issue it.

In the hundred-plus years since Smith’s address, higher education has excelled at teaching the mechanics of STEM disciplines. Our graduates can write code, work in the lab and perform all manner of calculations. But we have been less effective at teaching our students to think critically about what they are doing, let alone why they are doing it in the first place. In addition to the mechanics of analysing data, we should be teaching students how to reason about the statistical information they will inevitably confront on the job. We need to focus students’ attention on detecting the problems that will frequently arise in their own data analyses: data censoring, instances of Berkson’s paradox, observation selection effects and so forth. 

On our COVID-19 Hub: Transparency, communication and trust: The role of public communication in responding to the wave of disinformation about the new Coronavirus

To help students and the public develop these skills, we adopted the lessons from our classroom into a popular book: Calling Bullshit: The Art of the Skepticism in a Data-Driven World. Last January, when we sent the final proofs to our publisher, we still inhabited a world of restaurants, travel and live music. The book was released this August in a world that had been utterly transformed by COVID-19. Such massive change might have rendered the effort obsolete on arrival, but the opposite occurred. The pandemic and associated “infodemic” powerfully underscored the importance of our message. Today, social and traditional media are filled with graphs, tables and statistics about COVID-19 case counts, death rates, test sensitivities and reinfection risks. With the science around the pandemic so highly politicised, data are weaponised for persuasive effect as often as they are leveraged to advance our understanding of the novel coronavirus. Sorting through it all is difficult even for public health professionals. How can members of the public distinguish between reliable science and propaganda? 

We hope that our book provides some of the necessary skills. After finishing our chapter about causality and confounding, a reader might question the Forbes article [2] about how bald men face a higher risk of severe COVID symptoms. A peer-reviewed paper in the Journal of the American Academy of Dermatology [3] laid out the evidence and proposed an elaborate endocrine explanation. Our readers might have spotted the glaring problem that forced Forbes to issue a subsequent correction: “The study did not control for age, which is a risk factor for hair loss and severe COVID-19”. 

Calling BS on data is not just about looking smart in front of your friends: it is an act of social responsibility.

After reading our chapter on selection bias, readers might see through a story such as the one in the Washington Examiner [4] about “mask mouth”—the idea that wearing a mask for COVID causes dental problems. New York City dentists observed that the patients coming into their office in April had more severe dental problems than they were used to seeing. From this they inferred that masks must be the cause of the problem. But there is a far simpler explanation. During the peak of a respiratory pandemic, no one goes to the dentist for a routine cleaning or minor filling. People only see the dentist if they are in urgent need of immediate care. Of course the patients who arrived during the pandemic had more severe dental issues: everyone else stayed home. 

Calling BS on data is not just about looking smart in front of your friends: it is an act of social responsibility. As much as we would like to see our university issuing more statistics degrees, we are confident students in all majors can learn to spot – and refute – a large fraction of the quantitative misinformation that is out there. Common sense, clear thinking, and a little practice will get you a long way.

Related topics

Tackling COVID-19 Post-truth Future of Education & Skills

Whether you agree, disagree or have another point of view, join the Forum Network for free using your email or social media accounts and tell us what's happening where you are. Your comments are what make the network the unique space it is, connecting citizens, experts and policy makers in open and respectful debate.

Please sign in

If you are a registered user on The OECD Forum Network, please sign in