You are here

Who's afraid of evidence-based policymaking?

Oct 12,2021 - Last updated at Oct 12,2021

By Ori Heffetz and John A. List

ITHACA/CHICAGO — Without rigorous research and open inquiry, the breakthroughs that have defined the modern era, saving countless lives and delivering tremendous economic growth, never would have happened. From discovering the laws of physics and the germ theory of disease to developing public policies, scholars have used experimentation to move society forward.

Now, as societies struggle with reviving travel, reopening schools, and workplace safety in the shadow of new COVID-19 variants, social experiments are urgently needed to ensure that we implement policies with a proven record of success.

In doing so, we will be building on a storied tradition. In 1881, Hippolyte Rossignol, a famous French veterinarian who was sceptical of the germ theory of disease, challenged Louis Pasteur to test his hypothesis by vaccinating animals on his farm outside Paris. Pasteur had no choice but to accept the public challenge, even though no vaccine had ever been tested outside the laboratory.

On May 5, 1881, a couple dozen animals at Rossignol’s farm were inoculated against anthrax (and received another “protective injection” two weeks later). A similar group of animals received no vaccine. On May 31, both groups were injected with virulent anthrax. Two days later, a group of farmers, veterinarians, pharmacists, and agriculture officials gathered at Rossignol’s farm to observe the results. Pasteur’s theory was confirmed: all of the vaccinated animals were alive and well, while the unvaccinated were dead, dying, or in bad condition.

We owe much to such early experiments, which have officially come to be known as randomised controlled trials. RCTs play two important roles: they help scientists push science forward; and they help convince the rest of society to trust that science.

Just over 140 years after Pasteur’s experiment, the world held its breath waiting for the clinical-trial results for newly developed COVID-19 vaccines. Once these RCTs were concluded, with strikingly successful results, governments rushed to approve the new vaccines, and countries raced each other to get supplies.

Since Pasteur laid the groundwork for controlled medical trials, they have become the scientific gold standard. And since 1963, the US Food and Drug Administration has required RCT-based evidence before allowing commercial use of new pharmaceuticals. Much of the world now is either vaccinated or eagerly awaiting doses, because people implicitly trust the science and the reliable, transparent, publicised experiments that underpin it.

But RCTs are not limited to the medical sciences. Since the mid-twentieth century, many social experiments also have followed this process. One such experiment grew out of the political debate over existing and alternative welfare programs. Sponsored by the Office of Economic Opportunity, the “New Jersey Income Maintenance” experiment explored the behavioural effects of income-supplementation programmes, yielding insights that still influence the design of public policy today.

Unlike medical trials, however, social experiments, large-scale, publicly funded public-policy RCTs, have not become the gold standard. In many cases, critics argue that such experiments are inherently unfair. For example, why should some people or activities benefit from a lower income-tax rate than others?

But such concerns are often misplaced. We need far more social experiments, and governments should conduct such experiments routinely as an essential part of the policymaking process. Economists’ embrace of the experimental approach in recent years, in the lab, online, in firms, and in remote places, has led to many scientific breakthroughs, duly recognised with several Nobel prizes.

The problem is that governments have been painfully slow to follow suit. Our own attempts to convince policymakers to conduct large-scale social experiments have so far been disappointing. The main objections we hear are based on “fairness” arguments that are inherently at odds with the scientific concept of a control group. Ironically, this resistance to experimentation could eventually lead to less progress and even less fairness.

Working in New York City with fellow economists, we attempted to use the scientific approach to explore the effects of fines, penalties, and deadlines associated with driving and parking violations. The stock response from city officials went something like, “It is not fair to randomise, to charge people different amounts.”

Similarly, when we launched a pre-kindergarten programme for underprivileged children in Chicago a decade ago, school boards, administrators, and the public pushed back against the idea that only some underprivileged children would be served. Never mind that the point of the study was to help all underprivileged children by identifying best practices for teaching cognitive and executive function skills.

And most recently, when advising a foreign government on its economic response to COVID-19, we encountered policymakers who vehemently resisted using RCTs, even for answering life-and-death questions related to lockdowns, mobility restrictions and school reopenings.

Without evidence about what works best, governments end up effectively running a grand experiment on all of us, only without the appropriate controls. Policies based on weak evidence have been applied nationally and even globally, with costly results. Indeed, almost two years into the COVID-19 pandemic, and despite an unprecedented flow of daily data from the entire world, we still have a surprisingly limited understanding of the health, economic, and other effects of even extreme policies such as lockdowns, school closures, and mobility restrictions. Policymakers have, to a large extent, been flying blind.

Evidence obtained through formal experimentation can save lives, especially when guided by theory and combined with other types of evidence. Professional researchers are expected to design experiments carefully, abide by the highest ethical standards with respect to consent and potential harm, put all appropriate safeguards in place to minimise the risk to participants, and open their work to assessments by institutional review boards.

Scientists and scholars need to understand the political implications of their experiments from a policymaker’s perspective, and to build trust and strong partnerships throughout the process. But for evidence-based policy to win out, governments ultimately must recognise that they cannot afford to exclude the crucial evidence that social experiments provide.


Ori Heffetz is professor of Economics at Cornell University and the Hebrew University of Jerusalem. John A. List is professor of Economics at the University of Chicago. Copyright: Project Syndicate, 2021.

42 users have voted.

Add new comment

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
9 + 7 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.


Get top stories and blog posts emailed to you each day.