In my last blog, I discussed how one of the challenges with policy analysis is that it relies on data and effect estimates from other research. This means policy analysts need to be capable of understanding the quality of outside research and determining what things to include in their research. Part of this process is understanding the methods that other researchers use in their experiments.
That’s why today I wanted to talk about randomized controlled trials. A randomized control trial is often considered the strongest experimental setup under most circumstances, but as with everything in policy analysis there are better or worse implementations of it. Because of its relative importance in the research world, analysts should be aware of exactly what they are, how they work, and what to look out for when making judgments about their relative strength.
What is a randomized control trial?
A randomized controlled trial is an experiment where participants are randomly assigned to either receive treatment or serve as a control. The key insight is that because we randomly assign people to each group, we avoid selection bias. If we don’t randomly assign people to the treatment group, then we might worry that the participants are selected because they are more likely to have a positive outcome.
Imagine a city wants to understand whether a job training program helps unemployed residents find work. Simply comparing people who complete the program with people who don’t participate would not necessarily answer this question. For example, people who sign up might be more motivated to find a job. The concern is that some unobserved factor could influence their employment outcomes independently of the program.
Instead, suppose the city has more eligible applicants than available spaces and uses a lottery to offer some applicants a spot. Researchers could compare employment outcomes among those offered a spot with outcomes among those who weren’t. Because the offer was assigned randomly, we have a stronger basis for attributing differences in outcomes to access to the program rather than those unobserved differences.
One problem people have with randomized control trials is ethical concerns about not offering assistance to people who might have a real need. A classic example of this is in clinical trials where participants are facing serious health consequences and there is a new drug that might help. In order to truly understand the effectiveness of the drug, a randomized control trial might be best, but that necessarily means some people who might benefit are structurally denied access. It also means that people may randomly be exposed to side effects unknown at the time. This is why informed consent is an important element of any randomized controlled trial: so participants are aware of potential risks in trials they take part in.
Internal vs. external validity
Randomized control trials succeed at reducing selection bias, but that doesn’t mean that their results are always widely applicable. When evaluating a randomized controlled trial, it is helpful to distinguish between internal and external validity. Internal validity concerns whether the study credibly identifies a causal effect for the people and circumstances it studied. External validity concerns whether that finding applies to other populations or settings.
Our job training experiment might give us a strong estimate of what offering the program did for the people who entered the lottery. However, those applicants might differ from other unemployed residents. In effect, there is still a selection bias problem, but instead of concerning what people receive treatment we are concerned about whether the population that is eligible for the study is representative of everyone else in our society.
When people talk about how randomized control trials reduce selection bias, they are mostly talking about the internal validity of the study. If we didn’t randomly assign people to receive treatment, then our results might not be able to tell us the treatment effect even for the sample we are studying.
In some cases, randomized control trials can have even worse external validity than other forms of experiments. This is because randomized control trials are difficult to organize. The barriers that make randomized control trials difficult to perform can lead to researchers relying on less representative samples that are easier to get as participants. As an example, an academic researcher might ask the students at their university to be the participants in a study.
What policy analysts should consider
For policy analysts, the key question is whether the findings from a study apply to the policy and population we are studying. A randomized controlled trial can provide strong evidence about what happened in one setting without necessarily telling us what would happen somewhere else. We need to look closely at who participated, how the program was implemented, and whether differences between that study and our own context might affect the results.
Identifying these differences does not necessarily mean we should discard a study with less than ideal external validity. Instead, we should consider whether there is a credible way to adjust its estimates. If the researchers report separate effects for different age groups or genders, we might be able to use those estimates to better reflect the population we are studying. For example, we could weight results based on the demographics of the area we are studying. Taking this kind of approach does introduce more assumptions into our models though, so it should be used with discretion.
In general, randomized control trials are exceptional sources of information. While external validity is important for analysts who want to take research and apply it to new situations, without internal validity there is no result in the first place. Our job as policy analysts is to build on that strong evidence while being clear about the assumptions needed to apply it to the decisions we are trying to inform.
