While I was still in school, I worked as a guide on wilderness expeditions. As part of that job, I was a certified Wilderness First Responder. In that training there is a mantra about working from “ideal to real.” We’d love to work in sterile, safe environments all the time but if someone needs help while sitting in the mud during a rainstorm that's where you need to help them. Policy analysis is similar.
One of the challenges of policy analysis is its reliance on external factors for its success. Things like demographic data, effect size estimates, or methodological frameworks are all prerequisites for a strong policy analysis. While it would be extremely nice to live in an ideal world where all the data and methods are well established and credible, our actual job is to provide information in the real world when conditions are often subpar.
There are many times when analysts don’t have all the ideal prerequisites they need for an analysis. Maybe a dataset doesn’t include the population we are interested in, or maybe the policy we are studying has never been implemented in the state where we are working.
These limitations don’t mean we can’t provide useful information or that an analysis is impossible. Situations where we have to work around these kinds of limitations are fairly frequent. Instead, we have to think carefully about what the available evidence can tell us and what assumptions are needed to fill in the gaps.
What can we do when we don’t have the right data?
Data concerns are always at the front of mind when conducting a policy analysis. All statistical models are threatened by the “garbage in, garbage out” principle. Understanding how to work around these limitations is a key part of quantitative policy analysis.
One way this manifests is a situation where we have good data on the implementation of a policy in one context (say a neighboring state), but we lack the information needed to estimate what the impact will be in the context we are studying.
Imagine we are studying a rental assistance program that has been implemented in a different city. The city we are studying is smaller than the city with the existing program, so we lack some of the requisite data needed to help us estimate uptake. As a starting point, we could compare the number of renter households in each city and scale the program size accordingly. If our city has half as many renter households, we might initially estimate that a similar program would have half as many participants.
This simple model requires a lot of underlying assumptions to hold. There must be a similar share of renter households who would qualify for and participate in the program. These households must have characteristics that imply that they would experience similar outcomes. If we have information on things like differences in income or housing cost burdens, we might be able to refine that estimate. If we don’t, we need to acknowledge that our estimate relies on those populations being sufficiently similar.
If there is no suitable data available and there isn’t a credible way to adjust an estimate from another context, then we might have to rely on describing outcomes qualitatively. When providing a qualitative description, it is important to distinguish between an outcome we have evidence to expect and one that is simply possible. We might be able to explain how rental assistance could help households avoid eviction without estimating how many evictions it would prevent. That still provides useful information, as long as we explain the evidence supporting that relationship and why we cannot confidently estimate its size.
What if the available methods don’t answer our question?
As analysts, we try not to reinvent the wheel as often as possible. If a method has been shown to work well before, our first instinct is to replicate and adapt rather than search for a novel approach. Established methods give us a starting point for organizing an analysis and understanding its limitations. However, we still need to check whether their requirements and assumptions fit the question we are trying to answer.
There are of course situations where there aren’t existing methods that directly address our question. When this happens, we have to find a creative way to understand the question at hand.
It can often be helpful to start thinking through the problem logically. Say we don’t know how this rental assistance will impact health outcomes directly. We could perhaps get around this limitation by estimating the impact it has on net household incomes, and then making the connection between higher net household incomes and better health outcomes.
Why is it important to explain our assumptions?
Every time we work around one of these limitations in data or methodology, we increase the number of assumptions we are making with our models. While this is not inherently bad, it does expose us to more variability and it requires us to be more detailed about the assumptions we make and what their implications may be.
It is essential that we make all of these choices visible. It should be readily apparent in the narrative of a report what parts of the model depend on our assumptions. It can also be helpful in some situations to explain qualitatively why we believe our assumptions are reasonable and where we have less confidence.
This matters because policymakers may reasonably disagree with an assumption while still finding the analysis useful. If they can see how we arrived at a result, they can better judge whether it applies to the decision they are facing. An estimate without that explanation can appear more certain than the evidence actually supports.
How can sensitivity analysis help?
Disclosing an assumption is an important first step, but we also want to know how much it matters. Sensitivity analysis allows us to change assumptions and see what happens to our results.
Returning to the rental assistance example, we could estimate outcomes assuming the program is less effective than the study suggests, then compare those results with our main estimate. We could also test different participation levels or program costs.
Note that when we do sensitivity analysis we aren’t just seeing what happens if our model is wrong. The goal is to understand how our assumptions affect our conclusions. When we say that a policy is more or less effective, that should always be based on the best available evidence about the variation in outcomes.
Sometimes these changes have little effect on the overall conclusion. Other times, a relatively small adjustment changes whether a policy’s estimated benefits exceed its costs. Both findings are useful. The first suggests that the conclusion holds across the conditions we tested, while the second tells policymakers that the decision depends heavily on something we don’t know with much confidence. Sensitivity analysis cannot make weak evidence stronger or account for every possible uncertainty. Its goal is to communicate the uncertainty in our results.
As with wilderness medicine, policy analysis is about working from ideal to real. It would be so much easier if there was always high-quality data and highly rigorous methodological frameworks to fall back on, but often there aren’t. If we want policymakers to have better information to inform their decisions, then we need to be able to work with what we have. There are situations where these concerns make some aspects of an analysis impossible, but our goal should always be thinking about “ideal to real.” What questions can we answer with the data and methods we have available?
