An underrated skill policy analysts need to have is understanding when something should not be included in an analysis. It often feels safer to always want to include more, but managing the balancing act of deciding when enough is enough can lead to better results.
How standing can limit results
One reason an analyst might choose to exclude something from their analysis is because they decide that it doesn’t have standing. Standing is the principle that a cost or benefit should only be included if it accrues to the group whose welfare is being evaluated. While this sounds straightforward, it can dramatically change the scope of an analysis.
Suppose a state government is evaluating a transportation project using a state-level cost-benefit analysis. The primary question is whether the project makes the residents of that state better off. If the project causes travelers from another state to save time, those benefits are certainly real, but they are not benefits to the population being evaluated. Likewise, if a policy simply shifts economic activity from one state to another, the gains in one jurisdiction are offset by losses in another and may not represent a net benefit from the perspective of the analysis.
For more information about standing, check out our State Handbook of Cost-Benefit Analysis.
When further analysis costs too much
Another reason an analyst might decide not to include something is because the marginal benefit of more research doesn’t meet the marginal cost. As with any project, there comes a point where additional effort produces very little additional value. Good policy analysis is not about measuring every conceivable impact, but about measuring the impacts that matter most.
Analysts can hone their intuition when it comes to prioritizing which inputs to measure. For example, it is probably always worth exploring a mortality impact if the policy you are analyzing might have one since reductions in mortality are often extremely valuable. Conversely, if you think that an impact is probably not very valuable, then it’s worth using your limited time and resources to explore other impacts first.
If the data isn’t good enough
One final reason you might want to exclude something from an analysis is if the quality of the data isn’t good enough to support something quantitatively. Nearly every policy has effects that are difficult to measure, and there is often a temptation to assign numbers to those impacts simply because they seem important. Unfortunately, a quantitative estimate built on weak evidence can be more misleading than acknowledging that the effect cannot yet be measured.
That does not mean these impacts should be ignored entirely. In many cases, the appropriate approach is to discuss them qualitatively while being transparent about the limitations of the available evidence. This allows decision-makers to consider potentially important effects without giving a false sense of precision.
Note that poor data quality and high variance estimates are not the same thing. At the start of this year, the Trump administration stopped counting the health benefits of environmental policies, claiming that the monetization step was too uncertain. What they should have done if they were concerned was spend more time unpacking how that uncertainty might affect results through sensitivity analysis.
Ultimately, deciding what to leave out is just as important as deciding what to include. Every policy analysis involves limited time, limited data, and limited resources. The best analysts recognize these constraints and focus their efforts where they can provide the greatest insight. By carefully considering standing, weighing the costs and benefits of additional research, and being honest about the quality of the available evidence, analysts can produce work that is not only more efficient, but also more credible.
