Last week, I was at the Association for Public Policy Analysis and Management’s annual research conference. In past years at this conference, I’ve spent most of my time at sessions talking about specific areas of policy research. This year however, I got to go to more sessions looking more generally about the tools of policy analysis.
One of these sessions had a presentation by someone from The People Lab at Harvard. This was not necessarily a presentation about methods like some of the more technical presenters, but rather it was what information researchers include in their papers compared to what information policymakers use to select policies.
To do this, the researchers are surveying as many local government employees as they possibly can. In the survey, government employees are shown two randomly generated policies and some key characteristics about them. For example, they might be shown two policies to address housing in a hypothetical city. Policy A has a large effect size but is high cost, Policy B has a smaller effect but a lower cost.
On each policy, the respondent is shown information about the experimental design (e.g. is it a randomized control trial or an observational study, etc), the policy’s general effects (e.g. effect size, variance, costs, etc), and some other contextual factors (e.g. political feasibility, would it require hiring new staff, etc).
Based on their preliminary results,* the researchers were able to determine which factors were most important in the decision making process for government officials. The five most important things that drove decision making were:
Short-term outcomes
Political context
Long-term outcomes
Study year
Whether a study had been replicated
The second part of this research project was to scrape data from research journals and see how often these different topics were brought up in academic research papers. Looking at the same list again, the reported percentage is the share of academic papers that include some mention of each of these categories.
Short-term outcomes - 95%
Political context - 5%
Long-term outcomes - 57%
Study year - 98%
Whether a study had been replicated - 9%
Compare this to the categories that policymakers cared the least about and we can clearly see that challenges that communicators of research need to overcome:
10. Sample size - 94%
11. Statistical significance - 93%
12. Evaluation method - 100%
Policymakers do not care as much about some of the things that researchers spend the most time thinking about. Those bottom three categories are three of the most important things I look for when trying to determine the relative quality of a research study. In an ideal world, policymakers would care more about these too since they are good indicators of the chances that a policy studied elsewhere will have similar results in their jurisdiction.
However, I don’t just think that policymakers need to care more about variance. This list also clearly shows academics some ways they can make their research more relevant. Take the relative importance of long-term outcomes. Only a little of half the studies these researchers looked at measured long-term outcomes, despite the fact that it is something policymakers care a lot about.
Additionally, policymakers care a lot about whether a study has been replicated. If an academic is only replicating a past paper, then they aren’t contributing nearly as much new information to the field. Replication studies are still extremely important for increasing our understanding of a topic, but there exists a bias against such studies.**
The final and largest gap between academics and policymakers is the discussion of the political context. While it may not be appropriate for an academic paper to take a political stance, it would be beneficial to policymakers if there was some information about the political context in which a program was studied.
There is a lot we can take away from these early survey results. We know this gap between academics and policymakers exists, and we at Scioto Analysis try to bridge that as best we can. Still, we’d be better off if policymakers and academics were more aligned on the things that matter most.
* This was presented as a work in progress. The researchers were receiving their first wave of survey results the week leading up to this conference, and they expect these results to change somewhat by the end of the project.
** Replication studies that find different effects than the original are actually preferred, but if a researcher thought a study was well done and that they would find similar results then they would be disincentivized to try and replicate it.