At this year’s Association for Public Policy Analysis and Management annual conference, the most talked about topic by far was artificial intelligence. As these tools become increasingly accessible, they are creating new opportunities for researchers and policy analysts to work more efficiently. However, using artificial intelligence as part of analytical workflows also forces us to be vigilant to make sure our results are actually valid. That’s why today, I want to talk about some of the ways people were using artificial intelligence at the conference and what they were doing to ensure their results were valid.
What do policy analysts need to think about when using AI?
When using artificial intelligence as a tool for policy analysis, we need to make sure that whatever we do is always both auditable and reproducible.
One of the pitfalls of today's large language models is that they will (almost) always give you an answer to the question you ask it. This means that it is our responsibility as analysts to make sure that we aren’t asking these models to perform tasks for us that they aren’t qualified to do.
For the purposes of this blog, I want to talk about artificial intelligence use in two broad categories: artificial intelligence as a brainstorming tool and artificial intelligence as a tool for performing analysis.
Brainstorming with AI
Artificial intelligence can be a really helpful brainstorming partner at the start of any project. Especially when you aren’t deeply familiar with the relevant academic literature, asking artificial intelligence to suggest some starting points or to help explain some concepts can help get a project started much more efficiently.
When I’m brainstorming with artificial intelligence, I like to think of it like a librarian rather than an expert. Librarians are extremely helpful and super knowledgeable but they aren’t experts on any one topic. The goal of a librarian is to help you find the resources you need to advance your research, not to be that resource themselves.
Current artificial intelligence models have a tendency to convince people that their outputs are never wrong. Even when the artificial intelligence models say we should take their results with a grain of salt, it feels so easy to just trust whatever they say. That’s why we have to be super vigilant: asking the librarian for help is usually a good idea, but we have to do the follow-up work to check sources and investigate these topics ourselves.
Analysis with AI
Using artificial intelligence for key parts of analysis can be extremely helpful, but this is where it becomes absolutely critical to make sure that whatever we ask AI to do is auditable and reproducible. Without these conditions being met, we can’t ensure the accuracy of the results.
Part of ensuring that our results are auditable and reproducible is making sure that we are only using AI for tasks that it is well equipped to handle. A good place to start is to make sure that we are only asking AI to produce outputs that are verifiably true or false. For example, we can check whether or not any code produced by AI works the way we want it to, but we can’t do the same if we’re asking it to offer subjective analysis.
One thing AI is extremely useful for is natural language analysis. The example I saw at the conference was a researcher who wanted to know how often tariffs were mentioned in every State of the Union address. This kind of analysis would take a long time to do manually, but is lightning quick for a large language model.
Using an example from my own research, I once was using records from a local police station and trying to classify arrest types based on the type of crime it was (violent, drug related, etc.). This ended up being a huge lift because I was dealing with the police station’s own descriptions, and there were some inconsistencies that I had to make decisions about. On top of that, there were a number of typos that I had to handle one-by-one!
This kind of language classification is perfectly suited for large language models. A careful prompt can enable AI to scan all of the text we’re studying and identify the categories we need to conduct a quantitative analysis.
Tips for using AI in policy analysis
Finally, I wanted to just highlight some of the advice other researchers were sharing regarding writing better prompts for large language models.
Be as specific as possible with analysis prompts. AI models have a tendency to go do more than they’re asked if there is any ambiguity.
Repeat your goals multiple times in the prompt. This isn’t a real person who will get annoyed with micromanaging. Repeating your request can help remove ambiguity.
Ask it for comments about its process along the way. This can help you understand what specific steps you need to pay attention to as you audit the results.
Start new chats frequently. Keeping the information the model has access to more limited will produce better results that are easier to replicate and audit.

