Do Politicians Even Care About Research? Two Experiments Say “Yes”

When I tell people I do policy analysis, one of the most common questions I get is this: “Do politicians even care about what the research says?”

It’s a fair question, and usually I respond with anecdotes from history or from my personal experience that show that (at least sometimes!) policymakers do listen to research.

New research published in the National Bureau of Economic Research’s Working Paper series is now lending experimental credence to this claim. Researchers at Harvard, Columbia, UC Davis, and Innovations for Poverty Action collaborated with the National Confederation of Municipalities in Brazil to run two large experiments on policymakers’ demand for and use of policy analysis.

The first experiment involved 900 municipal-level officials across 657 municipalities in Brazil and was conduced at a series of Nation Confederation of Municipalities meetings across the country. The study gave local officials surveys on tablets with information on early childhood education programs, which are funded and administered at the municipal level in Brazil. The officials were then given lottery entries for a trip to Boston and a private tour of Harvard University. They were given the option to “purchase” information on the impact of early childhood education programs on outcomes like test scores and cognitive skills down the road by trading in these lottery entries for the results of peer-reviewed studies.

So did the local officials play along, trading their opportunity for a trip to Boston for information on the impact of early childhood education? Actually, they did. The average official was willing to trade a number of lottery entries for the trip, on average amounting to a value of $36 US. This means that officials, learning about a topic they did not seek out, were willing to pay the equivalent of $36 on average to find out more information on that topic. The study also found that policymakers were willing to pay more for the results of studies with larger sample sizes and changed their beliefs about the impact of early childhood in line with the research presented after exposure to results.

The second study randomly invited 881 Brazilian mayors via email and text message to a session on reminder letters for tax compliance at a large National Confederation of Municipalities meeting. Tax compliance is an important issue in Brazil where some estimate that about 20% of property taxes go unpaid every year. Reminder letters are also a low-cost and administratively simple policy to implement with a strong evidence base of its effectiveness, making it a strong candidate for a policy.

Of the invited mayors, 38% attended the session.[1] An additional 1% of uninvited mayors at the conference attended the open session as well. At the session, an instructor presented on policy impact, cost-effectiveness, and impact evaluation research around taxpayer reminder letters, then gave an overview of the literature, a list of characteristics of effective letters, and sample letters. Both treatment and control municipalities were interviewed 15 to 24 months to see if they were using reminders for tax compliance more than a year after the conference.

It turns out that the session had a significant impact on participants compared to the control group. Mayors who attended the session were 33% more likely to be sending tax compliance reminder letters 15-24 months out than mayors that did not attend the session. They also were more likely to have beliefs in line with the research 15-24 months than control mayors who did not attend the session.

There are reasons to still be skeptical. Maybe Brazil has a different political environment than the United States (though their most recent national election suggests the two countries have at least one thing in common) that makes them more amenable to research than the United States. Maybe these issues just work because they’re below the “policy waterline” as we’ve wrote about here before.

Despite these objections, this study does show one thing: there are certainly times when research and analysis matters and policymakers actually want it. More sources of rigorous, nonpartisan analysis will only improve policymaking at all levels of government, including in states and localities.

[1] Notably, younger and college-educated mayors were 7 to 15 percentage points more likely to attend the session.

Good budgeting puts people first

Columbus City Council recently voted to authorize the 2019 Capital Improvement Budget, a $940 million infrastructure plan for the city for the upcoming year.

The department that was awarded the most capital funds by far this year was the Department of Public Utilities, which will manage almost 60 percent of this year’s capital budget. This includes more than $340 million for improvements to the city sewer system and more than $190 million in improvements to the city water system.

Behind the Department of Public Utilities comes the Department of Public Service, which will manage about 18 percent of the capital budget. The vast majority of this funding is $160 million in transportation system funding. Another $9 million will go towards refuse collection.

Next comes the Department of Recreation and Parks, which will manage about 8 percent of capital budget funds, with $72 million allocated towards improvements to the city parks system.

Other departments receiving substantial allocations in the capital budget are the departments of Development (4 percent), Public Safety (4 percent) and Finance and Management (2 percent). These allocations include $34 million of economic development funds, $21 million in fire system improvements, $20 million for construction management, $8 million in police system improvements and $7 million in housing funds.

The remaining 4 percent of the capital budget funds technology administration, education, the City Auditor’s office, neighborhood programs, municipal courts and local health and human services.

It is sometimes said that “a budget is a moral document.” What can we learn about the values of the City of Columbus from its capital budget?

Unfortunately, “moral audits” are not simple. A press release from the city trumpeted $14 million in early childhood spending, $11 million for a new fire station, $30 million in road maintenance, $182 million in sewer, water and street lighting improvements, and $10 million on housing in the capital budget. This articulation of a quarter of the spending in the capital budget shows us items the city is proud to fund.

Internal reports by the city, though, have said that the city should be spending twice as much on road maintenance as it currently spends. This estimate may be high, though, since it is drawn from engineering assessments of roads, not economic assessments of their ability to promote safe, free movement throughout the city. Advocates for spending on other categories might have other reasons to quibble with the spending decisions in the capital budget.

The biggest limitation we have in budgeting is the imbalance between fairly comprehensive accounting of costs and minimal accounting of benefits. What will the economic impact of this capital budget be? How will it impact poverty and inequality in the city? What will it do for educational attainment, health, housing and food security of city residents?

Ultimately, these outcomes are the true indicators of the success of a city, and budgets that adhere closest to these goals will be most successful at attaining them.

This column first appeared in Columbus Alive.

Scioto Analysis Research Highlighted by Society for Benefit Cost Analysis

This Spring, Scioto Analysis carried out a study of Ohio’s policy analytic landscape. Today, Scioto Analysis Principal Rob Moore contributed a piece to the official blog of the Society for Benefit-Cost Analysis on the study. The piece contains an overview of the major findings of the study as well as information on the methodology and context for the study.

Moore’s piece can be found here.

Scioto Analysis Principal Rob Moore Profiled in Denison Magazine

Scioto Analysis Principal Rob Moore was profiled in this spring’s issue of Denison Magazine. In the profile, Moore talks about happiness economics, Scioto’s partnership with Gross National Happiness USA to measure Ohio’s recovery from the Great Recession using an alternative economic indicator, and the importance of measurement and analysis in good public policymaking.

The full profile can be found here.

Hawks, Doves, and Wonks

It’s June of an odd-numbered year, which means that state legislatures across the country are debating and passing budgets. As I write this blog, about half the states in the country have passed a budget and another nine are awaiting the governor’s signature.

Luckily for state legislatures, budgeting this year has been easier than in past years with a strong national economy and bullish state revenue projections for the next year. This has led to less consternation than in many other years. On top of this, only one state in the country, Minnesota, has a partisan split between its two houses, and 37 states have same-party control of both houses and the governor’s office, leading to less reason for partisan dispute in the budgeting process.

Despite these trends, we still hear the same rhetorical dynamics that come to the fore during budgeting bandied about during this budget season: budget hawks worried about rising Medicaid costs and trying to rein in spending, budget doves advocating for more spending on social services, and policymakers trying to sort out the arguments coming from both sides.

In his 1964 public administration classic The Politics of the Budgeting Process, public policy theorist Aaron Wildavsky lays out a distinction between different actors in the budgeting process.

In central offices (such as the federal Office of Budget and Management or state budget offices), there exist guardians of public funds. These are the budget hawks: actors who are primarily concerned with fiscal stability. Their charge is to reduce spending, increase revenues, and to make sure the state balances commitments across a range of different policy goals. These hawks see expenditures as costs and revenues as benefits.

Agencies, by contrast, are primarily populated by spenders. These are budget doves: they are close to the programs and see these programs working on the ground. They see places where more money would make their programs more successful, so they are advocates for more spending on programs. These doves tend to see expenditures as benefits and often see considerations of costs as imposed externally, more abstract than the benefit that money could create on the ground in front of them.

Where does the analyst fit into this dynamic? Well, the dispassionate analyst’s job, whether she is working for an agency, a state budget office, or a legislature, is to help policymakers get a better grasp of how costs and benefits accrue to all of society. This is why costs in a cost-benefit analysis are measured by how a policy hurts the economy or makes people sicker or less educated, not just in how policies impact the state budget book. This is why benefits are measured by how policies grow the economy in a cost-benefit analysis or how they increase education, health, or other key outcomes in a cost-effectiveness analysis, not just by counting how many dollars are spent.

Hawks and doves will always be here, but increasingly wonks are raising the quality of discussion around public budgeting. Budgeting processes that look past pure revenue and spending and give policymakers a broader view of what costs and benefits they are levying on society lead to more informed decisionmaking. And in the end, that’s the role of analysis: to give decisionmakers the data they need to make more informed choices.

Vehicle registration fees curb Columbus’ car addiction

It’s been a banner year for transportation policy in Ohio.

Gov. Mike DeWine’s deal with the legislature this spring to raise Ohio’s gasoline tax will pour millions of new dollars into local infrastructure coffers.

A less high-profile change in the recent transportation budget was recently acted on when Columbus City Council increased the vehicle registration fee by $5, bringing the total annual vehicle registration fee in Columbus to $65.

Cars exact costs on society in the form of wear and tear on roads, congestion, crashes and emissions. The more cars on the road, the more taxpayers pay to maintain them, the more commuters pay in hours lost to traffic, the more riders pay as victims to crashes and the more residents pay with asthma and mortality associated with emissions.

Columbus’ growth rate will exacerbate these problems. Last week, Columbus officially passed San Francisco in central city population size. Over the past few years, Franklin County has become the most-populated county in the state, and the Columbus metropolitan area passed the Cleveland metropolitan area in size.

More people means more cars, which means more infrastructure damage, congestion, crashes and emissions.

Fees are an efficient way to internalize these costs, encouraging people to drive less and raising funds for infrastructure maintenance, poverty alleviation and economic development investments. While a per-mile fee would better capture these costs, a vehicle registration fee is a tool city leaders actually have at their disposal to control car costs.

A limitation of local tools like a municipal registration fee is its local nature. While a small increase in the cost of owning a car in the city will make changes on the margins in Columbus car ownership patterns, it does nothing for commuters living in suburbs and townships not subject to the fee.

That being said, Columbus is helped here by its relatively large geographic footprint and its strong enforcement mechanisms. We all know people who have received failure to register citations: The city takes it seriously.

In the future, the city could benefit from partnerships with the county and suburban communities if it wants to create an efficient vehicle registration fee. The Mid-Ohio Regional Planning Commission could be a facilitator for this collaboration, as a table of local governments in the Central Ohio region.

Ultimately, better abatement of the societal costs of driving will hinge on future collaboration with the state. The most recent transportation bill shows what a state listening to and collaborating with local government can lead to. Gas taxes and registration fees will bring the private cost of driving closer to the social cost.

Who knows, maybe this collaboration will open the door for even more innovative solutions like vehicle miles traveled fees and congestion pricing.

A wonk can dream, right?

This column first appeared in Columbus Alive.

How is “policy analysis” different from “policy research?”

When people at a party ask me what I do, I tell them literally: I’m a policy analyst. After the blank stare, I then decide how I want to answer.

If I’m feeling like pivoting towards talk about which movies were snubbed at the Oscars or what a post-Urban Ohio State athletic program looks like, I usually just say “I figure out if laws work.”

If there seems like potential interest, though, I’ll use this analogy:

“Policy analysis is to economics as engineering is to physics.”

Yes, policy analysis brings in insights from political science, sociology, and even sometimes psychology. But at its heart, the sort of work analysts do is figuring out how changes in public sector policy impact the amount and distribution of resources in society and the well-being of members of that society.

So just as the engineer applies the lessons of physics to practical problems, like building a bridge or wiring a house, the policy analyst applies the lessons of economics to practical problems such as local economic development, poverty alleviation, and improving population health.

David Weimer and Aidan Vining make a distinction in their policy analysis textbook Policy Analysis: Concepts and Practice between “academic social science research,” “policy research,” and “policy analysis.”

Academic social science research is, according to Weimer and Vining, focused on constructing and testing theories for the purpose of understanding society. Academic social science research uses rigorous methods to find “truth,” and rarely faces external time constraints. This makes it often irrelevant to the information needs of decision makers.

Policy research is one step closer to policy relevance. Policy research is the practice of trying to analyze the effects of public policies already in place or project large-scale social impacts of public policy structure. Policy research is how I would classify much of the work being done in think tanks: it is focused on analyzing policy to see what its impact is, but it still is rarely directly applicable to policy problems. Much of the work that happens at the National Bureau of Economic Research gets into policy research territory.

Policy analysis, by contrast, is designed to bear directly on present public policy problems. Policy analysis is the systematic comparison of policy alternatives in order to inform a decision with a short time window. Policy analysis is often housed directly within government offices, such as cost-benefit analysis conducted by agency analysts at the federal level. Analysis synthesizes policy research and sometimes academic social science research to compare alternative public policy options and project their impacts for specific public policy decision makers.

Policy analysis has its limitations. The necessary time constraints of the political process give analysts less time to divine absolute “truth.” That being said, these limitations are far from damning of the project of policy analysis. A policy decision making process without analysis is much scarier than a policy decision making process with it.

Analysis is the public sector’s version of the “valley of death.” While the United States boasts both the most robust research university system in the world and an extremely well-funded industry of lobbyists, advocates, and activists, the public sector and public sector-adjacent analysts who translate and distill research for policymakers are few and far between. Recent research by Scioto Analysis suggests that a full cost-benefit analysis, the most rigorous form of policy analysis, has not been carried out in Ohio in over a decade.

If you find this problem interesting, feel free to email me and we can talk about ways to work together solving it. If not, I’m certainly happy to talk film or football.

Meet Our Data Science Intern, Tong Zhou!

On Monday, analyst Tong Zhou started as a data analysis intern at Scioto Analysis.

Tong is an undergraduate student majoring in Mathematics and Data Analysis with a concentration in economics at Denison University. He brings a wealth of experience as an economic analyst and a data analyst, having done research on criminal justice policy and insurance business analysis for Deans & Homer.

Tong will be at Scioto this summer leading an analysis of the state earned income tax credit. This work will be focused on the economic growth, poverty alleviation, and human development impacts of proposed changes to the state earned income tax credit.

“I look forward to investigating a topic that touches the lives of millions of Ohioans every year,” Zhou said.

Join us in welcoming Tong!

Tax Incentives Aren't "Free Money"

City Council recently voted to provide about $3.4 million in tax incentives over the next six years to Root Insurance, Columbus’s digital auto insurance tech “unicorn.”

While the spectacle of last year’s Amazon HQ2 bidding war has soured many on economic development incentives, such incentives can still be helpful for cities looking to entice businesses. Incentives are not costless, though, and companies like Root don’t take relocation lightly and may have stuck around without incentives. Foregone tax revenue spent on incentives is revenue not spent on local programs like safety, education and infrastructure.

Upjohn Institute Economist Timothy Bartik published a report last year that sheds some light on this tradeoff. He found that incentives tend to boost local wages as a whole, though the impacts are usually regressive, accruing mainly to high- and middle-income workers and decreasing wages for low-income workers because of decreased revenue for programs to support them.

Bartik’s study provided some insights for policymakers interested in boosting local wages through economic development incentives. Let’s see how the Root deal stacks up to Bartik’s findings.

Most importantly, Bartik found that customized services such as job training and manufacturing extension partnerships can affect location and expansion decisions 10 times as much as tax incentives. These incentives are also progressive since low-income workers get jobs at local businesses providing this support. Unfortunately, the Root deal is being paid out through tax incentives, so Columbus was unable to fulfill this best practice.

Also important is targeting firms that utilize local suppliers and create jobs locally. Manufacturing firms are especially good at this, inducing as many as six local jobs for every direct job they create and indirectly creating jobs for low-income workers, making incentives for manufacturing more progressive than your average incentive.

Insurance company incentives are also more locally stimulating than average, but only create about 3.5 jobs for every direct job created, according to estimates by economist Josh Bivens. This means the Root deal does moderately well on this metric, though it falls short of a deal for a manufacturing plant.

Another tool is to front-load tax incentives in the first year. This takes advantage of the fact that private firms are more shortsighted than the public sector, making the incentive stronger. Unfortunately, the Root deal will be paid out through income tax abatements over the next six years, so Columbus did not take advantage of this difference.

All in all, the Root deal will probably grow local wages, though the wage growth will largely go to high earners. This will be paid for out of $3.4 million in foregone tax revenue that could pay for pre-k slots, police officers and road maintenance. While everyone wants a unicorn in their stable, we shouldn’t forget where its hay comes from.

This column originally appeared in Columbus Alive.

Let's Think Bigger about "The Economy"

What is “the economy?”

This might seem on its face an esoteric question, but it has ramifications for how we approach public policy questions. This question has special relevance for those of us who conduct cost-benefit analysis, which has been called the “gold standard in applied welfare economics.”

At times, I have referred to cost-benefit analysis as a tool to study how the state can grow or shrink the economy. This has often led to protests from even experienced policy analysts, saying that cost-benefit analysis transcends economics by studying impacts like nonmarket goods such as those with existence value and those generated via externality.

Superficially, these protests are either semantic arguments or misunderstandings of what the undertaking of economics is all about.

In his classic Essay on the Nature and Significance of Economic Science, mid-20th century economist Lionel Robbins puts forth the most commonly-accepted current definition of “economics”:

Economics is a science which studies human behaviour as a relationship between ends and scarce means which have alternative uses.

Robbins describes in his essay that economics is not an exercise in classification (picking out kinds of behavior, like purchases and work), but an exercise in analysis (picking out an aspect of behavior, in particular how human beings behave under conditions of scarcity).

Cost-benefit analysis is so powerful because it attacks the question of how to allocate scarce resources head-on by focusing on a specific question: how do we maximize use of social resources as defined by maximization of social welfare?

Thus, the right way to think about “the economy” is not “the S&P 500” or “gross domestic product,” it is how resources are used to give people more of what they want. Economics is not about increasing jobs or firms or even median income, it’s about maximizing social welfare.

But what does “social welfare” mean? This is where things can get tricky. Some say that maximization of social welfare is equal to the maximization of aggregate social surplus. This is what cost-benefit analysis directly answers. Others argue that welfare is maximized by conferring rights and minimizing inequality, justifying it only by improvement in the lot of the worst well-off. Some argue that welfare is maximized by giving everyone a baseline of basic goods needed to facilitate human development. Still others argue that welfare should be measured by how people assess their own lives.

This ambiguity about the definition of “welfare” has led some down some strange rabbit holes to justify the use of cost-benefit analysis, arguing that cost-benefit analysis’s job is to quantitatively estimate root moral intuitions about what is good for people’s lives. This approach tries to separate people’s desires from their well-being, saying that a desire for certain outcomes in the world do not impact an individual’s well-being and that value can only be defined as that which is directly “experienced” by an individual. Under this point of view, someone cannot “experience” wilderness that is preserved that they do not visit.

This definition of “experience” is arbitrary, though. Do people “experience” a collapsing house across the street from theirs that drives down property values? What about hate speech? The “direct experience” test relies much more on capricious judgment calls by the analyst than simply eliciting willingness to pay for outcomes.

Cost-benefit analysis doesn’t need to provide a comprehensive definition of welfare in order to be of value. The classical definition of welfare as defined as the maximization of social surplus on its own provides policymakers with much more information than they would have otherwise without having to smuggle egalitarian, human development, and subjective well-being considerations into its calculation. Maximization of social surplus can then be balanced against these other goals.

At Scioto Analysis, we ask people to think bigger about what constitutes “the economy.” We ask policymakers to consider the tradeoffs inherent in all policy decisions: that social surplus, poverty, inequality, human development, and subjective well-being all exist in an ecosystem that is characterized by scarcity of resources. The economy is much larger than stocks or GDP, it is about how we use social resources to make lives better.