What’s the difference between “policy analysis” and “policy evaluation?”

I still can’t believe I was in Ohio for years before I came across the Ohio Performance Evaluators’ Group.

When I attended my first OPEG event in May of 2019, I realized I was among my people. At this conference at Otterbein University, I met dozens of evaluators from across the state, people like me who thought policy and programs should be guided by evidence, not knee-jerk assumptions or ideology. People who believed that the best that social science has to offer us can help us improve lives by giving us insight into what works in the government and social sectors.

There was one strange difference I found between the work I was doing and what OPEG members did, though. While I called what I did analysis, what many of my new companions called their work was evaluation.

Analysis and evaluation are close cousins, but not synonymous with one another. Below are some of the biggest distinctions between the two approaches.

Forward vs Backward

While both analysis and evaluation are ultimately trying to help policymakers make better decisions, analysis is focused primarily on a pending decision while policy evaluation focuses on a policy or program that is already in place.

Policy analysis is often conducted on a policy that has not been implemented yet, for instance whether or not the state of Ohio should legalize sports betting. An evaluation of such a policy would have to be conducted while the policy is being implemented or retrospectively using data that was collected during the implementation of the policy.

Because of this difference, analysis often is focused on “projection”— what a layman might call “predicting the future.” Policy evaluation, on the other hand, is focused on whether a current policy is working or whether a past policy worked. 

Microeconomics vs Econometrics 

Because of this distinction, analysis and evaluation use two different toolkits. A rigorous form of policy analysis such as cost-benefit analysis is heavily rooted in microeconomic analytical techniques, such as models for supply and demand or the theory of the firm.

Evaluators, on the other hand, use data available from implementation of a policy to estimate the impact of the policy on a population. This makes evaluators focused heavily on the effectiveness criteria: how well was a policy or a program able to bring about the results it wished to bring about? Evaluators are keyed into the elements of randomization, quasiexperimental methodology, and pre/post data in a way that analysts are only interacting with secondarily.

Internal vs External Validity

Evaluators are, at their core, focused on the evaluation of a program. Thus, the internal validity of their work is very important: how can they prove they approached a program with an objective eye and designed an evaluation that did not presuppose its own results?

Analysts, on the other hand, are much more interested in external validity. They ask the question of how they can take analogous policies in other places and use their results to project what the impact of a given policy would be in a certain place.

Analysis and evaluation are cousins, but understanding the difference between them helps someone interested in the process of evidence-based policymaking and programming understand how they fit together to make better policy and programs. That being said, good analysis draws from good evaluations and good evaluations ask questions asked by past analyses. Analysts and evaluators both have an important part to play in making evidence-based policymaking a reality, which will ultimately mean a stronger economy, lower poverty and inequality, and better lives for the general population.

Expanded Amtrak coverage ball now in Ohio’s court

A few years ago, I took a trip out to the Pacific Northwest, a part of the country I had never visited before. I stayed with college friends in Portland and Seattle then bunked up in a hostel in Vancouver. On my trips between the cities, I took the train.

A lot of us have romantic attraction to the train as a mode of transportation. I recall waiting at old train stations and hearing the announcements for boarding the train. I even remember riding on the train and the conductor making a wry joke, dryly deeming every town we passed through “The Jewel of the Pacific Northwest.”

Today, Ohio has a chance to substantially expand its passenger rail service. Currently, the only rail that goes through Ohio is a Lake Erie route that passes from Chicagoland into Toledo, through Cleveland, then out to Buffalo and Pittsburgh and a Cincinnati line that goes out to Indianapolis to the West and through West Virginia in the East. Columbus has no passenger rail.

Part of the federal infrastructure bill is $100 million in rail investment to build lines between Cincinnati, Dayton, Columbus, and Cleveland. In order to get the money and the lines, though, the Ohio state government needs to agree with the feds to start splitting the operating costs with the federal government five years after the lines are in, which would cost the state about $9-10 million a year. This would be an about-face from past policy, when Governor John Kasich a decade ago turned down a deal for a high-speed rail line to connect Cincinnati, Cleveland, and Columbus.

I don’t want to diminish $10 million: you can do a lot with $10 million. But it’s worth noting that Ohio already does a lot with $10 million. The current FY 2022 budget for the state of Ohio has over 300 line items that total $10 million or more, ranging from grants for sports events to cultural facilities lease rental bond payments to the “meat processing investment program.” Overall, a $10 million operating expense would amount to one hundredth of a percentage point of the total state operating budget this year.

Another way to think about this is in the context of the massive Intel deal. The most recent reports suggest that the scope of state spending on the project puts incentives in the $2 billion range. This means that the cost of expanding Amtrak in Ohio would take over 200 years to catch up with the one-year cost of the Intel deal.

Intel purportedly will have a large impact on the state economy. But we also know that state incentives are only a fraction of the decision making criteria for a big company location like this and that somewhere from 75% to 98% of firms would decide to make location decisions without incentives at all. This stands in contrast to the clear decision the state has currently to spend $10 million a year or pass on expanded passenger rail.

I don’t know the total economic benefits of the Amtrak deal in Ohio. What I do know is that if the state is seeing its spending as a diversified investment pool for encouraging economic growth, reducing poverty and inequality, and improving lives, then expanded passenger rail seems like a prudent asset to add to its portfolio.

This commentary first appeared in the Ohio Capital Journal.

We’re about to find out what the Intel deal actually cost us

In case you have been living under a rock for the past few weeks, the big news in economic development not only locally but nationally is Intel’s plan to build a $20 billion microchip factory in New Albany, Ohio.

President Joe Biden has come out praising the new factory, talking about the national security implications of keeping microchip creation within the United States in the face of encroachment on the industry from China.

Locally, everyone is abuzz about the economic development ramifications. State boosters have been quick to crown Ohio “Silicon Heartland” in the wake of the announcement. Ohio U.S. Sen. Sherrod Brown said “today the term Rust Belt is officially buried. Dead and buried,” arguing that when it comes to Ohio losing young people, “this will turn that around.”

All this may be a bit overblown. Data from United Van Lines out earlier this month showed that 56% of Ohio migration is currently outbound, which means that more people are moving out of the state than are moving into it. One economic development project, even a large one, would have to pack a pretty big punch to reverse a trend like this.

So let’s take a look at this project under traditional terms. By that I mean let’s look at how the Intel plan currently looks using the “rules of thumb” that leading economic development economist Timothy Bartik uses to understand tax incentives. After all, an estimated 75% to 98% of business incentives have no impact on the decisions of firms to relocate, expand or retain workers. How do we know this project isn’t the same?

Something we know about the Intel project is its location. Bartik argues that areas of high unemployment will have a larger bang for public dollar buck since economic development projects will give people opportunities to work they would not have had otherwise. According to the Bureau of Labor Market Information at the Ohio Department of Job and Family Services’s Office of Workforce Development, Licking County is 72nd out of 88 Ohio counties in unemployment. This suggests Licking County is not a particularly well-targeted location for a large economic development project within the state.

Similarly, New Albany is famously one of, if not the, wealthiest cities in the state, suggesting the area is not particularly distressed or aching for employment opportunities.

Information is still coming forth about the incentives the state is providing Intel to open this factory. What we do know is that, according to Lt. Gov. John Husted, the state is spending over $1 billion on nearby infrastructure alone. Infrastructure spending is usually a better investment for a government than cash because if a business proposition does not work out, it is a lot harder for the company to uproot and run with roads and sewer lines than with cash incentives. But $1 billion is still a massive amount of money in the context of state finances.

We have much to learn about what customized business services and cash incentives the state is providing Intel. It is easy to focus on the benefits of a project such as this, but the costs are just starting to become clear. What we do know is this: Intel will likely have to be at least as transformational as everyone is saying it is in order to be worth what Ohio is spending on it.

This commentary first appeared in the Ohio Capital Journal.

Ohio economists think gas tax freeze would hurt economy

In a survey published by Scioto Analysis this morning, 26 of 32 surveyed Ohio economists disagreed with the statement that decreasing state highway spending for the next five years by repealing increases to the state gas tax would create economic benefits that outweigh the policy's economic costs.

Among those who disagreed, many commented on the efficiency of gas taxes, stating that gas taxes ensure the people who are using roads are those who pay for them. They also noted the impact of negative externalities of driving such as air pollution, congestion, and crashes, which are mitigated by the gas tax. Many also talked about the public value of strong infrastructure in the state.

Of the two economists who believed the gas tax freeze would help the economy, neither gave optional comments. Of those who were uncertain, two emphasized the importance of where highway funds were going and which specific projects were being spent on.

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists.

How do I assemble evidence for my policy analysis?

Last year, I wrote an article on problem definition, the first step of Eugene Bardach’s Eightfold Path to more effective problem solving. Problem definition is the first step in a good policy analysis, but when we think of what it is that makes up policy analysis, the second step, to assemble some evidence, feels like the beginning of the project.

The heart of assembling evidence is assembling data on the topic at hand, which consists of learning facts relevant to the problem you have defined. When data has meaning, we call it information, because it informs our understanding of a problem. Information then becomes evidence when it becomes valuable to people who are trying to understand a problem. 

Bardach urges analysts to think before you collect data. He stresses that data collection can often feel productive when it is not: pulling together a bunch of data that does not inform or provide evidence for the analysis is ultimately not time well spent.

Economy is an important element of policy analysis. Since policy analysis usually occurs under a time limitation, making sure time is well spent is a valuable skill for a policy analyst. Economizing the evidence assembly step means understanding the value of data (how likely it is to become information and evidence) and understanding the utility of the data (how hard it is to collect it and how much time it will take). It also means leaning on educated guesses to guide your analysis: having good instincts to know what kind of data you may need to solve a problem can save a lot of time in this phase of analysis.

One key way data can be found is through review of the available research. Looking at professional journals, particularly policy and economics journals, can yield valuable evidence at this step. One source I go back to over and over again is the Washington Institute for Public Policy’s benefit-cost database.

Bardach also suggests analysts survey “best practices,” or look at how policymakers in other jurisdictions have solved this problem before. Note that just because a policy is being used other places does not mean it will work for your policymaker (or that it’s working at all!). Despite this, policymakers have noted in surveys that they highly value information about what other jurisdictions are doing. This comparative information can be valuable in a policy analysis on its own.

One way to deal with a unique problem is to use analogies. Bardach talks about how a policy analysis on merit pay in the public sector could use data from merit pay studies in the private sector. He also talks about how an analysis of how the state can discipline incompetent attorneys could draw from studies on how states discipline incompetent physicians. If you can’t find exactly the policy you’re trying to study, find policies like them and use them as a guide for gathering data and information.

Requests for data can be difficult. If you want data that has bureaucratic barriers, you may need time to get it. That is why you should start early when searching for evidence. The longer you wait, the less access to data you will have for your analysis.

Assembling evidence can often have a political component to it. People could be trying to protect a program or change it while you are trying to understand it. An analyst must touch base, gain credibility, broker consensus, and work with people in order to get access to the data they need and make it information and evidence that informs the policymaking process.

Lastly, don’t commit yourself to answers at the start of the analysis. Free the captive mind. Contact people you may disagree with and get data from them because they might send you in directions you don’t expect to go. 

Assembling of data can be a fuzzy step and ultimately is one of the most iterated steps in the Eightfold Path. But doing it well means being critical, openminded, and efficient with your time. A good analyst knows how to do all these things, which is why assembly of evidence is so key to good policy analysis.

Social workers could bolster emergency services

In 2019, more than one in seven calls received by the Shaker Heights Police Department were related to mental health — over 5,000 mental health calls in total. Shaker Heights Police Chief Jeff DeMuth expects this number was even higher in 2020 due to increased stress and depression caused by the COVID-19 pandemic.

To deal with this growing problem in their community, Shaker Heights is trying out a new pilot program — embedding licensed social workers in its police and fire departments. These social workers will ride along with emergency response teams during mental health calls and will follow up with people who are engaged when they are off duty.

The logic of programs like these are clear. Police are trained to investigate violent crime and property crime. They are not trained to provide people with support during mental health and addiction crises like social workers are. By plugging more people into social programs, people can not only find help with the problems they are dealing with but could also lessen the load on police, freeing up resources to focus on matters they are better equipped to focus on.

This is a lighter-touch version of a program that has been in place in Eugene, Oregon since the early 90s. Crisis Assistance Helping Out On The Streets (CAHOOTS) is a program that staffs a team of social workers and EMTs to respond to mental health crises before police get on the scene.

The Oregon program is active. In 2019, CAHOOTS handled nearly 19,000 calls for service. It has been on the rise, too: CAHOOTS reports it took double the number of calls in 2019 that it did in 2014. Programs like this can have a double benefit: one of providing people on the scene with support that social workers are trained and outfitted to provide and a second of saving money for the city by diverting calls from police.

Most of the calls the CAHOOTS program responds to center on behavioral health issues. According to official releases by the clinic that administers the program, 19% of calls CAHOOTS responds to are for people with severe and persistent mental illness, 15% are for counseling, and another 15% are for anxiety. Other major categories they respond to are alcohol-related calls, medical calls, and issues with shelter.

These have led to savings for the police department. CAHOOTS reports it saved the City of Eugene’s police department $5.7 million in diverted calls in 2014. That number ballooned to $12 million by 2017. 

CAHOOTS also helps with transportation, transporting people to medical services, substance abuse treatment, shelter, and social services. They estimate these services saved $14 million for the emergency medical system, including ambulance transportation and emergency room services.

The state of Ohio could benefit from programs such as this. This pilot should be watched by the state and if it is successful should be a candidate for funding other programs throughout the state. This is a good way to go about piloting a program like this. Maybe we can find a way to improve community policing in Shaker Heights.

This commentary first appeared in the Ohio Capital Journal.

Omicron underscores the limits of local COVID policy

Omicron is upon us. From Dec. 20 to Dec. 23, over 40,000 Ohioans tested positive for COVID-19, the most of any four-day stretch ever. As a matter of fact, each of the days in that window topped the previous single-day high of 13,374 cases recorded on Nov. 30, 2020, giving us the new first, second, and third highest days for positive tests to date.

Despite 59% of the population receiving a COVID vaccine, mask mandates in cities across the state, and continued social distancing from persistent corporate work from home policies and reduced in-person shopping, coronavirus rages on in Ohio.

Don’t take this to mean these measures aren’t working. Despite the high case numbers, hospitalization has been leveling off and deaths have been dropping, suggesting vaccination could be lessening the symptoms of COVID. The evidence we have suggest mask mandates do reduce infection on the margin, though we haven’t studied this phenomenon with omicron variants yet. And social distancing is surely having some impact on slowing the spread of the virus.

Omicron continues to be a problem in Ohio, though, sending holiday plans into disarray and most dangerously running through hospital staffs and putting further strain on the health care system when it needs capacity the most.

The Centers for Disease Control and Prevention has issued new guidance this week reducing recommended quarantine time after exposure, partly because of new evidence suggesting transmission typically happens within three days of onset of symptoms and partly to allow hospital workers to get back on the job and help people when they are at lower risk of transmission. Policy continues to evolve as we learn more and the science advances in our understanding of this virus.

Ohio could be doing more with vaccinations. The state is only 59% fully vaccinated among working-age adults, good for 36th in the country among states in working-age vaccination rate. Hopefully the new OSHA vaccine mandate will push more people to be vaccinated, which will lesson spread and symptoms and save lives.

The harsh reality of this virus, though, is that vaccination rates in Ohio will do little to slow the evolution of the virus. Variants like delta and omicron are not developing in the relatively-highly-vaccinated United States, they are developing in parts of the world that still do not have access to the vaccine and have not been able to administer it to the general population.

Currently, three out of every four people in the U.S. and Canada has received at least one vaccination shot. Asia and Europe are close to that threshold, with about two out of every three people on those continents having received a vaccination.

In the Middle East, only about half the population has received a shot. In Africa, the number is abysmal: only 1 in 8 Africans have received a shot. This means over a billion Africans have not received a single COVID-19 shot.

Yes, vaccination in Ohio is important. But even if we got our 55% fully-vaccinated rate up to Pennsylvania’s 64% rate or even Vermont’s 77% rate, that would do little to slow the evolution of the virus, which has now lead to a strong breakthrough strain with omicron. 

Patting ourselves on the backs for increasing vaccination rates in Ohio would be like Europe congratulating itself for phasing out carbon emissions over the next twenty years while the rest of the world burns coal. Yes, our policy matters here, but this is an issue that the world needs to tackle, not one we can handle on our own.

This commentary first appeared in the Ohio Capital Journal.

We can do better than GDP

In Ruchir Shama’s opinion piece last month, he wrote about recent efforts to move “beyond GDP” and introduce new measures into policymaking circles.

Despite a combative headline and subheader, the article as a whole is thoughtful and overall fairly assesses the state of well-being metrics used by policymakers.

I currently serve as the president of Gross National Happiness USA, a grassroots network of activists, analysts, and advocates across the United States committed to changing the way we measure progress and success in this country. We believe that gross domestic product is an insufficient benchmark for measuring the well-being of a country and advocate for the use of other tools that help policymakers get a broader understanding of the well-being of their citizens.

Here I offer a few considerations that should be added to this conversation about the use of well-being metrics by policymakers.

First, for all the deification and villainization of gross domestic product in policy spheres, in political conversations, and among economists, the actual measure of gross domestic product weighs quite lightly on policymakers’ minds.

I offer you an experiment: if you are at a local candidate’s debate or having a meeting with a state legislator, congressmember, or any other politician, ask them what they think of the recent gross domestic product numbers. Read up on them beforehand or don’t: you can make up numbers. I would wager that at least three out of four times (conservatively), the politician you ask has no idea what the gross domestic product is for the state they represent or the country as a whole and could not guess reliably by how much it grew last quarter.

Does this mean gross domestic product is harmless? Not necessarily. Many who oppose the dominance of gross domestic product in policymaking believe it is the dominance of gross domestic product thinking that causes problems, not the day-to-day use of the metric in policymaking itself.

It is for this reason that I question Shama’s insistence that making per-capita gross domestic product the “main target of policymakers and the key measure of progress” will result in better policy. Policymakers consider a range of quantitative and qualitative information when drafting legislation and casting votes and I’m willing bet dimes to donuts that gross domestic product as a trend and a number is not particularly influential.

The next question is obvious: then why care? If policymaking is dominated by individual philosophies, constituent concerns, party politics, and information from interest groups, then why focus on measurement of progress and success at all?

This is a sticky question. Political Scientist Ron Haskins argues that, anecdotally, research only makes up 1% of the total consideration by policymakers when crafting policy. So are we bickering about an already-neglected slice of the pie?

The promise of evidence-based policymaking is in a sense utopian. We are fighting for facts in an age of convenient falsehoods. But anyone who has worked with policymakers or who knows one knows that, as a whole, they are people who want to know the truth.

While policymakers are often stereotyped as self-interested and political, researchers have found evidence that about a quarter of state legislators in at least one setting were enthusiastic users of research and only one in six are skeptical nonusers of research. Policymakers want information, but they want information that will help them to make better decisions and craft policy that will help the people they serve.

The way I understand good policy analysis is similar to the ancient Indian parable of the blind men and the elephant. In this story, five blind men touch an elephant. The man touching the ear describes the elephant as like a fan. The man touching the tusk describes it as like a spear. The man touching the trunk describes the elephant as like a thick snake. The man touching the leg describes it as a tree trunk. The man touching its side describes it as like a wall. And the man touching its tail describes it as like a rope. Of course each of these men are wrong in their exact assessment, but they are also right together: an elephant is all of these things.

Good public policy analysis acknowledges that good public policy is multifaceted as well. The economic analyst tells us that people as a whole are getting more of what they want. The poverty analyst is telling us how many count among the least well-off among us. The inequality analyst tells us how close or far we are from one another in income. The human development analyst tells us how much rich, educated, and healthy the population is. And the happiness analyst tells us how well people are assessing their own lives.

All of these considerations have a place in policymaking. And it is only through use of better measures that we will come closer to understanding the elephant that is better public policy in the United States and across the world.

This commentary first appeared on the Gross National Happiness USA Website.

Ohio economists tepid on legalized sports betting

In a survey published by Scioto Analysis this morning, surveyed economists presented a range of opinions on the impact of sports betting on the Ohio economy.

Of the 23 respondents, 10 agreed that legalizing sports betting would have benefits that would outweigh the economic costs of the intervention, though many of those who agreed noted the problem of gambling addiction.

Another nine were uncertain about the impact. Among that group was Bluffton University’s Jonathan Andreas, who said legalization “wouldn't be an economic benefit if it just replaces other less-addictive forms of entertainment in the state.” Four economists disagreed with the statement, saying benefits would likely be small and could be swamped by costs borne by people with addiction.

The panel was even more split on the question of whether sports betting legalization could reduce inequality in the state, with nine economists saying it could, seven uncertain, and seven in disagreement. Those in agreement said that targeted spending of revenue could reduce inequality.

Those who were uncertain said that they were unsure whether spending could be targeted correctly to make up for the harm caused by sports betting addiction. Economists who disagreed emphasized that new revenue from one source usually means reduction in revenue from another source, causing no new benefit overall.

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists.

Scioto Analysis Releases Analysis of Antipoverty Proposals

This morning, Scioto Analysis released an analysis of three antipoverty proposals—one to end poverty in Ohio, one to provide guaranteed income for Ohioans, and one to target deep poverty in Ohio.

All three of the proposals would double the income of individuals in deep poverty and would have substantial impacts on other individuals in poverty as well. All three proposals would also have large impacts on racial inequality of income and would sharply reduce poverty rates in urban and Appalachian Ohio.

“With the passage of the expanded Child Tax Credit in the American Rescue Plan Act, cash transfers are front and center in the American political discourse,” said Scioto Analysis Principal Rob Moore, “this analysis shows what the impacts of cash transfers could be on poverty in Ohio.”

While these interventions would be effective and equitable, they would also be costly, with the three proposals estimated at a cost of $10-12 billion per year.

“More work needs to be done on modeling the financing of proposals such as these,” said Moore.

This analysis builds on Scioto Analysis’s past work calculating the most accurate poverty measure in the state to date. The analysts used the dataset, built on the base dataset of the American Community Survey and supplemented with Current Population Survey data, to simulate the impacts of different policies on incomes in the state of Ohio.

This analysis was led by Madeleine Gaw, Mansi Kathuria, and Sky Mihaylo, Master of Public Policy candidates at the University of California, Berkeley’s Goldman School of Public Policy.