What is "Cost-Free Evaluation?"

What’s the most amazing thing about evaluation?

Is it that it lets us know if something works or not? That’s pretty amazing, no doubt. Understanding if a program has the effects people claim it does is a huge part of understanding the value of the program.

Is it that evaluation helps us understand how much a program works? This is almost more amazing than knowing if the program works. For instance, maybe a graduation-focused tutoring program works because it increases the number of people who graduate each year at a high school by one. Another program is found to work by increasing the number of people who graduate by ten. Knowing how much a program works can be even more valuable than just knowing if it works in the first place.

So what’s even better than knowing both if a program works AND how much a program works? Knowing how much a program works—in respect to its resources. Maybe the second tutoring program above graduates 10 new people at a cost of $10,000. But the first tutoring program graduates 1 at $100. This means program 1 is ten times as cost effective as program 2. While the first program may be more effective on an absolute basis, the second is more effective per dollar…which can be very important for a cash-strapped school. 

So if a school was running tutoring program 2 for ten people and it was costing them $10,000 but then switched those ten people into tutoring program 1, the school would save $9,000…and still graduate the same number of students. 

Theoretically, the school could have evaluated both programs for up to $9,000, implemented the recommendations that came from that evaluation, and ended up on top, the evaluation effectively paying for itself.

In the evaluation world, this concept is called “cost-free evaluation.” If evaluation focuses on this question of cost-effectiveness, it can be a strong tool for saving money and effecting better outcomes at the same time.

In Results for America’s book Moneyball for Government, members of the Bush and Obama administrations put forth a number of recommendations for improving the efficiency and effectiveness of government by being more results-oriented and understanding the interaction between costs of programs and results brought about by those programs.

One of their recommendations is for every federal agency to have 1% of their discretionary budget set aside for evaluation. This means that they can focus their evaluation efforts on programs that may be expensive or ineffective and potentially make that money back for the American people. Then money can be spent on better programs or even rebated back with less spending overall.

There are of course wrinkles to evaluation. Just because a program is cost-effective when small does not mean it will be cost-effective as it grows. Often programs have trouble when they try to scale up. But this is why evaluation needs to occur during the scale-up process, too. If anything, evaluation could be more important here because there are more resources to be lost here. But the promise rings true: evaluation, when done right, can be free. And that is truly amazing.

White Male Workers Still Out-Earning Women and Non-White Workers in Ohio

In 2019, the average non-white male worker in Ohio made $12,000 less than the average white male worker in the state. The average white female worker made $19,000 less and the average non-white female worker made $23,000 less than the average white male worker in the state.

This is all according to American Community Survey data from the United States Census Bureau. Considering the number of workers in each category, the gap in earnings between white male workers and the other three categories came out to $560 million in 2019.

This gap has persisted over the past five years. While the earnings gap fell to $470 million in 2011, it ballooned 21% from 2011 to 2015 on the backs of higher earnings for white male workers. Since then, the earnings gap has persisted in the $560 million to $570 million range every year.

The total gap is largely driven by the earnings gap between white male workers and white female workers. This is because of both the large difference in earnings between white male workers and white female workers and the fact that for every non-white worker in Ohio there are more than three white female workers.

This is despite the fact that white women became a smaller portion over the workforce and grew their earnings more than any other category over the past decade. While Ohio’s non-white male workforce grew by 23% from 2010 to 2019 and its workforce of white males and non-white females grew by 11% each, only 5% more white women were working in Ohio in 2019 than in 2010. At the same time, they were earning 11% more after adjusting for inflation in 2019 than in 2010, higher than the 8% increase for white male workers and the 5% increase for non-white males and females.

Despite the earnings growth white female workers experienced in the past decade, the gap between white male earnings and white female earnings continued to grow due to the high baseline earnings white male workers enjoyed in 2010. The 4% ten-year increase in the white male/white female earnings gap pales in comparison to the 14% ten-year increase in the gap between non-white female and white male earnings and the 25% increase in the gap between non-white male earnings and white male earnings.

Secondary data collection for this analysis was conducted by Masashi Hamano. Analysis conducted by Rob Moore.

Education reforms in state budget could narrow achievement gap

Last Monday, members of the Ohio House and Senate finally reconciled their respective versions of the state budget, passing it with bipartisan support in both chambers. While a $1.6 billion tax cut and funding for broadband and brownfield development will make headlines, the most historic provision of this budget is likely its change to the school funding formula.

The school funding formula in Ohio is incredibly complicated. I know: I briefly worked for the legislative research group that focuses on the formula. A broad range of different factors determine how much state funding goes to each school.

It is hard to tell whether a single reform, even one as substantial as that included in this year’s state budget, will have a big impact on educational equity. What we can do, though, is look at other states’ school funding reform and see what their impacts have been.

A study conducted by leading poverty policy researchers Julien Lafortune and Jessie Rothstein of the University of California, Berkeley and Diane Schanzenbach of Northwestern University gives us some insight into the impact of this reform.

What these researchers did was examine the times state school funding reforms were put in place, seeing how schools fared before and after school funding reforms were implemented. The two outcomes of interest the economists focused on in this study were (1) increases in funding and (2) increases in student achievement.

According to the results, school funding reforms tend to lead to quick, large and long-lasting increases in funding for low-income school districts. Their findings were that not only did per-pupil funding increase on average for a decade out from the reform, but that progressivity of the funding system increased for decades into the future. The average low-income district per-pupil funding amount increased by $1,200 after reforms, $700 higher than in higher-income districts.

But spending alone is only one outcome: What does this mean for student achievement? 

No immediate effects of student achievement were found in the study. This is to be expected: While you can turn a dial and inject cash into a system, it will take more time to use that cash to improve student outcomes. The long-term results from this study, however, are more promising.

Ten years out, student achievement in low-income districts had improved so much that reforms had closed one-fifth of the gap between low-income and high-income districts. Notably, this impact is twice as large on an achievement-per-dollar basis as the impact of the Tennessee STAR experiment, a high-profile study that estimated the impact of class size on student achievement.

This means that funding equity can be twice as cost-effective as reducing class sizes at closing the achievement gap between low- and high-income districts.

Will we see these impacts in Ohio? It’s clearly too early to tell. Certainly some reforms are more effective than others and the devil can be in the details when it comes to a complex reform such as this. This study found that late-90s school funding reforms in Ohio brought considerably more resources to low-income school districts. Only proper evaluation of this reform will tell us the impact of this round of reforms.

This commentary first appeared in the Ohio Capital Journal.

The numbers behind Ohio's natural gas boom

Ohio’s transformation into a natural gas state is truly astonishing. The best way to see this is to look at data from the United States Energy Information Administration, a federal agency and part of the Department of Energy charged with collecting, analyzing, and disseminating energy information in the United States.

Data: US Energy Information Administration

Looking at energy production in Ohio, we can see that total energy production in the state was steadily declining from the 1970s to the 2000s. Then, something amazing happened in the 2010s: fracking. New ways were found to tap into natural gas in the state, and energy production shot up to a rate twice as high as Ohio has ever experienced before.

The incredible thing is that this happened over less than a decade. Natural gas production went from only 8% of total energy production in the state in 2012 to nearly 80% in 2018.

Data: US Energy Information Administration

It may seem here like natural gas is crowding out coal, crude oil, nuclear and renewables, and in one sense, it is. Natural gas production in 2018 accounted for more energy production than most single-year total energy production of all energy sources combined in Ohio. But nuclear and nonrenewable production in Ohio was also at its highest in 2018 compared to any other year, and coal and oil was down to a quarter of its peak in the 1970s.

While these numbers help us understand the energy breakdown of Ohio, it does not tell us much about costs borne to society, which is the relevant statistic for policymakers. Natural gas tends to emit much less carbon than coal and oil, but more than nuclear and renewable energy. However, higher production means more carbon emissions even if cleaner options are being used.

Some analyses tend to say that Ohio has reduced carbon emissions in production, even if it has not on the consumption side, since so much of this energy is exported from the state. Research we conducted this year has suggested that renewable portfolio standards, cap-and-trade, and carbon taxes could all be effective in reducing carbon emissions in the state. Interesting to policymakers, too, should be the impact of local emissions on public health and productivity, a topic we would love to delve into at another time.

18 of 23 Ohio economists believe rural broadband will grow economy, reduce inequality

In a survey published by Scioto Analysis this morning, 18 of 23 Ohio economists agreed that rural broadband programs funded with income taxes will lead to higher state economic growth and lower inequality.

Rural broadband is currently a point of contention in state budget talks. The Ohio House budget included funding for a program, while the Ohio Senate version omitted it.

Economists agreeing with the statements emphasized the role of broadband as a utility and its purpose in a modern economy. Those who were more skeptical wondered how funds would be spent and the degree to which they would help with production. Those who agreed the program would reduce income inequality emphasized how rural broadband could bolster rural areas and Appalachia in particular.

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists.

How can we do higher quality early childhood programs in Ohio?

One provision in the Ohio Senate’s new $75 million budget passed last week that has garnered plenty of attention has to do with Step Up to Quality, the state’s system for promoting quality in early childhood programs in Ohio.

The Senate budget removes the Step Up to Quality child care standards mandate, allowing child care providers to continue to get more money for meeting higher quality standards but not stopping payments for programs that don’t meet standards.

The Columbus Dispatch reports that the Step Up to Quality mandate removal is a step to reduce costs for the state, which Senate President Matt Huffman’s staff estimates will cost the state an additional $640 million by 2024.

The strange thing is that Step Up to Quality is currently saving the state money — but not in the way you would think. The same Columbus Dispatch article quotes Allen County Job and Family Services Director Joe Patton. He says the number of child care providers taking public funds has dropped from 60 to 17 in the past decade, something he attributes to the mandate to participate in Step Up to Quality.

This means that the requirements in Step Up to Quality could be leading providers to stop taking public funds so they don’t have to deal with these requirements.

The evidence that we have suggests quality in early childhood education matters. We’ve seen positive examples of the impact of early childhood like the Perry Preschool Project and the Abecedarian Project. We’ve also seen the negative impact of expanding child care without quality controls in Quebec, leading to higher aggression and illness and lower motor and social skills among children and worse parenting relationships and health among parents.

That being said, the evidence for the effectiveness of programs like Step Up to Quality are mixed.

A 2019 evaluation of New Mexico’s “Step Up to Quality” equivalent conducted by the New Mexico Legislative Finance Committee found no evidence child care assistance led to improved educational outcomes. It did find that family income and child well-being improved among providers that participated in the program, but the specific ranking didn’t have any bearing on these outcomes.

What this means is that, while it helped families to be a “one-star” program, they couldn’t find any difference between “one-star” and “five-star” programs. These programs, at least in this case, were likely measuring and requiring the wrong things.

So what can we do better? One option is to focus more on outcomes than outputs. The New Mexico study above recommends creating evaluation plans for child care based on “measures of child health and social-emotional development, family economic improvement, and parental employment.”

Another option is to put the state in charge of assessment of quality the same way it is in charge of assessing health conditions. Having early childhood assessors who go on-site to assess conditions would reduce reporting costs borne by providers and could tie assessment to widely-used measures like the Early Childhood Environment Rating Scale.

While we have good reason to believe early childhood education can grow the economy, reduce poverty, and improve lives, we still have a lot to learn about how to best foster it from a public policy standpoint. This is a system that will likely endure some substantial tweaking in the coming decade.

This commentary first appeared in the Ohio Capital Journal.

Ohio's growth was sluggish even before COVID-19

This morning, Scioto Analysis released Genuine Progress Indicator (GPI) calculations for 2019 for the state of Ohio, providing the most comprehensive economic activity estimate for that year to date. Results show that GPI grew 0.9% from 2018 to 2019, Ohio’s slowest growth rate in three years, and about a third of the growth rate suggested by Gross Domestic Product (GDP) estimates made by the Bureau of Economic Analysis.

“While GSP measurements only estimate the value of traded goods, GPI calculations we make also include the cost of environmental damage and the value of goods such as unpaid housework to the state economy,” said Scioto Analysis Principal Rob Moore.

While total economic indicators were up 1.6%, environmental damage was up 3%, driven by increased nonrenewable depletion and carbon emissions. Social indicators were also up 0.8%, driven by increased value of housework and parenting and higher education.

The report also includes recommendations for improvement of the Genuine Progress Indicator, an indicator that four states (Hawaii, Maryland, Vermont, and Washington) have statutorily endorsed. These recommendations include calculating value of government expenditures and net exports and medical costs associated with food insecurity.

Detailed estimates of the Genuine Progress Indicator’s 26 indicators are included in Appendix B of the report.

Bail reform could reduce disparities and save money

Last month, state Sens. Rob McColley and Steve Huffman introduced a bill to reform the bail system in Ohio.

Ohio has already had some counties dip their toe in the bail reform water. Toledo’s Lucas County released twice as many defendants in 2015 in its first year using a new bail-setting system based on risk assessment.

Why do we have bail in the first place? According to the American Bar Association, “[bail] is not supposed to be used as punishment. The purpose of bail is simply to ensure that defendants will appear for trial and all pretrial hearings for which they must be present.”

Judges set bail based on a variety of factors, including risk the defendant will not show up for trial, the crime the defendant is accused of committing, how “dangerous” the defendant is considered to be, and how much of a risk the defendant poses to the community during the release period.

The problem with the current system is that there is strong evidence for bias in it. Last year, researchers at the University of California, San Diego, Harvard University, and the University of Chicago released a working paper at the National Bureau of Economic Research tackling this question. They found that about two-thirds of the average release rate disparity between white and Black defendants in New York City is due to racial discrimination.

These findings echo an earlier study by these researchers that found evidence of racial bias among bail judges in Miami and Philadelphia. The researchers found that this bias was driven by racially-biased prediction errors, using race as a proxy for more salient bail considerations. They also found bias more common among inexperienced and part-time judges.

Over the past decade or so, racial justice advocates have increasingly partnered with social conservatives on criminal justice reform, realizing that high levels of incarceration in the United States are both exacerbating racial disparities and costing taxpayers a lot of money. Bail reform is an important front in this alliance of strange bedfellows.

Other states have taken steps to change the way bail is done in their justice systems. Earlier this year, Illinois became the first state in the country to end mandatory cash bail. Other states have been slow on the uptake. Alaska and New York instituted bail reforms that have been rolled back or amended. Voters in California rejected an effort to reform bail in their state.

It can be easy to be swept away by single stories when it comes to bail reform. Inevitably, someone who is released under Illinois’s cash bail system will commit a crime and it will make headlines. Opponents of bail reform will happily jump on such a story as evidence that the reforms were a mistake.

This is why evaluation is so vitally important for a reform such as this. Researchers across the country will have their eyes on Illinois as it implements its bail reform this summer. Hopefully if Ohio passes bail reform, it, too will be following the impact of the program on disparities, public safety, and local finances. Let the data bear it out and policymakers judge the worthiness of tradeoffs: That is the stuff of good policymaking.

This commentary first appeared in the Ohio Capital Journal.

Vax-a-Million: Is it worth it?

Last weekend, the Ohio Capital Journal reported that a state representative is considering introducing legislation to halt the “Vax-a-Million” campaign, a public outreach campaign that is famously giving $1 million to five Ohioans who have been vaccinated and registered for the program.

The Journal has also covered the consternation with the program among legislators, with some arguing the state should be doing nothing to promote vaccination and others arguing that the campaign is “untested” and a “misuse of money.”

How do we test these claims? Are there more “tried and true” ways to spend federal covid funds? How does this constitute “misuse?”

One of the most thoughtful analyses I’ve seen on the program has come from Julie Washington at Cleveland.com. She estimates that the cost of the program would equal the cost of about 40 severe hospitalizations. So if the program increases vaccination rates to the point where it prevents 40 cases of severe hospitalization, the program would “pay for itself” under this logic.

If a federal regulator were analyzing a program like Vax-a-Million, she would privilege the lives saved as a central consideration of the effectiveness of such a program. Since most valuations of the value of a statistical life place the value of risk reduction of death at around $10 million per life saved, if Vax-a-Million saves one life through spurring vaccination take-up, it pays for itself in social value under standard cost-benefit methodology.

As an additional consideration, when conducting cost-benefit analysis, an analyst also does not count the “sticker price” of a program as the cost, but rather distortion of the economy caused by the price. So, for instance, the Washington State Institute of Public Policy, the leading state institute for use of cost-benefit analysis in analyzing state programs, places the marginal excess burden of taxation at 50% of the cost of the program. This is a conservative assumption since it is on the high end of the range of estimates for how much taxes impact the economy.

For those confused, this means the economic cost of the program is probably $2.5 million (50% of $5 million) or lower. So this means that just looking at the cost of tax distortions of the program (most of them borne outside of Ohio since this is federal money), and the benefits of lives saved at a standard estimate of $10 million per lives saved, Vax-a-Million just needs to have a one in four chance of saving a single life in order for economic benefits to exceed economic costs.

These projections become even more rosy if you think taxes are less distortionary than the Washington State Institute for Public Policy’s conservative estimate or if you think the value of a statistical life is higher than the standard $10 million estimate.

So there it is. Yes, the program is new. It’s untested. But vaccinations were up by 30,000 in the week after the announcement. Can I get my calculator out and estimate if Vax-a-Million curbs Ohioans’ “freedom?” Of course I can’t. Do I think this program has greater than a one in four chance of saving a life? If this is the goal of the program, it seems like a gamble worth considering.

This commentary first appeared in the Ohio Capital Journal.

Ohio economists agree state naloxone spending has economic benefits

In a survey published by Scioto Analysis this morning, 25 of 27 Ohio economists agreed that the economic benefits of state funding for opioid overdose reversal medication outweigh the economic costs.

Economists agreeing with the statement emphasized both the low costs of the medication—about $40 per dose—as well as the high valuation of a statistical life. Some economists went past the simple economic argument, saying the policy was “the right thing to do” or “ethically obvious.” Some were critical of the opinions of detractors, stating the impact of availability of opioid reversal drugs are likely to have little impact on drug use habits.

Of the two that did not agree with the statement, the one economist who was uncertain commented that additional resources would be needed to transition overdose victims into the labor force if drug use leads to unemployment.

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists.