What should a policy analyst know about the normal distribution?

Last year, Scioto Analysis conducted a policy analysis to evaluate alternatives to reduce carbon emissions in the state of Ohio. In order to test our models, we conducted a Monte Carlo simulation, the “gold standard” of sensitivity analysis in cost-benefit analysis and a tool we often employ to see what range of possible outcomes.

Below are the Monte Carlo simulation results for a strong cap-and-trade program. For this Monte Carlo simulation, we ran hundreds of thousands of simulations of alternate policy scenarios, randomly generating different social cost of carbon estimates, discount rates, and price elasticities of demand for electricity. You can see the results of the Monte Carlo simulation below.

If you’ve taken a statistics class, you’re probably familiar with the shape of this distribution. It is one of the most important shapes in statistical analysis and one we end up using a lot when we’re modeling policy outcomes: the normal distribution.

The normal distribution, often called a bell curve because of its shape, is one of the most universally recognized statistical concepts. It is intuitive, broadly applicable, and useful in simplifying complex concepts. Here we will briefly discuss a few important characteristics of the normal distribution and why they are important. 

Parameterization

The parameters of a statistical distribution are the things you need to know to fully understand it. For example, if we consider some binary outcome like flipping a coin (called a Bernoulli distribution), the only parameter we need to know to fully understand the range of outcomes is the probability of an event happening.

The normal distribution is special because it has two parameters, its mean and its variance. We can always calculate the mean and variance of observed data, but the fact that these tell us everything we need to know about the distribution of unobserved data from the same distribution is extremely powerful. 

Symmetry and Outliers

Two other properties we will discuss together are the facts that the normal distribution is symmetric. This means that we should expect to observe values above and below the mean with the same likelihood, and that big outliers are extremely uncommon such that the we should only observe a value only four standard deviations above or below the mean is about 0.001% of the time. These two characteristics often shape how we think about applying the normal distribution to real data.

Consider the distribution of incomes in the US. We know that a few small outliers skew the distribution heavily to the right which makes fitting these data to the normal distribution difficult. If we just calculate the observed mean and variance of all individuals in the US, we would expect there to be extreme outliers in the negative direction as well.

The Central Limit Theorem

Arguably the most important concept in statistics, the central limit theorem is certainly the most useful application of the normal distribution. There is a lot of rigorous math that we will skip over here, but in short the central limit theorem tells us that if we repeatedly take random samples from a population, those sample means will be approximately normal. 

Going back to the income example, if instead of measuring using the mean and variance of all incomes to approximate a normal distribution we took 500 random samples (with replacement) and calculated the means of all of those, we would find that those sample means did in fact follow a normal distribution quite well. 

The normal distribution gives us a way to mathematically describe what we expect to happen with lots of unobserved data quite well. Understanding it at a surface level is valuable for policy analysts and policy makers since it so often works its way into our assumptions, whether we realize it or not.

Conversion therapy bans could prevent hundreds of youth suicides over the next decade

Last Monday, the Akron City Council voted to ban conversion therapy in the city, making it the eleventh city in Ohio to do so. 

The pseudoscientific practice referred to as “conversion therapy” encompasses counseling aimed at children focused on changing sexual orientation. The practice has been condemned by the American Medical Association, the American Counseling Association, the American Academy of Pediatrics, and the American Association for Marriage and Family Therapy.

The Akron ordinance cited a 2019 study by the UCLA Williams Institute that found LGBT+ youth exposed to conversion therapy were twice as likely to consider and attempt suicide than those who hadn’t. Conversion therapy made worldwide news when transgender Ohio teenager Leelah Alcorn posted her suicide note on the social media site Tumblr in 2014, explaining how conversion therapy led to her death.

I still find myself surprised this is a practice that occurs in Ohio today. I’ll admit, I sometimes even think “is this a real problem or are these sorts of ordinances empty gestures?” I have been saddened to find that this problem is indeed very real.

In a survey conducted as a part of Kent State University’s LGBTQ+ Greater Akron Community Needs Assessment, 30 of 701 respondents reported they had received conversion therapy at some point. According to the UCLA Williams Institute, there are 72,000 LGBTQ+ Ohioans age 13-17. 

If Ohio rates of conversion therapy reflect the responses in this survey, that means nearly 3,000 Ohio teenager have been exposed to conversion therapy. The Trevor Project’s 2022 National Survey on LGBTQ Youth Mental Health found 18% of LGBTQ teens made a suicide attempt in the past year. 

If teens exposed to conversion therapy have roughly double the suicide attempt rate of the general population, this means over 1,000 LGBTQ teens in Ohio who have been exposed to conversion therapy attempt suicide every year. If the death rate of suicide attempts reflects the national average, this means 43 LGBTQ teens in Ohio who are exposed to conversion therapy die of suicide every year.

If conversion therapy doubles the suicide attempt rate for youth, this means banning the practice statewide could save over 200 youth lives from suicide in Ohio over the next decade.

Bans on conversion therapy have been enacted so far in the cities of Akron, Athens, Kent, Cincinnati, Cleveland, Cleveland Heights, Columbus, Dayton, Lakewood, Reynoldsburg, and Toledo. The largest cities without bans right now are Parma, Canton, Lorain, Hamilton, and Youngstown, each with total populations over 60,000 people.

City-level bans only go so far, though. Bans can be ducked by people providing conversion therapy across jurisdictional lines only a short drive away, making patchwork municipal bans less effective than a statewide ban. 21 states and the District of Columbia have now enacted statewide bans on conversion therapy, mainly concentrated in the northeast and west. This includes moderate political states like New Hampshire, Pennsylvania, Utah, and Virginia. 

The only state with a conversion therapy ban in the Midwest currently is Illinois, though it is the region of the country most blanketed with city-level bans. If state lawmakers want to reduce teen suicides in Ohio, they have a strong option in front of them.

This commentary first appeared in the Ohio Capital Journal.

Discounting in Cost-Benefit Analysis

This series of blog posts are about my first cost-benefit analysis with Scioto Analysis. As I am going through this process, I am writing about the challenges I come across and how I have been thinking about them. This week, I wanted to write about discounting, why it is important, and when it should be used.

Discounting is the process we use to estimate the difference in benefits policies create in the present from benefits they create in the future. Discounting is a cornerstone practice of good cost-benefit analysis.

Which would you rather have, $100 today or $100 in 10 years? It is pretty easy to understand why $100 in 10 years is the worse of these options. If you get hit by a bus tomorrow, that $100 later will be pretty useless. You could also invest your $100 today and end up with $200 in ten years. Having money now is better than having money later. 

When conducting cost-benefit analysis, discounting is extremely important because many impacts calculated incur short-term costs and long-term benefits. Appropriate discounting of future benefits is an essential step in determining the true value of a proposal from a social perspective.

In the context of my current cost-benefit analysis, I have spent a lot of time thinking about whether or not I should include future values. For context, the program I am analyzing is essentially government subsidies to encourage farmers to use less phosphorus fertilizer which in turn leads to less phosphorus runoff into Ohio’s waters.

The justification for including future impacts is that improved water quality is valuable year after year. However, there are two major assumptions that we must consider. First, the subsidy is a one-time investment meaning that all of our costs are accounted for in the present. Second, because the reduction in phosphorus is the result of using less fertilizer, we assume that going forward there are no new reductions in phosphorus. 

Whether or not discounting future benefits is appropriate depends on how we choose to monetize benefits. In this case, we model the benefits of this program as people’s willingness to pay for cleaner water. This means that the marginal decrease in phosphorus is valuable. We could discount the marginal increase in recreational use, but we should expect that benefit to be captured in the current willingness to pay for a marginal increase in water quality today. 

Choosing not to include future benefits in this analysis makes the most sense in this context, but as a policy analyst it is important to fully understand the implications of that decision. In this case, it leads to a more conservative estimate of the benefits of the program, which can be a useful direction to err.

This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.

Introducing Scioto’s Interactive Budget App

At Scioto Analysis, we believe in making information as accessible as possible in order to inform policymakers and the public. In that spirit, we are proud to unveil our first interactive budget app! This app uses data from the Ohio Legislative Service Commission about the state’s forecasted budget for fiscal year 2023. 

In its current state, you can use this app to see how changing revenues and expenditures affects the state’s budget surplus and see the tradeoffs for yourself. With this app, you can effectively create your own state budget! Want to increase spending on public programs? Here you can see some of the different possible ways of financing it.

You can either use the tool below or you can open it up in its own window by clicking here.

In the future, we plan to include more options so you can see the effects of policy options ranging from specific proposed tax cuts to pie-in-the-sky proposals like universal basic income. If you have any feedback or suggestions feel free to email michael@sciotoanalysis.com.

In the meantime, get budgeting!

Social vs. Private Costs in Cost-Benefit Analysis

I am currently working on a cost-benefit analysis on policy options to improve water quality in Ohio. At this point in the process, I’ve spent a few weeks thinking about how to approach this question and even have some preliminary models built out. There is one specific part of these models that has made me stop and think and I want to talk about it briefly. 

All of the policy alternatives I am considering for this cost-benefit analysis involve some amount of government spending to improve agricultural practices on private farms. Essentially, the government subsidizes certain farm practices that while slightly more expensive, can improve water quality and provide benefits for people who use lakes and streams. The question then for the analyst is what should count between public costs, private costs, and private time and labor.

The case for including all these impacts in this analysis is that under most circumstances, the subsidy won’t cover all of the upfront costs associated with the program, and instead farmers would be spending a smaller amount in the short term to get additional long-term benefits in the form of reduced spending on fertilizer. If farmers are trading short-term costs for long-term benefits, then the model should try to capture that. The case against including both measures relies on the fact that the program of interest is voluntary and farms that participate in it are profit maximizing firms. 

Consider for a moment that this cost-benefit analysis did not involve any government subsidy, and we were just concerned about farmers implementing these practices themselves. In this case, we would include the upfront costs in our model. With government subsidies, the intuition might be to just subtract the subsidy from the total cost in order to avoid double counting. 

However, acknowledging the fact that in order to spend money the government needs to tax its citizens, the marginal excess burden of taxation is the social cost of a subsidy (read more in the Ohio Handbook of Cost-Benefit Analysis). In short, we are measuring the lost economic activity that comes with raising taxes, not the transfer of funds from the government to a farm, which is itself a net zero transfer payment. 

If farmers are choosing to enter into this program, then there should not be any lost economic activity. The upfront costs of this program are just a transfer of funds from the farmer to the scientists they need to perform soil tests. Whatever portion of these funds comes from the subsidy is largely irrelevant, because the only economic loss is through the marginal excess burden of taxation. If this was a mandatory program, then we would consider the lost value of however else the farmers would have spent their time because it would be mandated changes that would not have happened otherwise. Voluntary compliance suggests the costs and benefits are internalized by the farmer. 

This detail is an illustration of the fact that cost-benefit analysis is not a tool for measuring private costs, but rather social costs. Without additional information about value added or lost from transferring funds between parties, there should not be an economic effect. 

This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.

Economists weigh in on proposal to abolish subminimum wage for workers with disabilities

In a survey published by Scioto Analysis this afternoon, economists in Ohio called into question the current tiered minimum wage system that allows for employer to pay lower wages to workers with disabilities.

Ohio Representatives Brigid Kelly and Dontavius Jarrells have put forth a bill to eliminate the current lower minimum wage for persons with disabilities. This lower wage threshold was originally put in place to encourage employment of people with disabilities. Advocates for its abolition argue the lower minimum wage has not yielded the results originally intended.

A majority of respondents to the Economic Experts Panel survey conducted last week said abolishing the lower minimum wage would reduce poverty. Some who agreed did so with the qualification that the overall poverty impact would be modest. Among those skeptical of a poverty reduction, economists cited the potential unemployment impact of a higher minimum wage and the small scale of the policy.

A majority of respondents also said the policy would not likely have negative human capital ramifications for people with disabilities. Economists who believed this would not hurt human capital development said that the current policy has not been shown to improve human capital for workers. Those who were uncertain said that employment loss could hinder human capital development for workers.

The Ohio Economic Experts Panel is a panel of over 40 Ohio Economists from over 30 Ohio higher educational institutions conducted by Scioto Analysis. The goal of the Ohio Economic Experts Panel is to promote better policy outcomes by providing policymakers, policy influencers, and the public with the informed opinions of Ohio’s leading economists.

Conducting Cost-Benefit Analysis: The role of assumptions

I am currently working on a cost-benefit analysis on water quality in Ohio. After spending two weeks preparing and thinking about this current cost-benefit analysis, I am starting assembling some preliminary models. This step is often known as projecting the outcomes and is almost always the most difficult part of a cost-benefit analysis. 

One of the main challenges when trying to project outcomes is fully understanding what sort of assumptions we as analysts have to make in order to make projections at all. Our assumptions define how we think about our analysis and they shape our results. Assumptions cannot be avoided in any sort of analysis, but as analysts it is our job to be fully aware of which ones we make. 

Generally speaking, each additional assumption makes the analysis easier to perform and to understand, but it makes it more difficult to generalize the results. A common misconception is that having stronger assumptions makes analysis less accurate or less meaningful, but sometimes making additional assumptions actually improves our models. 

This tradeoff is extremely common and is even described mathematically by the bias-variance tradeoff. In statistical terms, we evaluate estimators using a criteria called mean squared error which can be written as the sum of an estimator's variance and its squared bias. Decreasing one of those quantities often means increasing the other. In our context adding more assumptions simplifies the models and reduces the variance, but because we are assuming more things we may be adding some bias to our results. It is ok to keep some bias, especially if we understand that it exists and it lowers the variance of our estimate making it more useful.

At the beginning of my model building, I chose to make as many assumptions as I had to in order to get some preliminary results. It may not make sense to assume that certain things will remain constant well into the future, but because this is an iterative process it is important to get a working model to build off of. Building a simple model can give you a general idea of potential results and can give you an answer which then you can build from by refining your model.

Making assumptions is also useful because they help you identify what things you need to learn more about. After going through and making all of the first models, it is important to ask which assumptions can I potentially get rid of. Does removing these assumptions make the model sufficiently more generalizable?

As analysts, it is our job to make clear that predicting the future is extremely difficult. I once heard someone say that making predictions is like trying to drive a car by only looking in the rearview mirror. Sometimes trying to work with real data forces us into making strong assumptions. Still, by using the best available methods and fully understanding the ramifications of the assumptions we make, we can help policy makers decide on the best course of action with better information than they would have on their own.

This blog post is part of a series of posts on conducting cost-benefit analysis for newcomers by Scioto Analysis Policy Analyst Michael Hartnett.

Is it time for all Ohio children to get free meals at schools?

Last month, the Ohio State Board of Education officially recommended that the state of Ohio use American Rescue Plan Act dollars to provide free breakfast and lunch to Ohio students through the end of the year.

The federal government has been providing free lunch to children from low-income families since 1946, when President Harry Truman signed the National School Lunch Act into law. The program was expanded in 1966 through the Child Nutrition Act, which added breakfast and summer meals to the program.

The federal free school lunch and breakfast program is one of the largest antipoverty programs in the United States. In a study Scioto Analysis released last year, we estimated from American Community Survey and Current Population Survey data that more than 8,800 Ohioans were pulled out of poverty in 2018 by the federal free school lunch and breakfast program. 

Nationally, the impact is even larger. The Census Bureau estimates 300,000 Americans were pulled out of poverty by school lunch and breakfast programs in 2020, and an additional 3.2 million were pulled out of poverty by a combination of SNAP food assistance and free school lunch.

Free lunch has a big impact on the children who receive them. Half of the food consumption for these children comes from free or reduced-priced meals. Recent expansions in availability of free lunch has also meant better outcomes for children. Recent studies of expansions of free school lunch programs suggest the expansions have led to increases in math scores for students, especially among elementary school students and Hispanic students. These expansions have also led to decreases in suspensions among white elementary-school age boys.

School lunch programs have been expanding for years. In 2011, schools in Illinois, Kentucky, and Michigan began piloting a program that would make all children in low-income districts eligible for free lunch. By 2019, two-thirds of all low-income schools across the country were providing free lunch meals to their children.

The pandemic brought a seismic sea change to this landscape. The United States Department of Agriculture, which administers the free school lunch and breakfast programs, suspended all eligibility requirements for free and reduced meals, making free lunch universal with a single administrative change.

Last summer, this expansion lapsed, and Ohio reverted to a limited system of provision of free and reduced priced lunches.

Five states—California, Maine, Massachusetts, Nevada, and Vermont—have passed legislation providing no-cost meals to all students. Pennsylvania recently joined this list, passing legislation making breakfast free in schools.

It seems that the biggest reason free school lunch and breakfast has not been expanded even more aggressively in the United States is because of simple budget constraints. In the particular policy the state Board of Education calls for, American Rescue Plan dollars have a number of different alternate uses. I have not done an analysis of all possible uses for these funds, but I have to imagine this would be a use that would yield relatively high economic and equity benefits.

One potential objection to universal school meal programs is that they would mainly benefit middle-and upper-income households and do little for children from low-income households since low-income households are likely already covered by the current program. The benefits of administrative simplicity, though, have already led to universal provision in low-income schools. Providing this benefit to all children and then clawing back costs through progressive income taxation is likely a more efficient program design than a system focused on splitting hairs around eligibility.

The pandemic changed a lot of assumptions about the U.S. safety net. Maybe a permanent change could come to the program: maybe all children in Ohio should get free meals at school.

This commentary first appeared in the Ohio Capital Journal.

How should an analyst decide on a policy recommendation?

Deciding is maybe the strangest step of Eugene Bardach’s Eightfold Path of policy analysis. This is because the job of the analyst is generally not to say how the world should be, but to describe how it is and could be. One way to think of the Eightfold Path is to split the parts of the path into these three categories.

You could describe selecting criteria as an exercise in describing the world “should be.” After all, selecting criteria is about making value judgments about what a policymaker should care about. But we often lean on values universal enough (effectiveness, efficiency, equity) that any policymaker should be interested in them and we tailor our selection of criteria based on the desires put forth by the client, so that step is still more descriptive than prescriptive.

Deciding is a different story. 

In A Practical Guide to Policy Analysis, Bardach says that

[T]he object of all your analytic effort should not be merely to present the client with a list of well-worked-out options. It should be to ensure that at least one of them—and more than one, if possible—would be an excellent choice to take aim at solving, or mitigating the problem.

The reason this step is a leap from the previous six steps lies in its interaction with step four. Surely any policy analysis could include dozens of different criteria by which to evaluate a policy. But policy analysis is also by its nature time-limited, so selection of criteria cannot be exhaustive. Policy analysts are thus working with imperfect information.

In the classic understanding of the relationship between the policy analyst and the policymaker, the policymaker makes the decision of which policy to choose and the analyst provides information to make the decision with. Under this framework, the policy analyst deciding for the policymaker can be seen as inappropriate. It also can be seen as unhelpful: for some criteria—I’m thinking of “political feasibility” in particular—surely the policymaker has better information than the analyst does.

So why does Bardach include it? I would hazard to guess that it has to do with the actual desires of policymakers.

When I was a budget analyst at the Ohio Legislative Service Commission, one tension we dealt with when working with legislators was between analysis and recommendation. While our charge was to provide the policymakers with analysis, legislators would often ask analysts for their opinions on the policy. They weren’t looking just for advice, but for collaborators. And sure, maybe they wouldn’t make the exact same decision the analyst tells them to, but having that guidance was helpful to them.

Some analysts felt uncomfortable being asked a question like this because they felt that it undermined their credibility as analysts. They felt that the jump from “could be” to “should be” was one that an analyst shouldn’t make. But by putting yourself in the shoes of the decisionmaker, the analyst can bring clarity to their findings are guide a policymaker in a way that is helpful to them: by telling them what you would do if you were making the decision.

Ultimately, the policymaker will make the final decision. But if the policymaker is looking for a recommendation, give it to her. It will only make her job in deciding easier.

How do Policymakers Value Risk of Death Reduction?

How much should society pay to save a life? According to a recent meta-analysis published in the Journal of Benefit-Cost Analysis by the Cambridge University Press, $8 million is a good place to start. 

In policy analysis, this number is often referred to as the Value of Statistical Life (VSL). Many people are initially hesitant when they are presented with the idea of VSL, pointing out that it is unpleasant to assign a monetary value to life. So why should we put a dollar value on lives saved at all? 

Imagine that a local government is trying to figure out how many traffic lights to install in a city. Traffic lights are good because they make road intersections safer, but they also require resources to install and maintain. How many traffic lights should the city build?

If we assume that this local government has to raise taxes in order to finance these new stop lights, we need some way to measure the value they provide in order to reach the optimal solution. Each additional stop light may reduce traffic deaths by some small amount, but if they are only providing small benefits then it may be the case that those resources are better spent somewhere else. Would you be willing to pay an additional $1,000 in taxes in a year to reduce your chance of death by one in a million? These are the tradeoffs policymakers confront when they make decisions on behalf of the public.

Once we realize why it is important to have a measure for VSL, we must figure out the best way to calculate it. The article from the Journal of Benefit-Cost Analysis groups VSL studies into three main categories.

First are the studies that try to measure VSL through choices people make in the labor market. Hedonic wage studies use labor market data to see what sorts of workplace risk individuals are willing to accept for higher wages. While these estimates have some drawbacks (e.g. the labor market does not capture the entire population), they are based on reliable and easily accessible data. 

There are also studies that measure VSL by looking at individual decisions like people’s willingness to wear helmets while riding bikes. The authors mention that these sorts of studies often rely on researchers making significant assumptions and are therefore not used as often. 

The final type of VSL study tries to measure people's willingness to pay for safety through methods such as contingent valuation studies. The biggest benefit of these studies is that they can more specifically ask about different risks that may be more applicable to a public policy context. The drawback is that they hinge on stated preference for risk of death reduction, which can sometimes be different than the revealed preferences people make when faced with real decisions. Words don’t necessarily speak louder than actions.

This new study in the Journal of Benefit-Cost Analysis is unique because it does not get its information from other individual studies about VSL, but rather from other meta analyses about VSL. By taking into account the widest possible set of VSL estimates, the authors are able to get the best possible picture of where VSL is currently. 

The authors estimate that the central value for VSL is about $8.0 million, with the 90% confidence interval ranging from $2.4 – $14.0 million. This number largely falls in line with what we see policy makers actually use. The US Department of Transportation has said that they recommend the VSL be $10.9 million after adjusting for inflation. 

The literature on VSL continues to evolve as researchers work to better understand tradeoffs between small increases and decreases in risk of death and everything else in life. For policy analysts and policy makers, how human lives are impacted is the most significant part of any proposal. The better information we have about VSL, the more efficiently we can allocate our resources and reach the best possible outcomes.