Earlier this week, I attended the Minnesota Evaluation Association’s annual conference. In the morning, I had a conversation with a few other attendees about some of the challenges of performing quantitative analysis in a way that accurately reflects the groups we are trying to better understand.
At the heart of this conversation was how we as statisticians assign categories to certain questions and what the implications of those choices are. Grouping people together helps us identify patterns, but those same groupings can obscure important differences between the people we are studying.
Categorical variables
At the heart of many statistical analyses is the question “how is X related to Y?” How do poverty rates differ across racial groups, for example? What happens, though, when we dig into the categories we use to answer that question?
Categorical variables allow us to organize observations into groups. Instead of looking at each person’s exact age, we might compare children, working-age adults, and older adults. We may choose to do this when looking at age depending on our specific research question, but for non-numeric variables we are often forced into studying them categorically.
One of the key problems with discrete categorization is that certain people might be grouped together on one measure like race or age, but that might not be a good representation of that individual. As an example, someone who is 64 might have more in common with someone who is 65 than with someone who is 25, even though our categories might place the first two people in separate groups. The boundary is useful for organizing an analysis, but it doesn’t necessarily represent a meaningful change in someone’s circumstances.
Tradeoffs with sample size
One way to address this problem is to use more specific categories. If a broad group contains people with very different experiences, we could divide it into smaller groups and analyze those separately. However, we face a tradeoff as we go down this path. The more specific our model, the less generalizable it becomes.*
Imagine we are using a survey to study employment barriers in a particular city. We might have enough responses to estimate an overall employment rate with reasonable confidence, but only a handful of responses from a particular immigrant community. The experiences of those respondents are important, but a few people’s answers may not give us a reliable estimate for the entire community.
In these situations, we sometimes have to combine categories or leave a separate estimate out of a report. This can be especially frustrating when the smaller population is the one we are most interested in understanding. A dataset can be large enough to answer questions about the broader population while still being insufficient for the central question of our project.
The important distinction is that we are leaving out an estimate because we cannot support it with enough confidence, not because the population is unimportant. We should make that limitation explicit. Otherwise, readers might interpret a group’s absence from a report as evidence that it faces no particular challenges.
Depending on the project, we might address this by combining multiple years of data, collecting additional responses from that community, or using interviews to better understand people’s experiences. Each approach has tradeoffs. Combining years makes it harder to describe current conditions, while interviews provide depth without necessarily telling us how common an experience is.
Tradeoffs with race data
At Scioto Analysis, we rely on the Census Bureau’s race categories for our analyses. Because we rely so heavily on the American Community Survey and other Census Bureau products, this is the only option we have. Even in situations where we collect other data, we often need to compare it to estimates from the American Community Survey, so we still need to use the same categories.
The Census Bureau explains that its race data are based on self-identification and that its categories reflect social definitions of race rather than biological distinctions. These categories provide a common framework for analysis, but they cannot fully describe someone’s identity or lived experience.
Two people who identify with the same broad racial category might have different cultural backgrounds or speak different languages. Putting those people into the same bucket can hide those differences, which might be relevant to whatever policy decision we are studying.
At the same time, broad categories can still be relevant to understanding how people are treated. People with different backgrounds might encounter similar discrimination because others perceive them as belonging to the same racial group. For a policy question about unequal treatment, that shared experience may be important even when the category tells us relatively little about their individual identities. However, someone’s self-reported race does not directly tell us how other people perceive them.
The question, then, is whether the categories available to us capture the differences that matter for the policy we are studying. If we are evaluating access to a public program, language or immigration history might help explain barriers that a broad racial comparison alone would miss.
Quantitative analysis requires us to simplify complicated realities. Our models should increase our understanding of a topic, but we know that it will never be perfect. When dealing with categorical variables, especially with race data, we should be aware of the underlying assumptions that we are making by assigning categories, and we should be careful that those assumptions balance the tradeoff between understanding the data we have and preserving generalizability for the wider population.
* To see this tradeoff clearly, consider the trivial example where every individual is assigned their own category. Our analysis will give us plenty of information about each individual in our sample, but there is no way to make claims about people outside our sample, since their categories are by definition not included in the study sample.
