Development economics has a dirty secret hiding in plain sight. When we report that a cash transfer program raised incomes by 14 percent, or that a deworming intervention improved school attendance by 0.3 standard deviations, we are telling a story so compressed it borders on fiction. The average treatment effect—the workhorse statistic of every impact evaluation summary—collapses an entire distribution of individual experiences into a single number. And that number can deceive us profoundly.

Consider a microfinance program evaluated across 6,000 households. The ATE on business profits might be a modest but statistically significant positive. But disaggregate, and you might find that the top quartile of entrepreneurs doubled their profits while the bottom quartile actually took on debt they couldn't service and ended up worse off. The average is positive. The policy implication is ambiguous at best, harmful at worst. We designed the evaluation correctly. We just stopped analyzing too soon.

This is not a niche methodological concern. It is a fundamental challenge for how we allocate development resources, design targeting mechanisms, and make scaling decisions. The field has developed increasingly sophisticated tools—from quantile treatment effects to causal forests—for moving beyond the tyranny of the mean. The question is whether evaluation practice and policy design are keeping pace. For anyone designing, funding, or evaluating development interventions, understanding heterogeneous treatment effects is no longer optional. It is the difference between programs that work on average and programs that work for the people who need them most.

The average treatment effect is an elegant construct borrowed from clinical trials, where the primary concern is often binary—does the drug work or not. In development economics, the question is fundamentally more complex. We are intervening in heterogeneous populations with varying baseline characteristics, different constraint structures, and divergent capacity to absorb and respond to a given treatment. Reporting only the ATE assumes a uniformity of response that almost never holds empirically.

The problems run deeper than simple variation around the mean. In many interventions, the distribution of treatment effects is multimodal. A conditional cash transfer program might generate two distinct clusters of response: families who use the transfer to overcome a specific binding constraint—say, school fees—and families for whom the constraint is not financial at all but rather distance or safety. The first group shows large effects. The second shows near-zero effects. The ATE splits the difference and tells you something that is true of essentially nobody.

Worse, positive average effects can mask genuine harm to subpopulations. A job training program might raise average earnings while simultaneously displacing workers in adjacent labor markets who were not enrolled. An agricultural input subsidy might raise average yields while accelerating land concentration that pushes marginal farmers out entirely. These distributional consequences are not secondary. They are the policy-relevant question in contexts where equity and inclusion are stated objectives.

The standard approach to handling this—pre-specified subgroup analysis—helps but is inherently limited. You can only test for heterogeneity along dimensions you anticipated before the trial. Gender, age brackets, income quintiles: these are the usual suspects. But the most consequential sources of heterogeneity are often interactions between multiple baseline characteristics that no researcher would think to pre-specify. A program might work extraordinarily well for younger women in peri-urban areas with existing social networks, and barely register for anyone else. Pre-specification cannot find this.

The intellectual honesty required here is uncomfortable. It means acknowledging that many of our most celebrated impact evaluations have told us less than we thought. Not because the studies were flawed, but because the summary statistics we extracted from them were too coarse for the decisions we needed to make. Moving beyond ATE is not about discarding the randomized trial. It is about demanding more from the analysis we conduct after randomization has done its work.

Takeaway

An average treatment effect describes a population, not a person. Before scaling any intervention, ask not just whether it works on average, but whether the average describes anyone at all.

The methodological frontier for heterogeneous treatment effects has shifted dramatically with the introduction of machine learning approaches to causal inference. The most consequential of these is the causal forest, developed by Athey and Imbens, which extends the random forest algorithm to estimate conditional average treatment effects—individual-level predictions of how much a given person benefits from treatment, based on their observable characteristics.

The mechanics are worth understanding. A causal forest constructs thousands of decision trees, each splitting the sample along covariate dimensions to find subgroups with systematically different treatment effects. Crucially, it uses an honesty constraint: the data used to determine where to split is different from the data used to estimate effects within each leaf. This prevents the overfitting that plagues conventional subgroup mining and provides valid confidence intervals for heterogeneous effects. The result is a treatment effect estimate for every individual in the sample, not just the population average.

The practical advantage over pre-specified subgroup analysis is enormous. Causal forests explore the entire covariate space simultaneously, discovering interaction effects that no researcher would have hypothesized. In a recent application to a multi-arm nutrition intervention trial in South Asia, a causal forest analysis revealed that the strongest predictor of treatment response was not income or education—the pre-specified subgroups—but a three-way interaction between maternal age, distance to the nearest health facility, and the number of children already in the household. This kind of structure is invisible to conventional analysis.

Other approaches complement causal forests. Generalized random forests extend the framework to quantile treatment effects and instrumental variable settings. Bayesian Additive Regression Trees (BART) offer an alternative nonparametric approach with natural uncertainty quantification. The sorted effects method of Chernozhukov and colleagues provides a way to visualize the entire distribution of predicted treatment effects and test for meaningful heterogeneity without committing to a particular machine learning architecture.

A critical caveat: these methods discover heterogeneity along observable dimensions only. If the primary source of variation in treatment response is unobserved—motivation, social capital, local institutional quality—machine learning will not find it. The methods also require substantial sample sizes to detect meaningful heterogeneity with statistical precision. For the typical development RCT with 2,000 to 4,000 observations, causal forests may identify broad patterns but will lack power for fine-grained individual-level predictions. Understanding these limitations is as important as understanding the capabilities.

Takeaway

Machine learning does not replace theory in heterogeneity analysis—it reveals where theory was incomplete. Use these tools to discover patterns, then build the economic intuition for why those patterns exist.

The ultimate justification for heterogeneity analysis is not methodological sophistication—it is better program design. If we know who benefits most from an intervention, we can target resources toward responsive populations and either redesign or replace the intervention for those it fails to reach. This is the bridge between evaluation science and development practice, and it remains underdeveloped.

Consider the targeting framework formally. Suppose a cash transfer costs $500 per household to deliver and generates average benefits of $700. The program passes a cost-benefit test. But heterogeneity analysis reveals that the top tercile of responders gains $1,800 while the bottom tercile gains $50. If we could target the top tercile exclusively, the benefit-cost ratio triples. The savings from excluding low-responders could fund an entirely different intervention better suited to their constraints. This is not cherry-picking success—it is rational resource allocation under scarcity.

The implementation challenges are real but not insurmountable. Targeting based on predicted treatment effects requires that the covariates predicting heterogeneity are observable and verifiable at the point of enrollment. Proxy means testing already operates on similar principles in many social protection systems. The innovation is shifting the prediction target from poverty status to treatment responsiveness. Practically, this means building scoring algorithms from experimental data and embedding them in program eligibility systems.

There are legitimate ethical concerns. Targeting interventions away from those who respond least risks excluding the most vulnerable—precisely the populations development programs are designed to serve. The response is not to ignore heterogeneity but to use it constructively. Low responsiveness to one intervention is diagnostic information: it tells us the binding constraint lies elsewhere. The appropriate policy response is not continued delivery of an ineffective treatment but investigation into what would work for that subgroup. Heterogeneity analysis, properly used, is a tool for inclusion, not exclusion.

Scaling decisions benefit enormously from this framework. A program that shows modest average effects in a pilot might contain a subgroup with transformative impacts. Rather than abandoning the program or scaling it uniformly, heterogeneity-informed scaling would expand it selectively while investing in adapted designs for non-responding populations. This requires institutional cultures that value distributional analysis, donor frameworks that reward nuanced evidence, and M&E systems capable of ongoing heterogeneity monitoring. The methodology is ready. The institutional infrastructure is catching up.

Takeaway

Knowing that a program works on average tells you whether to keep funding it. Knowing for whom it works tells you how to redesign it. The second question is harder, more expensive, and far more valuable.

The average treatment effect served the credibility revolution well. It gave development economics a defensible, transparent summary of program impact at a time when the field desperately needed discipline. But we have outgrown it—not as a statistic, but as a sufficient basis for policy.

The tools now exist to look inside the average. Causal forests, quantile methods, and sorted effects analyses let us map the full distribution of who benefits, who is unaffected, and who might be harmed. The question is no longer whether we can examine heterogeneity rigorously. It is whether we will build the evaluation budgets, institutional practices, and policy frameworks that demand it.

Every development program is a bet on a theory of change. Heterogeneity analysis tells you where that theory holds and where it breaks down. The averages were always a starting point. It is time we stopped treating them as the destination.