Imagine you are designing a nationwide nutrition program for a low-income country. You turn to the evidence base—systematic reviews, meta-analyses, the accumulated wisdom of hundreds of impact evaluations. The literature tells you that a particular micronutrient supplementation protocol raises child hemoglobin levels by a meaningful margin. The effect sizes look robust. You commit resources. But what if the literature is lying to you—not through fabrication, but through omission?

Publication bias is one of the most corrosive threats to evidence-based development policy, and yet it receives remarkably little attention relative to its consequences. The mechanism is straightforward: studies that find statistically significant, positive results are far more likely to be written up, submitted, accepted, and published than studies that find null or negative effects. The result is a systematic distortion of the entire evidence base. We don't just miss individual findings—we warp the landscape upon which policy decisions are made.

This problem is not hypothetical. Recent registry-publication comparisons in development economics reveal alarming gaps between what is studied and what is published. The file drawer is not empty; it is overflowing. And the consequences compound: distorted primary studies feed into distorted systematic reviews, which feed into distorted policy guidance, which allocates scarce development resources on the basis of inflated expectations. Understanding the architecture of this bias—and the institutional mechanisms that might correct it—is essential for anyone serious about evidence-based development practice.

The term 'file drawer problem' dates to Robert Rosenthal's 1979 observation that journals are repositories of the 5% of studies that show significant results, while file drawers hold the other 95%. In development economics, the problem is arguably worse than in clinical medicine, where regulatory requirements force some degree of disclosure. Development research operates with fewer mandatory reporting structures, and the incentive gradient is steep: a clean, significant result in a top journal can define a career.

Recent work by researchers like Eva Vivalt and others who have systematically compared trial registrations with eventual publications provides sobering numbers. Studies registered on the AEA RCT Registry or RIDIE that report null findings are significantly less likely to appear in published form—and when they do appear, they take substantially longer to reach publication. Estimates vary, but some analyses suggest that roughly 40-60% of completed development RCTs with null primary outcomes remain unpublished years after completion. That is not a marginal gap. It is a structural deficit in the evidence base.

The bias operates at multiple stages. Researchers may choose not to write up null results because the opportunity cost is high—time spent publishing a null finding in a lower-ranked outlet is time not spent on the next project. Journal editors and referees systematically prefer novel, significant findings. Funders, too, may be less enthusiastic about disseminating results that suggest their investments did not work. Each filter compounds the selection effect.

There is also a subtler variant: specification searching and outcome switching. When a pre-specified primary outcome yields a null result, researchers may explore secondary outcomes, subgroup analyses, or alternative specifications until something significant emerges. Registry-publication comparisons have documented meaningful rates of outcome switching in development RCTs, where the published primary outcome differs from the registered one. The published record then overstates both the frequency and the magnitude of positive effects.

What makes this particularly dangerous in development economics is the heterogeneity of contexts. An intervention that works in one setting may genuinely fail in another, and those null results carry critical information about the boundary conditions of program effectiveness. When those results disappear, we lose not just evidence of failure but evidence about where, why, and for whom interventions do and do not work.

Takeaway

The absence of evidence is not evidence of absence—but in a biased literature, it is systematically treated as such. Every null result that stays in a file drawer inflates the apparent effectiveness of the interventions we think we understand.

The promise of systematic reviews and meta-analyses is that they aggregate across individual studies to produce more reliable estimates of intervention effects. They are, in principle, the pinnacle of the evidence hierarchy. But this logic depends on a critical assumption: that the studies being aggregated are a representative sample of all studies conducted. When publication bias systematically excludes null and negative findings, meta-analyses do not correct for individual study noise—they amplify a directional distortion.

Consider the practical implications. A meta-analysis of conditional cash transfer programs that draws on 30 published evaluations may yield a pooled effect size suggesting meaningful improvements in school enrollment. But if 15 additional evaluations with null or small effects were never published, the true pooled effect could be substantially smaller—possibly below the threshold of policy relevance. The meta-analysis becomes not a corrective but a laundering mechanism for inflated estimates, lending the authority of aggregation to a biased sample.

Statistical tools exist to detect and partially adjust for publication bias in meta-analyses—funnel plot asymmetry tests, trim-and-fill methods, p-curve and p-uniform analyses, selection models. Eva Vivalt's work applying these methods to development interventions has shown that standard meta-analytic estimates in development economics frequently overstate true effects by 30-50% or more once bias corrections are applied. These are not small adjustments. They can shift cost-effectiveness ratios enough to change which interventions a funder should prioritize.

The problem is compounded by the way evidence synthesis informs policy. Organizations like the World Bank, DFID (now FCDO), and GiveWell rely heavily on systematic reviews to allocate resources. If those reviews inherit publication bias, then resource allocation itself becomes biased—systematically directing funds toward interventions whose evidence base is inflated and away from alternatives that might perform equally well but lack a critical mass of published positive results. The entire evidence-to-policy pipeline is contaminated.

There is an additional methodological concern: heterogeneity masking. Published studies tend to cluster around significant, positive effects, which compresses the apparent variance across settings. This makes interventions look more consistently effective than they actually are, encouraging one-size-fits-all scaling when the underlying reality demands careful adaptation. The absence of null results doesn't just inflate the mean—it hides the true distribution of effects.

Takeaway

A meta-analysis is only as honest as the literature it draws from. When the evidence base is pre-filtered for success, aggregation doesn't reveal truth—it systematizes overconfidence.

The good news is that the development economics community has begun to take publication bias seriously, and a suite of institutional mechanisms has emerged to combat it. The most foundational is prospective trial registration. The AEA RCT Registry, launched in 2013, and the Registry for International Development Impact Evaluations (RIDIE) now host thousands of pre-analysis plans that specify hypotheses, outcomes, and analytical strategies before data collection begins. Registration creates a public record against which published results can be compared, making outcome switching and selective reporting detectable.

But registration alone is insufficient if null results still never reach publication. This is where results-blind review—sometimes called registered reports—offers a more structural intervention. Under this model, journals commit to publishing a study based on the quality of its design and pre-analysis plan, before results are known. The publication decision is decoupled from the findings. Several journals have adopted registered report formats, and early evidence suggests they produce a dramatically higher proportion of null results, consistent with the hypothesis that the traditional model suppresses them.

Funder policies represent another critical lever. Organizations like 3ie (the International Initiative for Impact Evaluation) and J-PAL have implemented requirements or strong incentives for grantees to publish all results, regardless of direction or significance. Some funders now require that final reports be made publicly available even if journal publication does not occur. The UK's DFID historically required publication of all funded evaluation reports through open-access repositories. These policies create a parallel dissemination channel that bypasses journal gatekeeping.

Pre-print servers and open-access working paper series—such as the NBER Working Papers or BREAD working papers—also help by lowering the cost of making null results available, even if they never reach a peer-reviewed journal. The growing norm of posting pre-prints before submission means that the community can access a broader slice of the evidence base, though discoverability and citation patterns still favor published work.

The most ambitious proposals go further: creating dedicated journals or sections for null and replication results, reforming tenure and promotion incentives to reward methodological rigor over novelty, and building infrastructure for living systematic reviews that continuously incorporate new evidence—including null findings—rather than relying on periodic snapshots. None of these solutions is sufficient in isolation. But collectively, they represent a serious institutional response to a problem that, if left unaddressed, undermines the foundational promise of evidence-based development.

Takeaway

Fixing publication bias requires changing the incentives at every stage—from researcher to journal to funder. The goal is an evidence ecosystem where what gets published reflects what was found, not what we hoped to find.

The evidence base in development economics is not a neutral archive. It is a curated collection, shaped by incentive structures that systematically favor positive, significant findings and suppress the rest. Every policy recommendation built on this distorted foundation carries an unknown margin of overoptimism—unknown precisely because the counterfactual evidence was never published.

This is not a counsel of despair. The tools to address publication bias exist and are improving: registries, pre-analysis plans, results-blind review, funder mandates, and open dissemination norms. What is required is the institutional will to implement them at scale and the professional culture to reward transparency over novelty.

For development practitioners, the practical implication is clear: treat every evidence synthesis with calibrated skepticism. Ask what might be missing. Seek out registrations without corresponding publications. Demand bias-adjusted estimates. The integrity of evidence-based development depends not on the studies we see, but on our willingness to reckon with the ones we don't.