In the summer of 1663, Christiaan Huygens crossed the North Sea carrying something that could not be written down. Two years earlier he had reported a strange phenomenon from his air pump in The Hague. Water purged of air remained suspended in a glass tube even after the surrounding vessel was evacuated. Robert Boyle, whose own pump was the most celebrated instrument in England, could not make it happen. The Royal Society's apparatus refused to cooperate. The anomaly existed, it seemed, only in Holland.
So Huygens came to London. Working alongside the Society's curators, he coaxed the English pump into producing the effect, and the anomalous suspension entered the shared ledger of natural philosophy. What crossed the sea was more than a claim. It was a practice: a feel for seals and valves, and the patience needed to purge water of its hidden air. The finding became knowledge only when it could travel.
That episode, reconstructed by historians Steven Shapin and Simon Schaffer, captures something the modern replication debate often obscures. Reproducibility is not a single test that a finding passes or fails. It is a negotiation between claims, instruments, people, and the unwritten skills that bind them together. To understand how science turns private observation into public, reliable knowledge, we need to understand the varieties of reproducibility and what their failures actually tell us.
Three Kinds of Doing It Again
The word reproducibility carries more weight than any single meaning can bear. In 2019 the US National Academies tried to impose order. They separated reproducibility, meaning consistent results from the same data and the same code, from replicability, meaning consistent results across studies that collect new data. Even this division hides important gradations, and different fields still use the two terms in opposite ways. The confusion is not merely semantic. Each type tests a different link in the chain from observation to conclusion.
Direct replication repeats an experiment as faithfully as possible: same procedures, same measures, new sample. It asks a narrow question. Was the original result a product of sampling, error, or chance? Its strength is diagnostic precision. If a carefully matched repetition fails, the space of possible explanations shrinks. Its weakness is that perfect duplication is impossible. Participants differ, reagent batches change, decades pass. Every replication is, strictly speaking, a new experiment wearing an old one's clothes.
Conceptual replication tests the same underlying hypothesis with different methods, operationalisations, or populations. This is how theories earn generality. A principle confirmed under only one protocol remains a fact about that protocol. Yet conceptual replication harbours a subtle asymmetry. When it succeeds, it is celebrated as converging evidence. When it fails, the difference in method offers an easy excuse. Psychologists Harold Pashler and Christine Harris warned that a literature built mainly on conceptual replications can pile up apparent support for effects that never existed.
The third type, reproducibility of analysis, concerns the step between data and claim. Identical data handed to different analysts should yield identical conclusions, but often does not. In a widely discussed 2018 project led by Raphael Silberzahn and Eric Uhlmann, twenty-nine teams analysed the same dataset. They asked whether football referees give more red cards to darker-skinned players. Estimated odds ratios ranged from slightly below one to nearly three. Twenty teams found a significant effect and nine did not. Nobody fabricated anything. They simply made different, defensible choices.
Taken together, the three types form a hierarchy of robustness. Analytical reproducibility establishes that a result follows from its data. Direct replication establishes that the data were not an accident. Conceptual replication establishes that the phenomenon is not an artefact of one particular method. A finding that survives all three has moved from observation toward something approaching knowledge. Most published findings, of course, have attempted none of them.
TakeawayEach form of reproducibility tests a different joint in the argument. Asking whether a finding replicates is incomplete until you specify which joint you are stressing.
Reading the Silence of a Failed Replication
When a replication fails, the result is ambiguous by construction. The original finding may have been false. The replication may have been flawed. Or the two studies may have differed in some way that matters. The sociologist Harry Collins called this the experimenter's regress. To know whether an experiment was performed competently, you need to know the correct result. To know the correct result, you need a competently performed experiment. Collins studied physicists building TEA lasers in the early 1970s and found that written instructions alone rarely sufficed.
Sometimes the hidden variable is real and strikingly mundane. In 2014, the laboratories of Mina Bissell in Berkeley and Kornelia Polyak in Boston described more than a year spent trying to understand why their measurements of breast tissue cells kept disagreeing. They used the same protocols and the same markers, yet got divergent results. The culprit turned out to be how the tissue was agitated during digestion. One lab stirred it vigorously, while the other rocked it gently. The failure did not expose a false claim. It exposed an unrecorded condition.
But hidden moderators can also become a refuge. Any failure can be explained away by pointing to some unmeasured difference, and an explanation that can absorb every outcome explains nothing. The Many Labs projects tested this directly by running identical protocols across dozens of sites and cultures. For most effects examined, variation between sites was modest. Effects that replicated tended to do so broadly, and those that failed tended to fail everywhere. Context matters, but it is not an infinitely elastic alibi.
Interpreting failure also demands statistical literacy. A non-significant replication is not evidence of absence if it lacked the power to detect a plausible effect. Uri Simonsohn proposed asking whether a replication can rule out effects large enough for the original study to have detected them, an approach he called the small telescopes framework. Equivalence tests and Bayesian estimates serve similar ends. They shift the question from whether an effect was found to how tightly the new result constrains what remains possible.
Finally, scientists weigh prior plausibility, and rightly so. In 2011 the OPERA collaboration reported neutrinos that appeared to travel faster than light. Few physicists abandoned relativity. The team itself invited scrutiny, the anomaly was traced to a loose fibre-optic connection, and the independent ICARUS experiment measured speeds consistent with light. The reasoning was not dogmatism but proportion. A finding that contradicts deep theory demands stronger evidence than one that sits comfortably within it. Interpretation is always comparative, never isolated.
TakeawayA failed replication is not a verdict but a new datum. Its meaning depends on statistical power, prior plausibility, and how precisely each study specified the conditions that produced its results.
Designing Work That Can Travel
The geophysicist Jon Claerbout offered a provocation that has aged remarkably well. As paraphrased by statisticians Jonathan Buckheit and David Donoho, an article about computational science is not the scholarship itself but merely advertising for it. The actual scholarship is the complete software environment and the instructions that generated the figures. Building reproducible research begins with taking this inversion seriously. The paper is a summary, and the underlying work must be packaged so that others can run it again.
For analytical reproducibility, the core practices are now well established. Keep raw data immutable and separate from processed data. Script every transformation instead of editing spreadsheets by hand. Use version control so that every result traces back to the exact code that produced it. Record software versions, or better still, capture the full environment in a container. Share data and code in persistent repositories with clear licences. The test is simple: could a stranger regenerate your figures without emailing you?
For replicability, the essential discipline is separating exploration from confirmation. Preregistration records hypotheses, sample sizes, and analysis plans before data are collected. This constrains what Andrew Gelman and Eric Loken called the garden of forking paths. Registered Reports, launched at the journal Cortex in 2013 under Chris Chambers, go further. Peer review happens before results exist, and acceptance does not depend on whether the hypothesis is supported. The pressure to produce a clean story weakens when the study is accepted in advance.
Methods sections remain the weakest link, because they are written to satisfy reviewers rather than to instruct successors. The lesson of Huygens and of the Bissell and Polyak collaboration is that tacit knowledge must be deliberately made explicit. Detailed step-by-step protocols, photographs and videos of procedures, reagent lot numbers, and honest notes about what went wrong all help. Where analytical choices are genuinely open, a multiverse analysis reports results across all reasonable specifications. It shows readers how fragile or robust a conclusion really is.
Beneath these techniques lies a cultural shift. Statistical power should be planned rather than hoped for. Internal replication before publication, adversarial collaborations with sceptics, and openness to independent checks turn reproducibility from a threat into a design principle. None of this eliminates creativity. It changes where creativity lives. Ingenuity belongs in the bold hypothesis and the inventive experiment, not in the flexible analysis that coaxes a result out of noise.
TakeawayDesign every study as though a capable stranger will try to rebuild it from your materials alone. The knowledge you produce is only as durable as what you leave behind.
Kuhn observed that normal science advances by solving puzzles within shared commitments, and that anomalies accumulate until a paradigm begins to strain. Reproducibility is what makes an anomaly trustworthy enough to matter. A strange result that cannot be repeated is a curiosity. One that recurs across laboratories becomes a crisis worth having.
This reframes the relationship between rigour and creativity. Replication is often cast as conservative housekeeping, the dull counterpart to discovery. Yet every failure to replicate is a small mystery. Some of the most instructive episodes in science, such as a stirring technique, a loose connection, or a method that refused to travel, emerged precisely there.
Perhaps the deeper lesson is that scientific knowledge is less a collection of facts than a set of performances that others can repeat. Huygens seemed to understand this when he set sail for London. The creative act lies not only in seeing something new, but in making it visible to everyone else.