Imagine a pharmaceutical company presenting regulators with three clinical trials, each showing that its drug outperforms placebo. Every study is genuine. Every statistical test is sound. Every inference from data to conclusion holds. And yet, if four other trials showing no effect sit unpublished in a filing cabinet, the argument as a whole is deeply misleading.

Formal logic has no tool for detecting this failure. Validity concerns the relationship between premises and conclusion. It says nothing about the premises that were never stated. An argument can be valid, its premises true, and its conclusion still unwarranted, because the case was built from a curated subset of what is known.

Philosophers Rudolf Carnap and Carl Hempel named the corrective: the principle of total evidence. Rational belief must rest on all available relevant evidence, not merely the portion that flatters a preferred conclusion. It sounds obvious. In practice, it is one of the most frequently violated norms in debate, law, and professional life.

Cherry-picking rarely looks like deception from the inside. Most people who argue selectively believe they are simply presenting the strongest case. The distortion operates upstream of conscious choice, in what we notice, what we search for, and what we remember. Confirmation bias directs attention toward supporting evidence before any deliberate selection has occurred. By the time we begin building an argument, the evidence pool has already been filtered.

Consider how this plays out in Toulmin's model of argument. A complete argument includes not only data and warrant but also rebuttal conditions, the circumstances under which the claim would fail. Selective evidence does more than tilt the balance. It quietly deletes the rebuttal. The argument appears unqualified precisely because the information that would qualify it was never gathered. What remains is a claim that sounds stronger than its evidence can support.

The effect is systematic rather than random. Random omissions might cancel out over time. Selective omissions accumulate in one direction. A lawyer who reviews only the depositions favourable to a client, a policy analyst who cites only the countries where a reform succeeded, a manager who recalls only the projects a favoured employee delivered on time: each compounds a small skew into a confident and unjustified conclusion.

There is also an audience dimension. Chaim Perelman observed that arguments are addressed to audiences, and that what counts as persuasive depends on what the audience already knows. Cherry-picking exploits the gap between the arguer's knowledge and the listener's. It persuades those who lack access to the full picture, and it collapses the moment someone who has that picture enters the room. An argument that only survives in ignorance has not earned its conclusion.

Takeaway

The strength of an argument cannot be judged by what it contains alone; it must be judged against what it leaves out. Before deciding how convinced to be, ask what a fair-minded opponent would add.

If selection bias operates below awareness, good intentions are not enough to correct it. We cannot easily search for evidence we are unconsciously disposed to overlook. The practical solution is structural: recruit people whose biases run in the opposite direction. This is the logic behind every well-designed institution of contested reasoning.

Daniel Kahneman popularised the term adversarial collaboration for an arrangement in which researchers who disagree design a study together. Before any data are collected, both sides agree on the methods, the measures, and, crucially, which results would count against their own position. The disagreement is not eliminated; it is harnessed. Each party polices the other's blind spots.

The principle extends well beyond science. Legal systems institutionalise it through adversarial procedure and discovery rules, which compel each side to disclose evidence the other might use. Intelligence analysts deploy red teams tasked with building the strongest case against a prevailing assessment. Competitive debaters are trained to argue both sides of a resolution, which forces them to know the opposing evidence as well as their own.

The less formal version is available to anyone. Before committing to a position, present it to the most capable person you know who disagrees, and ask a specific question: what evidence am I missing? Not 'what do you think,' which invites a counter-speech, but a request for the data points that should trouble you. Disagreers can be irritating, but they are uniquely motivated to find what you have overlooked. Treating them as an evidential resource rather than an obstacle changes the character of the exchange. The goal is not to be persuaded. It is to be fully informed before you decide.

Takeaway

Your opponents are the most efficient search engine for the evidence you are least likely to find on your own. Their motivation to prove you wrong is a resource, not merely a threat.

Collaborators are not always available, so the principle of total evidence also needs solitary methods. An evidence audit is a deliberate check on whether your case reflects the full body of relevant information or merely the part that arrived first or felt most agreeable. It works best before a conclusion hardens, while revising it still costs little.

Start with a search symmetry test. Review how you gathered your evidence. Did you search for the benefits of remote work but never its costs? Did you read sources you already trusted and skim the rest? Asymmetric search produces asymmetric evidence, however honestly you evaluate what you find. Rerun the search from the opposing direction and note what surfaces.

Next, apply a disconfirmation inventory. Write down what you would expect to observe if your conclusion were false, then check whether you have looked for those observations. Many confident beliefs survive only because nobody asked what their failure would look like. Pair this with a denominator check: when you cite successful cases, ask how many attempts there were in total. Three triumphant examples mean little without knowing whether they came from five tries or five hundred.

Finally, perform a steelman reconstruction. Write out the strongest version of the opposing argument, with its best evidence, in terms its advocates would recognise. If you cannot do this convincingly, you have not yet surveyed the evidence; you have surveyed your side of it. None of these methods guarantees completeness, since total evidence is an ideal rather than an achievable state. But each narrows the distance between the evidence you have considered and the evidence that exists, and that distance is where unwarranted confidence lives.

Takeaway

Before trusting a conclusion, ask not only whether the evidence supports it but how that evidence reached you. The path the evidence travelled often explains more than the evidence itself.

The principle of total evidence exposes a limit of formal analysis. An argument can be flawless in structure and still fail as reasoning, because its weakness lies in what was never put on the table. Evaluating real-world arguments means evaluating their evidential base, not just their inferences.

A practical framework follows. Recognise that selection operates before deliberate choice. Recruit disagreers who can locate what you missed. Audit your search, your disconfirmers, your denominators, and the strongest opposing case.

Applied consistently, these habits do more than protect you from being misled. They change what it means to win an argument. A conclusion that has faced the full evidence is not merely persuasive; it is earned.