In 1957, the government of New South Wales approved a design for an opera house in Sydney. Planners expected it to open in 1963 at a cost of seven million Australian dollars. It opened in 1973, and the final bill was about 102 million.

It is tempting to blame bad luck or incompetence. But the pattern is too consistent to be accidental. Kitchen renovations, software releases, doctoral theses, and the report due on Friday tend to follow the same path. They take longer and cost more than the people closest to them expected, often by a wide margin.

Daniel Kahneman and Amos Tversky named this the planning fallacy in 1979. The puzzle is not that forecasts are sometimes wrong. Uncertainty guarantees some error. The puzzle is that the errors run in one direction, and that people who have been burned before keep making them. Explaining this requires separating how we naturally think about our own plans from how we ought to think about them.

In the 1970s, Kahneman led a team writing a curriculum on judgment and decision-making for Israeli high schools. About a year in, he asked each member to estimate how long it would take to finish. The answers clustered between eighteen months and two and a half years.

Then he asked Seymour Fox, the team's curriculum expert, a different question: how long had similar teams taken? Fox paused. Roughly 40 percent, he admitted, never finished at all. Those that did took seven to ten years. Fox had known this all along. It simply never came to mind when he thought about his own project. The curriculum took eight years.

Kahneman called the first approach the inside view. It focuses on the specific case: the plan, the team, the steps ahead, and the obstacles you can foresee. It feels rigorous because it is detailed. The outside view sets those specifics aside and asks a statistical question: across projects like this one, what usually happens? It anchors on base rates rather than on a story about this particular project.

The inside view usually fails because its details are incomplete, even when each one is accurate. Delays rarely come from the risks you listed. They come from the many unlisted ones: illness, staff turnover, a supplier going under, a regulator changing course. Each is unlikely on its own, but over a long project something almost always happens. Base rates already include those surprises. Your plan does not.

Takeaway

Detail creates confidence, not accuracy. A richly specified plan can feel more trustworthy than a base rate while knowing far less about how projects like it actually unfold.

In a well-known study, Roger Buehler, Dale Griffin, and Michael Ross asked psychology students to predict when they would finish their honours theses. The average prediction was about 34 days. The average actual time was about 56. Only around a third finished by their own predicted date, and fewer than half finished by the date they were 99 percent sure of.

One explanation is motivational. We want projects to go quickly and cheaply, and wanting slides into believing. When Buehler and colleagues offered people an incentive to finish a task early, their predictions became more optimistic, but they did not finish any faster. The incentive changed the forecast, not the outcome. In organisations, the same motive can become what Bent Flyvbjerg calls strategic misrepresentation: low estimates win approval.

Wishful thinking cannot be the whole story, though, because people show the same bias on tasks they barely care about. Cognitive factors do much of the work. To forecast, planners imagine how the work will unfold, and that imagined sequence is usually the best case, with each step going as intended. Participants rarely drew on their past experience when forecasting. When they did recall past overruns, they put them down to external, one-off causes unlikely to recur.

A related failure involves unpacking. When a task is held in mind as a single unit, such as 'write the report', its hidden parts get compressed. Justin Kruger and Matt Evans found that asking people to list subtasks before estimating produced longer, more realistic forecasts. Outside observers also tend to be less optimistic than the people doing the work, partly because they lack the vivid scenario and partly because they look at the planner's track record.

Takeaway

The mind models the future as a story, and stories tend to run smoothly. Optimism in planning comes less from mood than from how we imagine what lies ahead.

If the problem is the inside view, the fix is to bring in the outside view on purpose. Reference class forecasting, developed by Bent Flyvbjerg from Kahneman and Tversky's work, turns this into a procedure with three steps. First, identify a relevant group of past projects. Second, establish the spread of outcomes for that group. Third, place your project within that spread.

The data are sobering. Flyvbjerg's studies of transport infrastructure found that roughly nine in ten projects ran over budget, with rail projects averaging overruns of around 45 percent. These figures describe a whole population of projects, not one unlucky case. A new rail line should start from that record, not from a bottom-up costing that quietly assumes it will beat the average.

The hardest step is choosing the comparison group. If it is too broad, such as 'all IT projects', the comparison loses meaning. If it is too narrow, such as 'ERP migrations at mid-sized logistics firms last year', there is little data. A useful group is large enough to give a meaningful sample and similar enough that its outcomes apply. Adjust for genuine differences only after anchoring on the base rate, and do it cautiously, because every planner believes their project is the exception.

The method has moved from theory into policy. The UK Treasury's Green Book directs appraisers to add 'optimism bias' uplifts based on reference class data, and Denmark has adopted similar practices. The approach also lets decision-makers set their risk tolerance openly. Budget at the median and you accept roughly even odds of an overrun. Budget at the 80th percentile and you buy more certainty at a visible price.

Takeaway

Start with how projects like yours usually turn out, then make the case for why yours might differ. The burden of proof belongs on the claim that you are the exception.

The planning fallacy persists because the inside view is our default. It is vivid, specific, and flattering. Base rates, by contrast, feel abstract, like a description of other people's projects.

Knowing about the bias does little on its own. Buehler's students had missed deadlines before and still underestimated. What works is changing the process: requiring a reference class before an estimate is accepted, breaking tasks into their parts, and asking people outside the project for their forecasts.

For professionals in finance, policy, and marketing, the practical lesson is about where estimates begin. A forecast that starts from the plan will drift toward the best case. A forecast that starts from history makes you justify every step away from it, and that justification is where good judgment happens.