Every dataset is a map, which means it shows a territory in simplified form and leaves things out, and both of those are the point rather than the flaw. The first two parts dealt with structures and metrics, meaning with questions that arise before the analysis. This one starts later. Assume the organization works, the metrics are cleanly defined and the numbers are computed correctly. Wrong conclusions still follow reliably.
This is the most uncomfortable part of the series, because these errors cannot be prevented through diligent execution. A report can be technically flawless and still lead somewhere wrong, because of what the map omits, because it has no scale, or because of what the reader brings to it. None of the four models here requires any statistics.
The Map Is Not the Territory
A map is useful precisely because it is not the territory. It throws away almost everything and keeps the few features that help you get somewhere, and a map that reproduced its territory faithfully would be the territory, and useless. Alfred Korzybski gets the credit for saying so, mostly because it is the kind of thing everybody knows and nobody applies.
Every star schema, every KPI report, every customer segment works this way. It keeps what someone once decided was decision-relevant and discards the rest, including the exceptions, the workarounds and the friction that the people doing the work deal with daily. This is not a flaw to be fixed. It is what makes the report readable. The failure comes later, when the abstraction has been in use long enough that everyone forgets it was one, and the green traffic light on the quarterly dashboard becomes the state of the business rather than a summary of eleven selected numbers about it.
The practical rule follows directly. When the dashboard and the lived experience of the people at the front line contradict each other, the sensible default is that reality is right and the map is incomplete. That does not mean the dashboard is wrong. It usually means it is measuring something adjacent to the problem.
Zillow paid for the lesson in public. Its home-buying business relied on a model that estimated what a house was worth and what it could later be sold for, and the model kept producing confident numbers while the market it described moved out from under it. The company exited the business in 2021, wrote down more than half a billion dollars and cut a quarter of its workforce, and its chief executive explained the decision by saying the unpredictability of house prices had far exceeded what they had planned for (Zillow, Q3 2021). The map had not stopped working. It had stopped corresponding.
The statistician George Box supplied the other half of this, and it is the half that keeps the idea from turning into an excuse to ignore data: all models are wrong, but some are useful. The question worth asking of a report is therefore not whether it is true, because it is a simplification and simplifications are never true. The question is whether it is useful for the decision on the table, and that depends on what it left out and whether what it left out matters here.
Survivorship Bias
During the Second World War, the US Air Force examined returning bombers in order to improve their armour. The bullet holes clustered on the fuselage and the wings, and the obvious recommendation was to reinforce exactly those areas. The statistician Abraham Wald disagreed: the reinforcement belonged where the returning aircraft showed no holes, because hits in those places apparently meant the aircraft never came back at all. The dataset consisted exclusively of survivors.
This is the most common way a map loses territory, because data almost always arises where a process was completed successfully. A churn analysis based on surveying current customers is surveying precisely the people who did not churn. A credit scoring model learns from approved applications and knows nothing about the rejected ones, even though their behaviour would be the genuinely interesting information. The support ticket statistics describe the customers who bothered to open a ticket, not the ones who quietly gave up. And the widespread search for the best practices of successful companies ignores that plenty of failed companies applied the same practices and no longer give conference talks.
The effective countermeasure is a single question, asked before any analysis: which cases are systematically missing from this dataset, and would they be relevant to the question at hand? If the answer is unclear, so is the result, no matter how cleanly it was computed.
Denominator Neglect
Forty complaints is not information. It becomes information the moment someone asks: out of how many? Forty out of four hundred is ten percent, forty out of four hundred thousand is 0.01 percent, and neither figure yet tells you whether there is a problem. Only one of them lets you ask.
This sounds too obvious to be worth a section, which is exactly why it survives in so many reports. Absolute counts are easier to produce, they need no agreement on what the base population is, and they are considerably more dramatic. So the incident report says the number of data quality issues rose from 60 to 90 without mentioning that the number of monitored tables tripled in the same period. The adoption slide says the new tool has 1,200 users without mentioning that 9,000 people have access. The AI pilot reports 340 successful interactions without reporting how many were attempted.
The pattern also runs in the other direction, and that version is harder to spot. A percentage without its base is just as empty: a conversion rate that doubled means one thing at ten thousand visitors and nothing at all at twelve. Rates and counts each hide what the other reveals, which is why the useful habit is to insist on both. Whenever a number arrives alone, the question is what it is a share of, and whether that base moved too.
Confirmation Bias
Walk into a dashboard with a suspicion and you will come out with evidence. Not because the data is manipulated, but because a dashboard with a dozen filters offers enough views that some of them will agree with you, and those are the ones that feel like insight while the others feel like noise.
This is the model that implicates the analyst rather than the data, which is why it tends to get skipped. It shows up in small, respectable forms. The analysis that is declared finished at the point where it confirms the expectation, and would have continued for another hour had it not. The segment definition adjusted twice until the effect appeared, each adjustment defensible on its own. The pilot evaluated by the team that proposed it. The stakeholder who asks for the numbers again because these ones do not look right, which is a reasonable request exactly often enough to be dangerous. Analysts have a name for the industrial version: metric shopping, meaning benchmarks, filters and cut-off dates tried in turn until the intervention comes out significantly positive. It is p-hacking carried out with a dashboard instead of a statistics package.
The countermeasures are cheap and almost nobody uses them. State what result you expect before you run the query, in writing, so that the surprise is visible when it happens. Ask what evidence would change your mind, and if there is no answer, the analysis is decoration. And let somebody who does not care about the outcome look at the method, which is the entire reason peer review exists in fields where being wrong is expensive.
Takeaway
The four models in this part share a property that makes them especially dangerous: they produce no error message. The dataset missing the relevant cases returns a result, the number without a base returns a result, the confirming query returns a result, and the report abstracted from reality returns a very tidy result. Everything looks correct because everything was computed correctly.
This is where the popular phrase becomes a problem. An organization that calls itself data-driven has, in the literal sense, handed the wheel to a map. The failure mode in the other direction is just as real: the highest paid person’s opinion overriding the evidence because the evidence is inconvenient, or a decision deferred for months because the result is not yet statistically significant when the cost of being wrong is a week of work. Both extremes are ways of avoiding judgement, one by outsourcing it to a number and one by refusing the number outright.
Mature data organizations are not data-driven but data-informed: the data supplies the evidence, the probabilities and the shape of the risk, while the judgement, and the responsibility for it, stays with a person who also knows what the data left out.
Part 4 (to be published soon), the final one, turns to complexity and simplicity, and to the question of why data platforms so reliably end up more complicated than they need to be.
