Data platform complexity is rarely something that happens to an organization. It is something the organization builds, one reasonable decision at a time. Many data platforms are not an architecture; they are the sediment of decisions nobody ever made. The first three parts dealt with problems arriving from outside, namely inherited organizational structures, misdirected incentives, and data that shows something other than it appears to. This final part deals with the problem we create ourselves.

Data landscapes almost never get simpler. Every new requirement adds a layer, every new technology another system, and every unclear responsibility produces a parallel solution. The result is a platform whose operation consumes more capacity than its development. The four models in this part describe how that happens and what counteracts it.

Occam’s Razor

Revenue in the report is seven percent higher than yesterday. Before anybody forms a hypothesis about market behaviour, check the join, check whether a file was loaded twice, check the time zone. Reaching for the interesting explanation first costs hours on a regular basis, and the boring one is usually waiting where nobody looked.

That ordering is what William of Ockham is remembered for: among several explanations of the same thing, prefer the one that needs the fewest additional assumptions. The scope of the rule matters, because it is not a proof and it does not promise that the simplest explanation is correct. It only says in which order to check. The same ordering applies to architectural decisions. Before a new component is introduced, it should be clear why the simpler option is insufficient, not the other way round. This is not merely a matter of taste. Researchers ran deep learning methods against ordinary tree-based models such as XGBoost across 45 datasets and a very large hyperparameter search, and on typical medium-sized tabular data the simpler models still came out ahead, before even counting how much faster they are (NeurIPS 2022). Most enterprise data is exactly that kind of data.

One caveat belongs here. Simple does not mean convenient. The rule says to check the explanation with the fewest assumptions first, not to stop checking after the first plausible answer. Confusing the two is the most common misuse of the model.

Chesterton’s Fence

G. K. Chesterton described a fence running seemingly pointlessly across a field. The hasty reformer wants to remove it because he sees no purpose in it. Chesterton’s answer is that this failure to see the purpose is precisely the reason to leave it standing: someone who does not know the purpose is not qualified to decide on its abolition.

Anyone who has inherited a grown data landscape knows the fence. It is the filter excluding three country codes. It is the transformation applying a different conversion factor to data before 2018. It is the nightly job nobody can explain any more, whose deactivation breaks reconciliation with finance. These constructs look like technical debt. Frequently they are badly documented business logic, which is not the same thing and calls for the opposite response. This is exactly why rebuild projects so often fail in the final third: the core logic is quickly reproduced, but the fences are missing, and without them the numbers do not add up.

The model should not be misread as an argument for standing still. The rule is “understand first, then decide”, not “never touch anything”. What works in practice is a time-boxed investigation whose outcome is documented either way, including when the outcome is that the purpose could not be reconstructed. At that point the fence can be removed deliberately and with a known risk.

The Law of the Instrument

Abraham Maslow put the idea as follows: to someone who owns only a hammer, every problem looks like a nail. Tools shape not only the solution but the perception of the problem itself.

In the data world this operates in both directions, which is easily overlooked. The conspicuous variant is Spark for data volumes that fit into a laptop’s memory, a vector database for two hundred documents, or a language model for a task that a regular expression solves more reliably and a thousand times more cheaply. The inconspicuous variant is just as expensive: the team that solves every requirement in SQL even where a small application would fit better, or the organization that answers every problem with another dashboard because dashboards are what it knows how to build.

The gap can be absurd. A developer took a Hadoop job that chewed through 1.75 gigabytes of chess records in about 26 minutes, rewrote it as a handful of piped command line tools, and finished in roughly 12 seconds on a laptop (adamdrake.com, 2014).That is not an argument against distributed processing. It is an argument about matching the tool to the problem. I would add that the same argument has come back with a different cast: in my experience a single machine running DuckDB will regularly beat a Spark cluster on datasets that fit comfortably into memory, and a great many corporate datasets do. The question I find worth asking is not which engine is faster. It is how many clusters exist because nobody measured the data first.

The price of these choices is rarely the licence. It consists of operational effort, of specialist knowledge held by two people, and of the cognitive load that every additional technology imposes on a team. A workable rule for decision makers is therefore to reverse the burden of proof. The boring solution is the default and needs no justification. The interesting solution needs one, and it has to rest on requirements rather than on anticipated growth.

Gall’s Law

Eighteen months of ingestion, modelling, governance and operating concepts, and not one use case delivered. That is not a project that is behind schedule. It is a project that has no way of finding out whether it is wrong. John Gall wrote the reason down in 1975, and it may be the most practical sentence in this series: a complex system that works has invariably evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched into working, and you have to start again with something simple.

This explains the failure of an entire project category. The large platform programme starts with a complete target architecture, then spends eighteen months on ingestion, modelling, governance and operating concepts, and delivers not a single use case in that time. When the first real user finally encounters the system, it turns out that central assumptions were wrong. By then the system is so interwoven that corrections cost more than a rebuild. Britain’s National Health Service ran the largest version of this experiment anyone has attempted: a complete target architecture for a nationwide electronic patient record, approved in 2002 and dismantled from September 2011. The Public Accounts Committee put the department’s own cost figure at 9.8 billion pounds, noted that this excluded further costs still to come, and found that benefits realised by 2012 amounted to roughly half of what had been spent by then (Public Accounts Committee, 2013). A more recent European example is Gaia-X, launched in 2019 to build a federated data infrastructure. Much of the early work went into architecture, policy rules and governance structures, and by 2021 observers noted that the basic services existed largely on paper while most use cases were still at a conceptual stage; a funding competition worth 190 million euros was then set up to get applications started (Cloudflight, 2021). Whatever one makes of the initiative overall, the sequence is the one Gall describes: the framework first, the users afterwards. The alternative is not less ambition but a different order: first a single complete path from source to decision, for one real use case and one real user. That path may be incomplete, but it has to work end to end. Only then does it get widened.

In AI initiatives this model currently carries especially far. Attempting to build an agent platform straight away, one that orchestrates several systems, selects tools and makes decisions autonomously, fails for exactly the reason described: the errors of the individual steps multiply, and without working individual steps there is no way to tell where things break down. A single, narrowly scoped use case that runs reliably and whose quality is measured is the precondition for everything else, not the unambitious warm-up.

The failure I have seen most often has a particular shape. The programme starts with the technology, the platform gets built, and then no use case arrives, because the people who own the use cases were never in the room and have meanwhile solved their problem some other way. My standing recommendation is the reverse order and it fits on one line: use case first, think big, start local. The target picture is still allowed to be ambitious, and it should be, because a first delivery with no direction of travel becomes the next thing to migrate away from. It just does not get built first.

Incidentally, the resistance to this approach is rarely technical. A narrow working path looks considerably more modest in a steering committee than a complete target architecture. That is where the actual leadership task lies.

Takeaway from Part 4

The four models in this part appear to contradict one another. Occam’s razor and the law of the instrument argue for simplicity, while Chesterton’s fence argues for caution towards things that look complicated. The contradiction dissolves once two kinds of complexity are distinguished. One has grown and encodes knowledge that exists nowhere else. The other has been added and mostly encodes the preferences of whoever introduced it. The first deserves respect, the second does not.

Looking back at the series

This series began with the thesis that data and AI initiatives rarely fail because of technology. Seventeen models later, that can be checked:

  • Part 1, Organization and Collaboration: Conway’s Law, Brooks’s Law, the Dunning-Kruger effect, bikeshedding.
  • Part 2, Metrics and Incentives: Goodhart’s Law, the McNamara fallacy, the streetlight effect, the Hawthorne effect, regression to the mean.
  • Part 3, Data and Judgement: the map is not the territory, survivorship bias, denominator neglect, confirmation bias.
  • Part 4, Complexity and Simplicity: Occam’s razor, Chesterton’s fence, the law of the instrument, Gall’s Law.

Not one of these models is about a technology, and not one is made obsolete by a better tool. They describe structures, incentives, perception and decisions, which are exactly the areas for which no roadmap exists.

The real value of such models does not lie in knowing them. It lies in recognizing the right one in a concrete situation, usually through a familiar sense of unease. When a pilot runs suspiciously well, when a metric improves faster than reality does, when a discussion about naming conventions enters its second hour: those are the moments in which one of these models already contains the answer. The models will not tell you what to do. They will tell you which question you have been avoiding.