07 · Software Engineering

This chapter is the floor where a program stops being a text that runs and becomes an object that has to survive being changed.

Why does software rot when it has no moving parts?

On 1 August 2012, the trading firm Knight Capital pushed new code to the eight servers that routed its orders to the New York Stock Exchange. It arrived on seven. The eighth still carried a routine called Power Peg, which the firm had stopped using in 2003 and never deleted, and the new code had reused an old feature flag that — on that one machine — switched the dead routine back on. Over roughly forty-five minutes Knight sent millions of unintended orders into the market and lost $460 million, more than the company was worth. The Securities and Exchange Commission's order records one further detail: the dormant code had last been disturbed in 2005, when a function it depended on was moved elsewhere and nobody retested the combination (SEC Release 34-70694, 2013). Knight paid a $12 million penalty and did not survive the year as an independent firm.

Not one line of Power Peg had changed in nine years. That is the whole difficulty of this floor. Software has no bearings to wear, no metal to fatigue, no parts to fall out of tolerance; a program left alone computes tomorrow exactly what it computed today. What moves is everything around it — the hardware beneath, the libraries beside it, the regulations above it, the people who understood it. Meir Lehman gave this its name in the 1970s as the first of his laws of software evolution: a program embedded in the real world must be continually adapted or it becomes progressively less satisfactory. His second law is the sting in the tail — as such a program evolves, its complexity increases unless deliberate work is done to hold it down (Lehman, Proc. IEEE 68(9), 1980). Rot is not decay of the artifact. It is drift of the ground.

Which is why this discipline is not the study of writing programs. Google's engineers put the distinction in one line: "software engineering is programming integrated over time" (Winters, Manshreck & Wright, 2020). Programming produces something that works. Engineering produces something that can be made to work again, by someone else, after the ground has moved. A script you run once and throw away needs none of this floor. A payroll system running in 2040 needs all of it.

The first thing that floor must confront is that not all of the difficulty can be removed. Fred Brooks, who had managed IBM's OS/360 — the largest programming project of its era, and a famous struggle — split the work in two: the essential tasks, "the fashioning of the complex conceptual structures that compose the abstract software entity," and the accidental tasks, the labor of representing those structures in a language and getting them onto a machine (No Silver Bullet, 1986). Better tools attack the accidental half, and they have done so brilliantly — nobody now fights the compiler for a week. But the essential half is the specification of what the thing must actually do, tangled with everything else it must do, and no notation makes a tangled requirement simple. Brooks's prediction was that no single advance would deliver an order-of-magnitude improvement within a decade, because the part that remained was the part that was never about tooling.

If complexity cannot be removed, it can be confined, and this is the field's one great structural idea. David Parnas asked what makes a decomposition into modules good, and rejected the obvious answer — that modules should mirror the steps of the process. His criterion instead: "Every module … is characterized by its knowledge of a design decision which it hides from all others. Its interface … was chosen to reveal as little as possible about its inner workings" (Parnas, CACM 15(12), 1972). A module is a bet about what will change. Hide the decision that will change, and change stays local; expose it, and change propagates. Knight Capital's flag was exactly this failure made visible — a name shared between new code and code no one remembered, a coupling that existed in the machine but in nobody's head.

The instrument that makes change survivable is the test, and its power is narrower than it looks. Dijkstra's corollary is exact: "Program testing can be used to show the presence of bugs, but never to show their absence!" (EWD249). A test suite proves nothing about the system as a whole. What it does is convert a hope — this probably still works — into a specific, mechanically re-checkable claim, and a large system is changeable in proportion to how many such claims it carries. Version control does the same job for history rather than behavior: since Git in April 2005 (Torvalds), a project's past is a queryable artifact, and the question "why is this line here" has an answer that outlives the person who wrote it.

The last principle is the one practitioners discover last. Melvin Conway, writing in Datamation in April 1968: "Any organization that designs a system … will produce a design whose structure is a copy of the organization's communication structure" (Conway's law). Two teams that rarely speak will build two components with a thin, brittle, under-specified interface between them, because that interface is a faithful cast of their conversation. Architecture is therefore partly a staffing decision, and reorganizations are silent design changes.

The field named itself in that same year, at a NATO conference at Garmisch in October 1968, where "software engineering" was chosen deliberately to provoke — an assertion that this work ought to rest on theory and discipline the way established engineering does, made at a moment when it visibly did not (Naur & Randell, eds., 1969). Nearly sixty years on, the aspiration is still doing more work than the achievement, and the honest members of the field say so.

Connections

Below this floor sit chapters 05 and 06: the operating system and the language give you a program that runs correctly once, and everything here begins after that. Chapter 02 supplies correctness for a single algorithm; this chapter is what happens when ten thousand of them must be changed by people who never met. Above and beside it, chapters 08, 09 and 10 are the domains where these disciplines are tested hardest, because a database, a network stack, or a distributed system cannot be stopped and rewritten. Chapter 11 is the case where a leak is not merely expensive but adversarial.

Open questions

The field's own evidence base is thin. Everyone quotes a figure for how much of software's cost is maintenance rather than initial construction, and the figures range from sixty to over ninety percent with no shared definition or measurement window behind them — a discipline that measures its central claim that loosely should be uneasy. Deeper: does machine-generated code shift Brooks's ratio, or does it merely make the accidental half cheaper while leaving the essential tangle untouched — and if reading and reviewing code becomes the bottleneck, is that a new floor or the same one? And if Conway's law holds, is it usable in reverse: can you get the architecture you want by first building the organization that would produce it?

Go deeper

• Titus Winters, Tom Manshreck & Hyrum Wright, Software Engineering at Google, O'Reilly, 2020 — the clearest modern statement of the time-integrated view; free in full at abseil.io.
• D. L. Parnas, "On the Criteria To Be Used in Decomposing Systems into Modules," Communications of the ACM 15, no. 12 (1972): 1053–1058 — full text.
• Frederick P. Brooks, Jr., "No Silver Bullet — Essence and Accident in Software Engineering," IFIP Congress, 1986 — UNC technical report.