The control that passes the test and does nothing

A security rule written correctly, applied to nine sites, validated with the server’s configuration test, read twice in the file, reloaded without error. And completely inert.

It was not a syntax error. It was not a logic error. The rule blocked archive extensions and blocked them exactly as written. The problem was that the server rewrote the requested path before the rule got to see it, and the rewrite happened only for files that exist on disk. For an invented path, the rule worked flawlessly. For the real file it was supposed to protect, it never fired.

A lock that engages only when the door is already open passes any test you can think to run.

I caught it because I applied the change to a single site first and verified against a file that exists. Had I verified against an invented path, as is usually done, I would have seen the expected 404 and declared success across all nine. The rule would have stayed there for months, decorative, in a compliance report stating that the endpoint was blocked.

The failure class, not the incident

The case above is small. What is interesting is the class it belongs to, because that class has nothing to do with nginx.

There is a category of defects that produce no signal at all. The command exits with code zero. The file reads correctly. The configuration test passes. The service restarts. And the promised effect does not happen. In a change log, such an intervention looks identical to a successful one, and in a report to an auditor it looks even better, because it generated no exception to explain.

From our own operational record, in a single day of applying seven previously prepared changes, each contained at least one defect in the commands or in the verification. Six in total. None was a reasoning error. All were plausible commands, written by competent people, looking in the wrong place: a listing command that paginated by default at 50 entries on a list of 180, a check that searched the application server’s log for blocks made by the proxy in front of it, a loop written for one shell and run with another, which moved no file and reported that it had finished.

The most dangerous of the series: four files deleted from a web directory, confirmed as non-existent on disk, of which two continued to be served publicly with their exact content, from the proxy’s cache. The failure mode that costs is not seeing that the file is still served. It is verifying a path that happened not to be in the cache, receiving a 404, declaring the exposure closed, and leaving the rest public. You turned off all the lights, look through a single window, see it dark, and declare the building empty.

Why human review does not catch this

A person reviewing an artifact looks for a mistake. Here there is no mistake to find, but an absence: something no one asked for, because everyone assumed it was there.

People are good at finding what is wrong in a text and poor at noticing what is missing from one that looks complete. This is not a competence or attention problem. It is a property of the task. The better written the material, the harder it is to challenge, and whoever would want to say “hold on” has to articulate quickly an intuition they cannot yet prove, against a coherent document.

Adding people does not solve it. Four reviewers reading the same artifact with the same mental model of the system produce four confirmations, not four independent checks. If the mental model has a hole, all four miss it. And if an impact-assessment document was produced before the review, it gets worse: the others verify the artifact against the document, not against the real system. The document becomes the anchor.

What generated code changes

An experienced reviewer does not read only the logic. They read the traces too: the place where the author hesitated, the defensive comment added next to a delicate line, the function written differently from the rest because there was something there that did not resolve the same way. These are signals about where uncertainty was, and attention goes there.

Code produced by a model does not have these traces. It is uniform, idiomatic, consistently commented. It looks equally tidy where the problem was understood and where a pattern was filled in. The surface is smooth everywhere, including over the hole. The reviewer loses the proxy they used to allocate attention, with nothing to announce it.

The second change is simpler and more brutal: the volume of artifact produced per unit of human attention has grown by an order of magnitude, and attention stayed the same. A change a team would have carried in their heads for two weeks is produced in two days. It is not just more to read. It is less lived, and understanding comes in good part from having built the thing, not from having read it.

The third, and the one that matters most: constraints not expressed in the code appear nowhere. Isolation between clients on shared infrastructure, for example, does not appear in a function’s signature, does not appear in the types, does not appear in a table’s schema. It lives in the team’s heads. A model generates code that is correct against what it was asked; what it was not asked appears neither as a gap nor as a question. And the people who review have the constraint so deeply internalized that they do not notice it is nowhere in the text. You assume it present precisely because it is obvious to you.

What works

The practical conclusion is not “less automation”. It is that verification has to move from reading to executing, and that invariants have to exist as an artifact separate from the code.

The invariant is written first and runs on its own. Not as a test of the current implementation, but as a statement about the system that is evaluated regardless of what changed and who wrote the change. “No row crosses the client boundary” is a query that returns zero or not. It does not depend on how good the change that just landed looks.

Aggregate, not sample. A check on a thousand correct rows says nothing about the existence somewhere of a wrong one. Isolation properties are relational: they are seen from the other party’s perspective, not from yours. A number computed over the whole set is cheap on an index and is the only one that means anything.

Verification is done from a different point than execution, and later. A check run seconds after a change, from the same context that produced it, hits the same layers that lied the first time: the cache has not expired, the wrong filter returns empty just as convincingly, the pagination looks the same. A re-evaluation at 24 hours, started from somewhere else, breaks all three and asks nothing of the operator.

An empty result is not a result. Before citing a zero as evidence, the filter is validated on a case you know must return something. If it does not find even that, the zero next to it is worth nothing.

Measure in two places. The configuration layer and the layer that actually enforces. When the two disagree, the one that enforces is right. Addresses can be in the tool’s database and absent from the set that filters, and the tool will report success.

What would contradict the thesis

The argument above would be wrong if there were evidence that human review consistently catches constraints not expressed in the artifact. I know of no such evidence, but the claim is verifiable and deserves to be contradicted with data, not with impressions.

It would also be wrong if the defects described were rare. From our record they are not: on a sample of seven carefully prepared changes, six verification defects. It is a small sample from a single environment, so it does not support a general rate, but it is enough to justify the question.

The Monday morning question

How many of your isolation, retention and access invariants exist as an executable assertion, and how many live in the team’s heads?

For each of them, the second question: when did it last run, against what, and how do you know it did not return empty because it was looking in the wrong place?

What this does not solve

Automated verification of the effect does not tell you whether the change was a good idea. It does not replace integrity verification: the fact that an attack endpoint is blocked says nothing about a file already placed on disk. And it does not cover the human preconditions, the kind such as “the owner was notified”, which cannot be verified technically. If one of them is skipped, the only solution is to write in the change log that it was skipped and who decided.

And the part that remains entirely with the human is not small: what the row you delete documents, whether the figure in the report was measured today on this system, and whether the nineteenth file found where the inventory said eighteen is junk or an active piece.