Insight
2,153 passing tests and a product that could not send.
A project of mine had 2,143 passing tests, a clean type check, a clean linter and a twenty-screenshot browser QA pass. I ran a release audit against it anyway and scored it 36 out of 100. NOT READY.
Open the diagram description
A diagram on a near black background with one acid green accent, showing why a fully green test suite said nothing about the one part of the product that was broken. On the left stands a tall panel titled The sending layer and labelled the call site. It says that every test exercised what to send, how to batch it, how to back off and what to do when the far end says wait, and that all of it worked. From the right edge of that panel two arrows fork out, and both cross a vertical dashed line running down the middle of the picture, which is labelled the seam. The upper arrow is labelled in tests and arrives at a panel titled A fake port, labelled in every test, which says the fake was the right tool while the real one was being written, and that it stayed. A further arrow to the right of that panel leads to the words 2,143 green, a clean type check and a clean linter. The lower arrow is labelled shipped and arrives at a panel outlined in acid green, titled The production adapter and labelled what shipped, which says the method the whole product existed to perform raised the error called NotImplementedError. A further arrow to the right of that panel leads to the words could not send, since the day the module was written. Between the fake port panel and the production adapter panel there is a vertical link drawn as two short faint dashed stubs with an open gap in the middle, standing for a comparison that was never made. Beside that gap, a warm gold label reads none, ever, at any point, and under it a fainter line reads, no evidence the fake behaves like its production counterpart. A rule runs across the foot of the picture. Under the rule a heading reads, the same shape one layer up, followed by a note that browser quality assurance drove the real front end against a mock bridge that made the launch step always succeed, and a closing line: so the screenshots showed a product working, they could not have shown anything else.
The core network call had never been implemented. The method the whole product existed to perform raised NotImplementedError in the production adapter, and had done since the day the module was written.
How does a suite that size miss the one thing that matters?
By testing everything around it.
The sending layer was built and tested against fake ports, which is normal and correct while the real one is being written. The fake ports stayed. Every one of those tests exercised the code that decides what to send, how to batch it, how to back off, what to do when the far end says wait. All of that logic was real and all of it worked. It called a port, and the port it called in tests was not the port that shipped.
The browser QA had the same shape one layer up. It drove the real front end against a mock bridge, and the mock bridge made the launch step always succeed and completed every job instantly. So the screenshots showed a product working. They could not have shown anything else.
What is the actual lesson, if it is not “write more tests”?
A green suite tells you the seams you mocked are the seams you mocked.
That is not a criticism of mocking. A test that reaches a real network is slow, flaky and often impossible in CI, and a fake port is the right tool. The defect is in the inference: 2,143 green means the mocked composition is sound, and it says nothing at all about whether the mock resembles the thing it stands for. Those are different claims and the number is the same for both.
The tell is structural rather than statistical. Ask which seams every test crosses in a fake, and then ask what evidence exists that each fake behaves like its production counterpart. On that project the answer for the send port was: none, ever, at any point. It was findable in an afternoon by asking the question, and unfindable in any length of time by reading the test count.
What does a release audit do that a test suite cannot?
It reads the artifact rather than the code.
The same audit found that the shipped installer contained an application that had been deleted from the repository, and that the documentation described that deleted application, so the operator’s recovery path pointed at software that no longer existed. No unit test has an opinion about the contents of an installer.
After a phase of fixes the suite read 2,153 green, ten more than before, and the verdict stayed NOT READY. That is the part worth keeping. What moved the number was ten tests around a real implementation; what kept the verdict was a list of things no test can substitute for, starting with one supervised run against the real thing.
A check nobody has watched go red is not a check, and a suite that has never crossed the seam it is protecting is a suite about something else.
I run this audit on client work before a release for the same reason. It is cheaper than the alternative, which is finding out from the person who was relying on it.
Green suites and broken seams
Which seam has never been crossed for real?
Tell me what your suite mocks and what it has actually run against. I will tell you where I would look first.
Ask Dan on WhatsApp (opens in a new tab)or email dan@burdetsky.xyz