A testing company had one tool write an app and another tool test it. Fixing the first real defect turned four passing tests red: they had been asserting the broken behaviour all along. Then it set its human testers and its automated sweep on the same app, and the two found almost entirely different bugs.
CLUJ-NAPOCA, Romania — BetterQA, an independent software testing company, has published what happened when it built an internal application largely with automated coding tools and then had automated tooling test it.
The suite was green. It was green because the defect was still present. When the defect was fixed, four tests went red. They had been asserting the broken behaviour from the moment they were written.
“The code was written by a machine, the tests were written by a machine, and success was reported with total confidence and no uncertainty at all,” said Tudor Brad, founder of BetterQA. “That is the part worth sitting with. Not that it was wrong, but that nothing in the output distinguished it from being right.”
The application is the company’s own, an internal project with no client involved, which Brad said is the only reason the findings can be described publicly.
The company’s human testers then went over the same application and found dozens of genuine defects, which Brad said surprised nobody. What surprised him was the overlap.
“The two sides found almost completely different bugs,” he said. “I expected a large intersection and an argument about who got there first. There was barely an intersection at all.”
Two lists, almost no overlap
The people found things with symptoms a user would feel: a menu that will not scroll on an iPhone after the customer scans the table code; a stock field that accepts a negative quantity and quietly reduces the count rather than refusing; a language setting that breaks the page.
The automated sweep found things with no symptom whatsoever: an audit log that had never recorded a single failed login attempt; database migration descriptions that did not match the database in production; two different password policies enforced on two different screens.
“Almost nothing in the second list has a symptom,” Brad said. “An empty log looks exactly like a quiet week. That one sat there for six weeks. And almost nothing in the first list is something you could search for. The stock field does precisely what its code says. The defect is that the code and the word printed on the button disagree.”
Whose eyes, not whose job title
BetterQA’s argument is that the familiar objection misses the mechanism. The objection is that a capable team needs no dedicated testing role, that quality is everyone’s job, and that the answer is to write better tests.
“I used to think this disagreement was about organisational charts,” Brad said. “It is not. It is about whose eyes. When the author and the reviewer share a blind spot, agreement is not corroboration. It is one mistake wearing two hats.”
He was explicit about what the tooling did better than any person would have. “Nobody on my team was going to read a hundred and four routes and a hundred and sixty-eight access policies, or notice a table that had been empty since July. That half belongs to the machines and I am not going to pretend otherwise.”
“But nobody except a person notices what is not on the screen. No option to resend the code. No way to cancel a bulk action. Automated checking verifies what exists. A human feels the gap.”
What a business can take from it
BetterQA suggests three questions that need no technical background and are more revealing than any coverage figure.
When did somebody last watch a test fail on purpose? A test nobody has seen fail is a hypothesis rather than a safeguard, and the usual answer is that nobody has tried.
What would be different if a given check stopped working? If the honest answer is that nobody would notice, the check is decoration and so is the report it feeds.
Who outside the change tried to break it? Not who reviewed it politely. The person who wrote a feature is the last one able to see how it fails, because the assumption that produced the defect also produced the test.
About BetterQA
BetterQA is an independent software testing company founded in 2018 and headquartered in Cluj-Napoca, Romania, with more than 50 engineers serving clients across 24 countries. It provides software testing, test automation and security testing, and holds ISO 27001, ISO 9001, ISO 14001 and ISO 13485 certification. Businesses wanting a second pair of eyes on their own software can start at https://betterqa.co/qa-outsourcing/
Media contact
Tudor Brad, BetterQA, brad@betterqa.co
