BrainIT Consulting Master Build No. 1

Review what the agent wrote

Part 3 of 5 · 3,206 words · about 15 minutes

▶Watch this part on YouTube

Master Builds takes one real application from an idea to a system you own, run, and remain accountable for. The application is Countercheck. It reads an accounts-payable export, reports the payments that look wrong, shows the rows behind each one, and never writes anything back.

About fifteen minutes to read. Two hours to complete the card at the end, and you can do all of it without reading a line of code.

Part 1 decided what Countercheck is for. Part 2 put it on accounts I own. This part opens it up and asks whether what was built is what was specified.

• What a person who cannot read code can actually check, which is more than it sounds
• A real review of this application, including what the review itself got wrong
• The Review Card: what to demand, and what to refuse to accept

1. You cannot read it, and that is not the problem you think it is

You have been told that agents can write your software now. You have probably seen one do it. The question that follows is not "can it write code" — you have your answer — but "how would I ever know if it were wrong?"

The honest starting point is that you cannot read it. Not this year, and probably not ever, because learning to read a codebase well enough to audit it is a career rather than an afternoon. Anyone who tells you to learn just enough to check the work is selling you a comfortable feeling.

That matters less than it sounds, because three things are checkable without reading a line of code:

1. Does it do what the specification said.
2. Does it refuse what the specification refused.
3. Do its own numbers agree with each other.

Every one of those is a comparison between two documents you can read: what was promised, and what the thing in front of you actually does. Neither of them is code.

Three of the most serious findings in the review of this application were found exactly that way, and I will show you all three.

2. What a review is, and what it is not

Six reviewers went through Countercheck, one lens each: the detection rules, the file import, storage and state, the interface, the in-app manual, and whether the product's claims about itself hold up.

Between them they raised sixty-five findings — six critical, nineteen high, thirty medium, ten low. The document's own summary states those counts, and counting the headings gives the same numbers. That is worth saying out loud, because it is the first thing that could have been wrong and was not.

Why six lenses rather than one reviewer six times

Because of the overlap, which is the most useful thing in the whole exercise.

• Three reviewers independently flagged the same thing: a question about date order that the product promises never to guess, arriving pre-answered.
• Four found the same broken button for reopening a decision.
• Two found the same headline total rounded to whole pounds.

Twelve of the sixty-five are the same defect seen through a second lens. That is not waste. Agreement between independent reviewers is the strongest signal you will get, and it exists only because the lenses ran separately. One reviewer looking six times produces one mention of each and no way to tell which mattered.

If you commission a review and it comes back as a single list from a single pass, you have lost that signal and nobody will tell you it is missing.

3. A finding is a claim

Here is the part that changes how you should read any review you are handed.

Every one of the sixty-five findings was passed to a different agent whose only instruction was to refute it. That refutation pass is the most valuable thing in the exercise, and it did not finish. It was cut short part way through.

The document says so in its own second paragraph. It calls the list, and I am quoting, "as raised, not as confirmed".

Two findings were checked against the running system before the cut. Both were real.

So sixty-five is a count of claims. It is not a count of defects, and the two words are not interchangeable.

The number I am not going to give you

You may want to know what share of the sixty-five were wrong. I am not going to tell you, because nobody measured it. The run stopped before it could be measured, and any percentage I put here would be a number I made up to make this part feel more finished than it is.

That is the exact failure this part is about. A confident figure replacing "we did not finish checking" is how a lead list turns into a defect list without anyone deciding that it should.

A finding nobody tried to disprove is a lead, not a defect. It may well be real — a lead does not stop being real because nobody got to it — but you should know which one you are holding.

4. Green tests, wrong product

Here is what one of those claims was worth.

A rule inside Countercheck stops a monthly cleaning contract being reported as a duplicate every month. It is the single most important piece of code in the detection engine, because without it every subscription and standing contract in a ledger becomes an accusation and people stop reading the findings.

It recognised monthly, and nothing else. Weekly, fortnightly, twice a month — none of them. Any of those fell inside the window the duplicate rule fires on, unguarded.

I added one legitimate weekly retainer to the demo ledger. The findings went from six to fifty-seven.

Now the part that should worry you: every test passed, before and after. The tests asked whether the code did what it was written to do, and it did. Nothing was broken. The product was unusable.

A test suite cannot catch that, because the test suite agrees with the mistake. It was written from the same understanding as the code, usually in the same sitting, often by the same agent. Green tests tell you the code matches its author's intention. They tell you nothing about whether that intention was right.

This is why review exists as a separate activity from testing, and why "all the tests pass" is not an answer to "is it correct".

5. What checking all sixty-five actually found

Seventeen defects were fixed from that review at the time, in three commits. Each commit subject says in plain English what was wrong — that is a thing you can ask for and a thing you can read.

Then I did what the review had not: I went through all sixty-five, opened the code each one names, and wrote down a verdict with the evidence beside it. Where a claim could be executed, I executed it. A verdict with no evidence beside it is not a verdict.

Eighteen were still live — more than a quarter of the list — three days after it was written. Not because anyone was careless. Because a backlog is a backlog, and a review that produces sixty-five findings produces more work than a week has room for.

Three of the five that remain are decisions rather than defects: a judgement about what a number means, a judgement about what a screen should say when there is nothing to report, and one narrow case where the fix moves more than it repairs. They are open because somebody has to decide, not because nobody looked.

6. Three the writing could have told you

Now the promise from section 1, kept. The three worst findings are ones you could have found, and all three are the same shape: the writing and the code disagree.

The check that was not being made

Countercheck's manual says it flags a round figure — a suspiciously neat amount from a supplier who normally invoices to the penny — above five thousand pounds.

The code fired only on exact thousands.

So £7,500 returned nothing. Not an error, not a warning. Nothing. And nothing reads as checked, and clean.

That is the worst kind of defect a checking tool can have, because its failure mode is silence. A tool that crashes tells you it failed. A tool that quietly does not perform a check it claims to perform tells you that you are fine.

You could have found this with the manual in one hand and the product in the other. No code required. It is fixed now — roundness is a multiple of five hundred, and the manual says so too.

Two claims, two chapters apart

Countercheck's manual quotes an industry band for how often this sort of problem appears in accounts payable, and uses it as the reason to care.

Its own demo ledger produces findings at a rate above the top of that band.

Nothing false is stated in either place. The band is quoted accurately, and the demo is what it is. But the two sit two chapters apart in the same document, and read together the product is quietly claiming a hit rate better than the problem it describes. The framing sells itself.

This is the hardest kind of finding to catch and it needs no code at all — only somebody willing to read a document against itself. It is still open, and it is open in public, which is the only honest place for it.

A promise the software makes and does not keep

Upload your own export to Countercheck and it reads the file, maps the columns, analyses the rows — and keeps them. There is no button anywhere that deletes them, and no function in the code that could. I checked every delete path in the storage layer: the demo ledger, one named test fixture, and abandoned uploads that were never confirmed. Nothing else.

The upload page now says so, in plain words. That is the honest half.

The button still does not exist. That is the other half, and it is not fixed in this part on purpose. Part 4 builds it. Patching it quietly between parts and presenting this one as clean would be the same move this series keeps arguing against.

Added 13 August 2026, after this part was published. The button now exists. Open the review queue for a ledger you imported and press Delete this ledger; it removes the rows, the findings, the decisions and the audit trail, and only the person who uploaded it can do it. The paragraph above is left as it was written, because it was true when it was filmed and the video still says so. Part 4 covers what had to be built first: until a ledger recorded whose it was, there was nobody this could safely be offered to.

7. What this cost, and what it did not

Some numbers about the review itself, because a method you cannot afford is not a method.

The review ran in a few hours of machine time and produced sixty-five findings across a codebase of about six thousand lines. Classifying all sixty-five by hand took most of a working day, and that day was the valuable part — not because the classification is clever, but because it is the step where claims become facts or stop being anything.

What it did not cost is instructive too. Nobody had to trust a reviewer's judgement, because every finding names a file and a line. That is the property that makes a review checkable by somebody who cannot read the code: you can hand the finding and the file to a second reader and ask whether the thing described is there.

Demand that property. A finding that says "the error handling is inconsistent" is an opinion. A finding that says "this line reports a count where the previous line reported a sum" is a claim you can settle.

8. One more, found while making this

While recording the video for this part, a screen capture of the review queue showed the demo ledger described as "12 months of accounts payable". The home page and the manual both say eleven months, and eleven is correct: the generated span is 336 days.

Two documents agreed, a third disagreed, and the third had been sitting inside the product for months. It was not found by reading the code. It was found by looking at a picture of the product.

There is a second lesson under it. Countercheck computes its findings once and stores them, so a fix to the wording of a finding does not change anything until the data is regenerated. The corrected wording had shipped two days earlier; the live product was still showing the old text, because the stored rows predated the fix. A deployed fix and a live fix are not the same event.

Both are in the video. Neither was planned to be.

9. The Review Card

Fill this in for any software you are handed, whoever built it.

1. What ran

• How many independent reviewers or passes, and did they run separately?
• What was each one looking at?
• Anything explicitly out of scope? Security is a common and legitimate exclusion; an unstated one is not.

2. What was verified

• Which findings were checked against a running system, and how?
• Did the verification pass finish? If not, say so on the document.
• Is the list labelled as raised, or as confirmed? These are different documents and only one of them is a defect list.

3. What each finding names

• Does every finding name a file and a line?
• Can a second person settle it without the first person's judgement?
• Reject "the error handling is inconsistent". Accept "this returns a count where the caller expects a sum".

4. What was fixed

• Does each fix say, in plain English, what was wrong?
• Is that description checkable against the change?
• A commit message reading "fixes" is a fix you cannot audit.

5. What is still open

• Get it written down, with a verdict on each: fixed, still true, never true.
• Note which open items are decisions rather than defects. They need a person, not a sprint.
• Ask when it was last checked. A backlog decays.

6. What you can check yourself

• Read the product's documentation against its own screens. Every claim in the manual is a testable statement.
• Look for checks that are promised and silently absent — an absent finding reads as clean.
• Look for two numbers in the same document that do not agree.

10. What to do this week

Ask for the review you already paid for. If software was built for you in the last year, ask whoever built it what review it had. The answer tells you what you bought.

Take one page of the documentation and check it against the product. One page, one hour. Do not choose the overview; choose the page that makes specific claims about what the software does. Note every place they disagree.

Ask for the open list. Not "is it done" — ask what is known to be wrong and not yet fixed. A team that can produce that list in an hour is a team that has one. A team that cannot is telling you something.

11. Where this leaves Countercheck

It has had a real review, and the review has had a review. Seventeen defects fixed from it at the time, thirteen more found live and fixed in a day, five open and named in public.

The application is at <https://countercheck-brainit.vercel.app> and the demo ledger is readable without an account. The code is at <https://github.com/brainit-consulting/countercheck> under an MIT licence, and the classification of all sixty-five findings is in the repository beside it.

Part 4 decides who may come in — and builds the delete button this part found missing. Part 5 runs it for a year.

Sources and limits

The review. REVIEW-2026-08-08.md in the application repository. Six reviewers, one lens each; sixty-five findings at six critical, nineteen high, thirty medium, ten low. Its own second paragraph records that the refutation run was cut short by a session limit and describes the list as "as raised, not as confirmed", and that two findings were verified empirically before the cut.

The classification. All sixty-five were classified against the code on 11 August 2026, with the evidence recorded per finding. Where a claim could be executed it was executed, including by driving a real browser with real mouse and keyboard events. The file is published in the repository.

No figure is given for how many findings were false. It was never measured, because the verification run did not finish. That absence is deliberate and it is the point of section 3.

The seventeen fixed at the time are the counts stated in three commit subjects: four, seven and six. Other commits in the same period fixed things the review pointed at without claiming a count, and they are deliberately not added to that total — a number assembled from two different counting rules is the exact defect this part is about.

Six findings becoming fifty-seven was measured in Part 1, at the commit that fixed the schedule guard. It is not re-photographed here: the demo ledger is generated from a fixed seed and its totals are spoken in two published videos, so breaking it again to take a picture would move numbers that readers have already been given.

The figures about Countercheck — 237 rows and 6 findings worth £27,401.25 — were measured against the running application, not quoted from a document. They are checkable in the repository and in the product.

What is not claimed here. Several widely repeated figures about duplicate payments and invoice fraud were traced during the writing of this series and are absent from it, because the trail ended at vendor posts citing one another rather than at a study. A citations page closing the series will publish with the final part.

This is not a security review. The access question belongs to Part 4 and the running question to Part 5. Saying so is the point of the second question on the Review Card.