The form looked finished. You typed a title, pressed submit, the spinner turned, and the page came back. Nothing was saved, for you or for anyone else. The agent that wrote the change had reported "done, tests pass", and both halves of that sentence were true.
The file it edited started with "use server", the directive that turns the exports of a Next.js module into server actions. The agent had added one small exported constant next to the actions. A "use server" module may only export async functions (React's reference spells out the rules), so the framework refused the module at runtime and every action in it failed. The unit tests imported the same file directly. To a test runner, "use server" is a string literal at the top of a file, nothing more. Green.
An AI coding agent is a fast, fluent and optimistic narrator, and "done" is the end of its story, not a fact about your system. The fix is not vague distrust. It is one question asked of every claim: what was actually checked, and what wasn't?
Three times "done" was wrong
The "use server" export was the first. Two more from the same family:
A template that was never filled in. A script generated a production config from a template containing tokens like {API_BASE_URL}. The substitution step never ran. A string with braces in it is still a perfectly valid string, so no parser complained, and the config shipped with the literal text {API_BASE_URL} where a URL should have been.
An edit that matched nothing. A find-and-replace ran across a handful of files. The files had changed since the agent last read them, so the pattern matched nothing. The script printed success, because it did succeed: a substitution that replaces zero occurrences is still a successful run.
You can reproduce the third one in five seconds:
printf 'timeout = 30\n' > app.conf
sed -i.bak 's/timeout = 60/timeout = 90/' app.conf
echo "exit code: $?" # exit code: 0
cat app.conf # timeout = 30, unchangedGNU sed's manual is explicit that status 0 means successful completion. "Did it complete?" and "did it change what I meant?" are different questions, and only the first one has an exit code.
In all three stories, every tool answered the question it was asked, and nobody asked the one that mattered: does the system now do the thing?
Why does an AI agent say "done" when the code is broken?
Not because it lies. Whether it is Claude Code, Codex or OpenCode, the agent's whole world is the text its tools returned: compiler output, test output, the result of a file write, the last line a script printed. "Done" summarizes the last success it observed, and each of those successes was real.
Three things make the gap systematic, not bad luck:
- Exit codes answer "did it run?", not "did it work?" A zero says the process finished without an error it knew how to detect.
- The agent's checks live where the agent lives. It runs unit tests in a terminal. It rarely boots the whole app, goes through the gateway and submits a real form, so failures at those boundaries are invisible to it.
- The task is framed as "make it pass". Give an agent a red test and it will make it green, which is also how you get a green test that no longer tests the thing.
A building inspector doesn't sign a sheet that says "building OK". A useful sign-off says: inspected the wiring in units 1 to 4 and the fire exits on floors 1 to 3; did not inspect the roof (no access) or unit 5 (locked); date; name. The second half is what makes the first half trustworthy. If a sheet doesn't say what wasn't inspected, assume nothing was.
An agent's "done" is the contractor telling you the building is finished. What you want is the inspector's sheet.
Why doesn't "please double-check" fix it?
The obvious fix is to ask for more care. Add "run the tests before you say done" to the prompt. Ask the agent to double-check. Have a second agent review the first one's report.
Each fails for the same reason: it asks the same kind of narrator to look harder at the same evidence. "Run the tests" reruns the tests that were already green. "Double-check" re-reads the diff with the same assumptions, so an agent that believes "use server" is just a directive finds the extra export perfectly fine. A reviewer reading a summary checks the story, not the system.
That is asking the contractor to promise harder. What changes the outcome is changing the evidence: a different kind of check, an artifact you read yourself, and a written list of what was not looked at. Three tools do that: labels, a ladder and receipts.
Seven labels for what you actually know
The first tool is vocabulary. The platform I work on, thirteen NestJS services in an Nx monorepo plus a frontend, keeps its project rules in one place, and one of those rules is that every implementation claim carries exactly one evidence label. There are seven:
| Label | What it means |
|---|---|
| Released | Contract, runtime OpenAPI, checks and a tracked release agree |
| Verified implementation | Current source plus a passing scoped check |
| Prototype observation | Seen in a stated source snapshot; nothing run |
| Architecture decision | An approved target rule, built or not |
| Planned | Intent without implementation proof |
| Alignment gap | Target and code disagree; follow-up needed |
| Unknown | No reliable evidence; infer nothing |
Most of the value sits in three distinctions.
Reading is not running. "The controller returns 404 for a missing order", said after reading the source, is a Prototype observation. It becomes a Verified implementation only after a check has exercised it. Agents read code so well that their observations sound like verification; the label drags the difference into the open.
Verified is not released. In the repository, one service gained three anonymous read endpoints. Typecheck passed, 31 test suites with 141 tests passed, the frontend client's checks passed, and the docs and Postman collection were updated. The record still says Verified implementation, not Released, because none of that proves gateway routing, an applied migration, generated runtime OpenAPI, browser behavior in a deployed environment or a tracked release.
Unknown is a legitimate answer. It is the label agents use least and need most. "I found no evidence either way" beats a confident guess, because a guess gets built on.
One more rule: a label is upgraded only when the required proof exists, and evidence is never rewritten to match the design you'd prefer. When target and code disagree, that is an Alignment gap recorded for follow-up, not a docs edit.
How do you verify code an AI agent wrote? Climb the ladder
Labels say what you know. The ladder says how to find out. Each rung costs more and sees more.
| Rung | Proves | Blind to |
|---|---|---|
| Types | Shapes agree across modules | Anything at runtime |
| Tests | What the test setup sees | Wiring the test bypasses |
| Runtime OpenAPI | Routes the booted app registered | Whether they behave |
| Scoped checks | Real boundaries, kept output | Paths not exercised |
| The running thing | What a user actually gets | Little, at a high cost |
The sentence at the bottom of the diagram is the whole method: climb until the rung matches the claim. "The types line up" needs rung one. "The endpoint exists" needs rung three. "Users can save a draft" needs rung five. An agent that says "users can save a draft" with evidence from rung two has made a claim three rungs above its proof.
Rung 1: types
tsc --noEmit or nx typecheck is cheap and catches a real class of mistake: a renamed field, a missing argument, an import that no longer exists. It catches none of the three stories. A string containing {API_BASE_URL} is a well-typed string.
Rung 2: tests
Tests prove what the test setup sees, and the setup is a model of the real system with pieces removed. The "use server" bug lived in exactly such a removed piece. When a test passes, ask what it bypassed: a direct import skips the framework, a mocked repository skips the database, a controller unit test skips the gateway.
One cheap guard against tests that are green because they test nothing: watch the test fail before you trust it to pass. If the agent fixed a bug, its new test should fail on the old code. Ask to see that run.
Rung 3: generated runtime OpenAPI
This rung is underrated. In the repository, each service builds its OpenAPI document from the booted application instance through a shared helper and serves it as openapi.json. That document lists the routes Nest actually registered, not the ones somebody wrote a controller for.
A controller whose module was never imported is a route claimed but never wired. Reading the source says it's there; the runtime document says it isn't. That is why the project rules require generated runtime OpenAPI, not a hand-written spec, before an endpoint can be called Released.
Rung 4: scoped checks against real boundaries
A scoped check runs the real thing across a real boundary, for a narrow scope, and keeps its output. End-to-end tests through the gateway against a real database are the obvious example.
The repository's deployment verifier shows the attitude in miniature. For every service it requests /health/live, /health/ready and /openapi.json, and it checks the content, not just the status code:
verify_openapi() {
local url="$1"
local response
if ! response="$(request_json "${url}")" \
|| ! jq -e '.openapi | type == "string"' >/dev/null 2>&1 <<<"${response}"; then
echo "OpenAPI verification failed for ${url}" >&2
return 1
fi
}The verifier has its own tests, and two target the failure modes in this article. One feeds it a truncated OpenAPI response and asserts it fails with the verification message, not a parser error. Another reads the script's source and asserts it contains no POST, login, refresh, logout or user calls, so verifying an environment can never change it.
Rung 5: look at the running thing
Open the page. Submit the form. Read the generated config. Call the endpoint with curl. This rung caught all three stories, at the highest cost, because it looks at the output instead of a proxy for it.
Two of those looks are cheap enough to automate as read-backs. One checks a rendered file for leftover placeholders:
import re
import sys
PLACEHOLDER = re.compile(r"\{[A-Z_]+\}")
def assert_rendered(path: str) -> None:
with open(path, encoding="utf-8") as rendered:
leftovers = sorted(set(PLACEHOLDER.findall(rendered.read())))
if leftovers:
sys.exit(f"{path}: unfilled placeholders {leftovers}")
assert_rendered(sys.argv[1])The other refuses an edit that changed nothing:
cp app.conf app.conf.before
sed -i.bak 's/timeout = 60/timeout = 90/' app.conf
if cmp -s app.conf app.conf.before; then
echo "edit matched nothing" >&2
exit 1
fiThe pattern is the same in both: check the artifact, not the tool's report about the artifact. A git diff --stat after an agent's bulk edit does this for free. An empty diff after "updated 12 files" is a finding, not a success.
What should a verification receipt contain?
A receipt is the inspector's sheet. Mine have five parts, and the last one is what makes it honest. Here is the read-endpoint case from above in that shape:
**Claim:** three anonymous read endpoints return only active items
**Label:** Verified implementation
**Against:** backend @ <short sha>, worktree with uncommitted changes
**Ran (2026-07-27, UTC):**
- `pnpm exec nx typecheck <service>`: pass
- service test suites: 31 suites, 141 tests, pass
**Does not prove:** gateway routing, an applied migration, runtime OpenAPI,
browser behavior in a deployed environment, a tracked releaseThe "Does not prove" line is the one people skip, and it protects the next reader. Without it, a receipt from rung two quietly gets read as rung five.
Two other entries in the repository show the extremes. One report described a successful local workflow (migrations, seed, build, HTTP calls through the gateway), but no command or output had been kept. It was downgraded to a Prototype observation until an auditable run exists. A sign-off with no sheet is not a sign-off.
The other entry synchronized config, CI commands, Terraform references, route docs and Postman, then listed every check it had not run, because verification was deferred on purpose. That receipt proves almost nothing and is completely trustworthy, because it says so. It even notes that some config values were wired in as deployment inputs while the modules still built placeholder adapters. Wired is not used.
A receipt you can rerun beats one you have to believe. This wrapper makes an agent's checks leave a trace instead of a sentence:
#!/usr/bin/env bash
# record.sh: run one check and keep exactly what it printed
set -uo pipefail
mkdir -p evidence
log="evidence/$(date -u +%Y%m%dT%H%M%SZ).log"
{
echo "when: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "revision: $(git rev-parse --short HEAD)"
echo "dirty: $(git status --porcelain | wc -l | tr -d ' ') files"
echo "command: $*"
echo "---"
} > "$log"
"$@" >> "$log" 2>&1
status=$?
echo "exit: $status" >> "$log"
echo "$log"
exit "$status"./record.sh pnpm exec nx test order-service leaves a log the agent did not write and cannot summarize away.
Why do receipts go stale?
An inspection is valid for the building as it stood that day. In the repository, one receipt covers an auth workflow verified on a given date. A day later the worktree had gained change-password, password-reset and role-management code, and the evidence file says plainly that the earlier receipt does not cover those changes. That one sentence stops anyone from reading "auth verified" as covering code that did not exist yet.
So every receipt names a revision and whether the worktree was dirty. Once the code moves, the receipt is history.
A definition of done for agent-written changes
This is what I ask of every agent and of myself. It lives in the shared rules file every tool reads, so it is never copied into five places (one rulebook for every agent).
- State the claim in one sentence, with one label. "Users can save a draft" is a claim. "Fixed the form" is not.
- Show evidence from the rung that matches the claim, as the command plus its kept output, not a paraphrase.
- Read back what changed. The diff, the generated file, the rendered config. An empty or surprising diff is a finding.
- List what was not verified. Every time. "Nothing" is almost never true.
- Prove nothing else moved. Snapshot
git statusin the root and in each submodule before and after. An unrelated change is a stop sign. - Validate the side artifacts. Parse every JSON file you touched and check the links in every doc you touched.
When I delegate, the brief names the rung that "done" requires, as a command the agent must run (briefs whose "done" is a command). "Done means pnpm exec nx e2e order-service-e2e passes and the log is attached" leaves much less room for narration than "make sure it works".
What it costs
- Time. Rungs four and five are slow. End-to-end suites for every small change aren't realistic, which is exactly why a claim should be scoped to the rung you can afford.
- Ceremony. Seven labels and a receipt per claim can become paperwork. When the receipt takes longer than the change, people stop writing them, or write dishonest ones.
- Freshness. With dirty worktrees, receipts go stale within hours. The repository's own snapshot admits it could not hash-lock a stable application state for that reason.
- Fabrication. An agent can type a plausible test summary. Only tool output, kept as a file and rerunnable by you, counts.
- Rungs can mislead too. A defined pipeline stage is not a stage that ran, and written Terraform is not built infrastructure; the Compose-versus-Kubernetes write-up runs into exactly that.
What I'd do differently
- Machine-written receipts from day one. The early receipts were prose the agent wrote. They should have been logs a wrapper wrote, with prose on top.
- Enforce instead of advising. A rule in a markdown file is advice. A Claude Code hook or CI job that refuses "complete" without an evidence log newer than the last edit is enforcement. I'd move the definition of done into tooling much earlier.
- Failing test first. For every bug fix, require the reproduction, a test that fails on the old code, before the fix.
- Verify on clean revisions. Several receipts in the repository describe dirty worktrees. They're honest about it, but a receipt tied to a commit anyone can check out is worth far more.
Key takeaways
- An agent's "done" summarizes the last success it saw. Treat it as a claim, never as a fact.
- Exit codes answer "did it run?". You still have to ask "did it change what I meant?".
- Asking for more care doesn't help. Changing the evidence does.
- Give every claim exactly one label. Reading is not running, verified is not released, and Unknown is a valid answer.
- Climb the ladder (types, tests, runtime OpenAPI, scoped checks, the running thing) until the rung matches the claim.
- A receipt names the revision, the commands, the kept output and what it does not prove.
That completes the four habits: Git stays in human hands, every agent reads one rulebook, work goes out as stand-alone briefs, and nothing is done without evidence. If you are starting from zero, start with the first: keeping Git in human hands.
Hay