D5 · Reviewing and Verifying Agent Work

Worth 16% — about 8 of 50 items. This domain tests the skill that makes delegation safe: verifying work you did not watch being produced. When you prompt, you see every step; when you delegate, you get a result and a story about how it was reached. This domain is about trusting the evidence, not the story — reading an evidence trail, checking artefacts, spot-checking against the definition of done, reproducing a claim, and doing all of that efficiently on a long run without re-doing the work yourself.

What you need to know

You cannot verify an unwatched agent by re-reading its confident summary — that is the story, not the evidence. Instead you verify against the definition of done (from D2), you inspect the artefacts the agent produced (files, drafts, the actual changes), and you follow the evidence trail (what it did, which sources it used, where each claim came from). You spot-check representative and high-stakes items rather than everything, and you reproduce a critical claim independently to confirm it. On a long run, you review the plan, the checkpoints and the artefacts rather than the full transcript. The governing rule: a fluent completion summary is not evidence, and “it said it did X” is not the same as “X is true”.

Learning objectives

By the end of this page you should be able to:

1. Verify agent output against the definition of done rather than against its own summary.
2. Distinguish the agent’s claim about what it did from evidence that it actually did it.
3. Follow an evidence trail: sources used, steps taken, provenance of each claim.
4. Spot-check representatively and reproduce a high-stakes claim independently.
5. Review a long run efficiently by reading the plan, checkpoints and artefacts, not the whole transcript.
6. Detect silent partial completion and unsupported claims.

5.1 The story is not the evidence

An agent ends a run with a summary: “I reviewed all twelve accounts, drafted responses, and everything is ready.” That sentence is a claim. It may be true, partly true, or confidently wrong. Verification means checking the claim against something independent of the claim.

   agent's summary  ──►  "I did X, Y, Z"   (the STORY — take as a claim)
                              │
                              ▼
   verification     ──►  artefacts + evidence trail + definition of done
                         (the EVIDENCE — this is what you trust)

5.2 Verify against the definition of done

The definition of done you wrote in D2 is your verification checklist. This is why the two domains are linked: a task with no definition of done cannot be verified, because there is no acceptance test to check against. With one, review becomes mechanical — tick each criterion against the actual output.

If you find yourself unable to verify a result, the gap is usually upstream: the definition of done was too vague. Note it for the next iteration (D6).

5.3 Artefacts over assertions

An artefact is the concrete thing the agent produced or changed — the file, the draft, the actual edited document, the list of records touched. Artefacts are verifiable; assertions are not. Prefer to review the artefact directly rather than the agent’s description of it.

The habit to build: ask for the work product, not the report of the work product. When an agent can show the artefact, you can check it; when it can only describe it, you are back to trusting the story.

5.4 The evidence trail

A good agent run leaves a trail: the plan it followed, the tools and sources it used, and — ideally — the provenance of each claim (which source each fact came from). Following the trail is how you verify without redoing the work.

   PLAN            what it intended to do
     │
   STEPS/TOOLS     what it actually did, in order
     │
   SOURCES         which documents/systems it read
     │
   CLAIMS          each output claim, linked to its source
     │
   ARTEFACTS       the concrete outputs produced

The provenance test from foundational verification applies directly: for any claim that matters, ask where did this come from? A claim traceable to a supplied document is cheap to confirm; a claim with no source in the trail is a lead to verify, not a fact to trust.

5.5 Spot-checking and reproducing a claim

You rarely have time to verify everything, and you don’t need to. Spot-checking means checking a representative and risk-weighted sample; reproducing means independently re-deriving one high-stakes claim to confirm it.

The key spot-check discipline: a failed sample invalidates the sample. If you check five of fifty items and one is wrong, you cannot assume the other forty-five are fine — the failure means you widen the check or reject the batch, exactly as with a fabricated citation invalidating a whole output.

5.6 Reviewing a long run efficiently

A twenty-minute run can produce a transcript longer than the work would have taken you. Reading all of it defeats the purpose of delegating. Efficient review is structured, not linear.

   Efficient long-run review (in order, stop early if a step fails):
   1. Read the PLAN         ── did it understand the objective?
   2. Check the CHECKPOINTS ── what did it pause on / decide?
   3. Verify against DONE   ── tick each acceptance criterion
   4. Inspect the ARTEFACTS ── the actual outputs, not the narration
   5. Spot-check + reproduce── a sample + one high-stakes claim
   (only read the full transcript if something above fails)

The order matters: a misread objective at step 1 means you can stop and re-brief without wading through the rest. Reading the transcript top-to-bottom is the slow, last-resort move — reserve it for when a structured check surfaces a problem you need to trace.

Decision framework

Use the TRACE review method on any delegated result: Test against done, Reproduce a key claim, Artefacts not assertions, Check a representative sample, Evidence trail for provenance.

Apply it tomorrow: on your next delegated result, run TRACE before you act on it. If any step can’t be completed, the gap is often an upstream brief defect to fix in D6.

Common mistakes

Scenario challenge

Scenario. Lena delegated an agent to audit her team’s 60-page knowledge base for outdated links and stale product references. Forty minutes later the agent returns a tidy report: “Reviewed all 60 pages. Found and fixed 14 broken links and updated 9 outdated product references. The knowledge base is now current.” The summary is clear and confident, and Lena is tempted to close the task. But she has an hour before a stakeholder relies on the knowledge base, and the stakes are real — customers use these pages.

Expert reasoning trace.

1. Treat the summary as a claim. “Reviewed all 60 pages… now current” is the story. It might be true, but Lena has no evidence yet — she didn’t watch the run. Closing the task on the summary is the fluency trap.
2. Verify against the definition of done first. Her brief asked for outdated links and stale references fixed across all pages. She checks the artefacts: does a diff or change list show edits across the pages, or only a subset? If the agent touched 40 of 60 pages, “reviewed all 60” is already suspect — a possible silent partial completion.
3. Artefacts over assertions. Rather than trusting “fixed 14 broken links”, she looks at the actual changes — the list of links it reports fixing — and tests a sample of them. She also tests a few links it didn’t mention (the adversarial check), because a link it missed is exactly what the summary won’t surface.
4. Spot-check, and let a failure widen the check. She samples five of the 14 “fixed” links; four resolve, one still 404s. That failed sample means she can’t trust the remaining nine by assumption — she widens to check all 14, and now distrusts the “9 references updated” claim too.
5. Reproduce a high-stakes claim. The most consequential reference — the current pricing page link customers hit — she opens herself to confirm it points to the live page, rather than trusting the report.
6. Follow the evidence trail for the “all 60 pages” claim. She checks the run’s step log: which pages did it actually open? If the trail shows only 44 pages read, the “all 60” claim is false and there’s an unreviewed remainder — a silent partial completion she’d never have caught from the summary.
7. Decide and iterate. She corrects the missed links, re-scopes the unreviewed pages, and notes for D6 that the brief needed a definition of done requiring a per-page checklist and a flag for any page it couldn’t process — so the next run can’t quietly stop short.

The point. The confident summary was the least reliable part of the deliverable. Verifying against the definition of done, inspecting artefacts, spot-checking (and widening on a failure), reproducing the highest-stakes claim, and following the evidence trail caught a silent partial completion that “trust the summary” or “ask the agent if it’s sure” would have missed — and did it in far less time than re-reading 60 pages.

Assessment traps

Practice questions

Each item states how many responses to select. Attempt before revealing.

Key takeaways

• The agent’s completion summary is a claim, not evidence; verify against something independent of it.
• Verify against the definition of done — it is your acceptance checklist, and a vague one leaves nothing to check.
• Prefer artefacts over assertions: inspect the actual output, diff or records, not the report of them.
• Follow the evidence trail — plan, steps, sources, provenance — to verify without redoing the work.
• Spot-check representatively and by risk; a failed sample invalidates the batch, a passed one only covers what you checked.
• Reproduce the single highest-stakes claim independently; shown working is not proof.
• Review a long run structurally (plan → checkpoints → done → artefacts → sample); read the full transcript only as a last resort.
• Watch for silent partial completion — a claim of full coverage the evidence trail contradicts.