D6 · Reliability and Iteration

Worth 16% — about 8 of 50 items. This closing domain tests whether you can make a delegation reliable, not merely successful once. A brief that worked today by luck will fail tomorrow on a slightly different input. Reliability comes from naming failures precisely (a taxonomy), making them visible (instrumentation), and iterating the brief on the real cause rather than guessing. It pulls the whole track together: every failure here maps back to a gap in the objective (D2), the context and access (D3), the boundaries (D4) or the verification (D5).

What you need to know

Agent failures fall into a small taxonomy: misunderstood objective, missing context, wrong tool or access, silent partial completion, and drift over long runs. Each has a different fix and a different upstream cause, so naming the failure is the first step to fixing it. Instrumentation means running delegations so failures are visible — capturing the plan, the evidence trail, checkpoints and outcomes — because you cannot iterate on a failure you cannot see. Iterating a delegation brief means changing one thing at a time based on the diagnosed cause: sharpen the objective, add the missing context, adjust access, add a checkpoint, or strengthen verification. The discipline is: diagnose the cause, change the brief (not the model, usually), and re-test — turning a lucky success into a repeatable one.

Learning objectives

By the end of this page you should be able to:

1. Classify an agent failure using the five-mode taxonomy.
2. Trace each failure mode to its upstream cause and the domain that fixes it.
3. Instrument a delegation so failures are visible rather than silent.
4. Iterate a brief by changing one diagnosed thing at a time and re-testing.
5. Distinguish a reliable delegation from one that succeeded by luck.
6. Decide when the fix is the brief, the context, the boundaries, or (rarely) the model.

6.1 The failure taxonomy

Vague debugging (“the agent didn’t work”) leads to vague fixes (“try a better model”). Precise classification leads to precise fixes. Learn these five modes cold — most exam items are asking you to name one and pick its fix.

   Agent failed. Which mode?
   ├─ wrong thing?          ─► MISUNDERSTOOD OBJECTIVE   → sharpen goal (D2)
   ├─ generic / off-brand?  ─► MISSING CONTEXT           → supply context (D3)
   ├─ stalled / wrong data? ─► WRONG TOOL/ACCESS         → fix provisioning (D3)
   ├─ "done" but wasn't?    ─► SILENT PARTIAL COMPLETION → add done-check (D2/D5)
   └─ degraded over time?   ─► DRIFT                      → checkpoints/scope (D4)

6.2 Silent partial completion, examined

This mode deserves its own treatment because it is the most dangerous: the agent reports success while having done only part of the work, so nothing signals a problem. It’s the failure D5’s verification exists to catch.

The reliability fix is a brief that makes silence impossible: “process all N items; for any you cannot complete, list it and why”. Combined with D5’s coverage check, silent partial completion becomes a caught partial completion.

6.3 Drift over long runs

On a long, unwatched run an agent can gradually wander — losing the thread of the objective, over-elaborating, or getting pulled off-task by something it found. Drift is a function of duration and lack of anchoring.

   short run                     long run without anchoring
   ──────────                    ───────────────────────────
   objective stays sharp    │    objective slowly blurs
   output stays on-task     │    output wanders / over-elaborates
   easy to keep on track    │    each step drifts a little further
                            │    → add checkpoints; scope the run;
                            │      break into shorter delegations

Fixes for drift are structural, not motivational: checkpoints that re-anchor to the objective, scoping the run into shorter bounded pieces, and the one→many pattern so a drifting pattern is caught after one item. Telling the agent “stay focused” is a respected boundary and won’t hold on a long run.

6.4 Instrumentation: you can’t fix what you can’t see

Instrumentation means running delegations so their behaviour is observable — the plan, the steps and tools used, the checkpoints, the evidence trail, and the outcome against the definition of done. Without it, a failure is just “it didn’t work”, and you’re guessing at the fix.

The mindset: treat a delegation like a process you can inspect, not a black box that emits an answer. When it fails, the instrumentation tells you which taxonomy mode you’re in — which tells you the fix.

6.5 Iterating the brief: one change at a time

Iteration is disciplined, not frantic. The rule is diagnose, change one thing, re-test. Changing several things at once means you won’t know which fixed it (or which made it worse), and reliability never accumulates.

   1. OBSERVE   the failure (via instrumentation)
   2. CLASSIFY  it (taxonomy)
   3. TRACE     to the upstream cause (which domain)
   4. CHANGE    exactly one thing in the brief/context/access/boundaries
   5. RE-TEST   on the same input, then on a varied one
   6. repeat until reliable across varied inputs

The model is the last lever, not the first — most failures are brief, context, access or boundary defects. Reaching for a bigger model before diagnosing is the single most common reliability anti-pattern.

6.6 Reliable versus lucky

A delegation that worked once may have worked by luck: the input happened to be easy, the ambiguity happened to resolve the way you wanted. Reliability means it works across varied inputs, including awkward ones.

The test for reliability: re-run the delegation on a different, harder input. If it still meets the definition of done, the brief is reliable; if it fails, you’ve found the next thing to iterate. A single success is a hypothesis, not a result.

Decision framework

Use the LEARN loop to turn any failed or lucky delegation into a reliable one: Locate the symptom, Examine the instrumentation, Attribute to a taxonomy mode, Revise one thing, re-test on a New input.

Apply it tomorrow: the next time a delegation disappoints, resist changing the model. Run LEARN, change one diagnosed thing, and prove reliability on a second input.

Common mistakes

Scenario challenge

Scenario. Sofia set up an agent on ChatGPT Work to triage her team’s incoming vendor invoices: extract the amount, match it to a purchase order, and produce a summary table for approval. It worked beautifully on her first test — one clean invoice, matched perfectly. She rolled it out for the week’s fifty invoices. The results are a mess: some amounts are wrong, several invoices are missing from the table entirely though the agent reported “all invoices processed”, and the later entries drift into inconsistent formatting and occasional editorialising about the vendors. Her instinct is to conclude agents “aren’t reliable enough for finance” and to try the most capable model.

Expert reasoning trace.

1. Reject the model-first instinct. A bigger model will produce the same shaped failures faster, because none of these symptoms is a raw capability problem. Diagnose before spending.
2. Classify each symptom with the taxonomy. Wrong amounts → likely missing context or wrong source (which document/field is authoritative?) — a D3 issue. Missing invoices with a “processed all” claim → silent partial completion — a D2/D5 issue. Later entries drifting into inconsistent formatting and editorialising → drift over the long run — a D4/D6 issue. Three distinct modes, three distinct fixes.
3. See why the first test misled her. One clean invoice was the lucky case: no ambiguity, short run, no scale. Reliability is proven on varied inputs, and she skipped that step.
4. Instrument to confirm. She examines the run: the step log shows the agent read 43 of 50 invoices (confirming silent partial completion), pulled amounts from an inconsistent field (confirming the source problem), and had no checkpoints across the long run (explaining the drift).
5. Iterate one change at a time. She doesn’t rewrite everything at once. First: fix the source of truth — specify exactly which field holds the amount and which system holds the PO (D3), and re-test. Then: add a definition of done requiring all 50 processed with any it couldn’t match explicitly listed (D2/D5), and re-test. Then: add a checkpoint after the first invoice (approve the pattern) and scope the run so drift can’t accumulate (D4), and re-test.
6. Prove reliability, not luck. After each change she re-tests on a varied set — a clean invoice, a messy one, one with no matching PO — until the delegation meets the definition of done across all of them. Only then is it trustworthy for the weekly run.

The point. “Agents aren’t reliable for finance” was the wrong conclusion. The delegation exhibited three separate, nameable failure modes, each with a brief-level fix; a more capable model addressed none of them. Reliability came from instrumenting the run, classifying each failure, iterating one change at a time, and proving it on varied inputs rather than trusting one lucky success.

Assessment traps

Practice questions

Each item states how many responses to select. Attempt before revealing.

Key takeaways

• Learn the five-mode failure taxonomy — misunderstood objective, missing context, wrong tool/access, silent partial completion, drift — because naming the mode points to the fix.
• Each mode traces to an upstream domain: objective (D2), context/access (D3), boundaries (D4), verification (D5).
• Silent partial completion is the most dangerous mode; defeat it with a coverage requirement in the definition of done plus a D5 coverage check.
• Drift on long runs needs structural fixes — checkpoints and shorter scope — not “stay focused”.
• Instrument delegations so failures are visible; you cannot fix what you cannot see.
• Iterate one change at a time, tracing to the diagnosed cause; the model is the last lever, not the first.
• A single success is a hypothesis — prove reliability by re-testing on varied, harder inputs.
• Use the LEARN loop (Locate, Examine, Attribute, Revise, New input) to turn a lucky or failed delegation into a reliable one.