D5 · Context Management and Reliability

This domain is worth roughly 9 of 60 items and underpins every scenario: a system that works in a demo but degrades as the context window fills, or falls over when the API returns 429, is not production-ready. It tests the economics of the context window, the two ways to reclaim it (context editing vs compaction), what must live in durable state, and the reliability patterns – caching, retries, fallback, monitoring – that keep an agentic system running.

Learning objectives

By the end of this page you should be able to:

1. Reason about context window economics and what consumes context.
2. Choose between context editing (clear tool results) and compaction (summarise preserving narrative).
3. Use the memory tool and external state for anything that must survive compaction, and subagent isolation to protect the main context.
4. Use prompt caching as a reliability and cost lever, and honour Fable 5.1’s append-only constraint.
5. Design retries/backoff/idempotency, graceful degradation and fallback models – and know what breaks when falling back from Fable 5.1.
6. Maintain provenance and citations, and design escalation / human-in-the-loop gates.
7. Set up monitoring (latency p95, error rate, cache-hit rate, cost per task) and design for rate limits, timeouts and partial results.

5.1 Context window economics

The window is a budget, not free space. Everything shares it:

┌───────── context window budget ─────────┐
│ system prompt                            │  stable
│ tool definitions                         │  stable
│ conversation history (turns)             │  grows every turn
│ tool results (can be huge)               │  grows fast
│ thinking tokens                          │  grows with reasoning
└──────────────────────────────────────────┘

As the window fills, latency and cost rise and quality can degrade (“context rot”). The architect’s job is to keep the working set small: cache the stable parts, evict what is no longer needed, and push durable facts to external state.

5.2 Context editing vs compaction

Two distinct mechanisms reclaim context. Choosing correctly is tested.

Context editing:  [sys][tools][turn1][BIG tool result ✗ cleared][turn2]…  ← keeps dialogue, drops bulk
Compaction:       [sys][tools][==== summary of turns 1..40 ====][turn41]… ← condenses the narrative

5.3 Memory tool and external state

Compaction and context editing are lossy. Anything that must survive – a decision, a customer fact, a checkpoint, canonical business data – belongs in durable state:

• Memory tool – model-managed persistence across sessions.
• Files – artefacts and checkpoints you control.
• Database – canonical, transactional business state (orders, tickets).

The conversation references durable state; it does not own it. This is the same rule as Domain 1’s state-vs-conversation table.

5.4 Subagent isolation protects the main context

A subagent runs in its own context window. Delegating a bulky sub-task (reading 50 files, searching a large corpus) to a subagent means the raw material never enters the coordinator’s window – only the distilled result returns. This is one of the most effective context-protection patterns, and it is why explicit context passing (Domain 1) matters: the subagent starts clean.

5.5 Long-context ordering and prompt caching

• Ordering. Put stable, reusable content first (system, tools, long reference docs); put the variable task last. This both enables caching and keeps the model’s attention on the right material.
• Prompt caching as a reliability + cost lever. Cache reads ≈ 0.1× base input price; writes ≈ 1.25× (5-min TTL) or 2× (1-hour TTL). Mark the last stable block with cache_control: {"type": "ephemeral"}. Minimum cacheable prefix ~1024 tokens (2048 on Haiku). A high cache-hit rate lowers cost and latency, improving reliability under load.

system = [
    {"type": "text", "text": SYSTEM_RULES},
    {"type": "text", "text": TOOLS_DOC},
    {"type": "text", "text": REFERENCE, "cache_control": {"type": "ephemeral"}},  # cache boundary
]

5.6 Fable 5.1: append-only history and thinking-block binding

Fable 5.1 introduces breaking changes an architect must design around:

• Append-only history. Editing, reordering or removing earlier turns invalidates later thinking blocks. Harnesses must be append-only: freeze system and tools, put mid-session changes in role: "system" messages, and trim server-side via context editing/compaction rather than rewriting the transcript.
• Thinking-block binding. Thinking blocks are readable only by the producing model or a newer one. Falling back to an older model silently drops those thinking blocks.
• Operational constraints. Requires 30-day data retention; not in the Priority Tier; thinking always on.

5.7 Retries, backoff, idempotency

Retry the transient errors with exponential backoff and jitter, honouring retry-after:

Only retry idempotent operations safely; give writes an idempotency key (Domain 1).

def call_with_backoff(fn, max_attempts=5):
    for i in range(max_attempts):
        try:
            return fn()
        except (RateLimitError, APIError, OverloadedError) as e:
            if i == max_attempts - 1:
                raise
            delay = min(2 ** i, 30) + random.random()      # backoff + jitter
            time.sleep(getattr(e, "retry_after", delay))

5.8 Graceful degradation and fallback models

When the primary model is unavailable or overloaded, degrade gracefully: fall back to another model, serve a cached/partial result, or queue for later. Fallback has costs.

5.9 Provenance and citations

For any output that will be relied on, preserve where it came from: keep citations from web search / retrieval, attach source IDs to extracted facts, and surface provenance so a human can verify. Provenance is both a quality and a governance control (and pairs with Domain 3’s per-type metrics and validation).

5.10 Escalation and human-in-the-loop gates

Irreversible or high-stakes actions need a human gate – the same escalation logic as Domain 1 (explicit request → immediate; capability gap → after attempting; never sentiment/self-report), plus mandatory human approval before irreversible effects (payments, deletions, external sends). The gate is deterministic (a permission/hook), not a prompt request.

5.11 Monitoring

Instrument the system with the metrics that predict failure:

Pair with per-agent traces and correlation IDs (Domain 1).

5.12 Rate limits, timeouts and partial results

• Rate limits are per-model-tier (RPM, ITPM, OTPM). Design concurrency and batching to stay under them; use the Batch API (50% discount, within 24h) for latency-tolerant work.
• Timeouts. Set them at every boundary (tool calls, subagents, the overall task); a hung tool must not hang the agent.
• Partial results. On timeout or partial failure, return what you have with provenance of the gap (Domain 1’s partial-failure handling), rather than nothing or a false-complete answer.

5.13 Context economics with token arithmetic

Treat the window as a budget you can compute against. A worked example for a long-running agent on Opus 5 (1M window, $5/M in, $25/M out):

Turn budget as the session grows (input tokens carried each turn):
  system + tools (stable)          =  12,000
  CLAUDE-style reference docs      =   8,000
  conversation history @ turn 30   =  60,000
  accumulated tool results         = 400,000   ← the runaway
  thinking (this turn)             =  10,000
  ----------------------------------------------
  input this turn                  = 490,000 tokens
  cost of THIS input turn          = 490,000 × $5 / 1e6 = $2.45   (and rising every turn)

The accumulated tool results dominate. Three levers, with their arithmetic:

• Context editing clears the 400k of stale tool results → input drops to ~90k → this-turn input cost falls from $2.45 to ~$0.45.
• Prompt caching the stable 20k prefix (system + tools + docs) → those tokens read at ~0.1× (20,000 × $0.5/1e6 = $0.01 vs $0.10).
• Subagent isolation: delegate the tool-heavy sub-task so the 400k never enters the main window at all — only the distilled result returns.

5.14 The Fable 5.1 harness: constraints and a compliant design

Fable 5.1 (claude-fable-5-1, 1M/128k, $10/$50) has the strictest harness rules on the exam. Design around all of them:

Compliant Fable 5.1 harness:
  ┌ freeze system + tools (never edited) ─────────────┐
  │ append user/assistant turns only                  │
  │ mid-session change? → append a role:"system" msg  │  (NOT edit an old turn)
  │ window pressure?    → server-side context editing  │
  │                       / compaction (append-safe)   │
  │ need structured out?→ output_config.format / strict │  (never forced tool_choice)
  └────────────────────────────────────────────────────┘

5.15 Reliability budgets: timeouts, concurrency and rate-limit math

Reliability is quantitative. A quick capacity check for a real-time workload:

Tier limits (example):  RPM = 4,000   ITPM = 400,000 input tokens/min
Per request:            ~10,000 input tokens
Token-bound throughput: 400,000 / 10,000 = 40 requests/min  ← ITPM binds first
Request-bound:          4,000 RPM
=> Effective ceiling = min(40, 4,000) = 40 rpm; the token limit dominates.

If you need 10,000 latency-tolerant jobs, the real-time ceiling (40/min ≈ 4+ hours and rate-limit risk) argues for the Batch API (50% off, ≤24h) instead. Set timeouts at every boundary (tool, subagent, overall task) so one hung call cannot stall the system, and on timeout return partial results with provenance of the gap.

Common misconceptions

Scenario walkthrough — a research agent that is fast in a demo and falls over in production

Situation. A multi-agent research agent works in demos but in production (1) degrades after ~30 turns as the window fills with large search results it no longer needs, while the dialogue thread must stay intact; (2) during an Anthropic capacity event it fails over from Fable 5.1 to an older model and starts behaving differently even with an identical prompt; (3) an SRE reports it is “fine on average” yet users occasionally wait 9 seconds and the monthly bill is a surprise; and (4) a nightly run of 10,000 extraction jobs keeps hitting rate limits.

Expert reasoning trace.

1. Problem 1 — filling window, keep dialogue. Bulky, stale tool results with the narrative preserved → context editing, not compaction (which would summarise the dialogue) and not a bigger model. Additionally, delegate the search-heavy work to subagents so raw results never enter the main window.

2. Problem 2 — fallback behaviour change. Fable 5.1 thinking blocks are readable only by that model or newer, so the older fallback silently drops them. The design must test the degraded path and not assume parity; window size and key expiry are red herrings.

3. Problem 3 — hidden tail and cost. Add p95/p99 latency (the 9-second tail averages hide) and cost per task (the surprise bill). Reject “total request count” and “model name” as non-diagnostic.

4. Problem 4 — rate limits on bulk. 10,000 latency-tolerant jobs belong on the Batch API (50% off, ≤24h), which eases rate-limit pressure. Reject real-time max concurrency (full price, limit risk) and one giant request (will not fit).

Exam-correct decision: context editing + subagent isolation for the window; a tested degraded path for the Fable 5.1 fallback; p95/p99 and cost-per-task monitoring; and the Batch API for bulk. Every rejected option is a named trap (bigger-model, parity-assumption, average-only monitoring, real-time bulk).

Exam traps in this domain

Practice questions

Key takeaways

• The context window is a shared budget consumed by system, tools, history, tool results and thinking; keep the working set small.
• Context editing clears stale tool results; compaction summarises the narrative — neither preserves everything, so durable facts go to the memory tool or a database.
• Subagent isolation keeps bulky raw material out of the main window; put stable content first and cache it for ~10× cheaper reads and lower latency.
• Do the arithmetic: the accumulated-tool-results term usually dominates a filling window, so context editing (or subagent isolation) — not a bigger model — is the cost-effective fix.
• Fable 5.1 is append-only (never edit old turns), thinking is always on, forced tool_choice is a 400, it has 30-day retention/no ZDR, and it is not Priority Tier — design around all of these.
• Retry only transient errors (429/5xx/529) with backoff, jitter and retry-after; only retry idempotent writes.
• Fallback is not free — from Fable 5.1 thinking blocks are dropped; Haiku 4.5 is 200k, not 1M; validate the degraded path.
• Size real-time throughput against ITPM/RPM (the token limit often binds first); use the Batch API for latency-tolerant bulk; set boundary timeouts and return partial results with provenance.
• Monitor p95/p99 latency, error rate by category, cache-hit rate and cost per task; gate irreversible actions on humans.