D3 · Integration (incl. RAG)

This is the largest domain on the exam – 19%, roughly 12 of 63 items. It is dominated by RAG architecture but also covers tool/agent capability governance, identity and authorization, integration-mechanism selection, observability at scale, and enterprise integration patterns. The recurring judgement being tested: when an answer is confidently wrong, suspect retrieval and indexing before the model; and when an agent has many tools, apply least privilege by removing capabilities, not by logging or confirming them.

Learning objectives

By the end of this page you should be able to:

1. Design a RAG pipeline stage by stage: ingestion → chunking → embedding → indexing → retrieval → rerank → grounding.
2. Match a chunking strategy to the data’s shape.
3. Choose dense vs sparse vs hybrid retrieval and an index/vector store with metadata filters.
4. Improve retrieval with reranking, query rewriting, HyDE, multi-query, MMR.
5. Evaluate retrieval (recall@k, MRR, faithfulness, answer relevance) and debug confident-but-wrong answers.
6. Decide between RAG, long context (1M) and fine-tuning; design agentic RAG.
7. Run capability-bloat / least-privilege analysis and close authn/authz gaps.
8. Choose integration mechanisms (MCP vs API/CLI vs agent-to-agent), design observability at scale, and apply enterprise integration patterns (queues, webhooks, idempotency, batch, freshness).

3.1 The RAG pipeline, stage by stage

Retrieval-Augmented Generation grounds the model in your data by retrieving relevant passages and placing them in context at query time. Learn the pipeline as a fixed skeleton; every design decision slots into one stage.

  INGESTION            INDEX-BUILD (offline)                    QUERY-TIME (online)
 ┌──────────┐   ┌───────────────────────────────┐   ┌───────────────────────────────────┐
 │ sources  │   │ clean → chunk → embed →        │   │ query → (rewrite/HyDE/multi-query) │
 │ (docs,   │──▶│ write vectors + metadata to    │   │  → retrieve top-k (dense+sparse)   │
 │  DB, web)│   │ vector store / search index    │   │  → rerank → assemble context       │
 └──────────┘   └───────────────────────────────┘   │  → Claude generates grounded answer │
      │  freshness / refresh pipeline ▲              │  → cite sources → validate          │
      └──────────────────────────────┘              └───────────────────────────────────┘

3.2 Chunking strategies matched to data shape

Chunking is the highest-leverage RAG decision. The right strategy depends on the structure of the source.

Document shape?
  ├─ Structured (headings/tables/code) → structural / document-aware chunking
  ├─ Long, cross-referential prose      → parent–child or late chunking
  ├─ Topically shifting prose           → semantic chunking
  └─ Uniform prose, need a baseline     → recursive (fall back to fixed)

3.3 Embeddings and indexing

Dense vs sparse vs hybrid

Most production RAG uses hybrid retrieval (e.g. reciprocal-rank fusion of BM25 and dense) because real queries mix concepts and exact identifiers.

Vector store and metadata filters

Metadata filters are also a security control: filtering by tenant_id and per-user ACL at retrieval time is how you prevent cross-tenant/permission leakage (see 3.9).

3.4 Retrieval and re-ranking techniques

A common high-precision recipe: retrieve wide (k=50, hybrid) → rerank → keep top 5–8 → ground. Reranking is often the single biggest precision win.

query ─▶ [rewrite / multi-query] ─▶ hybrid retrieve (k=50)
                                        │
                                        ▼
                                    reranker  ─▶ top 6 ─▶ context ─▶ Claude

3.5 Grounding and citations

Grounding means the answer is constrained to the retrieved passages, and every claim is traceable.

• Instruct the model to answer only from the provided context and to say when the context is insufficient (never invent).
• Use the Citations feature / structured references so each claim links to its source chunk and location.
• Return insufficient context rather than a parametric-memory guess when retrieval fails — a grounded system prefers “I don’t have that” to a confident fabrication.

{
  "system": "Answer ONLY from <context>. Cite the source id for each claim. If the context does not contain the answer, say so.",
  "messages": [
    {"role": "user", "content": "<context>{retrieved_chunks_with_ids}</context>\n\nQuestion: What is the termination notice period?"}
  ]
}

3.6 Evaluating retrieval

You cannot fix what you do not measure. Retrieval and generation are evaluated separately so you know which stage failed.

3.7 Debugging confident-but-wrong answers after a document refresh

This is a signature exam scenario. A document is updated; the assistant keeps giving the old answer, confidently. The instinct to blame the prompt or the model is wrong.

1. Suspect the index first. Was the refreshed document re-chunked, re-embedded and re-indexed? A refresh that updates the source store but not the vector index leaves stale vectors that retrieval faithfully returns.

2. Inspect what was retrieved. Log the retrieved chunk IDs and text for the failing query. If they are the old content, it’s an indexing/freshness bug, not a model bug.

3. Check chunk/version metadata. Stale updated_at or a missing re-index job confirms it.

4. Only then examine grounding/prompt. If retrieval returned the new content but the answer used the old, the grounding instruction or reranking is at fault.

Confident-but-wrong after a refresh?
  1. What did retrieval return?  ──stale── ▶ re-chunk/re-embed/re-index (fix freshness pipeline)
  2. Returned fresh content?     ──yes──── ▶ check grounding instruction / reranking
  3. Neither?                    ────────── ▶ check embedding-model mismatch / query rewrite

3.8 RAG vs long context vs fine-tuning

Does the knowledge change frequently or need citations? ── yes ─▶ RAG
Is the corpus small, stable, and fits comfortably in context? ── yes ─▶ long context
Is it about behaviour/format/style rather than changing facts? ── yes ─▶ fine-tuning

These are not mutually exclusive: fine-tune for style, RAG for facts, long context for a bounded session.

3.9 Agentic RAG, tool capability-bloat and least privilege

Agentic RAG lets the model decide when and what to retrieve, issue follow-up queries, and combine sources — more powerful than one-shot retrieval, but with more cost and failure surface. Use it when queries are multi-hop or exploratory; use plain RAG when a single retrieval suffices.

Capability-bloat and least privilege

An agent with too many tools (18 vs a recommended 4–5) is slower, more error-prone, and more dangerous. The correct remediation for an unneeded destructive tool is to remove it, not to log its use or add a confirmation prompt.

Authn / authz gap analysis

Tools act on real systems, so identity and permissions must propagate to the tool call — the agent must act as the user, with the user’s permissions, not as an omnipotent service account.

3.10 Choosing the integration mechanism

Will many clients/agents reuse this capability?  ── yes ─▶ MCP server (OAuth if remote)
One app, tight control/latency?                  ── yes ─▶ direct API/CLI tool
Need a specialised agent with its own context?   ── yes ─▶ agent-to-agent (subagent)

Progressive discovery vs monolithic context

Do not load every tool and document into context up front. Use progressive discovery: tool search + defer_loading: true for large tool catalogues, and Skills loaded on demand. A monolithic context is expensive, cache-hostile, and degrades tool-selection accuracy.

3.11 Observability at scale

At production scale you cannot debug what you cannot trace. Instrument every request end to end.

[req id: 9f3a] ── app ──▶ retrieve(k=50) ──▶ rerank(6) ──▶ opus-5 ──▶ tool:lookup ──▶ validate ──▶ resp
                 correlation id 9f3a threads through every span; token+cost tagged per span

Per-segment cost/latency telemetry is what lets you route, cache and optimise (D4) with evidence rather than guesses.

3.12 Enterprise integration patterns and data freshness

3.13 A concrete RAG pipeline configuration

Item writers reward candidates who can read a config and predict its behaviour. A defensible enterprise baseline:

ingestion:
  sources: [confluence, s3_pdfs, postgres_kb]
  dedup: content_hash
  pii_scrub: true
chunking:
  strategy: structural           # split on headings/sections
  fallback: recursive
  target_tokens: 400
  overlap_tokens: 60
  parent_child: true             # match child (~400), return parent (~1500)
embedding:
  model: text-embedding-3-large  # keep query + index model identical
  dimensions: 1024
index:
  store: managed_vector_db
  metadata: [tenant_id, doc_type, acl, updated_at, source_id]
  filters_before_search: [tenant_id, acl]   # security + precision
retrieval:
  mode: hybrid                   # BM25 + dense, RRF fusion
  k: 50
  rerank:
    model: cross_encoder_reranker
    keep_top: 6
  query_transform: [rewrite, multi_query]   # for conversational/sparse queries
generation:
  model: claude-sonnet-5
  grounding: answer_only_from_context
  citations: true
  on_insufficient_context: "say so; do not use parametric memory"
refresh:
  trigger: [webhook_on_update, nightly_schedule]
  action: re_chunk_re_embed_re_index

3.14 Retrieval evaluation with worked numbers

You must be able to compute the metrics, not just name them.

Setup. 5 queries; for each we know the single relevant chunk and where it ranked in the retrieved list:

recall@3 = (#queries whose relevant chunk is in top-3) / total = 3/5 = 0.60
MRR      = mean reciprocal rank = (1.00 + 0.25 + 0.50 + 0 + 1.00) / 5 = 0.55

Now suppose adding a reranker moves Q2’s relevant chunk to rank 2 and retrieves Q4’s at rank 3:

recall@3 → 5/5 = 1.00   (both now in top-3)
MRR      → (1.00 + 0.50 + 0.50 + 0.333 + 1.00)/5 = 0.667

The reranker lifted recall@3 from 0.60 → 1.00 and MRR from 0.55 → 0.67 — the single biggest precision/ranking win, without touching chunking or embeddings.

3.15 Multi-tenant isolation and ACL-aware retrieval

In a shared index, retrieval is a security boundary. A query must never surface another tenant’s or an unauthorised user’s chunk.

user (tenant_42, role: support) asks a question
   │  attach identity: {tenant_id: 42, acl_groups: [support]}
   ▼
filter BEFORE/ALONGSIDE vector search:
   WHERE tenant_id = 42 AND acl IN ('public','support')
   ▼
hybrid retrieve → rerank → ground   (only authorised chunks ever enter context)

3.16 Scenario walkthrough: a leaking, stale, over-privileged support RAG agent

Scenario. A B2B SaaS support agent serves 300 tenants from one shared vector index. Three complaints arrive: (1) a tenant occasionally sees another tenant’s runbook in an answer; (2) after customers update their docs, the agent still quotes yesterday’s version, confidently; (3) the agent has 22 tools including delete_ticket and refund it should never call. Retrieval is dense-only, k=8, no reranker.

Expert reasoning trace.

1. Isolation first (highest severity). Cross-tenant exposure is a security incident. Add a mandatory tenant_id (and ACL) pre-filter on every query, key caches by tenant, and propagate end-user identity. Do not rely on prompt wording to keep tenants apart.

2. Freshness second. Confident-wrong-after-update is stale retrieval: the refresh updated the source store but not the vector index. Inspect retrieved chunk IDs (they’ll be old), then fix the re-chunk/re-embed/re-index pipeline with webhook + nightly triggers.

3. Least privilege third. Remove delete_ticket, refund, and other unneeded tools from the allowlist — do not merely log or add “are you sure?”. Cut to ~4–5 tools; use tool search + defer_loading if the legitimate catalogue is large.

4. Then quality. Dense-only + k=8 + no rerank underperforms on exact identifiers and precision. Move to hybrid retrieval, k=50, rerank, keep top 6, and measure recall@k and faithfulness per tenant segment.

5. Close the loop. Instrument correlation IDs and per-segment retrieval telemetry so the next incident is reconstructable.

Why the tempting alternatives are wrong: “add a system-prompt rule not to reveal other tenants” is prompt-as-enforcement and leaves the leak; “switch to a bigger model” fixes neither isolation nor freshness; “log the dangerous tools” leaves excessive agency; “raise k to 500” adds noise instead of adding a reranker.

3.17 Common misconceptions

Exam traps in this domain

Practice questions

Key takeaways

• RAG is a fixed pipeline: ingest → chunk → embed → index → retrieve → rerank → ground → cite; every decision slots into a stage.
• Match chunking to data shape: document-aware for structured text, parent–child/late for cross-referential prose, semantic for shifting topics.
• Use hybrid retrieval (dense + sparse) for real corpora; rerank for precision; add rewriting/HyDE/multi-query for recall.
• Ground answers to retrieved context, cite sources, and prefer “insufficient context” over fabrication.
• Evaluate retrieval and generation separately (recall@k, MRR vs faithfulness); confident-wrong-after-refresh means inspect retrieval/indexing first.
• Choose RAG for changing/cited facts, long context for small stable corpora, fine-tuning for behaviour/style.
• Apply least privilege by removing unneeded (especially destructive) tools; keep agents to ~4–5 tools with tool search for larger catalogues.
• Propagate user identity to tools, enforce per-user ACLs, and require OAuth 2.1 on remote MCP servers.
• Instrument traces, correlation IDs and token/cost telemetry; use queues, idempotency keys, webhooks, Batch API and a freshness pipeline for enterprise integration.
• Compute retrieval metrics: recall@k = relevant-in-top-k ÷ queries; MRR = mean(1÷rank); a reranker typically lifts both the most.
• In a shared index, retrieval is a security boundary: filter by tenant_id/ACL before the vector search, propagate end-user identity, and key caches by tenant.
• Keep the query and index embedding models identical; the canonical recipe is retrieve wide (k≈50 hybrid) → rerank → keep ~6 → ground with citations.