D5 · Retrieval-Augmented Generation

This domain is about 16% of the OAI-API mock – roughly 10 of 60 items – and mirrors the Academy Build with Retrieval-Augmented Generation course (100 min). It tests whether you can ground a model’s answers in your own documents: chunk and embed content, store it in a vector store, retrieve with file search and hybrid methods, format citations, evaluate whether answers are actually grounded, and handle freshness and permissions so retrieval returns the right, allowed, current evidence.

What you need to know

RAG puts your documents in front of the model at answer time instead of relying on training knowledge. You chunk documents into passages, embed them into vectors, and store them in a vector store. At query time you retrieve the most relevant chunks (semantic, keyword, or hybrid) and pass them as context so the model answers from evidence and cites its sources. RAG is the right choice when knowledge is large, private, or changing – you update the store instead of retraining. The hard parts are not the plumbing: they are chunking well, retrieving the right passages, proving answers are grounded, keeping content fresh, and enforcing permissions so users only see what they may.

Learning objectives

By the end of this page you should be able to:

1. Chunk documents sensibly and explain why chunk size matters.
2. Create embeddings and a vector store, and answer with file search.
3. Use hybrid retrieval (semantic + keyword) and know when it beats pure semantic.
4. Format citations and evaluate grounded answers.
5. Handle freshness so stale content is not retrieved.
6. Enforce permissions so retrieval respects who may see what.

5.1 When RAG, and the pipeline

INGEST (offline)                      QUERY (online)
────────────────                      ──────────────
documents                             user question
   │ chunk                                │ embed query
   ▼                                       ▼
chunks ── embed ──► vectors ──► VECTOR STORE ──► retrieve top-k chunks
                                                     │
                                                     ▼
                                       prompt = question + chunks
                                                     │
                                                     ▼
                                          model answers WITH citations

RAG beats fine-tuning and beats stuffing everything in the prompt when the knowledge base is large, private, or frequently updated: you edit the store, not the model.

5.2 Chunking

Chunk size trades recall against precision and cost.

Overlap between adjacent chunks preserves context that would otherwise be cut at a boundary. Respect document structure – chunk on headings/paragraphs rather than fixed character counts where possible.

5.3 Embeddings and the vector store

Embeddings turn text into vectors so semantically similar passages sit near each other. A vector store indexes them for fast nearest-neighbour retrieval. With the OpenAI platform you can create a managed vector store and use file search without running your own vector database.

vs = client.vector_stores.create(name="hr-policies")
client.vector_stores.files.upload_and_poll(
    vector_store_id=vs.id, file=open("leave-policy.pdf", "rb"),
)

resp = client.responses.create(
    model="gpt-5.6-terra",
    input="How many days of paid parental leave do I get?",
    tools=[{"type": "file_search", "vector_store_ids": [vs.id]}],
)
print(resp.output_text)   # answer grounded in the uploaded files, with citations

5.4 Hybrid retrieval

Pure semantic search can miss exact terms (a product code, an error string, a legal clause number); pure keyword search misses paraphrases. Hybrid retrieval combines both and usually reranks.

If users search by exact identifiers (SKUs, error codes, clause numbers), hybrid retrieval materially beats pure semantic.

5.5 Citations and grounded answers

A grounded answer points to the passage that supports each claim. Two things to get right:

• Citation formatting: return the source (file, section, and ideally the quoted span) so a reader can verify. File search returns citations you surface to the user.
• Grounded-answer discipline: instruct the model to answer only from retrieved context and to say “not found in the provided material” when the context does not support an answer – rather than filling the gap from training knowledge.

Answer:  "Paid parental leave is 16 weeks."   [leave-policy.pdf, §4.2]
Check:   open §4.2 → confirms 16 weeks → grounded.
If the chunk said nothing → the model should say "not in the policy",
not invent a number.

5.6 Evaluating grounded answers

RAG has two failure surfaces, and you evaluate each:

This is where D5 meets D3: build a dataset of questions with reference answers and supporting passages, and grade both retrieval and grounding. “The answer sounds right” is not a grounding measurement.

5.7 Freshness and permissions

Decision framework

Use the GROUND framework to design and defend a RAG system.

The step most teams skip is D – they ship RAG that retrieves plausibly but never measure whether answers are actually supported.

Common mistakes

Scenario challenge

Scenario. A healthcare provider builds an assistant that answers clinician questions from internal protocols. Requirements: answers must cite the exact protocol section; clinicians must only retrieve protocols for their own department; protocols are revised monthly and the old version must never be quoted; and some questions reference exact protocol codes like PROT-CARD-014. An engineer proposes: fine-tune a model on all protocols, tell it in the system prompt to “only answer for the user’s department and cite sections”, and re-fine-tune monthly.

Expert reasoning trace.

1. Fine-tuning is the wrong core. Protocols change monthly and answers must cite exact sections. Fine-tuning bakes a snapshot into weights, cannot cite a source span, and forces a costly retrain each month. RAG is correct: update the store, retrieve, cite.
2. Permissions must be enforced at retrieval. A prompt instruction to “only answer for the user’s department” is not a control – the model still sees other departments’ chunks if they are in the candidate set. The safe design filters the candidate set to the clinician’s department before retrieval, e.g. partitioned stores or metadata filters, so out-of-scope protocols never reach context.
3. Freshness needs versioning, not just re-indexing. Re-index monthly, but also tag each chunk with an effective date/version and filter retrieval to the current version so a superseded protocol is never quoted.
4. Exact codes demand hybrid retrieval. PROT-CARD-014 is a rare exact token that pure semantic search may miss; hybrid (keyword + semantic) with reranking retrieves it reliably.
5. Ground and cite. Instruct the model to answer only from retrieved chunks and to return the protocol and section span; if the chunks do not answer, say so rather than inventing.
6. Measure it. Build an eval of clinician questions with the correct protocol section, and grade retrieval recall and answer faithfulness – especially that no cross-department leakage occurs.

Exam-correct decision: RAG with a vector store, retrieval-time department filtering, version/freshness filtering, hybrid retrieval for exact codes, grounded-and-cited answers, and a grounding eval. Not fine-tuning, not prompt-based permission “controls”.

Assessment traps

Practice questions

Each item states how many responses to select. Commit before revealing.

Key takeaways

• RAG grounds answers in your documents; prefer it over fine-tuning when knowledge is large, private, changing or must be cited.
• Chunk into coherent, right-sized, overlapped passages that respect document structure.
• Store embeddings in a vector store and answer with file search; use hybrid retrieval when exact IDs matter.
• Make the model answer only from retrieved context and cite the source span; have it say “not found” rather than invent.
• Evaluate both retrieval recall and answer faithfulness on a labelled dataset.
• Enforce permissions by filtering the candidate set before retrieval, never with a prompt instruction.
• Keep content fresh with version/date filtering and re-indexing so superseded documents are never quoted.