A model answers from what it can find, not only from what it remembers

In 2020, RAG was one retrieval call. In 2026, agents search, check and cite in a loop.

The user asks

“What changed in our refund policy last week?”

Closed-book model

guesses

Its weights stop at the training cutoff. It cannot see last week's change. It gives no source.

RAG model

finds the pagecites it

It retrieves the current policy page. It answers from that page and links to it.

0 Wikipedia passages in the first RAG index (2020)

Paper arXiv 2005.11401

0 top-20 retrieval failures with Contextual Retrieval + reranking

Anthropic Sep 2024

0 BrowseComp: GPT-4o + browsing vs Deep Research

Paper arXiv 2504.12516

“good context engineering means finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome.”

Anthropic Effective context engineering for AI agents

Start with the question on the slide. A closed-book model must guess, because last week's policy change is not in its weights. A RAG model finds the policy page, answers from it and cites it. Analogy: RAG is an open-book exam; the student still needs to know the subject, but checks the facts in the book and shows the page. The animation shows the 2020 pattern first (one retrieval call), then the 2026 pattern (an agent that thinks, searches, reads and searches again before it answers). Step 1 shows three numbers: the 21M-passage index of Lewis et al. 2020, Anthropic's drop in top-20 failures from 5.7% to 1.9%, and BrowseComp, where GPT-4o with browsing gets 1.9% and Deep Research gets 51.5%. Likely question: "Is RAG dead now that models have 1M-token windows?" Say no, and that chapter 4 gives the measurements (lost in the middle, context rot) and the decision rule.

The course moves from one retrieval call to agents that search, check and cite

09 eval · security · every stage

10 ops · cost · managed RAG

offline · index once

Parse

Chunk

Embed

Index

02 02 03 03

online · every query

Query

Retrieve

Rerank

Context

LLM

04 04 04 04

Agentenough evidence?

05 · 06

Graph

Page images

Memory

07 08 08

First principles

01 Why RAG

Mechanisms

02 Ingestion and chunking
03 Embeddings and vector indexes

04 Retrieval quality and context assembly

Building agents

05 Agentic RAG patterns
06 Research agents, RL-trained search and MCP

State of the art

07 Graph and structured RAG
08 Multimodal, code and memory

Production and open challenges

09 Evaluation and security
10 Production, managed RAG and best practices (Oct 2026)

agent, retrievertool, APIindex, data, memorymodel calluserrisk

Each chapter ends with interview questions.

This map is the whole course on one diagram. Step 1 is the offline lane: parse, chunk, embed and index, taught in chapters 2 and 3. Step 2 is the online lane that runs for every query: retrieve about 150 candidates with hybrid search, rerank to 20, assemble the context, call the model; that is chapter 4. Step 3 bends the line into a loop: an agent checks the evidence and searches again when it is not enough; chapters 5 and 6 cover routers, Self-RAG, CRAG, deep research and RL-trained search. Step 4 adds other stores: graphs in chapter 7, page images and memory in chapter 8. Step 5 puts evaluation, security and operations around every stage, in chapters 9 and 10. Analogy: the offline lane is a library that catalogues its books; the online lane is the librarian who answers one question. Likely question: "Where do I start when my RAG system gives bad answers?" Answer: measure recall of the first stage before you change the prompt; chapter 1 shows why.

01

Why RAG

Models need knowledge they were never trained on.

Frozen. Knowledge stops at the training cutoff. An update needs a retrain.
No source. The model cannot cite, so you cannot audit a hallucination.
No private data. Tickets, contracts and inventory were never in training.
No access control. You cannot filter weights per user.

The RAG answer. Keep knowledge in an external index that you can swap and filter by permission. At query time, retrieve the top-k passages. Condition the answer on them.

Update = re-index, not retrain.

Paper Lewis et al. 2020, arXiv 2005.11401 Video RAG From Scratch (LangChain)

external index

User

LLMparametric memory

cutoff 2025

Indexchunks + ACL

swap the corpus
filter by user
re-index on change

frozen at cutoff no citations no private data no per-user ACL ✓ fresh data ✓ cited sources ✓ private data ✓ per-user ACL

Lewis et al. 2020: a 400M generator plus an index beats an 11B closed-book model

Index: 21M passages 2016 index 2018 index

Query x

Query encoderBERT · DPR

MIPSFAISS · top-k

passage z1

passage z2

passage z3

GeneratorBART-large ~400M

The query encoder makes q(x). MIPS returns the top-k passages z. BART writes the answer y token by token.

RAG-Sequence. One passage conditions the whole answer: p(y|x) ≈ Σ_z p(z|x) · Π_i p(y_i|x, z, y_<i)

RAG-Token. Each token can use a different passage: p(y|x) ≈ Π_i Σ_z p(z|x) · p(y_i|x, z, y_<i). Both marginalize over the top-k.

Natural Questions, exact match (test, Table 1)~27x smaller generator wins

T5-11B closed-book

0

REALM

0

DPR

0

RAG-Token

0

RAG-Sequence

0

Parametric weights + non-parametric index

Retriever: DPR, a BERT bi-encoder. Fine-tune the query encoder. Keep the index frozen.
Generator: BART-large, about 400M parameters.
Index: 21M Wikipedia passages of 100 words. MIPS search with FAISS.

Hot-swap the index: world-leader questions

2016 Q · 2016 idx

0

2018 Q · 2018 idx

0

2016 Q · 2018 idx

0

2018 Q · 2016 idx

0

"we can update RAG's world knowledge by simply replacing its non-parametric memory."

Lineage: DrQA 2017 → ORQA 2019 → REALM, DPR 2020 → RAG 2020 → FiD, RETRO 2021 → prompt RAG with frozen LLMs → agentic RAG

Paper Lewis et al., arXiv 2005.11401 (NeurIPS 2020)

This is the paper that named RAG: Lewis et al., FAIR, UCL and NYU, NeurIPS 2020. Step 1 draws the pipeline: query encoder, MIPS over 21M passages, top-k passages, BART generator. Steps 2 and 3 show the two variants: RAG-Sequence keeps one passage for the whole answer, RAG-Token lets each token use a different passage; both sum over the top-k. Step 4 is the headline: a 400M generator with an index gets 44.5 exact match on Natural Questions, against 34.5 for closed-book T5-11B, a model about 27 times larger. Step 5 is the experiment to remember: swap the 2016 index for the 2018 index and the answers change with it, no retraining. Analogy: the weights are what you remember, the index is your notebook. Likely question: "Is the retriever trained end to end?" Only the query encoder is fine-tuned; the document encoder and the index stay frozen, so the index never needs a rebuild during training.

Every RAG system is an offline index pipeline plus an online retrieval funnel

OFFLINE · indexing, batch ONLINE · every query

SourcesPDF · HTML2

Parselayout · tables · OCR2

Chunk+ LLM context2

Embedbi-encoder3

indexes

MetadataACL · date filter9

BM25 indexinverted4

Vector indexANN · HNSW3

Query

Rewriteoptional4

Hybrid retrieveANN top-k + BM25 → RRF4

Rerankcross-encoder4

LLMchunks + query4

Answer+ cites9

Pitfall: you tune the prompt when recall is the problem. The LLM can use only the chunks that the retriever returns. If the gold chunk is not in the top-k, no prompt can fix the answer. Measure recall@k on a golden set first.

Failure rate = 1 − recall@20. Anthropic's baseline misses the gold chunk on 0 of queries. Badges show the chapter that teaches each box.

Blog Introducing Contextual Retrieval Repo langchain-ai/rag-from-scratch

Every RAG system, from a weekend demo to a managed service, has these two pipelines. Step 1 is the offline lane: parse, chunk, optionally add context, embed, and write three indexes. Step 2 is the online lane that runs on every query: rewrite, hybrid retrieve, rerank, then build the prompt for the LLM. Step 3 is the funnel: one million documents, 150 hybrid candidates, 20 after the reranker, 5 in the prompt. Step 4 names what each band decides: recall at the first stage, precision at the reranker, use at context assembly. Analogy: a library; the catalogue is built at night, and each visitor gets a librarian who narrows a million books to the five on the desk. Likely question: "Where do I start when answers are wrong?" Start at the top of the funnel: measure recall@k, because nothing downstream can bring back a chunk that the first stage dropped.

Chapter 01 · Why RAG Interview lens recall first

Interview: when RAG, when fine-tuning, and where to look first

When do you choose RAG over fine-tuning?

Use RAG for fresh, private or permissioned facts and for citations. An update is a re-index, not a retrain. Use fine-tuning for behaviour, format and style. Combine them when you need both.

Explain RAG-Sequence vs RAG-Token.

RAG-Sequence uses one retrieved document for the whole output. RAG-Token can use a different document for each token. Both marginalize over the top-k documents. On NQ, RAG-Sequence scored 44.5 EM and RAG-Token 44.1.

Your RAG bot gives wrong answers. What do you check first?

Check recall@k on a golden set. If the gold chunk is not in the top-k, the generator cannot fix the answer. Separate retrieval failures from generation failures, then fix the stage that fails.

Numbers to know.

Lewis 2020: NQ exact match 44.5 with retrieval vs 34.5 for closed-book T5-11B. Hot-swap: 70% with the matched index vs 4% mismatched. Anthropic Contextual Retrieval plus reranking: top-20 failures 5.7% → 1.9% (−67%).

Paper arXiv 2005.11401 Blog Introducing Contextual Retrieval Course DeepLearning.AI RAG course

02

Ingestion and chunking

Retrieval quality is decided at ingestion, before the first query. Tables, columns, scans and charts break naive text extraction.

Three questions for this chapter

Parse. How do you read layout, tables and reading order?
Chunk. Where do you cut the text?
Contextualize. How does each chunk keep its context?

0recall swing from the chunking choice (Chroma, 2024)

0top-20 failures, Contextual Retrieval without reranking (Anthropic, 2024)

1 · find the layout blocks 2 · chunks + context tags

Parse layout, tables and reading order first, because naive text extraction scrambles them

Naive extraction pdftotext

Revenue by region The board approved

rose in EMEA and a new buyback plan

Region Q1 Q2 EMEA 4.1 for next

4.6 APAC 3.2 3.9 year. Risk

Two columns interleave line by line. The table loses its rows and columns.

Layout parser Docling → Markdown

# Revenue by region
Revenue rose in EMEA and APAC.
| Region | Q1  | Q2  |
|--------|-----|-----|
| EMEA   | 4.1 | 4.6 |
| APAC   | 3.2 | 3.9 |

Reading order is kept. The table stays one unit.

VLM tier hard pages only

→

| Year | Units |
|------|-------|
| Y1   | 36    |
| Y2   | 54    |
| Y3   | 66    |

A VLM reads the chart and writes a table. It costs more per page, so send only hard pages here.

parse.py · Docling README quickstart


from docling.document_converter import DocumentConverter

source = "https://arxiv.org/pdf/2408.09869"  # local path or URL
converter = DocumentConverter()
result = converter.convert(source)   # layout, tables, OCR
md = result.document.export_to_markdown()  # -> Markdown
print(md)
# CLI: default pipeline, then the GraniteDocling VLM
# $ docling https://arxiv.org/pdf/2206.01062
# $ docling --pipeline vlm --vlm-model granite_docling <url>

Parser	What it gives you (Oct 2026)
Repo Docling	MIT, LF AI & Data (from IBM Research Zurich). PDF, DOCX, PPTX, XLSX, HTML, EPUB, images, audio, LaTeX. Layout, reading order, tables, OCR, MCP server. Python 3.10+.
Doc Unstructured	Typed elements: `Title`, `NarrativeText`, `Table`, `ListItem`. `by_title` never spans two sections. `max_characters` 500. Tables stay isolated; `orig_elements` kept.
Doc LlamaParse	v2 tiers: `fast`, `cost_effective`, `agentic`, `agentic_plus`.
Doc Gemini	1,000 pages and 50 MB per PDF; 258 tokens per page. Pages to Markdown, or no chunking for a small corpus.

Chunk boundaries move recall by up to 9 points, and recursive 200-token chunks are a strong default

Chroma eval: token-level recall and IoU

text-embedding-3-large · 5 chunks

strategyrecall (0–100)IoU (0–10) Recursive 200

88.1

6.9 Recursive 400

89.5

3.6 ClusterSemantic 200

87.3

8.0 LLMSemantic (GPT-4o)

91.9

3.9

Chroma: “default settings for certain popular chunking strategies can lead to relatively poor performance.” Less overlap gives a better IoU.

2026: no single chunk size fits every query. AI21, “Stop Chunking Like It's 2022” (AI Engineer World's Fair 2026): keep several indexes.

Research Evaluating Chunking Strategies for Retrieval Doc LlamaIndex node parsers

Why chunk: embedders take 2K–32K tokens, one vector averages away detail, context costs money, precision needs passage-level units.

Strategy	How it cuts · trade-off
Fixed-size	Every N tokens, overlap O. Fast; cuts sentences and tables.
Recursive	`\n\n` → `\n` → `". "` → `" "` until ≤ N. Strong default; ignores meaning.
Structure	Markdown/HTML headers, code AST, `by_title`. Needs a good parser.
Semantic	Embed sentences; cut where adjacent similarity drops (95th-percentile breakpoint). Extra embedding cost.
LLM-based	An LLM proposes the boundaries. Best recall in Chroma; slow, costly.
Parent-child	Index small, return the parent. LlamaIndex `HierarchicalNodeParser` [2048, 512, 128] + `AutoMergingRetriever`; sentence window 3; LangChain `ParentDocumentRetriever`.

Copying 1,000-character chunks with 200 overlap from a tutorial. Measure recall and IoU on your own golden set.

One short text passes through four strategies. The first paragraph is the Berlin passage from the late-chunking paper; S1 to S9 are its sentences in three paragraphs. Step 1: fixed-size cuts every 16 tokens, and every cut lands inside a sentence. Step 2: recursive splitting tries paragraph breaks first, so the cuts move to paragraph ends. Step 3: semantic chunking embeds each sentence and cuts where the similarity of neighbours dips below the breakpoint; it finds a topic change inside paragraph 2 that the recursive splitter misses. Step 4: the query matches child S5; parent-child indexes small sentences for precise matching but sends the whole parent paragraph to the LLM. Step 5: the Chroma numbers. Recall changes by only a few points, but IoU changes by 2x, so token waste is the bigger difference. Analogy: cutting a newspaper with a ruler versus cutting it along the articles. Likely question: "Why not always use the LLM chunker, since it has the best recall?" Answer: one LLM call per document section, slower ingestion, and its IoU is among the worst.

Chunks lose their context, and late chunking or Contextual Retrieval puts it back

Fix A · Late chunking

Paper Jina, arXiv 2409.04701

0“Its…” to “Berlin”, naive 0.708

BEIR nDCG@10: NFCorpus 23.46 → 29.98, SciFact 64.20 → 66.10. No extra training. Needs token-level access to a long-context embedder.

Fix B · Contextual Retrieval

Repo anthropic-cookbook

The LLM reads the full document plus one chunk and writes 50–100 tokens of context. Prepend it before both embedding and BM25.

The problem chunk

“The company's revenue grew by 3% over the previous quarter.”

Which company? Which quarter? “traditional RAG solutions remove context when encoding information.”

Blog Introducing Contextual Retrieval (19 Sep 2024)

Top-20 retrieval failure rate

Baseline chunks

5.7%— + contextual embeddings

3.7%−35% + contextual BM25

2.9%−49% + reranking (150 → 20)

1.9%−67%

0per million document tokens, one time

Prompt caching stores the document once. Every chunk call reads it from the cache. “All these benefits stack.”

situate.py · adapted from the blog prompt


import anthropic
client = anthropic.Anthropic()
def situate(doc: str, chunk: str) -> str:
    msg = client.messages.create(
        model="claude-haiku-4-5",  # illustrative; blog used Claude 3 Haiku
        max_tokens=150,
        system=[{"type": "text",
                 "text": f"<document>\n{doc}\n</document>",
                 "cache_control": {"type": "ephemeral"}}],  # cached once
        messages=[{"role": "user", "content":
            f"Here is the chunk we want to situate within "
            f"the whole document\n<chunk>\n{chunk}\n</chunk>\n"
            "Please give a short succinct context to situate "
            "this chunk within the overall document for the "
            "purposes of improving search retrieval of the "
            "chunk. Answer only with the succinct context "
            "and nothing else."}])
    return msg.content[0].text + "\n\n" + chunk  # index for dense AND BM25

Start with the problem chunk: alone, it does not say which company or which quarter, so neither the embedder nor BM25 can match a query that names them. Step 1, late chunking: run the long-context embedder over the whole document first, then mean-pool each chunk's token span; the token "Its" now carries attention from "Berlin", so the cosine rises from 0.708 to 0.825. Step 2, Contextual Retrieval: an LLM writes a short context note for each chunk, and you index note plus chunk in both the dense and the BM25 index. Step 3: the gains stack, from 5.7% to 1.9% failures with reranking. Steps 4 and 5: the cache_control block is the reason this is cheap, about $1.02 per million document tokens, because the document is written to the cache once and read for every chunk. Analogy: late chunking reads the whole book before it summarizes a page; Contextual Retrieval writes a header on each index card. Likely question: "Which fix do I choose?" Answer: late chunking needs an open long-context embedder with token outputs; Contextual Retrieval works with any embedder and also fixes BM25, but costs one LLM call per chunk.

Interview: parsing and chunking trade-offs

ingestionchunkingcontextual retrievaltables and scans

How do you choose a chunk size?

Run an eval, not a guess. Measure token-level recall and IoU on a golden set. Start with recursive ~200 tokens, no overlap (Chroma: recall 88.1, IoU 6.9). Move to structure-aware chunks when the parser is good.

Why is Contextual Retrieval cheap enough to run on every chunk?

Prompt caching. The document is written to the cache once. Each chunk call reads it from the cache and adds only the chunk. Cost: about $1.02 per million document tokens, one time. Failures fall 5.7% → 2.9% before reranking.

Financial PDFs full of tables give bad answers. Design the fix.

Use a layout parser that isolates tables (Docling, Unstructured). Send charts and scans to a VLM tier. Or retrieve page images with ColPali (chapter 8). Keep each table as its own chunk, with its caption and section as context.

Parent-child retrieval: what do you index and what do you send?

Index small children for a precise match. Send the parent or a sentence window for context (LlamaIndex AutoMergingRetriever, LangChain ParentDocumentRetriever). Cost: more storage and more bookkeeping.

03

Embeddings and vector indexes

An encoder maps meaning to a vector. Similar meanings land near each other. An index finds the nearest vectors fast.

Similarity is geometry

Use cosine, dot product on normalized vectors, or L2. In 2026 one space can hold text, images, audio, video and PDFs.

Exact search does not scale

Exact k-NN costs O(N × d) per query. ANN gives up a little recall for orders of magnitude more speed.

Three questions for this chapter

Which model?How many dimensions?Which index?

Choose the embedding model on your own data, because MTEB ranks differ by sub-board

embed.py · Gemini API


from google import genai
client = genai.Client()
q = client.models.embed_content(
    model="gemini-embedding-2",
    contents="task: question answering | "
             "query: What is late chunking?",
    config={"output_dimensionality": 768})  # MRL
d = client.models.embed_content(
    model="gemini-embedding-2",
    contents="title: Late Chunking | "
             "text: Late chunking pools ...",
    config={"output_dimensionality": 768})

Paper 2503.07891 Repo Qwen3-Embedding

Model	Dims · MRL	Input	Key fact
gemini-embedding-001	3072 · 128–3072	2,048	task_type; MMTEB 68.32
gemini-embedding-2	3072 · 128–3072	8,192	5 modalities; GA ~Apr 2026
text-embedding-3-large	3072 · `dimensions`	8,192	Jan 2024; MTEB 64.6
Voyage 4 family	1024 · 256–2048	32K	shared space; int8, binary
Cohere embed-v4.0	1536 · 256–1536	128K	text and images
Qwen3-Embedding-8B	4096 · MRL	32K	Apache-2.0; MMTEB 70.58

A new model means a full re-embed. Never mix two models in one index.

gemini-embedding-2: task in the text, not compatible with 001, Batch API at 50%. MTEB sub-boards (MMTEB, Eng v2, Code) rank differently; Voyage does not submit. Vendor-reported (Google Dev Blog, 30 Apr 2026): Nuuly Match@20 60% → ~87%; Supermemory +40% Recall@1; Harvey +3% Recall@20. Docs Gemini API embeddings Board MTEB leaderboard

Start with the picture: every chunk is a point, and the query is a new point in the same space. The index returns the five chunks with the smallest angle to the query, which is the highest cosine similarity. Then walk the table. The 001 paper reports an MMTEB mean of 68.32, 5.09 points above the next model; Qwen3-Embedding-8B was #1 on MMTEB on 5 Jun 2025 with 70.58, and it has open weights. Gemini Embedding 2 is the new default on Google because it puts five modalities in one space, but it uses text prefixes, not task_type, and it cannot share an index with 001. Analogy: two embedding models are two maps with different projections; a coordinate from one map points to the wrong place on the other. Likely question: "Which model is best?" Answer: the leaderboard sub-boards disagree and some vendors do not submit, so build a golden set from your own queries and measure recall@k before you choose.

Matryoshka dimensions and quantization shrink a 123 GB index to under 1 GB

Matryoshka Representation Learning

Train the encoder so that every prefix (the first 64, 128, 256 … dims) is itself a good embedding. Sum the loss over the nested sizes.

“up to 14x smaller embedding size … at the same level of accuracy”

Paper Kusupati et al., NeurIPS 2022

MRL is in every major API in 2026

Gemini 128–3072 (use 768 or 1536)
OpenAI dimensions parameter
Voyage 256–2048; output_dtype float, int8, uint8, binary, ubinary
Cohere v4 256–1536 · Qwen3 MRL

Docs Voyage embeddings

Re-normalize after you truncate (OpenAI docs). If you do not, cosine scores are wrong. Never compare truncated vectors with full vectors.

Funnel retrieval: search with short or binary vectors. Re-score the candidates with the full vectors.

HNSW descends from long links to short links, while IVF-PQ, DiskANN and ScaNN trade memory for recall

HNSW (Malkov & Yashunin, TPAMI 2018) is the default in pgvector, Qdrant, Weaviate, Milvus and FAISS.

PQ

Split a vector into M sub-vectors; code each with 2^nbits centroids. 768-d float32: 3,072 B → PQ96x8: 96 B.

DiskANN / Vamana

PQ codes in RAM; graph and full vectors on SSD. SIFT1B on a 64 GB box: >5,000 QPS, <3 ms, 95%+ 1-recall@1.

ScaNN + SOAR

Anisotropic quantization. SOAR spills each vector to a second, near-orthogonal cluster. Vertex AI Vector Search and AlloyDB use it.

Paper HNSW, arXiv 1603.09320 Wiki Faiss indexes Blog SOAR with ScaNN

index.py · FAISS


import faiss, numpy as np
d, nlist, m = 768, 4096, 96      # 96 x 8 bits = 96 B
quantizer = faiss.IndexFlatL2(d)
index = faiss.IndexIVFPQ(quantizer, d, nlist, m, 8)
index.train(xt)                  # representative sample
index.add(xb)
index.nprobe = 16                # query-time knob
D, I = index.search(xq, 10)

# graph index instead of IVF-PQ
hnsw = faiss.IndexHNSWFlat(d, 32)  # M = 32 neighbours
hnsw.hnsw.efSearch = 64

HNSW knobs and bytes per vector

M: graph degree → memory and recall
efConstruction: build quality → build time
efSearch: beam width → latency vs recall
Flat 4·d · IVFPQ ⌈M·nbits/8⌉+8 · HNSWFlat 4·d + M·2·4

Every index is a point on the recall, latency, memory and build-time surface; Elasticsearch and OpenSearch also use HNSW by default. HNSW works like a road network: the top layer is the highway with a few long links, and layer 0 is the street level with every vector. The search enters at the top, moves greedily to the neighbour nearest the query until it cannot improve, drops one layer, and repeats; on layer 0 it runs a beam search of width efSearch and returns the top k. Step 4 shows the other family: IVF cuts the space into nlist k-means cells and searches only the nprobe nearest cells, and PQ compresses each vector to 96 bytes. DiskANN keeps only PQ codes in RAM, which is how one workstation serves a billion vectors. Likely question: "HNSW or IVF-PQ?" Answer: HNSW for the best recall and latency when the vectors fit in RAM; IVF-PQ or DiskANN when memory is the limit, and always measure recall@k against exact k-NN.

Interview: embeddings and ANN index design

Design an index for 1B vectors on a tight memory budget.

Use DiskANN (PQ codes in RAM, graph and full vectors on SSD) or IVF-PQ. Cut to 768 dims with MRL, then use int8 or binary. Re-score the top candidates with the full vectors. Measure recall@k against exact k-NN on a sample. DiskANN served SIFT1B from one 64 GB workstation at >5,000 QPS.

What do HNSW M, efConstruction and efSearch change?

M is the graph degree: more memory (HNSWFlat costs 4·d + M·2·4 B) and higher recall. efConstruction sets build quality: higher values make the build slower. efSearch is the query beam width: higher values give more recall and more latency. You set efSearch at query time.

Can you query a gemini-embedding-001 index with gemini-embedding-2 vectors?

No. The two models make different vector spaces, so the distances have no meaning. Do a blue/green re-embed: build a second index with the new model, compare both on the golden set, then move the traffic.

Numbers to know.

10M × 3072 × 4 B ≈ 123 GB; 768-bit binary ≈ 0.96 GB. PQ96x8 = 96 B per vector (3,072 B for 768-d float32). MRL: up to 14× smaller at the same accuracy. Exact k-NN: O(N × d) per query.

04

Retrieval quality and context assembly

Hybrid search gives recall, reranking gives precision, and context assembly decides use.

Hybrid search + RRF Rerankers Query transforms + self-query Lost in the middle / context rot Long context vs RAG + caching Context engineering

This chapter follows one query through the retrieval funnel. The first band is recall: cheap retrievers (BM25, dense and learned sparse) search millions of documents, metadata filters run inside this stage, and RRF fuses the lists into about 100 to 150 candidates. The second band is precision: a reranker sees the query and each document together and keeps 5 to 20. The third band is use: the order, the number of distractors and the total length decide whether the model actually uses what we found. Analogy: a hiring funnel, where a CV keyword screen gives recall, an interview gives precision, and the final brief to the manager decides what gets read. A likely question: "Which stage should I improve first?" Measure first; a miss at the recall stage can never be fixed later, so check recall@100 before you tune the reranker.

BM25 and dense retrieval fail on different queries, so run both

BM25 wins. Dense vectors miss exact tokens: error codes, SKUs, names and rare acronyms.

Dense wins. "car" ≈ "automobile". BM25 misses synonyms and paraphrase.

Both hit. The failures barely overlap, so the union raises recall. BEIR (NeurIPS 2021): "BM25 is a robust baseline."

SPLADE (Formal et al., SIGIR 2021): a BERT MLM head projects each token onto the whole 30,522-term vocabulary. Log-saturation and a FLOPS regulariser keep a few hundred weights. Expansion adds "vehicle" and "automobile" to a "car" passage.

It is still an inverted-index vector: Lucene, Elastic, OpenSearch, Qdrant sparse. Sentence Transformers: SparseEncoder("naver/splade-cocondenser-ensembledistil").

Score fusion, if you fuse scores: Weaviate relativeScoreFusion (default since v1.24) min-max normalises, then weights by alpha (0.5 default, 1 = pure vector, 0 = pure BM25). Qdrant DBSF normalises each list with mean ± 3σ. Rank fusion is on the next slide.

Docs Weaviate hybrid search Paper BEIR Paper SPLADE

Start with three real query shapes. "error E1023" is an exact token, so BM25 finds it and a dense vector blurs it. "How to make the car faster" uses other words than the document, so only the dense retriever finds it. "Q3 revenue ACME" is found by both, and the point is that the two failure sets barely overlap, so running both raises recall. Then show the three vector types for one passage: BM25 keeps only the words that are there, dense spreads meaning over 768 numbers, and SPLADE learns weights over the whole vocabulary and adds expansion terms in blue, while it still runs on an inverted index. Analogy: one witness remembers exact names, another remembers what happened; ask both. Likely question: "Does SPLADE replace hybrid search?" It narrows the gap, but most production systems still fuse a sparse list and a dense list, and you measure on your own queries.

Reciprocal Rank Fusion merges ranked lists by rank only, with k = 60

RRFscore(d) = Σ 1 / (k + rank_r(d)) over all rankers r. Only ranks count, so BM25 (unbounded) and cosine (−1..1) need no calibration.

Pilot MAP: k=0 .2072 · k=60 .2145 · k=500 .2098. "k = 60 was fixed during a pilot investigation and not altered." RRF beats Condorcet, CombMNZ and the best single system by 4–5% on average.

Qdrant: set each prefetch limit ≥ the main limit + offset. If you do not, the result can come back empty.

Paper Cormack et al., SIGIR 2009 Docs Qdrant hybrid queries

rrf.py


def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

bm25 = ["A", "C", "E"]; dense = ["D", "B", "A"]
print(rrf([bm25, dense])[0])   # ('A', 0.03226...)

from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")
results = client.query_points(
    collection_name="my_collection",
    prefetch=[
        models.Prefetch(query=models.SparseVector(indices=[1, 42],
                        values=[0.22, 0.8]), using="sparse", limit=20),
        models.Prefetch(query=[0.01, 0.45, 0.67], using="dense", limit=20),
    ],
    query=models.RrfQuery(rrf=models.Rrf(k=60)),   # k param since v1.16.0
    limit=10,
)
# Weighted RRF (v1.17.0+): models.Rrf(weights=[3.0, 1.0])

RRF is the default way to merge a BM25 list and a dense list. Each list gives a document 1 over (60 plus its rank), and the scores add up. Walk the worked example: A is rank 1 in BM25 and rank 3 in dense, so it gets 0.0164 plus 0.0159, which is 0.0323; B is only in the dense list at rank 2, so it gets 0.0161; A wins because both lists agree on it. Then show the pilot table: k from 0 to 500 barely moves MAP, so you almost never tune k. Analogy: two judges rank the same contestants; you add up placings, not their raw scores, because one judge scores out of 10 and the other out of 100. Likely question: "When is score fusion better?" When the scores are calibrated and you have labels to tune alpha; otherwise RRF is the safe default, and Qdrant v1.17 adds weights if one list is more reliable.

Rerank about 150 candidates with a cross-encoder or LLM, then keep the best 20

Late interaction: ColBERTv2 (NAACL 2022) keeps one vector per token. Residual compression makes the index 6–10× smaller than v1. sentence-transformers v6.0.0 (18 Aug 2026) adds MultiVectorEncoder.
Cohere rerank-v5.0-pro / -fast: 32k context, 100+ languages, adds relevant_spans. rerank-v3.5: 4k. Voyage rerank-3 / rerank-3-lite: 32K.
Open: BAAI/bge-reranker-v2-m3 (0.6B, multilingual). jina-reranker-v3 (0.6B on Qwen3, listwise over 64 docs in one 131K context, BEIR nDCG@10 61.94, CC BY-NC 4.0).
LLM rerankers: RankGPT (EMNLP 2023) outputs a permutation with a sliding window: "a distilled 440M model outperforms a 3B supervised model on BEIR".
Reasoning rerankers: Rank1 (COLM 2025, 600K+ reasoning traces). ReasonIR-8B on BRIGHT: nDCG@10 29.9 → 36.9 with a reranker.

0candidates in

→

0kept for the prompt

Anthropic: retrieve 150, rerank, keep 20. Cost is one API round-trip or one GPU pass per pair. Cap the candidates. Keep docs ≤ max_tokens_per_doc (Cohere default 4096).

rerank.py


import cohere
co = cohere.ClientV2()
resp = co.rerank(
    model="rerank-v5.0-pro",        # or rerank-v5.0-fast, rerank-v3.5
    query="What is the capital of the United States?",
    documents=["Carson City is the capital city of Nevada.",
               "Washington, D.C. is the capital of the United States.",
               "Capital gains are profits from the sale of an asset."],
    top_n=2,
    relevant_spans={"mode": "enabled", "threshold": 0.1,
                    "max_spans": 3},   # v5+ only: spans behind the score
)
for r in resp.results:
    print(r.index, r.relevance_score)

Docs Cohere Rerank Codelab Advanced RAG Techniques (VertexAIRank)

Show the interaction spectrum one grid at a time. A bi-encoder makes one vector for the query and one for the document, so the only interaction is one cosine; it is fast because documents are encoded offline, but it loses detail. Late interaction keeps a vector per token: each query token takes its best match in the document (MaxSim) and the maxima add up. A cross-encoder reads the query and the document together with full attention, so it is the most accurate and also needs one forward pass per pair, which is why we only run it on about 150 candidates and keep 20. Analogy: the bi-encoder reads the book titles, ColBERT reads the index at the back, and the cross-encoder reads the chapter with your question in mind. Likely question: "Why not use a cross-encoder for the first stage?" Because you cannot precompute anything, so a million documents means a million forward passes per query.

Rewrite the query when the user's words do not match the documents, then measure end to end

Multi-query

1 questionLLM → 3 versionsunique union

LangChain default: "generate 3 different versions".

RAG-Fusion

1 questionLLM → N rewritesRRF fuse, k = 60

Multi-query + RRF. Repo: Raudaschl/rag-fusion.

HyDE

questionLLM → fake answerembed the answer

ACL 2023. The "dense bottleneck" filters fake detail.

Step-back

specific questionLLM → the principleretrieve for both

PaLM-2L: TimeQA +27%, Chemistry +11%.

Decomposition

multi-hop questionLLM → sub-questionsanswer one by one

Least-to-Most, Self-Ask. The seed of agentic RAG.

Self-query and conversational rewrite

"sci-fi movies rated above 8.5 from the 90s" → query="sci-fi" + filter rating > 8.5 AND 1990 <= year < 2000 (SelfQueryRetriever + AttributeInfo). "what about its price?" → one standalone query from the chat history.

Rule: apply tenant and ACL filters server-side and deterministically. Never let the LLM set them.

Medrano et al., arXiv 2603.02153 (2026): fusion gains "mostly disappear after re-ranking and context truncation". The lesson: "retrieval-level improvements do not reliably translate into end-to-end gains."

Hit@10 before

0.51

with RAG-Fusion

0.48

query_transforms.py


def hyde_search(question, llm, embed, index, k=5):
    fake_doc = llm("Write a short passage that answers the question.\n"
                   f"Question: {question}\nPassage:")
    vec = embed(fake_doc)            # embed the answer-shaped text
    return index.search(vec, k)      # real docs near the fake one
from langchain_classic.retrievers.multi_query import MultiQueryRetriever
mq = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    llm=ChatGoogleGenerativeAI(model="gemini-2.5-flash"),
    include_original=True)           # default False

Codelab Advanced RAG Techniques Paper Medrano 2026

Query transforms fix a vocabulary gap between how users ask and how documents are written. Multi-query and RAG-Fusion write several rewrites and merge the hits, HyDE embeds a fake answer because answers look more like documents than questions do, step-back asks for the general principle first, and decomposition splits a multi-hop question into steps, which leads into agentic RAG in the next chapter. Self-query turns words into a metadata filter, but the security rule is strict: tenant and ACL filters come from your auth system, never from the model. The "Advanced RAG Techniques" codelab runs HyDE and step-back on Cloud SQL with pgvector, and its whole HyDE prompt is one line: "Write a concise, hypothetical answer to the question." Analogy: a librarian rephrases your question before searching the catalogue. Likely question: "Should I always add RAG-Fusion?" No; the 2026 industry study saw recall gains vanish after reranking and truncation, so every transform costs an LLM call and must earn its place on an end-to-end metric.

More context is not better context: models miss facts in the middle and degrade with length

Lost in the Middle (Liu et al., TACL 2024): accuracy is highest when the relevant document is at the start or the end. In the middle, GPT-3.5-Turbo fell below its closed-book score. More documents made it worse than none.

Context Rot (Chroma, 14 Jul 2025, 18 models incl. Claude 4, GPT-4.1, Gemini 2.5, Qwen3): models "do not use their context uniformly". "Even a single distractor reduces performance." Shuffled haystacks beat coherent ones. Note: Chroma sells retrieval infrastructure. Kelly Hong: "Simply having the right information in the context is not enough."

Rerank, then send fewer chunks.
Put the best chunks first and last (LangChain LongContextReorder).
Remove near-duplicates before you build the prompt.

Raising top-k to 50 "to be safe" adds distractors and pushes the answer into the middle.

Paper Lost in the Middle Report Context Rot Video Kelly Hong, Context Rot

First the axes: the x-axis is where the relevant document sits among 20 documents. The curve is U-shaped: the model reads the start and the end well and the middle badly; I draw the shape only, the paper has the exact values. Then the dashed line: GPT-3.5-Turbo with no documents at all scored 56.1%, and with 20 or 30 documents and the answer in the middle it scored 53.8% and 50.5%, so the red area is where retrieval made the answer worse. Chroma's Context Rot report adds that a focused 300-token prompt beat the full 113k-token history in every model family, and that even one distractor hurts. Analogy: a long meeting where the key decision was made in minute 40 of 90; everyone remembers the opening and the close. Likely question: "Do 1M-token models fix this?" No; the report tested 2025 frontier models and still saw non-uniform use of the context, so rerank, send less, and put the best evidence at the edges.

Cache a small static corpus in the prompt; retrieve from a large, changing or permissioned one

Corpus < 200k tokens and static?one permission level

Cache in promptCAG / prompt cache

Hybrid + rerank RAGlarge, daily changes, per-user ACLs

Multi-hop question?model judges the context

Agentic / Self-Routeloop or fall back

Anthropic prompt-cache price, × base input

base input

1×

write, 5-min TTL

1.25×

write, 1-hour TTL

2×

read (0.05× Opus/Sonnet 5.5)

0.1×

Gemini: implicit caching is on by default for 2.5+ (minimum 2,048 / 4,096 tokens). Explicit client.caches.create has a 1 h default TTL. It is not supported on Gemini 4 Argon and later. Anthropic: up to 4 breakpoints, minimum 512–4,096 tokens by model.

Anthropic: if the knowledge base is "smaller than 200,000 tokens (about 500 pages)", skip RAG. Put it in the prompt with caching: >2× lower latency, up to 90% lower cost.
CAG (Chan et al., arXiv 2412.15605): preload the corpus and precompute the KV cache. No retriever. It works only when the corpus fits.
Self-Route (Google DeepMind, EMNLP 2024): long context wins on average when resourced, but RAG is far cheaper. The model checks if the retrieved context is enough. If not, it falls back to long context.
LaRA (ICML 2025, 2,326 cases, 11 LLMs): "no silver bullet". Llama 4 Scout claims 10M tokens, but it was trained at 256K.
2026 synthesis: not RAG vs long context, but RAG into long context. Retrieve and rerank to a focused set. Use the big window for breadth and caching.

cache_kb.py


import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-opus-5-5", max_tokens=1024,
    system=[{"type": "text", "text": BIG_KNOWLEDGE_BASE,
             "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
    messages=[{"role": "user", "content": question}])
print(resp.usage.cache_read_input_tokens)   # billed at the read rate

Docs Claude caching Docs Gemini caching Paper Self-Route

Walk the decision tree one branch at a time. If the corpus is under about 200k tokens, does not change, and every user may see all of it, put it in the prompt and cache it; Anthropic reports more than 2x lower latency and up to 90% lower cost, and CAG goes further by precomputing the KV cache. If the corpus is large, changes daily, or has per-user permissions, you need hybrid retrieval with reranking, because you cannot cache a different prompt for every user and every edit. If one retrieval pass misses facts that need several hops, move to an agentic loop or Self-Route, where the model decides if the retrieved context is enough. The cost bars show why caching pays: you pay 1.25x or 2x once to write and 0.1x on every read. Analogy: a small reference card lives on your desk; the full library needs a catalogue and a librarian. Likely question: "With 10M-token models, why retrieve at all?" Cost, latency, freshness and permissions, plus the context-rot results from the previous slide.

Context engineering treats retrieval as one way to select the smallest high-signal context

"good context engineering means finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome." Anthropic, 29 Sep 2025

Just in time

Context windowsmallest high-signal set

SelectRAG: hybrid + rerank

Compresscompaction of old turns

turn 1: search refunds
turn 2: read 4 docs
turn 3: tool error, retry
turn 4: read 6 docs
… 36 more turns …

summary: decisions + open tasks

Sub-agent Areturns 1–2k tokens

Sub-agent Bown context window

WriteNOTES.md, outside the window

tool call

path: refunds.md → loaded

query: refund tickets → loaded

link: status page → loaded

Definition: "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference." LangChain (2 Jul 2025) names four buckets: Write, Select, Compress, Isolate. RAG is Select. Sub-agents return "a condensed, distilled summary of its work (often 1,000-2,000 tokens)".

Just-in-time retrieval: the agent keeps "lightweight identifiers (file paths, stored queries, web links, etc.)" and loads the content through tools when it needs it. A hybrid strategy loads some data up front and lets the agent explore the rest (suited to legal or finance work).

Load agent skills the same way: by identifier, just in time.

Practitioners: Hamel Husain and Ben Clavié ("Stop Saying RAG Is Dead"): naive single-vector search is dead, retrieval is not. Bischof and Chaurasia: "Stop searching for the perfect embedding. Build specialized representations and route intelligently."

Post Anthropic: context engineering Post LangChain

Context engineering is the 2025 frame that replaced prompt engineering: the job is to choose which tokens the model sees, and RAG is only one of four ways to do it. Select brings in retrieved chunks; Compress replaces a long trace with a short summary; Isolate gives each sub-agent its own window and brings back only 1,000 to 2,000 tokens; Write keeps notes outside the window so the agent can read them later. The last step is just-in-time loading: the agent holds cheap identifiers such as a path, a stored query or a link, and pays for the content only when a tool call loads it. Skills and long instructions work the same way: keep a short index in the prompt and load the full text on demand. Analogy: a chef keeps the recipe names on the wall and takes a recipe card from the drawer only when an order needs it. Likely question: "Is RAG dead?" No; naive single-vector top-k is out of date, but selecting evidence is now a core part of every agent.

Interview: hybrid search, reranking and long context

Why use RRF instead of averaging BM25 and cosine scores?

The scores have different scales and distributions: BM25 is unbounded, cosine is −1..1. RRF uses ranks only, so it needs no calibration. k = 60 is stable: pilot MAP was .2072 / .2145 / .2098 for k = 0 / 60 / 500. Use score fusion (Weaviate alpha, Qdrant DBSF) only when you can tune it on labels.

You retrieve 150 and rerank to 20. What is the latency budget, and how do you cut it?

The cost is one cross-encoder pass per pair, or one API round-trip. Cap the candidates. Truncate docs (Cohere max_tokens_per_doc, default 4096). Use a fast tier (rerank-v5.0-fast, rerank-3-lite). Batch the pairs on a GPU. Check that a smaller candidate set keeps recall.

Your context has 30 chunks and accuracy fell. Why?

Lost in the middle and context rot. GPT-3.5-Turbo with 30 docs and the answer in the middle scored 50.5%, below its closed-book 56.1%. Even one distractor hurts. Send fewer, reranked chunks. Put the best first and last, and remove near-duplicates.

When would you skip RAG?

When the corpus is under ~200k tokens, static, and has one permission level. Put it in the prompt with prompt caching: write 1.25× (5 min) or 2× (1 h), read 0.1×. Anthropic reports >2× lower latency and up to 90% lower cost. For large, changing or ACL'd corpora, retrieve.

05

Agentic RAG patterns

Classic RAG is one fixed pass. The pipeline retrieves once, also when the question does not need it. Nothing checks the retrieved evidence before the model uses it.

Agentic RAG is a policy. An LLM decides whether to retrieve, where (vector DB, SQL, web, API, MCP server), what to query (rewrite, decompose), if the evidence is good enough (grade, reflect) and when to stop.

Agentic RAG works by “embedding autonomous AI agents into the RAG pipeline.”

Paper Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (Singh et al., arXiv 2501.09136, v4 Apr 2026)

Start with the simplest workflow, and add an autonomous loop only when one pass fails

Prompt chaining

Rewrite

Retrieve

Answer

RAG: rewrite → retrieve → answer

Routing

Router

Vec

SQL

Web

RAG: source router (vector, SQL, web)

Parallelization

Query

q1

q2

q3

RRF fuse

RAG: multi-query fan-out, then fuse

Orchestrator-workers

Lead

W1

W2

W3

RAG: deep research (lead plans, workers search)

Evaluator-optimizer

Retrieve

Grade

RAG: grade → rewrite loop (CRAG-style)

A workflow first, an agent only when needed

A workflow is a fixed code path with LLM steps. An agent chooses its own path. When you know the path, use a workflow: it is cheaper to test, trace and cap.

Survey taxonomy (Singh et al.): 4 dimensions, 4 patterns, 7 architecture families. The 5 workflows come from Anthropic's “Building effective agents”.

0tokens of chat, one agent

0tokens of chat, multi-agent

Blog Anthropic multi-agent research (Jun 2025)

Failure modes of loops: endless loops, query drift after rewrites, a grader that says “relevant” to everything, context bloat, and prompt injection from retrieved pages.

Repo asinghcsu/AgenticRAG-Survey

The survey by Singh et al. (arXiv 2501.09136, v4 April 2026) sorts agentic RAG by four dimensions and seven architecture families, but the practical lesson is simpler. Each of the five workflow patterns from Anthropic's "Building effective agents" maps to one RAG job: chaining is rewrite, retrieve, answer; routing picks the source; parallelization fans out queries and fuses them with RRF; orchestrator-workers is deep research; evaluator-optimizer is the grade-and-rewrite loop. A workflow is a fixed code path, so you can test it, trace it and put a cap on it. Use an autonomous agent only when one pass fails on real queries. Then show the cost: an agent uses about 4 times the tokens of chat, and a multi-agent system about 15 times (Anthropic, June 2025). Analogy: a workflow is a recipe, an agent is a chef who improvises; use the recipe until it fails. Likely question: "Where is the line between the two?" If code decides the next step, it is a workflow; if the model decides, it is an agent.

A router picks the source, and Adaptive-RAG picks how much retrieval a question needs

Question

Router LLMstructured output

{"source": "?"}

Vector DBproduct docs

SQLtext-to-SQL · orders DB

WebGoogle Search / Tavily

Noneanswer directly

refund policy? orders last week?

router.py · enum schema


class Route(BaseModel):
    source: Literal["vector", "sql", "web", "none"]

route = llm.with_structured_output(Route).invoke(q)

Constrain the router with an enum schema.
Log every route decision in the trace.
Fall back to none plus a clarifying question.
Measure routing accuracy on its own labels.

Adaptive-RAG: a classifier predicts complexity Jeong et al., NAACL 2024

ClassifierT5-size

A simple · LLM only

B moderate · 1 retrieval

C complex · multi-step

Index

Training labels come from the strategy that got the answer right.

Pitfall: a free-text router invents a source name, for example "docs_v2", and the call fails. An enum stops this.

Paper Adaptive-RAG (2403.14403) Paper Self-Route (2407.16833) Course Building Agentic RAG with LlamaIndex

The router is the simplest agentic RAG: one LLM call that returns structured output and picks a source. Step one: "refund policy?" goes to the vector DB of product docs. Step two: "orders last week?" goes to text-to-SQL over the orders database. The enum in the schema stops the model from inventing a source. Adaptive-RAG (Jeong et al., NAACL 2024) adds a second decision: how much retrieval. A small T5-size classifier sends simple questions to the LLM alone, moderate ones to one retrieval, and complex ones to a multi-step IRCoT-style loop; the training labels come from which strategy got the answer right. Analogy: a hospital triage desk sends a cold to the pharmacy and a fracture to surgery. Self-Route (arXiv 2407.16833) uses the same idea to choose between RAG and long context. Likely question: "How do you evaluate a router?" Keep a labeled set of questions with the correct source and measure routing accuracy apart from answer quality.

Self-RAG critiques itself with reflection tokens, and CRAG grades evidence and falls back to web search

Self-RAG Asai et al., arXiv 2310.11511 · one fine-tuned LM

Retrieveretrieve now? ISRELpassage relevant? ISSUPoutput supported? ISUSEoutput useful?

Per segment: one continuation per passage, then keep the best by a weighted critique score. Self-RAG 7B and 13B beat ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning and fact verification.

vs

CRAG Yan et al., arXiv 2401.15884 · plug-in, any generator

EvaluatorT5-large ≈0.77B

Correctmax > upper θ

Ambiguousin between

Incorrectall < lower θ

Refinestrips → filter

Bothrefined + web

Web searchquery → keywords

PopQA RAG → CRAG

RAG

0 CRAG

0

PubHealth generator: Self-RAG-LLaMA2-7B

RAG

0 CRAG

0

Pitfall: an uncalibrated LLM grader marks everything “relevant”. Then the corrective branch never fires. Check the grader against human labels first.

Two ways to check evidence. Self-RAG (Asai et al.) trains one model to write reflection tokens into its own output: Retrieve says whether to fetch, ISREL grades each passage, ISSUP checks that the sentence is supported, ISUSE scores usefulness. The model writes one continuation per passage and keeps the best one by a weighted critique score. CRAG (Yan et al.) leaves the generator alone and adds a T5-large evaluator of about 0.77B parameters: Correct refines the documents into knowledge strips, Incorrect rewrites the query into keywords and searches the web, Ambiguous does both. With the same Self-RAG-LLaMA2-7B generator, CRAG lifts PopQA from 52.8 to 59.8 and PubHealth from 39.0 to 75.6. Analogy: Self-RAG is a writer who learned to fact-check while writing; CRAG is an editor who checks the sources before the writer starts. Likely question: "Which one for a hosted API model?" CRAG, because you cannot fine-tune the hosted model to emit reflection tokens.

Dynamic RAG retrieves during generation: IRCoT, FLARE, DRAGIN and DRAG decide when and what to fetch

Paper IRCoT (arXiv 2212.10509) Paper FLARE (arXiv 2305.06983) Paper DRAGIN (ACL 2024) Repo oneal2000/DRAGIN Paper DRAG / IterDRAG (arXiv 2410.04343)

IRCoT Trivedi et al., ACL 2023
Each new chain-of-thought sentence becomes the next query. Up to +21 points retrieval and +15 points QA on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC.

FLARE Jiang et al., EMNLP 2023
Draft the next sentence. If a token has low confidence, use the draft as the query and regenerate the sentence.

DRAGIN Su et al., ACL 2024
RIND decides when: entropy × attention from later tokens × stopword filter > θ. QFS decides what: the tokens in the whole context that the trigger token attends to most. It needs logits and attention, so it cannot run on a closed API.

DRAG / IterDRAG Yue et al., Google DeepMind, ICLR 2025
DRAG adds many documents and in-context demos. IterDRAG splits the question into sub-queries with interleaved retrieval (max 5 iterations). Up to 58.9% over standard RAG: spend inference compute on retrieval.

Acronym trap: “DRAG” names at least four other papers. Always cite the arXiv ID, not the acronym.

Classic RAG retrieves once before generation. Dynamic RAG retrieves while the model writes. IRCoT uses each new chain-of-thought sentence as the next query, which gave up to 21 points better retrieval and 15 points better QA on four multi-hop sets. FLARE drafts the next sentence and retrieves only when a token has low confidence; the red token is the trigger. DRAGIN makes both decisions from the model's internals: RIND fires on an uncertain, important, non-stopword token, and QFS builds the query from the context tokens that this token attends to, not only from the last sentence. That is why DRAGIN needs logits and attention weights, which hosted APIs do not return. DRAG and IterDRAG from Google DeepMind show that RAG accuracy grows nearly linearly with effective context when you allocate compute well: up to 58.9 percent over standard RAG. Analogy: a student who checks the textbook each time she is unsure, not once at the start. Likely question: "Which of these works with Gemini or GPT through an API?" IRCoT and IterDRAG yes, FLARE only if the API returns token log-probabilities, DRAGIN no.

LangGraph builds the corrective loop as a graph: generate, retrieve, grade, rewrite

START

generatequery_or_respond

END

retrieveToolNode

grade_documentsstructured yes / no

rewritequestion

generateanswer → END

pass 1 grade: no

Add a max-iterations counter to the state. Log each grade.
Use a cheaper, different model as grader. Calibrate it.
Put graph.get_graph().draw_mermaid() in the docs.

Pitfalls: infinite loops, and query drift after many rewrites.

Docs Build a custom RAG agent Repo rag-from-scratch Video RAG From Scratch playlist

agentic_rag.py · langgraph 1.2.16 · langchain 1.4.7


@tool
def retrieve_blog_posts(query: str) -> str:
    """Search and return information about Lilian Weng blog posts."""
    return "\n\n".join(d.page_content for d in _get_retriever().invoke(query))
class GradeDocuments(BaseModel):
    binary_score: str = Field(description="'yes' if relevant, 'no' if not")
def grade_documents(state) -> Literal["generate_answer", "rewrite_question"]:
    q, ctx = state["messages"][0].content, state["messages"][-1].content
    msg = GRADE_PROMPT.format(question=q, context=ctx)
    r = response_model.with_structured_output(GradeDocuments).invoke(
        [{"role": "user", "content": msg}])
    return "generate_answer" if r.binary_score == "yes" else "rewrite_question"
wf = StateGraph(MessagesState)
wf.add_node(generate_query_or_respond)
wf.add_node("retrieve", ToolNode([retrieve_blog_posts]))
wf.add_node(rewrite_question); wf.add_node(generate_answer)
wf.add_edge(START, "generate_query_or_respond")
wf.add_conditional_edges("generate_query_or_respond",
    lambda s: "tools" if s["messages"][-1].tool_calls else END,
    {"tools": "retrieve", END: END})
wf.add_conditional_edges("retrieve", grade_documents)   # CRAG-style gate
wf.add_edge("rewrite_question", "generate_query_or_respond")  # loop
wf.add_edge("generate_answer", END); graph = wf.compile()

This is the LangGraph docs example "Build a custom RAG agent", trimmed. Step one: the retriever is a plain function with the @tool decorator, so the LLM in generate_query_or_respond can choose to call it or to answer at once; that is the router. Step two: the grader uses with_structured_output (green wire = yes, red wire = no) and a Pydantic model, so it can only say yes or no, and grade_documents returns the name of the next node. Step three: the conditional edges make the graph a CRAG-style gate after retrieval. Step four: the rewrite edge goes back to generate, which closes the corrective loop; watch the token take the second pass. Analogy: a quality gate on a factory line that sends a bad part back to rework. Likely question: "What stops this loop?" Nothing in this sample; in production add an iteration counter to the state and route to a "not found" answer when it hits the cap.

Interview: designing agentic RAG

Design a support assistant that sometimes needs SQL, sometimes docs and sometimes nothing.

Put an enum router first: vector | sql | web | none with structured output, so it cannot invent a source. Add an Adaptive-RAG complexity gate: simple questions skip retrieval, complex ones get a multi-step loop. Trace every route and evaluate routing accuracy separately from answer quality.

How do you stop an agentic RAG loop?

Use four stops: a max_iterations cap in the state, a sufficiency grader that ends the loop when the evidence is enough, a token or time budget, and a fallback answer that says “not found” and cites what was searched. Log each pass to detect query drift.

Self-RAG or CRAG for a closed API model?

CRAG. It wraps any generator with a separate retrieval evaluator (T5-large, about 0.77B) and a web-search fallback. Self-RAG needs a fine-tuned model that emits reflection tokens (Retrieve, ISREL, ISSUP, ISUSE). You cannot train those into a hosted model.

Why can DRAGIN not run on a hosted API?

RIND needs token entropy from the logits and the attention that later tokens give each token. QFS needs the self-attention weights over the whole context. Closed APIs do not return attention weights. Use IRCoT or IterDRAG instead: they need only text.

Paper Corrective Retrieval Augmented Generation (arXiv 2401.15884) Paper Self-RAG (arXiv 2310.11511) Paper DRAGIN (ACL 2024)

06

BrowseComp accuracy, 1,266 hard-to-find, easy-to-verify questions (OpenAI, Apr 2025)

GPT-4o

0.6%

GPT-4o + browsing

1.9%

o1 (reasoning)

9.9%

Deep Research

51.5%

Human trainers 29.2% · gave up on 70.8% after 2 h

Research agents, RL search and MCP

Deep research agents spend many tokens on many searches to answer one hard question.

ADK agentic RAG Deep research architecture RL-trained search agents MCP as the retrieval interface

“browsing alone is not sufficient—models must also be capable of reasoning strategically, identifying relevant search paths, and interpreting retrieved content.”

Paper BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (arXiv 2504.12516)

BrowseComp has 1,266 questions. Each answer is hard to find but easy to check, so the score is exact. Press through the bars: GPT-4o alone gets 0.6%, browsing lifts it only to 1.9%, the reasoning model o1 gets 9.9%, and Deep Research gets 51.5%. The dashed line is the human baseline: trainers solved 29.2% and gave up on 70.8% after two hours. Analogy: a detective case where every clue leads to the next clue; a search box alone does not solve it, the plan does. This chapter shows how to build that plan: ADK workflow agents, the deep-research architecture, RL-trained searchers and MCP as the standard retrieval port. Likely question: "Is Deep Research better than people?" On this benchmark and with a two-hour human budget, yes; the benchmark was made so that people find the answers hard to locate.

In Google ADK, retrieval is a tool the agent chooses to call, and loops are workflow agents

SequentialAgent · AgenticRAG

Initial retrieverLlmAgent → evidence

LoopAgent · max_iterations=4

Grader→ critique

Researcher→ evidence

RAG corpusAgentTool(kb)

Web searchAgentTool(web)

Writercited answer

Docs ADK Knowledge Engine Repo google/adk-recipes Codelab Build a Sports Shop Agent AI Assistant with ADK, MCP Toolbox and AlloyDB Codelab Build a Patent Search App with AlloyDB, vector search, and Java Agent Development Kit

rag_tool.py · google-adk 2.13.0


import os
from google.adk.agents import Agent
from google.adk.tools.agent_tool import AgentTool
from google.adk.tools.retrieval.vertex_ai_rag_retrieval import (
    VertexAiRagRetrieval)
from vertexai.preview import rag

ask_vertex_retrieval = VertexAiRagRetrieval(
    name="retrieve_rag_documentation",
    description="Retrieve documentation from the RAG corpus.",
    rag_resources=[rag.RagResource(
        rag_corpus=os.environ.get("RAG_CORPUS"))],
    similarity_top_k=10, vector_distance_threshold=0.6)

kb_agent = Agent(model="gemini-flash-latest", name="rag_agent",
    instruction="Fetch docs with the tool, then answer from them.",
    tools=[ask_vertex_retrieval])     # the tool stays alone here
kb_tool = AgentTool(agent=kb_agent)  # mixes with other tools

corrective_loop.py · adk.dev/agents/workflow-agents/loop-agents


from google.adk.agents import LlmAgent, LoopAgent, SequentialAgent
from google.adk.tools.tool_context import ToolContext

def exit_loop(tool_context: ToolContext):
    """Call only when the evidence is sufficient."""
    tool_context.actions.escalate = True   # stops the LoopAgent
    return {}
grader = LlmAgent(name="Grader", model="gemini-2.5-flash",
    include_contents="none", output_key="critique",
    instruction="Judge if {evidence} fully answers {question}. "
                "Reply 'SUFFICIENT' or list gaps.")
researcher = LlmAgent(name="Researcher", model="gemini-2.5-flash",
    output_key="evidence", tools=[exit_loop, kb_tool, web_tool],
    instruction="If {critique} is 'SUFFICIENT' call exit_loop; "
                "else search to fill the gaps.")
loop = LoopAgent(name="CorrectiveLoop",
    sub_agents=[grader, researcher], max_iterations=4)
pipeline = SequentialAgent(name="AgenticRAG",
    sub_agents=[initial_retriever, loop, writer])

Built-in RAG and search tools work only alone in an agent. Wrap each in a sub-agent; call it with AgentTool.

Start with the tool. VertexAiRagRetrieval reads a RAG Engine corpus; the ADK docs now call it Knowledge Engine, the cloud docs still say RAG Engine. The model decides whether to call the tool, so this is agentic RAG in about 20 lines. The docs say the built-in retrieval tool and google_search can only be used by themselves in one agent, so wrap each in a sub-agent and expose it with AgentTool; then one researcher can use both, which is a multi-source router. The loop is a LoopAgent with max_iterations=4: the grader writes "critique" to session state, the researcher writes "evidence", and exit_loop sets escalate=True to leave the loop early; the SequentialAgent then runs the writer. Analogy: an editor and a reporter who pass one shared notebook until the editor says "sufficient", with a hard limit of four rounds. Interview angle: workflow agents make the loop testable and capped, because the control flow is code, not a prompt.

Deep research plans, fans out, compresses and writes, with citations at every step

Blog How we built our multi-agent research system Repo assafelovic/gpt-researcher Docs Gemini Deep Research agent Repo langchain-ai/open_deep_research (archived)

OpenAI deep research (Feb 2025): 26.6% on Humanity's Last Exam. API o3-deep-research / o4-mini-deep-research with web_search_preview, file_search (max 2 vector stores), code_interpreter. Shut down 2026-07-23; replacement: gpt-5.6-sol.
Gemini Deep Research (Interactions API): deep-research-preview-04-2026 and a -max- variant; 1,048,576 input / 65,536 output tokens; MCP servers and File Search.
Open source: GPT Researcher (~30.3k stars) has an MCP retriever. open_deep_research got RACE 0.4943 with GPT-5 on Deep Research Bench, at $20–$100 per 100 tasks. Archived 2026-08-21; successor: Deep Agents.

gemini_deep_research.py · Interactions API


import time
from google import genai
client = genai.Client()
interaction = client.interactions.create(
    input="Research the history of Google TPUs.",
    agent="deep-research-preview-04-2026",
    background=True)            # a long job: poll it
while interaction.status not in ("completed", "failed"):
    time.sleep(10)
    interaction = client.interactions.get(interaction.id)
print(interaction.steps[-1].content[0].text)  # or .error

Walk the canonical loop top to bottom. Step 1: the planner writes a research brief and splits it into sub-questions; Gemini lets the user edit this plan. Step 2: one researcher sub-agent per sub-question runs search, read, note in its own context window, in parallel; Anthropic's Opus 4 lead with Sonnet 4 sub-agents beat single-agent Opus 4 by 90.2% on an internal eval. Step 3: each lane compresses its notes and keeps the URLs; "the essence of search is compression", and token use alone explains 80% of BrowseComp variance. Step 4: a supervisor looks for gaps and spawns another lane; this is why multi-agent costs about 15x the tokens of chat. Step 5: the writer produces a report with inline citations, then the hosted API returns it from a background job that you poll. Analogy: an editor who sends four reporters out, reads their short memos, sends one more, then writes the story. Likely question: "Can I still call o3-deep-research?" The OpenAI docs list a shutdown on 2026-07-23 with gpt-5.6-sol as the replacement, so check the docs before you build on it.

RL-trained search agents learn when to search from an answer-only reward

+24%Search-R1 7B vs RAG (paper v4/v5)

+20%Search-R1 3B vs RAG, 7 QA sets

+28.9DeepResearcher vs prompt agents (pts)

+7.2DeepResearcher vs RAG-based RL (pts)

Paper Search-R1 (arXiv 2503.09516) Repo PeterGriffinJin/Search-R1

Search-R1 · UIUC, UMass, Google Cloud AI Research

The policy writes <search> when it needs evidence. BM25, E5 dense or web search returns <information>.
The policy picks how many hops to make.
Tokens in <information> are masked from the loss. The model does not learn to write evidence.
Reward = exact match of the answer, plus a format check. PPO, GRPO or REINFORCE on veRL.

R1-Searcher (arXiv 2503.05592): two-stage outcome RL. Stage 1 rewards correct search calls and format; stage 2 rewards the answer. No SFT cold start.
ReSearch (Baichuan, arXiv 2503.19470, NeurIPS 2025): search inside the reasoning chain; it generalizes across datasets. DeepResearcher (arXiv 2504.03160) trains on the live open web.

Search-R1 README quick start


$ conda activate retriever
$ bash retrieval_launch.sh  # E5 over Wikipedia
$ conda activate searchr1
$ python infer.py           # edit question, line 7

Read the rollout as a transcript the policy writes. It thinks, emits a search, gets passages back from the environment, thinks again and makes a second hop, then answers. Step 3 is the key design choice: the retrieved tokens are masked from the loss, so the gradient trains the decisions and the reasoning, not imitation of Wikipedia text. Step 4: the only reward is exact match on the final answer plus a format check, and GRPO or PPO updates the policy; nobody wrote a rule for when to search. Analogy: you grade a student only on the final answer of an open-book exam, and the student learns by itself when to open the book. Quote the numbers with the paper version: v4/v5 report +24% for the 7B model and +20% for 3B over RAG baselines; v1 said +41%. Likely question: "What can go wrong with the reward?" A format reward can be hacked: the policy can learn to emit valid tags without good answers, so keep exact match as the main signal and read sampled rollouts.

MCP makes retrieval a standard search-and-fetch interface that any agent can call

ADK agentMcpToolset

LangGraph / Claudeany MCP client

Gemini Deep ResearchMCP servers

OpenAI deep researchneeds search + fetch

MCP serversearch · fetch

untrusted output

Knowledge baseAzure AI Search KB

MCP ToolboxSQL databases

Graph retrieveHippoRAG MCP tool

MCP spec 2026-07-28 (prior revision 2025-11-25)

Stateless: no initialize handshake; version and capabilities ride in _meta.
Servers MUST implement server/discover.
No Mcp-Session-Id in Streamable HTTP.
subscriptions/listen replaces the GET stream.
Multi Round-Trip Requests replace server-side sampling and elicitation.
tools/list carries ttlMs and cacheScope, in a fixed order. Roots, Sampling, Logging: deprecated.

kb_agent.py · ADK McpToolset


from google.adk.agents import LlmAgent
from google.adk.tools.mcp_tool import McpToolset
from google.adk.tools.mcp_tool.mcp_session_manager import (
    StreamableHTTPConnectionParams)

kb = McpToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://kb.example.com/mcp", timeout=5),
    tool_filter=["search", "fetch"],  # retrieval surface only
)
root_agent = LlmAgent(model="gemini-flash-latest",
    name="kb_agent", tools=[kb],
    instruction="Search the KB. Cite the fetched docs.")

A retrieval server is now a stateless, cacheable HTTP service. A search cursor travels as an explicit handle in the tool arguments.

Docs MCP spec changelog Docs ADK MCP tools Repo googleapis/mcp-toolbox (ex genai-toolbox)

MCP turns a corpus into a tool server. ADK, LangGraph, Claude, ChatGPT and Gemini Deep Research all call the same search and fetch tools, so you write no glue per framework. OpenAI deep research accepts an mcp tool only if the server implements search and fetch and require_approval is "never"; that makes search plus fetch a de facto retrieval contract. The 2026-07-28 spec removes the initialize handshake and the session id, so each request stands alone, and tools/list is cacheable and comes in a fixed order, which helps prompt caching. Analogy: a USB-C port for knowledge; any laptop plugs into any drive. In ADK, use tool_filter to expose only search and fetch, not every tool on the server. Likely question: "Is MCP output safe to put in the prompt?" No. Tool output is untrusted input; chapter 9 covers indirect prompt injection.

Interview: research agents and RL-trained search

deep researchSearch-R1MCPtoken budget

Multi-agent research costs about 15× the tokens of chat. When is it worth it?

Use it for broad questions that split into parallel parts, where token spend drives quality: token use explains 80% of BrowseComp variance. Do not use it for simple lookups. Cap the number of sub-agents and set a token budget.

Why does Search-R1 mask the retrieved tokens?

The loss must train decisions and reasoning: when to search, what to query, how to answer. The retrieved text is written by the environment, not the policy. Without the mask, the model learns to generate fake evidence.

Design a deep-research feature with a fixed budget.

Plan with a brief the user can edit. Run N capped sub-agents. Compress each lane to 1–2k tokens with URLs. Let a supervisor check for gaps. Write a cited report, then run a critic pass. Trace the token spend per lane.

What did the 2026-07-28 MCP spec change, and why does it help RAG?

Requests are stateless, and servers implement server/discover. There is no session id. tools/list is deterministic and cacheable. So a retrieval server scales like a plain HTTP service, and the tool list stays in the prompt cache.

07

Graph and structured RAG

Top-k similarity finds the chunks that look like the query. Two question shapes break it. Structure adds aggregation and traversal.

Global sensemaking

"What are the main themes across this 1M-token corpus?"

The answer is spread thin over all 100 chunks. The top-5 window (blue) sees 5% of the evidence. No single chunk holds the answer.

Multi-hop

"Which Stanford professor's student founded the company that acquired X?"

Only chunk C shares words with the query. Chunks A and B do not look like the query, so top-k never returns them.

entity graphcommunity hierarchy summary treerelational schema

Recent studies "report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks."

Paper When to use Graphs in RAG (GraphRAG-Bench, ICLR 2026)

This chapter is about questions that plain top-k similarity cannot answer. First, global questions: the answer is a summary of the whole corpus, so every chunk is a little relevant and no five chunks are enough. Second, multi-hop questions: each hop is in a different chunk, and the middle chunks do not look like the question. Analogy: top-k is a search light; global questions need a map of the whole room, multi-hop questions need a rope from one object to the next. Structured RAG builds that map or that rope at index time: an entity graph, communities, a summary tree or a SQL schema. Then end with the caution: GraphRAG-Bench shows that graphs often lose to vanilla RAG, so measure before you adopt. Likely question: "Can I not just raise k?" Larger k adds cost and distractors and still misses chunks that are not similar to the query.

GraphRAG extracts an entity graph, clusters it with Leiden and summarizes every community

Query mode	How it works	Result
global (default)	map-reduce over reports, level 2; dynamic selection: −77% tokens	72–83% win vs vector RAG; C0 reports: "over 97% fewer tokens"
local	nearest entities → edges, text units, reports	"Who is X, how is X related to Y?"
DRIFT	primer over top-K reports, then local follow-ups	beats local: 78% comprehensiveness, 81% diversity
basic	vector top-k over text units	baseline

Compose TextUnits. Default 1200 tokens (paper: 600, overlap 100).
Extract. An LLM finds entities and relationships per chunk. FastGraphRAG uses NLP.
Summarize. An LLM merges each entity description over all mentions.
Cluster. Hierarchical Leiden makes levels C0 (few, big) to C3 (many, small). No LLM.
Report. An LLM writes a report per community per level. Then embed. Output: parquet.

terminal · getting started


$ python -m pip install graphrag  # 3.3.0 (8 Oct 2026)
$ graphrag init                   # .env, settings.yaml, input/
$ curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt \
    -o ./input/book.txt           # GRAPHRAG_API_KEY in .env
$ graphrag index                  # --method standard | fast
$ graphrag query "What are the top themes in this story?"
$ graphrag query --method local \
    "Who is Scrooge and what are his main relationships?"
$ graphrag query --method drift \
    "How does Scrooge change, and why?"

Cost is set by corpus size, not query count. Each chunk, entity and report is an LLM call: about 1000x vector indexing. The repo is "largely in maintenance mode".

Paper From Local to Global Repo microsoft/graphrag Docs Default Dataflow Blog DRIFT Search

Walk the index pipeline one phase at a time and watch the LLM-call counter. Chunks first; then an LLM reads every chunk and pulls out entities and relationships; then an LLM merges the descriptions of each entity. Leiden is plain graph maths with no LLM: it finds dense groups, then groups of groups, so you get levels from C0 at the top to C3 at the bottom. Then the expensive part: one LLM report for every community at every level. Analogy: a newsroom that writes a briefing for every desk, every department and the whole paper before anyone asks a question. Global search map-reduces over those briefings; local search starts from entities; DRIFT mixes the two. Likely question: "Why is it so expensive?" Because every LLM phase scales with the corpus, which is about 1000x vector indexing, and the project is now in maintenance mode.

LazyGraphRAG and LightRAG cut graph cost by deferring LLM work or using one call

LazyGraphRAG: no LLM at index time

It "requires no prior summarization of the source data". Index: a noun-phrase co-occurrence graph; graph statistics make communities.

Query: 3–5 subqueries → best-first ranking → LLM relevance tests, breadth-first → deepen → stop at the budget → map-reduce claims.

5,590 AP articles, LLM judge. At 4% of GraphRAG global query cost, it beats all methods on local and global queries. Ships in Microsoft Discovery (Azure).

Blog LazyGraphRAG: a new standard for quality and cost

LightRAG: one keyword call, dual-level retrieval

EMNLP 2025pip install lightrag-hkuRAG-Anything (2026.05)

0tokens, 1 call. GraphRAG global: 610,000

0win vs GraphRAG (49.6–54.8%)

0win vs naive RAG

Query

LLM keywordsone call

Entity nodes"Scrooge"

Relation edges"redemption"

Context+1-hop, chunks

Index: LLM extracts entities and relations → profiles key-value pairs → dedupes. Update = graph union, no community rebuild.

examples/lightrag_openai_demo.py (trimmed)


from lightrag import LightRAG, QueryParam
from lightrag.llm.openai import gpt_4o_mini_complete, openai_embed

rag = LightRAG(working_dir="./rag_storage", embedding_func=openai_embed,
               llm_model_func=gpt_4o_mini_complete)
await rag.initialize_storages()        # inside async def main()
await rag.ainsert(f.read())            # incremental: graph union
for mode in ["naive", "local", "global", "hybrid", "mix"]:
    print(mode, await rag.aquery("What are the top themes in this story?",
                                 param=QueryParam(mode=mode)))
await rag.finalize_storages()

Paper LightRAG (EMNLP 2025) Repo HKUDS/LightRAG (40.3k stars)

Both systems attack the same problem: full GraphRAG spends LLM calls at index time on every chunk and every community. LazyGraphRAG moves that work to query time: the index is cheap NLP co-occurrence, the same cost as vector RAG and 0.1% of GraphRAG, and the LLM only tests chunks that look promising, until a budget runs out. That budget is the one knob: Z100_Lite costs the same as an 8k vector RAG call, and more budget gives more quality. LightRAG keeps an LLM-built graph but drops community reports: one LLM call makes low-level keywords that hit entity nodes and high-level keywords that hit relation edges, and an update is a graph union. Analogy: GraphRAG pre-cooks every meal; LazyGraphRAG cooks to order; LightRAG keeps a well-labelled pantry. Likely question: "Is LightRAG better than GraphRAG?" On LLM-judge win rates it is about even, near 50%, at a small fraction of the query tokens; the next slide shows that F1 tells a different story.

HippoRAG 2 spreads activation with PageRank, and RAPTOR retrieves from a tree of summaries

Bergum: "You don't need a graph database for GraphRAG." Han et al. (arXiv 2502.11371): combining both paradigms gives consistent gains.

HippoRAG 2: PageRank over a KG as memory

Offline: OpenIE triples → phrase nodes, synonym edges (similarity > 0.8), passage nodes linked by "contains".
Online: query → top-5 triples → LLM recognition filter → seeds: kept phrases + all passages (weight × 0.05).
Personalized PageRank spreads activation from the seeds. Passages rank by score → reader.

RAPTOR clusters chunks by similarity; GraphRAG clusters entities by connectivity. Collapsed tree, GPT-4: +20% on QuALITY.

LLM-judge win rate

~50%LightRAG vs GraphRAG (49.6–54.8%), judged on breadth.

Avg F1, 7 QA sets

HippoRAG 2

59.8

dense

57.0

GraphRAG

49.6

RAPTOR

48.8

LightRAG

6.6

Paper HippoRAG 2 Repo HippoRAG Paper RAPTOR Repo raptor

HippoRAG models the hippocampus: the LLM is the cortex, the knowledge graph is the index, and Personalized PageRank is pattern completion. Watch the heat: the query picks triples, an LLM drops the bad ones, the kept phrases become seeds, and PageRank spreads activation along edges until passages two hops away light up. That is associative recall: the passage about Rockland County never matched the query words, but it wins through Montebello. RAPTOR takes the other road: it clusters chunks by embedding similarity, writes a summary per cluster and repeats, so long documents get a tree of summaries; search all levels at once. Then the metric trap: graph papers report LLM-judge win rates, HippoRAG 2 reports F1, and LightRAG goes from about 50% to 6.6 depending on the metric. Likely question: "Which number do I trust?" The one that matches your task; for factual QA, F1 against a strong dense baseline.

When the answer lives in tables or a graph database, retrieve the schema and generate the query

Question

Schema retrieverDDL + column docs

Few-shot SQLsimilar Q→SQL

LLMwrite SQL

Executedatabase

Verbalizeanswer

(1) link the schema with embeddings (2) retrieve few-shot pairs (3) generate (4) execute (5) self-correct on error (6) verbalize the rows.

BIRD test execution accuracy

leaderboard, 23 Oct 2026

GrainSQL

82.95

human

92.96

Spider 2.0: o1-preview agent

632 tasks, BigQuery/Snowflake, >1,000 columns

enterprise

21.3%

Spider 1.0

91.2%

Which structure for which question?

Question shape	Use
Specific facts	hybrid search + reranker first
Multi-hop entity chains	HippoRAG 2, or vector → Cypher expansion
Whole-corpus themes	GraphRAG global search, LazyGraphRAG
Corpus changes daily	LazyGraphRAG, LightRAG (incremental)
Long single documents	RAPTOR tree
Numbers in tables	text-to-SQL agent; graph DB: Text2Cypher
Mixed	agentic router over vector, graph and SQL tools

Codelab Generate SQL using AlloyDB AI natural language Bench BIRD Paper Spider 2.0 Paper LinkedIn KG-RAG (SIGIR 2024)

neo4j-graphrag 1.22.0 · Text2CypherRetriever


from neo4j_graphrag.retrievers import Text2CypherRetriever

neo4j_schema = """
Node properties: Person {name: STRING, born: INTEGER};
  Movie {title: STRING, released: INTEGER}
Relationship types: (:Person)-[:ACTED_IN]->(:Movie),
  (:Person)-[:DIRECTED]->(:Movie)
"""
examples = ["USER INPUT: 'Who acted in The Matrix?' QUERY: "
    "MATCH (p:Person)-[:ACTED_IN]->(m:Movie {title: 'The Matrix'}) "
    "RETURN p.name"]
retriever = Text2CypherRetriever(driver=driver, llm=llm,
    neo4j_schema=neo4j_schema, examples=examples)
print(retriever.search(
    query_text="Which movies did Hugo Weaving star in?"))

Schema, not documents. The LLM writes Cypher from the schema and examples.
Bad Cypher raises Text2CypherRetrievalError. VectorCypherRetriever: vector entry, Cypher expansion.
Vanna 2.0: row-level security. LinkedIn KG-RAG: MRR +77.6%, resolution time −28.6%.

Some answers are not in prose at all; they are rows in a warehouse or paths in a graph database. Then retrieval changes target: you retrieve the schema and a few similar question-to-SQL examples, the LLM writes a query, the database runs it, and errors go back to the LLM for a retry. Analogy: you do not photocopy the whole library catalogue; you ask the librarian who knows the shelves to fetch the exact rows. The numbers are low: the BIRD leader is at 82.95 against 92.96 for humans, and on Spider 2.0, with enterprise schemas of over 1,000 columns, an o1-preview agent solves 21.3%. The graph version is the same pattern with Cypher, and neo4j-graphrag gives it to you as Text2CypherRetriever. Close with the decision table, which is the one-slide summary of the whole chapter. Likely question: "How do you stop the LLM from dropping a table?" Run with a read-only role and row-level security, and validate the query before execution.

Interview: graph and structured retrieval

GraphRAGLazyGraphRAGLightRAGHippoRAG 2RAPTORtext-to-SQL

When is GraphRAG worth its indexing cost?

When users ask corpus-wide sensemaking questions over a stable corpus: indexing costs about 1000x vector RAG, and global search wins 72–83% on comprehensiveness. For specific facts, use hybrid search + a reranker. For data that changes daily, use LazyGraphRAG (index = vector cost) or LightRAG (graph union updates).

Explain how HippoRAG 2 uses Personalized PageRank.

Offline, OpenIE triples make a KG of phrase and passage nodes, with synonym edges. Online, the query gets top-5 triples; an LLM filter keeps the good ones. Kept phrases and all passages (× 0.05) are the seeds. PPR spreads activation over the graph, and passages rank by their score. Result: associative multi-hop recall, 59.8 avg F1 vs 57.0 for dense.

Why do GraphRAG papers and HippoRAG disagree on LightRAG?

They use different metrics. GraphRAG-family papers use LLM-judge win rates on comprehensiveness and diversity: LightRAG is about 50% vs GraphRAG. HippoRAG 2 uses F1 on factual QA: LightRAG gets 6.6. Pick the metric that matches your task, and always include a strong dense baseline.

Text-to-SQL on a 1,000-column warehouse: your design?

Retrieve the schema, not documents: embed DDL + column docs and link only the relevant tables. Retrieve similar (question, SQL) pairs as few-shot examples. Execute, and send errors back for self-correction. Use a read-only role with row-level security. Why: on Spider 2.0, an o1-preview agent solves only 21.3%.

08

Page images

Video clips

Code

User history

Retrieverone loop

LLManswer

Visual documents, from text to pixels

Parse then text: Docling makes Markdown.
Caption then embed: VLM text, multi-vector retriever.
Embed the page image: ColPali or Gemini Embedding 2.

Code needs exact identifiers, AST structure and a fresh index.

Memory is retrieval over the user's own past sessions.

Multimodal, code and memory

The retrieval loop stays the same. The index changes.

ColPali embeds the page image and matches query tokens to image patches with MaxSim

Index size, 4,874 NQ passages

Dense MiniLM 7.5 MB

LateOn fp32 311.5 MB

+ fast-plaid 92 MB

+ pool_factor=4 78.8 MB

Multi-vector costs about 40x a dense vector before compression. colpali-engine is deprecated for new projects; v6 needs transformers v5.

One vector instead: gemini-embedding-2 puts text, image, video (up to 120 s), audio and PDF (up to 6 pages) in one space, 128 to 3072 dims.

Video: cut 10–120 s clips, embed them, index them with timestamps, and cite the timestamp. VideoRAG uses LongerVideos: more than 160 videos, 134 h.

Paper VideoRAG, arXiv 2502.01549

colqwen_search.py · sentence-transformers v6


# pip install -U "sentence-transformers[image]"
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("vidore/colqwen2.5-v0.2")
BASE = ("https://huggingface.co/datasets/"
        "sentence-transformers/example-documents/resolve/main/")
pages = [BASE + "doc1.jpg", BASE + "doc2.jpg"]   # page images

# many patch vectors per page image, 128 dims each
doc_emb = model.encode_document(pages)
# one vector per query token
q_emb = model.encode_query(
    ["Total outlay is maximum in which year?"])
# MaxSim: max over patches, then sum over query tokens
scores = model.similarity(q_emb, doc_emb)

ViDoRe nDCG@5: colpali 81.3 · colpali-v1.3 84.8 · colqwen2-v1.0 89.3 · colqwen2.5-v0.2 89.4 · tomoro-colqwen3-embed-4b 90.6 · colqwen3.5-4.5B-v3 90.9 · colSmol-256M 80.1

Paper ColPali, ICLR 2025 Repo illuin-tech/colpali Blog Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

ColPali (Faysse et al., ICLR 2025) skips OCR, layout parsing and chunking. It renders each PDF page as an image, and PaliGemma-3B gives about 1,000 patch vectors of 128 dims per page. The query becomes about 20 token vectors. Step 1 fills the token-by-patch similarity matrix; step 2 keeps the best patch per token; step 3 sums those maxima into the page score. This is ColBERT MaxSim applied to pixels. Step 4 shows the similarity map: it tells you where on the page the match is, which is good for debugging and for showing evidence. The Hugging Face blog reports that ColPali beats pipelines in which Claude Sonnet captions every visual element. Step 5 is the cost: multi-vector storage is about 40 times a dense vector until you use PLAID-style indexes and token pooling. Analogy: a dense vector is a one-line summary of the page; multi-vector keeps a sticky note on every patch. Likely question: "Why not just use one multimodal vector?" Answer: cheaper and fits any vector DB, but you lose token-to-patch detail on dense pages; a common design is single-vector recall plus ColPali reranking.

Code retrieval combines grep, symbols and embeddings, and re-indexes only what changed

grep / ripgrepexact identifiers, always fresh

LSP / symbol graphdefinitions, references

Embedding index"where is the auth logic?"

Coding agent looppicks a tool per step

4.03 hp99 time to first query
was 4.03 h

2.82 minp90 time to first query
was 2.82 min

7.87 sp50 time to first query
was 7.87 s

Code has exact names, syntax trees, cross-file links and constant change

cAST chunks along the AST: split large nodes, merge small siblings under a size budget. +4.3 Recall@5 on RepoEval, +2.67 Pass@1 on SWE-bench.

Cursor: re-index only what changed

A Merkle tree of file hashes finds the changed files. 50,000 files give about 3.2 MB of hashes.
One changed file changes every hash up to the root. Compare roots, then walk down.
Split changed files into syntactic chunks. Embed them asynchronously. Cache embeddings by chunk content.
A simhash seeds a new user's index from a teammate's (92% average similarity). Content proofs return only files the client has.

"Grep generally yields higher accuracy than vector retrieval" on 116 LongMemEval questions across Claude Code, Codex, Gemini CLI and a custom harness. The result depends strongly on the harness. Practice in 2026: grep for names, LSP for precision, embeddings for concepts, an agent loop on top. Streaming RAG uses the same delta idea: Pathway re-embeds only new data.

Blog Securely indexing large codebases Repo yilinjz/astchunk Paper cAST Paper Is Grep All You Need?

Code breaks three assumptions of text RAG: the user often searches for an exact name, the structure is a syntax tree, and the corpus changes every few minutes. cAST (Findings of EMNLP 2025) chunks along the AST and gains 4.3 Recall@5 on RepoEval. Steps 1 to 3 follow Cursor's January 2026 blog: a Merkle tree finds the changed files, only their syntactic chunks are re-embedded, and embeddings are cached by content, so a teammate's clone seeds yours through a simhash. Step 4 shows the result: p99 time to first query fell from 4.03 hours to 21 seconds. Step 5 is the 2026 debate: a May 2026 study found grep usually beat vector retrieval inside coding-agent harnesses, but the harness mattered a lot, so the practical answer is a hybrid toolset. Analogy: Git does not re-copy the repository on every commit; it only stores what changed. Likely question: "Does Claude Code use a vector DB?" Answer: say only what the published study shows, and design for grep plus symbols plus embeddings.

Agent memory is retrieval over the user's own history: extract, consolidate, retrieve, inject

Conversationsession events

ExtractLLM finds facts

ConsolidateADD / UPDATE / DELETE

Memory storescope: user_id

Retrievesimilarity, top_k

Injectinto the prompt

Graphiti: "Facts have validity windows. When information changes, old facts are invalidated — not deleted."

agent.py · ADK + Memory Bank


from google import adk
from google.adk.agents import Agent
from google.adk.memory import VertexAiMemoryBankService
from google.adk.tools import preload_memory

memory_service = VertexAiMemoryBankService(
    project="PROJECT_ID", location="LOCATION",
    agent_engine_id="AGENT_ENGINE_ID")
async def auto_save(callback_context):
    await callback_context.add_session_to_memory()

agent = Agent(model=MODEL, name="Generic_QA_Agent",
    instruction="Answer the user's questions",
    tools=[preload_memory],      # or load_memory: agent decides
    after_agent_callback=auto_save)
runner = adk.Runner(agent=agent, app_name="app",
    session_service=session_service,
    memory_service=memory_service)

preload_memory retrieves on every turn. load_memory is a tool the agent calls when it needs it. InMemoryMemoryService() is for prototypes.

Working: context window, session state · Episodic: past sessions · Semantic: facts, preferences · Procedural: learned instructions

LongMemEval (ICLR 2025): 500 questions, 5 abilities (extraction, multi-session, temporal reasoning, knowledge updates, abstention). Assistants lose about 30% accuracy over sustained use. LaMP (ACL 2024) retrieves from the user profile to personalize.

Letta / MemGPTVirtual context management. The agent edits its own in-context memory blocks. Mem0LLM picks ADD / UPDATE / DELETE / NOOP. LOCOMO: +26% vs OpenAI memory, 91% lower p95 latency, >90% fewer tokens. Zep / GraphitiTemporal knowledge graph. DMR 94.8% vs MemGPT 93.4%. LongMemEval up to +18.5%. Memory BankFormerly Vertex AI Agent Engine Memory Bank. Scoped memories, Gemini extraction and consolidation, IAM Conditions, VPC-SC, CMEK.

Scope every read and write by user_id, or facts leak between users. Invalidate stale preferences. Memories are personal data.

Docs ADK Memory Docs Memory Bank overview Repo getzep/graphiti Paper Mem0

RAG retrieves from a shared corpus. Memory retrieves from one user's own history. Steps 1 to 3 build the lifecycle: an LLM extracts candidate facts from the conversation, a consolidation step decides ADD, UPDATE, DELETE or NOOP against similar memories (this is Mem0's design), the store is scoped by user_id, and the agent retrieves and injects the top memories into the prompt. Step 4 is the Graphiti idea: when Alice moves from Acme to Google, the old edge is invalidated but kept, so the question about 2025 still has an answer. Steps 5 and 6 show the ADK version with Memory Bank: preload_memory retrieves every turn, and an after_agent_callback saves the session. Analogy: a good assistant keeps a notebook about you, crosses out old facts with a date, and never reads another client's notebook. Likely question: "Why not keep the whole history in the context window?" Answer: LongMemEval shows about a 30% accuracy drop over long interactions, and Mem0 reports over 90% token savings versus full context.

Interview: multimodal, code and memory retrieval

You must index 10M scanned pages. ColPali or parse-then-text?

ColPali wins on table and chart pages, but it stores about 40x more than one dense vector (311.5 MB vs 7.5 MB on 4,874 passages). Compress with PLAID and token pooling (78.8 MB). Or use a single-vector multimodal embedder for recall and ColPali to rerank the top pages.

Why can grep beat a vector index for a coding agent?

Exact identifiers, always fresh, no index to sync. A 2026 study found grep usually more accurate across Claude Code, Codex and Gemini CLI, but the harness changed the result. Keep embeddings for concept queries such as "where is the auth logic?".

The user says "I moved to Berlin." Design the memory update.

Extract the fact. Find conflicting memories for this user_id. Invalidate the old city with a validity window (Graphiti) or choose UPDATE (Mem0). Keep the history, so "where did I live last year?" still works. Never write outside the user's scope.

Single-vector vs multi-vector retrieval: what is the trade-off?

One vector per item is cheap and fits any vector DB. Gemini Embedding 2 also crosses modalities. Multi-vector keeps token-level detail (about 1,000 patch vectors per page) and wins on dense pages, at a higher cost in storage and compute.

09

Query

Retrieverhybrid + rerank

Contexttop-k chunks

LLMwrites answer

Answer+ citations

Documentsweb · email · PDF

Indexvectors + ACL metadata

Tools / MCPsend · fetch · write

1 · wrong chunks 2 · no support 3 · off-question injection · poisoning ACL leak · inversion exfiltration

Evaluation and security

A RAG system is as good as its measured retrieval and as safe as its least trusted document.

Fails in three places: retrieval, grounding, usefulness. Score each one separately.

Is attacked in three places: the documents, the index, the tools.

“LLM-Integrated Applications blur the line between data and instructions.”

Paper Greshake et al. 2023, Not what you've signed up for

This chapter answers two questions: how do you know the system works, and how do you stop it from being turned against its users. Step 1: three failure points pulse. Retrieval returns the wrong chunks, the model writes claims the context does not support, or the answer is grounded but does not answer the question. Step 2: three attack points appear. Attackers write into documents, read across the index without permission, and use tools to send data out. Step 3: the Greshake quote is the root cause: the model reads retrieved text and instructions in one token stream. Analogy: a new employee who obeys every sticky note in the filing cabinet. Likely question: "Is security not a separate team's job?" Answer: no, because the retriever decides what text the model reads, so the RAG engineer owns this boundary.

Score retrieval, grounding and answer relevance separately: the RAG triad

TruLens RAG triad: one metric per edge

Groundedness splits the answer into claims and looks for evidence for each claim. TruLens 2.16.0 traces on OpenTelemetry.

Docs TruLens, RAG Triad

Retrieval metrics need relevance labels

Recall@k: relevant chunks in the top-k. Main RAG metric.
Precision@k: relevant chunks in the top-k, divided by k.
MRR: mean of 1/rank of the first relevant hit.
nDCG@10: graded and rank-discounted. MTEB/BEIR headline.
Hit rate / Match@k: any relevant chunk in the top-k.

Faithfulness = supported claims / all claims. Here 3 of 4 claims have evidence: 0.75.

Context Precision@K is rank-aware: swap a relevant and an irrelevant chunk and ContextUtilization drops from ≈1.0 to ≈0.5. No LLM: IDBasedContextPrecision (2/4 = 0.5). Cheap judge: FaithfulnesswithHHEM (HHEM-2.1-Open, CPU). LangSmith adds correctness, which needs a reference answer.

faithfulness.py · ragas 0.4.3 collections API


from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
faith = Faithfulness(llm=llm)
result = await faith.ascore(
    user_input="When was the first super bowl?",
    response="The first superbowl was held on Jan 15, 1967",
    retrieved_contexts=[
        "The First AFL–NFL World Championship Game was an "
        "American football game played on January 15, 1967, "
        "at the LA Memorial Coliseum."])
print(result.value)          # 1.0

The old ragas.metrics + evaluate() API is removed in v1.0.

One end-to-end score hides which part failed.

Repo vibrantlabsai/ragas Docs LangSmith RAG eval

Calibrate LLM judges and keep a private golden set

Judge the judge: >80% agreement

Paper Zheng et al. 2023

GPT-4 judge ↔ humans

>80%

human ↔ human

≈ same

position biasverbosity biasself-enhancement bias

Recipe: binary labels with a rationale, structured output, swapped answer order, a judge from another model family, 100–300 human labels.

ARES: trained judges + PPI

Paper ARES, NAACL 2024

Synthetic triples train small judges. PPI corrects their bias and gives confidence intervals from “only a few hundred human annotations”. RAGAS 0.4.3 adds DSPy judge-prompt tuning.

2018HotpotQA113k multi-hop QA pairs

2020KILTmany tasks, one Wikipedia snapshot

2021BEIR18 datasets, zero-shot retrieval

2024 · COLMMultiHop-RAG2,556 queries over 2–4 articles

2024 · NAACL 25FRAMES824 questions, 2–15 articles each

2025 · ICLRMMTEB500+ tasks, 250+ languages

2025BrowseComp1,266 Qs; Deep Research 51.5%

FRAMES: multi-step retrieval

no retrieval

0

multi-step retrieval

0

Accuracy on 824 questions (Google, NAACL 2025).

Paper FRAMES

Public benchmarks leak into pretraining data. “Models that ace BEIR benchmarks often fail at real RAG tasks.” (Nandan Thakur)

Docs Stop Saying RAG Is Dead

Golden set: 100–500 private Q / A / context-ID triples. Make them with RAGAS TestsetGenerator or ARES. Add labelled production traces.

agentic RAG eval · ADK


$ adk eval <AGENT> <EVAL_SET> \
    --config_file_path=test_config.json

Criteria: tool_trajectory_avg_score, response_match_score, hallucinations_v1, rubric-based quality.

Docs ADK, Evaluate

Step 1: an LLM judge is useful because strong judges agree with humans over 80% of the time, which is the level humans reach with each other. The same paper names three biases: position, verbosity and self-enhancement, so swap the answer order and use a judge from another model family. Step 2: ARES trains small judges on synthetic data and uses Prediction-Powered Inference; watch the confidence band get narrow as human labels come in. Step 3: the benchmark line moves from multi-hop QA to zero-shot retrieval to real RAG pipelines and agentic browsing. Step 4: on FRAMES, multi-step retrieval lifts accuracy from 0.40 to 0.66. Step 5: public sets leak into pretraining, so your private golden set is the number you trust. Analogy: a public exam paper that circulated before the exam. Likely question: "How big must the golden set be?" Start with 100 to 500 items and grow it from failed production traces.

Retrieved text is untrusted input, so injection and poisoning attack through the index

Indirect prompt injection

The attacker writes into a page, email, PDF or ticket that your retriever reads. A normal query pulls it into the prompt.

Paper Greshake 2023 Paper PoisonedRAG 2025

0PoisonedRAG success: five texts per question, millions of documents

Retrieval condition: close to the question, so it ranks in the top-k.
Generation condition: it states the attacker's answer.

OWASP: a résumé hides white-on-white text. The model obeys it. Data leaves through an image URL or a tool call.

Docs OWASP LLM08:2025

1 · Control what enters the index

Trusted sources only, with provenance and signatures. Review before ingestion. Detect duplicates and anomalies. Keep immutable retrieval logs.

2 · Mark the data, split control from data

0attack success with Spotlighting, from >50%

0AgentDojo tasks, provably secure (CaMeL); 84% undefended

Paper Spotlighting, Microsoft Paper CaMeL, Google DeepMind

3 · Limit what the agent can do

Least-privilege tools. Human confirmation for side effects. No auto-rendered external URLs or images. Egress filters. NeMo Guardrails retrieval rails.

MCP tool output is the same untrusted input.

This slide replays one attack, then replays it with defences. Step 1: the attacker plants cv_17.pdf in the corpus. Step 2: a normal user query retrieves it, and it rises to rank 1 because PoisonedRAG texts are written to be close to the target question; five texts per question give 90% success. Step 3: the hidden text tells the model what to do, and data leaves through a tool call and a markdown image URL. Steps 4 to 6 add defence layers. Assume each layer can fail, so stack them: provenance at ingestion, Spotlighting and CaMeL at the model, least privilege and egress filters at the tools. Analogy: a letter in the in-tray that says "the boss told you to wire money". Likely question: "Can a better system prompt stop this?" No. Prompt text is not a security boundary; CaMeL gives provable guarantees because the plan comes only from the trusted query.

Filter by the end user's permissions inside the query, before ranking, and redact PII at ingestion

Embeddings drop the source document's ACL. Top-k returns what is similar, not what this user may see. The retriever becomes a confused deputy.

Docs OWASP LLM08: “permission-aware vector stores”

Index time: copy the source ACL onto every chunk as metadata. Use group ids. Deny overrides grant.
Query time: pass the end user's identity, not a service account. Filter inside the vector and BM25 queries, before ranking and fusion.
Re-sync chunk ACLs when permissions change.
Partition semantic caches and memories the same way.

Post-filtering can return fewer than k results and leaks through counts. Never let an LLM write tenant or ACL filters.

acl_and_pii.py


# illustrative Chroma-style filter: END USER's groups
hits = collection.query(query_texts=[q], n_results=5,
    where={"acl_groups": {"$in": user.groups}})

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

text = "My phone number is 212-555-5555"
results = AnalyzerEngine().analyze(
    text=text, entities=["PHONE_NUMBER"], language="en")
print(AnonymizerEngine().anonymize(
    text=text, analyzer_results=results))

PII: redact or pseudonymize at ingest, before embedding, because embeddings can be inverted (OWASP LLM08). Redact again at output. Right-to-erasure deletes must reach vector stores, caches and traces.

Products: OpenSearch and Elasticsearch document-level security. Glean and Microsoft 365 Copilot advertise permission-trimmed retrieval. Red-team: user A asks for user B's data; PII probes; poisoning canaries.

Docs Presidio, Getting started (text)

Two users send the same query. Step 1: without a filter, both lists contain "HR salaries", so user B from sales sees a document he may not open. Step 2: the permission filter moves inside the retriever query. B's restricted chunk is removed before ranking, and the list still has k = 4 results; a post-filter would show only 3 and leak through the count. Step 3: the same leak through a semantic cache: B's similar question hits A's cached answer, until the cache is partitioned by user. Steps 4 and 5: the filter in code, then Presidio removes PII before embedding. Analogy: a library that shows every book title in the catalogue but keeps the restricted shelf locked; the catalogue must be filtered too. Likely question: "Can the LLM decide which filter to apply?" No. Identity and ACL filters come from the authenticated session, never from model output.

Interview: evaluating and securing RAG

evaluationprompt injectionACL filtersLLM judges

Faithfulness is 0.95, but users still complain. What is wrong?

Faithfulness checks support, not usefulness or completeness. A grounded answer can miss the question. Measure answer relevance and retrieval recall@k on a golden set. Then check the judge against human labels for position and verbosity bias.

Your agent reads email and can send email. How do you stop indirect injection?

Treat retrieved text as data: mark it with Spotlighting. Split control from data as in CaMeL: the plan comes only from the user's query. Give tools least privilege. Ask for human confirmation before a send. Filter egress and never auto-render external URLs or images.

Pre-filter or post-filter for ACLs?

Pre-filter inside the ANN and BM25 queries, with the end user's identity, before ranking and fusion. A post-filter can return fewer than k results and leaks through counts. Partition caches and memories by the same identity.

How many human labels do you need to trust an LLM judge?

A few hundred, with Prediction-Powered Inference to get confidence intervals (ARES). Start with 100–300 labels. Recalibrate when the judge model, the prompt or the data changes.

10

Latency budget

Embed query → ANN search → rerank → LLM prefill (grows with context) → decode.

Cost budget

Tokens in context → reranker calls → agent loops: agents use ~4x, multi-agent systems ~15x the tokens of chat.

Freshness promise

Source changes → index lags → cache keeps the old answer. Zombie chunks answer from deleted or changed documents.

Production, managed RAG and best practices

Every answer spends time, tokens and trust. This chapter covers tracing and caching, incremental indexing, managed RAG services, a best-practice checklist and open challenges.

A demo answers one question. A product answers a million questions, and each answer spends three budgets: time, money and trust in freshness. Press three times: each gauge fills segment by segment and the needle passes its budget mark. The segment widths are illustrative, not measurements; the order is the point. Latency: prefill grows with the context, so the number of chunks you send drives time. Cost: tokens in context and agent loops dominate; Anthropic reports about 4x the tokens of chat for agents and about 15x for multi-agent systems. Freshness: a "zombie chunk" is a chunk whose source document was edited or deleted, but the chunk still answers. Analogy: a restaurant menu on the web that still shows last year's prices. Likely question: "Which budget breaks first?" Usually latency on long contexts and cost on agent loops; freshness breaks silently, so you need an SLO for it.

Trace every hop, cache what repeats and re-embed only what changed

Docs RedisVL LLM Caching Docs Prompt caching Docs LangChain index() Docs Langfuse Python SDK Docs Phoenix tracing quickstart

Trace on OpenTelemetry, track these KPIs

Langfuse 5.0.4 @observe · Phoenix 20.26.0 register(auto_instrument=True) · LangSmith @traceable

p50 / p95 per spanTTFTtokens + $ / querycache hit rate hit@k (golden set)groundedness drift"I don't know" ratethumbs-down

Five levers for latency and cost

Retrieve less: rerank the top-50 to the top-5.
Prompt cache: static prefix first; reads cost 0.1x (Anthropic).
Semantic cache: ~0.05 s hit vs ~1.8 s LLM call.
Route and cascade: small model first, escalate on low confidence.
Batch APIs for indexing and offline eval.

Freshness: stop zombie chunks

index() + RecordManager hashes each chunk and skips unchanged ones. Cleanup: incremental · full · scoped_full. Never point it at a store that another process filled. New embedding model = full blue/green re-embed.

semantic_cache.py · RedisVL


from redisvl.extensions.cache.llm import SemanticCache
from redisvl.utils.vectorize import HFTextVectorizer
from redisvl.query.filter import Tag

cache = SemanticCache(name="llmcache",
    redis_url="redis://localhost:6379",
    distance_threshold=0.1,     # Redis COSINE 0-2, lower = stricter
    vectorizer=HFTextVectorizer("redis/langcache-embed-v1"),
    filterable_fields=[{"name": "user_id", "type": "tag"}])
def answer(q: str, uid: str) -> str:
    hit = cache.check(prompt=q,
                      filter_expression=Tag("user_id") == uid)
    if hit:                               # ~0.05 s instead of ~1.8 s
        return hit[0]["response"]
    out = rag_answer(q)                   # retrieve, rerank, generate
    cache.store(prompt=q, response=out, filters={"user_id": uid})
    cache.set_ttl(3600)                   # freshness bound
    return out

Partition by user_id or tenant, or user A gets user B's answer. At distance_threshold=0.5, a "Nice" question hit the cached "Paris" answer. Tune the threshold with CacheThresholdOptimizer.

Step 1 draws one request as a trace. Every hop is a span, and each span records what you need to debug it: the retriever span keeps the top-k ids, scores, the filter and the index version. Step 2 turns on prompt caching and a semantic cache. The prefill bar shrinks because the static prefix is read from cache at 0.1x price; a repeated question returns in about 0.05 s instead of 1.8 s. The bar widths are illustrative; the 0.05 s and 1.8 s values come from the RedisVL docs demo. Step 3 is freshness: hash every chunk, re-embed only the changed ones, write tombstones for deleted ones, and invalidate cache entries tagged with that doc_id. Steps 4 to 6 walk the RedisVL code: the partition field, the filtered check, and the TTL. Analogy: a semantic cache is a librarian who remembers recent questions; without a partition, the librarian tells you someone else's answer. Likely question: "Why not cache by exact string?" Exact match misses paraphrases; a semantic cache catches them, but it needs a strict threshold.

Managed RAG services now parse, chunk, embed, store and cite for you

you manageGoogle manages (shares illustrative)

Docs Perform semantic search and retrieval-augmented generation Codelab Build a patent search app using AlloyDB, vector search, and Gemini Enterprise Agent Platform (formerly Vertex AI) Codelab AlloyDB AI Operators and Reranking Docs RAG quickstart Codelab Building a Google-quality Search System with Vertex AI Docs Gemini API File Search

Service	Primitive · API	Oct 2026 facts
Gemini API File Search	`FileSearchStore` + `file_search` tool (`google-genai` 2.33.0)	Storage and query-time embeddings free. Indexing at embedding rates ($0.15 / 1M tokens at launch). 100 MB per file; store 1 GB (Free) to 1 TB (Tier 3); keep stores under 20 GB. `metadata_filter`, `file_citation`.
Agent Platform RAG Engine	`client.rag.create_corpus / import_files / retrieve_contexts` (`google-cloud-agentplatform` 2.6.0)	Chunking 512 / 100, `text-embedding-005`. Bring your own vector DB (Pinecone, Vector Search). Agent Search backend. ADK tool.
OpenAI file_search	`vector_stores` + Responses API tool	Semantic + keyword search. `max_num_results`, `filters`, `include=["file_search_call.results"]`.
Amazon Bedrock Knowledge Bases	`retrieve` · `retrieve_and_generate`	`overrideSearchType: HYBRID`, rerankers, GraphRAG on Neptune Analytics.
Azure AI Search agentic retrieval	knowledge base `retrieve` or MCP endpoint	GA in REST 2026-04-01. LLM query planning, parallel subqueries, semantic rerank. 2,000 retrievals ≈ $4.32. Powers Foundry IQ.

Renamed in 2026: Vertex AI → Gemini Enterprise Agent Platform (22 Apr 2026) · Agent Engine → Agent Runtime · Vertex AI Search → Agent Search · google/adk-samples → google/adk-recipes · genai-toolbox → mcp-toolbox. Do not mix the old vertexai.rag style and the new agentplatform client in one project.

file_search.py · Gemini API, Interactions API


import time
from google import genai                       # google-genai 2.33.0
client = genai.Client()
store = client.file_search_stores.create(config={
    "display_name": "my-store",
    "embedding_model": "models/gemini-embedding-2"})   # images need -2

op = client.file_search_stores.upload_to_file_search_store(
    file="sample.txt", file_search_store_name=store.name,
    config={"display_name": "display-file-name"})
while not op.done:                     # service parses, chunks, embeds
    time.sleep(5); op = client.operations.get(op)

interaction = client.interactions.create(
    model="gemini-3.8-flash", input="Can you tell me about ...",
    tools=[{"type": "file_search", "file_search_store_names": [store.name]}])
for step in interaction.steps:
    if step.type == "model_output":
        for b in step.content:
            if b.type == "text":
                print(b.text, [a.file_name for a in (b.annotations or [])
                               if a.type == "file_citation"])

Step 1 builds the Google ladder from the bottom: BigQuery VECTOR_SEARCH, where you write one SQL statement and own every choice, up to Gemini File Search, where Google parses, chunks, embeds, stores and cites. The shading shows who manages the pipeline; the shares are illustrative. Step 2 compares the five managed services across clouds. Point at File Search pricing: storage and query embeddings are free, and you pay embedding rates only at indexing time. Point at Azure: agentic retrieval plans subqueries with an LLM and costs about $4.32 for 2,000 retrievals in the docs example. The rename callout matters in practice: in April 2026 Vertex AI became Gemini Enterprise Agent Platform, so old tutorials use old names and the old vertexai.rag SDK. Steps 3 to 6 walk the File Search code: create a store, upload and poll, call the file_search tool, then read the file_citation annotations. Analogy: managed RAG is a hotel; DIY RAG is your own house. The hotel is ready tonight, but you cannot move the walls. Likely question: "When would you leave File Search?" When you need custom chunking, hybrid weights, a specific reranker or ACL logic it does not expose.

Best practice in Oct 2026: measure first, retrieve hybrid, rerank, and add agents only where they pay

Golden set100–500 Q/A/IDs

CI gateRAGAS · adk eval

Ship + traceOTel spans

Online judges+ user feedback

Human labelsrecalibrate judge

Red-teaminjection · ACL

eval-driven lifecycle
failures become tests

Open challenges

Agentic + deep-research evalScarce agent trajectories Multi-agent token costInjection via tools / MCP When graphs pay offContext rot at 1M+ tokens

The survey also lists coordination, memory, efficiency and governance. GraphRAG-Bench asks when graphs beat vanilla RAG.

Paper Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (v4) Paper Data-Centric Perspectives on Agentic RAG: A Survey Report Context Rot

01Measure first. Build a 100–500 item golden set. Gate every merge in CI (RAGAS, LangSmith experiments, adk eval).
02Parse well. Use a layout parser for the bulk and a VLM tier for hard pages.
03Chunk simply. Start with ~200-token recursive or structure-aware chunks. Add Contextual Retrieval.
04Retrieve hybrid. Run BM25 + dense search and fuse the lists with RRF (k = 60).
05Rerank. Cut ~150 candidates to ~20. Send few chunks. Put the best chunks first and last.
06Skip the index when you can. Under ~200k static tokens, cache the whole corpus in the prompt.
07Workflows before agents. Use a router, Adaptive-RAG or a corrective loop with max_iterations. Budget multi-agent search at ~15x tokens.
08Load context just in time. Keep knowledge, instructions (skills) and tool context behind identifiers. Load them when a step needs them.
09Trust nothing retrieved. Treat retrieved text and MCP tool output as data. Filter ACLs inside the query. Remove PII at ingest.
10Operate it. Trace on OpenTelemetry. Set freshness SLOs. Re-embed blue/green.

Learn next

Video RAG From Scratch (LangChain) Course Retrieval Augmented Generation (RAG) (DeepLearning.AI) Repo NirDiamant/RAG_Techniques (30.0k stars) Series Stop Saying RAG Is Dead (Hamel Husain) Video Building the Document Context Layer for AI Agents (Jerry Liu)

This is the one slide to photograph. Step 1 is the lifecycle: a golden set, a CI gate, shipping with traces, online judges plus user feedback, human labels that recalibrate the judge, and red-teaming that feeds new cases back into the golden set. Step 2 is the ordered checklist. The order matters: measurement comes first, because every later choice (chunk size, hybrid weights, reranker, agents) is a guess without it. Item 7 answers the "workflows" question: a workflow is a fixed code path with LLM steps, so it is cheaper to test, trace and cap than an autonomous agent. Item 8 answers the "skills" question: load instructions and knowledge just in time by identifier, the same principle as retrieval. Step 3 lists what is still open in October 2026: how to evaluate agentic and deep-research systems, the lack of agent trajectory data, the token cost of multi-agent search, injection through tools and MCP, when graphs pay off, and context rot at very long inputs. Step 4 gives the best next resources. Analogy: the checklist is a pilot's pre-flight list; skipping item 1 is flying without instruments. Likely question: "If I can do only three things this week?" Build the golden set, add hybrid search with RRF, and add a reranker.

Interview: shipping RAG at scale

system designlatency + costindex migrationcaching risks

Design RAG for 50M documents, p95 under 1 second, with per-user ACLs.

Run hybrid BM25 + ANN (HNSW or DiskANN; MRL 768 dims + int8). Apply the ACL pre-filter inside both queries with the end user's identity. Fuse with RRF. Rerank the top-100 with a small, fast reranker. Cache the static prompt prefix. Stream the output. Give each span a latency budget and trace it.

Build your own pipeline, or use File Search / RAG Engine?

Use managed for speed, citations and low ops. Build DIY when you need custom chunking, hybrid weights, rerankers, ACL logic or a non-Google stack. Measure both on the same golden set before you decide.

How do you migrate to a new embedding model with zero downtime?

Blue/green. Build the new index in parallel. Compare both indexes on the golden set. Switch reads when the new index wins. Keep the old index for rollback. Never mix vectors from two models in one index.

What are the risks of a semantic cache?

Cross-user leakage: partition by user or tenant. False hits: tune the threshold; at 0.5 a "Nice" question hit the "Paris" answer (RedisVL docs). Stale answers: set a TTL and invalidate entries by doc_id when a document changes.

Ask the students to answer each question aloud before you reveal the model answer. Q1 is a classic system-design question. A strong answer names the budget per span, puts the ACL filter inside the ANN and BM25 queries (not after generation), and keeps the reranker small because it runs on every query. Q2 tests judgment, not loyalty to a vendor: the right answer is "measure both on the same golden set". Q3 tests whether the candidate knows that vectors from two models live in different spaces. Analogy: you cannot compare a GPS coordinate with a street address; you must re-encode everything. Q4 has three distinct risks; a strong candidate names all three and a fix for each. Likely follow-up: "How do you pick the threshold?" Use a labelled set of paraphrase and non-paraphrase pairs and an optimizer such as RedisVL CacheThresholdOptimizer.