Agentic RAG in Production.

I The Retrieval Ceiling

RAG systems hit a ceiling around the third week of production. The ceiling looks like this: the retrieval is working. The right documents come back. The answer is still wrong.

The failure is not retrieval. The failure is that the question required reasoning across the retrieved documents, and retrieval alone does not reason. It returns. The system was never going to solve the harder problem with a better index.

Every RAG system eventually meets this ceiling. The teams that keep tuning retrieval are solving the wrong problem. The teams that notice the ceiling introduce an agent.

II Where the Agent Actually Helps

The agent does four things retrieval cannot:

1. Plans. It decides what to retrieve, in what order, and whether the first retrieval was enough.
2. Reconciles. When two retrieved documents disagree, it picks one or flags both. Retrieval returns both and walks away.
3. Asks. When the retrieval is ambiguous, it asks the user a clarifying question. A retrieval system that cannot ask is guessing.
4. Stops. It knows when the answer is not in the corpus. A retrieval system always returns its best candidate, even when the best candidate is wrong.

Retrieval is not deprecated. It remains the agent’s primary input. But it is no longer the system.

III Hand-Off Patterns

We have shipped two stable hand-off patterns across six agentic RAG installs.

Pattern A: Retrieval first, agent second. The agent calls retrieval, examines the result, and decides whether to call again, reason over what it has, or escalate. Works when the corpus is large and the query space is broad. Used on the Home Inspection RAG install.

Pattern B: Agent first, retrieval second. The agent plans the query, decomposes it, and calls retrieval for each sub-question. Works when the queries are complex and require multi-hop reasoning. Used on a regulated-industry install we cannot name.

Pattern A is cheaper. Pattern B is more capable. The install chooses based on the user-facing latency budget.

IV Production Pitfalls

Three things have bitten us on production agentic RAG.

First, agent loops. An agent can call retrieval seven times when two would do. We cap loops and log the cap. If the cap fires frequently, the planning is broken and the agent needs retraining on decomposition, not retrieval.

Second, citation drift. The agent reasons over retrieved documents, then produces an answer that no longer cites them. The user cannot verify. We constrain the agent to cite the retrieval it used, and fail closed when it cannot.

Third, confidence theatre. The agent says “I don’t have enough information” when it has plenty, because the system prompt rewarded caution. We tune this per install. Too cautious is worse than too confident when the product ships with a human in the loop.

Every one of these pitfalls surfaces in simulation. Never ship an agentic RAG system that has not been simulated against real task patterns from the actual corpus.