RAG agent

The RAG Agent is an agent specialised in answering from your documents. For every question it does the same thing:

Understands the question. It completes a follow-up (“and how much is it?”) with the conversation, and if the question mentions a value your documents carry (a brand, a month), it prepares the filter.
Searches every connected retriever.
Scores the documents found, all in a single call, and keeps the useful ones.
Answers from those documents only, citing where each fact comes from: [1], [2]… and at the end the list of Sources with the file name and page.
Checks that the answer invents nothing and answers what was asked. If something fails, it searches once more with other words.

If no document is useful, it searches once more with other words and, if it still finds nothing, says it does not have that information, in the language of the question, without making anything up.

This video builds a RAG Agent over the documentation of a coffee-machine repair service and asks it a question, a follow-up and one the documentation does not answer.

What you see in the video:

Qdrant in Retriever mode, with 5 results, connected to Retrievers of the RAG Agent.
The options: Additional Instructions, Metadata Schema with brand, type, month and codes, the three checks (Enable Document Relevance Grading, Enable Hallucination Check and Enable Answer Quality Check), Show Sources and Use Memory.
Asked what error E07 means on the Vellani Aurora, it answers blocked flowmeter, part VL-1180 [1], and a real case [2], and lists the sources at the end: the Vellani Aurora manual and the July incident report.
Asked how long that part takes to arrive, memory lets it understand it is the VL-1180, and it answers five to seven working days.
Asked for the opening hours of the shop in Málaga, it says it cannot find that information.
Each question takes about four model calls and about five seconds.

How it did: the 40-question test

We measured it on the same knowledge base as the comparison: the documentation of the fictional Cafés del Sur repair service (manuals, warranty, maintenance guide, three incident reports and FAQ) and its 40 questions, 4 of which the documentation does not answer. DeepSeek deepseek-flash model, FastEmbed jina-embeddings-v2-base-es vectors in FAISS, 5 results per search. Every answer was checked strictly against the correct answer (17 October 2026).

Setup	Correct (of 36)	Wrong	Says “I don’t know” but it was there	”I don’t know” said right (of 4)	Calls per question	Input tokens per question	Time (median)
RAG Agent, previous version	20	5	10	4	12.8	6,800	9.4 s
RAG Agent, default	23–25	3–6	5–8	4	4.4	2,700	4.9 s
RAG Agent with Metadata Schema	27–28	1–2	6	4	4.3	2,700	4.7 s
RAG Agent without checks	24–25	3–5	5–8	4	3.5	2,200	4.1 s
RAG Agent with Local Rerank	26–27	4	4–5	4	4.4	3,000	6.7 s
Agent with RetrieverTool	24–27	4–5	3–4	4	2.4	9,700	3.0 s

The ranges come from 2 to 4 runs of the same setup: repeating the test moves the result by 1 or 2 answers. The Local Rerank row was measured again on 18 October with its current model, in 2 runs. The remaining answers in each row were partial.

What the numbers say:

Compared with the previous version, the RAG Agent gets more right (from 20 to 23–25), uses a third of the calls (from 12.8 to 4.4) and answers in half the time. Scoring all documents in a single call and searching with the complete question instead of keywords explain almost all of the improvement.
Compared with an Agent with a search tool, it gets the same right and takes about 2 seconds longer, but reads a quarter of the text (2,700 input tokens against 9,700), so each question costs less. It also always cites its sources. Both said “I don’t know” to the 4 questions with no answer.
With a Metadata Schema it gets more right than any other setup. On questions that depend on a brand or a month (“what does E07 mean on the Vellani?”), the filter brings the right documents: 5 of 6 instead of 2 of 6.
Local Rerank helps with exact codes and references: it got all 8 right, at the cost of about 2 more seconds per question on a machine without a graphics card.
The checks did not change the result in this test (24–25 with and without them) and cost one call, under a second. They are a safety net for smaller models or when a wrong answer is expensive.
The Agent is better at counts. It can search several times in a row and got 2 of 6 “how many…?” or “who has…?” questions spread over many documents right; the RAG Agent got none. For those questions, a graph (GraphRAG) or a database is the better tool.

RAG Agent or Agent with a search tool

	RAG Agent	Agent with a search as a tool
What it does with the documents	Scores them and only uses the useful ones.	Reads everything the tool returns.
Answer check	Built in: that it invents nothing and answers the question.	Whatever you ask for in its instructions.
”I can’t find it”	Automatic when no document is useful.	Depends on the instructions.
Sources	Cites `[1]`, `[2]` and lists the files at the end.	Whatever the model decides to mention.
Other tools	No: it only searches your documents (and the internet, if you turn it on).	Can combine the search with a graph, a CRM, email or any component.
Model calls per question	4 (3 without checks).	2 or 3.
When to choose it	Policies, terms, support: when you want cited sources and a reliable “I don’t know” at the lowest cost per question.	Assistants that combine several sources or actions, and questions that need several searches.

You can also have both: connect the RAG Agent to the Tools of an Agent, which will use it as the specialist in your documentation. As a tool it returns the answer with its sources in JSON format.

How to connect it

Connection	What to connect
Retrievers	The Retriever output of one or more vector databases, or of Local Rerank, Ensemble Retriever or Neo4j GraphRAG.
Model	The model that understands the question, scores, answers and checks.
Question	The question, usually from Chat Input.
Web Search Tool	Only if you turn on the web search fallback: a web search tool.

Several retrievers. If you connect several (for example manuals and FAQ), it searches all of them for every question, merges the results and removes duplicates.

How many results. Between 4 and 8 per retriever (Number of Results in the vector database). Scoring is a single call however many documents arrive, but each document makes that call and the answer longer.

What each option does and what it costs

Option	Default	What it does	Calls
Enable Document Relevance Grading	On	Scores the documents found and drops the useless ones. This is what lets it answer “I can’t find it”.	1
Enable Hallucination Check	On	Checks that every fact in the answer is in the documents. If not, it searches again.	1 (shared with the next one)
Enable Answer Quality Check	On	Checks that the answer addresses the question. If not, it searches again.	(the same one)
Show Sources	On	Cites `[1]`, `[2]` in the answer and adds the Sources at the end with the file name or title and page, in the language of the question.	None
Metadata Schema	Empty	Filters the search by the document data the question mentions.	None extra
Use Memory	Off	Completes follow-up questions with the conversation.	None extra
Enable Web Search Fallback	Off	If no document is useful, searches the internet with the tool connected to Web Search Tool.	1 when used
Stream	Off	Shows the answer as it is written; the sources arrive at the end.	None

On top of those, there is always one call to understand the question. A question the documentation does not answer costs 5: understand, score, search with other words, score again and say it cannot find it.

Recommended setups

	Default (recommended)	Fastest
For	Policies, terms, contracts, technical support	Internal FAQ, help chat
Enable Document Relevance Grading	On	On
Enable Hallucination Check	On	Off
Enable Answer Quality Check	On	Off
In the test	23–25 of 36, 4.9 s, 4.4 calls	24–25 of 36, 4.1 s, 3.5 calls

Always keep document scoring on: without it, the model receives everything the search returns, related or not, and loses the ability to say it does not know.

If your documents carry data such as brand, product or date, add the Metadata Schema: in the test it was the biggest improvement. If questions include exact codes or references, connect the Retriever output of Local Rerank: it got all 8 code and reference questions right, at the cost of about 2.5 more seconds per question on a machine without a graphics card.

Instructions, filters and memory

Additional Instructions. This is where you say who it speaks for, the tone and what to say when it cannot find the answer. For example:

You speak for Electrohogar Norte and address customers informally.
If the documentation does not have the answer, suggest writing to soporte@electrohogar-norte.example.

With those instructions, to “Do you have a physical store in Lisbon?” it replied that it could not find that information and suggested writing to support.

Metadata Schema. The same format as Metadata Tagger: a list with the name, type and description of each item. Write the possible values in the description:

[
  {"name": "brand", "type": "string", "description": "Brand of the machine in the manual: Jaltec, Vellani, Molinar"},
  {"name": "month", "type": "string", "description": "Month of the report, format YYYY-MM"}
]

It works with the databases that accept filters, such as FAISS, Chroma DB, Qdrant or PGVector. If the filter finds nothing, the agent searches without it, so a misspelt value does not leave the question unanswered.

Use Memory. Without memory, each question is answered on its own. With memory, the agent completes follow-up questions with the conversation:

Conversation	With memory	Without memory
”How long does standard shipping to the Canary Islands take?” → “and how much is it?"	"Standard shipping to the Canary Islands costs 9.95 euros.”	It found no information about the costs.
”What does the laptop warranty cover?” → “and how long does the battery part last?"	"The laptop battery warranty lasts 1 year.”	It found no information.

A complete example

The FAQ with verification recipe builds a RAG Agent with FAISS, with no server, step by step.