RAG Evaluator

RAG Evaluator checks the answers of an assistant that replies from your documents (a RAG). A language model acts as a judge and gives each answer several scores from 0 to 1, each with a sentence that explains it:

Faithfulness (faithfulness): how much of what the answer states is backed by the pieces the search found. It also lists the statements that are not (unsupported_claims): that is where whatever the assistant made up shows.
Answer relevance (answer_relevance): whether the answer replies to what was asked.
Context relevance (context_relevance): whether the pieces found contain what is needed to answer. A low score means the problem is in the search, not in the model.
Correctness (correctness): if you give the right answer, whether the answer matches it.

You can grade a single answer inside a flow or a whole list of test questions, and get the averages. That way you measure your assistant without writing code and compare, with the same questions, which setup answers better.

When to use it

Before publishing a document assistant, to check with 20 or 30 real questions that it answers well.
To compare two setups (another piece size, more results, reordering with Local Rerank, another model) with the same questions.
To find made-up answers: the statements that do not come from your documents are listed.
To know whether a failure comes from the search (it did not find the piece) or from the model (it had it and answered badly).

Configuration

Field	What it does
Question	The question that was asked. Not needed if you connect Rows.
Answer	The answer to grade, for example an agent’s. Not needed if you connect Rows.
Context	The search results the answer was written from. Without them, faithfulness and context relevance are not scored.
Reference Answer	Optional. The right answer. If you give it, the correctness score is added.
Rows	Optional. A list of rows to grade at once, each with the question, the answer and, if you have them, the pieces found and the right answer. With rows connected, the four fields above are ignored.
Judge Model (required)	The model that grades. It can be different from the one that answered. Use a capable model with temperature 0, so the scores do not change from one run to the next.

Advanced options

Field	What it does
Question Field	The field of each row with the question. Default: `question`.
Answer Field	The field of each row with the answer. Default: `answer`.
Context Field	The field of each row with the pieces found: a text or a list of texts. Default: `context`.
Reference Field	The field of each row with the right answer. Default: `reference`.
Instructions	Extra criteria for the judge, for example what counts as a complete answer in your business.
Parallel Requests	How many answers are graded at the same time, from 1 to 20. Default: 5.

Output

Scores: the scores of the answer, with their explanations and the unsupported statements. If you connected Rows, the average of each score and how many rows were graded (evaluated).
Graded Rows: one row per answer with the question, the answer, its scores and their explanations. Connect it to Data to CSV to review them in a spreadsheet.

Both outputs share the same grading: the judge is asked once per answer even if you use both.

How to read the correctness score: 1 means the answer gives the same as the right one. Below 0.9, something is missing, extra or wrong; the explanation says what. A score of 0 is usually a different answer or an “I don’t know”.

Example

A coffee machine repair service wants to know whether its assistant answers well before handing it to the technicians.

Prepare a sheet with four columns: question (real questions from the technicians), answer (what the assistant replied), context (the pieces it found) and reference (the right answer, written by a senior technician). Export it to CSV.
Load it with CSV to Data List and connect its output to Rows of RAG Evaluator.
Connect a model to Judge Model.
Run the flow. Scores gives the averages: if average faithfulness is high and average correctness lower, the assistant does not make things up but fails some questions. Graded Rows shows which ones, and the context relevance of those rows tells whether it was because the search did not find the piece.

You can also grade each answer inside the flow itself: connect the user’s text to Question, the model’s response to Answer and the Results output of the same search to Context.

How reliable it is: measured figures

Measured on 17 October 2026 on a test knowledge base (9 documents of a fictional repair service), with a low-cost model as the judge. These figures come from a single test set: they show how it behaves, not a promise for your data.

Correctness: on 90 answers to hard questions, already checked one by one with strict rules, a score of 0.9 or more matched that check in 89 of 90.
Made-up answers: on 25 answers where one fact had been changed (a number, a code, a name), faithfulness dropped below 1 in 24, and in 23 the changed statement appeared in the list of unsupported ones. The one that slipped through was an answer that only said it could not reply. Of 34 correct answers, only 1 got a faithfulness below 1.
Context relevance: it averaged 0.96 when the search had found the piece with the answer and 0.22 when it had not, or the documents had no answer.
Cost: one judge request per answer; 90 answers, graded 8 at a time, took about 17 seconds.

Good practices

Write the right answers for your test questions: the correctness score is the one closest to a person’s opinion.
Look at the three scores together. High faithfulness with low correctness usually means the search brought the wrong piece and the model repeated it faithfully; context relevance confirms it.
Use the same judge and the same questions when you compare setups, and repeat the evaluation if the differences are small: the judge can vary a little from one run to another.
Do not put it on every production answer: it adds one model request per answer. Use it with test questions or a sample.

Video

This video scores one answer inside a flow and then a set of test questions, including an answer altered on purpose.

What you see in the video:

After an agent, RAG Evaluator scores the answer to “Which part is JT-777 and how long does it take to arrive?”: faithfulness 1.0, answer relevance 1.0 and context relevance 1.0, each score with its reason.
A test set of 4 rows (question, answer, chunks and right answer), read from a JSON file in storage and connected to Rows.
The averages: faithfulness 0.875 and correctness 0.875.
The answer altered on purpose (“the first half hour of labour is free”) scores 0.5, and its unsupported claim is listed.

Local Rerank: reorders the search results; measure with RAG Evaluator whether it improves your answers.
Contextual Text Splitter: prepares pieces that are found more easily.
RAG Agent: an agent that answers from your documents.
CSV to Data List and Data to CSV: to load the test questions and save the scores.

To see how it fits in a complete system, see Retrieval techniques in the RAG guide.