Judge Agent

Judge Agent uses an AI model as a judge: it receives an answer, checks it against the criterion you choose and returns whether it passes or not, together with a comment explaining why. You can use ready-made criteria, such as whether the answer is correct, whether it is concise or whether it makes information up, or write your own.

It is useful to control the quality of other agents’ answers, whether by checking them one by one, assessing batches of tests or inside a Reflection Agent, which asks the main agent to improve its answer until the judge approves it.

When to use it

Before publishing a customer assistant, you want to assess its answers to a list of test questions.
You want to catch answers that include information not found in the documents of your knowledge base.
You need to check that an agent’s answers are short and to the point.
You want to check automatically whether the code an agent produces does what was asked.

Configuration

Field	What it does
Agent Name	The judge’s name. It is used when you connect it as an agent of another component. Default: `ReactAgent`.
Agent Description	What the judge assesses, in one sentence.
Evaluation Prompt Template	The assessment criterion. When you pick one, its text is copied into Custom Judge Prompt, where you can adapt it, and the selector goes back to `Custom`. See Criteria. Default: `Correctness`.
Custom Judge Prompt	The judge’s instructions: what to check and how to decide. Each variable in curly braces, for example `{outputs}`, appears as a new field where you type or connect that piece of data.

Connection	What to connect
Model	The AI model that acts as the judge, for example OpenAI.

Criteria

Criterion	What it checks	Data it asks for
`Correctness`	Whether the answer is correct and complete.	`inputs`, `outputs`, `reference_outputs`
`Conciseness`	Whether the answer is short and to the point.	`inputs`, `outputs`
`Hallucination`	Whether the answer includes made-up information or information the context does not support.	`inputs`, `outputs`, `context`, `reference_outputs`
`RAG Groundedness`	Whether the answer is backed by the retrieved documents.	`context`, `outputs`
`RAG Helpfulness`	Whether the answer really solves the question.	`inputs`, `outputs`
`RAG Retrieval Relevance`	Whether the retrieved documents are related to the question.	`inputs`, `context`
`Code Correctness`	Whether the code in the answer does what was asked.	`inputs`, `outputs`
`Custom`	Whatever you write in Custom Judge Prompt.	The variables you use in the text

In the data, inputs is the question or request, outputs the answer being assessed, reference_outputs a reference answer if you have one and context the supporting documents or data.

Batch processing

With the Batch Mode switch on, the component runs once for each item of the list you connect to Batch Input and returns all the results together as a list. Parallel Executions sets how many items are processed at the same time (1 to 5). Skip if Empty Batch Input skips the component when the list arrives empty, and Skip Branch if Empty Output stops the following components when the result is empty. More in Key concepts.

Using it as an agent tool

You can connect this component to an agent’s Tools input. In the Tool tab, Tool Name and Tool Description are the name and explanation the agent uses to recognise the tool and decide when to call it: write a clear description of what it does.

Output

Eval: the judge’s verdict as text, with the score (score, passed or not) and the comment that justifies it (comment). You can show it in a Chat Output or store it with your test results.
Agent: the judge as an agent, to connect it to the Judge Agent input of a Reflection Agent. There it assesses the main agent’s latest answer against the user’s latest question.
Tool: the judge as a tool, so another agent can use it to check its own answers.

Example

An insurance company wants to check that its policy assistant does not invent coverage.

Choose Hallucination in Evaluation Prompt Template.
Connect the customer’s question to the inputs field, the assistant’s answer to outputs and the policy text to context. Leave reference_outputs empty if you have no reference answer.
Connect an AI model to Model.
Turn on Batch Mode and connect the list of test questions with their answers to Batch Input.

The Eval output says, for each answer, whether it matches the policy and why.

Good practices

Use as the judge a model at least as capable as the one of the agent you assess.
If you write your own criterion, explain clearly when an answer passes and when it does not.
Check a sample of the verdicts by hand before relying on the judge for important decisions.

Reflection Agent: combines an agent and this judge to improve answers.
Agent: NappAI’s general agent.