How we measured it
The figures in this section come from a single test, run on 17 October 2026 with NappAI’s components. This page explains how it was done, so you know what the numbers mean and how far they go.
The knowledge base
We wrote the documentation of a fictitious company that repairs professional coffee machines, with made-up brands (Jaltec, Vellani and Molinar). It has 9 documents in Spanish, about 8,500 words:
| Documents | What they contain |
|---|---|
| 3 technical manuals | Error codes, parts and procedures of an espresso machine of each brand and of a grinder. |
| 1 warranty and pricing policy | 2026 terms and prices. |
| 1 preventive maintenance guide | What to check and how often. |
| 3 monthly incident reports | July, August and September: 24 incidents, each with its customer, technician, machine and status. |
| 1 FAQ document | 12 customer questions. |
It is deliberately hard: the same error code means different things on each brand (E07 is the temperature probe on one, the flowmeter on another and the motor on the third), many paragraphs do not name their machine, and the three reports share the same structure without stating their month in each section.
The incidents were also loaded into a Neo4j graph: each incident linked to its customer, technician and machine.
The 40 questions
The questions were in Spanish; the examples are translated.
| Type | Questions | Example |
|---|---|---|
| Phrased in other words | 10 | ”How often should the gaskets that seal the portafilter be renewed?” |
| Exact code or reference | 8 | ”Which part is JT-777?” |
| Needs a filter (brand, month, document type) | 6 | ”What does error E07 mean on the Vellani Aurora espresso machine?” |
| The chunk does not say what it is about | 6 | ”What did the September report recommend about the water softeners?” |
| Relationships or counts | 6 | ”How many open incidents is Lucía Ortega assigned?” |
| Not in the documents | 4 | ”What does error E21 mean on the Jaltec Brío 2G?” (that code does not exist) |
What we measured
For each question we knew which sentence of which document answers it. With that we measured:
- First result: the percentage of questions with the right answer in the first result.
- First five: the percentage with the right answer among the first five, which is what an agent usually receives.
- Time of each search and each ingestion, and the model calls each technique adds.
Relationship and counting questions were judged on the final answer: they only counted as right if they were exact (the right number, all the right people and nobody extra). A refusal to answer counted as wrong.
Common setup: 800-character chunks with a 150-character overlap (80 chunks), the default vector model of FastEmbed Embeddings (Local), and DeepSeek, in its fastest version, as the language model.
Main results
Vector databases
| Database | First result | First five | Time per search in a flow |
|---|---|---|---|
| FAISS | 60 % | 83 % | 16 ms |
| Chroma DB | 60 % | 83 % | 16 ms |
| Qdrant | 60 % | 83 % | 36 ms |
| PGVector | 60 % | 83 % | 35 ms |
All four returned the same first five results for all 40 questions. Details in Vector databases.
Search techniques (questions about documents)
| Technique | First result | First five | Cost per question |
|---|---|---|---|
| Basic search (5 results) | 60 % | 83 % | None |
| Hybrid search, equal weights | 53 % | 83 % | None |
| Basic search + Local Rerank | 77 % | 90 % | About 1.2 s |
| Multi Query Retriever | 63 % | 83 % | 1 call, about 1.3 s |
| SelfQueryRetriever with each document’s data | 57 % | 93 % | 1 call, about 1.1 s |
| Contextual chunks, basic search | 50 % | 93 % | None (1 call per chunk at ingestion) |
| Contextual chunks + hybrid + Local Rerank | 80 % | 100 % | About 2 s (1 call per chunk at ingestion) |
The rows with Local Rerank were measured again on 18 October with its current model. The breakdown by question type is in Retrieval techniques.
Vector models
| Model | First five | Time per question |
|---|---|---|
jinaai/jina-embeddings-v2-base-es (default) | 83 % | 10 ms |
intfloat/multilingual-e5-large | 87 % | 30 ms |
Snowflake/snowflake-arctic-embed-l-v2.0 | 83 % | 22 ms (e5-large, measured alongside: 22 ms) |
bge-m3 (Ollama) | 73–80 % | 20 ms |
paraphrase-multilingual-MiniLM-L12-v2 | 63 % | 6 ms |
nomic-embed-text (Ollama) | 53–57 % | 13 ms |
Details in Vectorising and tagging.
Graph versus chunks (6 relationship and counting questions)
| How it searched | Right answers |
|---|---|
| Best chunk combination | 2 of 6 |
| The three whole reports in the instructions | 3 of 6 |
| Neo4j GraphRAG, Vector + Graph | 3 of 6 |
| Neo4j GraphRAG, Text2Cypher without examples | 2 of 6 |
| Neo4j GraphRAG, Text2Cypher with 2 examples | 5 of 6 |
Details in GraphRAG with Neo4j.
What we learned
- The vector database does not change quality. Choose by server, filters and how you work.
- The vector model does matter. The default one is the best balance in Spanish; English-only models fail on questions phrased in other words.
- Context on every chunk is the most cost-effective improvement: you pay once, at ingestion, and together with reranking it gives the best measured combination.
- Reranking fixes exact codes, and its relevance helps drop what is unrelated. A similarity threshold between vectors cannot.
- Tagging needs care: a field that can hold several values must be a list, or filters fail.
- Counting is not searching. Counting questions need a query to a database, with examples.
- Some unanswerable questions are only caught by the agent, such as a code that does not exist on a model that does. That is why the instructions must ask it to say when it does not know.
Limits of the test
- The data is synthetic and small: 9 documents and 40 questions, written by the same person who prepared the right answers. The knowledge base was designed to contain the hard cases, so the results show how each technique behaves, not how often each case appears in real data.
- There are few questions per type (4 to 10). One question moves the overall result by about 3 points, so differences of one or two points are not meaningful.
- Everything ran once, on a laptop shared with other programs. Times are a guide; on your server they will differ. Reranking time in particular depends heavily on the machine’s load: we measured between 1.3 and 2.7 seconds for the same task.
- We did not test every component: Cohere Rerank, Multi Vector Retriever and external providers’ vector models were left out.
Measurement date: 17 October 2026.
Measure your own RAG
The figures in this section come from our test knowledge base. To find out how your assistant answers with your documents, use RAG Evaluator: a model acts as a judge and scores each answer. It tells you whether the answer rests on what the search found, whether it answers the question, whether the search brought what was needed and, if you give the right answer, whether it matches it. Prepare 20 or 30 real questions with their right answers and compare setups with the same questions.
This video scores one answer inside a flow and then a set of test questions, including an answer altered on purpose.
What you see in the video:
- After an agent, RAG Evaluator scores the answer to “Which part is JT-777 and how long does it take to arrive?”: faithfulness 1.0, answer relevance 1.0 and context relevance 1.0, each score with its reason.
- A test set of 4 rows (question, answer, chunks and right answer), read from a JSON file in storage and connected to Rows.
- The averages: faithfulness 0.875 and correctness 0.875.
- The answer altered on purpose (“the first half hour of labour is free”) scores 0.5, and its unsupported claim is listed.