Documentation Evaluations ▾
Evaluations
How retrieval and answers are measured, on what data, and which results exist so far.
Full run · 2026-10-15
All four retrieval configurations and the end-to-end answer evaluation were run on 2026-10-15 with OpenAI text-embedding-3-small, gpt-5.4-mini and Jev 1.13 (Jev served all 44 reranks; the fallback was never needed). Raw rows: evals/results/2026-10-15-openai-*. No number on this page is estimated.
Dataset
- Corpus:
data/synthetic/corpus.jsonl, Slack-style messages from a fictional student team (“Beacon”). Ingestion produces 2,233 message chunks and 158 thread chunks. - Queries:
evals/queries.yaml. 38 answerable and 6 unanswerable (nothing in the corpus answers them). Split into dev (23: 20 answerable, 3 unanswerable) and test (21: 18 answerable, 3 unanswerable). - Evidence groups: each answerable query lists the parts of its answer (for example “who owns it” and “when is it due”), each with every message verified to support that part. 68 groups in total (dev 41, test 27).
- Graded relevance: each query also has a
relevantmap with grades 1 to 3. Grade 1 messages mention the topic but were not verified as supporting the claim.
Configurations
scripts/run_eval.py calls the production functions in retrieval/service.py (retrieve, rerank, answer). It only wraps the API clients to record token usage, raw output, and which rerank model answered. Questions are sent exactly as /ask sends them, with no date or channel filters.
| Config | What runs |
|---|---|
| a) Dense-only | retrieve(q, limit=20, mode="dense") |
| b) Lexical-only | retrieve(q, limit=20, mode="lexical") |
| c) Hybrid RRF | retrieve(q, limit=20) |
| d) Hybrid + rerank | rerank(q, retrieve(q, limit=20)) |
In a and b, chunk scores are still summed per message with 1/(rank+10), as in production. End-to-end answers use config d: the top 12 reranked messages go to answer(), the same evidence /ask uses.
Metrics
Retrieval is scored on the 38 answerable queries; latency is timed on all 44.
| Metric | Definition |
|---|---|
| Recall@k | Per query, the share of its evidence groups with at least one message in the top k; then the mean over queries. 3 groups with 2 hit scores 0.667. |
| Group recall@k (micro) | Groups hit in the top k divided by all 68 groups. Queries with more parts weigh more. |
| Full coverage@k | Share of queries where every group is hit in the top k. |
| MRR@20 | Mean of 1 / rank of the first retrieved message in any evidence group, or 0 if none is in the top 20. |
| Latency p50 / p95 | Wall-clock time of the retrieval call (plus rerank for d), in ms, with linear interpolation. Includes the query embedding request for a, c, d. |
| Citation validity rate | Share of model outputs whose cited IDs are all among the 12 evidence messages. |
| Answer rate | Share of the 38 answerable questions whose final response has at least one verified source. |
| Correct abstention rate | Share of the 6 unanswerable questions whose final response has no sources. |
| Citation precision | Cited IDs that are in the query’s evidence groups, divided by all cited IDs (strict, micro). |
| LLM-judged correctness | gpt-5.4-mini compares the answer with the reference: 1, 0.5, or 0. An LLM opinion, not a human grade. |
Reranking only reorders the top 10, so Recall@10 and Recall@20 are identical for c and d by construction.
Results
Retrieval, 38 answerable queries
| Config | Recall@10 | Recall@20 | MRR@20 | p50 ms | p95 ms |
|---|---|---|---|---|---|
| a) Dense-only pgvector cosine top 30 chunks, no lexical arm | 0.849 | 0.882 | 0.657 | 637.0 | 1188.0 |
| b) Lexical-only ts_rank top 30 chunks over the query’s content words joined with OR | 0.616 | 0.675 | 0.460 | 15.7 | 21.8 |
| b) Lexical-only, before fix Previous websearch_to_tsquery: every word ANDed, stop words kept | 0.000 | 0.000 | 0.000 | 12.9 | 16.5 |
| c) Hybrid RRF Production default: both arms fused with score += 1/(rank+10) | 0.768 | 0.829 | 0.625 | 617.3 | 740.2 |
| d) Hybrid + rerank Decisions API rerank of the top 10 (Jev 1.13 → GPT-6 Luna Decisions → RRF order) | 0.768 | 0.829 | 0.782 | 1425.0 | 2443.7 |
By split
| Config | Split | Queries | Recall@10 | Recall@20 | MRR@20 | p50 ms | p95 ms |
|---|---|---|---|---|---|---|---|
| a) Dense-only | dev | 20 | 0.838 | 0.875 | 0.614 | 623.3 | 1168.7 |
| a) Dense-only | test | 18 | 0.861 | 0.889 | 0.706 | 648.6 | 1172.8 |
| b) Lexical-only | dev | 20 | 0.596 | 0.608 | 0.540 | 16.2 | 21.8 |
| b) Lexical-only | test | 18 | 0.639 | 0.750 | 0.372 | 15.4 | 19.3 |
| c) Hybrid RRF | dev | 20 | 0.733 | 0.775 | 0.681 | 658.7 | 740.3 |
| c) Hybrid RRF | test | 18 | 0.806 | 0.889 | 0.564 | 611.2 | 710.9 |
| d) Hybrid + rerank | dev | 20 | 0.733 | 0.775 | 0.757 | 1445.7 | 2459.4 |
| d) Hybrid + rerank | test | 18 | 0.806 | 0.889 | 0.809 | 1389.4 | 2127.6 |
The test split is the fairer number: the dev split was available while the pipeline was built. Raw rows are in evals/results/2026-10-15-openai-retrieval.jsonl and 2026-10-15-openai-e2e.jsonl.
End-to-end answers (config d)
| Metric | All | Dev | Test |
|---|---|---|---|
| Citation validity rate | 1.000 | 1.000 | 1.000 |
| Answer rate, answerable | 1.000 | 1.000 | 1.000 |
| Correct abstention rate, unanswerable (strict: formal refusal) | 0.167 | 0.333 | 0.000 |
| Unanswerable handled correctly (LLM-judged: says it is not in the chat) | 0.833 | 0.667 | 1.000 |
| Citation precision, strict (micro) | 0.574 | 0.630 | 0.521 |
| Citation precision, lenient (micro) | 0.734 | 0.783 | 0.688 |
| LLM-judged correctness, answerable (mean score) | 0.803 | 0.825 | 0.778 |
Rerank usage: Jev 1.13 answered all 44 rerank requests, so the GPT-6 Luna fallback and the plain-RRF path were never used.
What the numbers say
- Reranking is the biggest win. It moves the right message to the top: MRR@20 rises from 0.625 to 0.782, and the first result is relevant for 74% of questions instead of 53%. It only reorders the top 10, so recall does not change.
- Hybrid does not beat dense-only yet. Dense-only reaches Recall@10 0.849 against 0.768 for hybrid. With equal RRF weights, keyword hits push some good vector hits out of the top 10. Down-weighting the lexical arm, or using it only for exact tokens such as error codes, is the next thing to test.
- Citations are always valid, not always the best ones. Every citation pointed at retrieved evidence, but only 57% (strict) to 73% (lenient) of cited messages are in the hand-labelled evidence.
- Abstention mostly happens in prose. For 5 of the 6 unanswerable questions the answer says the chat does not contain the information, but only one took the formal “I couldn’t verify an answer” path; the rest still attach citations. One answer added an unsupported detail before saying the information was missing.
The lexical fix
Before 2026-10-15 the lexical arm returned zero chunks for all 44 questions. The query went through websearch_to_tsquery('simple', q), which joins every word with AND and keeps stop words, so a chunk had to contain “which”, “the”, and every other word of the question. In effect, “hybrid” retrieval ranked exactly like dense-only.
lexical_query() now drops common English stop words and ORs the remaining terms, while ts_rank still ranks chunks that match more terms higher. On the same 38 answerable questions, with no API calls:
| Lexical-only | Recall@10 | Recall@20 | MRR@20 |
|---|---|---|---|
| Before | 0.000 | 0.000 | 0.000 |
| After | 0.616 | 0.675 | 0.460 |
How the two arms are fused is described in Model architecture.
Limitations
- The data is synthetic: one fictional team, written by an LLM from a planted fact list. Real Slack history is messier.
- 44 queries is small. One query moves Recall by about 2.6 points overall and 5 points within the dev split. Six unanswerable questions give a very coarse abstention rate.
- The queries and gold evidence were written by the same project that built the system. It is not a blind benchmark.
- Gold evidence was verified with gpt-5.4-mini, so labels can contain model errors.
- Generation and reranking are remote, non-deterministic calls. A single run is reported.
- Latency was measured from one laptop against local Docker Postgres and will differ in deployment.
- Filters in the query file are not applied, matching /ask today.
Reproduce
docker compose up -d db
uv run python -m recallhq.cli migrate
uv run python -m recallhq.cli ingest data/synthetic/corpus.jsonl
uv run python scripts/validate_queries.py
# Full run: 4 retrieval configs, end-to-end answers, LLM judge
uv run python scripts/run_eval.py
# Cheaper variants
uv run python scripts/run_eval.py --no-judge
uv run python scripts/run_eval.py --no-e2e
uv run python scripts/run_eval.py --configs lexical --no-e2e # no API callsA full run needs OPENAI_API_KEY with access to text-embedding-3-small and gpt-5.4-mini, and OPEN_ROUTER for reranking. Outputs go to evals/results/<stamp>-*.jsonl and a summary JSON with all metrics and token usage.