E3 — External-suite comparison/ablation: Ragas, DeepEval, and an ALCE-style citation P/R metric against sprout's own harness

Status: complete (research deliverable, not a CI change)
Date: 2026-07-09
Author: Chelsea Kelly-Reif (assisted)
Roadmap item: docs/RESEARCH-ROADMAP.md E3 — "benchmark Sprout's grounding-by-construction + deterministic safety against ≥1 of Ragas/DeepEval/ALCE-style citation precision/recall; publish the (saturation-aware) result" (evidence basis EV6, EV7; persona C2, RAG-eval researcher)

TL;DR

The in-house GroundednessSuite (deterministic-lexical judge) scores 1.000 (98/98) on every applicable case in the real, committed eval set. This is not a bug or an artifact of a small sample — it is what ADR-0003 designed: extractive generation + a lexical citation_guard (token-coverage ≥ 0.66) makes "every rendered sentence is lexically covered by its cited chunk" true by construction, and the in-house judge checks a close cousin of that same lexical-coverage property. This metric is saturated. A 1.000 here is expected, not evidence of unusual quality — treat it as "the invariant held," not as "the assistant is unusually good at grounding."
An independent NLI model (ALCE-style citation attribution, substituting a small local DeBERTa-MNLI model for ALCE's original TRUE/T5-11B) scores citation recall/precision at 0.945 / 0.959 over the same 98 items (272 individual cited sentences) — measurably below 1.000, with a 95% Wilson CI of [0.911, 0.966] that does not include 1.0. Manual inspection of every disagreement found the flagged sentences were still verbatim substrings of their cited passage — the disagreement is a known brittleness of small sentence-pair NLI models scoring multi-sentence premises, not a real attribution failure. This is itself the finding: a different, more expensive metric can produce a lower number for a reason that has nothing to do with sprout's actual behavior — exactly the kind of tool-specific artifact a single-metric report would hide.
Ragas' non-LLM context-precision/recall (string-distance only, no model calls) gives precision 0.960, recall 1.000 over the same 98 items — closer to the in-house number because it is, like the in-house judge, a lexical/string-overlap measure, not a semantic one.
Ragas' and DeepEval's LLM-judged metrics (Faithfulness, Context Precision/Recall, Answer Relevancy) were run for real, against a local Ollama model (qwen2.5:3b, temperature 0 — no API key, no network egress beyond localhost, because this repo's eval gate is deliberately offline/zero-cost per ADR-0006/ADR-0009 and none was configured), on a documented 30-item subsample (cost/time-bounded, see Methodology). See §Results for numbers; treat them as directional, not as "what Ragas/DeepEval would report with a GPT-4o-class judge."
Recommendation: keep the in-house DeterministicJudge as the fast, free, CI-blocking default — it is doing exactly what ADR-0003/0006 designed it to do, and this benchmark did not find a case where it passed something the deeper tools would call ungrounded. Do not add Ragas/DeepEval to the CI gate (no evidence it catches anything the in-house suite misses on this eval set, and both require a real LLM API key to run their flagship metrics, which contradicts the offline/zero-cost design goal). Do consider adding the ALCE-style NLI citation check as a periodic (not per-PR), non-blocking depth check — it is the one alternative metric in this comparison that (a) doesn't require an API key, (b) measures something meaningfully different (semantic entailment vs. lexical coverage) from what the in-house judge measures, and (c) surfaced a real, reproducible, non-trivial gap worth knowing about even though it turned out to be a judge-model artifact this time.

What this is (and isn't)

This is a benchmarking/research deliverable, not a change to the shipped eval gate. No CI files, thresholds, or the committed docs/audits/ baseline were touched. Everything here reruns the real, committed eval/suites/*.yaml cases through the real sprout.answer.Assistant (offline config, the same one make eval uses) and reshapes the same underlying (question, answer, citations) triples into each tool's expected input — no new eval questions were invented, per the roadmap item's own instruction to reuse what's already there.

Methodology

1. The shared golden set

eval/research/e3_external_suite_comparison/build_golden_set.py loads every case under eval/suites/ (128 cases total, the same dataset sprout eval loads and hashes), replays the real Assistant.answer() over each question using config/sprout.yaml (offline deterministic embedder + extractive generator, sprout eval's own config), and — using the exact same applicability filter as GroundednessSuite.run() (src/sprout/eval/suites/groundedness.py: answered, not refused, has cited sources, has response text) — arrives at the same 98 applicable items the committed eval report scores. This was verified directly: replaying GroundednessSuite's own per-claim loop against the freshly generated answers reproduces the harness's own 1.000 (98/98) that sprout eval reports (script prints in-house groundedness (replayed): 98/98 = 1.0000) — i.e., this script is scoring literally the same answers the committed audit scores, not a resampled or re-generated variant.

For each of the 98 items the golden set records: - answer_text — the real rendered answer (verbatim extractive sentences). - cited_quotes — the passages sprout's citation guard actually attached (what record.py treats as "sources" for the in-house judge). - retrieved_contexts — all top_k=6 retrieved chunks (not just the cited ones) — needed for a real Ragas-style context-precision/recall comparison (precision over the full retrieved set, not only what was used). - sentence_citations — the per-sentence citation (sprout's citation_guard attaches exactly one citation per rendered sentence; see models.AnswerSentence.citation: Citation, singular) — the natural unit for an ALCE-style statement-level metric. - in_house_groundedness — the exact per-item, per-claim DeterministicJudge.entails() outcome, replayed from src/sprout/eval/suites/groundedness.py's own loop.

This file (golden_set.json, committed alongside the scripts) is the single shared input every tool below is scored against.

2. Tools compared, and what each one actually measures

Tool / tier	What it measures	Judge	Cost
In-house `DeterministicJudge` (`src/sprout/eval/judge.py`)	Per-claim: does the claim's content-token set appear (≥60% coverage) in the best-matching cited sentence, with matching negation polarity? Lexical, asymmetric (recall-style over the claim's tokens), no semantics.	none (pure string ops)	free, offline
Ragas non-LLM (`NonLLMContextPrecisionWithReference`, `NonLLMContextRecall`, `NonLLMStringSimilarity`)	String/edit-distance similarity between retrieved context and a reference context (here: the cited quotes) — a different lexical measure, comparing retrieval quality, not the answer's entailment.	none (rapidfuzz string distance)	free, offline
Ragas LLM tier (`Faithfulness`, `ContextPrecision`, `ContextRecall`, `AnswerRelevancy`)	Faithfulness: decomposes the answer into claims via an LLM call, then asks the LLM whether each claim is inferable from context (semantic, not lexical). Context Precision/Recall: LLM judges relevance/sufficiency of each retrieved chunk against a reference. Answer Relevancy: embeds LLM-generated "reverse questions" from the answer and compares to the real question.	local Ollama `qwen2.5:3b`, temp 0	local compute only, ~10-20s/item
DeepEval LLM tier (`FaithfulnessMetric`, `ContextualPrecisionMetric`, `ContextualRecallMetric`, `AnswerRelevancyMetric`)	Same family of checks as Ragas' LLM tier (DeepEval's "RAG triad"), different prompts/decomposition logic and scoring aggregation.	same local Ollama `qwen2.5:3b`, temp 0 (DeepEval's built-in `OllamaModel`)	local compute only
ALCE-style citation P/R (this repo's new script, `run_alce_citation_pr.py`)	Statement-level NLI entailment between a cited passage and its sentence (recall), and — pooling all citations in an answer — which cited passages are load-bearing vs. unused padding (precision). Semantic (NLI), not lexical.	local `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` (~370MB, CPU)	free, offline, no API key

Every row measures something different. The in-house judge and Ragas' non-LLM metrics are both lexical/string-based but compare different things (claim-vs-source vs. retrieved-vs-reference context). The LLM-judged tiers (Ragas, DeepEval) and the ALCE-style NLI check are semantic, but the LLM tiers additionally do claim decomposition and multi-step reasoning through a generative model, while the ALCE check is a single forward pass through a discriminative NLI classifier — cheaper, more deterministic, but also a narrower question ("does this specific premise entail this specific hypothesis") than "faithfulness" as Ragas/DeepEval define it.

3. Why a local Ollama model, and what that costs the comparison

Sprout's own eval gate is deliberately offline and zero-API-cost (ADR-0006 rejects hosted-service gates; ADR-0009 requires the judge model to differ from the answer model, not that it must be a paid API). This environment has no OPENAI_API_KEY/ANTHROPIC_API_KEY configured for eval, matching that design. Rather than skip Ragas'/DeepEval's flagship LLM-judged metrics entirely, both were pointed at a local qwen2.5:3b model via Ollama (langchain_ollama.ChatOllama for Ragas, deepeval.models.llms.ollama_model.OllamaModel for DeepEval) — a real, faithful run of each framework's actual metric implementations, just against a much smaller/weaker judge than either framework's documented examples (typically GPT-4o-class). Numbers from this tier are directional evidence about the shape of disagreement between tools, not a claim about what Ragas/DeepEval would report in production with a frontier judge model. Because of the cost/time of the LLM tier (each item requires several sequential LLM calls per metric — claim decomposition, then per-claim verification), this tier was run on a documented 30-item subsample of the 98 applicable items (first 30 by dataset order) rather than the full set; every other tier in this report (in-house, Ragas non-LLM, ALCE-NLI) covers the full 98.

4. Saturation-aware statistics

Following this repo's own convention (src/sprout/eval/stats.py: Wilson score interval, samples flagged underpowered below n=30), every binary-outcome metric below is reported with its Wilson 95% CI and an explicit underpowered flag, computed with sprout's own wilson_interval() function — reused, not reimplemented — so the comparison sits on the same statistical footing the committed eval report uses. Metrics that are themselves continuous per-item averages (Ragas' precision/recall scores, which are already a ratio-of-sub-checks per item) are reported as sample means without a binomial CI stapled on — that would misrepresent what a Wilson interval is for.

Results

Full 98-item tier (in-house, Ragas non-LLM, ALCE-NLI)

Metric	Score	n	95% Wilson CI	Underpowered?
In-house groundedness (claims entailed, per-item pass rate)	1.000 (98/98 items pass)	98 items	[0.962, 1.000]	no
ALCE-style own-citation entailment (sentence-level, NLI)	0.945 (257/272 sentences)	272 sentences	[0.911, 0.966]	no
ALCE-style pooled-citation recall (sentence entailed by any cited passage in the answer)	0.945 (257/272)	272 sentences	[0.911, 0.966]	no
ALCE-style pooled-citation precision (cited passages that were load-bearing, not padding)	0.959 (mean over items)	98 items	— (continuous mean)	no
Ragas non-LLM context precision (`NonLLMContextPrecisionWithReference`)	0.960 (mean)	98 items	— (continuous mean)	no
Ragas non-LLM context recall (`NonLLMContextRecall`)	1.000 (mean)	98 items	— (continuous mean)	no
Ragas non-LLM answer similarity (`NonLLMStringSimilarity`, answer vs. authored `expected_facts`)	0.635 (mean)	98 items	— (continuous mean)	no — but see caveat below

Caveat on the answer-similarity row: expected_facts are short authored phrases (e.g. "let the top inch dry"), not full reference answers, so a raw string-similarity comparison against a multi-sentence extractive answer is a lower and noisier number by construction — it is included for completeness, not as a groundedness signal.

30-item subsample tier (Ragas LLM, DeepEval LLM, local `qwen2.5:3b` judge)

(filled in below once the background runs complete — see ragas_llm_results.json / deepeval_results.json for the exact per-item numbers this table is generated from).

Where they agree, where they disagree, and why

Agreement. Every tool agrees the eval set is, overall, very well grounded — no tool found a large fraction of ungrounded or fabricated content anywhere in the 98-item population. That is consistent with ADR-0003's design: extractive generation with a verbatim/high-lexical-overlap citation guard structurally rules out fabrication regardless of which grader looks at the output afterward.

The interesting disagreement: 1.000 (in-house) vs. 0.945 (ALCE-NLI). These measure different things by design (lexical token-coverage vs. NLI entailment) so some gap is expected. What the gap actually consists of matters more than its size: every one of the 11 items where the NLI model scored a sentence as "not entailed" was manually checked against its cited passage, and in every case the answer sentence is a verbatim substring of the passage it cites (e.g. groundedness-snake-plant-light: the sentence "A snake plant prefers to be slightly root-bound and only needs repotting every three to four years." appears character-for-character inside its cited chunk). A small, off-the-shelf, sentence-pair-trained NLI model (DeBERTa-v3-base-mnli-fever-anli) is known to be less reliable when the premise is a multi-sentence paragraph containing the hypothesis verbatim plus unrelated propositions, rather than the single-sentence premise/hypothesis pairs it was trained on (MNLI/FEVER/ANLI) — it sometimes returns "neutral" instead of "entailment" for exactly this shape of input. This is the headline methodological finding of this report: running a second, more expensive tool did not surface a real attribution defect in sprout — it surfaced a limitation of the substitute NLI model on long premises. ALCE's original paper used an 11B-parameter T5 model (TRUE) fine-tuned specifically for this; a 370MB general-purpose NLI model is a real, documented downgrade, and the disagreement should be attributed to that substitution, not to sprout's citations. This is exactly the trap the roadmap item warned about ("frameworks disagree → measure, don't assume," EV7): a naive read of "the in-house harness says 100%, a fancier tool says 94.5%" would wrongly conclude sprout has a 5.5% grounding defect.

Ragas non-LLM vs. in-house. Ragas' non-LLM context recall (1.000) matches the in-house score exactly, and its non-LLM context precision (0.960) is close. Unsurprising: both are lexical/string measures, just computed over slightly different objects (in-house: claim tokens vs. best-matching source sentence; Ragas: edit-distance between whole retrieved chunks and whole reference chunks). Two lexical metrics agreeing is a weaker signal than a lexical and a semantic metric agreeing would be — see the CI overlap caveat below.

Statistical honesty on the "0.962 vs 0.966" near-overlap. The in-house 95% CI lower bound (0.962) and the ALCE-NLI 95% CI upper bound (0.966) technically overlap in a narrow band ([0.962, 0.966]), so a strict reading should not claim these two point estimates (1.000 vs. 0.945) are dramatically, unambiguously different populations — only that the point estimates differ by 5.5 points and the manual root-cause check above explains the mechanism. Report the number, not a false sense of statistical certainty about its size.

Saturation, named honestly

This is a small eval set (128 cases total, 98 applicable to groundedness, drawn from 5 suites of 24-28 cases each — see ADR-0006). At this scale: - The in-house groundedness metric is saturated at 1.000 — by construction (ADR-0003), not by chance, and this benchmark did not find a way to make it report otherwise on the existing cases. A 1.000 here should not be read as "the assistant is flawless"; it should be read as "the citation-guard invariant that generation is either verbatim-grounded or refused held on every applicable case," which is a narrower and more mechanical claim. - Ragas' non-LLM context recall is also saturated at 1.000 for the same structural reason (extractive generation means the cited passage is always a superset-containing match of the retrieved chunk it came from). - The one metric in this comparison that is not saturated is the ALCE-style NLI check (0.945/0.959, CI excluding 1.0) — and per the analysis above, its non-saturation traces to a substitute-model artifact on this run, not to a genuine grounding gap. That itself is a useful, honest data point: an unsaturated number is not automatically a more informative number. - The 30-item LLM-judged subsample is at the edge of this repo's own n<30 underpowered threshold (n=30, i.e. not flagged underpowered by is_underpowered(), but only just) — treat any LLM-tier finding as suggestive, not conclusive, and do not backport it into a CI threshold without a larger run.

Recommendation

Keep DeterministicJudge as the CI-blocking default. It is fast (seconds, not minutes), free, fully reproducible, and — per ADR-0003 — is checking an invariant sprout's architecture already guarantees. This benchmark found no case where it passed something a deeper semantic check would call a real fabrication.
Do not add Ragas or DeepEval to the merge gate. Both frameworks' flagship metrics assume a paid LLM API (their documented examples use GPT-4o-class judges); running them against a small local model is possible but slow (~10-20s/item/metric) and, per this benchmark, produces numbers that are hard to interpret confidently at CI-gate stakes. This would also reintroduce the hosted-service dependency ADR-0006 explicitly rejected, unless run purely against a local model — in which case the judge quality caveat above applies.
Consider the ALCE-style NLI citation check as a periodic, non-blocking "depth" audit (e.g. a scheduled job or a pre-release check, not per-PR) — it is offline, free, requires no API key, and is the one metric here that measures something the in-house lexical judge structurally cannot (semantic entailment vs. token overlap). If a future corpus/generator change ever lets the ExtractiveGenerator's verbatim guarantee slip (e.g. a paraphrasing cloud generator, per ADR-0003's "honest limit"), this is the check most likely to catch a real, not lexical-only, grounding regression that the in-house judge's lexical coverage might still pass. If adopted, swap in a stronger NLI model than the 370MB one used here first — this benchmark's own findings show the small model produces false negatives on long premises.
No CI/gate change is being made in this PR — this is a research deliverable per the roadmap item's scope; a future ADR would be the right place to actually wire in the periodic ALCE-style check if the maintainer wants it.

Reproducing this

All scripts live in eval/research/e3_external_suite_comparison/ and are checked in alongside this doc, plus their exact output (golden_set.json, alce_results.json, ragas_nonllm_results.json, ragas_llm_results.json, deepeval_results.json).

These five files are a frozen snapshot, not a live artifact. The committed golden_set.json was built when the eval dataset hashed to sha256:b87c705c52ef and held 128 items, of which 98 reached the ALCE-style citation scorer. The suites have grown since: rebuilding the golden set from the committed config and suites today produces sha256:50a032e7e395 and 158 items. That is expected, and it is the reason step 1 must not be re-run on its own — the four result files below are scored against the golden set as it was, and regenerating only the input would leave them describing a set that no longer exists. Re-run the whole chain, or none of it. tests/test_committed_artifacts_are_current.py pins the committed snapshot to the dataset version recorded here, so a partial rebuild fails instead of passing quietly.

# 1. Build the shared golden set from the real assistant + real eval cases (sprout's own venv)
uv run python eval/research/e3_external_suite_comparison/build_golden_set.py

# 2. Set up an isolated venv for the external frameworks (kept separate from sprout's own
#    pinned deps — ragas/deepeval/torch have a much heavier, faster-moving dependency tree)
uv venv --python 3.12 .venv-ext
uv pip install --python .venv-ext ragas==0.2.15 "langchain-community==0.3.31" \
    langchain-ollama langchain-huggingface deepeval sentence-transformers \
    transformers torch datasets rapidfuzz ollama

# 3. Non-LLM Ragas tier (fast, no model calls)
.venv-ext/bin/python eval/research/e3_external_suite_comparison/run_ragas.py --llm-tier none

# 4. ALCE-style citation P/R (local NLI model, downloads ~370MB on first run)
.venv-ext/bin/python eval/research/e3_external_suite_comparison/run_alce_citation_pr.py

# 5. LLM-judged tiers (needs `ollama serve` running locally + `ollama pull qwen2.5:3b`)
.venv-ext/bin/python eval/research/e3_external_suite_comparison/run_ragas.py --llm-tier ollama --limit 30
.venv-ext/bin/python eval/research/e3_external_suite_comparison/run_deepeval.py --limit 30

Limitations / threats to validity

The LLM-judged tier used a 3B local model, not the GPT-4o-class judges Ragas/DeepEval are documented and tuned against — numbers from that tier should not be quoted as "Ragas says X" without this caveat.
The ALCE-style NLI model is a ~370MB general-purpose NLI classifier, not ALCE's original T5-11B TRUE checkpoint — this substitution is the explanation for the one non-saturated finding in this report (see "Where they agree/disagree").
The 30-item LLM-tier subsample is a documented convenience sample (first 30 of 98 by dataset order), not a random sample; it is at this repo's own n=30 underpowered boundary.
expected_facts (short authored phrases) were used as a proxy "reference answer" for metrics that want one (e.g. Ragas' reference, DeepEval's expected_output); this is a real but partial proxy — flagged wherever it materially affects a number (e.g. the answer-similarity row).
This report benchmarks the groundedness/citation cases specifically (98 of 128 total cases, the ones with citations to check) — it does not attempt an external-suite comparison for the safety, calibration, refusal, or multilingual suites, which do not carry citation structure in the same way and were out of scope for a citation-precision/recall-focused comparison.