Sprout — roadmap, metrics ledger, and conformance declarations
This file records Sprout's public targets, measured numbers, declared tiers, and any justified deviation from the named engineering standards. Silent deviation from a standard is a defect, not a footnote.
Author: Chelsea Kelly-Reif · Last updated: 2026-07-12 (GenAI lifecycle telemetry and weekly
corpus-freshness gate; conformance-audit remediation pass began 2026-07-05 —
corrected several rows below that had gone stale since 2026-06-22; see the dated erratum/gap
notes throughout) · Status: In build (Phase 3).
AI-Evaluation-Standard: APPLIES (tiers: RAG, red-team, model-card)
Sprout retrieves-then-generates over a cited corpus and consults an LLM-as-judge, so the
AI Evaluation Standard binds in full. The three
eval layers (retrieval, generation, calibration) are all gated; the judge model
(claude-sonnet-4-6) is structurally different from the default answer model
(claude-haiku-4-5-20251001), and the offline deterministic generator is the default so the
entire eval runs with no network and no cloud account.
EU AI Act / NIST AI RMF classification (explicit, per §6): Not Annex III high-risk — no
recruitment, credit, law-enforcement, education, or critical-infrastructure decisioning; a
houseplant-care assistant. Not GPAI; API-only, training compute = 0. The named GenAI risks in
scope are confabulation (mitigated structurally: extractive generation + citation guard,
groundedness 100% by construction) and information integrity / safety (the
never-certify-"safe" toxicity rule and vet/poison-control routing). Reviewed 2026-06-22 by
Chelsea Kelly-Reif; recorded in docs/audits/ai-risk-register.md on completion of Phase 2.
Metrics ledger
The portfolio-standard Metric | Target | Measured by | Gate shape. Targets are this repo's
values; the gate mechanics are defined in the referenced standard, not restated here. Every
row is an AUTO-GATE (mechanically checked, merge-blocking) unless marked REVIEW. The one
command that reproduces the whole set locally is make verify
(lint · type · test · security · eval · a11y).
Code quality and coverage — CODE-QUALITY-STANDARD
| Metric | Target | Measured by | Gate |
|---|---|---|---|
| Lint | zero findings | ruff format --check + ruff check src tests |
AUTO |
| Type safety | zero errors, strict | mypy (strict; py.typed shipped) |
AUTO |
| Branch coverage | ≥ 90% (published-library floor) | pytest --cov=sprout --cov-fail-under=90 |
AUTO |
| Layout | src/ layout, importable as sprout |
uv build (packaging regression check, CQ-10) + tests/test_resources.py import |
AUTO |
AI evaluation suites — AI-EVALUATION-STANDARD
9 suites — calibration, completeness, conversation, groundedness, language-parity, multilingual, refusal, safety, toxicity-coverage
— over 158 committed YAML cases (exceeding the 120+ target),
scored by deterministic checks blended with an LLM-as-judge (judge ≠ answer model). Runs are content-hashed and byte-identical for
identical inputs; the gate is both the absolute threshold below and no regression past
tolerance from the committed baseline. Each PASS must also clear its Wilson lower bound when
the statistical gate is on (see runner.py::_apply_statistical_gate).
| Suite / metric | Target | Measured by | Gate |
|---|---|---|---|
| groundedness (every claim entailed by its cited passage) | ≥ 0.95 | eval/suites/groundedness + judge entailment |
AUTO |
| safety (cite a toxicity ref, never certify "safe," route to vet/poison-control) | ≥ 0.95 | eval/suites/safety + deny-list guard |
AUTO |
| refusal (out-of-scope, "just tell me it's fine," embedded injection) | 0.90 offline / ≥ 0.95 portfolio — sprout eval now auto-selects the gate from retrieval.embedding_provider (refusal.threshold_for): the offline hashing embedder keeps its documented 0.90 floor (per the suite's own docstring — it cannot fully separate every unknown-species/jailbreak phrasing from in-scope), and the run auto-raises the gate to the ≥ 0.95 portfolio target the moment embedding_provider: bedrock (Titan) is configured. CI still exercises the offline default only, so 0.90 remains what's actually measured today — corrected 2026-07-08, was a hardcoded 0.90 with no enforcement path to 0.95 at all. EXP-03's opt-in offline static embedder measurably narrows the gap to 0.9412 without cloud/network — ADR-0017 — but does not close it, so 0.90 remains the wired offline default's floor |
eval/suites/refusal |
AUTO |
| multilingual (ES preserves the facts + citations of its EN mirror) | ≥ 0.85 — fraction of non-reference-language cases matching their EN anchor on the refuse/answer decision + cited-plant set | eval/suites/multilingual + judge equivalence |
AUTO |
| calibration (stated confidence tracks correctness) | ECE ≤ 0.15 | eval/suites/calibration (reliability diagram + ECE) |
AUTO |
completeness (a multi-facet answer covers every authored expected_fact) |
≥ 0.90 | src/sprout/eval/suites/completeness.py over cases with ≥ 2 facets; single-fact cases are groundedness's job |
AUTO |
| conversation (a follow-up resolves species/topic from bounded history, and history is never a source) | ≥ 0.95 | src/sprout/eval/suites/conversation.py, replayed through the server's own SessionMemory; a history-injection case that changes which chunks ground the answer is an outright failure |
AUTO |
toxicity-coverage (every ASPCA-listed pet-toxic plant the corpus covers carries a routed ## Toxicity section) |
≥ 0.99 | src/sprout/eval/suites/toxicity_coverage.py — corpus-level, so no lucky generation can satisfy it |
AUTO |
| Abstention enforced below threshold | answered cases below 0.25 confidence must have been refusals (ADR-0012, supersedes ADR-0005; corrected 2026-07-05 — see execution log) | eval/suites/calibration invariant |
AUTO |
| EN/ES pass-rate parity (aggregate delta between languages) | |EN − ES| ≤ 5 pp (0.05) — the gap between the language slices' pass rates over the recorded per-case correctness label, with the EN anchors scored as a slice of their own | src/sprout/eval/suites/language_parity.py (the language-parity suite, metric en-es-pass-rate-gap) |
AUTO (wired 2026-09-05, closing the 2026-08-29 correction below). Was not implemented: the row declared AUTO and named the bilingual slice, but eval/suites/multilingual gates per-case structural parity at ≥ 0.85 (the row above), a different quantity — it only ever scores the non-reference member of a pair and never scores the EN anchors — and no code computed the delta. It does now, as its own suite with its own written metric, so the two quantities cannot be conflated again. Measured on the committed run: 0.011 (1.1 pp) — EN 0.826 (n=121) vs ES 0.838 (n=37) — inside the target. Read it with the caveats the suite itself publishes: the 95% Newcombe interval on the gap is [0.000, 0.127], so this corpus cannot yet rule out a gap far above 5 pp, and the EN/ES case sets are not matched, so the pooled figure carries a case-mix component. The suite's report-only stratum diagnostics all exceed 5 pp (matched pairs 0.083, behaviour=answer 0.068, behaviour=refuse-and-redirect 0.052). The gate is the aggregate this row declares; deepening the 37-case ES slice so the aggregate can carry more weight is separate, still-outstanding work. |
| Hallucination rate | 0% by construction (extractive + citation guard) | tests/test_rag.py citation-guard tests + eval/suites/groundedness |
AUTO |
| Judge ↔ human agreement (deterministic judge, CI floor) | ≥ 0.80 raw · Cohen’s κ ≥ 0.60 | sprout calibrate --gate on the dated probe set |
AUTO (gated 2026-07-08, P0-4) — probe set expanded 12 → 66 (within the 50–100 target) across all 16 corpus species, and the judge’s negation-only polarity guard gained has_antonym_conflict for antonym flips carrying no negation marker ("safe" vs "toxic"); measured on the combined set: agreement 0.955, κ 0.906, both clear threshold; the CI/make verify step passes --gate and fails the build on regression. This gates the reproducible offline judge as a coverage/polarity smoke-floor only — morphological synonyms and low-overlap paraphrase remain documented blind spots (3 known disagreements are committed, not hidden). See docs/audits/judge-calibration.md. |
| Judge ↔ human agreement (LLM judge, production gate) | ≥ 0.80 raw · Cohen’s κ ≥ 0.60 | sprout calibrate --judge llm |
not yet done — the LLM judge needs a live Anthropic credential and is deliberately never invoked in CI (see eval/llm_judge.py); calibrating and gating it (--judge llm --gate) before it backs any production judging decision is a separate, still-outstanding step, tracked here rather than conflated with the deterministic-judge CI floor above. |
| Judge-calibration freshness | probe set re-labeled within 30 days | labeled_date on eval/judge_probes.yaml, checked by sprout calibrate |
warn-only past 30 days; hard failure when the check cannot run (wired 2026-07-05, P1-19; split 2026-09-07) — re-labeled 2026-07-08 alongside the 12→66 expansion. A probe set past the 30-day target still only warns, and flipping that to a failure ties to P0-4 (issue #130): it happens once the LLM judge is calibrated and --gate is enabled. A probe set whose freshness cannot be read at all — no labeled_date, a labeled_date that is not an ISO date, or one dated in the future — exits 2 under --gate. All three used to take the same warn-and-pass path a 31-day-old set takes, so this row could have gone on claiming the check while the field it reads was absent; a future date was the worst of them, since a negative age satisfies the comparison indefinitely. |
| Fail-closed loader | hash mismatch / malformed case / empty suite / bad judge output → FAIL | eval/dataset.py + runner.fail_closed |
AUTO |
| Model card completeness | required HF front-matter present | tests/test_model_card.py |
AUTO (wired 2026-07-05 — see P1-11; previously declared AUTO with no lint) |
| Card honesty / limits framing | truthful, not box-ticking | owner review per release | REVIEW |
| Red-team (OWASP LLM01–LLM10) | 0 open critical findings | Promptfoo redteam on prompt/model PRs |
config committed 2026-07-08 (eval/redteam/promptfooconfig.yaml, eval/redteam/README.md) — covers OWASP LLM01–LLM10 against the live POST /api/chat pipeline in EN+ES; wired as an advisory, non-blocking redteam CI job (.github/workflows/ci.yml) that needs ANTHROPIC_API_KEY; gap now: the key is not yet provisioned as a repo secret and no run has completed, so it is not yet in ci-gate and "0 open critical findings" is not yet a measured number — the refusal/adversarial eval suite plus the manual dated red-team report (docs/audits/red-team-2026-06-22.md) remain the standing substitute until a run is observed clean and the job is promoted to blocking |
| Garak (LLM vulnerability scanner) | n/a | — | N/A-with-reason — the offline deterministic default has no LLM to scan (extractive generation, no model in the loop); revisit when the Bedrock/Anthropic generator seam is activated in a production configuration. Added 2026-07-05 (previously unrecorded — AIEV-14). |
Provider note (per standard §0): Sprout standardizes on Anthropic Claude — Haiku to answer, Sonnet to judge — behind a config switch; the deterministic offline generator is the default and is what CI exercises (no network, no key). No "rejected because" deviation is recorded.
GenAI lifecycle measurement
The optional Anthropic and Bedrock paths adopt the shared OpenTelemetry GenAI runtime vendored
byte-for-byte under src/sprout/_vendor/genai_telemetry/ from immutable STANDARDS commit
e8150c82fc35267f022af46ac71fe5a851e2d042; .standards-version pins the boundary and
src/sprout/genai_telemetry.py is only the Sprout record/sink wrapper. Native
Anthropic answers, Bedrock Claude answers, Titan embeddings, and the native Anthropic judge all
record success/error duration, the locally selected request model, allowlisted protocol finish
reasons, and normalized usage fields, without reflecting provider strings or capturing content.
Claude answer/judge calls and Titan embeddings receive shared-table estimates;
Titan's AWS catalog price is selected by the configured Bedrock region carried on Usage.
The operational provider wrapper rejects answer calls before transport when the model is unpriced
or the estimate exceeds generation.max_cost_usd; Titan activation likewise rejects a missing or
unsupported region rather than borrowing a false rate. The wrapper forwards the original
query/context/limit into the behavior-bearing provider unchanged. Provider-separated fresh,
cache-creation, and cache-read tokens are summed into canonical total input before cache-hit and
Claude cost math; Titan's input-only row rejects unsupported output/cache usage.
Streaming first-chunk latency is N/A because none of these adapters streams. The implementation
and privacy proof are recorded in
docs/audits/genai-lifecycle-telemetry-2026-07-12.md and gated by
tests/test_genai_telemetry.py.
The offline eval/calibration loop remains merge-blocking, and the corpus has a weekly scheduled freshness gate. The production trace-to-weekly-judge loop remains an explicit external launch gate until the optional cloud API, credentials, and trace store exist; it cannot produce an honest measured distribution before there is production traffic.
Accessibility — ACCESSIBILITY-STANDARD
| Metric | Target | Measured by | Gate |
|---|---|---|---|
| Conformance level | WCAG 2.2 AA | .github/workflows/ci.yml pa11y + lighthouse jobs (both ci-gate dependencies): axe + pa11y (merge-blocking) and Lighthouse accessibility (merge-blocking, threshold 0.95) on the reference question UI + HTML eval report; transcript view not yet built (see row below) |
AUTO |
| Structural a11y check | zero violations | sprout a11y-check on web/dist/index.html + docs/audits/eval-report.html |
AUTO |
| Non-chat alternate view | static, paginated Q/A/citations renders | tests/test_a11y_and_judge.py (render_transcript) |
AUTO |
| Color independence | severity + provenance never color-only | manual SR review (NVDA, VoiceOver) | REVIEW |
| ACR (VPAT 2.5 Rev 508) | committed, regenerated on release | docs/accessibility/ACR.md |
REVIEW |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD
| Metric | Target | Measured by | Gate |
|---|---|---|---|
| App-security level | OWASP ASVS L1 (offline mode; no auth, no persistence, no network) | review-gate checklist | REVIEW |
| Dependency audit | zero unresolved advisories | pip-audit (blocking in CI; never \|\| true) |
AUTO |
| Secret scanning | zero leaks | gitleaks |
AUTO |
| Static analysis | zero high findings | semgrep + .github/workflows/codeql.yml |
AUTO |
| Actions pinning | SHA-pinned, least-privilege tokens | zizmor workflow-SAST scan (.github/workflows/ci.yml) |
AUTO |
| SBOM | emitted on release | .github/workflows/release.yml (Generate SBOM step; never \|\| true) |
AUTO |
| PII in logs | zero — logger whitelists low-cardinality fields only, never question text | obs.py _ALLOWED_FIELDS + Semgrep/bandit |
AUTO (never N/A) |
Internationalization — INTERNATIONALIZATION-STANDARD
| Metric | Target | Measured by | Gate |
|---|---|---|---|
| EN/ES key + placeholder parity | complete, no orphan keys | tests/test_i18n_parity.py |
AUTO (wired 2026-07-08 — see FIX-02; previously declared AUTO with no such diff implemented) |
| EN/ES eval per-case structural parity | ≥ 0.85 — each ES case matches its EN anchor on refuse/answer + cited plants | eval/suites/multilingual (also in the AI ledger above) |
AUTO |
| EN/ES eval pass-rate parity (aggregate delta between languages) | |EN − ES| ≤ 5 pp — measured 0.011 (EN 0.826 n=121 vs ES 0.838 n=37) | src/sprout/eval/suites/language_parity.py (language-parity, also in the AI ledger above) |
AUTO (wired 2026-09-05; was "not implemented" per the 2026-08-29 correction — see the fuller row in the AI-evaluation ledger above for the measured number and the interval/case-mix caveats it comes with) |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD
| Metric | Target | Measured by | Gate |
|---|---|---|---|
| First-token latency (offline) | p95 < 200 ms | tests/test_latency.py |
AUTO (wired 2026-07-05 — previously declared AUTO with no test; see P1-10) |
| Reproducibility | byte-identical report from identical inputs | tests/test_eval_suites.py (test_run_is_byte_identical) |
AUTO |
| Versioning | SemVer; Keep-a-Changelog; signed tags on first release | .github/workflows/release.yml |
AUTO (mechanism wired; never yet exercised — no tag has ever been cut, corrected 2026-07-05; see CHANGELOG.md) |
| Publish | PyPI Trusted Publishing (OIDC), distribution sprout-plantcare |
.github/workflows/release.yml |
AUTO (wired; unexercised — same caveat). The distribution name is not sprout: that name on PyPI belongs to an unrelated library, so a release under it was never possible. sprout-plantcare was free when chosen (2026-09-07). Publishing additionally needs a one-time PyPI Trusted Publisher registration, which is a web-UI action only the owner can take: project sprout-plantcare, owner ChelseaKR, repository sprout, workflow release.yml, environment pypi. Until it exists the publish-pypi job fails at the upload and nothing is published. |
| CI parity | make verify uses the same tools/thresholds as the required ci-gate checks |
sprout ci-parity-check (src/sprout/ci_parity.py, tests/test_ci_parity.py) mechanically diffs .github/workflows/ci.yml's required jobs against their Makefile target(s); run via make ci-parity-check (also a make verify prerequisite) and the ci-parity CI job (a ci-gate dependency) |
AUTO (wired 2026-07-08, closing ci-parity-no-mechanical-diff; the checker's first run also surfaced two real gaps — docs and the zizmor workflow-SAST scan were required by ci-gate but absent from make verify — now closed via new docs/workflow-lint prerequisites) |
Observability tier
Per the Observability Standard §0, the tier is declared here and any skipped control is recorded as N/A-with-reason; silent omission is a defect caught by the tier-declaration gate. Sprout has two surfaces and states both.
Tier C — the offline CLI (the default, and what CI exercises).
OTEL_SERVICE_NAME / service.name = sprout.
- Logging is opt-in structured JSON via
observability.log_format: json(textby default), emitted bysrc/sprout/obs.py. It is PII-free by construction: the logger drops any field outside a whitelist of low-cardinality keys (language,refused,refusal_reason,is_safety_query,confidence,n_retrieved,n_sentences,injection_categories,status,route,index_size) — the user's question text is never logged. - N/A-with-reason — OTel tracing, RED/USE metrics, SLOs, burn-rate alerts,
/livez+/readyz: out of scope for a local-only CLI with no network surface (standard §10). The valid-JSON and required-field log gates apply only when--log-format jsonis selected. - NOT N/A — the PII/secrets-in-logs gate. It is non-tiered and binds here exactly as in any Tier-A service; the whitelist above is its enforcement.
Tier A — the optional serverless API (infra/, behind observability.tier: A).
When the cloud generator and serverless deploy are enabled, the API surface adopts the full
Tier-A stack, and every piece of it is now wired and code-reviewable, not aspirational:
- OTel traces + metrics,
src/sprout/otel.py: a tracer/meter provider per process, W3Ctraceparentextraction on inbound requests, and RED-per-endpoint instruments (sprout_http_requests_total,sprout_http_request_duration_secondson the standard's fixed second-scale buckets,sprout_http_request_errors_total) recorded byREDMiddleware, keyed on the matched route template (never the raw path) so cardinality stays bounded.configure_observabilityno-ops (never crashes the server) for tier B/C or if theobservabilityextra is not installed. - Trace-correlated structured logs:
src/sprout/obs.pystampstrace_id/span_id/trace_flagsfrom the active span onto every Tier-A JSON log record. /livez+/readyz: already existed insrc/sprout/server.pybefore this change; unaffected.- An SLO file:
slos/sprout-api-availability.yamlandslos/sprout-api-latency.yaml, schema-checked bysprout slo-check(wired intomake verifyand CI'seval-a11yjob). - Multi-window burn-rate alerts:
alerts/burn-rate.yml— critical (14.4×, 1h+5m) and high (6×, 6h+30m) tiers, both required and both present, per the standard §5. - A deployable manifest:
infra/is a real AWS CDK (Python) app —infra/sprout_stack.py— deploying the existingsprout servecontainer to Lambda (via the AWS Lambda Web Adapter,infra/Dockerfile) behind an API Gateway HTTP API, with a monthlyCfnBudgetalarm. It synthesizes cleanly (cdk synth, verified against aws-cdk-lib 2.261 while building this); seeinfra/README.md.
What is verified vs. what is not. The OTel wiring, RED metrics, log correlation, and
SLO/alert schema checks are exercised in tests/test_otel.py and tests/test_slo.py, and
were additionally run end-to-end against a live local OTel Collector +
Tempo + Mimir (docker-compose.observability.yml) while building this change — a real
/api/chat request produced a queryable trace in Tempo and the exact
sprout_*_http_requests_total/..._duration_seconds series in Mimir with the expected
method/route/status_code labels. What is not yet exercised: an actual cdk
deploy against a live AWS account, and the burn-rate alerts firing against real traffic
(no production traffic exists yet). That is the same "wired, not yet exercised" posture
this doc already uses for PyPI Trusted Publishing and signed release tags — record the
first real cdk deploy here when it happens. The offline Tier-C path remains fully
functional with the serverless surface absent, and needs none of the above.
Per-standard applicability — none N/A
Every portfolio standard APPLIES to Sprout. There is no standard marked N/A at the repo level. The single deferred scope is noted explicitly.
| Standard | Applies | Posture / values |
|---|---|---|
| Quality & Metrics (ISO 25010 / DORA) | APPLIES | Ledger above; make verify = CI gate set; latency + reproducibility budgets |
| Code Quality | APPLIES | ruff + mypy --strict; branch coverage ≥ 90%; src/ layout |
| Security & Supply Chain | APPLIES | ASVS L1 (offline mode — no auth/persistence/network to defend); pip-audit, gitleaks, Semgrep, SHA-pinned actions, SBOM |
| CI/CD | APPLIES | Single ci-gate required check; least-privilege tokens; local/CI parity via make verify |
| Release & Versioning | APPLIES | SemVer; Keep-a-Changelog; PyPI Trusted Publishing (OIDC) wired but unexercised; signed tags on first release (no tag has ever been cut — corrected 2026-07-05, see docs/ROADMAP.md REL-03 note below) |
| Accessibility | APPLIES | WCAG 2.2 AA target; structural sprout a11y-check, axe/pa11y, and Lighthouse accessibility (threshold 0.95) are all merge-blocking as of 2026-07-08 (previously axe/pa11y advisory-only and Lighthouse unwired); transcript view; ACR (VPAT 2.5 Rev 508) |
| Observability | APPLIES | Tier C (offline CLI) + Tier A (optional serverless); see section above; Tier-A controls wired and unit/e2e-tested, cdk deploy unexercised |
| Internationalization | APPLIES | EN/ES key + placeholder parity; per-case EN/ES structural parity gated at ≥ 0.85 by eval/suites/multilingual; the aggregate |EN − ES| pass-rate delta gated at ≤ 0.05 by language-parity (measured 0.011, with the interval/case-mix caveats in the ledger row) |
| AI Evaluation | APPLIES (RAG, red-team, model-card) | groundedness/safety/refusal/multilingual/language-parity/calibration gates; judge ≠ answer model; κ + reliability; model/data cards |
| Documentation | APPLIES | Full docs/ set; ADRs; dated, regenerated audit artifacts |
| Responsible-Tech Framework | APPLIES | docs/RESPONSIBLE-TECH-AUDITS.md §A–F + AI-EVAL + I18N; no audit is N/A; added to this table 2026-07-05 (was silently omitted — DOC-11) |
Family Greenhouse personalization. Phase A is implemented behind a feature flag: strict
minimized context, HMAC authentication, provenance labeling, sentinel-PII proofs, and a scoped
ASVS L2 review. Corpus-only remains the privacy-preserving default. Phases B (proactive
notifications) and C (confirmed write-back) remain deferred. See Family Greenhouse integration in
../CLAUDE.md for the full plan and phasing (A → B → C).
Build plan and current status
The four phases from the spec, with honest status as of 2026-06-22. A phase is done only
when make verify is green for its scope.
Phase 1 — corpus + retrieval
Fetch and snapshot open-licensed care/toxicity references with a dated manifest; chunk by care
topic; hybrid retrieval against smoke questions; guards.py v1 (safety-assertion ban, scope,
PII).
Status: done (corrected 2026-07-08 — the dedicated smoke suite below closed the last
outstanding Phase 1 gap; corrected 2026-07-05 — this was stale since 06-22 and had not caught
up with actual repo state per DOC-15).
- Done: hybrid retrieval (retrieve.py — BM25 + deterministic dense via reciprocal rank fusion,
topic filter, min_score threshold gate), guards.py v1 (citation guard, never-certify-safe
deny-list EN/ES, scope via retrieval threshold, PII redaction + injection labeling), ingest /
chunk / store pipeline, config-over-code (config/sprout.yaml). The synthetic CC0 corpus is
committed at corpus/processed/ — 32 files (16 species × EN/ES) with a dated, licensed
corpus/manifest.yaml — not the "ships no passages yet" state a prior version of this line
claimed.
- Done: a dedicated CI smoke suite of corpus-derived questions, beyond what the eval harness
(Phase 2) already exercises — sprout smoke (src/sprout/smoke.py), wired as its own
merge-blocking smoke job in ci.yml. Every case is templated mechanically from the
ingested corpus's own species slugs and ## <topic> headings (one question per
(species, topic) pair actually present in the store), not hand-authored, so coverage tracks
the corpus automatically as species/topics are added. Runs the offline deterministic
generator only (no judge, no network) — a fast, judge-free canary distinct from the
hand-authored Phase 2 harness (whose case count is stated once, above). 80 cases pass over the shipped corpus; report
committed at docs/audits/smoke-report.md.
Phase 2 — eval first
Runner, judges, report. Author 60 cases (groundedness, safety, refusal) from the corpus. Wire the CI smoke suite. Commit a baseline scoreboard, mediocre numbers included.
Status: substantially done (corrected 2026-07-05 — see the same DOC-15 staleness note above).
- Done: the eval engine — fail-closed dataset loader, run fingerprint (reproducible), all
9 suites registered (src/sprout/eval/suites/; six of
them authored as YAML under eval/suites/, three derived — completeness,
toxicity-coverage, language-parity), deterministic + Anthropic judges behind one Protocol
(judge ≠ answer model), report generation (MD + HTML + JSON; JUnit + SARIF), Wilson statistical
gate, ECE/reliability calibration. 158 YAML cases are
committed under eval/suites/ (exceeds the 120+ target). docs/audits/eval-baseline.json is committed and, as of 2026-07-05,
actually gates sprout eval (previously computed but never loaded by the CLI — AIEV-26,
fixed). The eval job (eval-a11y in ci.yml) is inside the required ci-gate check. The
judge-calibration probe set (eval/judge_probes.yaml, expanded 2026-07-08 from 12 to 66
probes across all 16 corpus species — within the 50–100 target) is committed and its κ is
measured, and now clears the standard’s threshold (agreement 0.955 ≥ 0.80, κ 0.906 ≥ 0.60,
measured with the P0-4 antonym-polarity guard in place); the CI step passes --gate and fails
the build on regression — see AI evaluation suites table above and
docs/audits/judge-calibration.md.
- Outstanding: this gates the deterministic judge as a reproducible coverage/polarity
smoke-floor only — morphological synonyms and low-overlap paraphrase remain documented blind
spots (3 known disagreements are committed, not hidden). Calibrating and gating the LLM
judge (--judge llm --gate) before it backs a real production judging decision is separate,
still-outstanding work that needs a live Anthropic credential and cannot run in CI.
Phase 3 — quality + multilingual
Tune retrieval/prompts against eval failures only; add calibration suite and abstention; Spanish to parity; model card. Deploy the accessible, stateless reference-and-assurance web surface behind a real URL.
Status: in progress (current phase).
- Done: calibration suite + two-threshold abstention (confidence.py), EN/ES throughout
(lang.py, per-language bundles, parity suite), framework-free WCAG 2.2 reference surface
shipped at https://sprout.chelseakr.com through the zero-server TypeScript port (one
stateless corpus question, claim chain, evaluation evidence, and docs;
household/photo/reminder workflows excluded per ADR 0015), structural a11y check,
structured PII-free logging, the ACR
(docs/accessibility/ACR.md, VPAT 2.5 Rev 508) and a dedicated OWASP-LLM red-team report
(docs/audits/red-team-2026-06-22.md, LLM01–LLM10:2025 coverage table, 0 open critical
findings) — both committed in the 2026-07-05 conformance pass. Corrected here 2026-07-08: this
bullet previously still listed the ACR and the red-team report as outstanding after they had
already been committed (DOC-defect, same class as the other 2026-07-05/07-08 "declared vs.
actual" corrections in this file). Caveat that remains real: the red-team report is a
structured manual exercise, not yet backed by an automated, per-PR mechanical check — see the
"Red-team (OWASP LLM01–LLM10)" row in the AI evaluation ledger above, which honestly carries
that gap (no Promptfoo run has completed against a provisioned key yet).
- Done: tuning only against committed eval failures is mechanically enforced by sprout
check-tuning-scope, a required CI job. Changes to retrieval, generation, guards,
calibration, lexical logic, or config must cite a case already recorded in the committed
eval baseline via a Tunes-Against: commit trailer. Comment-only YAML and the exact named
operational lifecycle wrapper are excluded by semantic/AST comparison. The initial lifecycle
module is pinned to one reviewed bootstrap digest; all later lifecycle hunks are gated.
Authorization comes from the merge-base baseline, and adversarial tests keep model, prompt,
decoding, real-config, retrieval/guard, lifecycle-output, and unknown provider edits fail-closed.
- Done: sprout fit-confidence (ADR-0016, 2026-07-08) — fits the confidence logistic on a
held-out train split (eval/train/, never eval/suites/) instead of hand-tuning it; wired
but not yet adopted as the shipped default (a real fit against the current 24-item train
split measures ECE 0.263, worse than the shipped 0.108 — see ADR-0016's evidence).
- Outstanding: get a clean Promptfoo redteam run wired and
promoted into the blocking ci-gate so the committed OWASP-LLM report is backed by a mechanical
check rather than only the manual, dated one (tracked in the ledger row above); expand
eval/train/calibration_train.yaml and re-fit before adopting confidence.fit as the
shipped default (ADR-0016).
Phase 4 — generalize
A corpus.yaml so any care corpus can be swapped in; "adapt this to your domain" doc.
Status: guide and read-only personalization Phase A done; Phases B–C deferred.
- The seam exists (config/sprout.yaml already points the whole system at a corpus path,
manifest, languages, models, and thresholds; the eval runner is corpus-agnostic). The
"adapt this to your domain" guide is written (docs/ADAPT.md, linked in the site
nav) and walks an adopter through swapping the corpus, manifest, domain vocabulary,
retrieval/abstention tuning, languages, and generator/embedding provider using only that
config seam. Remaining Phase 4 scope is Family Greenhouse notification and confirmed-write
phases B–C.
Definition of done (the bar each phase is held to)
A fresh user can install Sprout from source (and, after the first release, via pipx), ask a
plant question offline, get a cited answer (or
an honest refusal), run make eval to regenerate the committed report with no cloud account, and
read a model card that states the limits plainly — with every CI gate green. make verify
reproduces the full gate set locally; if it is not green, the phase is not done.