Gate inventory
Generated by sprout gate-inventory from docs/ROADMAP.md (FIX-02) — do not hand-edit.
Maps every ledger row to the mechanism its Measured by cell names, and mechanically
resolves that mechanism against Makefile / .github/workflows/*.yml / the repo tree.
An AUTO row that cannot be resolved is a declared-but-unenforced gate.
33 AUTO rows · 0 unresolved.
| Section | Metric | Gate | Measured by | Resolution |
|---|---|---|---|---|
Code quality and coverage — CODE-QUALITY-STANDARD |
Lint | AUTO | ruff format --check + ruff check src tests |
✅ resolved |
Code quality and coverage — CODE-QUALITY-STANDARD |
Type safety | AUTO | mypy (strict; py.typed shipped) |
✅ resolved |
Code quality and coverage — CODE-QUALITY-STANDARD |
Branch coverage | AUTO | pytest --cov=sprout --cov-fail-under=90 |
✅ resolved |
Code quality and coverage — CODE-QUALITY-STANDARD |
Layout | AUTO | uv build (packaging regression check, CQ-10) + tests/test_resources.py import |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
groundedness (every claim entailed by its cited passage) | AUTO | eval/suites/groundedness + judge entailment |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
safety (cite a toxicity ref, never certify "safe," route to vet/poison-control) | AUTO | eval/suites/safety + deny-list guard |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
refusal (out-of-scope, "just tell me it's fine," embedded injection) | AUTO | eval/suites/refusal |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
multilingual (ES preserves the facts + citations of its EN mirror) | AUTO | eval/suites/multilingual + judge equivalence |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
calibration (stated confidence tracks correctness) | AUTO | eval/suites/calibration (reliability diagram + ECE) |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
completeness (a multi-facet answer covers every authored expected_fact) |
AUTO | src/sprout/eval/suites/completeness.py over cases with ≥ 2 facets; single-fact cases are groundedness's job |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
conversation (a follow-up resolves species/topic from bounded history, and history is never a source) | AUTO | src/sprout/eval/suites/conversation.py, replayed through the server's own SessionMemory; a history-injection case that changes which chunks ground the answer is an outright failure |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
toxicity-coverage (every ASPCA-listed pet-toxic plant the corpus covers carries a routed ## Toxicity section) |
AUTO | src/sprout/eval/suites/toxicity_coverage.py — corpus-level, so no lucky generation can satisfy it |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Abstention enforced below threshold | AUTO | eval/suites/calibration invariant |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
EN/ES pass-rate parity (aggregate delta between languages) | AUTO (wired 2026-09-05, closing the 2026-08-29 correction below). Was not implemented: the row declared AUTO and named the bilingual slice, but eval/suites/multilingual gates per-case structural parity at ≥ 0.85 (the row above), a different quantity — it only ever scores the non-reference member of a pair and never scores the EN anchors — and no code computed the delta. It does now, as its own suite with its own written metric, so the two quantities cannot be conflated again. Measured on the committed run: 0.011 (1.1 pp) — EN 0.826 (n=121) vs ES 0.838 (n=37) — inside the target. Read it with the caveats the suite itself publishes: the 95% Newcombe interval on the gap is [0.000, 0.127], so this corpus cannot yet rule out a gap far above 5 pp, and the EN/ES case sets are not matched, so the pooled figure carries a case-mix component. The suite's report-only stratum diagnostics all exceed 5 pp (matched pairs 0.083, behaviour=answer 0.068, behaviour=refuse-and-redirect 0.052). The gate is the aggregate this row declares; deepening the 37-case ES slice so the aggregate can carry more weight is separate, still-outstanding work. |
src/sprout/eval/suites/language_parity.py (the language-parity suite, metric en-es-pass-rate-gap) |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Hallucination rate | AUTO | tests/test_rag.py citation-guard tests + eval/suites/groundedness |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Judge ↔ human agreement (deterministic judge, CI floor) | AUTO (gated 2026-07-08, P0-4) — probe set expanded 12 → 66 (within the 50–100 target) across all 16 corpus species, and the judge’s negation-only polarity guard gained has_antonym_conflict for antonym flips carrying no negation marker ("safe" vs "toxic"); measured on the combined set: agreement 0.955, κ 0.906, both clear threshold; the CI/make verify step passes --gate and fails the build on regression. This gates the reproducible offline judge as a coverage/polarity smoke-floor only — morphological synonyms and low-overlap paraphrase remain documented blind spots (3 known disagreements are committed, not hidden). See docs/audits/judge-calibration.md. |
sprout calibrate --gate on the dated probe set |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Judge ↔ human agreement (LLM judge, production gate) | not yet done — the LLM judge needs a live Anthropic credential and is deliberately never invoked in CI (see eval/llm_judge.py); calibrating and gating it (--judge llm --gate) before it backs any production judging decision is a separate, still-outstanding step, tracked here rather than conflated with the deterministic-judge CI floor above. |
sprout calibrate --judge llm |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Judge-calibration freshness | warn-only past 30 days; hard failure when the check cannot run (wired 2026-07-05, P1-19; split 2026-09-07) — re-labeled 2026-07-08 alongside the 12→66 expansion. A probe set past the 30-day target still only warns, and flipping that to a failure ties to P0-4 (issue #130): it happens once the LLM judge is calibrated and --gate is enabled. A probe set whose freshness cannot be read at all — no labeled_date, a labeled_date that is not an ISO date, or one dated in the future — exits 2 under --gate. All three used to take the same warn-and-pass path a 31-day-old set takes, so this row could have gone on claiming the check while the field it reads was absent; a future date was the worst of them, since a negative age satisfies the comparison indefinitely. |
labeled_date on eval/judge_probes.yaml, checked by sprout calibrate |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Fail-closed loader | AUTO | eval/dataset.py + runner.fail_closed |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Model card completeness | AUTO (wired 2026-07-05 — see P1-11; previously declared AUTO with no lint) | tests/test_model_card.py |
✅ resolved |
AI evaluation suites — AI-EVALUATION-STANDARD |
Card honesty / limits framing | REVIEW | owner review per release | — exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Red-team (OWASP LLM01–LLM10) | config committed 2026-07-08 (eval/redteam/promptfooconfig.yaml, eval/redteam/README.md) — covers OWASP LLM01–LLM10 against the live POST /api/chat pipeline in EN+ES; wired as an advisory, non-blocking redteam CI job (.github/workflows/ci.yml) that needs ANTHROPIC_API_KEY; gap now: the key is not yet provisioned as a repo secret and no run has completed, so it is not yet in ci-gate and "0 open critical findings" is not yet a measured number — the refusal/adversarial eval suite plus the manual dated red-team report (docs/audits/red-team-2026-06-22.md) remain the standing substitute until a run is observed clean and the job is promoted to blocking |
Promptfoo redteam on prompt/model PRs |
— exempt (not an unconditional AUTO claim) |
AI evaluation suites — AI-EVALUATION-STANDARD |
Garak (LLM vulnerability scanner) | N/A-with-reason — the offline deterministic default has no LLM to scan (extractive generation, no model in the loop); revisit when the Bedrock/Anthropic generator seam is activated in a production configuration. Added 2026-07-05 (previously unrecorded — AIEV-14). | — | — exempt (not an unconditional AUTO claim) |
Accessibility — ACCESSIBILITY-STANDARD |
Conformance level | AUTO | .github/workflows/ci.yml pa11y + lighthouse jobs (both ci-gate dependencies): axe + pa11y (merge-blocking) and Lighthouse accessibility (merge-blocking, threshold 0.95) on the reference question UI + HTML eval report; transcript view not yet built (see row below) |
✅ resolved |
Accessibility — ACCESSIBILITY-STANDARD |
Structural a11y check | AUTO | sprout a11y-check on web/dist/index.html + docs/audits/eval-report.html |
✅ resolved |
Accessibility — ACCESSIBILITY-STANDARD |
Non-chat alternate view | AUTO | tests/test_a11y_and_judge.py (render_transcript) |
✅ resolved |
Accessibility — ACCESSIBILITY-STANDARD |
Color independence | REVIEW | manual SR review (NVDA, VoiceOver) | — exempt (not an unconditional AUTO claim) |
Accessibility — ACCESSIBILITY-STANDARD |
ACR (VPAT 2.5 Rev 508) | REVIEW | docs/accessibility/ACR.md |
— exempt (not an unconditional AUTO claim) |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
App-security level | REVIEW | review-gate checklist | — exempt (not an unconditional AUTO claim) |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Dependency audit | AUTO | pip-audit (blocking in CI; never || true) |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Secret scanning | AUTO | gitleaks |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Static analysis | AUTO | semgrep + .github/workflows/codeql.yml |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
Actions pinning | AUTO | zizmor workflow-SAST scan (.github/workflows/ci.yml) |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
SBOM | AUTO | .github/workflows/release.yml (Generate SBOM step; never || true) |
✅ resolved |
Security and supply chain — SECURITY-AND-SUPPLY-CHAIN-STANDARD |
PII in logs | AUTO (never N/A) | obs.py _ALLOWED_FIELDS + Semgrep/bandit |
— exempt (not an unconditional AUTO claim) |
Internationalization — INTERNATIONALIZATION-STANDARD |
EN/ES key + placeholder parity | AUTO (wired 2026-07-08 — see FIX-02; previously declared AUTO with no such diff implemented) | tests/test_i18n_parity.py |
✅ resolved |
Internationalization — INTERNATIONALIZATION-STANDARD |
EN/ES eval per-case structural parity | AUTO | eval/suites/multilingual (also in the AI ledger above) |
✅ resolved |
Internationalization — INTERNATIONALIZATION-STANDARD |
EN/ES eval pass-rate parity (aggregate delta between languages) | AUTO (wired 2026-09-05; was "not implemented" per the 2026-08-29 correction — see the fuller row in the AI-evaluation ledger above for the measured number and the interval/case-mix caveats it comes with) | src/sprout/eval/suites/language_parity.py (language-parity, also in the AI ledger above) |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
First-token latency (offline) | AUTO (wired 2026-07-05 — previously declared AUTO with no test; see P1-10) | tests/test_latency.py |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
Reproducibility | AUTO | tests/test_eval_suites.py (test_run_is_byte_identical) |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
Versioning | AUTO (mechanism wired; never yet exercised — no tag has ever been cut, corrected 2026-07-05; see CHANGELOG.md) | .github/workflows/release.yml |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
Publish | AUTO (wired; unexercised — same caveat). The distribution name is not sprout: that name on PyPI belongs to an unrelated library, so a release under it was never possible. sprout-plantcare was free when chosen (2026-09-07). Publishing additionally needs a one-time PyPI Trusted Publisher registration, which is a web-UI action only the owner can take: project sprout-plantcare, owner ChelseaKR, repository sprout, workflow release.yml, environment pypi. Until it exists the publish-pypi job fails at the upload and nothing is published. |
.github/workflows/release.yml |
✅ resolved |
Quality, release, CI/CD — QUALITY-AND-METRICS · RELEASE-AND-VERSIONING · CI-CD-STANDARD |
CI parity | AUTO (wired 2026-07-08, closing ci-parity-no-mechanical-diff; the checker's first run also surfaced two real gaps — docs and the zizmor workflow-SAST scan were required by ci-gate but absent from make verify — now closed via new docs/workflow-lint prerequisites) |
sprout ci-parity-check (src/sprout/ci_parity.py, tests/test_ci_parity.py) mechanically diffs .github/workflows/ci.yml's required jobs against their Makefile target(s); run via make ci-parity-check (also a make verify prerequisite) and the ci-parity CI job (a ci-gate dependency) |
✅ resolved |