MCP · LangGraph · Pydantic · Evals
A vehicle-diagnostic agent that answers with cited evidence — or refuses, grounded, when its data source genuinely can't answer. Built above a source-agnostic seam, so swapping OBD for raw CAN changes nothing above it.
The GenAI layers are kept ignorant of where a number came from. A normalizer sits between the data and the intelligence, so the tools, agent, and evals never learn whether a reading was an OBD PID or a decoded CAN frame. A seam-enforcement test fails CI if that knowledge leaks upward.
Everything shown below runs against the synthetic source — deterministic, so it doubles as the eval fixture. Phase 5 swaps in raw CAN behind the same normalizer; the eval harness is meant to keep passing unchanged. That's the proof the seam held.
A structured error taxonomy — not thumbs-up/down — so every disagreement points at which defense failed. Here is where the judge is trustworthy and where it isn't.
Perfect on the mechanically
checkable failures. Every single disagreement was an overconfident call — the
one judgment a rubric only partly disciplines. Read the judge as approaching, not
exceeding, the reliability ceiling of its ground truth.
The 85% above is the machinery on
hand-seeded labels; this is the first run on real review labels. Agreement is 100% on
every failure mode except false_refusal (50%) — the judge waved through
four refusals a human marked as answerable questions wrongly declined. No self-agreement
ceiling yet (needs review pass B). The same over-refusal appears independently in the
regression suite (scripts/eval.py → 3/6 pass, all three failures answerable
questions refused).
uv run python scripts/calibrate.py --real.Reproducible with no API key: uv run python scripts/calibrate.py.
The seed labels are hand-authored (a solo project has no panel) — the calibration machinery
is the deliverable, and it firms up as real reviewed failures land (the callout above is the
first of those).
Every trace below is an actual run pulled from data/evals/traces/
— the full tool-call record, the structured answer or the grounded refusal. Nothing here is
mocked for the demo. Pick a question: