Case study · RFC
Sentinel
An LLM gateway that fails a PR when your model gets worse.
§ 01 · problem
Prompt and model changes get merged on vibes. There's no CI signal when a tweak quietly makes answers less grounded or more hallucinated — until someone complains in Slack.
§ 02 · constraints
- Zero-config drop-in: an app points at Sentinel and keeps its OpenAI/Anthropic SDK call.
- Logs every request with cost, latency, and both sides of the conversation.
- Runs eval checks on the live response, not just a canned set.
- A GitHub Actions job can call `sentinel verify` on a branch and get a pass/fail.
§ 03 · architecture
The system, from the outside in.
Click a component for the trade-off and a snippet.
§ 04 · decisions
Why not a Python gateway?
Hot path is p99 latency. Go's runtime makes the proxy cost predictable; evals stay in Python where I already have LangGraph tooling.
Why fail a PR, not a dashboard?
A dashboard you look at sometimes. A failing check you can't ignore. The point is to put the eval signal where code review already lives.
§ 05 · results
- Design complete; proxy and trace schema working end-to-end on OpenAI.
- Eval worker runs groundedness + a hallucination check from my resume-reviewer research on sampled traffic.
- GitHub Action POC fails on a seeded regression.
§ 06 · next
- Harden for streaming responses.
- Add Anthropic + OpenRouter adapters.
- Public v0 release and a docs page about the eval taxonomy.