Agent accuracy benchmark

The benchmark is built and runs weekly. Results appear here after the first full run, with accuracy, tool calls and tokens per task for each model, with and without our tools.

Tasks

Realistic questions an agent gets asked: billable ICD-10-CM codes, ICD-10-PCS and HCPCS Level II lookups, RxNorm and NDC mapping, NPI checks, dataset freshness, and requests it must decline (CPT and CDT codes, which we don't license).

With tools

The model gets our MCP server, built from the same commit, and its published docs. Nothing else. It calls the live API.

Without tools

The same model and the same questions, with no tools: what it knows on its own. The gap between the two is the lift.

Grading

Deterministic checks on the final answer (patterns it must and must not contain), never another model's opinion. With tools, a task also needs the right tool to have been called, and fails if patient details reach a tool call.

Cost

Tool calls per task count every call the model makes. Tokens per task count every input and output token, cached or not, including the model's reasoning.

Release gate

The benchmark runs weekly and after every release that changes the API, the MCP server or the task suite. If accuracy with tools drops more than 2 points for any model, the release fails the gate and we open an issue to fix it.