Evaluation Methodology & Fact Sheet

Evaluation Basis

• Measurement campaign conducted on 2026-09-27 (UTC) against live Anthropic API endpoints (Claude Sonnet 5 and Claude Haiku 4.5), driven by the Claude Code command-line agent, not a mock or simulated backend. Other agents and other model providers have not been measured.
• 444 runs recorded. 6 were set aside because the cache state seen on the first call did not match the plan, and 83 were excluded (below), leaving 355 kept as clean, comparable measurements: 176 on Claude Sonnet 5 and 179 on Claude Haiku 4.5. A “run” is one task executed once; the 355 kept runs contain 2,390 individual API calls.
• 83 runs excluded from the results: 74 were superseded/retired task-wording versions (a task prompt was revised mid-campaign, both old and new versions can't be mixed under one task id), 8 were a PII probe task intentionally excluded from savings numbers, and 1 was excluded for a cross-task answer contamination incident.
• Task sizes span three bands — Small (light chat / status checks), Medium (Excel/PDF/invoice-scale documents), Heavy (SQL joins, directory scans, Parquet merges) — each run against both an unaided baseline and Opto, on both a cold cache and a warm cache, on both models.

Query Size Range

Across all 355 kept runs (both models), individual run size ranged:

Reduction Range, By Query Type

Opto's advantage is not a single number — it depends heavily on query size and whether the cache is warm. Per-run average reduction, unaided vs. Opto, across the six measured size/cache-state combinations. Claude Sonnet 5 (176 runs):

Claude Haiku 4.5 (179 runs):

Sonnet: token reduction 57.5% – 81.4%, cost reduction 14.3% – 79.1%. Haiku: token reduction 57.1% – 81.5%, cost reduction 34.9% – 80.7%. Cost reduction narrows on warm-cache Medium/Heavy queries because Anthropic's own cache discount already lowers the unaided baseline's price — Opto's token savings are still real and large there, they just translate into a smaller dollar gap once the baseline itself gets cheaper.

Simulation Basis

The headline figures (46% lower spend, 68% fewer tokens) come from a simulation of a 50-person organization, not from a customer deployment.

• 30 simulated workdays (seeds 1001 to 1030), each 600 minutes (8:00 AM to 6:00 PM) with morning ramp, lunch dip and afternoon peak. Each day's workload is generated once and replayed identically through all four cases, so every comparison is like-for-like. Results are averages across the 30 days.
• Query sizes come from the measured campaign above. The workforce mix, query mix, session lengths and daily pattern are modeled assumptions, set in a published configuration file.
• Opto and the cost governor are separate. Opto trims what a model is sent and works with any model. The cost governor is an optional feedback controller that runs in a gateway and steers each request toward a frontier or cheaper model to follow a daily spend target (this simulation uses $55 a day). It steers; it does not block requests or enforce a cap.
• Reproducible. The full run replays bit for bit from its recorded seeds (120 of 120 case runs identical).

With Opto and the governor together, the busiest five minutes of the day fall by about 70%, and daily spend varied far less from day to day than without. The governor saves money by serving some frontier-model requests with a cheaper model: about 85 of roughly 234 per day in this simulation. It does not look at what a request is when choosing, so which requests are downgraded is an area still being improved.

Not modeled: rate-limit rejections, user retries, and the cost of waiting. Retries and waiting would add to the unaided baseline, so leaving them out understates the savings; rate-limit rejections are not part of the results.