Benchmarks
Does the same coding task cost less through Anyray, with the same result?
The paired agent bench runs real coding-agent sessions twice: direct to the provider and through Anyray. Both arms get the same task and pinned repository, each in a fresh checkout. The agent chooses its own steps. Cost includes the whole session, including subagents and prompt-cache reads and writes.
The end state decides whether the task was solved: tests pass, required facts appear in the answer, or source citations resolve. Only pairs where both arms solved the task count toward cost results. A cheaper failure is never a win.
Reading the verdict
Rule 0 is that a session must never cost more because of Anyray. Its verdict requires all of these:
| Check | Pass when |
|---|---|
| Win rate | Anyray is cheaper in at least 63% of rounds, above the 53% noise floor |
| Third-quartile cost ratio | Anyray / direct is below 1 |
| Quality parity | Anyray solves as many rounds as direct |
Short, cheap tasks sit at parity by design. Run-to-run variance on direct alone is large, so a few runs cannot establish a difference.
Latest results
The 2026-10-15 results compare direct Amazon Bedrock sessions with a development Anyray gateway on Claude Sonnet 5. There was one round per scenario, with a rerun when a pair was unusable. This is a spot check; the separate paired cost verdict needs at least 8 valid rounds per scenario.
Ratios are total session cost through Anyray divided by direct cost. Lower is cheaper.
| Scenario | Anyray / direct |
|---|---|
cobra-flag-groups, 200-tool MCP seat | 0.21× |
pyrepo-pillar-docs | 0.24× |
cobra-dispatch, 200-tool MCP seat | 0.25× |
sdk-docs | 0.27× |
framework-docs | 0.35× |
pyrepo-docs | 0.35× |
gin-test-triage | 0.36× |
pyrepo-state-docs | 0.52× |
gin-long-session | 0.63× |
cobra-pause | 0.76× |
cobra-dispatch | 0.78× |
pyrepo-loader | 0.80× |
cobra-flag-groups | 0.96× |
gin-doc-audit | 1.05× |
cobra-web-flags | 1.60× |
pyrepo-long-session | Not counted |
The short tasks (cobra-flag-groups, cobra-dispatch, pyrepo-loader) are close to parity;
an optional agent step can outweigh their small savings. gin-doc-audit cost more when the
direct arm finished without subagents. cobra-web-flags cost more because loading a deferred
tool triggered an extra model step. pyrepo-long-session was excluded because the direct arm
wrote outside its checkout. Both arms solved every round.
Replay suite
The replay suite measures single-request compression on fixed payloads, not session cost. It has no session turns or prompt-cache effects. Its reductions are not evidence of savings.