Benchmarks

Reproducible numbers: the real optimizer over fixed synthetic payloads.

These isolated request measurements show transformation efficiency, not total session spend. Provider prompt-cache writes, persistent client transcripts, retrieval turns, and tool reruns can dominate the bill, so session-level regressions are monitored separately.

For the numbers behind every claim here, and to reproduce them, see the benchmarks repository.

Benchmarks

How it's measured

1

Two configs per workload

Each workload runs as control (the baseline) and optimized (the workload's strategy, pinned at one knob). The Anyray analog of a before/after.

2

Optimizer

The harness calls a live optimizer over its HTTP contract — pin one strategy via /admin/optimizer/settings, run the request through /v1/optimize, read back the transformed request. No optimizer internals are vendored.

3

Token accounting

Token counts are real BPE tokens (gpt-tokenizer o200k_base, the GPT-4o/4.1 family), measured the same way for the control and optimized runs.

Patterns

The common token-waste patterns users and coding agents fall into every day — the ones Anyray benchmarks against.

Waste pattern	Why it's costly
Paste a whole log or data dump and ask one question	the single most common thing engineers do with an assistant — billed in full, every turn
Agents re-read entire files when the question needs signatures	~70% of a coding agent's tokens are irrelevant file reads
Tool-schema bloat rides along every call	55k+ tokens of tool definitions before the first message
RAG over-retrieval	3–5× more chunks fetched than the answer uses
Agents resend the whole history every turn	the #1 pattern — 4–15× chat tokens; a 50-turn session bills ~25:1 input:output
Redundant / near-duplicate requests	40–60% of enterprise traffic is repetitive (18% exact, ~47% similar)

Headline savings

Whole-request input-token reduction, measured by running each workload's strategy through a live optimizer at a fixed knob. Grouped into suites:

Suite	Workloads	Before (tok)	After (tok)	Saved
Logs & structured data	3	126,998	18,383	86%
Code context	5	15,161	7,745	49%
Tools & retrieval	3	12,953	3,354	74%
Agent operations	3	89,013	24,033	73%
Cross-session memory recall	5	18,809	3,180	83%

Run it yourself

The suite needs a reachable optimizer (the Anyray stack brings one up in-network on :8088) and its admin token. Then:

Run the benchmarks

git clone https://github.com/anyrayHQ/benchmarks && cd benchmarks
cp .env.example .env          # set ANYRAY_OPTIMIZER_URL + ANYRAY_ADMIN_TOKEN
./run.sh --all                # writes <suite>/results/*.json

Each suite writes results/control.json and results/optimized.json — the committed, real scores. Full per-workload tables and a live-provider cross-check are in the repo's RESULTS.md.