Evaluation

Quality preservation is measured against an unoptimized control arm on your own traffic — not asserted.

Trimming input tokens is only worth it if the answers stay as good. Four layers check that, and each one can fail visibly with a named cause.

Layer	What it establishes	Where it runs
Construction	A transformation is deterministic, structural, and reversible — or it doesn't run.	Every request
Pre-ship suite	Properties CI re-checks on every change: never inflate, never nest a handle, never lose coverage.	Every pull request
In-flight tripwires	Content-free evidence that a specific trim cost the model something — and an automatic stand-down.	Every request, per tenant
Randomized holdout	Outcome parity against a byte-identical control arm, on your traffic, with 95% CIs.	Your deployment, on demand

No layer reads a prompt or a response. Everything here is metadata — counts, tokens, latency, status codes, and a cohort label — so the evaluation itself never violates the content-privacy boundary.

Layer 1 — preserved by construction

Some quality risks are removed by how the optimizer is built, not by measuring afterward.

• Decisions are structural, not generative. A strategy decides from the text's shape — message roles, tool-output size, duplicate spans, token budgets. Same input, same output, every time. No model is asked what to keep, so there is no sampling variance to evaluate in the first place.
• No second model on the path. The two components that could have been model calls aren't: vision_ocr runs a local Tesseract WebAssembly worker, and semantic recall uses a local embedder. Both degrade to a clean pass-through when unavailable.
• Lossy means reversible. When a retrieval-capable strategy drops content it stores the original and leaves a content-free ctx_… handle. The model pulls it back through POST /v1/retrieve if it turns out to matter — see Reversibility.
• Anything unsafe is skipped, not guessed. A strategy whose preconditions don't match passes the request through untouched, and the whole optimizer fails open.
• The control arm is byte-identical. Holdout requests skip every mutating strategy, so the comparison below is against the request you would have sent without Anyray — not a reconstruction of it.

Layer 2 — the pre-ship regression suite

Determinism only helps if the deterministic thing is still correct after the next change. These are properties, not example outputs, and CI re-checks them on every pull request across the optimizer and gateway test suites.

Property pinned in CI	The regression it prevents
Never inflate	Every strategy, run against the most wasteful request it exists to fix, must leave the request no larger than it found it.
Savings floor	The same run must still clear a conservative floor — a strategy that silently degrades to ~0% saved fails the build rather than quietly shipping.
Coverage superset	A strategy that replaces another must handle everything the old one did; code_graph is pinned as a true superset of the skeletonizer, byte-for-byte on the empty-working-set case.
Marker protection	A strategy that meets a handle another strategy already left must pass it through verbatim. Nesting handles would hand a retrieving agent a second marker instead of the original.
Cross-package parity	The gateway, optimizer, and connect each hold their own copy of the retrieve-capability contract; a test reads the other packages' files and fails when they drift. A flat scan that missed nested tool namespaces once cost the retrieval evidence entirely.
Cache safety	Turn-varying rewrites are pinned against the provider prompt cache, since a busted cache costs far more than the trim saves.
Content-free decisions	Every decision a strategy emits must carry a non-empty, content-free summary and finite numeric signals.

What the offline benchmark is, and isn't

Benchmarks measures compression ratio on fixed synthetic payloads. It is deliberately not a quality benchmark — it makes no accuracy claim, and no number on that page should be read as one. Quality evidence comes from the layers on this page.

Layer 3 — in-flight regret tripwires

A test can't answer whether a specific trim, on a specific tenant's traffic, cost the model something it needed. Four content-free signals stand in for that — a retrieved original, a tool re-run with identical arguments, a retry after a downshift or window crop, and a tool the model reached for after tool_pruning dropped it. When the rolling regret rate crosses its threshold, the breaker suppresses that one strategy for that one tenant and emits a guard_suppressed decision naming why.

Full thresholds, phases, and per-strategy coverage: Task-outcome regression guard.

These are tripwires, not a quality measure. They catch a strategy that is actively losing information; they say nothing about whether answers are as good. That needs a control group.

Layer 4 — measured on your own traffic

The audited holdout turns the quality claim into an A/B test your own deployment runs and you read the result of.

How assignment works

Enable the holdout and a deterministic slice of attributable traffic is assigned to a control arm that runs unoptimized — every mutating strategy suppressed, the request forwarded byte-identical. Everything else is the treated arm. Assignment is a stable hash per user (falling back to team, then session), so a given developer stays in one arm rather than flipping mid-session, and their spend rows can be compared directly.

Assignment is per identity, not per request. That's deliberate — a session split across both arms would compare nothing — but it is also why the balance check below exists.

What's compared

Four outcome proxies, each a way a quality drop would surface in metadata:

Proxy	A regression looks like…	Counts toward the verdict
Turns per session	Worse answers make users and agents loop more, so treated sessions run longer.	Yes — the headline
Output tokens per request	Degraded context tends to produce longer, more hedged replies.	Yes
Error rate	A higher share of 4xx / 5xx responses.	Yes
Latency	Higher wall time per request.	No — the treated arm carries the optimizer's own overhead by design, so it's shown, never judged

The statistics

The read has to hold in both directions: no false alarm off a noisy week, and no clean bill of health off eight requests.

Confidence intervals, not raw deltas

Continuous proxies (turns per session, output tokens per request) get a Welch 95% interval on the difference of means — unequal variances, unequal sample sizes, no pooled-variance assumption. Error rate, a proportion, gets a two-proportion 95% interval. A metric with no sample-stddev basis (n < 2) reports no interval rather than a fabricated one.

Student-t, not a flat 1.96

The critical value is the two-sided 95% Student-t quantile at the Welch–Satterthwaite degrees of freedom — exact table values through df 30, interpolated in 1/df beyond, tending to 1.96 as df → ∞. A hard-coded 1.96 understates uncertainty on small arms, which is precisely where a false regression would be minted.

Significant is not enough — it must also be material

With a large enough sample any trivial difference turns "statistically significant." A proxy counts against optimization only when its interval excludes zero in the worse direction and the delta exceeds 10%. The one exception: a zero-error holdout baseline with any treated errors is the most extreme relative regression there is, so it always counts.

Sample gates

Below 200 requests per arm the read refuses to call it either way. Turns per session additionally needs 30 sessions per arm — per-session turn counts are heavy-tailed, and below that a t interval rests on assumptions the data can't support, so the metric reports no basis rather than a shaky verdict.

Workload balance

Because assignment is per identity, a few heavy users can load one arm with different work. The read compares average input tokens per request across the arms and refuses to judge outside a 0.8–1.25 ratio — outcome deltas there would reflect workload, not the optimizer.

The verdict

Precedence runs top to bottom; the first condition that matches wins.

Verdict	Meaning
holdout_off	No cohort rows and the optimizer confirms the holdout is disabled. An unreachable optimizer never yields this — it would tell you to enable something that may already be on.
insufficient	Fewer than 200 requests in an arm. Wait for traffic.
imbalanced	Average input tokens per request across the arms fall outside 0.8–1.25. No call is made.
regression	At least one verdict proxy is both statistically real and materially worse in the treated arm. The failing proxy is named.
parity	Every gate passed and no proxy regressed — optimization held quality over this window.

Read it

GET /admin/spend/quality-parity — admin-gated, requires the config:read capability, and accepts the same ?window= values as the spend dashboard (24h, 7d, 30d, 90d, mtd, prev-month; default mtd). An unknown window is a 400 rather than a silent widening.

Read the quality-parity verdict

curl -s "http://localhost:8787/admin/spend/quality-parity?window=30d" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN"

Response (abridged)

{
  "window": "30d",
  "from": "2026-06-29T00:00:00.000Z",
  "to": "2026-07-29T00:00:00.000Z",
  "holdout":  { "cohort": "holdout", "requests": 1841, "sessions": 96, "requestsPerSession": 19.2, "avgInputTokensPerRequest": 41302, "avgOutputTokensPerRequest": 612, "errorRate": 0.004 },
  "treated":  { "cohort": "treated", "requests": 34907, "sessions": 1783, "requestsPerSession": 18.7, "avgInputTokensPerRequest": 39118, "avgOutputTokensPerRequest": 604, "errorRate": 0.005 },
  "metrics": [
    { "id": "turns_per_session", "label": "Turns per session", "holdout": 19.2, "treated": 18.7, "deltaPct": -2.6, "ci95": { "lo": -1.94, "hi": 0.94 }, "countsTowardVerdict": true },
    { "id": "output_tokens_per_request", "label": "Output tokens per request", "holdout": 612, "treated": 604, "deltaPct": -1.3, "ci95": { "lo": -21.4, "hi": 5.4 }, "countsTowardVerdict": true },
    { "id": "error_rate", "label": "Error rate", "holdout": 0.004, "treated": 0.005, "deltaPct": 25.0, "ci95": { "lo": -0.002, "hi": 0.004 }, "countsTowardVerdict": true },
    { "id": "latency_ms", "label": "Latency", "holdout": 8420, "treated": 8611, "deltaPct": 2.3, "ci95": null, "countsTowardVerdict": false }
  ],
  "balance": { "inputTokensRatio": 0.947, "balanced": true },
  "sample":  { "minRequestsPerArm": 200, "sufficient": true },
  "verdict": "parity",
  "optimizerHoldout": {
    "enabled": true, "fraction": 0.05, "windowSeconds": 604800, "holdoutShare": 0.0501
  }
}

Note the error-rate row: a +25% delta that is not a regression, because its interval spans zero. Materiality and significance both have to hold.

The endpoint serves even when the optimizer is unreachable — optimizerHoldout degrades to null rather than failing the read.

Run it on your own traffic

1

Turn the holdout on

Default off Enable it in the top-level holdout block through PUT /admin/optimizer/settings — not in the strategies array, which the read-only audited_holdout marker belongs to.

PUT /admin/optimizer/settings

{
  "config": {
    "holdout": { "enabled": true, "fraction": 0.05, "windowSeconds": 604800 }
  }
}

fraction is the share held back — 5% by default. Higher arrives at a verdict sooner and costs more; see the trade-off below.

2

Confirm assignment is landing

GET /admin/optimizer/settings reports live holdout.arms counters. holdoutShare should track the configured fraction; if it doesn't, traffic isn't attributable (assignment needs the x-anyray-metadata identity).

3

Wait for volume, then read the verdict

Both arms need 200 requests — and 30 sessions each before the headline turns-per-session interval is emitted. Until then the verdict is honestly insufficient.

4

Cross-check the savings number separately

Quality parity says optimization didn't cost you answers. It does not audit the savings figure — for that, sample a shadow baseline, which measures the real trim next to the estimate on a sampled fraction, with no extra upstream call.

You have a verdict

GET /admin/spend/quality-parity returns "verdict": "parity" with sample.sufficient: true and balance.balanced: true — optimization held quality over that window, on your own traffic.

The holdout has a real cost

The control arm receives no trimming and no cache serving — you are deliberately paying full price on that slice of traffic to buy the measurement. At the 5% default that's a small, bounded cost; raise fraction only for as long as you need the read, then lower it again.

What this does not prove

• These are proxies, not graded answers. Turns per session, reply length, and error rate are how a quality drop shows up in metadata — they are not a correctness score. Anyray never reads your prompts or responses, so it cannot grade them. A content-level quality score would require reading content.
• Parity is a window, not a guarantee. A parity verdict covers the requests in that window on that traffic mix. It doesn't forecast a workload you haven't run.
• Latency is not judged. The treated arm carries the optimizer's own overhead by design. It's reported so you can see it, and never counted against optimization.
• Balance can defeat the read. With few attributed users, one heavy user can imbalance the arms; the read says imbalanced rather than pretending.
• The tripwires are asymmetric. They catch information loss, not subtle degradation that produces a confident, shorter, worse answer. That's exactly the gap the holdout exists to cover.

Guardrails

The runtime protections — reversibility, fail-open, the regret breaker, the kill switch.

Benchmarks

Reproducible compression numbers, with an explicit non-quality disclaimer.