Guardrails
If a change can't be made safely, your request goes through untouched.
Quality
Optimizations decide by looking at the text's structure, never by asking another model, so the same input always produces the same result. A strategy runs only when its conditions match, and when a change might lose meaning, Anyray keeps the original instead of guessing. An edit is also priced before it is made, and one that would not pay back even in a short session is skipped (pricing switches).
Reversibility
When a retrieval-capable optimization drops content, it stores the original and leaves a
short handle (ctx_…) in its place; fetch the original before its TTL expires with
POST /v1/retrieve. A handle is an opaque id bound to the user whose key
minted it. It resolves for that user's keys and their sub-agents. A handle an agent minted also
resolves for the person the agent is assigned to, unless the agent is a shared bot with delegated
attribution. Anyone else gets a 404. A handle minted with no attribution, or before handles were
bound to users, is readable across the org until it expires (each replay extends it), so treat it
like the content it points to. Originals live in bounded memory; when
durable recall is configured they may also be stored as AES-256-GCM ciphertext under the
deployment's content mode (off mode writes nothing), and each strategy sets its own TTL.
Callers that cannot restore a handle in place get only self-contained transformations; a one-shot Batch request may instead use an explicit handle-free window crop to avoid a provider context-limit rejection.
curl -X POST http://localhost:8088/v1/retrieve \
-H "content-type: application/json" \
-d '{"handle": "ctx_abc123"}'
# → { "handle": "ctx_abc123", "content": "…original…" }
Reliability
The optimizer is best-effort and fails open: a strategy that errors is skipped, and if the whole optimizer is down or misses its hard timeout, the gateway forwards your original request to the model unchanged. The customer Docker Compose stack does not wait for optimizer health before starting the gateway. The optimizer hook cannot break or stall a request beyond its bounded budget.
The gateway also smooths over provider hiccups: it retries transient upstream errors (502 / 503 /
504, network drops) and rate limits (429, honoring the provider's retry-after), and can
fall back across providers when configured.
Kill switch
Any optimization can be turned off at runtime, with no redeploy, through the admin settings API
(GET / PUT /admin/optimizer/settings). Changes
are audit-logged and can be scoped by endpoint, model, user, or team. A session with a warm prompt
cache keeps a disabled strategy's settled edits until the cache goes cold. To stop a strategy on
warm sessions too, use the per-session gate's force_off.
Quality audit
Two independent checks confirm optimization isn't degrading results. Both read request counts, token totals, latencies, status codes, and a cohort label, never content.
Quality parity from the holdout
Enable the holdout (the Measurement card on Console → Optimizer, or holdout.enabled in the
optimizer settings) and Anyray runs a deterministic
slice of your traffic unoptimized, with every mutating strategy suppressed. Assignment is stable
per session and drawn within each user, so one heavy user cannot pull the whole tenant into the control arm, and a session never changes arms.
Four proxies are compared at 95% confidence: turns per session (the headline), output tokens per
request, error rate, and latency. They resolve to one verdict: parity, regression,
insufficient, imbalanced, or holdout_off. A regression needs a damaging interval and a
delta above 10%, so a significant but trivial move stays parity.
Read it at GET /admin/spend/quality-parity. To measure one strategy
instead of the whole optimizer, set holdout.kinds and read that arm with ?cohortKind=:
curl -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
"https://gateway.example.com/admin/spend/quality-parity?window=mtd&cohortKind=thinking_trim"
An upgrade drops a holdout.kinds entry only when no config, rule, or request header can turn
that strategy on.
Per-session strategy gate
Every byte-changing strategy earns its place per tenant. Tenants that already had spend when the
store first connected keep theirs on (grandfathered). Tenants first seen after that start in
shadow.
A brand-new deployment is the exception: its store holds no spend at all when the gate first
connects, so there is no warm session and no cost baseline to protect, and every strategy would
otherwise sit at shadow for good. Those tenants are seeded on instead. The seed is not an
override, so the evaluator can still demote a strategy that loses.
| State | Where the strategy may run |
|---|---|
off | Withheld, except in the bounded treated arm it keeps so it can earn its way back. A sufficient win returns automatic off to shadow. force_off is the kill switch: no run anywhere, and a session with a warm prompt cache drops the strategy's settled edits and pins on its next turn (one cache rewrite). |
shadow | Withheld, except in the same bounded treated arm. Sessionless or otherwise ineligible traffic stays withheld. |
on | Eligible traffic, except sessions selected as this kind's holdout. The console distinguishes grandfathered, promoted, and forced approval. |
sessionGate.measurementFraction is the per-kind measurement rate (maximum 0.5). Left unset, it
is 0 for a tenant seeded on. A tenant that starts in shadow gets a draw sized on its own
traffic, so its strategies can earn their way on:
| Tenant | Unset draw |
|---|---|
| Under 7 days of history | 0.05 |
| 7 days or more | The smallest step from 0.01 to 0.065 that fills each strategy's arms in one evidence window (windowSeconds) to detect a 10% cost saving at alpha with 80% power (806 sessions per arm at the defaults) |
Volume is the tenant's distinct sessions over the 7 full UTC days before today. At five strategies,
2,000 sessions a day draws 0.015 (7% of sessions run an unproven strategy). 20 a day draws the
0.065 cap (25% of sessions), about 30 sessions per arm per 30-day window.
Until a shadow strategy on such a tenant reaches a verdict, its evidence pools across windows
instead of restarting each one. Checks fall due as the evidence doubles in age, and alpha covers
all checks of the pool together. A verdict, or a change to the strategy's policy, starts a new pool.
At 20 sessions a day a 30% saving is decided in about 3 windows and a 10% saving can take years.
The gate draws separate control and treated slices of that size, and an unproven kind runs only in
its treated slice. The audited holdout uses holdout.enabled
and holdout.fraction.
sessionGate.measurementFraction: 0 stops the gate draw. Strategies already on then run
unmeasured (evidenceStatus: session_gate_disabled). Strategies in shadow or off never run and
are never measured (evidenceStatus: withheld_unmeasured). The audited holdout has its own switch.
A strategy whose sessions split across client tools is also judged per tool, so a loss on one tool turns it off there only. The API reference has the unit names.
A strategy turned on per request with the
x-anyray-optimization-kinds header is
gated and measured the same way. It is drawn only in sessions where no configured strategy was
drawn.
Two lanes outside the optimizer change bytes too, and the gate covers them with the same draw and evaluator:
| Kind | Lane | Drawn per |
|---|---|---|
connect_hook | Connect's PostToolUse trims. In the holdout, Connect forwards the output untrimmed. | Claude Code session. Codex and Cursor are in neither arm. |
tool_defer | The gateway's tool deferral. The arm is held with the session's deferral mode. | Session |
Both start on on every tenant: the gate can demote them on evidence but never withholds them
unmeasured. A key's connect_hook rows count only from the UTC day after its Connect first honored
the draw, and only while that Connect holds the current measurementFraction. A tool deferral
session counts only when the mode it holds matches its arm.
The evaluator checks hourly. It ranks whole sessions in the control and treated arms on cost
(adjusted for task size), errors, turns and output. A strategy is demoted only on a significant
loss and promoted only on a significant cost win, at an error rate alpha (default 5%) spread
across every look. Anything else changes nothing. A loss demotes on to shadow, and three losses
with no win between them turn the strategy off. The console's Optimizer -> Session gate table
shows the last look.
Thresholds and every field: API reference.
Task-outcome regression guard
A per-tenant and client-tool circuit breaker for lossy but reversible strategies. It watches for regret: a
call to /v1/retrieve for an elided original, a tool re-run after its result became a marker, or
a request retried after a downshift or crop. Past a threshold it suppresses that strategy for that client tool within the tenant and emits a guard_suppressed
decision. Every signal is attributed to the strategy that made the change, never to message text.
With the spend database configured, replicas share the guard's counts and state.
Existing rows without a client tool remain in the unknown family.
A new edit counts as one fire; replaying it later adds none. A file re-read is a take-back only
when its line range overlaps the elided span. The default net_outcome basis counts a retrieval
only when the edit's saved cache reads do not cover its retrieval cost, summed across replicas
when the durable context store is set. Set regretBasis: retrieve to count every model retrieval.
Savings are credited to each handle's emitted or replayed edit, rather than split equally between
handles. Discarded edits earn no credit; separate spans sharing a handle add their savings.
These are tripwires, not a quality measure, so the defaults are conservative and tunable. The guard counts sessions, not edits: a window's effective sessions is its edits squared over the sum of each session's edits squared, so a session that fires 30 edits weighs as one. Regrets scale the same way, so one confused session cannot open the guard for the whole tenant.
| Phase | Default |
|---|---|
| Rolling window | 30 min (30 buckets) |
| Open (suppress) | ≥ 4 edits and ≥ 2 regrets, when an exact binomial test over the effective sessions puts the regret rate above 20% |
| Open (accumulated) | evidence gathered across windows, each edit weighted by its window's sessions per edit, crosses its threshold |
| Cool-off | 10 min, doubling on consecutive relapses (4x cap, and never past the rolling window), then half-open probing |
| Close (restore) | the probes show the rate is below 20% at 95%: 14 clean probe sessions, or 1 regret in 22 |
| Reopen | ≥ 4 probe edits at a ≥ 20% probe regret rate |
| Resolve (quiet timeout) | one rolling window (30 min) into half-open without a verdict: any probe regret reopens it, none closes it |
The test's confidence and the probe share follow the tenant's own traffic:
| Number | How it is set | Low traffic (10 to 40 sessions) | High traffic (300 to 400 sessions) |
|---|---|---|---|
| p-value to open | 0.01 per window, split over 1 + ln(0.2 × edits) looks | 0.0032 (40 edits) | 0.0013 (3,200 edits) |
| Probe share | enough sessions for twice the 14 recovery needs, between 1 in 10 and 1 in 2 | 1 in 2 | 1 in 10 |
| Opens at threshold | measured share of windows a strategy at 20% opens | under 1% | under 2% |
A tenant with fewer than about 28 sessions in a window cannot show 14 clean probe sessions in one, so its half-open ends as a quiet timeout.
tripTest: raw_ratio restores the earlier rule: open at ≥ 10 edits and a ≥ 20% raw rate (or the
binomial test below 10 edits), every edit its own trial, 1 in 10 probes, close after 4 clean probes.
The raw rate opened a strategy with a true 10% regret rate in about a quarter of 10-edit windows.
Quiet-lane recovery requires enough clean trials that their probability at the unacceptable regret
rate is below 5%: ceil(log(0.05) / log(1 - regretThreshold)). A probe fires only when the
strategy's own gates admit an edit, so a probe cohort can go a whole window without firing. After
one rolling window (never less than the minimum probe observation time), half-open reopens on any
probe regret and otherwise closes with reason quiet_timeout. That close certifies nothing: the
accumulated cross-window evidence is kept and every Open arm above stays armed.
This is evidence for the circuit breaker, not a session-quality guarantee. Behaviour at or below the configured rate drains the score, so a healthy strategy never
accumulates its way into suppression.
Boundary guards
- Token caps. An optional per-user monthly limit returns
402once a user is over budget, before any model call. Unattributed or uncapped requests pass through. - Request validation. Every request is checked against its schema at the edge, with guards that block requests to internal hosts.