Guardrails

If a change can't be made safely, your request goes through untouched.

Quality

Optimizations decide by looking at the text's structure, never by asking another model, so the same input always produces the same result. A strategy runs only when its conditions match, and when a change might lose meaning, Anyray keeps the original instead of guessing.

Reversibility

When a retrieval-capable optimization drops content, it stores the original and leaves a short handle (ctx_…) in its place; fetch the original before its TTL expires with POST /v1/retrieve. Handles are opaque ids, safe to log and show in traces. Originals live in bounded memory; when durable recall is configured they may also be stored as AES-256-GCM ciphertext under the deployment's content mode (off mode writes nothing), and each strategy sets its own TTL.

Callers that cannot restore a handle in place get only self-contained transformations; a one-shot Batch request may instead use an explicit handle-free window crop to avoid a provider context-limit rejection.

Recover the original with POST /v1/retrieve

curl -X POST http://localhost:8088/v1/retrieve \
  -H "content-type: application/json" \
  -d '{"handle": "ctx_abc123"}'
# → { "handle": "ctx_abc123", "content": "…original…" }

Reliability

The optimizer is best-effort and fails open: a strategy that errors is skipped, and if the whole optimizer is down or misses its hard timeout, the gateway forwards your original request to the model unchanged. It can never break or stall a request.

The gateway also smooths over provider hiccups: it retries transient upstream errors (502 / 503 / 504, network drops) and rate limits (429, honoring the provider's retry-after), and can fall back across providers when configured.

Kill switch

Any optimization can be turned off at runtime, with no redeploy, through the admin settings API (GET / PUT /admin/optimizer/settings). Changes are audit-logged and can be scoped by endpoint, model, user, or team.

Quality audit

Two independent checks confirm optimization isn't degrading results. Both read request counts, token totals, latencies, status codes, and a cohort label, never content.

Quality parity from the holdout

Enable the holdout and Anyray runs a deterministic slice of your traffic unoptimized, with every mutating strategy suppressed. Assignment is stable per session and drawn within each user, so one heavy user cannot pull the whole tenant into the control arm, and a session never changes arms.

Four proxies are compared at 95% confidence: turns per session (the headline), output tokens per request, error rate, and latency. They resolve to one verdict: parity, regression, insufficient, imbalanced, or holdout_off. A regression needs a damaging interval and a delta above 10%, so a significant but trivial move stays parity.

Read it at GET /admin/spend/quality-parity. To measure one strategy instead of the whole optimizer, set holdout.kinds and read that arm with ?cohortKind=:

curl -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  "https://gateway.example.com/admin/spend/quality-parity?window=mtd&cohortKind=thinking_trim"

Per-session strategy gate

Every byte-changing strategy earns its place per tenant. Tenants that already had spend when the store first connected keep theirs on (grandfathered). Tenants first seen after that start in shadow.

A brand-new deployment is the exception: its store holds no spend at all when the gate first connects, so there is no warm session and no cost baseline to protect, and every strategy would otherwise sit at shadow for good. Those tenants are seeded on instead. The seed is not an override, so the evaluator can still demote a strategy that loses.

State	Where the strategy may run
off	Withheld, except in the bounded treated arm it keeps so it can earn its way back. force_off disables every run. A sufficient win returns automatic off to shadow.
shadow	Withheld, except in the same bounded treated arm. Sessionless or otherwise ineligible traffic stays withheld.
on	Eligible traffic, except sessions selected as this kind's holdout. The console distinguishes grandfathered, promoted, and forced approval.

holdout.fraction (default 5%, capped at 50%) is the per-kind measurement rate, so an unproven kind never runs on more than that share of sessions.

The evaluator reads seven days of evidence hourly. A win needs 30 sessions per arm, 30 matched pairs, a 95% Wilson lower bound above 53%, a Q3 cost ratio below one, and quality parity. A loss demotes on to shadow, and three losses in a row turn the strategy off. Insufficient evidence never changes state, and the console's Optimizer -> Session gate table shows the last verdict. Thresholds and every field: API reference.

Task-outcome regression guard

A per-tenant circuit breaker for lossy but reversible strategies. It watches for regret: a call to /v1/retrieve for an elided original, a tool re-run after its result became a marker, a request retried after a downshift or crop, or a call to a tool tool_pruning dropped. Past a threshold it suppresses that one strategy for that one tenant and emits a guard_suppressed decision. Every signal is attributed to the strategy that made the change, never to message text.

These are tripwires, not a quality measure, so the defaults are conservative and tunable:

Phase	Default
Rolling window	30 min (30 buckets)
Open (suppress)	≥ 50 fired and ≥ 10 regrets, at a ≥ 20% regret rate
Open (low traffic)	20 to 49 fired with ≥ 10 regrets, when a binomial test rules out chance at the 20% rate (p ≤ 0.01)
Open (accumulated)	evidence gathered across windows crosses its threshold, once at least 50 decayed trials and ≥ 10 regrets are on record
Cool-off	10 min, doubling on consecutive relapses (4x cap, and never past the rolling window), then half-open probing
Close (restore)	~1 in 10 eligible requests probe the strategy; restore after ≥ 20 clean probe fires over one probe-cohort rotation
Close (quiet lane)	14 clean probes at the default 20% regret threshold, after the minimum probe observation time

Quiet-lane recovery requires enough clean trials that their probability at the unacceptable regret rate is below 5%: ceil(log(0.05) / log(1 - regretThreshold)). Time alone cannot restore a lane. This is evidence for the circuit breaker, not a session-quality guarantee. Behaviour at or below the configured rate drains the score, so a healthy strategy never accumulates its way into suppression.

Boundary guards

• Token caps. An optional per-user monthly limit returns 402 once a user is over budget, before any model call. Unattributed or uncapped requests pass through.
• Request validation. Every request is checked against its schema at the edge, with guards that block requests to internal hosts.