Guardrails

If a change can't be made safely, your request goes through untouched.

Quality

Each optimization either applies safely or not at all, so answer quality is preserved.

• Same input, same result. Optimizations decide by looking at the text's structure — never by asking another model — so there's no randomness.
• Only when it fits. A strategy runs only when its conditions match; otherwise it passes the request through unchanged.
• When unsure, keep the original. If a change might lose meaning, Anyray keeps the original instead of guessing.

Reversibility

When a retrieval-capable optimization drops content, it stores the original and leaves a short handle (ctx_…) in its place. Fetch the original before its TTL expires with POST /v1/retrieve. Callers that cannot restore a handle in place use only self-contained transformations; a one-shot Batch request may instead use an explicit handle-free window crop to avoid a provider context-limit rejection.

• Handles carry no content — safe to log and show in traces.
• Originals live in bounded memory. When durable recall is configured, they may also be stored as AES-256-GCM ciphertext under ANYRAY_CONTENT_MODE; off mode writes nothing. Each strategy sets a TTL.
• The model sees trimmed context; a later step pulls the original back only if it's needed.

Recover the original — POST /v1/retrieve

curl -X POST http://localhost:8088/v1/retrieve \
  -H "content-type: application/json" \
  -d '{"handle": "ctx_abc123"}'
# → { "handle": "ctx_abc123", "content": "…original…" }

Latency

The optimizer runs before each request with a hard 800 ms timeout (ANYRAY_OPTIMIZER_TIMEOUT_MS). Miss it and the original request goes through unchanged — it can never stall a request.

When it does run, it usually adds ~1–2 seconds. That normally pays back: cleaner context means more accurate answers and fewer back-and-forth loops, so total task time often goes down. Cache hits are faster still — a match serves a stored response and skips the model entirely.

Reliability

The optimizer is best-effort and fails open. A strategy that errors or times out is skipped; if the whole optimizer is down, the gateway forwards your original request to the model, unchanged. It can never break or stall a request.

The gateway also smooths over provider hiccups: it retries transient upstream errors (502 / 503 / 504, network drops) and rate limits (429, honoring the provider's retry-after), and can fall back across providers when configured.

Kill switch

Any optimization can be turned off at runtime — no redeploy — through the admin settings API (GET / PUT /admin/optimizer/settings). Changes are audit-logged and can be scoped by endpoint, model, user, or team.

Quality audit

Two independent checks confirm optimization isn't degrading results. Both run on metadata only — request counts, token totals, latencies, status codes, and a cohort label — never your prompts or responses.

Quality parity from the holdout

Enable the holdout and Anyray assigns a deterministic slice of traffic to a control arm that runs unoptimized — every mutating strategy is suppressed, so the request reaches the provider byte-identical. Everything else is the treated arm. Assignment is stable per user (falling back to team, then session), so the two arms behave like a randomized A/B test on your own traffic and their spend rows can be compared directly.

Anyray compares four outcome proxies — each a way a quality drop would show up in the numbers:

Proxy	A regression looks like…
Turns per session (headline)	Worse answers make users and agents loop more, so treated sessions run longer.
Output tokens per request	Degraded context tends to produce longer, more hedged replies.
Error rate	A higher share of 4xx / 5xx responses.
Latency	Display-only — the treated arm carries the optimizer's own overhead by design, so it's shown but never counts toward the verdict.

A proxy only counts against optimization when the gap is both statistically real and material (over 10%) — a small or noisy difference doesn't. The read resolves to a single verdict:

Verdict	Meaning
holdout_off	No control traffic — enable the holdout to start collecting.
insufficient	Fewer than 200 requests in an arm — wait for more traffic. (Turns per session also needs ≥ 30 sessions per arm.)
imbalanced	Average input tokens per request across the arms fall outside a 0.8–1.25 ratio — the arms aren't carrying comparable work, so no call is made.
parity	No proxy regressed — optimization held quality.
regression	At least one proxy is materially worse in the treated arm; the failing proxy is named.

Read it at GET /admin/spend/quality-parity, or in the console under Observability.

Task-outcome regression guard

A per-tenant circuit breaker for lossy but reversible strategies. It watches for regret — content-free signals that a trim cost the model something — and, when they cross a threshold, suppresses that one strategy for that one tenant (emitting a guard_suppressed decision so you can see why it stood down) until it looks healthy again. Every signal is attributed to the strategy that made the change, never to prompt or response text.

Regret signals:

• The model calls POST /v1/retrieve to pull back an original that was elided.
• A tool is re-run with the same arguments after its earlier result was replaced with a retrieval marker.
• A reasoning downshift or window crop is followed by the same request being retried in-session.
• Tool miss — the model invokes a tool that tool_pruning had dropped from the request.

These are tripwires, not a full quality measure, so the defaults are conservative (all tunable via the settings API):

Phase	Default
Rolling window	30 min (30 buckets)
Open (suppress)	≥ 50 fired and ≥ 10 regrets, at a ≥ 20% regret rate
Cool-off	10 min, then half-open probing
Close (restore)	~1 in 10 eligible requests probe the strategy; restore after ≥ 20 clean probe fires over ≥ 2 min

Boundary guards

• Token caps. An optional per-user monthly limit returns 429 once a user is over budget, before any model call. Unattributed or uncapped requests pass through.
• Request validation. Every request is checked against its schema at the edge, with guards that block requests to internal hosts.