Guardrails

If a change can't be made safely, your request goes through untouched.

Quality

Optimizations decide by looking at the text's structure, never by asking another model, so the same input always produces the same result. A strategy runs only when its conditions match, and when a change might lose meaning, Anyray keeps the original instead of guessing.

Reversibility

When a retrieval-capable optimization drops content, it stores the original and leaves a short handle (ctx_…) in its place; fetch the original before its TTL expires with POST /v1/retrieve. Handles are opaque ids, safe to log and show in traces. Originals live in bounded memory; when durable recall is configured they may also be stored as AES-256-GCM ciphertext under the deployment's content mode (off mode writes nothing), and each strategy sets its own TTL.

Callers that cannot restore a handle in place get only self-contained transformations; a one-shot Batch request may instead use an explicit handle-free window crop to avoid a provider context-limit rejection.

Recover the original with POST /v1/retrieve

curl -X POST http://localhost:8088/v1/retrieve \
  -H "content-type: application/json" \
  -d '{"handle": "ctx_abc123"}'
# → { "handle": "ctx_abc123", "content": "…original…" }

Reliability

The optimizer is best-effort and fails open: a strategy that errors is skipped, and if the whole optimizer is down or misses its hard timeout, the gateway forwards your original request to the model unchanged. It can never break or stall a request.

The gateway also smooths over provider hiccups: it retries transient upstream errors (502 / 503 / 504, network drops) and rate limits (429, honoring the provider's retry-after), and can fall back across providers when configured.

Kill switch

Any optimization can be turned off at runtime, with no redeploy, through the admin settings API (GET / PUT /admin/optimizer/settings). Changes are audit-logged and can be scoped by endpoint, model, user, or team.

Quality audit

Two independent checks confirm optimization isn't degrading results. Both read request counts, token totals, latencies, status codes, and a cohort label, never content.

Quality parity from the holdout

Enable the holdout and Anyray assigns a deterministic slice of traffic to a control arm that runs unoptimized: every mutating strategy suppressed, the request byte-identical at the provider. Assignment is stable per session and drawn within each user (or team): fraction sets the share of requests held out, not the share of people, so one heavy user cannot pull the whole tenant into the control arm. A session never changes arms, so the prompt prefix stays byte-stable and the arms behave like a randomized A/B test on your own traffic.

Four quality proxies are compared across the arms with 95% confidence intervals: turns per session (the headline), output tokens per request, error rate, and display-only latency. They resolve to one verdict: parity, regression, insufficient, imbalanced, or holdout_off. Turns per session and error rate are one-sided (only a higher treated value is damage). Output tokens per request is two-sided, because volume has no good end: far more can be repetition, and far less is what a trim looks like when it cost the model something it needed. A regression needs both a confidence interval excluding zero in a damaging direction and a delta above 10%, so a significant but trivial move stays parity.

The same response carries grossCostObjective: actual list-price provider cost per observed session in the window (input, output, cache reads and writes, every recorded turn), tested against a 30% reduction objective on the unoptimized arm. target_met requires quality parity plus a 95% confidence-interval lower bound reaching 30%; a point estimate alone returns target_not_proven. Workload balance uses input before optimization (billed plus removed), so a successful trim doesn't make the treated arm invalidate itself as smaller traffic.

Read it at GET /admin/spend/quality-parity. The admin settings response also reports per-arm request counts, input-token estimates, and holdoutShare.

Measuring one strategy

By default the control arm skips every strategy, so the verdict cannot say which one caused a difference. Set holdout.kinds in the optimizer config to one or more strategy ids and a control session keeps the rest of the pipeline, withholding only that strategy; each listed strategy draws its own independent arm, and read-only strategies are rejected because withholding one would compare two identical arms. Read a per-strategy arm with ?cohortKind=:

curl -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  "https://gateway.example.com/admin/spend/quality-parity?window=mtd&cohortKind=thinking_trim"

Omitting cohortKind keeps the whole-optimizer view, which excludes per-strategy control rows so the two experiments never mix. Testing one strategy at a time keeps both arms at full size.

Task-outcome regression guard

A per-tenant circuit breaker for lossy but reversible strategies. It watches for regret, signals that a trim cost the model something, and past a threshold it suppresses that one strategy for that one tenant until it looks healthy again, emitting a guard_suppressed decision so you can see why it stood down. Every signal is attributed to the strategy that made the change, never to prompt or response text. The regret signals:

• The model calls POST /v1/retrieve to pull back an elided original.
• A tool is re-run with the same arguments after its result was replaced with a retrieval marker.
• A reasoning downshift or window crop is followed by the same request retried in-session.
• The model invokes a tool that tool_pruning had dropped (a tool miss).

These are tripwires, not a full quality measure, so the defaults are conservative (all tunable via the settings API):

Phase	Default
Rolling window	30 min (30 buckets)
Open (suppress)	≥ 50 fired and ≥ 10 regrets, at a ≥ 20% regret rate
Open (low traffic)	20 to 49 fired with ≥ 10 regrets, when a binomial test rules out chance at the 20% rate (p ≤ 0.01)
Open (accumulated)	evidence gathered across windows crosses its threshold, once at least 50 decayed trials and ≥ 10 regrets are on record
Cool-off	10 min, doubling on consecutive relapses (4x cap, and never past the rolling window), then half-open probing
Close (restore)	~1 in 10 eligible requests probe the strategy; restore after ≥ 20 clean probe fires over one probe-cohort rotation
Close (quiet lane)	a lane too quiet to reach 20 probes is restored anyway after a long, entirely clean look

The low-traffic and accumulated rows protect quiet lanes: a strategy firing 20 times an hour never reaches 50 fired inside one window, and a rolling window discards its evidence on every roll. The binomial test suppresses a small sample only when its regrets are implausible at the configured rate (the rate and significance level are operator policy; neither adapts to the data it judges), and the decayed cross-window score lets sustained underperformance eventually suppress even a very quiet lane: tens of decisions for a badly broken strategy, roughly a day for one only slightly worse than policy. Behaviour at or below the rate drains the score, so a healthy strategy never accumulates its way into suppression.

Cool-off doubles only while probes confirm the regret is still live, and resets on the first clean recovery, so a relapsing strategy waits longer between probes instead of flapping. The quiet-lane restore keeps the flip side fair: a strategy too quiet to earn 20 probes would otherwise stay suppressed far longer than a busy one.

Boundary guards

• Token caps. An optional per-user monthly limit returns 402 once a user is over budget, before any model call. Unattributed or uncapped requests pass through.
• Request validation. Every request is checked against its schema at the edge, with guards that block requests to internal hosts.