Optimizer
On every request the gateway hands it, the optimizer runs an ordered pipeline of strategies: small, self-gating transforms (context compression, relevance filtering, tool pruning, semantic caching, and more), each deciding on a best-effort basis how to serve the request before passing it to the next.
It's a standalone HTTP service (optimizer:8088, internal-only). The gateway reaches it
through a thin adapter (gateway/src/services/optimizer.ts) over HTTP only, and never imports
its internals. Two rules make it safe on the hot path: it does nothing unless
ANYRAY_OPTIMIZER_URL is set, and any timeout or error serves the request without it
(fail-open). That's why it sits beside the request path as a
decision engine instead of in front of it as a second gateway.
The flow
- Before routing, the gateway calls POST
/v1/optimize. - The optimizer runs the ordered pipeline from
optimizer.config.json; each org composes its own set from the library. - Each enabled strategy runs in declared order, threading its (possibly transformed) request into the next. Before strategy execution the pipeline applies the task-outcome regression guard, a per-tenant and client-tool breaker that suppresses a lossy or steered strategy while its tripwire signals show elevated regret.
- After the response, the gateway records a spend row. On a
cacheEligiblemiss, and only then, it also writes the response back via POST/v1/cacheso the next identical call can hit. On a lane the cache can never serve (a streamed response, a caller that cannot short-circuit) the write-back is measurement-only:shadowOnly: true.
What it returns
/v1/optimize returns a (possibly) transformed request plus decisions:
| Outcome | Meaning |
|---|---|
| request unchanged | No strategy changed it, so the gateway forwards as-is. |
| transformed request | A strategy rewrote messages, tools, or params, so the gateway forwards the returned request. |
| cacheHit | When the caller canShortCircuit, semantic_cache may return cacheHit:true + cachedResponse to serve directly. |
Only exact semantic-cache hits serve. Pressure-based strategies use the remaining input budget and observed exchange growth; cached-history mints additionally need trusted route and pricing evidence. Explicit legacy overrides remain supported. The strategy reference describes the individual decisions and limits.
Each decision carries kind, a human-readable summary, token/cost estimates, and sometimes a
numeric metric ({ name, value }). A decision never carries prompt content, so the console can
show them in every content mode. Full wire shapes:
Optimizer Protocol.
Prompt caching
Providers cache the start of each request. When the next request begins with the same bytes, that part is a cache read, which Anthropic bills at 0.1x the input price. From the first changed byte on, the provider processes the rest again and writes it to the cache at 1.25x (2x for the hour-long cache).
Anyray edits only text the provider has not cached yet, and resends every earlier edit byte for byte, so the cached part is read back, not rewritten.
Fail-open is a hard invariant
The optimizer can never break a request. The gateway calls it with a hard 1200 ms timeout (up to 2.5 times that on large requests) and forwards the request without it on any failure. Inside the pipeline, a strategy that throws is skipped, not fatal. A failure costs you the savings on that request. It never costs you a wrong answer.