Optimizer

A decision engine the gateway calls on every request, to serve it more cheaply.

On every request the gateway hands it, the optimizer runs an ordered pipeline of strategies: small, self-gating transforms (context compression, relevance filtering, tool pruning, semantic caching, and more), each deciding on a best-effort basis how to serve the request before passing it to the next.

It's a standalone HTTP service (optimizer:8088, internal-only). The gateway reaches it through a thin adapter (gateway/src/services/optimizer.ts) over HTTP only, and never imports its internals. Two rules make it safe on the hot path: it does nothing unless ANYRAY_OPTIMIZER_URL is set, and any timeout or error serves the request without it (fail-open). That's why it sits beside the request path as a decision engine instead of in front of it as a second gateway.

The flow

1. Before routing, the gateway calls POST /v1/optimize.
2. The optimizer runs the ordered pipeline from optimizer.config.json; each org composes its own set from the library.
3. Each enabled strategy runs in declared order, threading its (possibly transformed) request into the next. Before strategy execution the pipeline applies the task-outcome regression guard, a per-tenant breaker that suppresses a lossy or steered strategy while its tripwire signals show elevated regret.
4. After the response, the gateway records a spend row. On a cacheEligible miss, and only then, it also writes the response back via POST /v1/cache so the next identical call can hit. On a lane the cache can never serve (a streamed response, a caller that cannot short-circuit) the write-back is measurement-only: shadowOnly: true.

What it returns

/v1/optimize returns a (possibly) transformed request plus decisions:

Outcome	Meaning
request unchanged	No strategy changed it, so the gateway forwards as-is.
transformed request	A strategy rewrote messages, tools, or params, so the gateway forwards the returned request.
cacheHit	When the caller canShortCircuit, semantic_cache may return cacheHit:true + cachedResponse to serve directly.

Only exact semantic-cache hits serve. Pressure-based strategies use the remaining input budget and observed exchange growth; cached-history mints additionally need trusted route and pricing evidence. Explicit legacy overrides remain supported. The strategy reference describes the individual decisions and limits.

Each decision carries kind, a human-readable summary, token/cost estimates, and sometimes a numeric metric ({ name, value }). A decision never carries prompt content, so the console can show them in every content mode. Full wire shapes: Optimizer Protocol.

Fail-open is a hard invariant

The optimizer can never break a request. The gateway calls it with a hard 800 ms timeout (ANYRAY_OPTIMIZER_TIMEOUT_MS) and forwards the request without it on any failure. Inside the pipeline, a strategy that throws is skipped, not fatal. A failure costs you the savings on that request. It never costs you a wrong answer.

A bypassed turn is not a rewind. The gateway remembers, per session, what the optimizer turned each tool description, system block, and message into, and replays those same substitutions on the turn it skips. So the bytes the provider sees stay identical to the turns before it, and your prompt cache stays warm through a timeout instead of being rewritten from the first changed byte. The rest of the safety story is Guardrails.