Skip to main content

Optimizer

A decision engine the gateway calls on every request, to serve it more cheaply.

On every request the gateway hands it, the optimizer runs an ordered pipeline of strategies: small, self-gating transforms (context compression, relevance filtering, tool pruning, semantic caching, and more), each deciding on a best-effort basis how to serve the request before passing it to the next.

It's a standalone HTTP service (optimizer:8088, internal-only). The gateway reaches it through a thin adapter (gateway/src/services/optimizer.ts) over HTTP only, and never imports its internals. Two rules make it safe on the hot path: it does nothing unless ANYRAY_OPTIMIZER_URL is set, and any timeout or error returns the original request untouched (fail-open). That's why it sits beside the request path as a decision engine instead of in front of it as a second gateway.

The flow​

  1. Before routing, the gateway calls POST /v1/optimize.
  2. The optimizer runs the ordered pipeline from optimizer.config.json; each org composes its own set from the library.
  3. Each enabled strategy runs in declared order, threading its (possibly transformed) request into the next. Before strategy execution the pipeline applies the task-outcome regression guard, a per-tenant breaker that suppresses a lossy or steered strategy while its tripwire signals show elevated regret.
  4. After the response, the gateway records a spend row. On a cacheEligible miss, and only then, it also writes the response back via POST /v1/cache so the next identical call can hit. On a lane the cache can never serve (a streamed response, a caller that cannot short-circuit) the write-back is measurement-only: shadowOnly: true.

What it returns​

/v1/optimize returns a (possibly) transformed request plus decisions:

OutcomeMeaning
request unchangedNo strategy changed it, so the gateway forwards as-is.
transformed requestA strategy rewrote messages, tools, or params, so the gateway forwards the returned request.
cacheHitWhen the caller canShortCircuit, semantic_cache may return cacheHit:true + cachedResponse to serve directly.

Only exact semantic-cache hits serve.

Each decision carries kind, a human-readable summary, token/cost estimates, and sometimes a numeric metric ({ name, value }). A decision never carries prompt content, so the console can show them in every content mode. Full wire shapes: Optimizer Protocol.

Fail-open is a hard invariant​

The optimizer can never break a request. The gateway calls it with a hard 800 ms timeout (ANYRAY_OPTIMIZER_TIMEOUT_MS; image-bearing requests get their own 10 s budget, where vision_ocr needs the room) and forwards the original request on any failure. Inside the pipeline, a strategy that throws is skipped, not fatal. A failure costs you full price on that request. It never costs you a wrong answer. The rest of the safety story is Guardrails.