Optimizer

A decision engine the gateway calls on every request to serve it optimally.

Intro

On every request the gateway hands it, the Optimizer runs an ordered pipeline of strategies: small, self-gating transforms (prompt/context compression, relevance filtering, tool pruning, param tuning, semantic cache, and more) that thread the request through in declared order. Each strategy decides how to serve a request on a best-effort basis.

Concretely, it's a standalone HTTP service (optimizer:8088, internal-only). The gateway calls it through a thin adapter (gateway/src/services/optimizer.ts) — over HTTP only, never importing its internals.

Two rules make it safe to sit on the hot path:

• Opt-in — a no-op unless ANYRAY_OPTIMIZER_URL is set.
• Fail-open — any timeout or error returns the original request untouched.

So it sits beside the request path as a decision engine, not in front of it as another gateway.

The flow

1. Before routing, the gateway calls POST /v1/optimize.
2. The optimizer runs an ordered pipeline of strategies from optimizer.config.json — each org composes its own set (prompt/context compression, code skeletonization, relevance filtering — with an optional local-embedding re-rank for vocabulary-mismatch recovery, default off — window budgeting, tool pruning, tool-schema compression, semantic cache, cache prefix stabilization, and more). See the strategy menu.
3. Each enabled strategy runs in declared order, threading its (possibly transformed) request into the next.
4. After the response, the gateway records a spend row and — only on a cacheEligible miss — writes the response back via POST /v1/cache so the next identical call can hit.

Before ownership decisions and strategy execution, the pipeline also applies the task-outcome regression guard: a per-tenant breaker that suppresses a lossy or steered strategy when retrieval, tool re-run, or reasoning-downshift tripwire signals show elevated regret. Suppression is reversible through the same runtime settings surface.

What it returns

/v1/optimize returns a (possibly) transformed request plus decisions:

Outcome	Meaning
request unchanged	No strategy changed it — the gateway forwards as-is.
transformed request	A strategy rewrote messages, tools, or params — the gateway forwards the returned request.
cacheHit	When the caller canShortCircuit, semantic_cache may return cacheHit:true + cachedResponse to serve directly.

Only exact semantic-cache hits serve.

Each decision carries kind, a human-readable summary, token/cost estimates, and sometimes a numeric metric ({ name, value }) — never prompt content, so the console shows them in every content mode. Full wire shapes: Optimizer Protocol.

Fail-open is a hard invariant

The optimizer can never break a request. The gateway calls it with a hard 800ms timeout (10s for image-bearing requests) and forwards the original request on any failure. Inside the pipeline, a strategy that throws is skipped, not fatal. A failure costs full price, never a wrong answer. See Guardrails.