Optimizer settings

The optimizer runs an ordered strategies[] array; a working seed is baked into the image, and admin edits reach every replica live.

The pipeline lives in optimizer.config.json (array order = run order; ANYRAY_OPTIMIZER_CONFIG overrides the path). Edit it on Settings or via GET / PUT /admin/optimizer/settings; the GET also lists every available strategy and its params (Strategies, Guardrails).

Admin edits land in the shared Postgres and every optimizer replica re-reads them every 15 seconds. That takes ANYRAY_SPEND_DB_URL on the optimizer service; docker compose sets it, and the gateway pushes its copy at boot so older installs need no env edit.

An optimizer without that URL keeps its config per pod

It falls back to a file under ANYRAY_DATA_DIR, where an admin edit reaches only the replica that served the request and is discarded on the next deploy. The optimizer logs runtime config is PER-POD and lost on restart at boot when it is in that state.

The hook is wired by env: ANYRAY_OPTIMIZER_URL (unset = disabled; the gateway fails open) and ANYRAY_OPTIMIZER_TIMEOUT_MS (default 800; past it, the request is forwarded un-optimized).

Halfway through that budget the optimizer starts standing strategies down so it can finish the turn with what it has, instead of running out of time and forwarding the original request. Those turns name budget_exhausted in their trace.

It stands down only the strategies it can drop without changing bytes the session already sent: the read-only ones, the semantic cache, and any strategy whose earlier edits it can replay. The rest keep running past the halfway mark, because dropping them would send the original text back into a prompt the provider has already cached. So budget_exhausted in a trace means some strategies were skipped, not that the turn stopped optimizing.

Per-target rules

overrides.rules in the optimizer config runs a different pipeline per team, user, model, endpoint, or metadata.<key>. A when matcher takes globs, ANDed across dimensions and ORed within one; the action is enable, disable, or params, and disable wins:

{ "overrides": { "rules": [
  { "when": { "teams": ["platform"], "models": ["gpt-4o*"] }, "disable": ["prompt_compression"] }
] } }

Shadow baseline (estimate-vs-actual)

To audit reported savings, sample a shadow baseline: on the sampled fraction the gateway also measures the real trim next to the estimate (no extra upstream call) and stores the drift on the spend row. ANYRAY_OPTIMIZER_SHADOW_BASELINE (default false): true measures every request, a number up to 100 measures that percentage. Audited-holdout requests are always measured; the holdout itself stays off by default.

Batch API input

With ANYRAY_OPTIMIZER_URL set, a purpose=batch JSONL upload (POST /v1/files) is optimized line by line: fail-open, retrieval-independent transformations, plus a handle-free window_budget crop when the alternative is a provider context-limit rejection. Budget: ANYRAY_BATCH_OPTIMIZER_TIMEOUT_MS (default 30000); files over 100 MB skip optimization.