Optimizer settings

The optimizer runs an ordered strategies[] array; a working seed is baked into the image, and admin edits reach every replica live.

The pipeline lives in optimizer.config.json (array order = run order; ANYRAY_OPTIMIZER_CONFIG overrides the path). Edit it on Settings or via GET / PUT /admin/optimizer/settings; the GET also lists every available strategy and its params (Strategies, Guardrails).

Admin edits land in the shared Postgres and every optimizer replica re-reads them every 15 seconds. That takes ANYRAY_SPEND_DB_URL on the optimizer service; docker compose sets it, and the gateway pushes its copy at boot so older installs need no env edit.

An optimizer without that URL keeps its config per pod

It falls back to a file under ANYRAY_DATA_DIR, where an admin edit reaches only the replica that served the request and is discarded on the next deploy. The optimizer logs runtime config is PER-POD and lost on restart at boot when it is in that state.

The hook is wired by env: ANYRAY_OPTIMIZER_URL (unset = disabled; the gateway fails open) and ANYRAY_OPTIMIZER_TIMEOUT_MS (default 800; past it, the request is forwarded un-optimized).

Per-target rules

overrides.rules in the optimizer config runs a different pipeline per team, user, model, endpoint, or metadata.<key>. A when matcher takes globs, ANDed across dimensions and ORed within one; the action is enable, disable, or params, and disable wins:

{ "overrides": { "rules": [
  { "when": { "teams": ["platform"], "models": ["gpt-4o*"] }, "disable": ["prompt_compression"] }
] } }

Shadow baseline (estimate-vs-actual)

To audit reported savings, sample a shadow baseline: on the sampled fraction the gateway also measures the real trim next to the estimate (no extra upstream call) and stores the drift on the spend row. ANYRAY_OPTIMIZER_SHADOW_BASELINE (default false): true measures every request, a number up to 100 measures that percentage. Audited-holdout requests are always measured; the holdout itself stays off by default.

Batch API input

With ANYRAY_OPTIMIZER_URL set, a purpose=batch JSONL upload (POST /v1/files) is optimized line by line: fail-open, retrieval-independent transformations, plus a handle-free window_budget crop when the alternative is a provider context-limit rejection. Budget: ANYRAY_BATCH_OPTIMIZER_TIMEOUT_MS (default 30000); files over 100 MB skip optimization.