Optimizer settings
The optimizer runs an ordered strategies[] array; a working seed is baked into the image,
and admin edits reach every replica live.
The pipeline lives in optimizer.config.json (array order = run order; ANYRAY_OPTIMIZER_CONFIG
overrides the path). Edit it on Settings or via GET / PUT /admin/optimizer/settings; the
GET also lists every available strategy and its params
(Strategies,
Guardrails).
Batch read appears in the strategy grid with a Client managed badge. Change its Connect setting on each machine and restart the client.
Admin edits land in the shared Postgres and every optimizer replica re-reads them every 15
seconds. That takes ANYRAY_SPEND_DB_URL on the optimizer service; docker compose sets it,
and the gateway pushes its copy at boot so older installs need no env edit.
It falls back to a file under ANYRAY_DATA_DIR, where an admin edit reaches only the replica
that served the request and is discarded on the next deploy. The optimizer logs runtime config is PER-POD and lost on restart at boot when it is in that state.
The hook is wired by env: ANYRAY_OPTIMIZER_URL (unset = disabled; the gateway fails open). The
gateway waits up to 1200 ms for it, then forwards the request un-optimized. Requests with more than
about 30k tokens of content get more time, up to 2.5 times that, because the optimizer's work grows
with the content. The budget is set by the gateway build: ANYRAY_OPTIMIZER_TIMEOUT_MS is retired
and ignored.
Halfway through that budget the optimizer starts standing strategies down so it can finish the turn
with what it has, instead of running out of time and forwarding the original request. Those turns
name budget_exhausted in their trace.
It stands down only the strategies it can drop without changing bytes the session already sent: the
read-only ones, the semantic cache, and any strategy whose earlier edits it can replay. The rest keep
running past the halfway mark, because dropping them would send the original text back into a prompt
the provider has already cached. So budget_exhausted in a trace means some strategies were skipped,
not that the turn stopped optimizing.
Client strategies
Symbol context and Read-only workflows use the same cards and per-user rules as request strategies. Both start off. Changes apply when a supported client or app starts a new session; existing sessions keep their tool definitions.
Open Console → Optimizer, select its card, and turn it on. To limit it to a user, set On in a targeting rule instead. A matching Off rule wins.
Complete the symbol context setup, including separate tool approval, then restart Claude Code. Apps register the managed workflow tool and dispatch its calls. A local opt-out still prevents registration.
Open Users, select the person or agent, and read Client strategies. The report includes its observation time. Unknown, not reported, unavailable and off are distinct. Availability does not confirm tool approval, app integration, tool use or savings.
Test log reads (log_read) is the test-log Read lane in Claude Code. It starts on and
shortens a Read of a test log the session's own test command wrote
(how it works). Its switch is the team policy's hooks.logRead,
so the card reads and writes the org-level setting, and changing
it needs the teamskills:write capability. A team or user override set in Tool policy still
wins for that team or user, and it has no targeting rule here. It needs a Connect release with the
lane. Each Read it leaves whole is recorded on that call's trace with a named reason
(strategy_declines, log_read), and its savings are booked as log_read.
Open the card to see the fleet: seats seen in the last 14 days that run the lane, seats the policy
holds off (by org, team or user), seats the regret guard holds back, seats without Claude Code hooks,
and seats that are Not reporting. That last group runs a Connect older than the lane fixes, or one
that has not synced since it updated. Each seat's own status is on its key and user pages. A seat
reports after each policy pull, and only to a gateway that lists client-strategy-log-read-v1 in its
policy capabilities.
The card also shows the lane's regret guard, the same one the optimizer's strategies have (the
guard setting turns it off). A re-read of lines a digest omitted is its regret. If about half the
org's digests are read back over some 200 digests in 6 hours (about 70% over some 50), the guard holds
the lane back for an hour: Connect reads those logs whole and records the reason guarded. It then runs
the lane in every session to check recovery. Rates up to 40% never trip it.
To measure the lane, turn on Holdout in the Measurement card, choose Custom and tick Test log reads. A share of Claude Code sessions (5% by default) is held out for good: Connect reads their test logs whole, as with the lane off, and the gateway tags their requests as the control. Both decide from the org and the session id alone, so they agree without a call. The card shows provider cost per session in held-out sessions against the rest. It counts only sessions where the lane would have shortened a test log and Connect 0.11.415 or later did what the draw said. A session that never reads a test log, or one whose older Connect served the digest anyway, is in neither group. Beside that, the card shows what the held-out digests would have saved against what the others booked per digest (counters on one gateway replica, a cross-check). A held-out session serves no digest, so the lane's booked savings never include it.
The config stores the other choices in clientStrategies. Targeting rules store their choices in
overrides.rules[].clientStrategies. Client-session policy matches verified user and team;
rules requiring a request model, endpoint or other metadata do not apply at session startup.
Both features remain opt-in experiments with no booked savings or gateway holdout measurement.
Measurement
The Measurement card on Console → Optimizer edits two top-level fields of the same config. A save carries the config revision, so if another admin changed the config first, your save is refused and the card reloads.
| Field | What it does |
|---|---|
holdout | enabled, fraction (share of sessions, 0 to 1, 0.05 by default) and kinds. A fixed slice of sessions runs unoptimized, forfeits its savings, and is the control quality parity reads. At a 0 share nothing is drawn, and the card says so. Leave kinds empty to hold out the whole optimizer, or name strategies to measure each on its own arm. The card offers only switched-on strategies that change requests, because the optimizer drops a read-only kind from the list, and an emptied list holds out the whole optimizer. It also offers Test log reads (log_read), the one client strategy it can measure. That arm is its own: naming log_read replaces any strategies in the list, and the whole-optimizer holdout does not hold it out. |
mintSessionSurvival | Off by default. On (true), a cache rewrite is priced against this deployment's own completed sessions (main threads up to their first compaction, the last 14 days, at least 100) instead of a fixed prior. The horizon is capped where too few sessions were still running. "shadow" builds the same curve and reports what it would have priced as shadow_* metrics on each trace, without pricing on it. Claude Code subagent threads get their own curve under either value, reported as shadow_* metrics only and never priced on. The card shows "shadow" as off and leaves it in place. Set it with PUT /admin/optimizer/settings. |
Per-target rules
overrides.rules in the optimizer config runs a different pipeline per team, user, model,
endpoint, or metadata.<key>. A when matcher takes globs, ANDed across dimensions and ORed
within one; the action is enable, disable, or params, and disable wins:
{ "overrides": { "rules": [
{ "when": { "teams": ["platform"], "models": ["gpt-6*"] }, "disable": ["thinking_trim"] }
] } }
Shadow baseline (estimate-vs-actual)
To audit reported savings, sample a shadow baseline: on the sampled fraction the gateway also
measures the real trim next to the estimate (no extra upstream call) and stores the drift on the
spend row. ANYRAY_OPTIMIZER_SHADOW_BASELINE (default false): true measures every request, a
number up to 100 measures that percentage. Audited-holdout requests are always measured; the
holdout itself stays off by default.
Batch API input
With ANYRAY_OPTIMIZER_URL set, a purpose=batch JSONL upload (POST /v1/files) is optimized
line by line: fail-open, retrieval-independent transformations only. Budget:
ANYRAY_BATCH_OPTIMIZER_TIMEOUT_MS (default 30000); files over 100 MB skip optimization.