Skip to main content

How the optimizer learns

The optimizer works out which strategy to apply to your requests, and when to apply it. It learns that from your own sessions, inside your deployment.

Which strategy, and when​

A strategy that saves money in one org can cost money in another, depending on your coding tools, how long your sessions run and how often the model asks for what was trimmed. So the optimizer tests each strategy on your own traffic and keeps the ones that pay.

A strategy that saves goes on for everyone. One that costs more goes back to testing, and one that keeps losing is switched off. A coding tool with enough sessions is judged on its own, so a strategy can be on for one tool and off for another.

The org policy narrows at the coding-tool level, then again for a bound user or verified agent key. A narrower level changes only the settings its own evidence supports. Everything else comes from its parent level.

Within a session, the optimizer also decides when a trim is worth making. A trim pays only if it saves more over the turns the session has left than it costs when the model asks for the original back. When a session keeps asking for trimmed content, the optimizer trims less in it. When that happens across sessions, it pauses the strategy for that coding tool and tries it again later.

The optimizer also coordinates the test share across the org. It writes a daily shadow recommendation for promising strategy combinations without changing traffic. Active learned settings return to paired testing when measurement is enabled. A new win renews them; a loss restores the last validated version or the built-in defaults. On a tenant whose strategies start withheld, the test share goes to one strategy at a time so each reaches a useful sample sooner. An explicit measurement share of 0.05 or more uses a 90% paired lane while they calibrate.

The gateway combines evidence for the exact same strategy policy across tenants. A new tenant can start from a durable fleet result instead of waiting for a personal verdict. The approval remains provisional. With measurement enabled, local control traffic can show a loss and turn the strategy off. Borrowed strategies are validated before still-unproven ones. A policy change withdraws the prior.

What the test costs

Strategy measurement is off by default. Strategies already on keep running. The draw adds no new control or treated sessions. To enable the test, set a positive sessionGate.measurementFraction (see evaluator settings). Control sessions then run without the strategy and forfeit its savings. Setting the share to 0 stops the test; evaluatorEnabled only pauses evaluations.

Learning stays in your deployment​

The gateway and optimizer do the learning, and everything they learn is stored in your deployment's database: your own Postgres when self-hosted, your company's own stack on Anyray Cloud. Test results, strategy states and learned settings never leave it. Anyray receives the content-free Billing App report (usage, savings and health counters), never what the optimizer learned.

Data boundary lists every field the report carries.

What you control​

  • Force any strategy on or off for your org with a session gate override.
  • Settings the optimizer fits from your traffic start in shadow: they are priced next to the real decision and change nothing. When the policy line is idle, the learner starts a bounded trial. Only a winner becomes active. A trial with no result after 30 days ends as stalled once a newer fit is ready, and the newer fit takes its place. A tenant-wide or per-line force_off pauses new trials. An admin can stop one or restore an older winner through the optimizer API.
  • Nothing is learned from prompt or response content, only from counts, sizes and prices, and nothing is shared between orgs.