Optimizer protocol

The HTTP contract between the gateway (or your adapter) and the optimizer.

Endpoint	Gateway hook	Purpose
POST /v1/optimize	pre-call	Transform the request; may return a cache hit.
POST /v1/optimize-response	post-call	Transform the response.
POST /v1/cache	post-call	Write a live response back for semantic_cache.
POST /v1/census	post-call	Count a Codex seat turn's tool output for the census.
POST /v1/retrieve	on demand	Fetch original content a reversible strategy stashed, by handle.
POST /v1/recall	on demand	Semantic search over the durable stash: find originals by meaning.
POST /v1/record	post-call	BYO only, opt-in. Persist the call when no Anyray gateway is in the path.

An adapter wires whichever hooks its gateway supports. Every endpoint rejects a body over ANYRAY_OPTIMIZER_MAX_BODY_BYTES (default 32 MiB) with 413.

The optimizer serves its own spec at GET /openapi.json and Swagger UI at GET /docs. Both show only the API shape. When prose and spec disagree, the route handlers win.

Optional request selection

POST /v1/optimize accepts selection: {profile?: "interactive" | "exact", only?: string[], shadow?: string[]}. It is separate from attribution metadata. Unknown profiles, unknown strategy IDs and extra fields fail validation. Omitting it keeps existing behavior.

Selection narrows the strategies resolved from tenant policy, after lane and session gates. exact admits only diagnostics and cache anchoring. only restricts other fresh work.

A kind named in only also runs when it is merely off by default, as if a matching rule enabled it, unless the config sets requestKindsEnable: false. exact enables nothing, and semantic_cache is never enabled this way. If a later turn stops naming the kind, its settled edits stay (it stands down as request_selection, tracked by a request_kind_enabled pin). Configured disables (a matching rule, byEndpoint) and every gate still apply to a kind enabled this way. The session gate draws these kinds in a second tier, only in sessions where no configured kind was drawn, so they get holdout and treated arms without taking one from a configured kind. A turn whose only leaves out a kind policy would run supplies no arm to any kind.

protectClientManagedCache is an optimizer org setting, off by default. Set it to true to hold every new edit that could land at or before the client's last cache breakpoint. A breakpoint is a cache_control marker its route honors, or Anthropic's request-level cache_control (automatic caching, through the last message) on a route that forwards it. The gateway reports which routes forward the request-level field (providerAutomaticCacheControl). Bedrock's Converse route drops it; InvokeModel leaves it unknown. Off, editors run on Claude Code traffic and pay for the cache rewrite through the mint payback check.

An older optimizer may still report rolling_breakpoint in suppressedKinds: the retired context_dedupe stood down there when the client's last honored breakpoint was on the last message. No current strategy reports it.

With it on, a kind is held when:

Strategy	Held when
observation_mask, relevance_filter	the last breakpoint is on the last message
other history editors, such as code_graph, thinking_trim, command_digest	any message carries a breakpoint

When the last breakpoint sits before the last message, observation_mask and relevance_filter still run, but only on the messages after it. They do not mint a new edit at or before the breakpoint, and their settled pins there keep replaying. observation_mask masks only observations from completed exchanges, before the active turn. When the breakpoint sits after all of those, the usual case, it mints nothing new and only replays.

A breakpoint on only the system prompt or tools leaves the messages editable. Deterministic strategies such as tool_schema_compression and columnar_fold always run. Each kind the pipeline holds reports client_managed_cache in both suppressedKinds and strategyDeclines, and its session-gate arm reads cooloff. Existing settled pins still replay with the same bytes, including after a retrieval rollback. content_census, cache_lint, and cache_optimizer keep running.

protectThinkingBinding is an optimizer org setting, off by default. Claude Fable and Mythos 5.1, Opus 5.5, Sonnet 5.5 and newer check every re-sent thinking block against the exact prompt it was written for. Off, observation_mask, relevance_filter and code_graph still edit at or before a retained thinking block on those models. On the first-party API that edit makes the provider drop that block and every thinking block after it: the call returns 200, the blocks bill nothing, and the model answers without that reasoning. Bedrock keeps and bills them. Set it to true to hold those edits instead. Each of the three kinds then reports thinking_binding in strategyDeclines. thinking_trim keeps its own hold either way.

Settled recipes can replay, and strategies marked for session continuation can stay active after prior treatment. Declined fresh work appears in suppressedKinds with reason request_selection, which still permits settled replay.

The response acknowledges handling with selection: {eligible: string[], continued: string[]}. eligible is the configured candidate set before safety gates. continued lists session-scoped strategies that stayed active despite the selection.

shadow names kinds to evaluate without applying. After the pipeline, the optimizer runs each one on a copy of the request as the client sent it, with a fresh context, and keeps only whether it produced a decision that saves tokens. The kind resolves as if only had named it, so a disable still wins. Nothing it returns reaches the request, the pins, the decisions or the spend record. The acknowledgment then carries shadow: {wouldApply: string[], unsupported: string[]}. With no cache boundary, pins or economics in that context, wouldApply is an upper bound: the kind found something to act on, as it would on a fresh session. A kind can run this way only if its strategy is registered shadowSafe, which is never true of a kind that writes retrieval handles. A shadow kind starts only before the strategy admission deadline and checks it before each segment, so only the last kind admitted can run past it, by one segment plus its setup and teardown. A kind that isn't shadowSafe, runs out of time, or fails is unsupported. shadow never narrows the kinds that run. An optimizer older than the field rejects it, so deploy the optimizer before a gateway that sends it.

A gateway must discard an optimizer rewrite if it requested selection and the response has no valid acknowledgment, then follow its usual fail-open behavior. The SDK header contract and transition limits are in the SDK reference.

Request signing

Optional, off unless configured. The gateway holds an Ed25519 private key and the optimizer only the public half, so a workload that can read the optimizer's environment (shared bearer token included) still cannot produce a valid request.

Once the optimizer has a verify key, POST /v1/optimize, /v1/optimize-response, /v1/cache, /v1/census, and /v1/recall require an x-anyray-signature header and answer 401 without a valid one. /v1/record is never gated.

/v1/retrieve is mid-rollout. The gateway signs it, and the optimizer refuses a bad signature, but a request carrying no signature at all is still served: an older gateway does not sign this route, and refusing it would leave every handle already sitting in a cached prefix unresolvable. The signature becomes required once every gateway signs it. Retrieve is signed at all because the optimizer scopes each read to the user named in its metadata, and the signature is what proves the gateway wrote that name. Until it is required, that proof is advisory: an in-network caller holding the shared bearer token can omit the header and name anyone, so the scoping separates colleagues, not attackers.

GET /health reports retrieveSignatures so you can see when the flip is safe. It counts the state of every retrieve this replica has served:

{ "signed": 4412, "unsigned": 0, "invalid": 0, "unverified": 0,
  "firstUnsignedAt": null, "lastUnsignedAt": null }

unsigned counts calls that arrived with no signature at all, which are exactly the ones that would start failing if a signature became mandatory. unverified counts a signature that arrived but could not be checked, because this deployment has no verify key. That is not the same as a verified one: a valid header and a malformed one both land there.

So unsigned: 0 on its own is not proof that the flip is safe. Roll it out in this order:

1. Set the verify key. Enforcement turns on for the other routes, /v1/retrieve keeps serving an unsigned call, and nothing can strand. This is what turns unverified into a real verdict.
2. Wait for real retrieve traffic, then confirm both unsigned: 0 and invalid: 0, on every replica. Together those say every gateway sends a signature and every signature verifies.
3. Then require it. Skipping step 1 and flipping on unsigned: 0 alone will 401 any gateway that sends a malformed header, which is the stranding this rollout exists to avoid.

The counters are per-replica and reset when the process does, so read them on each pod.

The header is <timestamp>.<signature>:

Part	Form
timestamp	Unix ms, canonical decimal: plain digits, no leading zero
signature	Ed25519 over the payload below, unpadded base64url (86 chars)

The signed payload is <timestamp>.<METHOD>.<path>.<sha256-hex of the raw body>. It carries the body hash, never the body, so a signature never becomes a second copy of prompt content, and method and path bind a signature to its route. Both halves must be spelled canonically, since base64url has several spellings for the same bytes. Signatures are single-use within a 60-second window: outside it they are stale, and a repeat inside it is refused as a replay.

Enable the gateway first

The optimizer requires a signature the moment it has a verify key, so a deployment that configures the optimizer ahead of the gateway refuses every optimize call. The gateway fails open on that 401 rather than erroring, so nothing looks broken. The deployment just stops optimizing. GET /health reports requestSignatures: "enforced" once it is live.

On a deployment that already enforces signatures, upgrade the gateway before the optimizer. A gateway older than signed retrieval sends /v1/retrieve unsigned, and until it is upgraded the model gets retrieval unavailable for every handle.

Entitlement

The bearer token proves a caller is inside the deployment, not that the subscription is current, so the optimizer checks that separately. It reads the Ed25519-signed entitlement lease the gateway stores in the shared database and verifies it against the key pinned in its own image. The check is a memoized read refreshed on a timer, never a database call on the request path.

Three lease states stop /v1/optimize, /v1/optimize-response, /v1/cache, /v1/census, and /v1/recall:

Lease state	Response	What it means
optimizerEnabled: false	402	Optimization is paused, typically a trial that ended with no subscription. The lease itself stays active, so this is the state that matters most in practice.
status: "suspended"	403	The subscription was suspended.
Expired, or the signature does not verify	402	No current, vouched-for entitlement.

POST /v1/retrieve and POST /v1/record are never gated by entitlement, whatever the lease says. Refusing retrieve would strand live sessions with unresolvable handles, and record is the BYO lane that writes spend rows. Retrieve does carry a signature where that is configured, for a different reason: entitlement is about whether the subscription is current, the signature about which user the read is scoped to.

A deployment with no lease at all serves normally (attach mode, or an in-network stack with no Anyray gateway). A lease that exists and has expired is refused, including one that expired before the optimizer last started: restarting never clears the gate.

What GET /health reports

{ "gateway": 18432, "foreign": 0, "firstForeignAt": null, "lastForeignAt": null }

callers counts POST /v1/optimize calls since start that carried a gateway-stamped session id against those that did not. A non-zero foreign means something other than an Anyray gateway is driving this optimizer. That is legitimate for attach mode and BYO adapters, but where a gateway should be the only caller those calls are optimized and their savings reach no spend row. The field is advisory, not a security control.

GET /health is the readiness probe. Its readiness is unarmed for embedded use, warming while the first stash-policy and session-gate reads are open, ready after both settle, or ceiling_elapsed after 30 seconds. It returns 503 with ok: false while warming and 200 otherwise, with the full content-free body (including configStore and retrievalStore) at either status. GET /livez is the public, dependency-free liveness probe and always returns plain-text ok with 200.

{ "state": "reported", "calls": 18402, "rows": 16110, "foreignCalls": 0, "checkedAt": "…" }

reporting compares optimize calls served over a rolling window against the spend rows that landed in the same window. reported is healthy. underreported is the shape produced by driving this optimizer around the gateway, where savings reach no invoice. insufficient_data, unknown, no_store, and not_optimizing are honest non-answers. The check never refuses a request.

retrievalStore is a content-free durable-write state. It controls fresh handle creation, not optimizer liveness: the gateway still fails open, and handle-free strategies keep running.

State	Meaning
ready	Fresh retrieval spans can be durably committed.
cooldown	A bounded write or connection failed. New handles stay disabled until the five-minute retry circuit closes.
unconfigured	No durable context store is configured, or no secret to key it with (neither ANYRAY_CONTENT_KEY nor ANYRAY_ADMIN_TOKEN). New handles stay disabled.
policy_pending	This replica has not read the console content mode yet. New stash writes are held; settled handles can still replay.
content_disabled	The configured content mode is off. New handles stay disabled; rows stashed before the switch still serve.
placeholder_key	No ANYRAY_CONTENT_KEY, and ANYRAY_ADMIN_TOKEN is the public example token, so the only stash key is public. New handles stay disabled; rows stashed earlier still serve. Set a private admin token or a content key.

POST /v1/optimize

Pre-call. Runs the configured pipeline and returns a transformed request (unchanged if nothing fired) plus decisions.

endpointstringrequired

Logical route (e.g. /v1/chat/completions). Selects the per-endpoint config.

requestobjectrequired

The incoming OpenAI-compatible body (model, messages, …) to transform.

metadataobject

Attribution (user, team), never content. The gateway may add optional signals: modelContextTokens, clientCanRetrieve, and billing/tenant scope. See Proving retrieval.

enabledKindsstring[]

Allow-list that further narrows the pipeline to these strategy kinds.

capabilitiesobject

Gateway-owned controls. They live outside honor-system metadata so callers cannot grant themselves an irreversible strategy or misclassify their cache behavior. See Capabilities.

pinsobject[]

The session's current decision pins. Sending this array, even empty, opts the session into pinning; omit it for pre-pinning behavior.

subjectUserstring

Erase only, set by the gateway. When an agent key with delegated attribution acts for an end user other than itself, this is that user's id. Every stash row the request writes adds it to the row's set of end users, so erasing any of them removes the row. The set holds at most 32; a later end user is not recorded, and that person's erase misses the row until it expires. The field changes no request bytes and grants no reads. It is read from the top level only, never from metadata, and a malformed value is ignored (no 400).

POST /v1/optimize

Request

{
  "endpoint": "/v1/chat/completions",
  "request":  { "model": "...", "messages": [ ... ] },
  "metadata": { "user": "u1", "team": "t1" },   // OPTIONAL attribution
  "enabledKinds": ["tool_schema_compression"],  // OPTIONAL allow-list
  "capabilities": {                             // OPTIONAL, gateway-owned
    "canShortCircuit": true,
    "clientContextBudgetTokens": 950000
  },
  "pins": [ ... ]                               // OPTIONAL decision pins
}

Response

{
  "protocolVersion": 1,
  "features": { "persistentTranscriptPolicyV1": true },
  "optimizationId": "opt_000001",
  "request": { ...transformed body... },   // FORWARD THIS
  "decisions": [
    { "kind": "tool_schema_compression", "summary": "shortened 1 tool description",
      "estimatedTokensSaved": 120, "estimatedSavingsUsd": 0 }
  ],
  "estimatedTokensSaved": 120,
  "cacheHit": false,        // true => serve cachedResponse, skip the provider
  "cachedResponse": null,   // present only when cacheHit
  "cacheEligible": true,    // true => write the live response back via /v1/cache
  "cacheKey": "<opaque key>",
  "cacheTtlSeconds": 3600,
  "cacheShadowOnly": true,  // present with cacheEligible on a lane that can never serve
  "cohort": "treated",      // present only when audited holdout is enabled
  "cohortKind": "thinking_trim", // present only on a per-strategy control row
  "pins": [ ... ],          // present only when the request carried a `pins` array
  "vouchedHandles": [ { "handle": "ctx_1x2y3z_9k", "expiresAtMs": 1790000000000, "epoch": "42" } ],
  "suppressedKinds": [ { "kind": "relevance_filter", "reason": "no_retrieve" } ],
  "strategyTimingsMs": { "relevance_filter": 512, "phase_retrieval_commit": 40 },
  "strategyDeclines": { "observation_mask": { "payback_retrieval": 1 } }
}

The read-only regret_guard_refetch decision counts repeated and same-file calls that did not prove a take-back. Its metric name is edited_between when a file-changing call ran between the masked call and its repeat (an edit, then the same test run), range_disjoint when known line ranges do not overlap omitted lines, range_unknown when a range cannot be resolved, or search_content_unknown when a file search does not show that removed content was sought. The metric value is the number of skipped calls. No file path or search text is sent in this decision.

Mint admission emits numeric mint_economics metrics: rewrite_premium_usd, future_read_saving_usd, and net_expected_saving_usd, alongside retrieval risk and cost. They describe the selected candidate group, or a declined group when none qualifies. These are admission estimates, not measured or billed savings. With economics.tailAdmission on, the group also reports tail_payback_turns and tail_net_saving_usd: the same price at the horizon three sessions in four still reach. An edit after the cache breakpoint reports suffix_payback_turns, suffix_expected_retrieval_usd, suffix_rewrite_premium_usd, suffix_future_read_saving_usd and suffix_net_expected_saving_usd (economics.priceAllLanes). observation_mask's open lanes report open_lane_payback_turns, open_lane_expected_retrieval_usd, open_lane_future_read_saving_usd, open_lane_net_expected_saving_usd, and open_lane_would_decline: 1 when the price says no (enforced only under economics.enforceOpenLanes).

Anchored trims add two mint_economics counts. cache_anchor_verified_tails is the number of attested tails that match the request about to be sent and can take a marker, reported each time the late anchored pass runs. cache_anchor_placed is 1 when that pass places its marker. The pass reports its own admission estimate under the same names prefixed anchored_, so one request still carries one net_expected_saving_usd verdict. All of them need observation_mask.mintAnchor: true and attested cacheAnchorTails; the param defaults to false.

The request planner exhaustively searches groups of up to six proposals. For larger groups, it also searches every compatible combination when bounded discovery proves they fit within the 80-quote budget, including required dependencies. Otherwise, the original search keeps its 64-quote allowance and an independent recovery schedule explores shared rewrite regions and groups ranked by expected read savings. Both schedules share quotes within the same 80-quote limit. Every group must pass the same cost and tail checks.

Warm rewrites must also repay if their useful lifetime is up to one turn shorter. The planner reserves one turn of net recurring savings, capped at the quoted horizon. Cold-cache edits and edits with no rewrite premium have no reserve. request_plan_horizon_reserve_usd reports this admission floor alongside committed and unselected expected-scenario estimates. Modeled savings and horizons stay unchanged, so a positive estimate can still receive a payback decline when it falls below the reserve. This cushion does not guarantee savings if the session stops immediately.

The request planner (economics.requestPlanning) adds its own mint_economics counts. A committed plan reports request_plan_edits (edits selected), request_plan_min_net_saving_usd (the worst certified scenario), request_plan_quotes (search and completion quotes plus at most one quote for the per-strategy selectors' combined candidate) and request_plan_search_exhausted: 1 when the 80-quote budget ran out, so the committed group is the best of a bounded search. The combined candidate passes the same safety checks but does not guarantee parity with a separate planner-off run. After ranking, the planner can retain additional certified fresh edits that pay on the first write, add no retrieval risk or requirement, and leave existing edits' horizons unchanged. This uses no extra joint quotes. request_plan_min_net_saving_usd can be lower for the larger plan because the added edit introduces a shorter scenario; it still reports the actual bound. A completion pass tries unquoted combinations of compatible fresh edits under those same rules. It splits failed batches within eight trials and at most eight extra quotes. The total selection cap, including the combined per-strategy candidate, is 89 quotes.

Metric	Meaning
request_plan_opportunity_turns	With experimental shared-handle pricing, expected reads before a later cold opportunity. Present only when this is shorter than the session horizon.
request_plan_completion_quotes	Extra joint quotes spent completing the plan.
request_plan_completion_edits	Edits added by completion.
request_plan_completion_exhausted	1 when the trial cap leaves groups unprocessed.
request_plan_unselected_certified	Unselected proposals admitted in at least one observed trial.
request_plan_unselected_economic_rejection	Unselected proposals with an economic rejection and no admission.
request_plan_unselected_structural_rejection	Unselected proposals tried only in groups rejected for conflicts or dependencies.
request_plan_unselected_unquoted	Unselected proposals absent from selector trials.

The unselected counts partition the captured proposals left out of the plan, from attempted trials and admitted local alternatives. A rejection does not prove another combination would fail. Counts include zero; exhaustion appears only when reached. These metrics publish with the committed plan; materialization failures use the abort metrics below.

Fresh edits that need no retrieval handle preserve eligible peers' cache-expiry pricing. Cached edits and new retrieval handles still require their own cold eligibility; avoided-write credit remains limited to eligible cached edits. A plan that kept the mandatory baseline reports request_plan_materialization_failed: 1 with a request_plan_abort_<reason>: 1 twin naming why: render_mismatch (a strategy rendered bytes other than the quoted edit), edit_precondition, anchor_conflict, anchor_placement, stale_certificate, or capture_error, select_error, render_error, anchor_error when that phase threw. request_plan_baseline_unmatched counts replay decisions the plan could not replace; a non-zero value means a replay was reported twice.

Three content-free decision fields reach billing. Each is optional; absent means unknown.

Field	What it carries
cachedSavedFraction	Share of estimatedTokensSaved, in [0, 1], whose bytes the provider had already cached (pin replays plus in-prefix mints). Billing prices that share at the cache-read tier and the rest at the not-yet-cached tier.
cacheExpired	true only. The pipeline stamps it when the gateway attested the prompt cache expired this turn (capabilities.promptCacheExpired) on every decision that claims cached bytes (a cachedSavedFraction above 0 or pinsReplayed above 0). Billing then prices that cached share at the not-yet-cached tier, but only on a row that read nothing from cache. Never sent by a strategy; anything but an exact true is dropped at the gateway.
maxBillableTokensSaved	A proven ceiling on what the trim could ever have billed, in provider tokens. A count, so a downstream refinement of the estimate never lifts it.
prefixRewriteTokens	On a turn where a strategy minted a fresh edit inside the cached prefix: the retained prefix tokens the edit made the provider re-write, which the unedited turn would have read. A count. Billing debits it from that turn's saving, capped at what that round wrote to the cache (or, on a route that caches implicitly and shows the bust as fresh input, at that fresh input). Never sent on a replay turn, a cache-expired turn, or under a legacy mintPaybackRatio. Sent by every strategy that prices its mint: observation_mask, relevance_filter, code_graph, command_digest and thinking_trim.
prefixReadTokens, headerReadTokens	Sent together, beside prefixRewriteTokens, only when the mint was priced on a measured tail-lookback route (the provider reads back to the nearest earlier tail within its lookback window). That price is opt-in per strategy (mintTailLookbackPrice: true, default off), so by default no decision carries the pair. prefixReadTokens is the cached read the price assumed: the header plus the cached messages ahead of the re-written region. headerReadTokens is the header alone. Never priced. The gateway compares them with the row's real cache read to check the window (see mintEvidence.tailLookbackMissed). Withdrawn whenever prefixRewriteTokens is, and when an earlier edit on the same turn moved the read below the prediction.
tailBreakEvenMissRate	Sent with prefixReadTokens: the miss rate, from 0 to 1, at which this mint's expected net saving reaches zero, when a window miss re-writes the messages the price assumed were read. 1 means even a miss leaves the mint paying. Never priced. The gateway stores it with the verdict and latches from it (see mintEvidence.tailLookbackMissed). Withdrawn with the read pair.

When the gateway sends a measured session survival curve (mintEvidence.sessionSurvival) and this optimizer's config does not set mintSessionSurvival: true, a group priced on a main thread also reports shadow_payback_turns and shadow_net_expected_saving_usd: the horizon and net saving it would have had under the curve. shadow_tail_payback_turns and shadow_tail_net_saving_usd are the same price at the curve's lower quartile (measured, or estimated from the mean when the curve carries none). They are estimates, never read by admission and never billed. Set mintSessionSurvival: "shadow" to have the gateway build the curve without the gate deciding on it. They appear once the curve exists: 100 sessions of a client tool the gateway knows, completed after the gateway upgrade that counts each session up to its first compaction.

The request-wide planner prices the same shadow on its own horizons and emits the same four metrics, and the response sums them per strategy as shadowHorizon. The shadow never changes the plan: the forwarded request and the declines match a request with no curve.

The curve applies only at elapsed turns covered by its arrays. A session that outlives that coverage uses the existing lifetime prior, still capped by its context budget and any mintPaybackTurns override. A measured zero within the array remains zero.

The 14-day window selects recent sessions. Their lifetimes include earlier retained summary hours; the window does not cut a session short. Sessions with incomplete lifetime counts are skipped.

A billed prompt dropping below half the previous successful prompt on the same model marks compaction. A comparable pair of complete Messages or Chat Completions requests vetoes that cut when the original input did not halve, so optimizer edits do not shorten the learned lifetime. Both counts must share originalInputBasis and provider. Missing counts, partial shadows and older measurements keep the billed-prompt detector. originalInputTokens is a static body estimate, never billed usage. With mintSessionSurvival: true or "shadow", the gateway counts complete original requests after the response. Existing diagnostic samples reuse their original count. Other traffic keeps its existing sampling behavior.

Once the gateway also sends the subagent curve (mintEvidence.subagentSessionSurvival, needs 100 subagent threads), a group priced on a Claude Code subagent thread reports the same four metrics from it instead, even when mintSessionSurvival is true: no subagent decision reads a measured curve yet.

warning

Shadow metrics show what the curve would have priced, not what a session repaid. Before a tenant sets mintSessionSurvival: true, run a paired live bench against pass-through on that tenant's session shape (Rule 0).

rewrite_premium_usd is priced over the bytes the mint re-sends, which is not the same region on every route. Where the bytes ahead of the edit are provably still cached, it covers only the edit point to the end of the cached prefix; otherwise it covers the whole prefix. The two differ by orders of magnitude on a long session, so compare this metric only within one route, and expect a step change on a deployment that alters which branch applies.

On a turn priced as cache-expired (cache_expired: 1), rewrite_premium_usd is 0 unless the mint adds a retrieval tool header. observation_mask also counts the write the provider skips that turn for the content the mint removed (mintColdWriteCredit, on by default; false turns it off). Set coldTurn.writeCredit: true in the config, or mintColdWriteCredit: true on relevance_filter, to count it for those kinds too. It is priced at the 5-minute write rate and capped at future_read_saving_usd. It is reported as cold_write_saving_usd and is already part of net_expected_saving_usd. command_digest is never credited.

The credit applies only where the turn has evidence the removed content was uncached. With the credit asked, any other cache-expired turn keeps the zero premium and reports why, as a metric with value 1:

Metric	The turn
cold_write_credit_withheld_implicit_route	is on a route that caches automatically
cold_write_credit_withheld_markerless	has no cache_control breakpoint from the client covering all of the content
cold_write_credit_withheld_thread_unproven	carries no parent conversation id from the gateway
cold_write_credit_withheld_subagent_thread	is a sub-agent thread
cold_write_credit_withheld_sibling_finished	was started by a background task finishing
cold_write_credit_withheld_tool_call_gap	resumes a tool call

A turn_shape decision says the request declared no callable tool, so the strategies that depend on decision pins or a retrieval handle stood down. It records a fact about the request envelope, never a gate: it claims no savings, and it does not appear in suppressedKinds.

Read it beside a no_retrieve row, whose two causes are independent and often both true. Its metric name says which case you have: tool_calls_forbidden means the turn's shape alone stood the strategies down, so nothing needs fixing (whether the client can retrieve is judged on turns that declare a tool); tool_calls_forbidden_and_retrieval_incapable means the shape applied AND this client cannot resolve a handle at all, which is a real client-level finding that outlives this turn. The absence of a turn_shape decision on a no_retrieve row means the client is retrieval-incapable on a turn that did declare a tool.

A paste_gap decision comes from content_census when the live user turn is a large pasted log or data dump. Its paste_gap_tokens metric is what a trim would have removed. The paste is never trimmed and the decision claims no savings. The gateway adds the metric to the verified user's advisory potentialPaste total on /connect/savings, except on Cursor rows, whose prompt hook reports it.

content_census also sizes the request prefix on every turn that carries a system prompt or tools, one decision per number: prefix_system_tokens (the system prompt plus any leading system or developer messages), prefix_tools_tokens, prefix_mcp_tools_tokens (tools whose name starts mcp__), prefix_first_message_tokens and prefix_tool_count. They are estimates from each field's JSON length, carry no text, and claim no savings.

A force_off_revert decision says the per-session gate's force_off kill switch dropped a strategy's settled decision pins on this turn, one decision per strategy, named in the summary. The spans went out as the client sent them and the pins do not return. It claims no savings; the count rides the force_off_pins_dropped metric. The strategy itself is listed in suppressedKinds as session_gate_force_off, never in the applied set.

A columnar_fold_dictionary_census decision comes from columnar_fold. On each tool result the fold considered fresh this turn (and, on app tool loops, each JSON message the app wrote itself), and only when the gateway's firstAppearance claim says the result is new, it measures what a dictionary of repeated values (the values lane) or of shared URL and path roots (the prefixes lane) would take off the text the fold sends. Nothing is applied: the request carries the fold's bytes alone, a replayed result is not measured again, and the decision claims no savings and carries no text. It reads at most 64 KB of one result and 64 KB per request, in message order. A result past either bound is counted and its size recorded, but not read. A result with more JSON bodies than its walk allows, such as dense JSON lines with short rows, is counted as read in part, with the characters it did not reach. It is never reported as measured, because what it did not reach is unknown. Each number is its own decision:

Metric	Value
dictionary_census_segments	Results counted this turn, read or not.
dictionary_values_segments, dictionary_prefixes_segments	Results that lane would shorten.
dictionary_values_tokens, dictionary_prefixes_tokens	Estimated tokens that lane would remove.
dictionary_<lane>_declined_<reason>	Results the lane left alone, by reason. Only reasons with a count appear.
dictionary_census_too_large_chars	Characters in results over 64 KB, which were not read. Absent when zero.
dictionary_census_budget_exhausted_chars	Characters in results after the request's 64 KB was spent, which were not read. Absent when zero.
dictionary_census_partial_read_chars	Characters the walk did not reach in results read in part. Absent when zero.

The reasons, in the order they are checked: too_large, budget_exhausted, partial_read, not_json, too_deep, duplicate_key, prototype_key, unsafe_numeric, non_canonical, reserved_key, no_candidates, expands, inverse_mismatch, then census_failed when the census itself threw, which leaves the fold's bytes and decision as they were. A result read in part declines partial_read on both lanes, whatever its read part shows, so each lane's counts still add up to dictionary_census_segments.

A turn that declares no callable tool withholds its decision pins, so the settled history it sends is untrimmed while the turns either side of it send it trimmed. The optimizer config's toollessSettledReplay lets such a turn replay the settled pins it already holds instead. It is off by default, and whether the difference costs anything is not yet established: while it is off, a turn_shape decision carries toolless_revertable_pins, the count of settled edits that turn discards, so the lane can be sized per deployment before anyone turns it on. Handle-bearing pins are never replayed on these turns, on or off: the model cannot call the retrieval tool, so a retrieve ctx_… marker would be unresolvable.

When the caller sends capabilities.acceptRequestPatch, the response carries requestPatch instead of request: only the messages that changed (by index), the top-level keys that changed or appeared, and the top-level keys that were removed. The caller rebuilds the forwarded body from its own original request and the patch, and the result is byte-identical to the echo it replaces, key order included. The optimizer sends the full request whenever it cannot guarantee that, such as when the message count changed. It is on by default; set the optimizer config's requestPatch to false to turn it off.

Every response field beyond request and decisions is additive and optional, and protocolVersion stays 1.

Field	What it carries
suppressedKinds	Each configured strategy a gate dropped this turn, and why. Reasons: no_retrieve, toolless_turn, cache_guard, prefix_stabilizer, regret_guard, pin_thrash, rolling_breakpoint (the client's last breakpoint is on the last message, so a cache-busting kind that is not suffix-safe has nowhere to edit but the cached prefix; its settled pins still replay), session_participation (the session's participating kinds were fixed on its first cached turn: a kind enabled since then does not start, and a kind disabled since then keeps its settled pins replaying until the prefix goes cold), budget_exhausted, strategy_failed, cache_anchor, warm_prefix, holdout_kind (legacy control), holdout_whole (the whole-optimizer control, which withholds every strategy but the read-only sensors), session_gate_holdout, session_gate_shadow, session_gate_off, session_gate_force_off (the operator's kill switch: the kind's settled pins stop replaying, its spans revert to the client's bytes once, and its pins are dropped), session_gate_unavailable, no_durable_stash (content mode off left no stash to hold an original), untrusted_terminal_trim (a terminal-trim hint the gateway could not verify, so the request goes out unchanged). This is how "suppressed" stays distinguishable from "had nothing to do". An empty array is reported when no kind was suppressed; absence means the peer did not report a ledger (optimizer_ledger_missing on a skipped gateway record); an array with only invalid entries yields optimizer_ledger_invalid. no_retrieve means the client itself cannot resolve a retrieval handle; toolless_turn means only that this request declared no callable tools, as compaction and title turns do. A turn that is both reports no_retrieve, so that the real client finding is not outvoted; the turn_shape decision's metric then says whether the client is also retrieval-incapable.
suppressedKinds rows of kind first_appearance_shadow	A measurement tier on the same ledger. It is not a gate: it holds nothing back and applies nothing. When the gateway sends a verified firstAppearance claim, the optimizer measures what an edit to the new part of the request would have covered and reports it as a first_appearance_shadow decision with estimatedTokensSaved: 0; the count is in the metric fa_shadow_tokens. A turn with nothing to measure gets a row of this kind instead, and its reason says why. With no claim on a pinned, prompt-cached turn that is not toolless or in a cooloff, the reason is fa_no_marker: the gateway sends a claim only when it verified every clause, so the marker was erased or never written. With a claim, the first check that fails names the reason, in this order: fa_field_invalid (the claim's priorExtent is not a whole number of at least 0, so it is rejected and never rounded), fa_toolless (the request declared no callable tools), fa_no_pins (the request carried no pins array), fa_cooloff (a pin_thrash cooloff is active for the session), fa_region_empty (nothing is left to measure: the request has no messages array, or the new part after the prior extent and the last assistant message is empty). The gateway leaves rows of this kind out when it names why a turn was skipped, so they never appear as skipped:*. It also writes its own fa_* rows under this kind.
strategyTimingsMs	Wall time per leg of the hook (id → ms). Keys are the strategy ids that ran, plus reserved phase_ keys for work that is not a strategy: phase_retrieval_commit, phase_pin_replay, and phase_response_patch (diffing the request for a requestPatch, on every attempt, including one that falls back to the full request). The gateway adds six more before persisting: phase_gateway_pin_read (the decision-pin store read, up to its deadline when that deadline fires), phase_gateway_fa_hash (hashing the transcript for first-appearance evidence), phase_gateway_marker_write (the write-ahead stamp, on claim turns only), phase_gateway_loop_lag (how late a pre-call deadline fired because the event loop was busy, plus, for the pin read, the wait after its deadline until the gateway recorded it; only when one fired), phase_gateway_wire (the round trip to this service, minus the handlerMs it reports) and phase_gateway_post (the gateway's own work on the answer, after it arrives), so a trace carries up to nine. It also adds diagnostic keys under phase_gateway_diag_ (hook_offset, pin_read_acquire, pin_read_queued, pin_read_graced, pin_read_gc) that measure time inside or before those legs, so they are never added to the totals. None of the gateway keys travels over this protocol.
handlerMs	Wall time this optimizer spent on the request, from receipt (before auth, signature verification and the body read) to just before the response is serialized. The gateway subtracts it from its own measured round trip to record phase_gateway_wire. An optimizer that predates this field omits it, and the gateway then records no wire leg rather than charging the whole round trip to transit.
vouchedHandles	Retrieval handles on the forwarded messages whose stored row this optimizer saw a write commit for. Each entry has the earliest time that row can expire and the erase epoch the write landed under (epoch, a decimal string). A handle held only in one replica's memory is never listed. The gateway uses it to repeat a trimmed message on a turn the optimizer misses (a timeout or a cooloff hold). It repeats a message only while every handle in it is listed under the erase epoch the gateway read when the request started, more than 5 minutes from expiry, and the turn can call the retrieval tool. Otherwise that turn sends the original. Each gateway replica reads the erase epoch every 30 seconds, and refuses every such repeat after 90 seconds without a successful read. So a replica that did not serve an erase or purge can keep repeating a message that names an erased row for up to 30 seconds after it, or up to 90 seconds while its reads are failing. A retrieve of that handle then returns not found. The replica that served the erase stops at once. At most 256 entries. Omitted when empty.
strategyDeclines	Why a strategy made no edit this turn, as strategy id → reason → count. Route and pricing: route_provider_unproven (the gateway proved no provider for this request: a pass-through seat, an inline config, a custom host, or a default route that resolves to several providers or none, where on the Messages and Chat Completions lanes a load balance counts only the target this conversation is routed to; the fix is the route), seat_no_retrieve, seat_toolless_turn, seat_no_durable_stash and seat_terminal_trim (the route is proven, but the pipeline withheld the seat's evidence because an edit on this turn could not be read back: the client cannot call anyray_retrieve, the request declares no callable tool (a title or summary call, or tool_choice none, from a client that can retrieve otherwise; nothing to fix), the deployment has no store to serve a retrieval handle (content capture off, or no usable content key), or the connect hook made its trim final), route_cache_mode_unknown (the provider is known but its cache contract for this model is not), route_or_price_unknown (the model/provider pair prices only at a fallback rate), retrieval_tools_unknown. Shape: media (the cached prefix carries media), thinking_binding (a retained thinking block leaves no span a mint may start at: thinking_trim and command_digest always, observation_mask, relevance_filter and code_graph only when protectThinkingBinding is true), no_candidate_span (the gate ran and had nothing to price: every span was rejected before pricing, by role, the keep-recent cutoff, the cache boundary, a minimum size, an existing marker, or another strategy's claim), cache_boundary_at_tail (observation_mask or relevance_filter skipped its scan: the client's cache breakpoint is on the last message and no pin has settled a span it may edit; the verdict when no settled span is editable yet on a last-message breakpoint, Claude Code's shape. On a turn the gateway attests as cache-expired, relevance_filter scans through the last completed exchange instead, so there it reads this only when no such exchange is editable), no_query (relevance_filter found no session task, recent prose, or usable tool-call value), no_discriminating_query (a query source exists but has no terms that rank lines), client_managed (the client supplied its own cache breakpoints or context-management edit; code_graph also reports it when protection is off and its only source sits under a client breakpoint), client_managed_cache (the pipeline withheld a new edit that could land inside the prefix the client cached; see protectClientManagedCache), fresh_input_spared (code_graph found source only in tool output the model has not read yet, which stays whole unless freshTailMinLines is set), fresh_tail_below_min_lines (code_graph with freshTailMinLines set: every fresh source read was shorter than the floor), fresh_tail_ranged_read (code_graph with freshTailMinLines set: every fresh source read large enough to outline named a line range, so it stayed whole), fresh_tail_fetched_back (code_graph with freshTailMinLines set: the model already fetched enough of this session's first-sight outlines back that another no longer pays), active_file_spared (code_graph: the only source in reach is a file the agent is editing or the turn names by path, which stays whole), too_recent (relevance_filter or code_graph: a source-file result has fewer than sourceReadMinAge later assistant requests in its own session), still_referenced (a later tool call read overlapping or adjacent lines of that file), source_history_limit (the recorded-read count or overlap-work limit was reached; new source masks are held), durable_stash_unavailable (code_graph found an affordable rewrite but could not store its original for retrieval), masked_anchor (sent only by older optimizers: the retired context_dedupe left a repeat whole because the earlier copy it would point at was already masked or rewritten by another strategy), arrival_unproven (with firstSightOnly, command_digest found a fresh exact repeat but could not prove it was not cached yet, or the request had no pin state to replay from), json_not_byte_stable (the retired context_compression with firstSightOnly left a fresh JSON result whole because minifying it would change more than whitespace: a number re-rendered, an escape rewritten, a duplicate key collapsed, or keys reordered), below_savings_floor (tool_schema_compression found a reduction below its minimum gain), no_lint_finding (cache_lint found no reportable churn or established its baseline), within_token_cap (the output or thinking budget already fits its configured cap), no_action_reported (the strategy returned no decision or decline; investigate its unreported guard), no_applicable_input (no message, tool, or thinking field the strategy can edit), below_activation_threshold (below the size, turn-count, or pressure trigger), routine_turn_not_found (no qualifying routine tool-resume turn), protected_tool_catalog (catalog safety retained every tool), unsupported_request_shape (the endpoint, model, provider, billing lane, body dialect, or missing session context cannot carry the edit), protected_input (another edit, retrieval handle, or pin owns the candidate bytes), no_census_sample (the read-only census emitted no sample: disabled, no compatible messages, or no tool output or paste to classify), admission_closed (observation_mask admission is off by configuration: an explicit mintPaybackRatio with no fill bar and no context pressure), warm_prefix (thinking_trim held a finished turn's reasoning because it sits in the client's cached prefix, and removing it would re-write the cached transcript). Search: search_exhausted is no longer sent by current builds and stays listed because an older optimizer still emits it (it meant the local subset-improvement search hit its step bound and the gate discarded the group; the gate now keeps that group, which was already admitted and profitable, so the bound turns nothing down). mint_batch_deferred means observation_mask passed the payback gate but is waiting for a larger batch; mint_batch_deferred_tokens records the held mass. Older optimizers report these holds as payback_premium. Then four economic verdicts naming which term sank the trim: payback_horizon (no later turn is expected, so nothing repays any cost), payback_premium (the rewrite premium is the larger term), payback_retrieval (expected retrieval cost is the larger term), and payback_short (everything priced, neither term dominant). payback_tail means the edit paid on average but not when the session ends early: priced at the horizon three sessions in four still reach, it lost money, so it was held (the tailAdmission switch in the optimizer config's economics block). payback_rederive is thinking_trim's alone: with trimCurrentLoop on, it held reasoning from the running tool loop because the extra turn the model may spend rebuilding it costs more than the replay it saves. The economic four are worth reading separately: the gate carries two independent cost terms, and a deployment losing to one is not helped by fixing the other. observation_mask checks the route reasons before it scans, so it counts them once on every run that could not mint, and no_candidate_span likewise reports a run that priced nothing at all. code_graph and relevance_filter report no_candidate_span too, for a run whose strategy offered no span at all; when a span survives past the cache boundary they stay silent instead, because the strategy may still edit it and be counted as a firing, and one run must never be counted on both sides. The source-read reasons count blocked candidate spans; every other count is one strategy guard or gate evaluation. Omitted when nothing declined.
shadowHorizon	The shadow horizon's verdicts, as strategy id to { priced, wouldAdmit, wouldAdmitTail, shadowNetUsd, shadowTailNetUsd }, for each strategy that priced a mint on a shadow horizon: the measured session curve while the config did not honor it, or else the client tool's shadow policy version (tenantPolicyShadowVersion). priced counts groups that carried the shadow price; wouldAdmit counts those the gate declined on a payback_* reason that the curve priced positive (wouldAdmitTail on the curve's lower quartile); the two USD fields sum those positive nets. Shadow estimates, never billed. Omitted, never zeros, when no shadow curve rode in. The gateway stores it as counts and dollars only.
horizonEconomics	Up to 12 content-free gate or planner price records, one per decision, with at most 16 members per record. Each carries horizon facts, numeric cost terms, net, and verdict. Admitted records come first when capped. Omitted when the request priced no group. The gateway clamps and stores the array as JSONB for later replay.
cohort	"holdout" or "treated" when audited holdout is on and the request is attributable. A holdout response carries one audited_holdout decision with estimatedTokensSaved: 0.
cohortKind	The single strategy id a holdout response withheld, present only when the optimizer runs a per-strategy arm (holdout.kinds in its config). Absent on a whole-optimizer control. A response never names more than one.
strategyCohorts	One session-gate assignment, as a sparse map of strategy ids to holdout or treated. Omitted kinds carry no evidence; an empty map still marks this protocol. Needs the gateway-derived per-conversation ID. A subagent inherits its parent's draw. A request with metadata.clientTool also carries each drawn kind's arm under its per-tool unit, <kind>__<tool>, with the kind's policy key.
sessionGatePolicies	Strategy kinds to 64-character hex hashes of the assignment epoch plus each kind's tenant parameters and registry flags. Omitted when empty. The older singular sessionGatePolicy stays readable.
tenantPolicyVersion	The version of this client tool's learned policy the request could price with. Versions count per tool, so read it with the row's client tool. Present on any lane where the policy's horizon or retrieval priors apply (the transcript lane gets the horizon only); omitted without a policy and on a whole-optimizer holdout turn. The gateway stores it on the spend row.
tenantPolicyShadowVersion	The client tool's shadow version the shadow slot priced: this response's shadowHorizon counts are what that version would have admitted. Present when a shadow version with a horizon for the request's tool rode in, the thread is a main thread, and the tenant runs no survival shadow of its own; omitted under force_off and on a whole-optimizer holdout turn. Never decided on. The gateway stores it on the spend row.
tenantPolicyLine, tenantPolicyShadowLine	The heavy user's own line (<tool>.u-<digest>) that tenantPolicyVersion, or tenantPolicyShadowVersion, belongs to, when it is not the client tool's own. Version numbers count per line, so read them together. Omitted when the tool's own version priced. The gateway stores both on the spend row.
features.persistentTranscriptPolicyV1	Legacy compatibility assertion: v1.10.120 gateways fail open without it. Newer gateways ignore it.
features.mintSessionSurvival	On /v1/optimize, present (true) only when this tenant's config sets mintSessionSurvival to true or "shadow". The gateway builds the tenant's session survival curve only after it sees this.
features.servedRetrievalPriors	On /v1/optimize, present (true) when the tenant can enable observation_mask's measuredRetrievalPrior: true, or resolves command_digest, code_graph or relevance_filter with measuredRetrieval enabled. The gateway reads handle counts (mintEvidence.handleRetrievals) only after receiving this flag.
features.agentContinuationV1	On /v1/optimize, true only when that response used the activated, handle-free Responses continuation path. On /health, true asserts image support.

cache_optimizer reports one fixed strategyDeclines reason when it makes no edit: client_managed, invalid_request, non_claude_model, route_ignores_markers, subscription_billing, injection_disabled, prefix_below_floor, conversation_breakpoint_disabled, unsupported_anchor_endpoint, turn_varying_strategy_fired, no_settled_anchor, no_session_key, implicit_cache_unavailable, model_without_implicit_cache, client_cache_key, app_harness_client, stamp_disabled, no_followup_expected, ttl_gap, no_session_history, below_min_cacheable, content_replaced, unannotatable_tail, or breakpoint_cap. A handled request with no decision has optimizationStatus: skipped. If its producer omits the reason, spend and trace records use optimizer_reason_missing. A gateway lane counts as a strategy: a turn the optimizer skipped, or that bypassed it, on which a lane booked a saving (laneCredits: cache_ttl_5m, cache_keep_warm, cache_prefix, batch_read, batch_input) reads optimizationStatus: applied, with optimizationReason: lane_applied on the spend row and no reason on the trace. An optimizer timeout or error keeps its status.

The gateway adds metadata.cacheSessionHistory with priorRequests and gapMs from its verified session id. These are counts and elapsed time, never prompt content. An opening request, an unknown session, or a gap of at least five minutes does not get a new cache write. A placed marker names placed in its decision summary.

Sending a session id your adapter can use

Cohort assignment hashes the session within its user (or team). The session comes from metadata.gatewaySessionId when the gateway supplies it, falling back to metadata.sessionId then metadata.session.

Send your adapter's real conversation id, not a per-install grouping label: a label that never varies puts a whole user in one arm and makes fraction sample people instead of requests. Send the same metadata bag on /v1/optimize and /v1/optimize-response, since each leg re-derives the arm and dropping the session on one of them splits a single request across both arms. gatewaySessionId is injected by the Anyray gateway and your adapter cannot reproduce it, so supply your own session or sessionId on both legs.

What budget_exhausted, cache_anchor and warm_prefix mean

budget_exhausted means the turn passed half of the hookBudgetMs the caller sent, and this kind was one the optimizer could stand down without changing bytes the session already sent. Kinds it cannot drop safely keep running past that mark, so the reason names a skipped strategy rather than a stopped turn. Strategies that already ran are kept, settled pins still replay, and the prefix cache anchor is still placed, so the turn goes out optimized instead of forwarding the original bytes.

cache_anchor means the conversation history is cached: cache_optimizer anchors it this turn, or the route caches it implicitly. A strategy whose edits would rewrite cached history stands aside there on every billing lane; no shipped strategy does today. Settled pins keep replaying.

warm_prefix means the system prompt is cached: the client marked it and the route honors the marker, the route caches it itself, or cache_optimizer anchors it this turn. With a marker on the tools alone, the strategy leaves the cached tools untouched and still shortens the system prompt. No shipped strategy is suppressed for it as a whole today. The retired client_prefix_compression was, on every billing lane: a client's stock prefix is the same bytes in every session and already cached, so rewriting it swapped a cheap read for a new cache write. The same name still appears in strategyDeclines, where thinking_trim and reasoning_budget report a single edit they held to keep the cached prefix intact.

strategy_failed means that strategy raised while running, so its result for this turn was discarded. The turn still goes out optimized, and the failed kind's settled pins replay. Treat it as a bug in that strategy, not a property of the request.

How strategyCohorts slices a session

With sessionGate.measurementFraction above 0, each configured kind draws two equal, disjoint slices at that fraction. A session-level selector measures at most one, so a session is never a control for more than one strategy. The selected kind is labelled holdout (withheld) or treated (run). Every other kind runs on its promotion state alone and carries no arm label. A session that selects nothing labels nothing, and per-turn enabledKinds allowlists do not redraw the selection.

holdout.enabled controls the separate audited-savings experiment. Its per-kind control uses holdout_kind; the session gate records session_gate_holdout with its holdout arm.

Static session/sessionId fallbacks do not qualify. Lane stand-asides are excluded turn by turn and do not make the remaining session mixed. A control draw with its own settled pins replays them without new mints and omits the kind. cooloff retains own-kind pin-thrash cost without supplying an arm. New responses do not emit excluded; older explicit values stay readable.

A request selection that permits every configured kind keeps the usual labels. One that restricts a configured kind emits cooloff and supplies no new experimental arm. In a session already labelled by an unrestricted turn, the restricted turn's full cost stays in the session total. A session made entirely of restricted turns has no arm label.

Capabilities

Capability	What it does
cacheAnchorTails	Up to 4,096 { index, hash } entries for live earlier request tails, attested by the gateway. index is a non-negative safe integer; hash is 32 lowercase hex characters from cachePrefixHash(request, index) in @anyray/observability. It hashes { tools, system, messages: messages.slice(0, index + 1) } with absent fields as null, and removes cache_control only where a provider reads a breakpoint (a tool, a system block, a message, a content block, a block inside a tool result), using the first 32 hex characters of keyed HMAC-SHA-256 under the shared handle key. The Claude Code billing line in system[0] hashes as a fixed placeholder. The optimizer checks each hash once, against the request it is about to send, in a pass after every other strategy. Used only with observation_mask.mintAnchor: true on explicit Bedrock routes, past the measured lookback window. The Anyray gateway sends it on the routed Messages lane for explicit Bedrock routes once the optimizer answers features.cacheAnchorTails: true, and only for provable 1h entries. Out-of-range indices are ignored; any malformed entry drops the whole field to absent without failing the call. Never derive this from client metadata.
canShortCircuit	Lets the optimizer answer from semantic_cache instead of the provider.
acceptRequestPatch	The caller can rebuild the forwarded request from a requestPatch (see below) instead of a full request echo. The Anyray gateway always sends it.
hookBudgetMs	How long the caller will still wait for this hook, from the moment it sent the request. Sizes the durable-stash write allowance (three quarters) and the point where strategies start standing down (half). Omit it and the optimizer falls back to a fixed 300 ms write allowance and no admission deadline.
hookBudgetWireDebited	Sent only as true when optimizerWireDebit is on and this session's published wire debit (a measured median in 100 ms steps) is above zero, under the 400 ms cap, and withheld whole from hookBudgetMs. A durable-stash write may then run to the write allowance instead of stopping at 300 ms. Absent: 300 ms.
singleOutput	Marks the connect hook's synthetic [intent, tool output] request on the reversible branch (anyray_read, anyray_search, or a Read that can retrieve): relevance_filter stands its recent-tool and fresh-input spares down and keeps retrieval handles.
terminalTrim	A handle-free trim of one tool output. It suppresses every strategy, so the output comes back byte-identical and each configured kind appears in suppressedKinds with reason terminal_trim. It wins when it arrives together with rangedTrim.
rangedTrim	The connect hook's ranged lane. When the tenant's relevance_filter sets rangedHookTrim: true, only relevance_filter runs, without a retrieval handle, and each omitted run reads … [N line(s) omitted by anyray · lines A-B], numbered as 1-based lines of the tool output; every other kind is suppressed with reason ranged_trim_scope. Without the opt-in, every kind is suppressed with reason ranged_trim_opt_out and the output comes back byte-identical.
retrieveEvidence	The gateway's own evidence that the caller can resolve handles.
retrieveEvidenceClass	Why the gateway reached that verdict. On a turn that declares no callable tool, toolless means the gateway had nothing to inspect, so the optimizer reports toolless_turn instead of no_retrieve. Any other class keeps no_retrieve.
allowIrreversibleWindowFit	Reserved for one-shot Batch API lines.
agentContinuation	A Responses transcript with a verified trailing typed function_call_output and no callable tool catalog. See the accordion below.
providerPromptCache, providerPromptCacheKnown	Marks an upstream that implicitly caches prompts whatever the model id (GitHub Copilot, say). With …Known, the optimizer trusts the answer in both directions: true protects a model id it would not recognize, false releases cache-busting on a host that provably does not auto-cache. Without it, it classifies by model id.
providerExplicitCacheControl, providerExplicitCacheControlKnown	Whether the resolved upstream honours the client's own cache_control breakpoints. Some routes accept an Anthropic Messages body and translate it to OpenAI chat, which has no equivalent field, so markers are dropped before the request leaves the gateway. The model id cannot see this, and a marker the upstream never received is not a cache boundary.
providerAutomaticCacheControl, providerAutomaticCacheControlKnown	Whether the resolved upstream honours Anthropic's request-level cache_control (automatic caching). A separate answer because a route can keep block breakpoints and drop this field: Bedrock's Converse lane turns breakpoints into cache points and has no field for it, so Bedrock reads false. A Bedrock route that may take the native InvokeModel lane (anthropicInvokeNative), which forwards the body as sent, sends nothing. Absent, the providerExplicitCacheControl answer decides, except that a route whose mintEvidence names Bedrock reads as uncached.
mintEvidence	{ provider, cacheMode: "explicit" | "implicit", retrievalToolsStable, pendingRetrievalToolTokens? }. The gateway attests a direct upstream and whether retrieval tools are already stable in the prefix. A direct upstream is a client-pinned provider, or the org default route once it resolves to one provider for the request (a conditional tree narrowed to the matching branch, a load balance narrowed to the one target dispatch picks for this conversation on the /v1/messages and /v1/chat/completions lanes, or the sole configured key). Unknown, custom-host, mixed, inline-config, or pass-through routing supplies no evidence here (pass-through Claude seats use passthroughMintEvidence). retrievalToolsStable is true when no edit on this turn can add the retrieval tools to the wire: the session's stored first-turn choice keeps them there (eager, the default for a client that declares them), the gateway's session latch proves they were on the previous turn's wire, a retrieval marker sits in the part of the conversation the previous turn already sent, the session row records that an earlier request kept them, or the session holds a pin carrying a retrieval handle. Every other turn that declares them reports false, and the optimizer then prices a mint with those tool definitions counted as uncached instead of declining it. With false, the optional baselineRetrievalActivation: true says the conversation already carries a retrieval marker, so the gateway keeps the lazy tools on this turn whatever the optimizer does: that header re-write is the baseline's, and no edit is charged or debited for it. The optional pendingRetrievalToolTokens counts gateway-served schemas appended after optimization; admission charges their activation and later reads. Automatic cached mints need known pricing and estimated payback, including the shared 5% retrieval reserve for handle-bearing edits, within mintPaybackTurns later turns (derived from the post-edit context shape when unset, where observation_mask counts a Lindy estimate of the remaining turns from 24 elapsed turns on; any explicit value keeps the older third-of-exchanges prior; 0 for the mint turn alone); missing evidence declines them. On a route priced by the rewritten region, observation_mask also holds an admitted group until its rewrite is repaid within mintBatchTurns turns (default 3; 0 mints on the admitting turn); the hold declines as mint_batch_deferred. Old gateways may omit this capability safely. Never promote client metadata into it.
mintEvidence.sessionSurvival	Optional { turnsLeft: number[], turnsLeftQ25?: number[], aliveByTurn?: number[], sessions }. At elapsed index t, turnsLeft[t] is the mean further requests and turnsLeftQ25[t] is the most further requests three in four surviving sessions made (same array length). The gateway measures this tenant and client tool's completed main-thread sessions over 14 days, stopping each life at the first compaction (prompt size below half the previous request). Requires 100 sessions; mean and quartile support ends below 20 survivors. With mintSessionSurvival: true (default off), main-thread admission uses this curve, capped at 60 turns; subagents keep the prior. Otherwise it reports shadow metrics only. aliveByTurn[t] counts sessions longer than t: expected pricing sums conditional survival through the operator and context limits, at most 60 future turns. Its maximum 572 entries must cover the elapsed index plus 60. Missing or unsupported counts retain mean-based pricing; the quartile basis still reads turnsLeftQ25. Invalid curves fall back to the prior without failing the call; invalid turnsLeftQ25 or aliveByTurn columns drop alone. mintPaybackTurns still narrows the horizon. The gateway builds and sends curves only for known client tool ids whose tenant received features.mintSessionSurvival within 30 minutes; other traffic adds no trace-store reads.
mintEvidence.subagentSessionSurvival	Optional, the same shape as sessionSurvival, measured per Claude Code subagent thread: each thread's requests up to its own first compaction (its prompt falling below half its previous one on the same model), over the same tenant, client tool, window and minimum (100 threads). Sent only on a subagent request, for the same opted-in tenants. Never read by admission, whatever mintSessionSurvival says: a subagent group reports it as the shadow_* metrics and decides on its prior. A curve outside the schema drops to absent; it never fails the call.
mintEvidence.handleRetrievals, passthroughMintEvidence.handleRetrievals	Optional { ageMs, kinds: [{ kind, minted, retrieved }], classes?: [{ kind, retrievalClass, minted, retrieved }] }. Counts handles first stashed in 48 hours, excluding the newest hour, and those retrieved at least once. Each replica refreshes from the shared context store in the background, one tenant at a time, at most every 30 minutes. Requires features.servedRetrievalPriors within 30 minutes. kinds holds at most 64 pooled strategy totals. observation_mask reads them with measuredRetrievalPrior: true (default off) and maskSourceReads: "never" (default): at 200 handles and 5 retrieves, its output prior becomes the 95% upper bound and the 95% lower bound floors its price. Below either minimum, it retains the measured 5.4% prior. classes holds at most 64 unique strategy/class pairs (source_read, output, or unknown). With measuredRetrieval enabled, code_graph and relevance_filter use matching class counts on retrieval-capable lanes; the 95% lower bound can only raise their price. Pooled totals never price these mixed strategies; lossless edits retain zero retrieval cost. First-mint classes cannot change; legacy rows remain unclassified. Counts older than 2 hours or malformed in any entry are ignored. Invalid payloads drop to absent without failing the call. Never derive this field from client metadata.
mintEvidence.tenantPolicy, passthroughMintEvidence.tenantPolicy	Optional { version, retrieval: [{ kind, cls, p, n }] }: the tenant's active learned policy. Each entry replaces the typed retrieval prior for one strategy and content class (source_read, output, or unknown) with the fetch-back rate p, above 0, fitted on n handles. An entry may lower the prior only with at least 200 handles, and never below the live served upper bound on the mask's measured lane. A configured floor, the session's own fetch-back rate and the live measured raises still apply on top. Summarizing strategies (command_digest) keep their flat reserve. At most 192 unique entries. The optional horizon holds, per client tool, a main-thread survival curve frozen at fit time (the sessionSurvival shape): that tool's payback horizon prices on it, in place of the prior, the live curve and the slope table, and it may widen as well as narrow. Sent only once a version is active; each replica refreshes it every 5 minutes. The retrieval entries are withheld on the transcript lane; the horizon applies on every lane. A malformed policy drops to absent without failing the call. Never derive this field from client metadata.
mintEvidence.tenantPolicyCandidate, passthroughMintEvidence.tenantPolicyCandidate	Optional, the same shape: the tenant's version under measurement. Both are the request's own client tool's line (metadata.clientTool); a request with no client tool gets neither. The candidate prices only in sessions on the treated arm of that tool's session-gate unit, tenant_policy__<tool>, which draws only where no strategy is measured; the holdout arm prices with tenantPolicy. Those sessions carry strategyCohorts.tenant_policy__<tool> and a sessionGatePolicies key per tool and version. force_off on the tool's unit, or on the tenant-wide tenant_policy, prices with neither version.
mintEvidence.tenantPolicyShadow, passthroughMintEvidence.tenantPolicyShadow	Optional, the same shape: the request's client tool's newest unpromoted (shadow) version. It changes no decision and no byte. The shadow slot prices every mint on its horizon as if it were active and reports the counterfactual in shadowHorizon, named by tenantPolicyShadowVersion. force_off on the tool's unit or on tenant_policy drops it.
mintEvidence.tenantPolicyCohort, passthroughMintEvidence.tenantPolicyCohort	Optional { line, active?, candidate?, shadow? }: a heavy user's own policy line, the key's verified user and never a header claim. line is <tool>.u-<12 hex>; each version has the shape above, and a malformed one drops alone. Each version it holds replaces the tool's for that field, and one it omits stays the tool's. A candidate on it is judged under the line's own unit, tenant_policy___<digest of line>, with a sessionGatePolicies key per line and version, never the tool's. force_off on the line's unit, the tool's, or tenant_policy prices with neither. An optimizer that predates the field ignores it and prices on the tool's policy.
mintEvidence.currentWriteCacheTtl, passthroughMintEvidence.currentWriteCacheTtl	Optional "5m": the gateway has fixed this request's cache writes at the 5-minute tier, including its retry body. Sent for proven native Claude main-thread lowering. On explicit Anthropic routes, warm rewrites with unchanged retrieval tools use that tier; positive fresh-tail credits cannot exceed it. Original markers still price future retrieval, cold turns, and first retrieval-tool activation. This does not change the rewritten region or payback horizon. Missing or malformed values keep conservative pricing. Never derive this field from client metadata.
mintEvidence.tailLookbackMissed, passthroughMintEvidence.tailLookbackMissed	Optional true: this tenant and provider's tail-lookback misses (rows whose real cache read fell to about the header where the price assumed it read back to the edit) outrun what its mints can absorb, or the gateway has not read the shared verdict history recently enough to know. The threshold is the first quartile of the pair's reported tailBreakEvenMissRate. The window is the fewest whole hours holding enough checks to rule that rate out at 95%, up to 24. The latch needs more misses than that rate makes likely in the window at 95%, and never fewer than two. It clears as the misses leave the window or hits outweigh them. The optimizer then prices every mint at the whole-prefix price, exactly as mintWholePrefixPrice: true does. Any other value is treated as absent. Never derive this field from client metadata.
passthroughMintEvidence	Same shape as mintEvidence, sent only for a claude-* pass-through request pinned to anthropic with no custom host or inline config (a subscription seat, which the gateway dispatches straight to Anthropic). It has its own field so an optimizer that predates it drops it instead of honoring it. The optimizer uses it as mint evidence only when mintEvidence is absent, the passthroughMintEvidence setting (PUT /admin/optimizer/settings) is not false (on by default), and the turn can retrieve. A seat that cannot retrieve would otherwise get edits it can never restore. Turning the setting off stops new mints; edits already pinned keep replaying until observation_mask and command_digest are disabled.
transcriptHandles	Runtime optimizer setting (PUT /admin/optimizer/settings), on by default. Under content mode off, permits handles into a current connect seat's local Claude Code or Codex transcript when the gateway verifies local retrieval. On Claude Code that means the turn declares connect's own stdio tool, mcp__anyray__anyray_retrieve; a turn whose only retrieve tool is another server (the gateway's remote /mcp, an SDK's own tool) gets stashed handles or none, because only the local server can read the transcript. Set false to keep the previous no-stash behavior. No server content row is written for these handles.
hiddenByMessage	One entry per request.messages entry, { chars, reasoningChars, media, unpriceableMedia }: what that message's wire form carries that the request does not show. The gateway sends it for Codex /v1/responses traffic, whose optimizer request drops reasoning items, call arguments and images. A cached mint re-writes those bytes too, so the mint gate counts them when it prices a mint. The three are split because the provider bills them at rates two orders of magnitude apart: chars tokenize as prose, reasoningChars are opaque encrypted_content billed far under what they tokenize to, and media counts whole image parts, each billed by its dimensions rather than the size of its payload, and unpriceableMedia counts binary document parts, whose cost follows a page count nothing on the wire reveals and which stand the mint gate down instead of being priced. The reasoningChars rate is per model family, and a family with no measurement of its own is priced as prose. Counts only, never content. Replaces hiddenCharsByMessage, a single character count per message priced entirely as prose. Both go on the wire during the rolling update, and the optimizer reads the older field only when this one is absent, so neither half of a mixed fleet loses the hidden bytes.
semanticRerank	Enables the semantic rerank path.
clientContextBudgetTokens	A lower client-local context ceiling for callers whose transcript compactor runs before the provider's model window. Mint admission caps repayment using the original client transcript; wire compression does not move that local compaction clock. For Codex the gateway derives it only from the user's configured model_auto_compact_token_limit or model_context_window, with a 5% safety margin. No user setting means no client ceiling. Client-supplied modelInputErrorTokens is ignored.
promptCacheExpired	Sent only as true: the session sat idle longer than the route's prompt-cache TTL, so the provider re-bills the whole prefix on this turn. The gateway measures the gap on the database clock from the session row it stamps every turn. It sends nothing when the request's transcript does not show that the row's last stamp is the previous turn's, or, on a Claude Code main thread, a sibling's (a request with the same history and a different last message, such as the background request Claude Code sends a few minutes after a reply). On a session it keeps warm with pings, the gap must also outrun the last moment a ping could have refreshed the cache. With promptCacheRecordedExpiry on (off by default), the session row decides instead: it stores the lifetime the gateway sent (5m on a session the 5m setting or header lowered) counts each ping before it is sent, and notes each request of the session's sub-agents, so no ping may be out and the last ping and sub-agent request must be older than that lifetime. The kinds in the config's coldTurn.premiumKinds (observation_mask and relevance_filter by default) then price a cached-prefix mint's rewrite premium at zero, and thinking_trim a strip's on its own (relevance_filter also scans through the last completed exchange when nothing has settled yet); the payback horizon and retrieval risk still apply, and mintPaybackTurns: 0 still declines. code_graph and command_digest do the same only when the list names them, and only when every priced mint has a retrieval handle and the optimizer is not placing its own conversation cache breakpoint on the request. Absent means unknown and changes nothing.

hiddenCharsByMessage is the pre-split form of hiddenByMessage, still accepted and still sent. It prices every hidden character as prose, which over-counts an image by about 130x. Retire it once no optimizer in the fleet reads it.

persistentTranscriptPolicyActive was accepted here through v1.10.120. Newer optimizers drop it unread.

Activate agent continuations after the rolling update

agentContinuation is cache-affecting and defaults off under config.rollouts.agentContinuationV1ActivateAt. Roll every gateway and optimizer replica to a version whose features.agentContinuationV1 is true. Then set the field once through PUT /admin/optimizer/settings to an RFC 3339 instant with an explicit Z or numeric offset, at least 60 seconds in the future. The API rejects a first activation less than 60 seconds ahead, and makes the timestamp immutable after it is set. Keep the field absent until the fleet is compatible. Replicas refresh shared config every 15 seconds, so a current or past activation time can briefly alternate original and optimized prefixes. Before activation, continuation requests fail open and enter the normal session cooldown.

What agentContinuation permits, and the catalog test

The capability permits byte-stable decision-pin replay and settled-history optimization. It does not grant retrieval: degradable strategies stay non-reversible, emit no retrieval handles, and preserve the fresh function result.

The catalog test runs over the RESOLVED request the optimizer receives, not the client's raw body. Codex 0.146 and later omit top-level tools and carry the catalog in an additional_tools input item, so a gateway reading the raw field alone asks for a lane the optimizer refuses, and the refusal holds the session.

Proving retrieval

An exact anyray_retrieve declaration, or a hosted MCP connector aimed at the gateway's own /mcp, proves retrieval is callable on that turn and renews the authenticated lease. Valid MCP activity, model retrieval calls, and the Connect MCP server's heartbeat also renew it.

Claude Code's measured parent signature (callable ToolSearch plus its exact anonymous DeferredToolPlaceholder) may use the fresh lease when deferral hides the declaration on a later turn. Generic MCP tools, generic defer_loading, either half of that pair, and restricted Read/Bash agent shapes fail closed.

Claude Desktop sends the same signature through its bundled Claude Code, often on the same key, but its MCP registration is separate. A Desktop request uses only Desktop's own lease on that key, written by its declarations, heartbeats from the Connect MCP server Desktop launched, or Desktop's calls to the gateway's /mcp. A lease from Claude Code or a heartbeat from a server another client launched does not count, and a re-minted key starts without one.

Whatever those say, a request that declares no tools, or sets tool_choice to none, normally runs the no-retrieve lane and withholds decision-pin replay: the model cannot call the retrieval tool that turn, so no handle is emitted or replayed. A Responses agent continuation is identified separately through agentContinuation. Other tool-less requests keep the compaction and summarization stand-down behavior.

Clients that cannot retrieve

A caller with no retrieval path is handled by billing lane:

Lane	Behavior
Subscription seat	The degradable eliders still run, forced to reversible:false. They trim in place and emit no retrieve ctx_… handle the caller could not resolve, which keeps savings for seats that would otherwise get none.
API key	They are skipped. A trimmed byte there is billed money, and an agent handed a placeholder it cannot expand re-reads the input and spends more than the trim saved.

config.requireReversibleElision (PUT /admin/optimizer/settings) overrides the lane rule: true skips on every lane, an explicit false restores the in-place trim on api-key lanes. Unset, it follows the lane. It gates only the no-retrieve degrade, never a run that can stash.

A client with a read path keeps the reversible strategies. The gateway serves the tools at POST /mcp (anyray_retrieve, anyray_recall), authenticated with a client key as either x-anyray-api-key or Authorization: Bearer. It also answers them itself (served evidence) for an API-key request whose class an admin opted into servedRetrievalClasses; none is opted in by default, so a tool loop with no read path of its own goes through unchanged.

Decision pinning

A decision pin records one settled-history rewrite, so a later turn can replay it byte for byte instead of re-deciding it against the live turn. Re-deciding is what busts the provider prompt cache on warm sessions. The gateway persists the pin list keyed by session and round-trips it through /v1/optimize. An adapter treats each pin as opaque.

Each pin carries a message index, an HMAC spanHash fingerprint (of the original bytes, tool universe, or model, never content), and a replay recipe. On each turn the optimizer replays every pin whose spanHash still matches and fresh-decides only the suffix. A mismatch, meaning the client edited or compacted that span, discards just that pin.

A pin stored by an older release may also carry toolUseId, the call id of the tool output it trimmed (the retired Codex seat lane). Nothing mints such pins any more, and the optimizer drops one at intake: it is not replayed, not counted as a discard, and not returned in the response pins.

Field	Meaning
Request pins	The session's pins so far.
Response pins	The pin list after this turn: still-matching plus newly recorded. The gateway persists it. Absent when the request sent none.

Three rules an adapter must honor:

• The response list is a full REPLACE, not a merge. A pin missing from it is deleted. A pin the optimizer declined to apply this turn is still returned, so persist the array verbatim rather than filtering it against what visibly changed in the request.
• Fail open. No pins on the request (first turn, older gateway, or a store read error) means normal warm-cache behavior.
• A pin is keyed to the client's original bytes, which the client re-sends every turn, so its validity does not depend on the optimizer having applied it last turn. A pin keeps replaying across turns where its strategy did not run at all. Leaving those spans alone would send the original bytes, which rewrites the cached prefix exactly as re-deciding would.

The thrash guard

When a turn's discards outnumber its replays (two live conversations sharing one pin session, for example), re-minting every turn would re-write the cached prefix every turn. After two consecutive mostly-miss turns the optimizer suppresses cache-busting strategies for a few turns (suppressedKinds reason pin_thrash) and carries the guard's two counters in an opaque pin_thrash_guard marker inside the pins array. Adapters need no special handling: the marker is an opaque pin like any other.

A single all-miss turn, such as a client-side compaction, never triggers it. The session's existing pins are returned and keep replaying throughout the cooloff, handle-bearing ones included, because dropping them would make the turn the cooloff lifts re-record the whole settled prefix at once. A handle the store cannot confirm on a cooloff turn renders its placeholder without the handle rather than the original span. No new edit is minted while the cooloff runs.

Session participation

The kinds that take part on a session's first prompt-cached pinned turn stay the participating set until the prefix goes cold: a cache expiry the gateway attests (capabilities.promptCacheExpired), or a cleared pin list. A config change, a holdout draw, a regret-guard trip or an optimizer upgrade mid-session neither starts a new kind's edits (reported as session_participation in suppressedKinds) nor drops a participating kind's settled pins, which keep replaying. The set rides the pins array as an opaque session_participants marker: one recipe key per kind plus treated (1 when the session began in the treated arm of the whole-optimizer holdout, 0 in control; that arm holds for the session too). Adapters persist it like any other pin.

POST /v1/optimize-response

Post-call hook to transform the response. Only output-stage strategies act. Skip it for streaming responses.

{ "endpoint": "/v1/chat/completions",
  "request":  { ...the request that was sent... },  // OPTIONAL context
  "response": { ...the provider response... },
  "metadata": { "user": "u1" } }
// → { "protocolVersion": 1, "response": { ... }, "decisions": [ ... ] }

POST /v1/cache

After a successful response, write it back so the next identical request hits cache. Use the cacheKey from the optimize response. Echo it unchanged (it carries an opaque purge fence). A write-back without cacheKey returns 400.

{ "cacheKey": "<opaque key>", "response": { ... }, "ttlSeconds": 3600,
  "endpoint": "/v1/chat/completions",        // the route this answer came back on
  "metadata": { "user": "alice" } }          // the same attribution you optimized under
// → { "ok": true }
// → { "ok": true, "stored": false }   // invalid or stale key, or no partition

• Send back the same metadata the matching /v1/optimize call carried. A cacheKey addresses one caller's partition, and the server refuses a key that was not minted for the scope this request's metadata resolves to. That is a consistency check, not authorization; authenticating the caller is request signing's job.
• An invalid or stale cacheKey is refused with stored: false. A peer may accept a write-back before it polls a new erase epoch; that poll clears the entry.
• Send the endpoint too. It records which wire dialect produced the answer. The same messages on /v1/chat/completions and /v1/messages want different response shapes and the optimizer returns a cached response unchanged, so an entry is served only to the route it came from. Omitting it is safe on an older gateway.
• Measurement-only write-back. Add "shadowOnly": true when the optimize response carried cacheShadowOnly, and for every streamed response (send the reassembled message, not the SSE). The optimizer fingerprints it for the shadow tier and never stores it as servable, so a stream can never be replayed to a later caller. A key minted on such a lane stays measurement-only even if the flag is dropped.
• Hook results are not stored. A top-level hook_results key in response is the gateway's guardrail report for one request, and it can quote that caller's prompt. The optimizer drops it and stores the rest, so a hit never replays it.
• Multi-tenant. A caller using a per-tenant bearer must also send request. The server re-derives the key under the authoritative tenant, so a forged cacheKey cannot cross namespaces, and returns 400 if it is absent. The shared-token gateway lane uses cacheKey as-is.
• Caller partition. Entries are partitioned per caller: the tenant, plus, in order, the attributed user, the team, or the gateway-stamped gatewaySessionId. The userId and teamId aliases are deliberately not read, because honouring them would let a caller name someone else's partition. A request none of those resolve for has no partition: nothing is stored and the reply says stored: false. For the same reason /v1/optimize returns no cacheKey and cacheEligible: false for such a request.

POST /v1/retrieve

The recoverable side of any reversible strategy (context_compression, observation_mask, output_externalize, …). When one drops (observation_mask, relevance_filter, code_graph, …). When one drops content, the original is stashed under an opaque handle left in the request. Fetch it by handle:

{ "handle": "ctx_3kf9q", "metadata": { "source": "model" } }
// → { "handle": "ctx_3kf9q", "content": "...the original...",
//     "totalLines": 4000, "totalChars": 181204, "partial": false }

Status	Meaning
404	Unknown or expired handle, or a handle minted for a key the caller cannot read as.
503	Unavailable durable store, or an unreadable encrypted row. The gateway relays it as retrieval unavailable instead of telling the model the handle never existed.

Every attributed handle is bound to the key's own metadata.user and answers that user. The gateway may add a top-level readers list, resolved from the verified key: for a person, the agents assigned to them (ownerUser), so an owner reads back what their agents stashed. It is one way: an agent never reads its owner's handles, and a shared bot with delegated attribution is never a reader. The end users such a bot acts for don't read its handles either: the subjectUser tag on a row only lets their erasure reach it. A handle minted with no attribution, or before handles were bound, stays readable by anyone in the tenant for as long as sessions keep replaying it.

Partial retrieval is opt-in. A bare { handle } always returns the full original, unchanged. To fetch only part of a large stash, add a line range, a grep, or both (grep searches within the range):

{ "handle": "ctx_3kf9q", "startLine": 3180, "endLine": 3240 }   // verbatim 1-based inclusive slice
{ "handle": "ctx_3kf9q", "grep": "FAILED" }                     // case-insensitive LITERAL, ±3 context lines
// → { "handle": "ctx_3kf9q", "content": "@@ lines 3178-3186 @@\n…", "partial": true,
//     "totalLines": 4000, "totalChars": 181204, "startLine": 1, "endLine": 4000,
//     "grepMatches": 27, "grepTruncated": true }

startLine and endLine are rounded and clamped server-side, so a fractional or out-of-range value never errors. grep is a literal, never a regex, whitespace-significant, at most 256 characters (whitespace-only or oversized gives 400). Every partial response reports the bounds actually searched and the stash's totals. grepMatches is the total match count, with only the first 20 windows rendered (grepTruncated: true announces the cut).

Version skew fails loud, never silently full

A slice request against an optimizer that predates these fields is answered by the gateway with 503 and code: "slice_unsupported", so the model is told to re-call without slice params rather than silently receiving (and pinning) the full copy it asked not to get. In the mirror case, a new connect client against an older gateway that ignores slice fields, the client detects the non-slice-aware response shape and returns the same guidance as a tool error.

A caller qualifies for fresh reversible trimming when the current turn directly declares retrieval, or when a deferred catalog has authenticated retrieval activity less than 6 hours old. The Connect stdio server renews that lease every 5 minutes while its initialized transport is alive; valid calls through the stdio or remote MCP path renew it too.

Some clients name no MCP tool in the request at all and discover them at runtime instead, so their catalog can never carry the proof. There the Connect server reports its own client name and whether the retrieval tool is approved for it, and that approval is what qualifies the turn. The verdict is fixed for the length of a session: it would otherwise change mid-conversation when a lease ages out, and each change rewrites bytes the provider has already cached.

The gateway records an evidence class (direct, deferral_latched, unlatched_deferred, approval_latched, unapproved_client, no_discovery, toolless, forced_handler, or served) in the spend row's existing retrieval JSONB and exposes the split from /admin/spend/dashboard. served means the gateway served the retrieval tool itself because the catalog had none. The optimizer contract does not change for served: the gateway sends clientCanRetrieve: true and retrieveEvidence: true exactly as it does for a declared retrieval tool.

Tiers and the commit rule

The per-process in-memory tier is a bounded read-through cache. Entries expire after the strategy's ttlSeconds, 4 hours by default, and are never logged.

Strategies stage new handles without doing I/O. After they finish, the optimizer persists every required row in one request-scoped batch, and a fresh ctx_… handle is emitted only if its row committed. A failed required row rolls back that strategy's transform and every later transform that may have observed it; earlier independent transforms may survive.

Durable rows are AES-256-GCM blobs, gated by the content mode and never written when it is off. An outage can make an older pinned handle unavailable, but it cannot mint a new unusable handle or rewrite the settled prefix.

ageBucket, sliding expiry, and the guard signal

A successful retrieve returns ageBucket, a coarse content-free bucket for how old the marker was (lt_1m, lt_5m, lt_30m, lt_2h, lt_4h, lt_12h, lt_24h, lt_48h, gte_48h), absent when the age is unknown. The console reports the distribution as retrieval.ageByBucket. It never reaches the developer's client.

A successful retrieve also extends the handle's life by 30 minutes in both tiers, bounded by an absolute 48-hour read ceiling. Re-storing the same content starts a fresh window, so that ceiling bounds how far retrievals carry one original, not how long recurring content stays stored. A handle the model keeps using stays resolvable; one that is never read expires on its original schedule.

Model-requested retrieves feed the task-outcome regression guard. Automated callers must set metadata.source ("splice", "console", …) so they are excluded: a mechanical fetch is not evidence the model needed the content back.

How a client reaches it

The optimizer is in-network only. The gateway proxies retrieval on its client surface at post /connect/retrieve (port :8787, client-key gated), and anyray-connect exposes it to the model as the MCP tool anyray_retrieve(handle), with the same optional startLine/endLine/grep params. When a strategy leaves a · retrieve ctx_… marker, the model passes that handle to anyray_retrieve and gets the original back. The trim is lossless, and for a large stash the model can pull back just the slice it needs. A partial result opens with an [anyray: partial — …] marker line saying how to get the rest. The tool stays out of context until used.

POST /v1/recall

The semantic counterpart to /v1/retrieve: find a stashed original by meaning when its · retrieve ctx_… marker has scrolled out of context. Pass a natural-language query. The optimizer ranks it against the durable stash's encrypted embeddings and returns ranked handles plus a preview, and the model then calls /v1/retrieve on the chosen handle.

{ "query": "the failing pytest run", "k": 5 } // k optional (default 5, capped at 20)
// → { "matches": [
//      { "handle": "ctx_3kf9q", "score": 0.91, "preview": "pytest: 2 failed, 28 passed…",
//        "startLine": 3180, "endLine": 3240 }   // range present on chunk-indexed matches
//    ] }

Long stashes are indexed in chunks, so a match can point at the exact line range where the meaning lives, and the model passes that range straight to /v1/retrieve instead of re-fetching the whole output. Matches from older, head-only index rows omit the range.

• Durable-only. Recall reads only the durable tier. Content mode off stops new rows being stashed but does not hide rows stashed beforehand. It fails open: empty matches with no embedder, no durable backend, or nothing stashed, never an error.
• User-scoped. Recall ranks only the requesting user's own stash, so one user can never discover a co-worker's externalized output, even within one tenant. As strong as the user attribution itself.

Like retrieve, recall is in-network only. The gateway proxies it at post /connect/recall, exposed to the model as anyray_recall(query).

POST /v1/record (BYO only, opt-in)

In a BYO-gateway deployment (your gateway, no Anyray gateway in the path) the optimizer becomes the trace writer. The adapter POSTs one normalized record per call, and the optimizer writes the same trace the Anyray gateway would, so the console looks identical either way.

{
  "id": "litellm-call-uuid",        // required → trace id (idempotent: a retry upserts)
  "ts": 1750000000000,              // epoch ms or ISO; defaults to now
  "endpoint": "/v1/chat/completions", "method": "POST", "status": 200,
  "model": "gpt-6.1-sol", "provider": "openai",
  "promptTokens": 9000, "completionTokens": 5, "totalTokens": 9005, "durationMs": 1200,
  "attribution": { "user": "u1", "team": "t1", "sessionId": "s1" },
  "cacheReadTokens": 8000, "cacheWriteTokens": 500,   // optional: parts of promptTokens, never additive
  "decisions": [ { "kind": "tool_schema_compression", "estimatedTokensSaved": 100,
                   "cachedSavedFraction": 1, "pinsReplayed": 2 } ],   // optional cache-tier evidence
  "content": { "input": ..., "output": ... }   // RAW, gated/encrypted server-side
}
// → { "ok": true }   ·   404 when recording is disabled (the default)

• Disabled by default. 404 unless the optimizer starts with a trace-store DB configured (ANYRAY_OBSERVABILITY_DB_URL, falling back to ANYRAY_SPEND_DB_URL). Set it only in a BYO deployment: with an Anyray gateway in front, the gateway is the sole writer, so this would double-record. content is gated by the deployment's content mode.
• Send content.input and content.output as JSON objects. Hosted MCP connector credentials are redacted by reading the body's structure, so a body sent as a string keeps any credential in it.
• Spend write. With ANYRAY_SPEND_DB_URL set, /v1/record also appends a SpendRecord (model, provider, tokens, cost, latency, attribution), so BYO-adapter savings show up in spend and ROI reports like native-path savings.
• Cache evidence prices the trim. cacheReadTokens, cacheWriteTokens and cacheWrite1hTokens are parts of promptTokens, as the provider reported them; without them the row's input prices fresh and its trims earn no cache tier. For openai and azure-openai a cache write is a subset counter, as on the native path: it never lifts a trim to the write tier. A decision may carry cachedSavedFraction (0 to 1), pinsReplayed and prefixRewriteTokens, the same fields the optimizer's own decisions carry; a malformed value is dropped, never clamped. savedTier is ignored.

GET|PUT /admin/optimizer/settings (admin)

The strategy registry is runtime-mutable: toggles, params, and per-endpoint overrides for every strategy, hot-reloaded and audit-logged.

Method	Returns
get	The current config, plus live guard, census, cache-lint, holdout, context-recall, and semantic-cache-shadow telemetry.
put	Replaces the config. Accepts the bare object or { "config": { … } }.

Gated by the admin key (ANYRAY_ADMIN_TOKEN) as Authorization: Bearer <admin_token>, not the ANYRAY_OPTIMIZER_TOKEN that gates /v1/optimize. Everything returned is settings data: strategy ids, toggles, params, counts, rates.

Client-session strategies use the optional config.clientStrategies map, with boolean symbol_context and read_only_workflows entries. Both default off. The optional capabilities.clientStrategies catalog supplies their cards and runtime (connect or app). They are outside the request-transform registry and optimizer holdouts. Rules may carry the same map under overrides.rules[].clientStrategies; verified user/team matchers apply before a session starts. A matching false wins. Missing entries stay absent on readback.

The gateway calls post /admin/optimizer/client-strategies/resolve with { metadata: { user, team } } and optional ?tenant= under the admin credential. It returns { revision, strategies }. Use the key-gated client route from clients and SDK apps.

GET /admin/optimizer/memo-stats (same admin key) gives the command_digest memo's hits, misses, evicted and oversize for each of its three caches, plus computes, since the replica started or was last purged. The gateway does not proxy it; the support bundle includes it.

{
  "config": {
    "strategies": [ ... ],
    "guard":   { "enabled": true },
    "holdout": { "enabled": false, "fraction": 0.05, "windowSeconds": 604800 },
    "sessionGate": { "evaluatorEnabled": true },
    // Both default false. true holds edits inside the client's cached prefix, and
    // mask/filter/graph edits before a retained thinking block, respectively.
    "protectClientManagedCache": false,
    "protectThinkingBinding": false,
    // The default. Which strategies price a cache-expired turn's rewrite at zero,
    // and whether they ask for the avoided-write credit. Also allowed: code_graph,
    // command_digest; [] turns the cold price off. A rule's coldTurn replaces
    // these fields for the requests it matches.
    "coldTurn": { "premiumKinds": ["observation_mask", "relevance_filter"], "writeCredit": false },
    // The default. How every edit is priced before it is made; false restores
    // each part's older behavior. A rule's economics replaces these fields.
    "economics": { "tokenizer": true, "unifiedRuler": false, "realTurnHorizon": true, "measuredRetrieval": true,
                   "priceAllLanes": true, "enforceOpenLanes": false, "tailAdmission": true,
                   "quartileGrowth": true, "markedHeaderRead": true, "subsetSelection": true, "freshTailGroups": true,
                   "requestPlanning": true, "horizonSlopes": false }
  },
  // GET also returns live telemetry blocks:
  "guard":   { "strategies": [ { "kind": "observation_mask", "state": "open",
                                 "regretRate": 0.3065 } ] },
  "census":  { "requests": 48211, "toolOutputTokens": 88214003, "classes": [ ... ] },
  "cacheLint": { "regions": { "system": { "changedTurns": 221 } },
                 "prefix": { "comparedTurns": 4102, "divergedTurns": 388,
                             "byKind": { "tool_def": 41, "system": 221, "message": 126 },
                             "systemBlocks": { "first": 214, "later": 7 },
                             "toolChanges": { "order": 37, "description": 4, "set": 0 } } },
  "holdout": { "arms": { "holdout": { "requests": 0 }, "treated": { "requests": 0 } } },
  "contextRecall": {
    "observed": true,
    "sinceMs": 1788307200000,
    "listEmbeddings":      { "lookups": 241, "truncated": 3, "maxCandidatesSeen": 2001,
                             "lastTruncatedAtMs": 1788480000000 },
    "listChunkEmbeddings": { "lookups": 241, "truncated": 9, "maxCandidatesSeen": 2001,
                             "lastTruncatedAtMs": 1788566400000 }
  },
  "semanticCacheShadow": { "lookups": 4210, "hits": 63, "strictPrecision": 0.9828,
                           "durable": true }
}

The live telemetry is per-replica, not a fleet total

Every live counter here (guard, census, cacheLint, holdout, contextRecall) is the answering replica's in-memory state. Run more than one optimizer task and each read is served by whichever one the load balancer picked, so the counts are a fraction of fleet traffic and they jump between reads. Treat them as a liveness signal for that task, never as a measurement.

This matters most for holdout, which carries "scope": "replica" and an authoritativeSource field for exactly this reason: its arm balance is unreadable here, because a control arm can be busy on a replica your read never reached and still show 0. For any conclusion about the experiment use the gateway's GET /admin/spend/quality-parity, which folds durable per-request attribution.cohort across the whole fleet.

cacheLint.prefix counts turns that dropped part of the previous turn's cached prefix, by the first segment that changed (a tool definition, a system block, or a message). systemBlocks splits the system count: first is the first system block, which Claude Code rewrites on every request, and later is any other block, a real prompt change. A system prompt sent as one string counts as first. toolChanges says what changed in the tools: order means the same tools arrived in a new sequence, description means only descriptions changed, and set means tools were added, removed or changed schema.

census.mcpServers splits MCP tool output by server, largest first (at most 32 servers, the rest under (other)). Only this read answers it: the gateway heartbeat and support bundle drop it.

The gateway relays Codex seat turns without calling /v1/optimize, so after each one it posts the turn's fresh tool output to POST /v1/census with "source": "codex_seat". That route runs only the census and answers 204. It takes only assistant and tool messages, answers 400 to any other shape and 413 to a body over 1 MiB. census.seatRequests counts those samples, and they are included in census.requests and census.requestsWithToolOutput.

Which counters roll over, and reading contextRecall

guard, census, cacheLint, and holdout roll over a window. contextRecall accumulates from process start and never decays.

semanticCacheShadow is the exception to per-replica scope. When the optimizer can reach the spend database, every replica flushes its counters there every 30 seconds, and the read returns the fleet total plus the answering replica's unflushed remainder, with "durable": true. Without that store it falls back to the per-replica window and says "durable": false.

contextRecall splits the head-vector and chunk-vector readers. Each caps candidates at RECALL_MAX_CANDIDATE_VECTORS (2000 today) and reads one row past the cap to detect a miss, so maxCandidatesSeen tops out at the cap plus one. Hitting that ceiling means the reader dropped at least one live candidate before relevance scoring, and truncated counts how often.

Read onset from lastTruncatedAtMs, null until that reader first truncates. A lifetime ratio cannot tell you the same thing: on a long-lived pod a rate that is 100% right now still dilutes toward zero. {"observed": false} is its own answer, not counters at zero: this replica served no recall for the tenant, or the tenant fell out of the counter LRU (1000 tenants).

Reachable two ways

With an Anyray gateway in front, the optimizer is in-network only, so the gateway proxies this at the same path on :8787, gated by its admin RBAC (config:read / optimizer:write / optimizer:purge). That proxy is not on the fail-open path: if the optimizer is unreachable it returns 502, or 503 when ANYRAY_OPTIMIZER_URL is unset. A BYO deployment can call the optimizer's :8088 endpoint directly. A purge clears the replica that takes it at once, and its receipt counts that replica only. peers.status: "pending" covers replicas with the shared store: they clear within 30 seconds, or 2 minutes if reads fail. A replica awaiting the gateway's database push clears on its first read after the push; one never given a database is outside the receipt's coverage. not_signalled means no other replica was told.

POST /admin/optimizer/bootstrap-durable

Admin-gated, in-network. The gateway calls this at boot, and on a heartbeat, to hand the optimizer the durable config it already holds: the Postgres URL (ANYRAY_SPEND_DB_URL) and, when it has one, the content key (ANYRAY_CONTENT_KEY). That turns on the durable tier behind /v1/retrieve and /v1/recall with no per-service config, and is what makes handles survive a restart on an image-only deployment.

Request body

{ "spendDbUrl": "postgres://…", "contentKey": "<64-hex>",
  "contentMode": "encrypted", "allowPlaintext": "false" }
// → { "durable": true|false }

• The two tiers travel independently: the shared runtime-config store needs the URL alone. The durable stash encrypts under contentKey when one is pushed, else under a key the optimizer derives from its own ANYRAY_ADMIN_TOKEN, so a deployment with no content key still gets a cross-replica stash. Only a configured mode of off stops new stash rows.
• The gateway sends nothing unless it has an ANYRAY_ADMIN_TOKEN. The body carries a database credential, so it is never audit-logged.
• contentMode is the mode the admin configured, not the key-folded effective one: the optimizer folds it itself for its trace paths, and the stash gates on the configured value. contentKey and allowPlaintext are the only fields that can resolve a mode upward. A keyless push clamps the gate to "false", so the optimizer's trace store stays at off. allowPlaintext is always an explicit "true" or "false": a missing field would leave a previous "true" in place through every later quiet push.
• Adoption is checked, not assumed. The URL must be a postgres:// or postgresql:// string, and the optimizer reads the store once before committing to it. A URL it cannot use is unwound, and the next heartbeat retries, so a database that comes back heals with no operator action.
• While that check is in flight, and for as long as a pushed store stays unusable, PUT /admin/optimizer/settings answers 503 with Retry-After: 5 and persists nothing. Optimizing traffic is unaffected: /v1/optimize keeps serving the config already in memory.
• Set-once: config already in the optimizer's own environment wins. contentMode is reconciled every heartbeat from the shared settings row (console Privacy page), so turning content capture off at runtime stops the durable tier on the next push. With no gateway in the loop (BYO /v1/record, attach mode) nothing pushes, and the optimizer's own ANYRAY_CONTENT_MODE governs.

Multi-tenant auth

With ANYRAY_MULTI_TENANT=true, each tenant gets a dedicated bearer registered in ANYRAY_TENANT_TOKENS (a JSON map of token to tenantId). A registered token pins the tenant and ignores metadata.tenantId; the shared ANYRAY_OPTIMIZER_TOKEN falls back to metadata.tenantId. That is the trusted in-network lane, so never expose it on an internet-facing listener: a holder can claim any tenant. Single-tenant deployments use the ANYRAY_OPTIMIZER_TOKEN path unchanged.

• Cross-tenant isolation. A handle stashed under tenant A returns 404 to tenant B, whatever metadata.tenantId claims, and /v1/recall ranks only the resolved tenant's own stash.
• Per-tenant settings. GET|PUT /admin/optimizer/settings?tenant=<id> reads and writes one tenant's config. Omitting ?tenant defaults to 'default'.
• Rate limits. Per-tenant concurrency (ANYRAY_OPTIMIZER_CONCURRENCY_LIMIT, default 50) and RPS (ANYRAY_OPTIMIZER_RPS_LIMIT, default 1000). Exceeding either returns 429.
• TLS. The optimizer binds HTTP only (:8088). Front /v1/* with a TLS-terminating proxy before any internet exposure.