Optimizer protocol

The HTTP contract between the gateway (or your adapter) and the optimizer.

Endpoint	Gateway hook	Purpose
`POST /v1/optimize`	pre-call	Transform the request; may return a cache hit.
`POST /v1/optimize-response`	post-call	Transform the response.
`POST /v1/cache`	post-call	Write a live response back for `semantic_cache`.
`POST /v1/retrieve`	on demand	Fetch original content a reversible strategy stashed, by handle.
`POST /v1/recall`	on demand	Semantic search over the durable stash: find originals by meaning.
`POST /v1/record`	post-call	BYO only, opt-in. Persist the call when no Anyray gateway is in the path.

An adapter wires whichever hooks its gateway supports. Every endpoint rejects a body over ANYRAY_OPTIMIZER_MAX_BODY_BYTES (default 32 MiB) with 413.

The optimizer also serves its own machine-readable spec: GET /openapi.json (OpenAPI 3.1, all routes + their auth) and GET /docs (Swagger UI). Both expose only the API shape, never secrets or content. When prose and spec disagree, the route handlers win.

Request signing

Optional; off unless configured. The gateway holds an Ed25519 private key and the optimizer only the public half, so a workload that can read the optimizer's environment (shared bearer token included) still cannot produce a valid request.

When the optimizer has a verify key, POST /v1/optimize, /v1/optimize-response, /v1/cache and /v1/recall require an x-anyray-signature header and answer 401 without a valid one. /v1/retrieve and /v1/record are never gated, for the same reasons entitlement never gates them.

The header is <timestamp>.<signature>:

part	form
`timestamp`	Unix ms, canonical decimal: plain digits, no leading zero
`signature`	Ed25519 over the payload below, unpadded base64url (86 chars)

The signed payload is <timestamp>.<METHOD>.<path>.<sha256-hex of the raw body>: the body hash, never the body, so a signature never becomes a second copy of prompt content, and method and path bind a signature to its route. Both halves must be spelled canonically; the optimizer re-encodes the signature and compares, since base64url has several spellings that decode to the same bytes. Signatures are single-use within a 60-second window: outside it they are stale, and a repeat inside it is refused as a replay.

Enable the gateway first. The optimizer requires a signature the moment it has a verify key, so a deployment that configures the optimizer ahead of the gateway refuses every optimize call. The gateway fails open on that 401 rather than erroring, so nothing looks broken; the deployment just stops optimizing. GET /health reports requestSignatures: "enforced" once it is live.

Entitlement

The bearer token proves a caller is inside the deployment, not that the subscription is current, so the optimizer checks that separately: it reads the Ed25519-signed entitlement lease the gateway stores in the shared database and verifies it against the key pinned in its own image. The check is a memoized read refreshed on a timer, never a database call on the request path.

Three lease states stop the four value-delivering endpoints (POST /v1/optimize, /v1/optimize-response, /v1/cache and /v1/recall):

Lease state	Response	What it means
`optimizerEnabled: false`	`402`	Optimization is paused, typically a trial that ended with no subscription. The lease itself stays `active`, so this is the state that matters most in practice.
`status: "suspended"`	`403`	The subscription was suspended.
Expired, or the signature does not verify	`402`	No current, vouched-for entitlement.

Two endpoints are deliberately never gated. POST /v1/retrieve returns the original of content a reversible strategy already replaced with a placeholder; refusing it would strand live sessions with unresolvable handles. POST /v1/record is the BYO lane, which makes no entitlement claims and is the path that writes spend rows.

A deployment with no lease at all serves normally (attach mode, or an in-network stack running without an Anyray gateway, where no lease is ever written and none is expected). A lease that exists and has expired is refused, including one that expired before the optimizer last started: restarting the service never clears the gate.

Seeing who is calling

GET /health reports a callers block: how many POST /v1/optimize calls since start carried a gateway-stamped session id, and how many did not.

{ "gateway": 18432, "foreign": 0, "firstForeignAt": null, "lastForeignAt": null }

A non-zero foreign means something other than an Anyray gateway is driving this optimizer. Legitimate for attach mode and the BYO adapters; when a gateway is supposed to be the only caller, those calls are optimized but their savings reach no spend row, because the gateway is what writes them. The field is advisory, not a security control: anyone who can reach the endpoint with a valid token can populate it, and restricting who can reach the endpoint is the network's job.

Is the optimization being reported?

GET /health also reports a reporting block: the optimize calls this service served over a rolling window, against the spend rows that landed in the same window.

{ "state": "reported", "calls": 18402, "rows": 16110, "foreignCalls": 0, "checkedAt": "…" }

reported is healthy. underreported means optimization is going out faster than it is being recorded, the shape produced by driving this optimizer around the gateway, where the savings reach no invoice. insufficient_data, unknown, no_store and not_optimizing are honest non-answers: too little traffic to judge, the spend store could not be read, there is no shared store, or optimization is currently being refused. The check never refuses a request, and it is not an authentication check: it measures whether usage is reported, which a caller cannot forge without doing the reporting.

GET /health also reports retrievalStore, a content-free durable-write state:

State	Meaning
`ready`	Fresh retrieval spans can be durably committed.
`cooldown`	A bounded write or connection failed. New handles stay disabled until the existing five-minute retry circuit closes automatically.
`unconfigured`	No durable context store is configured. New handles stay disabled.
`content_disabled`	Content mode or encryption does not permit durable context storage. New handles stay disabled.

This state controls fresh handle creation, not optimizer liveness. The gateway still fails open and handle-free strategies continue to run.

`POST /v1/optimize`

POST /v1/optimize

Pre-call. Runs the configured pipeline and returns a transformed request (unchanged if nothing fired) plus decisions.

endpointstringrequired

Logical route (e.g. /v1/chat/completions). Selects the per-endpoint config.

requestobjectrequired

The incoming OpenAI-compatible body (model, messages, …) to transform.

metadataobject

Attribution (user, team), never content. The gateway may also add optional signals: modelContextTokens (the model's real context window), clientCanRetrieve (the caller can restore shortened content in place), and billing/tenant scope. An exact anyray_retrieve declaration or a hosted MCP connector aimed at the gateway's own /mcp proves retrieval is callable on that turn and renews the authenticated lease. Valid MCP activity, model retrieval calls, and the active Connect MCP server's heartbeat also renew it. Claude Code's measured parent signature (callable ToolSearch plus its exact anonymous DeferredToolPlaceholder) may use the fresh lease when deferral hides the declaration on a later turn. Generic MCP tools, generic defer_loading, either half of that pair, and restricted Read/Bash agent shapes fail closed. Whatever these say, a request that declares no tools, or sets tool_choice to none, normally runs the no-retrieve lane and withholds decision-pin replay: the model cannot call the retrieval tool on that turn, so no retrieval handle is emitted or replayed. The gateway can identify a Responses agent continuation separately through the trusted agentContinuation capability described below. Other tool-less requests retain the compaction and summarization stand-down behavior.

enabledKindsstring[]

Allow-list that further narrows the pipeline to these strategy kinds.

capabilitiesobject

Gateway-owned controls: canShortCircuit, retrieveEvidence, terminalTrim, allowIrreversibleWindowFit, providerPromptCache, providerPromptCacheKnown, semanticRerank, singleOutput, agentContinuation, mintEvidence, hookBudgetMs, and clientContextBudgetTokens. These live outside honor-system metadata so callers cannot grant themselves an irreversible strategy or misclassify their cache behavior.

hookBudgetMs is how long the caller will still wait for this hook, measured from the moment it sent the request. The optimizer sizes two things against it: the durable-stash write allowance (three quarters) and the point at which it starts standing strategies down (half). Past that second mark a strategy is dropped with reason budget_exhausted only when dropping it cannot change bytes the session already sent, which means the read-only kinds, semantic_cache, and any kind whose settled edits the stand-down replay can reproduce. A deterministic rewriter with nothing to replay it from keeps running, because its stand-down would put the client's original bytes back into an already-cached prefix. Settled pins still replay and the prefix cache anchor is still placed either way, so a slow turn degrades to fewer optimizations instead of timing out and forwarding the client's original bytes. Omit it and the optimizer falls back to a fixed 200 ms write allowance and no admission deadline.
singleOutput marks the connect hook's synthetic [intent, tool output] request: relevance_filter stands its recent-tool and fresh-input spares down. terminalTrim implies it on the terminal branch; singleOutput also covers the reversible anyray_read branch, which keeps retrieval handles.
agentContinuation marks a Responses transcript in which the gateway verified a trailing typed function_call_output input item and no callable tool catalog. The catalog test is over the RESOLVED request the optimizer receives, not the client's raw body: Codex 0.146 and later omit top-level tools and carry the catalog in an additional_tools input item, so a gateway reading the raw field alone asks for a lane the optimizer refuses, and the refusal holds the session. The capability permits byte-stable decision-pin replay and settled-history optimization but does not grant retrieval: degradable strategies remain non-reversible, emit no retrieval handles, and preserve the fresh function result.
providerPromptCache marks an upstream host that implicitly caches prompts regardless of model id (for example, GitHub Copilot); the normal cache-safety suppression/replay guard then applies. providerPromptCacheKnown says the gateway resolved the route and the value above is an answer rather than a default. With it, the optimizer trusts that answer in both directions: true protects a model id it would not recognize, and false releases cache-busting on a host that provably does not auto-cache. Without it, the optimizer falls back to classifying by model id.
providerExplicitCacheControl answers the opposite question: does the resolved upstream honour the client's own cache_control breakpoints? Some routes accept an Anthropic Messages body and translate it to OpenAI chat, which has no equivalent field, so the markers are dropped before the request leaves the gateway. providerExplicitCacheControlKnown marks the value as a resolved answer, with the same contract as providerPromptCacheKnown: without it the optimizer classifies by model id. It matters because the model id cannot see this — a claude- model on a translating route keeps its id while losing its markers — and a marker the upstream never received is not a cache boundary.
mintEvidence is { provider, cacheMode: "explicit" | "implicit", retrievalToolsStable }. The gateway attests a direct upstream and whether retrieval tools are already stable in the prefix. Unknown, default, custom-host, or mixed routing does not supply this evidence. A first lazy-tool activation sets retrievalToolsStable: false. Automatic cached mints require known pricing and current-turn estimated input-cost payback; missing evidence declines them. Old gateways may omit this capability safely. Never promote client metadata into it.
clientContextBudgetTokens is a lower client-local context ceiling for callers whose transcript compactor runs before the provider's model window; window_budget uses the lower ceiling. For Codex, the gateway derives it only from the user's configured model_auto_compact_token_limit or model_context_window, with a 5% safety margin. No user setting means no client ceiling. Automatic model budgeting separately subtracts requested output and estimator uncertainty from metadata.modelContextTokens. Client-supplied modelInputErrorTokens is ignored.
Irreversible window fitting is reserved for one-shot Batch API lines. (persistentTranscriptPolicyActive was accepted here through v1.10.120; newer optimizers drop it unread.)

Clients that cannot retrieve

A caller with no retrieval path is handled by billing lane. On a subscription seat the degradable eliders still run, forced to reversible:false: they trim in place and emit no retrieve ctx_… handle the caller could not resolve, which keeps savings for seats that would otherwise get none. On an api-key lane they are skipped: a trimmed byte there is billed money, and an agent handed a placeholder it cannot expand re-reads the input and spends more than the trim saved.

config.requireReversibleElision (PUT /admin/optimizer/settings) overrides the lane rule: true skips on every lane, an explicit false restores the in-place trim on api-key lanes. Unset, it follows the lane. It gates only the no-retrieve degrade, never a run that can stash.

Prefer wiring retrieval, which recovers the savings and the bytes. The gateway serves the tools itself at POST /mcp (anyray_retrieve, anyray_recall), authenticated with a client key as either x-anyray-api-key or Authorization: Bearer.

Activate agent continuations after the rolling update

agentContinuation is cache-affecting and defaults off under config.rollouts.agentContinuationV1ActivateAt. Roll every gateway and optimizer replica to a version whose features.agentContinuationV1 is true. Then set the field once through PUT /admin/optimizer/settings to an RFC 3339 instant with an explicit Z or numeric offset, at least 60 seconds in the future. The API normalizes it to UTC before persistence. Optimizer replicas refresh shared config every 15 seconds, so a current or past activation time can briefly alternate original and optimized prompt prefixes. The admin API rejects a first activation less than 60 seconds ahead and makes the timestamp immutable after it is set. Keep the field absent until the fleet is compatible. Before activation, continuation requests fail open and enter the normal session cooldown. Those existing sessions begin optimizing after their cooldown expires.

pinsobject[]

The session's current decision pins. Sending this array (even empty) opts the session into pinning; omit it for pre-pinning behavior.

// request
{
  "endpoint": "/v1/chat/completions",
  "request":  { "model": "...", "messages": [ ... ] },
  "metadata": { "user": "u1", "team": "t1" },   // OPTIONAL attribution
  "enabledKinds": ["prompt_compression"],       // OPTIONAL allow-list
  "capabilities": {                              // OPTIONAL, gateway-owned
    "canShortCircuit": true,
    "clientContextBudgetTokens": 950000
  },

  "pins": [ ... ]                               // OPTIONAL decision pins
}

// response
{
  "protocolVersion": 1,
  "features": { "persistentTranscriptPolicyV1": true },
  "optimizationId": "opt_000001",
  "request": { ...transformed body... },   // FORWARD THIS
  "decisions": [
    { "kind": "tool_pruning", "summary": "pruned 1/2 unused tools",
      "estimatedTokensSaved": 120, "estimatedSavingsUsd": 0 }
  ],
  "estimatedTokensSaved": 120,
  "cacheHit": false,        // true => serve cachedResponse, skip the provider
  "cachedResponse": null,   // present only when cacheHit
  "cacheEligible": true,    // true => write the live response back via /v1/cache
  "cacheKey": "oj2po6",
  "cacheTtlSeconds": 3600,
  "cacheShadowOnly": true,  // present with cacheEligible on a lane that can never serve
  "cohort": "treated",      // present only when audited holdout is enabled
  "cohortKind": "thinking_trim", // present only on a per-strategy control row
  "pins": [ ... ],          // present only when the request carried a `pins` array
  "suppressedKinds": [ { "kind": "relevance_filter", "reason": "no_retrieve" } ],
  "strategyTimingsMs": { "relevance_filter": 512, "phase_retrieval_commit": 40 }
}

All response fields beyond request and decisions are additive and optional, and protocolVersion stays 1:

features.persistentTranscriptPolicyV1. Legacy compatibility assertion: v1.10.120 gateways fail open on the original request when it is absent, so the optimizer keeps emitting it. Newer gateways ignore it.
features.agentContinuationV1. On /v1/optimize, true only when that exact response used the activated, handle-free Responses continuation path. A gateway requesting the path fails open and holds the session when the value is false or absent. On /health, true asserts image support.
cohortKind. The single strategy id a holdout response withheld, present only when the optimizer runs a per-strategy arm (holdout.kinds in its config). Absent on a whole-optimizer control, where every strategy was skipped, and that absence is what keeps the two experiments separate in the gateway's parity fold. A response never names more than one: each configured strategy draws an independent arm, and a row attributed to two would belong to neither.
cohort. "holdout" | "treated" when audited holdout is on and the request is attributable. A holdout response carries one audited_holdout decision with estimatedTokensSaved: 0. Assignment hashes the session within its user (or team). The session comes from metadata.gatewaySessionId when the gateway supplies it (its own verified per-conversation id), falling back to metadata.sessionId then metadata.session. An adapter should send its real conversation id rather than a per-install grouping label: a label that never varies puts a whole user in one arm and makes fraction sample people instead of requests. Send the same metadata bag on /v1/optimize and /v1/optimize-response, since each leg re-derives the arm and dropping the session on one of them splits a single request across both arms. gatewaySessionId is injected by the Anyray gateway and is not reproducible by your adapter, so supply your own session/sessionId on both legs.
strategyCohorts. One session-gate assignment expressed as a sparse map of strategy ids to holdout or treated. Each canonical configured kind draws two equal, disjoint slices at holdout.fraction, and a session-level selector then measures at most one of them, so a session is never a control for more than one strategy. The selected kind is labelled holdout (withheld) or treated (run); every other kind runs on its promotion state alone and carries no arm label, so it cannot contaminate another kind's evidence. A session that selects nothing labels nothing. Per-turn enabledKinds allowlists do not redraw that selection. Omitted kinds carry no evidence, and an empty map still marks this protocol. cooloff retains own-kind pin-thrash cost in a labelled session without supplying an arm. New responses do not emit excluded; older explicit values remain readable. These assignments require the gateway-derived per-conversation ID; static session/sessionId fallbacks used by the separate legacy cohort above do not qualify. Lane stand-asides are excluded turn by turn; they do not make the remaining session mixed. Foreign kinds' pins never remove a clean control stamp or start a withheld rewriter. A control draw with its own settled pins replays those pins without new mints and omits the kind. Handle replay refreshes the stash at the strategy's TTL while preserving the settled bytes. Pin-less rewriters carry a content-free treatment marker in the shared pin list and continue on later gate stand-downs only if that same kind previously ran, excluded from controls.
sessionGatePolicies. Map of strategy kinds to 64-character hexadecimal hashes of the assignment epoch plus each kind's own tenant parameters and registry flags. Only labelled participants and own-kind cooloff costs carry hashes; the map is omitted when empty. Releases and unrelated config saves preserve these hashes. An epoch change resets all evidence without changing durable gate state. The gateway validates and persists them for per-kind evidence. The older singular sessionGatePolicy field remains readable for rolling compatibility.
suppressedKinds. Names each configured strategy a gate dropped this turn and why (no_retrieve, cache_guard, prefix_stabilizer, regret_guard, pin_thrash, budget_exhausted, billed_lane, holdout_kind (legacy control), session_gate_holdout (gate control), session_gate_shadow, session_gate_off, session_gate_unavailable), so "suppressed" is distinguishable from "had nothing to do". billed_lane is the api-key stand-aside: prompt_compression and context_dedupe on every billed lane, plus code_graph and context_compression when the route caches without an honored marker; settled pins keep replaying under it. budget_exhausted means the turn passed half of the hookBudgetMs the caller sent, and this kind was one the optimizer could stand down without changing bytes the session already sent. Kinds it cannot drop safely keep running past that mark, so the reason names a skipped strategy rather than a stopped turn. The strategies that already ran are kept, settled decision pins still replay, and the prefix cache anchor is still placed, so the turn goes out optimized rather than timing out and forwarding the original bytes.
strategyTimingsMs. Wall time for each leg of the hook (id → ms), so the gateway's trace span shows what consumed the optimize budget. Keys are strategy ids for the strategies that ran, plus reserved phase_ keys for work that is not a strategy. A phase saves no tokens and never fires, so read it as a slice of the budget, not as an optimization. The optimizer reports two: phase_retrieval_commit (the durable context-stash commit) and phase_pin_replay (replaying settled decision pins). The gateway adds phase_gateway_pin_read (its own decision-pin store read) to the same map before persisting it, so a trace carries three — that key is never sent over this protocol and an adapter neither receives nor emits it.

Decision pinning

A decision pin records one settled-history rewrite so a later turn can replay it byte-for-byte instead of re-deciding it against the live turn; re-deciding is what busts the provider prompt cache on warm sessions. The gateway persists the pin list keyed by session and round-trips it through /v1/optimize; an adapter treats each pin as opaque.

Each pin carries a message index, an HMAC spanHash fingerprint (of the original bytes, tool universe, or model, never content), and a replay recipe. On each turn the optimizer replays every pin whose spanHash still matches and fresh-decides only the suffix; a mismatch (the client edited or compacted that span) discards just that pin.

A pin is keyed to the client's original bytes, which the client re-sends every turn, so its validity does not depend on the optimizer having applied it last turn. A pin therefore keeps replaying across turns where its strategy did not run at all (a cache guard stood it down, or the thrash guard below is active); leaving those spans alone would send the original bytes, which rewrites the cached prefix exactly as re-deciding would.

Field	Meaning
Request `pins`	The session's pins so far.
Response `pins`	The pin list after this turn (still-matching + newly recorded). The gateway persists it. Absent when the request sent none.

The response list is a full REPLACE, not a merge: a pin missing from it is deleted. A pin the optimizer declined to apply this turn is still returned, so an adapter must persist the array verbatim rather than filtering it against what visibly changed in the request.

Fail open: no pins on the request (first turn, older gateway, or a store read error) ⇒ normal warm-cache behavior.

Thrash guard: when a turn's discards outnumber its replays (two live conversations sharing one pin session, for example), re-minting every turn would re-write the cached prefix every turn. After two consecutive mostly-miss turns the optimizer suppresses cache-busting strategies for a few turns (suppressedKinds reason pin_thrash) and carries the guard's two counters in an opaque pin_thrash_guard marker inside the pins array; adapters need no special handling, the marker is an opaque pin like any other. A single all-miss turn (a client-side compaction) never triggers it. The session's existing pins are returned and keep replaying throughout the cooloff; dropping them would make the turn the cooloff lifts re-record the whole settled prefix at once, which is the rewrite the guard exists to avoid.

`POST /v1/optimize-response`

POST /v1/optimize-response

Post-call hook to transform the response. Only output-stage strategies act. Skip for streaming responses.

{ "endpoint": "/v1/chat/completions",
  "request":  { ...the request that was sent... },  // OPTIONAL context
  "response": { ...the provider response... },
  "metadata": { "user": "u1" } }
// → { "protocolVersion": 1, "response": { ... }, "decisions": [ ... ] }

`POST /v1/cache`

POST /v1/cache

After a successful response, write it back so the next identical request hits cache. Use the cacheKey from the optimize response. Powers semantic_cache.

{ "cacheKey": "oj2po6", "response": { ... }, "ttlSeconds": 3600,
  "metadata": { "user": "alice" } }          // the same attribution you optimized under
// → { "ok": true }
// → { "ok": true, "stored": false }   // no partition, or the key is not yours

Measurement-only write-back. Add "shadowOnly": true when the optimize response carried cacheShadowOnly, and for every streamed response (send the reassembled message, not the SSE). The optimizer fingerprints it for the semantic-cache shadow tier and never stores it as servable, so a stream can never be replayed to a later caller. A key minted on such a lane stays measurement-only even if the flag is dropped.

Send back the same metadata the matching /v1/optimize call carried: a cacheKey addresses one caller's partition, and the server refuses a key that was not minted for the scope this request's metadata resolves to. That is a consistency check, not authorization (the scope comes from metadata the caller itself supplies); authenticating the caller is request signing's job. Where signing is not configured, /v1/cache is as trustworthy as the deployment network around it.

Multi-tenant: a caller using a per-tenant bearer must also send request. The server re-derives the key under the authoritative tenant (so a forged cacheKey can't cross namespaces) and returns 400 if it's absent. The shared-token gateway lane uses cacheKey as-is.

Caller partition. Cache entries are partitioned per caller: the tenant plus, in order, the attributed user, the team, or the gateway-stamped gatewaySessionId from metadata. The userId / teamId aliases are deliberately NOT read: the gateway does not shadow them with the verified identity, so honouring them would let a caller name someone else's partition. A request none of those resolve for has no partition: nothing is stored, and the reply says stored: false (a shared fallback bucket would let every unattributed caller read back what every other one wrote). For the same reason /v1/optimize returns no cacheKey and cacheEligible: false for such a request, so an adapter following the contract never reaches this endpoint with one.

`POST /v1/retrieve`

POST /v1/retrieve

The recoverable side of any reversible strategy (context_compression, window_budget, observation_mask, output_externalize, …). When one drops content, the original is stashed under an opaque handle left in the request. Fetch it by handle:

{ "handle": "ctx_3kf9q", "metadata": { "source": "model" } }
// → { "handle": "ctx_3kf9q", "content": "...the original...",
//     "totalLines": 4000, "totalChars": 181204, "partial": false }   // 404 if unknown/expired

The route keeps a real miss distinct from an outage. An unknown or expired handle returns 404. An unavailable durable store or an unreadable encrypted row returns 503, which the gateway relays as retrieval unavailable instead of telling the model that the handle never existed.

Partial retrieval is opt-in. A bare { handle } always returns the full original, unchanged. To fetch only the needed part of a large stash, add a line range, a grep, or both (grep searches within the range):

{ "handle": "ctx_3kf9q", "startLine": 3180, "endLine": 3240 }        // verbatim 1-based inclusive slice
{ "handle": "ctx_3kf9q", "grep": "FAILED" }                          // case-insensitive LITERAL match, ±3 context lines
// → { "handle": "ctx_3kf9q", "content": "@@ lines 3178-3186 @@\n…", "partial": true,
//     "totalLines": 4000, "totalChars": 181204, "startLine": 1, "endLine": 4000,
//     "grepMatches": 27, "grepTruncated": true }

startLine/endLine are rounded and clamped server-side (a fractional or out-of-range value never errors); grep is a literal, never a regex, whitespace-significant, at most 256 chars (whitespace-only or oversized ⇒ 400). Every partial response reports the bounds actually searched (startLine/endLine) and the stash's totalLines/totalChars; grepMatches is the total match count, with only the first 20 windows rendered (grepTruncated: true announces the cut).
Version skew fails loud, never silently full. A slice request against an optimizer that predates these fields is answered by the gateway with 503 and code: "slice_unsupported", so the model is told to re-call without slice params rather than silently receiving (and pinning) the full copy it asked not to get. In the mirror case, a new connect client against an older gateway that ignores slice fields, the client detects the non-slice-aware response shape and returns the same guidance as a tool error instead of the silently-full content.

A caller qualifies for fresh reversible trimming when the current turn directly declares retrieval, or when a deferred catalog has authenticated retrieval activity less than 6 hours old. The Connect stdio server renews that lease every 5 minutes while its initialized transport is alive; valid calls through the stdio or remote MCP path renew it too. The gateway records an evidence class (direct, deferral_latched, unlatched_deferred, no_discovery, toolless, or forced_handler) in the spend row's existing retrieval JSONB and exposes the split from /admin/spend/dashboard.

Tiers and commit rule. The per-process in-memory tier is a bounded read-through cache (entries expire after the strategy's ttlSeconds, 4 hours by default, and are never logged). Strategies stage new handles without performing I/O; after they finish, the optimizer persists every required row in one request-scoped batch under the shared write deadline. A fresh ctx_… handle is emitted only if its row committed. If a required row fails, that strategy's transform and every later transform that may have observed it are rolled back before cache finalization; earlier independent transforms may survive. Durable rows are AES-256-GCM blobs, gated by the content mode (never when off). A settled decision pin still replays byte-identically for provider-cache safety, while refreshing its existing handle through a bounded best-effort queue off the request path. An outage can therefore make an older pinned handle unavailable, but it cannot mint a new unusable handle or rewrite the settled prefix.
ageBucket. A successful retrieve also returns a coarse, content-free bucket for how old the marker was (lt_1m, lt_5m, lt_30m, lt_2h, lt_4h, lt_12h, lt_24h, lt_48h, gte_48h), absent when the age is not known. The gateway records it on the retrieval event row and the console reports the distribution as retrieval.ageByBucket; it is never returned to the developer's client.
Sliding expiry. A successful retrieve extends the handle's life by a further 30 minutes, in both tiers, bounded by an absolute 48-hour read ceiling. Re-storing the same content starts a fresh window, so that ceiling bounds how far retrievals carry an original, not how long recurring content stays stored. A retrieval is treated as proof the handle is live context, so a marker the model keeps using stays resolvable; one that is never read still expires on its original schedule, and no amount of re-reading carries an original past the ceiling.
Guard signal. Model-requested retrieves feed the task-outcome regression guard. Automated callers must set metadata.source ("splice", "console", …) so they're excluded; a mechanical fetch isn't evidence the model needed the content back.

How a client reaches it

The optimizer is in-network only. The gateway proxies retrieval on its client surface at post /connect/retrieve (port :8787, client-key gated), and anyray-connect exposes it to the model as the MCP tool anyray_retrieve(handle), with the same optional startLine/endLine/grep params. So when a strategy leaves a · retrieve ctx_… marker, the model passes that handle to anyray_retrieve and gets the original back. The trim is lossless, and for a large stash the model can pull back just the slice it needs (a partial result opens with an [anyray: partial — …] marker line saying how to get the rest). The tool stays out of context until used.

`POST /v1/recall`

POST /v1/recall

The semantic counterpart to /v1/retrieve: find a stashed original by meaning when its · retrieve ctx_… marker has scrolled out of context. Pass a natural-language query; the optimizer ranks it against the durable stash's encrypted embeddings and returns ranked handles plus a preview. The model then calls /v1/retrieve on the chosen handle.

{ "query": "the failing pytest run", "k": 5 } // k optional (default 5, capped at 20)
// → { "matches": [
//      { "handle": "ctx_3kf9q", "score": 0.91, "preview": "pytest: 2 failed, 28 passed…",
//        "startLine": 3180, "endLine": 3240 }   // range present on chunk-indexed matches
//    ] }

Long stashes are indexed in chunks, so a match can point at the exact line range where the meaning lives, and the model then passes that range straight to /v1/retrieve (startLine/endLine) instead of re-fetching the whole output. Matches from older, head-only index rows omit the range.

Durable-only. Recall reads only the durable tier (embeddings stored at stash time). Content mode off stops new rows being stashed but does not hide rows stashed beforehand, so recall keeps answering from what is already there. It fails open: empty matches with no embedder, no durable backend, or nothing stashed, never an error.
User-scoped. Recall ranks only the requesting user's own stash, so one user can never discover a co-worker's externalized output, even within one tenant. As strong as the user attribution itself.

Like retrieve, recall is in-network only; the gateway proxies it at post /connect/recall, exposed to the model as anyray_recall(query).

`POST /v1/record` (BYO only, opt-in)

In a BYO-gateway deployment (your gateway, no Anyray gateway in the path), the optimizer becomes the trace writer: the adapter POSTs one normalized record per call, and the optimizer writes the same trace the Anyray gateway would, so the console looks identical either way.

{
  "id": "litellm-call-uuid",        // required → trace id (idempotent: a retry upserts)
  "ts": 1750000000000,              // epoch ms or ISO; defaults to now
  "endpoint": "/v1/chat/completions", "method": "POST", "status": 200,
  "model": "gpt-4o", "provider": "openai",
  "promptTokens": 10, "completionTokens": 5, "totalTokens": 15, "durationMs": 1200,
  "attribution": { "user": "u1", "team": "t1", "sessionId": "s1" },
  "decisions": [ { "kind": "prompt_compression", "estimatedTokensSaved": 100 } ],
  "content": { "input": ..., "output": ... }   // RAW, gated/encrypted server-side
}
// → { "ok": true }   ·   404 when recording is disabled (the default)

Disabled by default. 404 unless the optimizer starts with a trace-store DB configured (ANYRAY_OBSERVABILITY_DB_URL, falling back to ANYRAY_SPEND_DB_URL). Set it only in a BYO deployment: with an Anyray gateway in front, the gateway is the sole writer, so this would double-record. content is gated by the deployment's content mode.
Spend write. With ANYRAY_SPEND_DB_URL set, /v1/record also appends a SpendRecord (model, provider, tokens, cost, latency, attribution), so BYO-adapter savings show up in spend and ROI reports like native-path savings.

`GET|PUT /admin/optimizer/settings` (admin)

The strategy registry is runtime-mutable: toggles, params, and per-endpoint overrides for every strategy, hot-reloaded and audit-logged.

get returns the current config plus live guard / census / cache-lint / holdout / context-recall / semantic-cache-shadow telemetry.
put replaces the config (accepts the bare object or { "config": { … } }).

Gated by the admin key (ANYRAY_ADMIN_TOKEN) as Authorization: Bearer <admin_token>, not the ANYRAY_OPTIMIZER_TOKEN that gates /v1/optimize. Everything returned is settings data (strategy ids, toggles, params, counts, rates).

The live telemetry is per-replica, not a fleet total

Every live counter in this response (guard, census, cacheLint, holdout, contextRecall) is the answering replica's in-memory state. Run more than one optimizer task and each read is served by whichever one the load balancer picked, so the counts are a fraction of fleet traffic and they jump between reads. Treat them as a liveness signal for that task, never as a measurement. guard, census, cacheLint and holdout roll over a window. contextRecall does not: it accumulates from process start.

semanticCacheShadow is the exception. When the optimizer can reach the spend database (ANYRAY_SPEND_DB_URL, the same store its runtime config uses) every replica flushes its counters there every 30 seconds, and the read returns the fleet total since the first flushed day plus the answering replica's unflushed remainder, with "durable": true. Without that store it falls back to the per-replica window and says "durable": false.

This matters most for holdout, which carries "scope": "replica" and an authoritativeSource field for exactly this reason: its arm balance is unreadable here, because a control arm can be busy on a replica your read never reached and still show 0. For any conclusion about the experiment use the gateway's GET /admin/spend/quality-parity, which folds durable per-request attribution.cohort across the whole fleet and reports error rates, token/cost spread, and a parity verdict.

contextRecall splits the legacy head-vector and chunk-vector readers. Each one caps candidates at RECALL_MAX_CANDIDATE_VECTORS (2000 today) and reads one row past the cap to detect a miss, so maxCandidatesSeen tops out at the cap plus one. That ceiling means the reader dropped at least one live candidate before relevance scoring, because the cap orders by expiry rather than by relevance. truncated counts how many lookups hit it.

These counters never decay. lookups and truncated accumulate for the life of the process and maxCandidatesSeen is a high-water mark that does not come back down after a TTL sweep clears the backlog. So read onset from lastTruncatedAtMs, which is null until that reader first truncates and then carries the time it last did. A lifetime ratio cannot tell you the same thing: on a long-lived pod a rate that is 100% right now still dilutes toward zero.

{"observed": false} is its own answer, not counters at zero. It means this replica has served no recall for the tenant, or the tenant fell out of the counter LRU (1000 tenants). Ask another replica before concluding the cap is idle.

{
  "config": {
    "strategies": [ ... ],
    "guard":   { "enabled": true },
    "holdout": { "enabled": false, "fraction": 0.05, "windowSeconds": 604800 }
  },
  // GET also returns live telemetry blocks:
  "guard":   { "strategies": [ { "kind": "observation_mask", "state": "open",
                                 "regretRate": 0.3065 } ] },
  "census":  { "requests": 48211, "toolOutputTokens": 88214003, "classes": [ ... ] },
  "cacheLint": { "regions": { "system": { "changedTurns": 221 } } },
  "holdout": { "arms": { "holdout": { "requests": 0 }, "treated": { "requests": 0 } } },
  "contextRecall": {
    "observed": true,
    "sinceMs": 1788307200000,
    "listEmbeddings":      { "lookups": 241, "truncated": 3, "maxCandidatesSeen": 2001,
                             "lastTruncatedAtMs": 1788480000000 },
    "listChunkEmbeddings": { "lookups": 241, "truncated": 9, "maxCandidatesSeen": 2001,
                             "lastTruncatedAtMs": 1788566400000 }
  },
  "semanticCacheShadow": { "lookups": 4210, "hits": 63, "strictPrecision": 0.9828,
                           "durable": true }
}

Reachable two ways

With an Anyray gateway in front, the optimizer is in-network only, so the gateway proxies this at the same path on :8787, gated by its admin RBAC (config:read / optimizer:write / optimizer:purge). That proxy is not on the fail-open path: if the optimizer is unreachable it returns 502 (or 503 when ANYRAY_OPTIMIZER_URL is unset). A BYO deployment can call the optimizer's :8088 endpoint directly.

`POST /admin/optimizer/bootstrap-durable`

Admin-gated, in-network. The gateway calls this at boot (and on a heartbeat) to hand the optimizer the durable config it already holds: the Postgres URL (ANYRAY_SPEND_DB_URL) and, when it has one, the content key (ANYRAY_CONTENT_KEY). That turns on the durable tier behind /v1/retrieve and /v1/recall with no per-service config, and is what makes handles survive a restart on an image-only deployment.

Request body
{ "spendDbUrl": "postgres://…", "contentKey": "<64-hex>",
  "contentMode": "encrypted", "allowPlaintext": "false" }
// → { "durable": true|false }

The push carries the two tiers independently: the shared runtime-config store needs the URL alone, while the durable stash also needs a content key. A deployment running content mode off holds no key and still gets a shared config store.
The gateway sends nothing unless it has an ANYRAY_ADMIN_TOKEN to authenticate with. The body carries a database credential, so it is never audit-logged either.
contentKey and allowPlaintext are the only fields that can resolve a mode upward (to encrypted and to plaintext); the URL is also the trace store's fallback target, and a push that supplies it turns /v1/record from disabled into live, so a keyless push sends no key and clamps the gate to "false", leaving the optimizer's effective mode at off. contentMode always travels, since on its own it can only resolve downward.
allowPlaintext is always an explicit "true" or "false", never omitted. A missing field leaves the previous value in place, so an optimizer already holding a "true" would keep it through every later push that stayed quiet; clearing the gate has to be said out loud.
A push that supplies a URL the optimizer did not already have also brings up the shared runtime-config store (migration 0058) on a process that booted without one, so admin strategy edits stop being per pod. A repeat push carrying the same URL changes nothing.
Adoption is checked, not assumed. The URL must be a postgres:// or postgresql:// connection string, and the optimizer reads the store once before it commits to it. A URL it cannot use (wrong scheme, or a database reachable from the gateway but not from the optimizer) is unwound, leaving the service exactly as the push found it; the next heartbeat retries, so a database that comes back heals with no operator action.
While that check is in flight, and for as long as a pushed store stays unusable, PUT /admin/optimizer/settings answers 503 with Retry-After: 5 and persists nothing. Once a deployment has a shared config store, a settings save either reaches it or is refused: writing to the pod's local file instead would report success for a change no other replica has, and the next refresh tick (15s) would overwrite it. Optimizing traffic is unaffected throughout; /v1/optimize keeps serving the config already in memory.
Set-once: config already in the optimizer's own environment wins. contentMode is reconciled every heartbeat: the gateway reads it from the shared settings row (console Privacy page), so turning content capture off at runtime stops the durable tier on the next push. On paths with no gateway in the loop (BYO /v1/record, attach mode) nothing pushes, and the optimizer's own ANYRAY_CONTENT_MODE governs instead. No new environment variables.

Multi-tenant auth

With ANYRAY_MULTI_TENANT=true, each tenant gets a dedicated bearer registered in ANYRAY_TENANT_TOKENS (a JSON map of token → tenantId). A registered token pins the tenant and ignores metadata.tenantId; the shared ANYRAY_OPTIMIZER_TOKEN falls back to metadata.tenantId. That is the trusted in-network lane, so never expose it on an internet-facing listener (a holder can claim any tenant). Single-tenant deployments use the ANYRAY_OPTIMIZER_TOKEN path unchanged.

Cross-tenant isolation. A handle stashed under tenant A returns 404 to tenant B, whatever metadata.tenantId claims; /v1/recall ranks only the resolved tenant's own stash.
Per-tenant settings. GET|PUT /admin/optimizer/settings?tenant=<id> reads/writes one tenant's config; omitting ?tenant defaults to 'default'.
Rate limits. Per-tenant concurrency (ANYRAY_OPTIMIZER_CONCURRENCY_LIMIT, default 50) and RPS (ANYRAY_OPTIMIZER_RPS_LIMIT, default 1000) caps; exceeding either returns 429.
TLS. The optimizer binds HTTP only (:8088); front /v1/* with a TLS-terminating proxy before any internet exposure.

Request signing​

Entitlement​

Seeing who is calling​

Is the optimization being reported?​

POST /v1/optimize​

Clients that cannot retrieve​

Decision pinning​

POST /v1/optimize-response​

POST /v1/cache​

POST /v1/retrieve​

POST /v1/recall​

POST /v1/record (BYO only, opt-in)​

GET|PUT /admin/optimizer/settings (admin)​

POST /admin/optimizer/bootstrap-durable​

Multi-tenant auth​