Optimizer protocol

The HTTP contract between the gateway (or your adapter) and the optimizer.

Endpoint	Gateway hook	Purpose
POST /v1/optimize	pre-call	Transform the request; may return a cache hit.
POST /v1/optimize-response	post-call	Transform the response.
POST /v1/cache	post-call	Write a live response back for semantic_cache.
POST /v1/retrieve	on demand	Fetch original content a reversible strategy stashed, by handle.
POST /v1/recall	on demand	Semantic search over the durable stash — find originals by meaning.
POST /v1/record	post-call	BYO only, opt-in — persist the call when no Anyray gateway is in the path.

An adapter wires whichever hooks its gateway supports. Every endpoint rejects a body over ANYRAY_OPTIMIZER_MAX_BODY_BYTES (default 32 MiB) with 413.

The optimizer also serves its own machine-readable spec: GET /openapi.json (OpenAPI 3.1, all routes + their auth) and GET /docs (Swagger UI). Both expose only the API shape, never secrets or content. When prose and spec disagree, the route handlers win.

POST /v1/optimize

POST /v1/optimize

Pre-call. Runs the configured pipeline and returns a transformed request (unchanged if nothing fired) plus decisions.

endpointstringrequired

Logical route (e.g. /v1/chat/completions) — selects the per-endpoint config.

requestobjectrequired

The incoming OpenAI-compatible body (model, messages, …) to transform.

metadataobject

Attribution (user, team) — never content. The gateway may also add optional signals: modelContextTokens (the model's real context window), clientCanRetrieve (the caller can restore shortened content in place), and billing/tenant scope.

enabledKindsstring[]

Allow-list that further narrows the pipeline to these strategy kinds.

capabilitiesobject

Gateway-owned controls: canShortCircuit, retrieveEvidence, terminalTrim, allowIrreversibleWindowFit, providerPromptCache, and clientContextBudgetTokens. These live outside honor-system metadata so callers cannot grant themselves an irreversible strategy or misclassify their cache behavior. providerPromptCache marks an upstream host that implicitly caches prompts regardless of model id (for example, GitHub Copilot); the normal cache-safety suppression/replay guard then applies. clientContextBudgetTokens is a lower client-local context ceiling for callers whose transcript compactor runs before the provider's model window; window_budget uses the lower ceiling. For Codex, the gateway derives it from the user's model_auto_compact_token_limit, then model_context_window, then the resolved request-model window, with a 5% safety margin. A request whose window cannot be resolved retains the 190k fallback. Irreversible window fitting is reserved for one-shot Batch API lines. (persistentTranscriptPolicyActive was accepted here through v1.10.120; newer optimizers drop it unread.)

pinsobject[]

The session's current decision pins. Sending this array (even empty) opts the session into pinning; omit it for pre-pinning behavior.

// request
{
  "endpoint": "/v1/chat/completions",
  "request":  { "model": "...", "messages": [ ... ] },
  "metadata": { "user": "u1", "team": "t1" },   // OPTIONAL attribution
  "enabledKinds": ["prompt_compression"],       // OPTIONAL allow-list
  "capabilities": {                              // OPTIONAL, gateway-owned
    "canShortCircuit": true,
    "clientContextBudgetTokens": 950000
  },

  "pins": [ ... ]                               // OPTIONAL decision pins
}

// response
{
  "protocolVersion": 1,
  "features": { "persistentTranscriptPolicyV1": true },
  "optimizationId": "opt_000001",
  "request": { ...transformed body... },   // FORWARD THIS
  "decisions": [
    { "kind": "tool_pruning", "summary": "pruned 1/2 unused tools",
      "estimatedTokensSaved": 120, "estimatedSavingsUsd": 0 }
  ],
  "estimatedTokensSaved": 120,
  "cacheHit": false,        // true => serve cachedResponse, skip the provider
  "cachedResponse": null,   // present only when cacheHit
  "cacheEligible": true,    // true => write the live response back via /v1/cache
  "cacheKey": "oj2po6",
  "cacheTtlSeconds": 3600,
  "cohort": "treated",      // present only when audited holdout is enabled
  "pins": [ ... ],          // present only when the request carried a `pins` array
  "suppressedKinds": [ { "kind": "relevance_filter", "reason": "no_retrieve" } ],
  "strategyTimingsMs": { "relevance_filter": 512, "tool_schema_compression": 9 }
}

All response fields beyond request and decisions are additive and optional — protocolVersion stays 1:

• features.persistentTranscriptPolicyV1 — legacy compatibility assertion: v1.10.120 gateways fail open on the original request when it is absent, so the optimizer keeps emitting it. Newer gateways ignore it.
• cohort — "holdout" | "treated" when audited holdout is on and the request is attributable. A holdout response carries one audited_holdout decision with estimatedTokensSaved: 0.
• suppressedKinds — names each configured strategy a gate dropped this turn and why (no_retrieve, cache_guard, prefix_stabilizer, regret_guard, pin_thrash) — so "suppressed" is distinguishable from "had nothing to do".
• strategyTimingsMs — per-strategy wall time for the strategies that ran (strategy id → ms, content-free), so the gateway's trace span shows which strategy consumed the optimize budget.

Decision pinning

A decision pin records one settled-history rewrite so a later turn can replay it byte-for-byte instead of re-deciding it against the live turn — re-deciding is what busts the provider prompt cache on warm sessions. The gateway persists the pin list keyed by session and round-trips it through /v1/optimize; an adapter treats each pin as opaque.

Each pin is metadata only — a message index, an HMAC spanHash fingerprint (of the original bytes, tool universe, or model — never content), and a replay recipe. On each turn the optimizer replays every pin whose spanHash still matches and fresh-decides only the suffix; a mismatch (the client edited or compacted that span) discards just that pin.

Field	Meaning
Request pins	The session's pins so far.
Response pins	The pin list after this turn (still-matching + newly recorded). The gateway persists it. Absent when the request sent none.

Fail open: no pins on the request (first turn, older gateway, or a store read error) ⇒ normal warm-cache behavior.

Thrash guard: when a turn's discards outnumber its replays (two live conversations sharing one pin session, for example), re-minting every turn would re-write the cached prefix every turn. After two consecutive mostly-miss turns the optimizer suppresses cache-busting strategies for a few turns (suppressedKinds reason pin_thrash) and carries the guard's two counters in a content-free pin_thrash_guard marker inside the pins array. Adapters need no special handling — the marker is an opaque pin like any other. A single all-miss turn (a client-side compaction) never triggers it.

POST /v1/optimize-response

POST /v1/optimize-response

Post-call hook to transform the response — only output-stage strategies act. Skip for streaming responses.

{ "endpoint": "/v1/chat/completions",
  "request":  { ...the request that was sent... },  // OPTIONAL context
  "response": { ...the provider response... },
  "metadata": { "user": "u1" } }
// → { "protocolVersion": 1, "response": { ... }, "decisions": [ ... ] }

POST /v1/cache

POST /v1/cache

After a successful non-streaming response, write it back so the next identical request hits cache. Use the cacheKey from the optimize response. Powers semantic_cache.

{ "cacheKey": "oj2po6", "response": { ... }, "ttlSeconds": 3600 }
// → { "ok": true }

Multi-tenant: a caller using a per-tenant bearer must also send request — the server re-derives the key under the authoritative tenant (so a forged cacheKey can't cross namespaces) and returns 400 if it's absent. The shared-token gateway lane uses cacheKey as-is.

POST /v1/retrieve

POST /v1/retrieve

The recoverable side of any reversible strategy (context_compression, window_budget, observation_mask, output_externalize, …). When one drops content, the original is stashed under an opaque handle left in the request. Fetch it by handle:

{ "handle": "ctx_3kf9q", "metadata": { "source": "model" } }
// → { "handle": "ctx_3kf9q", "content": "...the original..." }   // 404 if unknown/expired

A caller qualifies for reversible trimming only when the gateway sees the model-callable retrieval tool (or fresh key-latched evidence from an earlier declaration). Merely calling this endpoint beside a headless script does not opt that script into retrieval-dependent strategies.

• Tiers. Stashes live in a per-process in-memory tier (expires after the strategy's ttlSeconds, never logged). When the durable tier is configured, every stash is also mirrored to it fire-and-forget (bounded — a declined write leaves that row memory-only), so a handle survives an optimizer restart and resolves from another replica. output_externalize goes further: it requires a confirmed durable write before eliding, since it removes content from the wire. Durable rows are AES-256-GCM blobs, gated by ANYRAY_CONTENT_MODE (never when off).
• Guard signal. Model-requested retrieves feed the task-outcome regression guard. Automated callers must set metadata.source ("splice", "console", …) so they're excluded — a mechanical fetch isn't evidence the model needed the content back.

How a client reaches it

The optimizer is in-network only. The gateway proxies retrieval on its client surface at post /connect/retrieve (port :8787, client-key gated), and anyray-connect exposes it to the model as the MCP tool anyray_retrieve(handle). So when a strategy leaves a · retrieve ctx_… marker, the model passes that handle to anyray_retrieve and gets the original back — the trim is lossless. The tool stays out of context until used.

POST /v1/recall

POST /v1/recall

The semantic counterpart to /v1/retrieve: find a stashed original by meaning when its · retrieve ctx_… marker has scrolled out of context. Pass a natural-language query; the optimizer ranks it against the durable stash's encrypted embeddings and returns ranked handles plus a preview. The model then calls /v1/retrieve on the chosen handle.

{ "query": "the failing pytest run", "k": 5 } // k optional (default 5, capped at 20)
// → { "matches": [
//      { "handle": "ctx_3kf9q", "score": 0.91, "preview": "pytest: 2 failed, 28 passed…" }
//    ] }

• Durable-only. Recall reads only the durable tier (embeddings stored at stash time), gated by ANYRAY_CONTENT_MODE. It fails open — empty matches when off, no embedder, or no durable backend; never an error.
• User-scoped. Recall ranks only the requesting user's own stash, so one user can never discover a co-worker's externalized output — even within one tenant. As strong as the user attribution itself.

Like retrieve, recall is in-network only; the gateway proxies it at post /connect/recall, exposed to the model as anyray_recall(query).

POST /v1/record (BYO only — opt-in)

In a BYO-gateway deployment (your gateway, no Anyray gateway in the path), the optimizer becomes the trace writer: the adapter POSTs one normalized record per call, and the optimizer writes the same trace the Anyray gateway would, so the console looks identical either way.

{
  "id": "litellm-call-uuid",        // required → trace id (idempotent: a retry upserts)
  "ts": 1750000000000,              // epoch ms or ISO; defaults to now
  "endpoint": "/v1/chat/completions", "method": "POST", "status": 200,
  "model": "gpt-4o", "provider": "openai",
  "promptTokens": 10, "completionTokens": 5, "totalTokens": 15, "durationMs": 1200,
  "attribution": { "user": "u1", "team": "t1", "sessionId": "s1" },
  "decisions": [ { "kind": "prompt_compression", "estimatedTokensSaved": 100 } ],
  "content": { "input": ..., "output": ... }   // RAW — gated/encrypted server-side
}
// → { "ok": true }   ·   404 when recording is disabled (the default)

• Disabled by default — 404 unless the optimizer starts with a trace-store DB configured (ANYRAY_OBSERVABILITY_DB_URL, falling back to ANYRAY_SPEND_DB_URL). Set it only in a BYO deployment — with an Anyray gateway in front, the gateway is the sole writer, so this would double-record. content is gated by ANYRAY_CONTENT_MODE.
• Spend write. With ANYRAY_SPEND_DB_URL set, /v1/record also appends a metadata-only SpendRecord (model, provider, tokens, cost, latency, attribution — no content), so BYO-adapter savings show up in spend and ROI reports like native-path savings.

GET|PUT /admin/optimizer/settings (admin)

The strategy registry is runtime-mutable: toggles, params, and per-endpoint overrides for every strategy, hot-reloaded and audit-logged.

• get returns the current config plus live guard / census / cache-lint / holdout / semantic-cache-shadow telemetry.
• put replaces the config (accepts the bare object or { "config": { … } }).

Gated by the admin key (ANYRAY_ADMIN_TOKEN) as Authorization: Bearer <admin_token> — not the ANYRAY_OPTIMIZER_TOKEN that gates /v1/optimize. Everything returned is metadata only (strategy ids, toggles, params, counts, rates) — never handles, tool names/args, or prompt/response content.

{
  "config": {
    "strategies": [ ... ],
    "guard":   { "enabled": true },
    "holdout": { "enabled": false, "fraction": 0.05, "windowSeconds": 604800 }
  },
  // GET also returns live telemetry blocks, all metadata-only:
  "guard":   { "strategies": [ { "kind": "observation_mask", "state": "open",
                                 "regretRate": 0.3065 } ] },
  "census":  { "requests": 48211, "toolOutputTokens": 88214003, "classes": [ ... ] },
  "cacheLint": { "regions": { "system": { "changedTurns": 221 } } },
  "holdout": { "arms": { "holdout": { "requests": 0 }, "treated": { "requests": 0 } } },
  "semanticCacheShadow": { "lookups": 4210, "hits": 63, "strictPrecision": 0.9828 }
}

Reachable two ways

With an Anyray gateway in front, the optimizer is in-network only, so the gateway proxies this at the same path on :8787, gated by its admin RBAC (config:read / optimizer:write / optimizer:purge). That proxy is not on the fail-open path: if the optimizer is unreachable it returns 502 (or 503 when ANYRAY_OPTIMIZER_URL is unset). A BYO deployment can call the optimizer's :8088 endpoint directly.

POST /admin/optimizer/bootstrap-durable

Admin-gated, in-network. The gateway calls this at boot (and on a heartbeat) to hand the optimizer the durable-stash config it already holds — Postgres URL (ANYRAY_SPEND_DB_URL) and content key (ANYRAY_CONTENT_KEY) — so the durable tier behind /v1/retrieve and /v1/recall turns on with no per-service config. This is what makes handles survive a restart on an image-only deployment.

Request body

{ "spendDbUrl": "postgres://…", "contentKey": "<64-hex>",
  "contentMode": "encrypted", "allowPlaintext": "false" }
// → { "durable": true|false }

Set-once: config already in the optimizer's own environment wins. contentMode is reconciled every heartbeat, so turning content capture off at runtime stops the durable tier on the next push. The body carries secrets, so it's never audit-logged. No new environment variables.

Multi-tenant auth

With ANYRAY_MULTI_TENANT=true, each tenant gets a dedicated bearer registered in ANYRAY_TENANT_TOKENS (a JSON map of token → tenantId). A registered token pins the tenant and ignores metadata.tenantId; the shared ANYRAY_OPTIMIZER_TOKEN falls back to metadata.tenantId — the trusted in-network lane, so never expose it on an internet-facing listener (a holder can claim any tenant). Single-tenant deployments use the ANYRAY_OPTIMIZER_TOKEN path unchanged.

• Cross-tenant isolation. A handle stashed under tenant A returns 404 to tenant B, whatever metadata.tenantId claims; /v1/recall ranks only the resolved tenant's own stash.
• Per-tenant settings. GET|PUT /admin/optimizer/settings?tenant=<id> reads/writes one tenant's config; omitting ?tenant defaults to 'default'.
• Rate limits. Per-tenant concurrency (ANYRAY_OPTIMIZER_CONCURRENCY_LIMIT, default 50) and RPS (ANYRAY_OPTIMIZER_RPS_LIMIT, default 1000) caps; exceeding either returns 429.
• TLS. The optimizer binds HTTP only (:8088); front /v1/* with a TLS-terminating proxy before any internet exposure.