API reference

The gateway's inference, MCP, admin, and SCIM surfaces.

Surface	Serves	Credential
/v1/*	OpenAI-compatible inference plus the Anthropic Messages API	Client key or service key
/mcp, /mcp/org	Retrieval tools; the organization's remote connectors	Client key, or an AuthKit token on /mcp/org
/admin/*, /admin/v1/*	Configuration and reporting	Admin token, console SSO session, or aak_ key
/scim/v2/*	Inbound user and group provisioning	SCIM bearer

Your gateway serves its own explorer at GET /docs, over the OpenAPI 3.1 spec at GET /openapi.json (load that into Postman or a client generator). Both show only the API shape, but "try it out" sends a real request, so it still needs your key. Without a deployment in front of you, read the published copy at docs.anyray.ai/openapi.json.

Open the explorer

open http://localhost:8787/docs      # or https://<your-anyray-gateway>/docs

Route by route, the index is the Endpoint reference. The optimizer's own hooks are the separate optimizer protocol.

Inference POST requests accept optional optimization selection headers. Absent headers preserve configured behavior; preferences can only narrow the server's policy.

Base URL

The gateway is self-hosted. It runs on your own infrastructure, default port 8787, not on docs.anyray.ai. Put it behind TLS for any networked deployment.

Base URL

https://<your-anyray-gateway>        # e.g. http://localhost:8787 in-network

Provider keys, content mode, and the optimizer URL: Configure.

Authentication

Send your Anyray client key as a bearer token, in x-anyray-api-key, or, for Anthropic-native clients on /v1/messages, in x-api-key. The real provider key stays server-side.

Authenticated request

curl https://<your-anyray-gateway>/v1/chat/completions \
  -H "Authorization: Bearer $ANYRAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H 'x-anyray-provider: openai' \
  -d '{
    "model": "gpt-4o",
    "messages": [{ "role": "user", "content": "Summarize the release notes." }]
  }'

The six credentials

Credential	Send it on	It grants
Client key ark_…	/v1/*	The holder's user and team attribution. An admin mints it, or the developer does with anyray-connect --sso.
Service key ark_svc_…	/v1/*	A revocable non-human identity for an agent, CI job, or SDK script, with an optional monthly dollar budget. No interactive SSO.
AuthKit access token	/mcp/org	A WorkOS user in the customer organization. See Organization MCP with SSO.
Admin token ANYRAY_ADMIN_TOKEN	Console, /admin/*	Break-glass owner access, kept server-side. Each route still needs its capability.
Admin API key aak_…	/admin/*	Automation, with either a capability subset or v1 scopes. Never inference, never trace content. See Scoped admin API keys.
SCIM bearer	/scim/v2/*	One tenant's IdP provisioning. Only its SHA-256 hash is stored.

Every /v1/* request needs a minted key. The key's own user and team set attribution, and x-anyray-metadata cannot override them. OAuth never runs on /v1/* itself.

Status	code	Cause
401	missing_key, key_unknown, key_expired, key_revoked, user_deactivated	The identity is absent or rejected. The body carries a safe hint.
403	passthrough_only_key, path_key_refused	The key is valid but not accepted this way. A passthrough-only key (ark_pt_) works only in the base URL path, and no other key kind may be carried there.
503	key_store_unavailable	The gateway cannot reach the key store. Verification fails closed, so a database outage never reads as a bad credential.

Organization MCP with SSO

An MCP client that cannot attach a static header adds one URL:

https://<your-anyray-gateway>/mcp/org

The unauthenticated request returns 401 and points at the protected-resource metadata:

WWW-Authenticate: Bearer resource_metadata="https://<your-anyray-gateway>/.well-known/oauth-protected-resource"

That public metadata names WorkOS AuthKit as the authorization server. AuthKit runs the flow through the enterprise SSO connection already attached to the customer's WorkOS organization. The gateway implements no authorization codes, PKCE, client registration, consent, refresh, or revocation.

The access token must pass every check:

Claim or property	Required value
signing algorithm	RS256
iss	the managed AuthKit issuer
aud	one scalar string equal to https://<your-anyray-gateway>/mcp/org
time and subject	valid sub, iat, and exp
organization	non-empty WorkOS org_id

No mcp:org scope is advertised or required. The exact resource audience is what separates one gateway from another. A client key still works on /mcp/org and takes precedence when presented, which keeps Connect-managed tools compatible.

Require SSO in the WorkOS policy

Keep non-SSO methods disabled in the WorkOS domain policy, and require SSO in the organization policy. Otherwise a manually enabled Google or Magic Auth method can bypass the enterprise IdP.

What happens after JWT verification

The gateway sends only sub and org_id to Billing under its deployment bearer. The AuthKit token never reaches Billing or storage. Billing requires the tenant's activated SSO marker and a live WorkOS connection, fetches the current WorkOS user, requires exactly one active membership in the configured organization, resolves the current groups to an Anyray team, and applies allowed domains and revocation.

Throttles, outages, and vendor confirmation

Pre-verification and live identity throttles return 429 with Retry-After. Concurrent config, JWKS, and metadata refreshes share one outbound request. A Billing, WorkOS, or AuthKit outage fails closed with 503.

Billing also returns 503 until a vendor operator confirms the current WorkOS organization policy and the exact Resource Indicator derived from this deployment's public gateway URL. Rebinding the WorkOS organization or changing the gateway URL invalidates that confirmation.

This lane never provisions WorkOS for an ordinary tenant. The WorkOS organization is created only after an authenticated owner chooses Set up SSO, or an operator binds an existing organization. Non-SSO tenants keep using client-key MCP authentication.

Setup: Organization MCP endpoint. Every route: endpoint reference.

Service keys for agents and CI

Method	Path	Capability	Shape
post	/admin/service-keys	clientkeys:manage	{ "name": string, "team": string, "monthlyBudgetUsd"?: number } → 201 { "key": "ark_svc_…", "record": {...} }
get	/admin/service-keys	config:read	{ "keys": [...] }, service keys only, source and type set to service
delete	/admin/service-keys/:id	clientkeys:manage	{ "ok": true }

Mint a budgeted CI key

curl https://<your-anyray-gateway>/admin/service-keys \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "synthetic-ci-release",
    "team": "platform",
    "monthlyBudgetUsd": 25
  }'

Store the returned key at once. The gateway keeps only its SHA-256 hash and never returns the raw token again. A budget counts successful request cost per UTC calendar month. At or above it, covered inference routes return:

HTTP 402

{
  "error": {
    "code": "service_key_budget_exceeded",
    "message": "This service key has reached its monthly budget. Ask an administrator to raise the budget or wait for the next UTC month."
  }
}

The cap is soft and fails open. Completed requests update the shared counter after their responses, so concurrent requests can overshoot, and a counter outage lets requests through. A key with no monthlyBudgetUsd, or with 0, is unlimited.

Scoped admin API keys

An admin API key (aak_…) scripts /admin/* without the break-glass ANYRAY_ADMIN_TOKEN.

• Mint it with exactly one grant shape: a subset of the admin capabilities, or native Management API v1 scopes.
• The grant is fixed at mint time and can never exceed the minter's own access. Content, content mode, SSO config, self-update, and key management stay ungrantable.
• Minting needs adminkeys:manage (security_admin and up).
• Keys expire after 90 days. Pass expiresInDays for another horizon, or neverExpires: true.

A scope-minted key is v1-only. It holds no capabilities, so every legacy /admin/* capability gate rejects it. Use it on /admin/v1/*, plus the scope-less GET /admin/v1/me. A v1 403 names the exact scope to mint in required_scope.

Method	Path	Capability	Shape
post	/admin/api-keys	adminkeys:manage	Exactly one of caps: string[] or scopes: string[], plus optional label and expiry → 201 { "key": "aak_…", "record": {...} }
get	/admin/api-keys	config:read	{ "keys": [...] }, metadata only, hashes stripped
delete	/admin/api-keys/:id	adminkeys:manage	{ "ok": true }

Store the returned key at once. Revocation takes effect on the next request, on every replica. Admin writes made with a key are attributed in the audit log as apikey:<id>.

Mint by scope

Mint by scope

curl -X POST https://<your-anyray-gateway>/admin/v1/admin-keys \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "ci-dashboard",
    "scopes": ["spend:read"]
  }'

Use it

Use it

curl https://<your-anyray-gateway>/admin/v1/spend/dashboard \
  -H "Authorization: Bearer $ADMIN_API_KEY"

On the legacy surface a key needs a capability grant instead. A budget bot that reads config and writes one user's caps takes ["config:read", "usercaps:write"]:

Mint by capability

Mint by capability

curl https://<your-anyray-gateway>/admin/api-keys \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "budget-bot",
    "caps": ["config:read", "usercaps:write"]
  }'

Use it

Use it

curl -X PATCH https://<your-anyray-gateway>/admin/user-caps/alice \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "maxUsd": 250 }'

That PATCH merges a partial edit into one user's entry. Fields, conditional writes, and the whole-map PUT: per-user caps and budgets.

Use keys against the gateway origin directly

The console host's proxy replaces the Authorization header with the browser login cookie, so an aak_ key only works against the gateway itself (port 8787 by default), not through the console hostname.

If the key store is unreachable, aak_ verification fails closed with 503 and admin_key_store_unavailable, never 401, and the break-glass admin token keeps working. Console sessions work the same way: signing out revokes the session on every replica, and if the gateway cannot read that shared list, a session cookie gets 503 session_store_unavailable.

Management API v1

Stable /admin/v1/* is the /admin/* surface re-cut for scripts and API keys: plural-noun resources, one permission rule you read off the URL, one error envelope, and a request id on every response. Response shapes are the documented contract, and a breaking change would arrive only as /admin/v2.

For spend charts, GET /admin/v1/spend/overview?window=30d returns totals and breakdowns without latency, cache, retrieval, or strategy diagnostics. GET /admin/v1/spend/active-users?window=7d returns the complete list of attributed users with billed or saved tokens in that window.

Legacy /admin/* data routes still answer, now with Deprecation: true and a Link: <successor>; rel="successor-version" header naming their v1 path. Plan their retirement into your scripts. Requests are capped at 300 per minute per principal, then 429 with Retry-After.

Beyond the re-served routes, v1 adds:

Addition	Routes
Key lifecycle	GET/PATCH /admin/v1/keys/{id}; POST …/{id}/rotate, which returns the fresh token exactly once; POST …/{id}/secret, which returns an agent key's stored value (service keys only, sealed at rest, cache-control: no-store, audited on every read)
Provider key writes	PUT/DELETE /admin/v1/providers/{slug}/key
Audit trail	GET /admin/v1/audit/events?limit=&cursor=&category=, a { "data": [...], "next_cursor": ... } page over the admin and GDPR trail, audit:read

Developers get self-service on the inference plane instead, with no admin credential: GET /v1/me, /v1/me/usage, and /v1/me/limits answer with the verified key holder's identity, month-to-date usage, and the ceilings that would 402 them.

The scope rule

A scope is <resource>:<action>, where the resource is the first path segment after /admin/v1/:

Request	Scope it needs
GET /admin/v1/<resource>/…	<resource>:read
POST, PUT, PATCH, DELETE /admin/v1/<resource>/…	<resource>:write
POST /admin/v1/<resource>/…/query (a read whose filter needs a body)	<resource>:read

Resources: keys, admin-keys, users, spend, providers, routing, optimizer, policies, settings, sso, enrollment, provisioning, fleet, traces, audit, playground. GET /admin/v1/me needs no scope and returns the scopes the caller holds.

Each exception is declared on its operation in the OpenAPI spec as x-required-scope:

Scope	Gates	Grantable to a key
users:revoke	Revoking or reinstating a developer	no
users:gdpr	A user's export and erase	yes
optimizer:erase	Erasing one tenant's context stash at offboarding	yes
providers:write on GET /admin/v1/providers	A credential-grade read: provider keys come back in full	yes
fleet:enroll	The enroll secret, evidence-connector bearers, installer download, the MDM profile, enabling a policy	yes
fleet:remediate	Running a remediation script, retiring a host, pushing policy	yes
playground:run	The admin playground	yes
fleet:direct	Queuing a directive for a developer seat	no
settings:content	Content mode and trace retention	no
settings:update	The self-update policy	no
sso:write	IdP and SCIM configuration	no
admin-keys:write	Minting and revoking aak_ keys	no
traces:content, fleet:content	Prompt and response bodies; remediation script output	no

How existing capability grants map to scopes

A role or an aak_ key still carries capabilities, and each capability translates to the scopes it gated on the legacy paths. config:read is every <resource>:read except the audit, trace, and fleet lanes; usercaps:write is users:write; enrollment:manage is enrollment:write plus fleet:enroll; provisioning:manage is provisioning:write, fleet:write, and fleet:remediate. The legacy trust boundaries between enrolling, provisioning, and directing never collapse into one another.

Four differences are deliberate, each pinned by a test:

• The audit lane (/admin/v1/audit/*) needs audit:read, which config:read no longer implies. Auth-rejection events are not in that lane: they record what a developer's tool did rather than what an admin did, so they are GET /admin/v1/users/auth-events at users:read.
• The fleet listings, including GET …/mdm-servers, GET …/installers, and GET …/evidence-connectors, are fleet:read.
• Single-route capabilities of one tier fold into one write scope. routing:write covers routing config and aliases, policies:write covers team skills and the Claude Desktop policy, and optimizer:write covers settings and purge.
• The non-content runtime settings (seatOverflow, seatOverflowHourlyMaxRequests, anthropicCacheLookback, adaptiveMcpInstructions, servedRetrieval, servedRetrievalClasses, harnessRetrieval, optimizerWireDebit, promptCacheTtl, promptCacheKeepWarm, promptCacheKeepWarmIdleMaxMinutes, promptCacheKeepWarmSubscription, promptCacheRecordedExpiry, claudeCodeRemoteMcp, toolDefer, ccBudgetNotice, and the deprecated promptCache1h and promptCacheKeepWarmSubscriptionIdle) are settings:write. The prompt-cache fields are held off in this release: a write to any of them answers 400, and reads report client, off and false whatever is stored. promptCacheKeepWarmOffSince is read-only: the gateway stamps it when keep-warm is switched off, and three hours later a session's cache is attested cold by its lifetime alone. promptCacheSiblingWriteAheadSince is read-only too: the gateway stamps it once, when the first replica that records sub-agent requests before forwarding them starts, and two hours later a sub-agent launch in the transcript stops refusing the attestation on its own. Only a launch that can read the conversation counts: a fork, or one of unknown shape. A launch of a named agent type starts that agent fresh and never refuses it. promptCacheRecordedExpiryChangedAt is read-only: the gateway stamps it with every write of promptCacheRecordedExpiry, and a launch keeps refusing until 61 minutes 20 seconds after it. Every other field of PUT /admin/v1/settings, including content mode, trace retention, the heartbeat tier, storeCapacityBytes, hookStash and codexSeatSourceTrim, stays behind settings:content.

anthropicCacheLookback

Defaults to false pending paired session validation. Set it through PUT /admin/v1/settings (or legacy PUT /admin/settings).

When enabled, native Anthropic API-key Messages requests can use a spare cache breakpoint after a large tool batch. It adds a 5-minute marker inside the client's cached conversation without editing tools, system text, or message content. Four-marker requests, extended and scoped caching, custom hosts, subscription passthrough, and other providers are excluded. x-anyray-optimize: off bypasses it. Setting it back to false stops annotations, and peer replicas refresh within 30 seconds. No keep-alive requests are sent.

Tenant and deployment resources

Each operation carries x-scope: tenant or x-scope: deployment. In fleet mode (ANYRAY_MULTI_TENANT=true) a tenant-bound principal reaches tenant resources only (keys, spend, optimizer, traces, the GDPR routes, health); a deployment resource answers 403 with deployment_scope_required. Off fleet mode the level is informational.

Errors

Every error uses one envelope, whatever the status:

Management API error envelope

{
  "error": {
    "code": "insufficient_scope",
    "message": "This operation requires the 'audit:read' scope, which this API key was not granted.",
    "required_scope": "audit:read",
    "request_id": "8c2f0b1e-synthetic-request-id"
  }
}

code is stable: unauthenticated, insufficient_scope, deployment_scope_required, invalid_request, not_found, conflict, rate_limited, unavailable, plus legacy reason codes such as admin_key_store_unavailable and session_store_unavailable. 401 means the credential is missing or bad. 403 means it is fine but lacks the scope, and names it. request_id repeats the x-request-id response header; send your own identifier in that request header and it is echoed back.

Extend one user's budget on v1

curl -X PATCH https://<your-anyray-gateway>/admin/v1/users/alice/limits \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "maxUsd": 250 }'

The resource-by-resource map, with the legacy twin of every v1 path, is on the endpoint reference. Every v1 operation in the spec also names its twin in x-legacy-operation.

Per-session strategy gate

Method	Path	Scope	Does
get	/admin/v1/optimizer/session-gate	optimizer:read	Returns the tenant's tenant, evaluator settings, and strategies.
put	/admin/v1/optimizer/session-gate/{kind}	optimizer:write	Takes {"override":"force_on"}, {"override":"force_off"}, or {"override":null}.

Legacy twins: /admin/optimizer/session-gate and …/session-gate/{kind}; legacy read capability config:read. A deployment owner may select ?tenant=synthetic-team, but a tenant-bound principal cannot widen its scope that way. A concurrent change returns 409. State and audit commit together in Postgres.

The strategy record

Field	Values
kind	The strategy id.
state	Durable off, shadow, or on.
effectiveState	state with any override applied. It has no expiry.
stateLabel	off, shadow, on:grandfathered, on:promoted, on:forced, off:forced.
origin	grandfathered or new. Only cohort capture sets it.
reason	existing_spend, fresh_deployment, new_tenant, promoted, measured_loss, consecutive_losses, recovered, operator_override, operator_cleared, legacy_loss_reverted.
evidenceStatus	available, session_gate_disabled, no_spend_store, store_loading, store_unavailable. session_gate_disabled means sessionGate.measurementFraction is 0. Older responses may say holdout_disabled.
lastEvaluation	null before the first check. Carries evaluatedAt, from, to, policyKey, qualityVerdict, metrics, inputTokensRatio, pairedSession, the settings used, passed, and decision (the epoch's progress and its lastLook, with each test's p, level and rejected). imbalancedEvaluations appears only on evaluations the retired paired rule wrote.
Also present	enabled, override, failedEvaluations, revision, updatedAt, evidenceSince, policyKey.

Clearing an override restores grandfathered tenants to on:grandfathered and new tenants to shadow, with a fresh evidence cutoff. force_on bypasses statistical approval, but strategy configuration and lane guards still apply.

Paired session fields

GET /admin/spend/quality-parity?cohortKind=prompt_compression, and its /admin/v1/spend/quality-parity twin, add tenant (null for an owner-wide query) and pairedSession (null without cohortKind). Existing metrics and grossCostObjective remain. The session gate no longer decides on these fields; they stay for display.

pairedSession field	Meaning
method	same_shape_without_replacement_v1
sessionsPerArm.holdout, .treated	Observed sessions in each arm
pairs, wins, ties	Distinct matched pairs; strictly cheaper treated sessions win
winRate, winRateWilson95.lo, .hi	Wins divided by matched pairs, with a two-sided 95% Wilson interval; null without observations
costRatio.median, .q3, .max	Nearest-rank treated/control cost ratios; null if absent or unbounded
unpairedTreated, unusedControls	Sessions with no remaining same-shape partner
unpairedTreatedCostShare	Unmatched treated cost divided by total treated cost; null when total cost is zero
unboundedRatios	Matched positive cost against a zero-cost control
excludedMixedSessions	Sessions observed in conflicting arms or policies
invalidCostSessions	Excluded sessions with missing or invalid usage on a successful provider turn
excludedLongSessions	Excluded labels spanning UTC dates or more than 200 observed turns
excludedLongSessionShare	Long-session exclusions divided by classified candidate sessions, excluding error-only sessions
errorOnlySessions	Sessions whose every turn has status >= 400; excluded from arm counts and pairs

Evaluator settings

Set config.sessionGate in the optimizer settings JSON through PUT /admin/v1/optimizer (legacy /admin/optimizer/settings). No environment variable is needed. The measurement fraction is per tenant; the other settings govern the evaluator across tenants.

Field	Default
evaluatorEnabled	true; false pauses evaluations and evidence resets
measurementFraction	0; per-kind size of each control and treated arm, maximum 0.5
evaluationIntervalSeconds	3600
windowSeconds	2592000 (30 days); the longest one evidence epoch runs
failedEvaluationsBeforeOff	3
alpha	0.05; may only be lowered
alphaSpending	information; look_count restores the per-look schedule

minSessionsPerArm, winRateFloor, maxQ3Ratio, maxUnpairedTreatedCostShare and maxExcludedLongSessionShare are retired. Stored values still validate and are ignored.

Durable approvals never expire, and explicit overrides stay until cleared. Pausing evaluation stops state changes while existing gate permissions keep applying.

When a strategy is promoted or demoted

The gate compares whole sessions in the two arms. Each session gets a cost score: its list-price cost, adjusted for the size of the request the gateway sent the optimizer on the session's first turn. That size is measured before any strategy runs, so a strategy that inflates prompts cannot inflate its own adjustment. Sessions over 200 turns, crossing a UTC date, failing every turn or carrying unpriceable usage rank as the most expensive outcome, so pushing sessions out of the sample cannot make a strategy look cheap. Only sessions labelled by the per-kind draw count.

Test	A loss when the treated arm
cost	ranks higher on cost score
cost_tail	is over-represented above the pooled cost-score Q3
errored_sessions	has more sessions with a failed turn
turns	ranks higher on turns per session
output	ranks lower on output tokens per successful turn

An epoch spends alpha by information. t is the sessions the epoch holds over the sessions the tenant's traffic will hold by windowSeconds, projecting the rest of the window at the rate since the last look (at least half the epoch's average rate). At t the epoch has spent 2(1 - Φ(z/√t)), scaled to reach alpha at t = 1 (O'Brien-Fleming type). The first hours spend almost nothing and the last look runs near alpha. Each look spends what that adds since the last look that could reject. With alphaSpending: look_count, look k spends alpha / (k(k+1)) instead. An epoch keeps the schedule it started spending under. A look's budget goes only to verdicts that can change the state: a loss for on, a win for off, and half each for shadow. The loss tests share their part by Tarone's procedure, so only tests that can reach significance at that sample size take a share. Rank tests are exact for small arms. Any rejection is a loss. A win needs cost to rank lower within the win's part, no more treated sessions above the Q3 than chance predicts, and no loss. Anything else is inconclusive and changes nothing.

Evidence builds up from the start of an epoch. A look falls due one evaluationIntervalSeconds in, again each time the epoch's age doubles, and a last time at windowSeconds, when a fresh epoch starts. The budgets sum to alpha, so a strategy that does nothing changes state in at most that share of epochs. Evaluations between looks read nothing.

A loss moves on to shadow. failedEvaluationsBeforeOff losses with no win between them reach off. Every win or loss starts a new epoch.

Automatic off keeps one bounded recovery arm: it runs only in sessions that select it with the treated arm. A win returns off to shadow with reason recovered, and a later win may promote. force_off disables recovery sampling.

A strategy demoted by the retired paired rule is judged again over a full window. Once that window holds enough sessions to test, no significant loss returns it to its seed state with reason legacy_loss_reverted, and a significant loss keeps the demotion. Until then it stays demoted.

How sessions are paired

The paired read is for display; decisions use the tests above. A session key is the tenant, the attributed user, and the store's clientSessionId. Sessions group by client tool, then match on task size inside that group: each arm is ordered by the entry turn's size (promptTokens + tokensSaved), and evenly spaced positions through each arm are paired off. Both samples therefore span their own full size range, so an unbalanced arm never compares treatment against systematically smaller control tasks. Pairing never sorts by cost, crosses tenants, or reuses a control.

sessionGate.measurementFraction (default 0, capped at 0.5) is the per-kind measurement rate. Each configured kind draws equal, disjoint holdout and treated slices of that size. A session-level selector then measures at most one kind, so no session controls two strategies. Selection competes among a session's candidates, so each kind's realized rate lands below sessionGate.measurementFraction. Only gateway-derived conversation IDs qualify; static metadata.session and metadata.sessionId labels do not. Subagents inherit the parent conversation's draw.

Cost uses costUsdFrom and official list prices. Failed requests and semantic-cache hits cost zero, and telemetry rows are not turns. Mixed-arm sessions are dropped from both the pairs and the gate's sample, and counted.

Cohort capture and policy keys

On its first successful store connection, the gate captures tenants with pre-upgrade spend as grandfathered, and every byte-changing kind starts on for them. A capture that finds no spend marks the deployment fresh (fresh_deployment), and its tenants also start on. Where there was spend at capture, later tenants start shadow. The cohort is immutable.

Each kind's policyKey fingerprints the assignment epoch, that kind's merged tenant parameters, and registry flags. Changing those parameters clears the kind's evaluation and restarts its evidence window, preserving its state. An epoch change resets every evidence window and preserves every durable state. Without a readable permission snapshot or the metadata-only boot cache (sessionGateOrigins.json under ANYRAY_DATA_DIR), unknown tenants withhold byte-changing strategies (session_gate_unavailable).

Request headers

Beyond Authorization and Content-Type, the gateway reads a few x-anyray-* headers:

x-anyray-providerstring

Provider to route to: openai, anthropic, vertex-ai, bedrock, azure-openai, … Optional when a default provider or routing config is set server-side.

x-anyray-metadatastring (JSON)

Attribution, e.g. {"user":"alice","team":"platform"}. Drives spend attribution and per-user caps. Never carries prompt/response content.

x-anyray-configstring (JSON)

Per-request routing: fallback list, load-balance weights, retry policy. No credentials; they resolve from the server-held key store.

Response headers

Every response carries x-anyray-provider (the provider the request was routed to). x-anyray-trace-id is the id of the request's trace; read it with GET /admin/v1/traces/{id} once written. It is absent when the request records no trace (no trace store, or a token count only).

When the provider returns a 5xx, such as an Anthropic 529 overloaded_error, the gateway adds the markers below. The status and the error type are forwarded unchanged; only the human-readable error.message gains a sentence naming the provider:

Relayed Anthropic 529

{
  "type": "error",
  "error": {
    "type": "overloaded_error",
    "message": "Overloaded — this error was returned by Anthropic (api.anthropic.com) and relayed unchanged by the Anyray gateway; it is not an Anyray gateway failure. Check https://status.claude.ai/ for provider status."
  }
}

x-anyray-error-sourcestring

Present and set to upstream when the error originated at the provider, not the gateway. Absent on success and on gateway-generated errors.

x-anyray-upstream-providerstring

The upstream provider that returned the error: anthropic, openai, vertex-ai, … (omitted when the provider is unknown).

x-anyray-upstream-statusstring

The provider's own HTTP status (e.g. 529, 503).

Telling provider errors from gateway errors

No x-anyray-error-source: upstream header on a 5xx means the gateway itself failed, so check your deployment. With it, the fault is on the provider's side, and retrying shortly usually clears it.

Endpoints

/v1/* serves an OpenAI-compatible inference API plus the Anthropic-native Messages API. /admin/* is the admin-gated configuration and reporting API. /sso/cli/* is the pre-key self-service login handshake Connect uses. /scim/v2/* is an inbound SCIM 2.0 server. The complete capability-annotated index is the Endpoint reference; full schemas are in the OpenAPI spec.

POST /connect/optimize-output also accepts Claude's native PostToolUse HTTP hook envelope. It returns {} to preserve the original tool result while session retrieval is unverified. Command hooks keep using the Connect envelope.

Not every endpoint runs on every provider

Chat completions, embeddings, and the Anthropic Messages API work across all supported providers. Image, audio, Responses, file, and Batch endpoints are OpenAI / Azure-only passthrough (audio also works on groq).

On POST /v1/chat/completions and POST /v1/messages, an API-key request that declares tools but no retrieval tool goes through unchanged by default. Once an admin opts its class in (servedRetrievalClasses below), the gateway may resolve its anyray_retrieve calls itself (up to 3 rounds) and return the model's final answer. A streaming request keeps streaming: the gateway holds back only the tail it must decide on. The spend row sums usage from every upstream round. The rounds the gateway added cost, and come off the row's savings, which never go below zero. See servedRetrieval below.

Retrieval via MCP: POST /mcp

The gateway is a remote MCP server (streamable HTTP). It serves anyray_retrieve and anyray_recall behind the same client key as every other route, so an MCP-capable agent runtime closes the retrieval loop with one server entry and no code:

Any MCP client registration

{
  "type": "http",
  "url": "https://<your-anyray-gateway>/mcp",
  "headers": { "x-anyray-api-key": "<client key>" }
}

The path is /mcp, not /v1/mcp

/v1/mcp is not a route: an MCP client's POST there answers 404 with a hint that lists the main inference routes. Point the client at /mcp on the gateway origin.

The wire rules:

• One JSON-RPC 2.0 message per POST: initialize, ping, tools/list, tools/call.
• Notifications answer 202 with no body. Batch arrays are rejected. GET /mcp and DELETE /mcp are 405: the gateway keeps no MCP session to stream or end.
• initialize returns an Mcp-Session-Id, and every call still authenticates independently.
• Successful initialization and other valid messages renew the presenting key's 6-hour retrieval lease, the same evidence the stdio server's heartbeat gives. Rejected initialization claims no capability.
• Tool failures come back in-band (isError: true, with "unknown or expired handle" or "retrieval unavailable"), never as protocol errors, so the model can read them and adapt.

A client that declares the two tools keeps them on the upstream wire from a session's first request, unless that request sends x-anyray-retrieval-tools: lazy or the session already has a recorded request, as Claude Code main threads do (details). The lane counts as retrieve-capable from the first request either way, so reversible strategies run as before. In an app tool loop, harnessRetrieval decides both: while it is off, the tools never reach the model and the lane has no read path.

Hosted connectors (Anthropic mcp_servers, OpenAI Responses type:"mcp") may point here and count as retrieve evidence when they target this gateway's own /mcp. First-party provider APIs only (not Bedrock or Vertex), and the endpoint must be reachable from the provider's cloud.

servedRetrieval

servedRetrieval defaults to true. It is the master switch for gateway-served retrieval, which arms only for the evidence classes in servedRetrievalClasses (none by default). For an armed API-key request that declares tools but carries no retrieval tool, the gateway serves anyray_retrieve and anyray_recall after the first elider marker so the reversible eliders can run. Set it to false through PUT /admin/v1/settings (or legacy PUT /admin/settings) to turn every class off at once; on a warm session that rewrites the cached prefix once. Requests with extended thinking enabled are not served: a thinking signature would bind to rounds the client never sees.

servedRetrievalClasses

servedRetrievalClasses defaults to [], so no request is served and a tool loop goes through unchanged. It lists the evidence classes where served retrieval may arm. Each class is its own opt-in:

Class	The client	Wire
unlatched_deferred	Claude Code with a deferred tool catalog and no Anyray MCP server	/v1/messages
unapproved_client	Codex 0.153 or later whose Anyray MCP tool was never approved	/v1/responses over HTTP
no_discovery	A client with its own tools and no read path, such as an agent SDK loop	/v1/chat/completions, /v1/messages, and /v1/responses over HTTP

Set it with PUT /admin/v1/settings, for example {"servedRetrievalClasses": ["unapproved_client"]}. servedRetrieval must also be true, and every exclusion above still applies. On /v1/responses the request must send store: false (Codex always does; an agent SDK has to set it). The gateway also stands aside for a tool_choice other than auto and for background: true, and never serves Codex's WebSocket transport. An opted-in session changes its tools block once, on the first turn that carries a marker, and keeps the tools if the client later approves or registers its own retrieval path.

Opt a class in only after a paired comparison against pass-through on that session shape clears. no_discovery on the two chat wires failed one: on a LangGraph tool loop, the served rounds re-sent the whole context and cost more than the trims saved.

harnessRetrieval

harnessRetrieval defaults to false. It decides whether the gateway trims tool output in an app tool loop: a request whose x-anyray-metadata tool is sdk-agent, langchain or langgraph and that declares anyray_retrieve itself, as the Add agent prompts wire it. Other clients are unaffected, and the gateway never serves retrieval to these requests.

While it is false, such a request has no read path. Strategies that need the model to read trimmed content back stand aside, and the gateway removes the declared anyray_retrieve and anyray_recall tools from the upstream request on every path, even when the optimizer skips or fails open. /connect/self-check reports the runtime's verdict as parity.

Turn it on in Console → Optimizer → Harness retrieval, or set it to true through PUT /admin/v1/settings (or legacy PUT /admin/settings). The gateway may then trim tool output behind a ctx_ handle, and the verdict reads wired.

Changing it rewrites the tools block of each live app session once, so its next request re-reads the cached prompt at full price. Replicas pick up a change within 30 seconds. Sessions that load tools through tool search are never gated.

hookStash

hookStash defaults to false. When on, a Claude Code hook digest keeps its full output in the gateway's encrypted context stash (48 hours) instead of a local hook-tee file, and its footer names a ctx_ handle for anyray_retrieve. Only a seat with fresh evidence it calls anyray_retrieve gets a handle; every other seat keeps its local file, as does every seat while content mode is off. It sends command output to the gateway, so it needs settings:content. Turning it off sends new digests back to the local file; issued handles keep working until they expire.

codexSeatSourceTrim

codexSeatSourceTrim defaults to false, and this release does not read it yet. The Codex ChatGPT seat lane relays Codex's bytes unchanged over the WebSocket and HTTPS. The switch is for trimming that lane's new tool outputs on the socket, repeating each trim when Codex re-sends the output after a reconnect or an HTTPS fallback. Trims keep what they remove in the encrypted context stash, so it needs settings:content. Turn it on only after a paired comparison against pass-through on long Codex sessions clears.

claudeCodeRemoteMcp

claudeCodeRemoteMcp defaults to false. When on, GET /connect/policy carries retrievalMcp: { url: "https://<gateway>/mcp" }, offering the gateway's MCP server to Claude Code seats in place of connect's local stdio one. The field is withheld while content mode is off (only the stdio server can read a local-transcript handle) and when ANYRAY_GATEWAY_PUBLIC_URL is not set. Claude Code reads its MCP servers at session start, so a running session keeps its transport. If content mode turns off under a session already on /mcp, that session gets no reversible trims until it restarts on the stdio server.

The remote server serves anyray_retrieve and anyray_recall only. Seats lose anyray_read_batch and anyray_history, the hook-tee id route in the retrieve description, and the stdio server's MCP instructions, so the tool list in every Claude Code prefix changes. Before turning it on, run a paired comparison against the stdio transport on that deployment's session shape. Judge the tail, not the mean: the win rate must sit clearly above the pass-through noise floor, the third quartile of per-round paired cost ratios must stay below 1, and the maximum cost, the long-tail turn counts and retrieve re-read loops (repeated anyray_retrieve calls for the same handle) must not grow.

Once a seat has switched to /mcp, its hook stops handing out local hook-tee handles (digest and ranged footers, green-collapse and re-read stubs) to any session that has not retrieved through the stdio server since the switch, because /mcp cannot open them. This includes a resumed session. Those outputs pass through unchanged, or take a gateway ctx_ handle when hookStash is on. Include that lost trimming in the comparison. A missing field sends the seat back to stdio only after two policy pulls in a row leave it out. The seat also stays on stdio when another MCP server already uses the name anyray-retrieval, or when the URL is not on the seat's own gateway. Tool approvals for the transport a seat leaves stay in place for sessions still running on it.

Each switch, either way, costs a resumed session one full prompt-cache write, because its tool list changes. Also check two things only the remote server can do. First, it can re-initialize mid-session, for example during a rolling deploy. If its instructions or tool text change when it does, the session's cached prefix is lost. Second, a session whose /mcp connection failed at launch has no retrieve tool, yet it can still be handed gateway ctx_ handles.

The remote entry carries the seat's key as a static header for the whole session. Claude Code 2.1.282 ran headersHelper once per connection and did not re-run it after a 401, so a helper would not help. Refresh extends the same key. A new key is issued only on re-enrollment or when the local key is lost; connect then writes it into the entry for new sessions, and the old key stays valid until its own expiry. A session that outlives that expiry, or whose key an admin revokes, can no longer retrieve. Check for that in the comparison too.

optimizerWireDebit

optimizerWireDebit defaults to false. When on, the gateway withholds a session's measured round trip to the optimizer (phase_gateway_wire, the median of its last six measurements in 100ms steps, cap 400ms, and never more than half the hook budget) from the hookBudgetMs it advertises. The optimizer then stops admitting strategies before the gateway's abort fires, instead of timing out and forwarding the original bytes. The gateway itself waits exactly as long as before. A session's first six measurements set its debit at once. After that the debit moves only when five consecutive medians agree on a new value, so it can lag the latest median by a few turns, and two optimizer replicas of unequal speed cannot flap it from turn to turn. Sessions whose round trip stays under 100ms are unchanged. Turn it on per deployment through PUT /admin/v1/settings once the paired gate clears on that session shape.

promptCacheRecordedExpiry

promptCacheRecordedExpiry defaults to false, pending paired session validation. It decides how the gateway tells the optimizer that a pass-through Claude Code session's prompt cache has expired (promptCacheExpired in the optimizer protocol).

Off, the gateway judges against the client's own markers and, on a pinged session, waits out the longest a ping could have kept the cache alive (44 minutes after a finished turn under a 30-minute idle cap). On, the session row answers: it stores the lifetime the gateway sent (5m on a session the 5m setting or a 5m header lowered from 1h), counts every keep-warm ping before the ping is sent, and notes each request of the session's sub-agents. The cache is expired when no turn, ping or sub-agent request touched it for that lifetime.

An expired turn lets observation_mask, relevance_filter and thinking_trim rewrite the cached prefix at no premium, so turning this on changes what the model sees. Turn it on per deployment once a run paired against pass-through clears on long sessions with idle gaps:

Judge expiry by the session row

curl -X PUT https://<your-anyray-gateway>/admin/v1/settings \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"promptCacheRecordedExpiry": true}'

Send false to go back. Replicas follow within 30 seconds, and one that has not read its settings yet reads this as off. Keep-warm refreshes already scheduled when you turn it off stay recorded, and the session row can still hold them back until the session's next request.

adaptiveMcpInstructions and session IDs

adaptiveMcpInstructions defaults to true. In this mode initialize always omits the instructions key, and recovery guidance lives in the retrieval tool descriptions, which reach the model with the first elision.

The signed Mcp-Session-Id holds only a decision and a random nonce. It binds to the authenticated identity and grants no access. Reuse it to keep the decision through a settings change. Invalid or foreign IDs return 404 on initialization, and admin-token rotation also invalidates IDs.

Set the field to false through PUT /admin/v1/settings to restore the legacy instruction text on new MCP sessions; peers refresh within 30 seconds. Apply the switch between conversations, since it changes the prefix of a continuing one. A cold replica that cannot read the setting answers 503 with Retry-After: 30 rather than guessing. Older gateway versions always announce instructions, so reconnecting a warm conversation during a mixed-version upgrade can change its prefix.

GET /admin/v1/health reports process-local mcpInstructions handshake counters by the closed reasons adaptive_omitted and runtime_disabled. They carry no identity or content.

Retrieval: POST /connect/retrieve

An agent you build yourself needs a read path, or optimization runs one way only. Declare a tool named anyray_retrieve and implement it against this route with the client key you already send. Three independent requirements:

Requirement	What proves it
Declaration	The model can select the tool this turn. Matched by name (anyray_retrieve, or a namespaced mcp__…__anyray_retrieve), so renaming it silently turns retrieval off.
Implementation	The route call that resolves a handle.
Liveness	An authenticated empty POST /connect/mcp-heartbeat at worker start and every 5 minutes while the worker can execute tools.

Without both a callable path and a fresh lease, the gateway keeps clientCanRetrieve false. Retrieval-dependent strategies stand down, and degradable ones keep their handle-free mode.

Resolve a ctx_ handle

curl -X POST https://<your-anyray-gateway>/connect/retrieve \
  -H "x-anyray-api-key: $ANYRAY_CLIENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"handle":"ctx_abc123"}'

handlestringrequired

The ctx_… handle from an [anyray: … retrieve ctx_…] marker in the message the model received.

startLine / endLinenumber

Read back a slice instead of the whole original.

grepstring

Return matching lines with context. Max 256 characters.

Answers {"status":"ok","content":"…","handle":"ctx_…"}. An unknown or expired handle is 404, an unavailable retrieval store is 503, and a missing or invalid key is 401. POST /connect/recall takes a natural-language query instead of a handle and returns ranked handle + score + preview matches, for when the marker has scrolled out of the model's view.

This is the route the /mcp endpoint's own anyray_retrieve tool calls. Implement against it directly only when the runtime cannot register an MCP server. A human or CLI caller that is only probing sends x-anyray-retrieve-probe: 1 so the call does not latch capability. Worked examples per SDK: Retrieval reference.

Retrieval read-back

Read-back shows whether a lane's handles actually resolve. Every read (served rounds, /mcp, /connect/retrieve, /connect/recall) is counted under the reading request's client tool and, for a service key, its agent.

GET /admin/v1/spend/dashboard returns retrieval.byClientTool, one entry per tool with read-back activity in the window. The tool id is the tool in x-anyray-metadata, else the User-Agent match.

Field	Meaning
clientTool	The tool id. A missing or malformed one reads unknown.
freshHandlesEmitted	Handles this tool's requests minted. The only valid denominator for a read-back rate. Omitted when an older gateway's rows make it partial.
retrieves, recalls	Read events from every reader.
hits, misses	Those events by outcome.
servedRounds	Rounds the gateway served.
servedOutcomes	The served loop's own counts: hits, misses, unavailable, malformed, capped (still asking at the round limit), mixed (client and served calls together), errors.

GET /admin/v1/keys/:id and GET /connect/self-check add readBack to a measured retrievalRuntimes, per tool over the same 24 hours:

A served lane that resolves its handles

"readBack": [
  {
    "clientTool": "langchain",
    "emitted": 12,
    "retrieved": 9,
    "missed": 1,
    "served": { "unavailable": 0, "capped": 0, "errors": 0 }
  }
]

emitted counts handles minted; retrieved and missed count anyray_retrieve reads. served counts the served loop's failures on this key and tool: reads the stash could not answer, requests that hit the round cap, and loop errors. The Agents card and the control plane's read-back alert read these counts (Read-back). emitted is omitted when an older gateway's rows make it partial. An empty list means nothing was minted or read; a missing readBack means an older gateway. Reads through /mcp or /connect/retrieve count under the caller's own tool id, so send the same x-anyray-metadata tool to land on the same entry.

Claude Desktop managed bootstrap

An organization routing Claude Desktop through Anyray manages its third-party profile through the deployment-owner API:

Set the organization policy

curl -X PUT https://<your-anyray-gateway>/admin/claude-desktop-policy \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "settings": {
      "inferenceModels": [{"name": "claude-sonnet-4-6"}],
      "managedMcpServers": {},
      "allowedPluginMarketplaces": [],
      "chatTabEnabled": true,
      "coworkEgressAllowedHosts": ["*"]
    }
  }'

Route	Does
GET /connect/claude-desktop/bootstrap?os=macos|windows|linux	Returns the effective secret-free policy, with the gateway provider and base URL, helper credential mode, and that OS's inferenceCredentialHelper path injected by the server. The gateway's own Skills and Guidance marketplace is merged into allowedPluginMarketplaces as one more element, leaving any the admin set in place. os defaults to macos. 404 while disabled.
GET /admin/claude-desktop-policy/mobileconfig	Downloads the MDM trust profile. It pins the bootstrap endpoint and /usr/local/bin/anyray-bootstrap-headers-helper without embedding a key, and names /usr/local/bin/anyray-credential-helper.
POST /admin/claude-desktop-policy/connectors/probe	Takes {"url": "https://..."} and asks that MCP server for its name and tool list. Answers ok (with name, version, tools, truncated), needs-auth, or unreachable with a reason. HTTPS only, nothing stored.
GET /admin/claude-desktop-policy/connectors/readiness	Returns the most-blocking organization sign-in state and the /mcp/org endpoint URL. Never probes vendors. v1 twin GET /admin/v1/policies/claude-desktop/connectors/readiness (policies:read).
POST /admin/v1/policies/claude-desktop/connectors/readiness/refresh	Refreshes the process-local access cache with at most four probes in flight (policies:write).
POST /admin/claude-desktop-policy/connectors/authorize	Takes {"name": "Atlassian"} and returns a vendor authorizationUrl. No response contains a token.

Current Claude Desktop sends Authorization: Bearer <client key> on the bootstrap fetch through the MDM-pinned bootstrapHeadersHelper. The route verifies a presented bearer (401 when rejected) and still serves an unauthenticated fetch in the no-OIDC lane. The write boundary rejects credential-shaped string values at every depth, including under otherwise allowed free-text keys.

After consent, the authorize callback shows a short claim code. Redeem it with POST /connect/mcp-oauth/complete and a client key. The grant belongs to the account that redeems it, never the one that started the flow, and developers start their own flow with POST /connect/mcp-oauth/authorize. On the readiness read, OAuth adoption is a count, fewerThan, or unavailable union, and counts of one and two return only as fewerThan: 3.

Import and fleet deployment: Desktop apps.

SCIM v2 provisioning

Point the IdP at this base URL and send the separately configured SCIM bearer on every request:

SCIM base URL

https://<your-anyray-gateway>/scim/v2

Resource	Routes
Discovery	GET /ServiceProviderConfig, GET /ResourceTypes, GET /Schemas
Users	POST /Users, GET /Users, GET /Users/:id, PUT /Users/:id, PATCH /Users/:id, DELETE /Users/:id
Groups	POST /Groups, GET /Groups, GET /Groups/:id, PUT /Groups/:id, PATCH /Groups/:id, DELETE /Groups/:id

User lists support filter=userName eq "dev@example.com", startIndex, and count. Group lists support the same paging fields. User PATCH accepts active:false for immediate offboarding. Group PATCH accepts add, remove, and replace member operations, and a missing member can be created when it carries a verified email in display or value.

Configure the bearer, admin group, and team mappings through the deployment-owner admin lane:

Configure SCIM

curl -X PUT https://<your-anyray-gateway>/admin/scim/settings \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bearerToken": "synthetic-scim-bearer-token-replace-me-0001",
    "adminGroup": "Anyray Admins",
    "groupTeamMap": { "Engineering": "platform" }
  }'

GET /admin/scim/settings returns configured, adminGroup, groupTeamMap, and updatedAt, never the bearer or its hash. A later PUT can omit bearerToken to keep the current credential.

SCIM errors always use application/scim+json and this envelope:

SCIM error

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:Error"],
  "detail": "invalid bearer token",
  "status": "401"
}

Errors

Most inference errors use the OpenAI envelope ({ "error": { "message", "type", "code" } }). Client-key authentication uses the fixed machine envelope described under Authentication.

Status	Meaning
400	Malformed request: failed validation, bad JSON, or an unknown provider or model.
401	Missing, malformed, unknown, expired, revoked, or SCIM-deactivated client identity. The body carries status: failure, a reason code, the fixed valid client key required message, and a safe hint.
402	A governance ceiling was reached: the deployment entitlement or spend cap, a service key's monthly budget, or a per-user token cap or USD budget. error.code is service_key_budget_exceeded, user_token_cap_exceeded, or user_budget_exceeded. Terminal, not retryable: only an admin raising the limit, or the window rolling, clears it.
429	Rate limit exceeded. Honors Retry-After.
503	A dependency is temporarily unreachable. Client-key verification returns code: key_store_unavailable. This is infrastructure, not a credential problem, so re-running anyray-connect will not help. Verification fails closed, so the request never reaches the provider. If it persists, check GET /admin/health.

Source trim evidence

The admin key list and detail, and the GET /v1/me key object, carry these UTC ISO fields:

Field	Meaning
sourceTrimSince	When observation began.
sourceTrimRequestCount	Successful inference requests from source-capable clients since observation began, or since the last recorded source trim. A missing count means an older gateway, not zero.
sourceTrimLastRequestAt	The last such request.
sourceTrimAt	The last recorded source trim. A successful hook spend event resets the count and start stamp.

Migration 0097_client_keys_source_trim adds the columns without backfilling historical traffic. Key records also expose short-lived sourceHook installation and retrieval reports, separately from retrieveCapableAt.

Hook contract, supported clients and grace period.