API reference

The gateway's inference, organization MCP, admin, and inbound SCIM v2 surfaces.

Browse and try every endpoint, inference (/v1/*) and admin (/admin/*), in the interactive API explorer your gateway serves itself at GET /docs, backed by the OpenAPI 3.1 spec at GET /openapi.json (load that into Postman or codegen too). Both routes are unauthenticated and expose only the API shape; "try it out" calls run against the same gateway, so real requests still need your client key or admin token. A copy of the spec is published at docs.anyray.ai/openapi.json if you don't have a deployment in front of you. The capability-annotated index of every route is the Endpoint reference.

Open the explorer

open http://localhost:8787/docs      # or https://<your-anyray-gateway>/docs

POST /connect/optimize-output also accepts Claude's native PostToolUse HTTP hook envelope. It returns {} to preserve the original tool result while session retrieval is unverified. Command hooks continue using the Connect envelope. Key records expose short-lived sourceHook installation and retrieval reports separately from retrieveCapableAt.

Base URL

The gateway is self-hosted: it runs on your own infrastructure (default port 8787), not on docs.anyray.ai. Put it behind TLS for any networked deployment.

Base URL

https://<your-anyray-gateway>        # e.g. http://localhost:8787 in-network

Deploying and configuring the gateway (provider keys, content mode, the optimizer URL): Configure.

Authentication

Send your Anyray client key as a bearer token, in x-anyray-api-key, or, for Anthropic-native clients on /v1/messages, in x-api-key. The real provider key stays server-side.

Authenticated request

curl https://<your-anyray-gateway>/v1/chat/completions \
  -H "Authorization: Bearer $ANYRAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H 'x-anyray-provider: openai' \
  -d '{
    "model": "gpt-4o",
    "messages": [{ "role": "user", "content": "Summarize the release notes." }]
  }'

Six credentials:

Credential	Used on	Grants
Client key (Authorization: Bearer, x-anyray-api-key, or ark_ in x-api-key)	/v1/* inference	The holder's user/team attribution. Minted by an admin enrollment flow or self-service SSO with anyray-connect --sso.
Service key (ark_svc_…, sent through the same headers)	/v1/* inference	A revocable non-human service identity for an AI agent, CI job, or SDK script, with an optional per-key monthly dollar budget. No interactive SSO.
AuthKit access token (Authorization: Bearer)	/mcp/org	A WorkOS user in the customer organization. The audience must equal this gateway's exact /mcp/org URL. The gateway verifies it and Billing checks live membership.
Admin token (ANYRAY_ADMIN_TOKEN)	The console + /admin/* config API	Break-glass owner access, kept server-side. Each /admin/* route requires a capability; enterprises can add console SSO + RBAC.
Admin API key (aak_…, Authorization: Bearer)	/admin/* config API	A revocable automation credential carrying either an explicit capability subset or native Management API v1 scopes. Never inference; never trace content. See Scoped admin API keys.
SCIM bearer (Authorization: Bearer)	/scim/v2/*	Lets the configured IdP provision users and groups for one tenant. Only its SHA-256 hash is stored.

Inference uses a minted client key

Every /v1/* request needs a valid minted key. The key's user and team control attribution; x-anyray-metadata cannot override them. Once the organization configures SSO, a developer mints their own key with the enrollment command from Users → SSO enrollment; an org admin can also mint and rotate keys. OAuth/OIDC never runs on /v1/* itself.

Missing, unknown, expired, revoked, and SCIM-deactivated identities return HTTP 401 with missing_key, key_unknown, key_expired, key_revoked, or user_deactivated and a safe fix hint. An unreachable key store is not a 401: verification fails closed with HTTP 503 and key_store_unavailable, so a database outage is never mistaken for a bad credential.

Organization MCP access with SSO

An MCP client that cannot attach a static header adds one URL:

https://<your-anyray-gateway>/mcp/org

The unauthenticated request returns 401 and points the client to the protected-resource metadata:

WWW-Authenticate: Bearer resource_metadata="https://<your-anyray-gateway>/.well-known/oauth-protected-resource"

That public metadata names WorkOS AuthKit as the authorization server. AuthKit runs the authorization flow through the enterprise SSO connection already attached to the customer's WorkOS organization. The gateway does not implement authorization codes, PKCE, client registration, consent, token refresh, or revocation.

The access token must meet all of these checks:

Claim or property	Required value
signing algorithm	RS256
iss	the managed AuthKit issuer
aud	one scalar string equal to https://<your-anyray-gateway>/mcp/org
time and subject	valid sub, iat, and exp
organization	non-empty WorkOS org_id

No custom mcp:org scope is advertised or required. The exact resource audience separates one gateway from another.

After JWT verification, the gateway sends only sub and org_id to Billing under its deployment bearer. Billing requires the tenant's activated SSO marker and a currently active WorkOS connection. It fetches the current WorkOS user, requires exactly one active membership in the configured tenant organization, resolves the current groups to an Anyray team, and applies allowed domains and revocation. The AuthKit token never reaches Billing or storage. Keep non-SSO methods disabled in the WorkOS domain policy and require SSO in the organization policy. Otherwise a manually enabled Google or Magic Auth method can bypass the enterprise IdP.

A client key still works on /mcp/org and takes precedence when presented. This keeps Connect-managed tools compatible. Pre-verification and live identity throttles return 429 with Retry-After; concurrent config, JWKS, and metadata refreshes share one outbound request. Billing, WorkOS, or AuthKit key-discovery outages fail closed with 503. See Organization MCP endpoint for setup and the endpoint reference for every route.

Billing also returns 503 until a vendor operator confirms the current WorkOS organization policy and the exact Resource Indicator derived from this deployment's public gateway URL. A WorkOS organization rebind or gateway URL change invalidates the matching confirmation. This lane never provisions WorkOS for an ordinary tenant. The WorkOS organization is created only after an authenticated owner explicitly chooses Set up SSO, or an operator deliberately binds an existing organization. Non-SSO tenants continue to use client-key MCP authentication.

Service keys for agents and CI

Three admin routes manage service keys:

Method	Path	Capability	Shape
post	/admin/service-keys	clientkeys:manage	{ "name": string, "team": string, "monthlyBudgetUsd"?: number } → 201 { "key": "ark_svc_…", "record": {...} }
get	/admin/service-keys	config:read	{ "keys": [...] }; filtered to service keys, with source and type set to service
delete	/admin/service-keys/:id	clientkeys:manage	Revokes the key and returns { "ok": true }

Mint a budgeted CI key

curl https://<your-anyray-gateway>/admin/service-keys \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "synthetic-ci-release",
    "team": "platform",
    "monthlyBudgetUsd": 25
  }'

Store the returned key immediately; the gateway stores only its SHA-256 hash and never returns the raw token again. A budget counts successful request cost per UTC calendar month. At or above the budget, covered inference routes return:

HTTP 402

{
  "error": {
    "code": "service_key_budget_exceeded",
    "message": "This service key has reached its monthly budget. Ask an administrator to raise the budget or wait for the next UTC month."
  }
}

The cap is soft and fails open: completed requests update the shared counter after their responses, so concurrent requests can overshoot, and a counter outage lets requests proceed. A key without monthlyBudgetUsd, or with a value of 0, is unlimited.

Scoped admin API keys

An admin API key (aak_…) scripts the /admin/* surface without the break-glass ANYRAY_ADMIN_TOKEN. Mint with exactly one grant shape: an explicit subset of the admin capabilities, or native Management API v1 scopes. The grant is fixed at mint time and can never exceed the minter's own access; content, content-mode, SSO config, self-update, and key management remain ungrantable. Keys expire after 90 days by default; pass expiresInDays for a different horizon or neverExpires: true to opt out explicitly. Minting requires the adminkeys:manage capability (security_admin and up).

A scope-minted key is deliberately v1-only. It has no capabilities, so every legacy /admin/* capability gate rejects it; use it with /admin/v1/* (and the scope-less GET /admin/v1/me). A v1 403 names the exact scope to mint in required_scope.

Method	Path	Capability	Shape
post	/admin/api-keys	adminkeys:manage	Exactly one of caps: string[] or scopes: string[], plus optional label and expiry → 201 { "key": "aak_…", "record": {...} }
get	/admin/api-keys	config:read	{ "keys": [...] }; metadata only, hashes stripped
delete	/admin/api-keys/:id	adminkeys:manage	Revokes the key and returns { "ok": true }

Store the returned key immediately; only its SHA-256 hash is kept. Revocation takes effect on the next request, on every replica. Admin writes made with a key are attributed in the audit log as apikey:<id>.

For a CI dashboard, mint the one scope its read needs, then call the v1 dashboard:

Mint a CI dashboard key by scope

curl -X POST https://<your-anyray-gateway>/admin/v1/admin-keys \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "ci-dashboard",
    "scopes": ["spend:read"]
  }'

Read the spend dashboard with the scope-minted key

curl https://<your-anyray-gateway>/admin/v1/spend/dashboard \
  -H "Authorization: Bearer $ADMIN_API_KEY"

On the legacy surface a budget bot needs a capability grant instead: a key that can read config and write per-user caps.

Mint a budget-bot key (as an admin)

curl https://<your-anyray-gateway>/admin/api-keys \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "label": "budget-bot",
    "caps": ["config:read", "usercaps:write"]
  }'

Extend one user's budget (with the key)

curl -X PATCH https://<your-anyray-gateway>/admin/user-caps/alice \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "maxUsd": 250 }'

PATCH /admin/user-caps/:user (capability usercaps:write) merges a partial edit into one user's entry: a field you omit is kept, null clears it, a value replaces it. Accepted fields are monthlyTokens, maxUsd, softUsd, and reset (never | daily | weekly | monthly), plus an optional expectedRevision for a conditional write (a stale revision returns 409). Unknown fields are rejected with 400. The whole-map PUT /admin/user-caps remains for bulk edits.

Use keys against the gateway origin directly

The console host's proxy replaces the Authorization header with the browser login cookie, so an aak_ key only works against the gateway itself (port 8787 by default), not through the console hostname.

If the key store is unreachable, aak_ verification fails closed with HTTP 503 and admin_key_store_unavailable (never a 401), and the break-glass admin token keeps working.

Management API v1

/admin/v1/* is the /admin/* surface re-cut for scripts and API keys: plural-noun resources, one permission rule you can read off the URL, one error envelope, and a request id on every response. The surface is stable (x-anyray-api-status: stable): response shapes are the documented contract, and breaking changes would arrive only as /admin/v2. Legacy /admin/* data routes still answer, now with Deprecation: true and a Link: <successor>; rel="successor-version" header naming their v1 path; plan their retirement into your scripts. Requests are capped at 300 per minute per principal (429 + Retry-After past it).

Beyond the re-served routes, v1 adds a key lifecycle (GET/PATCH /admin/v1/keys/{id}, POST /admin/v1/keys/{id}/rotate returning the fresh token exactly once, and POST /admin/v1/keys/{id}/secret returning an agent key's stored value — service keys only, sealed at rest, cache-control: no-store, and audited on every read), per-provider key writes (PUT/DELETE /admin/v1/providers/{slug}/key), and the audit trail (GET /admin/v1/audit/events?limit=&cursor=&category=, a { "data": [...], "next_cursor": ... } page over the admin/GDPR trail, audit:read). Self-service for developers lives on the inference plane instead: GET /v1/me, /v1/me/usage, and /v1/me/limits answer with the verified key holder's identity, month-to-date usage, and the ceilings that would 402 them; no admin credential involved.

The scope rule

A scope is <resource>:<action>, where the resource is the first path segment after /admin/v1/:

Request	Scope it needs
GET /admin/v1/<resource>/…	<resource>:read
POST, PUT, PATCH, DELETE /admin/v1/<resource>/…	<resource>:write
POST /admin/v1/<resource>/…/query (a read whose filter needs a body)	<resource>:read

Resources: keys, admin-keys, users, spend, providers, routing, optimizer, policies, settings, sso, enrollment, provisioning, fleet, traces, audit, playground. GET /admin/v1/me needs no scope and returns the scopes the caller holds. The named exceptions, each declared on its operation in the OpenAPI spec as x-required-scope:

Scope	Gates	Grantable to a key
users:revoke	Revoking or reinstating a developer	no
users:gdpr	A user's export and erase	yes
providers:write on GET /admin/v1/providers	A credential-grade read: provider keys come back in full, so possession is the privilege, not the verb	yes
fleet:enroll	Enrolling machines: the enroll secret (read and rotate), evidence-connector bearers, installer download, the MDM profile, enabling a policy	yes
fleet:remediate	Running a remediation script, retiring a host, pushing policy	yes
playground:run	The admin playground	yes
fleet:direct	Queuing a directive for a developer seat (revert, re-mint, update)	no
settings:content	Content mode and trace retention: PUT /admin/v1/settings/content, and any PUT /admin/v1/settings body naming a field outside the runtime-settings allowlist	no
settings:update	The self-update policy	no
sso:write	IdP and SCIM configuration	no
admin-keys:write	Minting and revoking aak_ keys	no
traces:content, fleet:content	Prompt/response bodies; remediation script output	no

Existing grants keep their reach: a role or an aak_ key still carries capabilities, and each capability translates to the scopes it gated on the legacy paths (config:read is every <resource>:read except the audit, trace, and fleet lanes; usercaps:write is users:write; enrollment:manage is enrollment:write plus fleet:enroll; provisioning:manage is provisioning:write, fleet:write, and fleet:remediate; and so on). The legacy trust boundaries between enrolling, provisioning, and directing never collapse into one another.

The deliberate differences, each pinned by a test:

• The audit lane (/admin/v1/audit/*) needs audit:read, which config:read no longer implies.
• The fleet listings (including GET …/mdm-servers, GET …/installers, and GET …/evidence-connectors) are fleet:read.
• Single-route capabilities of one tier fold into one write scope: routing:write covers routing config and aliases, policies:write covers team skills and the Claude Desktop policy, optimizer:write covers settings and purge.
• The non-content runtime settings (seatOverflow, seatOverflowHourlyMaxRequests, anthropicCacheLookback, adaptiveMcpInstructions) are settings:write; every other field of PUT /admin/v1/settings (content mode, trace retention, the heartbeat tier, storeCapacityBytes, and anything added later) stays behind settings:content.

anthropicCacheLookback defaults to false pending paired session validation. Set it through PUT /admin/v1/settings (or legacy PUT /admin/settings). When enabled, native Anthropic API-key Messages requests can use a spare cache breakpoint after a large tool batch. It adds a 5-minute marker inside the client's cached conversation without editing tools, system text, or message content. Four-marker requests, extended/scoped caching, custom hosts, subscription passthrough, and other providers are excluded. x-anyray-optimize: off bypasses it. Set the setting to false to stop annotations; peer replicas refresh within 30 seconds. No keep-alive requests are sent.

Tenant and deployment resources

Each operation carries x-scope: tenant or x-scope: deployment. In fleet mode (ANYRAY_MULTI_TENANT=true) a tenant-bound principal (an SSO operator, or a key minted for one tenant) reaches tenant resources only (keys, spend, optimizer, traces, the GDPR routes, health); a deployment resource answers 403 with deployment_scope_required. Off fleet mode the level is informational.

Errors

Every error, whatever the status, uses one envelope:

Management API error envelope

{
  "error": {
    "code": "insufficient_scope",
    "message": "This operation requires the 'audit:read' scope, which this API key was not granted.",
    "required_scope": "audit:read",
    "request_id": "8c2f0b1e-synthetic-request-id"
  }
}

code is stable: unauthenticated, insufficient_scope, deployment_scope_required, invalid_request, not_found, conflict, rate_limited, unavailable, plus the legacy reason codes such as admin_key_store_unavailable. request_id repeats the x-request-id response header; send your own identifier in that request header and it is echoed back. 401 means the credential is missing or bad; 403 means it is fine but lacks the scope, and names it.

Extend one user's budget with a scoped key, on v1

curl -X PATCH https://<your-anyray-gateway>/admin/v1/users/alice/limits \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "maxUsd": 250 }'

The resource-by-resource map, with the legacy twin of every v1 path, is on the endpoint reference. Every v1 operation in the spec also names its twin in x-legacy-operation.

Per-session strategy gate

GET /admin/v1/optimizer/session-gate returns the authenticated tenant's tenant, global evaluator settings, and strategies. The legacy twin is /admin/optimizer/session-gate. Read scope: optimizer:read; legacy capability: config:read. A deployment owner may select ?tenant=synthetic-team; a tenant-bound principal cannot widen its scope with that parameter.

Each strategy record contains kind, durable state (off, shadow, on), effectiveState, enabled, override, failedEvaluations, revision, updatedAt, evidenceSince, policyKey, and lastEvaluation (null before the first check). origin is grandfathered or new; reason records existing_spend, fresh_deployment, new_tenant, promoted, measured_loss, consecutive_losses, recovered, operator_override, or operator_cleared. fresh_deployment means cohort capture found no spend in a new install, so strategies start on without an override.

stateLabel distinguishes off, shadow, on:grandfathered, on:promoted, on:forced, and off:forced. effectiveState applies an override to the durable state; it has no expiry. evidenceStatus is available, holdout_disabled, no_spend_store, store_loading, or store_unavailable. The console separates evidence collection from the last quality verdict. Without a prior evaluation it reports no evidence: no spend store, no evidence: optimizer loading, or the corresponding collection cause; an outage never hides a recorded verdict.

On its first successful store connection, the gate captures tenants with pre-upgrade spend as grandfathered, including historical and sessionless rows. Every byte-changing kind starts on for those tenants, including kinds configured off. If capture finds no spend, the deployment is marked fresh and its discovered tenants start on as grandfathered. On deployments with spend at capture, later tenants start shadow. The immutable cohort survives spend retention, restarts and releases. Capture uses indexed pages with a persisted cutoff and cursor, independently of evaluator pause. Failed pages retry; unclassified tenants stay at parity until an origin snapshot arrives. Cached origins and permissions survive store outages. Configuration and lane restrictions still apply.

lastEvaluation contains evaluatedAt, from, to, policyKey, qualityVerdict, quality metrics, inputTokensRatio, imbalancedEvaluations, pairedSession, the settings used, and passed. Each kind's policy key fingerprints the session-assignment epoch, its own merged tenant parameters, targeted parameter rules, and registry flags. It excludes optimizer release versions, unrelated configuration, rollout settings and evaluator settings. A change to that kind's parameters clears its evaluation and restarts its evidence window while preserving its current state. Other kinds retain their evidence. An assignment-epoch change resets all evidence windows while preserving every durable on, shadow, and off state.

PUT /admin/v1/optimizer/session-gate/{kind} takes {"override":"force_on"}, {"override":"force_off"}, or {"override":null}. Its legacy twin is /admin/optimizer/session-gate/{kind}. Both require optimizer:write. Clearing an override restores grandfathered tenants to on:grandfathered and new tenants to shadow, with a fresh evidence cutoff. force_on bypasses statistical approval; strategy configuration and lane guards still apply. A concurrent change returns 409. State and audit commit together in Postgres. Overrides record forced provenance; only capture establishes origin.

Paired session fields

GET /admin/spend/quality-parity?cohortKind=prompt_compression and its /admin/v1/spend/quality-parity twin add tenant (null for an owner-wide query) and pairedSession (null without cohortKind). Existing metrics and grossCostObjective remain.

pairedSession field	Meaning
method	same_shape_without_replacement_v1
sessionsPerArm.holdout, .treated	Observed sessions in each arm
pairs, wins, ties	Distinct matched pairs; strictly cheaper treated sessions win
winRate, winRateWilson95.lo, .hi	Wins divided by matched pairs, with a two-sided 95% Wilson interval; null without observations
costRatio.median, .q3, .max	Nearest-rank treated/control cost ratios; null if absent or unbounded
unpairedTreated, unusedControls	Sessions with no remaining same-shape partner
unpairedTreatedCostShare	Unmatched treated cost divided by total treated cost; null when total cost is zero
unboundedRatios	Matched positive cost against a zero-cost control
excludedMixedSessions	Sessions observed in conflicting arms or policies
invalidCostSessions	Excluded sessions with missing or invalid usage on a successful provider turn
excludedLongSessions	Excluded labels spanning UTC dates or more than 200 observed turns
excludedLongSessionShare	Long-session exclusions divided by classified candidate sessions, excluding error-only sessions
errorOnlySessions	Sessions whose every turn has status >= 400; excluded from arm counts and pairs

A session key is tenant, attributed user, and the store's clientSessionId. Sessions are grouped by client tool (or the sorted set for a mixed-tool session), then matched on task size within that group: each arm is ordered by the entry turn's size, reconstructed as promptTokens + tokensSaved, and evenly spaced positions through each arm are paired off. Both samples therefore span their own full size range, so an unbalanced arm does not compare treatment against systematically smaller control tasks.

Size orders the arms rather than bucketing them. A bucket edge between two comparable tasks would destroy the pair, and a strategy that shifts every prompt would move its whole arm across the edge and stop pairing at all, hiding its own effect. Ordering survives any such shift. Within a group, the session key breaks ties between equal sizes. Pairing never sorts by cost, crosses tenants, or reuses a control. Missing tools share an unknown-tool group. Pairing stays per group, but promotion and demotion use the pooled per-tenant, per-kind result. They do not issue separate per-group approvals. Window bounds apply to observed turns; there is no inferred task-completion signal. Only gateway-derived conversation IDs qualify for trial assignment; static metadata.session and metadata.sessionId labels do not.

holdout.fraction (default 5%, capped at 50%) is the per-kind measurement rate: each canonical configured kind draws equal, disjoint holdout and treated slices of that size. A session-level selector then measures at most one kind, so no session is a control for two strategies at once. The selected kind is withheld (holdout) or run (treated); every other kind runs on its promotion state alone and is left unlabelled, and a session that selects nothing labels nothing. Selection competes among a session's candidates, so each kind's realized rate is somewhat below holdout.fraction and falls as the fraction rises. The configured candidate set is canonicalized and does not change with a turn's enabledKinds allowlist.

Cost uses costUsdFrom and official provider list prices for input, output, cache reads, 5-minute writes, and 1-hour writes. Failed requests and semantic-cache hits cost zero; telemetry rows do not count as turns. Later unlabelled turns of a labelled session still count. Unmatched treated sessions contribute only to the unmatched count and cost share. Wilson and ratio quartiles use pairs only. All-error sessions are excluded; they cannot form zero/zero ties. Mixed-arm and malformed-usage sessions are dropped and counted. Those exclusions block state changes above 5% of classified candidate sessions. Evidence is also insufficient when unpaired treated cost or the excluded-long-session share exceeds its bound (default 25% each). Old-policy rows are dropped before grouping instead of creating mixed sessions. Lane stand-asides and non-selected kinds in a control session omit the kind from strategyCohorts after their safety filters run. Those turns are dropped from cost, quality and pairing; they do not make the remaining session mixed. Own-kind pin-thrash cooloff turns carry cooloff: their cost and turn count remain in the session, but that marker supplies no arm label. Control draws carrying that kind's settled pins replay those pins, withhold new mints, and are excluded from evidence. Foreign pins leave the kind's control stamp and original bytes intact.

A sufficient cost evaluation has at least minSessionsPerArm sessions in each arm and 30 matched pairs, within the exclusion limit. Promotion requires a Wilson lower bound strictly above winRateFloor, Q3 strictly below maxQ3Ratio, and quality parity (including its existing 200-request-per-arm requirement). Decisions pool the tenant's shape-matched pairs.

Insufficient evidence never changes state. A sufficient loss means Wilson lower bound at or below 0.53 and Q3 at or above 1 (including an unbounded Q3), or quality regression. A treated/control baseline-input ratio above 1.25 on three consecutive fresh sufficient imbalanced windows also becomes a loss, starting with the third window. This catches sustained treated-input inflation, including the AIG-9 failure shape. Control-heavy imbalance below 0.8 does not count. An insufficient tick preserves the imbalance streak; a sufficient window without treated-side imbalance resets it. A loss moves on to shadow. failedEvaluationsBeforeOff consecutive sufficient losses reach off; a sufficient win or neutral evaluation resets the count. Every sufficient verdict advances the evidence cutoff, so each loss needs at least 30 fresh pairs. Insufficient ticks preserve the streak. Automatic off keeps one bounded recovery arm and nothing more: it runs only in the sessions that select it with the treated arm, never in a session that selected another kind and never in the unlabelled baseline. A sufficient win returns off to shadow with reason recovered; a subsequent fresh win may promote. force_off disables recovery sampling. On records retain grandfathered provenance until an actual demotion and later promotion.

evaluatorEnabled: false pauses evaluations and evidence resets. Existing permissions keep applying indefinitely. Changing evaluation cadence does not expire approval. With holdout.fraction: 0, no experimental evidence accrues: existing on stays on and new shadow stays shadow. The separate whole-optimizer holdout enable switch does not disable these slices.

SQL aggregates the paired result on the analytics pool; only summaries reach the gateway. A bounded model/provider lookup is shared across kinds to use canonical pricing. Its limit is 4,096 distinct model/provider combinations per tenant/window; exceeding it retains state and reports an unavailable evaluation. Work is interleaved across tenants with three concurrent queries. Tenant discovery drains 1,000-row indexed pages for a five-second budget per tick, independently of evaluation cadence. Each query spans at most one hour of backlog and has a two-second server timeout. The durable timestamp/row-ID cursor continues next tick; a cancelled transaction retries from its last committed cursor. Known tenants can be evaluated while discovery catches up. A tenant or kind failure does not abort the others. The reporting route can inspect historical windows, including the existing 90-day window.

The gateway's authenticated durable bootstrap also carries content-free permission snapshots. An optimizer that cannot adopt the spend backend applies those snapshots, preserving grandfathered on and new shadow. A metadata-only boot cache, sessionGateOrigins.json under ANYRAY_DATA_DIR, restores known origins and restrictions before serving requests. Without a readable cache or snapshot, unknown tenants withhold byte-changing strategies (session_gate_unavailable). Gate flips and control draws replay settled pins without new mints, confirming durable stash refresh at the owning strategy's TTL before emitting handles. If persistence fails or exceeds the shared request budget, that span goes out at its original bytes and keeps its pin for a later retry. Pin-less prefix rewriters continue only on sessions that kind already treated, excluded from control evidence. Legacy holdout.kinds still uses replay-blocking holdout_kind; gate draws use session_gate_holdout. New gate responses represent non-evidence by omitting the kind; explicit excluded remains accepted only for rolling compatibility with stored and older optimizer rows.

Runtime settings

Set config.sessionGate in the global optimizer settings JSON through PUT /admin/v1/optimizer (legacy /admin/optimizer/settings). No new environment variable is needed. These settings govern the evaluator across tenants.

Field	Default
evaluatorEnabled	true; false pauses evaluations
evaluationIntervalSeconds	3600
windowSeconds	604800
failedEvaluationsBeforeOff	3
minSessionsPerArm	30 (may be raised)
winRateFloor	0.53 (may be raised)
maxQ3Ratio	1 (may be lowered)
maxUnpairedTreatedCostShare	0.25 (may be lowered)
maxExcludedLongSessionShare	0.25 (may be lowered)

The gateway elects one background evaluator with a Postgres advisory lock. Optimizers refresh permission snapshots every 15 seconds without request-path database reads. Byte-changing strategies use the local boot cache while shared or authenticated bootstrap snapshots refresh. Unknown tenants remain at parity before the first snapshot. Durable approvals do not expire; explicit overrides remain until cleared. Disabling evaluation pauses state changes while existing gate permissions keep applying.

Request headers

Beyond Authorization / Content-Type, the gateway reads a few x-anyray-* headers:

x-anyray-providerstring

Provider to route to: openai, anthropic, vertex-ai, bedrock, azure-openai, … Optional when a default provider or routing config is set server-side.

x-anyray-metadatastring (JSON)

Attribution, e.g. {"user":"alice","team":"platform"}. Drives spend attribution and per-user caps. Never carries prompt/response content.

x-anyray-configstring (JSON)

Per-request routing: fallback list, load-balance weights, retry policy. No credentials; they resolve from the server-held key store.

Response headers

Every response carries x-anyray-provider (the provider the request was routed to) and x-anyray-trace-id. When the provider returns a server error (any 5xx, such as an Anthropic 529 overloaded_error), the gateway also stamps the markers below. The status and the error type are forwarded unchanged; only the human-readable error.message gains a sentence naming the provider that produced it:

Relayed Anthropic 529

{
  "type": "error",
  "error": {
    "type": "overloaded_error",
    "message": "Overloaded — this error was returned by Anthropic (api.anthropic.com) and relayed unchanged by the Anyray gateway; it is not an Anyray gateway failure. Check https://status.claude.ai/ for provider status."
  }
}

x-anyray-error-sourcestring

Present and set to upstream when the error originated at the provider, not the gateway. Absent on success and on gateway-generated errors.

x-anyray-upstream-providerstring

The upstream provider that returned the error: anthropic, openai, vertex-ai, … (omitted when the provider is unknown).

x-anyray-upstream-statusstring

The provider's own HTTP status (e.g. 529, 503).

Telling provider errors from gateway errors

No x-anyray-error-source: upstream header on a 5xx means the gateway itself failed, so check your deployment. With it, the fault is on the provider's side (capacity/overload); retrying shortly usually clears it.

Endpoints

/v1/* serves an OpenAI-compatible inference API plus the Anthropic-native Messages API; /admin/* is the admin-gated configuration and reporting API; /sso/cli/* is the pre-key self-service login handshake used by Connect; /scim/v2/* is an inbound SCIM 2.0 server. The Endpoint reference is the complete, capability-annotated index of every route; full request/response schemas live in the OpenAPI spec. The optimizer's own before/after-request hooks are the separate optimizer protocol.

Retrieval via MCP: POST /mcp

The gateway is a remote MCP server (streamable HTTP). It serves the retrieval pair, anyray_retrieve and anyray_recall, behind the same client key as every other route, so an MCP-capable agent runtime closes the retrieval loop with a single server entry and no code:

Any MCP client registration

{
  "type": "http",
  "url": "https://<your-anyray-gateway>/mcp",
  "headers": { "x-anyray-api-key": "<client key>" }
}

The path is /mcp, not /v1/mcp. /v1/* is the inference proxy, and it catches every path it does not recognize, so /v1/mcp is sent upstream as an inference request and comes back 500 rather than 404.

Registering the server puts the two tools in every request the client sends. The gateway forwards them to the model when the optimizer first reports a stored handle or a tool result carries a retrieval marker, and on every request after it in the same gateway process. This includes JSON compression markers and cropped user messages; exposure does not wait for another handshake. Until then the model sees the same tool list it would without the server: the two definitions on their own changed how an agent worked a lint task, before anything had been shortened, and a session that never shortens an output has no use for them. The lane still counts as retrieve-capable from the first request, so the reversible strategies run as before.

Hosted connectors (Anthropic mcp_servers, OpenAI Responses type:"mcp") may point here and are credited as retrieve evidence when they target this gateway's own /mcp. First-party provider APIs only (not Bedrock/Vertex), and the endpoint must be reachable from the provider's cloud.

One JSON-RPC 2.0 message per POST (initialize, ping, tools/list, tools/call); notifications answer 202 with no body; batch arrays are rejected; GET /mcp is 405 (no server-initiated stream). initialize returns a Mcp-Session-Id; every call still authenticates independently. Successful initialization and other valid messages renew the presenting key's 6-hour retrieval lease, the same evidence the stdio server's heartbeat provides. Rejected initialization does not claim retrieval capability. Tool-level failures come back in-band (isError: true with "unknown or expired handle" or "retrieval unavailable"), never as protocol errors, so the model can read them and adapt.

adaptiveMcpInstructions defaults to true. In this mode initialize always omits the instructions key. Recovery guidance lives in the retrieval tool descriptions, which reach the model with the first elision. There is no live-handle snapshot, expiry window, or tenant cap controlling the announcement.

A new client process, resumed conversation, replica handoff, or client discarding its MCP session ID gets the same announcement while the setting stays unchanged. The signed Mcp-Session-Id contains only a decision and random nonce; it binds to the authenticated identity and grants no access. Reuse it to keep the decision even through a settings change. Invalid or foreign IDs return 404 on initialization. Admin-token rotation also invalidates IDs; retrying without the ID preserves the announcement under the same setting.

Set adaptiveMcpInstructions to false through PUT /admin/v1/settings to restore the exact legacy instruction text on new MCP sessions; peer settings refresh within 30 seconds. This is an explicit prefix change for a continuing conversation whose client does not retain its session ID. Apply the switch between conversations. If a cold replica cannot read the setting, initialization without a valid ID returns 503 with Retry-After: 30, instead of guessing. Valid signed sessions and retrieval calls remain usable; failed settings refreshes retain the last known value. Older gateway versions always announce instructions. Reconnecting a warm conversation across mixed gateway versions during an upgrade can therefore change its prefix.

GET /admin/v1/health reports process-local mcpInstructions handshake counters by the closed reasons adaptive_omitted and runtime_disabled. These counters carry no identity or content.

Retrieval: POST /connect/retrieve

An agent you build yourself has to be given a read path, or optimization runs one-way. Declare a tool named anyray_retrieve and implement it against this route with the same client key you already send. These are three independent requirements:

• The declaration proves the model can select the tool on this turn. It matches on the tool name (anyray_retrieve, or a namespaced mcp__…__anyray_retrieve), so renaming the tool silently turns retrieval off.
• The implementation is what resolves a handle once the model calls it.
• The liveness signal proves that implementation is running. Send an authenticated empty POST /connect/mcp-heartbeat at worker start and every 5 minutes while it can execute tools.

Without both a callable path and a fresh live lease, the gateway keeps clientCanRetrieve false. Retrieval-dependent strategies stand down, and degradable strategies keep their handle-free mode.

Resolve a ctx_ handle

curl -X POST https://<your-anyray-gateway>/connect/retrieve \
  -H "x-anyray-api-key: $ANYRAY_CLIENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"handle":"ctx_abc123"}'

handlestringrequired

The ctx_… handle from an [anyray: … retrieve ctx_…] marker in the message the model received.

startLine / endLinenumber

Read back a slice instead of the whole original.

grepstring

Return matching lines with context. Max 256 characters.

Answers {"status":"ok","content":"…","handle":"ctx_…"}. An unknown or expired handle is 404, an unavailable retrieval store is 503, and a missing or invalid key is 401. This is the route the /mcp endpoint's anyray_retrieve tool calls into; implement against it directly only when the runtime cannot register an MCP server. POST /connect/recall takes a natural-language query instead of a handle and returns ranked handle + score + preview matches, for when the marker has scrolled out of the model's view.

A human or CLI caller that is only probing sends x-anyray-retrieve-probe: 1 so the call does not latch capability; only the model's own calls prove it can pull spans back. Worked examples per SDK: Retrieval reference.

Claude Desktop managed bootstrap

An organization that routes Claude Desktop through Anyray manages its third-party profile through the deployment-owner API:

Set the organization policy

curl -X PUT https://<your-anyray-gateway>/admin/claude-desktop-policy \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "settings": {
      "inferenceModels": [{"name": "claude-sonnet-4-6"}],
      "managedMcpServers": {},
      "allowedPluginMarketplaces": [],
      "chatTabEnabled": true,
      "coworkEgressAllowedHosts": ["*"]
    }
  }'

GET /connect/claude-desktop/bootstrap?os=macos|windows|linux returns the effective secret-free policy, with the gateway provider/base URL, helper credential mode, and that OS's inferenceCredentialHelper path injected by the server (os defaults to macos). Current Claude Desktop sends Authorization: Bearer <client key> on this fetch through the MDM-pinned bootstrapHeadersHelper; the route verifies a presented bearer (401 when rejected) and still serves an unauthenticated fetch in the no-OIDC lane. It returns 404 while disabled. The write boundary rejects credential-shaped string values at every depth, including values placed under otherwise allowed free-text keys. The authenticated admin routes remain available in headless gateway deployments. GET /admin/claude-desktop-policy/mobileconfig downloads the MDM trust profile, which pins that endpoint and /usr/local/bin/anyray-bootstrap-headers-helper without embedding a key; the response names /usr/local/bin/anyray-credential-helper. POST /admin/claude-desktop-policy/connectors/probe takes {"url": "https://..."} and asks that MCP server for its own name and tool list, so a connector row can be filled in from its address: {"status": "ok", "name", "version", "tools": [...], "truncated"} when it answers openly, {"status": "needs-auth", ...} when it wants a sign-in first, and {"status": "unreachable", "reason"} otherwise. HTTPS only, and nothing is stored.

Import and fleet deployment: Desktop apps.

GET /admin/claude-desktop-policy/connectors/readiness returns the most-blocking organization sign-in state and the /mcp/org endpoint URL. Its v1 twin is GET /admin/v1/policies/claude-desktop/connectors/readiness (policies:read). The read never probes connector vendors. POST /admin/v1/policies/claude-desktop/connectors/readiness/refresh (policies:write) refreshes the process-local access cache with at most four probes in flight. OAuth adoption is an explicit count, fewerThan, or unavailable union; counts of one and two are returned only as fewerThan: 3.

POST /admin/claude-desktop-policy/connectors/authorize takes {"name": "Atlassian"} and returns a vendor authorizationUrl. After consent, the callback page shows a short claim code. Redeem it with POST /connect/mcp-oauth/complete and a client key. The grant belongs to the verified account that redeems the code, never the account that started the flow. Developers start their own flow with POST /connect/mcp-oauth/authorize. Neither response contains a token.

Not every endpoint runs on every provider

Chat completions, embeddings, and the Anthropic Messages API work across all supported providers. Image, audio, Responses, file, and Batch endpoints are OpenAI / Azure-only passthrough (audio also works on groq).

SCIM v2 provisioning

Point the IdP at this base URL and send the separately configured SCIM bearer on every request:

SCIM base URL

https://<your-anyray-gateway>/scim/v2

Resource	Routes
Discovery	GET /ServiceProviderConfig, GET /ResourceTypes, GET /Schemas
Users	POST /Users, GET /Users, GET /Users/:id, PUT /Users/:id, PATCH /Users/:id, DELETE /Users/:id
Groups	POST /Groups, GET /Groups, GET /Groups/:id, PUT /Groups/:id, PATCH /Groups/:id, DELETE /Groups/:id

User lists support filter=userName eq "dev@example.com", startIndex, and count. Group lists support the same paging fields. User PATCH accepts active:false for immediate offboarding; Group PATCH accepts add/remove/replace member operations. A missing group member can be created when the member carries a verified email in display or value.

Configure the bearer, admin group, and team mappings through the deployment-owner admin lane:

Configure SCIM

curl -X PUT https://<your-anyray-gateway>/admin/scim/settings \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "bearerToken": "synthetic-scim-bearer-token-replace-me-0001",
    "adminGroup": "Anyray Admins",
    "groupTeamMap": { "Engineering": "platform" }
  }'

GET /admin/scim/settings returns configured, adminGroup, groupTeamMap, and updatedAt. It never returns the bearer or bearer hash. A later PUT can omit bearerToken to retain the current credential.

SCIM errors always use application/scim+json and this envelope:

SCIM error

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:Error"],
  "detail": "invalid bearer token",
  "status": "401"
}

Errors

Most inference errors use the OpenAI envelope ({ "error": { "message", "type", "code" } }). Client-key authentication uses the fixed machine envelope shown below.

Status	Meaning
400	Malformed request: failed validation, bad JSON, or an unknown provider/model.
401	Missing, malformed, unknown, expired, revoked, or SCIM-deactivated client identity. The body includes status: failure, a reason code, the fixed valid client key required message, and a safe hint.
402	A governance ceiling has been reached: deployment entitlement/spend cap, a service key's monthly dollar budget, or a per-user monthly token cap / USD budget. The body carries error.code, one of service_key_budget_exceeded, user_token_cap_exceeded, or user_budget_exceeded. Terminal, not retryable: retrying cannot clear it, only an admin raising the limit or the window rolling.
429	Rate limit exceeded. Honors Retry-After.
503	A dependency the gateway needs is temporarily unreachable. Client-key verification returns status: failure with code: key_store_unavailable when the gateway cannot reach the store holding minted keys. This is an infrastructure fault, not a credential problem, so re-running anyray-connect will not help. Verification fails closed, so the request is never forwarded upstream. Retry shortly; if it persists, an operator should check GET /admin/health for the deployment's database legs.

Source trim evidence

The admin key list/detail and GET /v1/me key object include sourceTrimSince, sourceTrimRequestCount, sourceTrimLastRequestAt, and sourceTrimAt. All timestamps are UTC ISO strings. The count covers successful inference requests from source-capable clients since observation began or the last recorded source trim. A successful hook spend event resets the count and start stamp. Missing count means an older gateway, not zero events. Migration 0097_client_keys_source_trim adds the metadata columns without backfilling historical traffic.

Hook contract, supported clients and grace period.