Gateway

An OpenAI-compatible middleware that sits in front of every LLM provider.

It routes, governs, and meters every call. Provider keys stay server-side and are injected at request time.

Request lifecycle

Every /v1/* request runs an ordered middleware chain (each gate fails fast), then a handler that calls the provider, then metering.

Stage	Middleware
Edge hardening	securityHeaders, decompressRequest, bodySizeLimit, authFailureGuard
Identity and entitlement	clientKeyAuth binds the key's user and team; entitlement rejects suspended or expired tenants with 402/403
Admission	rateLimit/ipRateLimit, concurrencyLimit, guardrail hooks, enforceUserCaps (402 over the monthly token cap or USD budget)
Cache and validate	Optional memoryCache, then the Zod requestValidator
Handler	Picks the route and provider, calls POST /v1/optimize, serves a semantic_cache hit or translates and forwards to the provider
Meter	logHandler writes one SpendRecord and exports the trace

Fallback & load-balancing

A request can name a list of targets instead of one provider:

Mode	Behavior
fallback	Tries targets in order, advancing on a status in onStatusCodes (e.g. 429, 500, 503) within the same request.
loadbalance	Splits by weight, sticky per conversation.

Each target has its own retry policy (attempts plus status codes, honoring Retry-After), so a blip is retried before fallback advances. Credentials come from the server-held key store; the config carries no secrets.

Adapter for external gateways

Already running LiteLLM, TrueFoundry, or an OpenAI-compatible proxy? Anyray runs its full pipeline, then forwards to it as the anyray-upstream provider.

provider config

{
  "provider": "anyray-upstream",
  "custom_host": "https://litellm.acme.com", // required: your gateway's base URL
  "api_key": "sk-acme-litellm-..."           // optional: omit it for a keyless gateway
}

Credentials, custom headers, the ANYRAY_CUSTOM_HOST_ALLOWLIST open-relay guard, custom inference paths, dedicated provider ids (LiteLLM, TrueFoundry, OpenRouter, Nebius, Oracle), and the optimizer-only Attach to LiteLLM install: gateway reference.

Spend and metering

Each call writes one spend row, metadata only.

Group	Fields
Call	Endpoint, status, provider, model, latency
Tokens	Prompt, completion, total, cache read and write
Money	Cost, savingsUsd, tokensSaved, billing mode
Who	User and team, from the x-anyray-metadata header
Optimizer	optimizationStatus (applied, skipped, disabled, timeout, error), optimizationReason, optimizerLatencyMs
Prompt cache	Lifetime asked and sent (5m, 1h, mixed) and what changed it, what reads saved and writes cost above fresh input, the optimizer's cache rebuild cost and mint gate verdicts. Visibility only

Rows land in an in-memory ring (live view), the monthly per-user counters (caps), and Postgres (ANYRAY_SPEND_DB_URL, auto-created anyray_spend table). reasoningEffort and batch attribution: gateway reference.

The same database holds SCIM identity state

IdPs call /scim/v2/* with a static bearer; the gateway stores only its hash plus user and group metadata. Every key lookup re-reads the user's active state, so a deactivation blocks the key on every replica. The read fails closed if Postgres is down.

Health & support diagnostics

Endpoint	Returns
get /admin/health	Probes every leg live and names the failing one
get /admin/support/bundle	Versions, health, redacted config, optimizer strategy config, request/error telemetry, to send to support

Both are admin-gated. See Troubleshooting & support bundles and the API reference.