Skip to main content

Gateway

An OpenAI-compatible middleware that sits in front of every LLM provider.

The gateway is the request-path component: it routes, governs, and meters every call. Provider keys stay server-side and are injected at request time.

Request lifecycle​

Every /v1/* request runs an ordered middleware chain (each gate fails fast), hits a handler that calls the provider, and is metered after the response.

  1. Edge hardening. securityHeaders, decompressRequest, bodySizeLimit, and authFailureGuard reject malformed, oversized, or abusive requests before real work.
  2. Identity & entitlement. clientKeyAuth validates the minted key and binds its user/team; the entitlement kill-switch rejects suspended/expired tenants with 402/403.
  3. Admission control. rateLimit/ipRateLimit, concurrencyLimit, guardrail hooks, then enforceUserCaps (402 once over the monthly token cap or USD budget).
  4. Cache & validate. Optional memoryCache lookup, then the Zod requestValidator.
  5. Handler. Selects the route/provider and calls the optimizer adapter (POST /v1/optimize). On a semantic_cache hit it serves the cached response; otherwise it translates the request to the provider's wire format and forwards it.
  6. Meter. logHandler writes one SpendRecord and exports the trace.

Fallback & load-balancing​

A request can name a list of targets instead of one provider:

ModeWhat it does
fallbackTries targets in order; on a configured status (onStatusCodes, e.g. 429, 500, 503) it advances to the next target on the same request.
loadbalanceDistributes requests across targets by weight, sticky per conversation (below).

Each target carries its own retry policy (attempts + status codes, honoring Retry-After), so a transient blip is retried before fallback advances. Per-target credentials come from the server-held key store at request time; the config never carries secrets.

Load balancing is sticky per conversation: every turn of one conversation goes to the same target, and the weighted split happens across different conversations. That is a prompt-cache requirement, not a preference. Each upstream account keeps its own cache, so a conversation that moved targets mid-flight would re-write its whole prefix on the new account, at roughly 12x the cost of a cache read. Traffic with no derivable conversation (a one-shot API call) is spread per request, and changing a weight, or adding a target, re-deals live conversations once.

Adapter for external gateways​

Already running LiteLLM, TrueFoundry, or a homegrown OpenAI-compatible proxy? Anyray receives the request, runs its full pipeline (validate → attribute → optimize → encrypt → meter → entitlement), then forwards to your gateway as an OpenAI-compatible upstream, the anyray-upstream provider. Every privacy and governance guarantee holds because Anyray stays on the path.

provider config
{
  "provider": "anyray-upstream",
  "custom_host": "https://litellm.acme.com", // required: your gateway's base URL
  "api_key": "sk-acme-litellm-..."           // optional: omit it for a keyless gateway
}

The full knob table (credentials, custom headers, the ANYRAY_CUSTOM_HOST_ALLOWLIST open-relay guard, gateways that serve inference on their own path), the dedicated provider ids for LiteLLM, TrueFoundry, OpenRouter, Nebius, and Oracle, and the optimizer-only Attach to LiteLLM install are in the gateway reference.

Spend and metering​

After each call the gateway writes one spend row. It holds metadata only, never the prompt or the reply.

The row recordsFields
The callEndpoint, status, provider, model, latency
The tokensPrompt, completion, total, and the cache read and write split
The moneyCost, plus the optimizer's savingsUsd and tokensSaved, and the billing mode
Who ran itUser and team, from the x-anyray-metadata header
What the optimizer didoptimizationStatus (applied, skipped, disabled, timeout, or error), optimizationReason, and optimizerLatencyMs

That last group is why you can query the unoptimized rate and its cause.

Each row lands in three places: an in-memory ring for the live view, the monthly per-user counters that drive caps, and Postgres (ANYRAY_SPEND_DB_URL, in the auto-created anyray_spend table).

reasoningEffort records the level the request was sent with, after any optimizer downshift. The levels are the union of what providers accept, not one ranked scale, so they only compare inside one provider family. A batch is attributed per line, never as one zero-token row. How each field is read: gateway reference.

The same database holds SCIM identity state

IdPs call /scim/v2/* with a static bearer, and the gateway stores only its hash plus user and group metadata. Every successful key lookup re-reads the user's active state, so deactivating someone on one replica blocks their key everywhere. That read fails closed if Postgres is down.

Health & support diagnostics​

Two admin-gated endpoints make a self-hosted deployment debuggable from the outside: get /admin/health probes every leg live and names the failing one, and get /admin/support/bundle assembles a one-shot diagnostic snapshot (versions, health, redacted config, optimizer strategy config, request/error telemetry) an operator can send to Anyray support. See Troubleshooting & support bundles; the full route surface and key environment variables are in the API reference.