Gateway

An OpenAI-compatible middleware that sits in front of every LLM provider.

Intro

The gateway is a request-path component that sits in front of LLM providers and routes, governs, and meters every request. Provider keys stay server-side and are injected at request time.

Request lifecycle

Every /v1/* request runs an ordered middleware chain (each gate fails fast), hits a handler that calls the provider, and is metered after the response.

1. Edge hardening — securityHeaders, decompressRequest, bodySizeLimit, and authFailureGuard reject malformed, oversized, or abusive requests before real work.
2. Identity & entitlement — clientKeyAuth validates the minted key and binds its user/team; the entitlement kill-switch rejects suspended/expired tenants with 402/403.
3. Admission control — rateLimit/ipRateLimit, concurrencyLimit, guardrail hooks, then enforceUserCaps (429 once over the monthly token budget). Each self-gates.
4. Cache & validate — optional memoryCache lookup, then the Zod requestValidator.
5. Handler — selects the route/provider and calls the optimizer adapter applyOptimization (POST /v1/optimize). On a semantic_cache hit it serves the cached response; otherwise it translates the request to the provider's wire format and forwards it.
6. Meter — logHandler writes one SpendRecord and exports the trace.

Fallback & load-balancing

A request can name a list of targets instead of one provider:

Mode	What it does
fallback	Tries targets in order; on a configured status (onStatusCodes — e.g. 429, 500, 503) it advances to the next target on the same request.
loadbalance	Distributes requests across targets by weight.

Each target carries its own retry policy (attempts + status codes, honoring Retry-After), so a transient blip is retried before fallback advances. Per-target credentials come from the server-held key store at request time — the config never carries secrets.

Adapter for external gateways

Adapter to other gateways. Anyray receives the requests and runs its full pipeline (validate → attribute → optimizer → encrypt → meter → entitlement), then forwards to your gateway as an OpenAI-compatible upstream — the anyray-upstream provider. Every privacy and governance guarantee holds because Anyray stays on the path.

provider config

{
  "provider": "anyray-upstream",
  "custom_host": "https://litellm.acme.com", // required — your gateway's base URL
  "api_key": "sk-acme-litellm-..."           // optional — your gateway's own key
}

Knob	Purpose
custom_host (required)	Your gateway's base URL; Anyray appends /v1/chat/completions etc. Config without it fails validation.
Custom headers	Extra headers merged onto the upstream request — for a non-Bearer auth scheme (api-key, a virtual key) or org/routing headers a plain Authorization: Bearer can't express. Set them on the Custom (OpenAI-compatible) provider in the console (a JSON object); an authorization entry overrides the API-key Bearer.
ANYRAY_CUSTOM_HOST_ALLOWLIST	On a public Anyray gateway, custom_host is rejected unless its host matches this CSV (open-relay guard).
TRUSTED_CUSTOM_HOSTS	The only knob that lets an internal host (private IP / .internal) past the SSRF block.

Routing to OpenRouter or Nebius

OpenRouter and Nebius (Nebius AI Studio) are first-class providers — set "provider": "openrouter" or "provider": "nebius" (or save that key in the console) with just the provider key. Their base URLs and endpoints are built in, so you don't need custom_host or ANYRAY_CUSTOM_HOST_ALLOWLIST. Reserve anyray-upstream for arbitrary or internal OpenAI-compatible gateways (LiteLLM, a homegrown proxy).

Attach to LiteLLM

Prerequisites

• Docker Engine (v24+) and Docker Compose v2 on the Anyray host
• Python 3.9+ and httpx in the LiteLLM environment
• Ports:
  • 3000 for the Anyray console, reachable from your org network / VPN
  • 8088 for the optimizer, reachable only by LiteLLM
• git and openssl available on the Anyray host

Install

1

Clone the install repo

git clone https://github.com/anyrayHQ/install anyray && cd anyray

2

Generate secrets

./setup.sh --host <anyray-host>

setup.sh writes .env, including the admin key, optimizer token, the trace store DB URL (ANYRAY_OBSERVABILITY_DB_URL), and content-encryption key.

3

Start attach mode

docker compose -f docker-compose.attach.yml up -d

By default the optimizer binds to 127.0.0.1:8088, which is correct when LiteLLM runs on the same host. If LiteLLM runs on another machine, set ANYRAY_OPTIMIZER_BIND=0.0.0.0 in .env, keep :8088 reachable only over a private network or VPN, then restart the stack.

4

Configure LiteLLM

Copy attach/litellm/anyray_optimizer.py next to your LiteLLM config.yaml, then add:

litellm_settings:
  callbacks:
    - anyray_optimizer.proxy_handler_instance

Set these environment variables in the LiteLLM process:

ANYRAY_OPTIMIZER_URL=http://<anyray-host>:8088
ANYRAY_OPTIMIZER_TOKEN=<ANYRAY_OPTIMIZER_TOKEN from .env>

5

Verify

docker compose -f docker-compose.attach.yml ps
curl -fs http://localhost:8088/health && echo "optimizer ok"
curl -fso /dev/null http://localhost:3000/anyray-login && echo "console ok"

Open http://<anyray-host>:3000 and sign in with the admin key setup.sh printed (also in .env as ANYRAY_ADMIN_TOKEN).

Self-service SSO key minting

anyray-connect login uses a separate pre-key handshake because inference endpoints already require a valid personal key. The gateway creates a ten-minute session in shared Postgres, returns a human code plus a separate high-entropy poll secret, and asks the deployment-authenticated Billing app to open WorkOS SSO. The browser confirms the code before redirecting to the IdP.

After WorkOS verifies the identity, the Billing app JIT-provisions the user membership and returns only governed metadata: user, eligible teams, mapped/default role, and key lifetime. The terminal chooses one eligible team. The gateway atomically claims that team, mints the renewable personal key, and returns the raw key once. User codes and poll secrets are stored only as salted hashes; the raw ark_… key is never persisted in the login-session table.

Spend and metering

After each call, logHandler writes one SpendRecord: endpoint, status, provider, model, prompt/completion/total tokens, cache read/write tokens, latency, cost, optimizer savingsUsd/tokensSaved, billing mode, user/team attribution (from x-anyray-metadata), and the optimizer hook outcome — optimizationStatus (applied / skipped / disabled / timeout / error), a normalized optimizationReason token, and optimizerLatencyMs — so the unoptimized rate and its cause are queryable straight from the spend store.

Records land in an in-memory ring plus monthly per-user counters (which drive caps), and a durable Postgres store (ANYRAY_SPEND_DB_URL, the anyray_spend table, auto-created).

The same shared database holds inbound SCIM identity state. IdPs call /scim/v2/* with a dedicated static bearer; the gateway stores only its hash plus user/group metadata. Client-key verification reads the user's active state on every successful key lookup, so a deactivation on one replica blocks the key on all replicas. That access read fails closed if Postgres is unavailable.

The async Batch API is attributed too: GET /v1/batches/<id>/output returns JSONL with one usage block per line, so the gateway writes one row per line (model, tokens, cost, status) instead of a single zero-token row. Re-fetching the same output is idempotent — a batch already attributed in-process isn't counted twice.

Health & support diagnostics

Two admin-gated endpoints make a self-hosted deployment debuggable from the outside: get /admin/health probes every leg live and names the failing one, and get /admin/support/bundle assembles a one-shot diagnostic snapshot (versions, health, redacted config, optimizer strategy config, request/error telemetry) an operator can send to Anyray support. See Troubleshooting & support bundles.

The full API surface (routes) and key environment variables are in the API reference.