Gateway reference

Configuration detail behind the adapter for external gateways: the anyray-upstream knobs, path resolution, the dedicated provider ids, and the optimizer-only attach install.

anyray-upstream knobs

Knob	Purpose
api_key (optional)	Your gateway's own credential, forwarded as Authorization: Bearer. Leave it unset for an upstream that needs no auth and the request goes out with no Authorization header at all; Anyray never substitutes the caller's Anyray client key (ark_…), which would leak it upstream. A caller's own non-Anyray key is still forwarded (BYOK).
custom_host (required)	Your gateway's base URL; Anyray appends /v1/chat/completions etc. Config without it fails validation. If your gateway serves inference somewhere else, set the full endpoint URL here instead and Anyray sends the request there unchanged (see below).
Custom headers	Extra headers merged onto the upstream request, for a non-Bearer auth scheme (api-key, a virtual key) or org/routing headers a plain Authorization: Bearer can't express. Set them on the Custom (OpenAI-compatible) provider in the console (a JSON object); an authorization entry overrides the API-key Bearer.
ANYRAY_CUSTOM_HOST_ALLOWLIST	On a public Anyray gateway, custom_host is rejected unless its host matches this CSV (open-relay guard). The subscription-seat backends Connect pins itself (chatgpt.com) are built in, so ChatGPT seats work with this unset.
TRUSTED_CUSTOM_HOSTS	Deprecated. Lets a caller-supplied internal host past the SSRF block. An upstream you configure in the console no longer needs it.

Inference on a different path

Not every OpenAI-compatible gateway exposes chat at /v1/chat/completions. If yours doesn't, put the complete endpoint URL in custom_host and Anyray uses it as-is rather than appending a path of its own:

a gateway with its own inference path

{
  "provider": "anyray-upstream",
  "custom_host": "https://gateway.acme.com/internal/infer/chat/completions"
}

Anyray treats the host as the endpoint itself when its path ends in a recognised inference suffix (/chat/completions, /messages, /completions, /embeddings). Anything else is a base URL and keeps the usual appended path, so a prefix like TrueFoundry's /api/llm still resolves to /api/llm/v1/chat/completions. A trailing /v1 or / is ignored either way. This applies to anyray-upstream, litellm, and truefoundry alike.

The suffix has to match the operation being served, so an endpoint URL covers only the operation it names. Point custom_host at .../chat/completions and an embeddings call on the same deployment still resolves to its own /v1/embeddings path. The one exception is Claude's count_tokens, which Anthropic serves directly off the messages endpoint: a host ending in /messages gets /count_tokens appended to it rather than a second full path.

Claude requests are worth one note. anyray-upstream translates them into OpenAI chat and posts them to its chat endpoint, so a host ending in /chat/completions serves your Claude traffic too. litellm and truefoundry forward Claude natively, so give those a host ending in /messages if that is where your gateway serves it.

Required body fields

Some gateways validate the request body against a schema of their own and reject anything that omits a field they require, such as organization_id or a trace name.

Send those fields exactly as you do today and Anyray forwards them. A field Anyray does not recognise is passed through to your gateway untouched, so a client that worked before Anyray was in the path keeps working — on /v1/chat/completions and on Claude's /v1/messages alike:

curl https://anyray.acme.com/v1/messages \
  -d '{"model":"…","max_tokens":1024,"messages":[…],
       "organization_id":"acme","trace_name":"nightly-eval"}'

Your callers keep control of the values, per request, with no gateway-side setup.

For a client that cannot add fields to its own request body, an operator can fill Required body fields on the provider in the console with a JSON object, applied to every request to that upstream:

{ "organization_id": "acme" }

A field in the request always wins over the configured one.

Two limits apply:

• Only the anyray-upstream, litellm, and truefoundry providers. Your gateway's schema is yours; a hosted vendor's is not, and an unknown field there would be rejected.
• model, messages, max_tokens, stream, tools, tool_choice, n, and best_of are never settable through the console setting. On Claude's translated lane n and best_of are dropped from the request body too: they bill for completions the Anthropic response format cannot return.

LiteLLM and TrueFoundry

Both have their own provider id: set "provider": "litellm" or "provider": "truefoundry" with the same custom_host and optional key as above. They are reached at a per-deployment base URL, so the host is still required and still subject to the allowlist. TrueFoundry's host keeps its /api/llm prefix; a trailing /v1 on either is ignored.

Use them in preference to anyray-upstream. Both serve the Anthropic Messages spec on their own /v1/messages, and these provider ids forward Claude requests there in their native wire format. The generic upstream instead translates them into OpenAI chat, which carries no cache_control breakpoint and drops thinking blocks, so a warm Claude session loses its prompt cache. Non-Claude traffic keeps using /v1/chat/completions either way. Details: LiteLLM, TrueFoundry.

OpenRouter, Nebius and Tensormesh

OpenRouter, Nebius (Nebius AI Studio) and Tensormesh (Serverless Inference) are first-class providers. Set "provider": "openrouter", "nebius" or "tensormesh" (or save that key in the console) with just the provider key. Their base URLs and endpoints are built in, so you don't need custom_host or ANYRAY_CUSTOM_HOST_ALLOWLIST. Reserve anyray-upstream for arbitrary or internal OpenAI-compatible gateways (a homegrown proxy, or a vendor gateway with no dedicated provider id).

Oracle (OCI Generative AI)

Oracle Cloud Infrastructure Generative AI is a first-class provider. Set "provider": "oracle" (or save the credentials in the console) and its region-scoped base URL is built in, so you don't need custom_host. Models keep OCI's dotted vendor prefix on the wire: xai.grok-4.3, meta.llama-4-scout-17b-16e-instruct, openai.gpt-oss-120b, cohere.command-a-03-2025.

Cost reporting on the Meta and Cohere families

Anyray ships list prices for OCI's xAI and OpenAI models, so their spend and savings are exact. The Meta and Cohere models route and meter normally, but Oracle bills them per character rather than per token, and publishes no characters-per-token ratio, so their cost is reported at a fallback rate rather than Oracle's. Read the spend figures for those two families as indicative, and take Oracle's own invoice as authoritative. Each such row carries a priceSource recording that it was priced at a fallback, and the per-model breakdown on GET /admin/spend/dashboard counts them, so you can measure exactly how much of a bill rests on an estimated rate.

oracleRegion is required and has no default: the host is region-scoped and a Generative AI API key only works in the region it was created in, so guessing one can only produce an upstream 401 that reads like a bad key. The gateway rejects the request and names the missing field instead. Beyond that, Oracle accepts two credentials and the gateway picks the mode from whichever is present:

• Generative AI API key. Set apiKey, which rides as a plain Bearer token. Oracle scopes this to testing and early development.
• IAM request signing. Oracle's production path, and the one that works with an existing tenancy's API key pair. Set oracleTenancy, oracleUser, oracleFingerprint and oraclePrivateKey; each request is then signed rather than carrying a key at all. Signing is offered on the chat paths only: a signature binds your tenancy's identity to a specific request target, so the gateway will not sign one a caller supplied.

Setting only some of the four signing fields is rejected, so a half-configured tenancy fails loudly instead of quietly routing production traffic on the weaker credential. Nor is an Anyray client key ever forwarded to Oracle as a fallback bearer token: an unconfigured tenancy fails closed rather than leaking that key upstream. Full field list and key-format requirements: Choose your setup.

Attach to LiteLLM

Keeps LiteLLM as the front door and runs only the optimizer beside it. The recommended topology is the gateway in front of LiteLLM, with per-developer enrollment and the full optimization surface: see Integrations → LiteLLM.

Prerequisites: Docker Engine (v24+) and Docker Compose v2, git, and openssl on the Anyray host; Python 3.9+ and httpx in the LiteLLM environment; port 3000 (the Anyray console) reachable from your org network / VPN and port 8088 (the optimizer) reachable only by LiteLLM.

1

Clone the install repo

git clone https://github.com/anyrayHQ/install anyray && cd anyray

2

Generate secrets

./setup.sh --host <anyray-host>

setup.sh writes .env, including the admin key, optimizer token, the trace store DB URL (ANYRAY_OBSERVABILITY_DB_URL), and content-encryption key.

3

Start attach mode

docker compose -f docker-compose.attach.yml up -d

By default the optimizer binds to 127.0.0.1:8088, which is correct when LiteLLM runs on the same host. If LiteLLM runs on another machine, set ANYRAY_OPTIMIZER_BIND=0.0.0.0 in .env, keep :8088 reachable only over a private network or VPN, then restart the stack.

4

Configure LiteLLM

Copy attach/litellm/anyray_optimizer.py next to your LiteLLM config.yaml, then add:

litellm_settings:
  callbacks:
    - anyray_optimizer.proxy_handler_instance

Set these environment variables in the LiteLLM process:

ANYRAY_OPTIMIZER_URL=http://<anyray-host>:8088
ANYRAY_OPTIMIZER_TOKEN=<ANYRAY_OPTIMIZER_TOKEN from .env>

5

Verify

docker compose -f docker-compose.attach.yml ps
curl -fs http://localhost:8088/health && echo "optimizer ok"
curl -fso /dev/null http://localhost:3000/anyray-login && echo "console ok"

Open http://<anyray-host>:3000 and sign in with the admin key setup.sh printed (also in .env as ANYRAY_ADMIN_TOKEN).