Gateway reference

Configuration detail behind the adapter for external gateways: the anyray-upstream knobs, path resolution, the dedicated provider ids, and the optimizer-only attach install.

anyray-upstream knobs

Knob	Purpose
api_key (optional)	Your gateway's own credential, forwarded as Authorization: Bearer. Leave it unset for an upstream that needs no auth and the request goes out with no Authorization header at all; Anyray never substitutes the caller's Anyray client key (ark_…), which would leak it upstream. A caller's own non-Anyray key is still forwarded (BYOK).
custom_host (required)	Your gateway's base URL; Anyray appends /v1/chat/completions etc. Config without it fails validation. If your gateway serves inference somewhere else, set the full endpoint URL here instead and Anyray sends the request there unchanged (see below).
custom_endpoints (optional)	Per-endpoint upstream paths, for a gateway whose routes Anyray cannot read off custom_host. Anthropic-shaped keys are messages and messagesCountTokens; OpenAI-shaped are chatComplete, complete, embed. Each value is a path appended to custom_host as written, or "" when custom_host is already that endpoint. Endpoints you leave out keep their default path. Endpoint paths in the console (see below).
Custom headers	Extra headers merged onto the upstream request, for a non-Bearer auth scheme (api-key, a virtual key) or org/routing headers a plain Authorization: Bearer can't express. Set them on the Custom (OpenAI-compatible) provider in the console (a JSON object); an authorization entry overrides the API-key Bearer.
ANYRAY_CUSTOM_HOST_ALLOWLIST	On a public Anyray gateway, custom_host is rejected unless its host matches this CSV (open-relay guard). The subscription-seat backends Connect pins itself (chatgpt.com) are built in, so ChatGPT seats work with this unset.
TRUSTED_CUSTOM_HOSTS	Exact hosts allowed for an intentional private or on-prem organization MCP connector. It also keeps the legacy caller-supplied custom_host exception. Configured inference upstreams no longer need it. Cloud metadata is never allowed.

How reasoningEffort is recorded

A spend row's reasoningEffort is the level the request was sent with, after any optimizer downshift. The gateway reads whichever field names one (reasoning_effort, reasoning.effort, output_config.effort, Gemini's thinkingLevel, and the same fields inside Bedrock's Converse envelope) and stores the value as given: none, minimal, low, medium, high, xhigh, max.

Those are the union of what providers accept, not one ranked scale. minimal is OpenAI-only, max is Anthropic-only, and low is Anthropic's floor but OpenAI's second rung, so a comparison only holds inside one provider family.

A numeric thinking budget records no level, because a budget's meaning depends on that model's own ceiling. Only an explicit off-switch is read from those fields (thinking.type: "disabled", or a budget of 0), which records none. A request that names no effort records nothing, since the provider's own default ran, and that is not the same as none.

A batch is attributed per line. GET /v1/batches/<id>/output returns one usage block per JSONL line, so the gateway writes one row per line rather than a single zero-token row. Fetching the same output twice does not double-count it, and a batch-output row records no effort level.

Inference on a different path

Not every OpenAI-compatible gateway exposes chat at /v1/chat/completions. If yours doesn't, put the complete endpoint URL in custom_host and Anyray uses it as-is rather than appending a path of its own:

a gateway with its own inference path

{
  "provider": "anyray-upstream",
  "custom_host": "https://gateway.acme.com/internal/infer/chat/completions"
}

Anyray treats the host as the endpoint itself when its path ends in a recognised inference suffix (/chat/completions, /messages, /completions, /embeddings). Anything else is a base URL and keeps the usual appended path, so a prefix like TrueFoundry's /api/llm still resolves to /api/llm/v1/chat/completions. A trailing /v1 or / is ignored either way. This applies to anyray-upstream, litellm, and truefoundry alike.

The suffix has to match the operation being served, so an endpoint URL covers only the operation it names. Point custom_host at .../chat/completions and an embeddings call on the same deployment still resolves to its own /v1/embeddings path. The one exception is Claude's count_tokens, which Anthropic serves directly off the messages endpoint: a host ending in /messages gets /count_tokens appended to it rather than a second full path.

Naming the paths yourself

When your gateway serves an endpoint somewhere the suffix rule cannot recognise, or serves two of them at unrelated paths, set custom_endpoints and Anyray stops guessing. Each key is an endpoint, each value a path appended to custom_host exactly as written. LiteLLM mounts its Claude passthrough under a prefix, so one key covers it and the OpenAI lane keeps its usual path:

Claude under a prefix, OpenAI at the root

{
  "provider": "litellm",
  "custom_host": "https://litellm.acme.com",
  "custom_endpoints": {
    "messages": "/anthropic/v1/messages"
  }
}

The keys, by the shape Anyray sends on the wire:

Wire shape	Key	Default path
Anthropic	messages	/v1/messages (/v1/chat/completions on anyray-upstream)
Anthropic	messagesCountTokens	/v1/messages/count_tokens
OpenAI	chatComplete	/v1/chat/completions
OpenAI	complete	/v1/completions
OpenAI	embed	/v1/embeddings

An endpoint you leave out keeps its default path, so name only the ones that differ. A value of "" means custom_host is already that endpoint and nothing is appended.

Like custom_host, it can sit on a routing group and applies to every target under it. A target that names its own paths replaces the group's map rather than merging with it.

Values are paths on custom_host, not URLs, and are appended exactly as written. A scheme, a //host, a backslash, a .. segment, or any percent-encoding is rejected: gateways that decode before resolving a path (nginx, and so LiteLLM) would let %2e%2e climb out of a prefix you set in custom_host.

Claude traffic always follows the messages key, whichever provider you are on. What differs is its default: litellm and truefoundry forward Claude natively, so messages defaults to /v1/messages, while anyray-upstream translates it into OpenAI chat and defaults messages to /v1/chat/completions. Setting chatComplete there does not move your Claude traffic.

In the console the same field is Endpoint paths on the provider key.

Required body fields

Some gateways validate the request body against a schema of their own and reject anything that omits a field they require, such as organization_id or a trace name.

Send those fields exactly as you do today and Anyray forwards them. A field Anyray does not recognise is passed through to your gateway untouched, so a client that worked before Anyray was in the path keeps working — on /v1/chat/completions and on Claude's /v1/messages alike:

curl https://anyray.acme.com/v1/messages \
  -d '{"model":"…","max_tokens":1024,"messages":[…],
       "organization_id":"acme","trace_name":"nightly-eval"}'

Your callers keep control of the values, per request, with no gateway-side setup.

For a client that cannot add fields to its own request body, an operator can fill Required body fields on the provider in the console with a JSON object, applied to every request to that upstream:

{ "organization_id": "acme" }

A field in the request always wins over the configured one.

Configure only values that stay the same for the whole deployment, such as an organization id. A value that varies per request, such as a trace or generation name, belongs in the caller's own request body. The configured one is identical on every call, so it clears your gateway's schema while stamping every trace with the same name, and the validation error you were fixing is replaced by a silent one in your own traces.

Any field your gateway's schema defines is allowed. Two limits apply:

• Only the anyray-upstream, litellm, and truefoundry providers. Your gateway's schema is yours; a hosted vendor's is not, and an unknown field there would be rejected.
• A value is a string, up to 256 characters, under a name of the form organization_id. A field needing a number, a boolean, or a nested object goes in the caller's request body.

Take care with a field that changes what the upstream generates. n and best_of make it produce completions the Anthropic response format cannot return, and reasoning_effort multiplies thinking tokens, so a value set here bills the org for work no client ever sees. On Claude's translated lane n and best_of are dropped from the request body for that reason.

LiteLLM and TrueFoundry

Both have their own provider id: set "provider": "litellm" or "provider": "truefoundry" with the same custom_host and optional key as above. They are reached at a per-deployment base URL, so the host is still required and still subject to the allowlist. TrueFoundry's host keeps its /api/llm prefix; a trailing /v1 on either is ignored.

Use them in preference to anyray-upstream. Both serve the Anthropic Messages spec on their own /v1/messages, and these provider ids forward Claude requests there in their native wire format. The generic upstream instead translates them into OpenAI chat, which carries no cache_control breakpoint and drops thinking blocks, so a warm Claude session loses its prompt cache. Non-Claude traffic keeps using /v1/chat/completions either way. Details: LiteLLM, TrueFoundry.

OpenRouter, Nebius and Tensormesh

OpenRouter, Nebius (Nebius AI Studio) and Tensormesh (Serverless Inference) are first-class providers. Set "provider": "openrouter", "nebius" or "tensormesh" (or save that key in the console) with just the provider key. Their base URLs and endpoints are built in, so you don't need custom_host or ANYRAY_CUSTOM_HOST_ALLOWLIST. Reserve anyray-upstream for arbitrary or internal OpenAI-compatible gateways (a homegrown proxy, or a vendor gateway with no dedicated provider id).

Oracle (OCI Generative AI)

Oracle Cloud Infrastructure Generative AI is a first-class provider. Set "provider": "oracle" (or save the credentials in the console) and its region-scoped base URL is built in, so you don't need custom_host. Models keep OCI's dotted vendor prefix on the wire: xai.grok-4.3, meta.llama-4-scout-17b-16e-instruct, openai.gpt-oss-120b, cohere.command-a-03-2025.

Cost reporting on the Meta and Cohere families

Anyray ships list prices for OCI's xAI and OpenAI models, so their spend and savings are exact. The Meta and Cohere models route and meter normally, but Oracle bills them per character rather than per token, and publishes no characters-per-token ratio, so their cost is reported at a fallback rate rather than Oracle's. Read the spend figures for those two families as indicative, and take Oracle's own invoice as authoritative. Each such row carries a priceSource recording that it was priced at a fallback, and the per-model breakdown on GET /admin/spend/dashboard counts them, so you can measure exactly how much of a bill rests on an estimated rate.

oracleRegion is required and has no default: the host is region-scoped and a Generative AI API key only works in the region it was created in, so guessing one can only produce an upstream 401 that reads like a bad key. The gateway rejects the request and names the missing field instead. Beyond that, Oracle accepts two credentials and the gateway picks the mode from whichever is present:

• Generative AI API key. Set apiKey, which rides as a plain Bearer token. Oracle scopes this to testing and early development.
• IAM request signing. Oracle's production path, and the one that works with an existing tenancy's API key pair. Set oracleTenancy, oracleUser, oracleFingerprint and oraclePrivateKey; each request is then signed rather than carrying a key at all. Signing is offered on the chat paths only: a signature binds your tenancy's identity to a specific request target, so the gateway will not sign one a caller supplied.

Setting only some of the four signing fields is rejected, so a half-configured tenancy fails loudly instead of quietly routing production traffic on the weaker credential. Nor is an Anyray client key ever forwarded to Oracle as a fallback bearer token: an unconfigured tenancy fails closed rather than leaking that key upstream. Full field list and key-format requirements: Choose your setup.

Attach to LiteLLM

Keeps LiteLLM as the front door and runs only the optimizer beside it. The recommended topology is the gateway in front of LiteLLM, with per-developer enrollment and the full optimization surface: see Integrations → LiteLLM.

Prerequisites: Docker Engine (v24+) and Docker Compose v2, git, and openssl on the Anyray host; Python 3.9+ and httpx in the LiteLLM environment; port 3000 (the Anyray console) reachable from your org network / VPN and port 8088 (the optimizer) reachable only by LiteLLM.

1

Clone the install repo

git clone https://github.com/anyrayHQ/install anyray && cd anyray

2

Generate secrets

./setup.sh --host <anyray-host>

setup.sh writes .env, including the admin key, optimizer token, the trace store DB URL (ANYRAY_OBSERVABILITY_DB_URL), and content-encryption key.

3

Start attach mode

docker compose -f docker-compose.attach.yml up -d

By default the optimizer binds to 127.0.0.1:8088, which is correct when LiteLLM runs on the same host. If LiteLLM runs on another machine, set ANYRAY_OPTIMIZER_BIND=0.0.0.0 in .env, keep :8088 reachable only over a private network or VPN, then restart the stack.

4

Configure LiteLLM

Copy attach/litellm/anyray_optimizer.py next to your LiteLLM config.yaml, then add:

litellm_settings:
  callbacks:
    - anyray_optimizer.proxy_handler_instance

Set these environment variables in the LiteLLM process:

ANYRAY_OPTIMIZER_URL=http://<anyray-host>:8088
ANYRAY_OPTIMIZER_TOKEN=<ANYRAY_OPTIMIZER_TOKEN from .env>

5

Verify

docker compose -f docker-compose.attach.yml ps
curl -fs http://localhost:8088/health && echo "optimizer ok"
curl -fso /dev/null http://localhost:3000/anyray-login && echo "console ok"

Open http://<anyray-host>:3000 and sign in with the admin key setup.sh printed (also in .env as ANYRAY_ADMIN_TOKEN).