LiteLLM

Keep LiteLLM exactly where it is. The Anyray gateway sits in front of it, so every request is optimized and attributed before LiteLLM routes it upstream.

What's specific to LiteLLM

Property	Value
Provider id	litellm
Gateway speaks	Anthropic /v1/messages for Claude, OpenAI /v1/chat/completions otherwise
Credential	Each developer's LiteLLM virtual key, forwarded verbatim, or one server-held org key
Client base URL	The tool's native base-URL env → the gateway's :8787

Requests flow client tool → Anyray gateway → LiteLLM → provider. The gateway accepts both the Anthropic and OpenAI dialects and forwards each in its own wire format, so client tools stay on their native managed lanes, where the optimizer earns most, while LiteLLM keeps owning upstream routing and key management. Everything shared (self-hosting, provider config, credential storage) is in the gateway.

Use the litellm provider, not the generic custom upstream

Both reach a LiteLLM proxy, but only litellm forwards Claude requests to LiteLLM's own /v1/messages. The generic anyray-upstream provider translates them into OpenAI chat, which cannot carry a cache_control breakpoint or a thinking block: every turn of a warm Claude session then re-processes a prefix LiteLLM could have read from cache. One production deployment that ran Claude over the translated lane saw per-request savings fall roughly 35x.

Running TrueFoundry instead? Same arrangement, same reasoning.

Onboard a developer

One command per machine. The enrollment link comes from your admin (onboard users); the upstream key is the developer's own LiteLLM virtual key:

curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- \
  --enroll https://app.anyray.ai/enroll/enl_… \
  --user dev@example.com \
  --upstream https://litellm.example.com \
  --upstream-key sk-synthetic1234 \
  --yes

This enrolls the developer (a personal ark_… key, so spend is attributed to them), points every detected tool at the gateway on its managed lane, and writes the passthrough pins that make the gateway forward the developer's LiteLLM key verbatim to your LiteLLM host. --upstream and --upstream-key go together; see CLI commands. Without per-developer virtual keys, omit both flags and set one org key on the LiteLLM provider in the console instead: its Base URL is your proxy root (omit a trailing /v1), and the virtual key is optional if your proxy is unauthenticated or takes its credential on a custom header.

warning

The LiteLLM hostname must be authorized on the gateway, so add it to ANYRAY_CUSTOM_HOST_ALLOWLIST (or TRUSTED_CUSTOM_HOSTS for a private/internal upstream), or every passthrough request is rejected with "Invalid custom host".

Success

Run one session from any connected tool, then open the console dashboard: the request appears under the litellm provider with the developer's name attached, and the optimizer's savings show within a few turns.

A hand-rolled provider block already points at the gateway

Machines wired up before Connect often carry a custom provider block (commonly named litellm) pointing client tools at the gateway over the OpenAI-compatible dialect, which keeps the optimizer earning a fraction of the managed lane. When everything that block serves is a Claude model, Connect migrates it automatically on the next apply: the block is removed, its model references move to anthropic/*, and anyray-connect revert restores the original from the backup. Details per tool: OpenCode.

The other topology: the optimizer inside LiteLLM

If LiteLLM must stay the front door, with nothing between clients and it, attach the Anyray optimizer to LiteLLM as a request hook instead; there is no gateway in the request path. See "Attach to LiteLLM" in the gateway. Fronting LiteLLM with the gateway (this page) is the recommended topology: it adds per-developer enrollment, client-native dialects, and the full optimization surface rather than the hook subset.