LiteLLM
Keep LiteLLM where it is. The Anyray gateway sits in front of it, so every request is optimized and attributed before LiteLLM routes it upstream.
Requests flow client tool → Anyray gateway → LiteLLM → provider. The gateway speaks both dialects and forwards each in its own wire format, so client tools stay on their native managed lanes, where the optimizer earns most, and LiteLLM keeps owning upstream routing and keys.
| Property | Value |
|---|---|
| Provider id | litellm, never the generic custom upstream (see below) |
| Gateway speaks | Anthropic /v1/messages for Claude, OpenAI /v1/chat/completions otherwise |
| Credential | Each developer's LiteLLM virtual key, forwarded verbatim, or one server-held org key |
| Client base URL | The tool's native base-URL env, pointed at the gateway's :8787 |
Set it up
Add it to ANYRAY_CUSTOM_HOST_ALLOWLIST, or every passthrough request fails with "Invalid custom
host". Skip this when LiteLLM is already the gateway's configured upstream, a private address
included.
One command per machine. Your admin supplies the enrollment link (onboard users), and the upstream key is that developer's own LiteLLM virtual key:
curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- \
--enroll https://app.anyray.ai/enroll/enl_… \
--user dev@example.com \
--upstream https://litellm.example.com \
--upstream-key sk-synthetic1234 \
--yes
This mints a personal ark_… key, so spend is attributed to that developer. It points every
detected tool at the gateway on its managed lane, and writes the pins that forward the virtual
key verbatim on both the Anthropic and the OpenAI lane, so every model the proxy serves is paid
for by that developer's own key. --upstream and --upstream-key go together
(CLI commands).
The request appears under the litellm provider with the developer's name attached. Savings show
within a few turns.
No per-developer virtual keys? Omit both flags, and set one org key on the LiteLLM provider in
the console instead. Its Base URL is your proxy root, without a trailing /v1. The virtual key is
optional when your proxy is unauthenticated or takes its credential on a custom header.
Why the `litellm` provider
Both litellm and the generic anyray-upstream reach a LiteLLM proxy, but only litellm
forwards Claude requests to LiteLLM's own /v1/messages. The generic provider translates them
into OpenAI chat, which carries neither a cache_control breakpoint nor a thinking block.
Every turn of a warm Claude session then re-processes a prefix LiteLLM could have read from
cache. One production deployment on the translated lane saw per-request savings fall roughly 35x.
Running TrueFoundry instead? Same arrangement, same reasoning.
A hand-rolled provider block already points at the gateway
Machines wired up before Connect often carry a custom provider block, commonly named litellm,
that points client tools at the gateway over the OpenAI-compatible dialect. That keeps the
optimizer earning a fraction of the managed lane.
When everything the block serves is a Claude model, Connect migrates it on the next apply: the
block is removed, its model references move to anthropic/*, and anyray-connect --revert
restores the original from the backup. Per-tool detail: OpenCode.
A block that also serves other model families is kept as it is. So is a canonical anthropic or
openai block you pinned at your own upstream by hand: Connect recognizes the passthrough
headers as yours, leaves your credential and host alone, adds only the attribution header, and
says so in the apply output. Move those lanes onto a managed passthrough with --upstream.
The other topology: the optimizer inside LiteLLM
If LiteLLM must stay the front door, with nothing between clients and it, attach the Anyray optimizer to LiteLLM as a request hook. There is no gateway in the request path: Attach to LiteLLM.
Fronting LiteLLM with the gateway, which is this page, is the recommended topology. It adds per-developer enrollment, client-native dialects, and the full optimization surface rather than the hook subset.