LiteLLM

The Anyray gateway sits in front of LiteLLM, which keeps its routing and virtual keys.

Client tool → Anyray gateway → LiteLLM → provider, each dialect in its native wire format.

Property	Value
Provider id	litellm, never the generic custom upstream (see below)
Gateway speaks	Anthropic /v1/messages for Claude, OpenAI /v1/chat/completions otherwise
Credential	Each developer's LiteLLM virtual key, forwarded verbatim, or one server-held org key
Client base URL	The tool's native base-URL env, pointed at the gateway's :8787

Set it up

1

Allow the LiteLLM hostname

Open Providers, then LiteLLM, then Personal keys. Turn on Ask each person for their key, enter the LiteLLM https:// URL, and save. The gateway then accepts that exact origin, only for model requests (messages, chat completions, responses, embeddings, models) that carry the developer's own key. The gateway also serves the URL in the connect policy. Another port, a subdomain, another path or a request without the developer's key is refused with "Invalid custom host".

Without the setting, add the hostname to ANYRAY_CUSTOM_HOST_ALLOWLIST. Not needed if LiteLLM is already the configured upstream.

2

Onboard each developer

Per machine, with the enrollment link and the developer's virtual key:

curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- \
  --enroll https://app.anyray.ai/enroll/enl_… \
  --user dev@example.com \
  --upstream https://litellm.example.com \
  --upstream-key sk-synthetic1234 \
  --yes

Mints a personal ark_… key and forwards the virtual key on both lanes. --upstream needs a key; prefer --upstream-key-stdin so the key stays out of the process list (CLI commands).

3

Run one session and check the console

Requests appear under the anthropic provider with the developer's name.

No per-developer keys? Omit both flags and set one org key on the LiteLLM provider. Base URL is the proxy root, no trailing /v1; the key is optional for an unauthenticated proxy or custom-header auth.

Developers who use the desktop app

They pass no flags. If Claude Code already points at the allowlisted LiteLLM host with a virtual key, turning Claude Code on keeps that key and routes it through the gateway. The same holds for a hand-written config that points Claude Code at the gateway with the virtual key as its token and an x-anyray-api-key header, once your organization sets the personal URL. The key is checked first; if LiteLLM does not accept it, nothing changes. An admin-assigned billing lane takes precedence. With Ask each person for their key on, the app asks for the key after sign-in (Your LiteLLM key).

Why the `litellm` provider

The generic anyray-upstream translates Claude into OpenAI chat, dropping cache_control and thinking blocks (savings fell roughly 35x in one deployment). litellm forwards Claude to LiteLLM's /v1/messages. Same for TrueFoundry.

A hand-rolled provider block already points at the gateway

A Claude-only custom block (often litellm) is migrated on the next apply: removed, models moved to anthropic/*; anyray-connect --revert restores it (OpenCode). Mixed-family blocks, and anthropic / openai blocks you pinned yourself, are kept with only the attribution header added. Move them with --upstream.

The other topology: the optimizer inside LiteLLM

If nothing may sit in front of LiteLLM, use the hook-subset optimizer: Attach to LiteLLM.