Routing

One default config decides where requests go; a client can override per request, and conditional lanes match on verified identity.

Set the default on Routing (or GET / PUT /admin/routing-config); a client can override per request with the x-anyray-config header. Routing applies to API-key traffic only; passthrough subscription traffic keeps the user's own token and provider. anyray-default, what anyray-connect sends to mean "gateway, you pick", resolves to ANYRAY_DEFAULT_MODEL (default anthropic/claude-sonnet-4-5); override per alias with ANYRAY_MODEL_ALIASES or PUT /admin/model-aliases.

Mode	Behavior
single	One target for every request.
loadbalance	Spread across targets by weight, sticky per conversation.
fallback	Try targets in order; advance on failure.
conditional	Pick a target from request conditions (below).

Transient failures (502 / 503 / 504, network) get 2 attempts; a 429 is retried for non-streaming requests only. A retry block changes it: { "attempts": 0 } opts out, max 5.

Upstream timeouts

Bound the wait on a provider; set via .env + restart.

Env var	Default	Bounds
ANYRAY_UPSTREAM_CONNECT_TIMEOUT_MS	10000	Connecting to the provider.
ANYRAY_UPSTREAM_HEADERS_TIMEOUT_MS	60000	First response byte; raise for slow providers.
ANYRAY_UPSTREAM_BODY_TIMEOUT_MS	600000	Gap between streamed chunks.

Conditional routing

conditional mode picks a named target from an ordered conditions list, else default (no match and no default → 400):

{ "strategy": { "mode": "conditional",
    "conditions": [ { "query": { "metadata.team": "research" }, "then": "frontier" } ],
    "default": "economy" },
  "targets": [ { "name": "frontier", "provider": "anthropic" },
               { "name": "economy", "provider": "openai" } ] }

A query matches metadata.<key>, params.<field>, and url.pathname, with operators ($eq, $ne, $gt / $gte / $lt / $lte, $in, $nin, $regex) combined by $and / $or. First match wins.

Identity fields (metadata.user / userId, metadata.team / teamId, metadata.agent / agentId) are evaluated on the verified key-bound identity, never the header, so a client can't steer an identity lane by editing x-anyray-metadata; every other metadata.<key> field stays client-controlled. The team is the seat's current SCIM-mapped team when SCIM manages the roster, otherwise the team set at enrollment. A bound key with no team ignores a header-supplied team: team lanes need team-bearing keys (SSO enrollment groupTeamMap / defaultTeam).

Pinning a target to a named provider key

Any target may carry provider_key_id naming one of the provider's named keys; the stored config holds the id only, never a credential. With verified team conditions this gives each department its own upstream account:

{ "strategy": { "mode": "conditional",
    "conditions": [ { "query": { "metadata.team": "ml" }, "then": "ml-lane" } ],
    "default": "org" },
  "targets": [ { "name": "ml-lane", "provider": "openai", "provider_key_id": "team-ml" },
               { "name": "org", "provider": "openai" } ] }

Key selection is admin-config only: a provider_key_id inside a client-sent x-anyray-config header is ignored, so a user can't bill another team's key. A save naming an id the provider-key store doesn't hold is rejected; if the key is deleted later, requests routed to that pin fail with 424, never a silent fallback to a different key.

Per-agent routing

An agent (a service key minted on the Agents page) can carry its own provider and named key, so unattended traffic bills a separate upstream account from your developers. Bind it on the agent itself, on the Agents page or at mint time (Add an agent); the Routing page lists such bindings read-only under From Agents. Precedence: the request's own x-anyray-config / x-anyray-provider wins, then the agent's binding, then the org routing config.

When the condition is broader than one agent, express it as a lane instead. metadata.agent is the agent's name and metadata.agentId its key id, which survives a rename:

{ "strategy": { "mode": "conditional",
    "conditions": [ { "query": { "metadata.agent": "release-bot" }, "then": "ci-lane" } ],
    "default": "org" },
  "targets": [ { "name": "ci-lane", "provider": "bedrock", "provider_key_id": "ci-account" },
               { "name": "org", "provider": "openai" } ] }

Both fields derive from the verified key, never the header, so no developer key, shared org key, or BYO pass-through can reach an agent lane; being an agent is a property of the key, not of the name.

Targeted routing on the Routing page

The Routing page builds conditional lanes for you (Targeted routing), wrapping whatever default strategy you already configured. Each lane picks what it matches on:

Match	Condition it writes	Matched against
agent name	metadata.agent	The verified service key's name
agent id (survives rename)	metadata.agentId	The verified service key's id
user	metadata.user	The verified key-bound user
team	metadata.team	The verified key-bound team
model	params.model	The model named in the request

The first four are identity, derived from the key that signed the request; model says what was asked for, never who asked. Match an agent on its id if you may ever rename it: a name lane stops matching on rename and its traffic silently falls back to the default. A lane naming a revoked agent is flagged on the page rather than blocked. The page edits only the shape it generates (one equality test per lane); a config with hand-written conditions is shown read-only rather than flattened on the next save.