Keep an agent running when the gateway is down

While the gateway is unreachable, anyray-agent sends an agent's calls straight to the provider. They return to the gateway once it recovers.

Seats get this from Connect (direct fallback). An agent reads its base URL once, so its fallback lives in the SDK's HTTP client: its fetch in Node, its httpx2 transport in Python.

Add it

Node

Node

npm install anyray-agent

Python

Python

pip install anyray-agent

Keep the client pointed at the gateway with the agent's key, and add one option:

agent.ts

agent.ts

import OpenAI from 'openai';
import { anyrayFetch } from 'anyray-agent';

const client = new OpenAI({
  baseURL: 'https://<your-anyray-gateway>/v1',
  apiKey: process.env.ANYRAY_CLIENT_KEY,
  fetch: anyrayFetch(),
});

agent.py

agent.py

import os

import httpx2
from openai import OpenAI
from anyray_agent import AnyrayTransport

client = OpenAI(
    base_url="https://<your-anyray-gateway>/v1",
    api_key=os.environ["ANYRAY_CLIENT_KEY"],
    http_client=httpx2.Client(transport=AnyrayTransport()),
)

Client	Node	Python
OpenAI SDK	fetch: anyrayFetch()	http_client=httpx2.Client(transport=AnyrayTransport())
Anthropic SDK	fetch: anyrayFetch()	http_client=httpx2.Client(transport=AnyrayTransport())
LangChain ChatOpenAI	configuration: { fetch: anyrayFetch() }	http_client=httpx2.Client(transport=AnyrayTransport())
LangChain ChatAnthropic	clientOptions: { fetch: anyrayFetch() }	use_fallback(llm, transport=AnyrayTransport())

Node 18 or later, with no dependencies. Python 3.11 or later, on httpx2, which the current OpenAI and Anthropic SDKs already use. In Python:

• use_fallback comes from anyray_agent.langchain, because ChatAnthropic takes no HTTP client of its own. Call it on the ChatAnthropic(...) you construct, not on what bind_tools returns.
• Async calls, LangChain's ainvoke and astream included, only reach an AsyncAnyrayTransport in an httpx2.AsyncClient (http_async_client= on ChatOpenAI, async_transport= on use_fallback).
• Build each transport once per process and reuse it. Closing an httpx2 client closes its transport.
• An SDK that still requires httpx 0.x instead of httpx2 takes the same classes from anyray_agent.httpx, in an httpx client (pip install "anyray-agent[httpx]"). use_fallback builds httpx2 clients, so upgrade an anthropic still on httpx 0.x before you use it.

Add agent shows this step under the Python and TypeScript snippets on the LangChain and LangGraph tabs.

The provider key

A call can go direct only with a key for its provider on the agent's host. Without one, that provider's calls fail during an outage as they would without the package, and the agent reports itself blocked.

Calls	Go direct to	Key
OpenAI models (gpt-*, o*, embeddings)	api.openai.com	OPENAI_API_KEY, or providerKeys: { openai }
Claude models (claude-*)	api.anthropic.com	ANTHROPIC_API_KEY, or providerKeys: { anthropic }
Claude models, with bedrock: { region }	AWS Bedrock in that region	AWS_BEARER_TOKEN_BEDROCK, or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (plus AWS_SESSION_TOKEN), or bedrock.apiKey / bedrock.credentials

Python takes the same options in snake_case: provider_keys, bedrock={"region": ...}, api_key, credentials.

Node

Node

anyrayFetch({ bedrock: { region: 'us-east-1' } });

Python

Python

AnyrayTransport(bedrock={"region": "us-east-1"})

Bedrock uses known dated IDs directly. For undated IDs, it lists inference profiles, then foundation models, and picks the newest matching snapshot. If discovery fails or finds no match, it tries the original candidates. Set bedrock.models (models inside Python's bedrock dict) to override a model's Bedrock ID. Streaming works.

When a call goes direct

A call goes to the provider only when the gateway provably never handled it: the connection never opened (refused, DNS, connect timeout) and the gateway's /connect/ready check reads down. A 404 falls back to /livez. If readiness reports provider_unreachable, a provider this link has used must answer a reachability check before fallback can start. The Bedrock lane checks its runtime origin. With no provider used yet, it stays on the gateway. Once the outage has lasted your org's fallback time, calls go direct without trying the gateway first. They come back once the gateway has stayed healthy for your org's recovery time. Both follow When the gateway is unreachable, under the Direct fallback switch on the console Health page, the settings seats use:

Setting	Agents switch after	Agents come back after	Agents check /livez
Switch fast	1 min	10 min	Every 60 s, failed after 3 s
Balanced Default	2 min	10 min	Every 60 s, failed after 5 s
Ride out blips	4 min	10 min	Every 60 s, failed after 5 s
Custom	(failed checks - 1) × check interval	(passing checks - 1) × check interval	Every check interval, failed after the check timeout
Off	Never

Agents check /livez only while the gateway seems down or calls go direct. A gateway older than these settings gets a check every 15 s, failed after 3 s on Switch fast and 5 s otherwise.

A call is never sent direct when:

• it may already have reached the provider: a reset mid-request, or any 5xx. Nothing is billed twice.
• it names a model or route only the gateway serves: a model the provider does not host, a routing header, an MCP server on the gateway, or a Responses call that continues an earlier one
• it carries no agent key (ark_…), or goes to another base URL
• your org turned Direct fallback off, or the agent has not read that setting from the gateway yet

The agent key is never sent to a provider.

See its state

The agent reports its state to the gateway, with no request content. Agents > the agent > General info > Direct fallback shows it, and the roster row shows Running direct while calls go direct.

State	Meaning
Armed	Ready. The last direct period, if any, follows with its call count.
Running direct	Calls are going to the provider.
Blocked	Fallback cannot carry these calls: no provider key, a model or route only the gateway serves, a request the agent cannot resend, or the org setting not read yet.
Turned off by the org	The org turned Direct fallback off.
Not reported	The agent does not use anyray-agent, or has not made a call yet.
Needs gateway upgrade	The gateway predates this report.

Advanced setup

Pass these options to anyrayFetch, or in snake_case to AnyrayTransport in Python. Use the result in the same place as above. Add agent builds one snippet from these options under Advanced setup.

Option	When you need it	What to pass
Keys from a secret store	The provider key is not in the host's environment.	providerKeys with your provider's key string, read once at construction.
Bedrock with an IAM role	The agent calls Claude through AWS Bedrock.	bedrock with a region and async credentials provider.
Wrap your own fetch or transport	You already pass the SDK a fetch (Node) or an httpx2 transport (Python) for a proxy or custom dispatcher.	Yours as fetch (Node) or transport (Python).
Report the fallback state	Your monitoring or health check needs the state: armed, direct, blocked, off.	onStatus (on_status) for changes; status() for the current state.

Keys from a secret store

For OpenAI, use openai and OPENAI_API_KEY instead.

Node

Node

import { anyrayFetch } from 'anyray-agent';

const fallback = anyrayFetch({
  // yourSecretStore: your secret store's client; get() returns the key string.
  providerKeys: { anthropic: await yourSecretStore.get('ANTHROPIC_API_KEY') },
});

Python

Python

from anyray_agent import AnyrayTransport

transport = AnyrayTransport(
    # your_secret_store: your secret store's client; get() returns the key string.
    provider_keys={"anthropic": your_secret_store.get("ANTHROPIC_API_KEY")},
)

Bedrock with an IAM role

The package does not load AWS profiles or instance roles. On EC2, ECS or EKS, pass bedrock.credentials as an async function that returns { accessKeyId, secretAccessKey, sessionToken }. In Python, pass boto3's credential chain, which reads the same profiles and roles. With bedrock set, Claude fallback uses Bedrock instead of the Anthropic provider key.

Node

Node

// npm install anyray-agent @aws-sdk/credential-providers
import { anyrayFetch } from 'anyray-agent';
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';

const fallback = anyrayFetch({
  // Replace us-east-1 with your Bedrock region.
  bedrock: { region: 'us-east-1', credentials: fromNodeProviderChain() },
});

Python

Python

# pip install anyray-agent boto3
import boto3
from anyray_agent import AnyrayTransport

transport = AnyrayTransport(
    # Replace us-east-1 with your Bedrock region.
    bedrock={"region": "us-east-1", "credentials": boto3.Session().get_credentials},
)

Wrap your own fetch or transport

Node

Node

import { anyrayFetch } from 'anyray-agent';

const fallback = anyrayFetch({
  // yourFetch: the fetch you pass the SDK today.
  fetch: yourFetch,
});

Python

Python

from anyray_agent import AnyrayTransport

transport = AnyrayTransport(
    # your_transport: the httpx2 transport you give the SDK's client today.
    transport=your_transport,
)

Report the fallback state

State	Meaning
armed	Ready to send calls direct during an outage.
direct	Sending calls straight to the provider.
blocked	Cannot send calls direct; the reason says why.
off	The org turned direct fallback off.

Node

Node

import { anyrayFetch } from 'anyray-agent';

const fallback = anyrayFetch({
  onStatus: (status) => console.log('anyray-agent', status.state),
});

// Health check: armed, direct, blocked, off. blocked carries a reason.
const fallbackHealth = () => fallback.status();

Python

Python

from anyray_agent import AnyrayTransport

transport = AnyrayTransport(
    on_status=lambda status: print("anyray-agent", status.state),
)

# Health check: armed, direct, blocked, off. blocked carries a reason.
def fallback_health():
    return transport.status()