Skip to main content

LangChain and LangGraph

LangChain agents and LangGraph graphs run on Anyray with a base URL and a client key. There is no package to install.

1
Add the agent

Console Users > Agents > Add agent. Name it, then open the LangChain tab, or LangGraph for a graph.

2
Hand the prompt to your coding agent

Copy the tab's prompt and give it to the coding agent working on your app. The code it writes reads ANYRAY_GATEWAY_URL and ANYRAY_CLIENT_KEY. Put the key in the app's environment yourself; never paste it into the chat. The prompt ends by polling /connect/self-check until problems is 0.

The prompts are also served at https://app.anyray.ai/agents/langchain.md and https://app.anyray.ai/agents/langgraph.md, where the agent asks you for the gateway URL.

An admin decides whether tool output is trimmed

The prompt always wires the Anyray MCP server. Trimming in your app's tool loop is the gateway setting harnessRetrieval, on by default (Console → Optimizer → Harness retrieval). While it is on, the gateway trims tool output behind handles the model reads back. An admin can turn it off; the gateway then trims nothing in the loop and keeps the retrieval tools it declares away from the model. An app that reads local files and takes input only from trusted people gets the anyray-connect stdio server, which adds batch reads (anyray_read_batch).

3
Or wire it manually

Open Wire it manually on the same tab. It prints the code below with your gateway and key filled in. Use the tab for the package your app already uses.

Anthropic Python
import asyncio

from langchain.agents import create_agent
from langchain.tools import tool
from langchain_anthropic import ChatAnthropic


@tool
def ping(text: str) -> str:
    """Echo the text back."""
    return text


tools = [ping]
model = ChatAnthropic(
    model="claude-sonnet-5-5",
    base_url="https://gateway.example.com",
    api_key="ark_synthetic-123456",
    default_headers={"x-anyray-metadata": '{"tool":"langchain"}'},
)
agent = create_agent(model, tools=tools)
# In a notebook, await the call instead of asyncio.run.
result = asyncio.run(agent.ainvoke({"messages": [{"role": "user", "content": "Ping hello"}]}))
print(result["messages"][-1].content)

The Anthropic package takes the bare gateway origin; the OpenAI package needs /v1. The header labels the traffic in the console. The key already carries the agent's name and team.

4
Run it once
Confirm it connected

Within a minute the dialog's last step reads Connected, and the agent's row on the Agents card reads Active · forwarded unchanged, or Active · no tool calls for a chain without tools. An app with the Anyray MCP server reads Active, unless an admin turned harnessRetrieval off.

LangGraph​

The LangGraph tab prints this graph. The model is built as in step 3, with "tool":"langgraph" in the header.

Python
import asyncio

from langchain_core.tools import tool
from langchain_anthropic import ChatAnthropic
from langgraph.graph import START, MessagesState, StateGraph
from langgraph.prebuilt import ToolNode, tools_condition


@tool
def ping(text: str) -> str:
    """Echo the text back."""
    return text


tools = [ping]
model = ChatAnthropic(
    model="claude-sonnet-5-5",
    base_url="https://gateway.example.com",
    api_key="ark_synthetic-123456",
    default_headers={"x-anyray-metadata": '{"tool":"langgraph"}'},
).bind_tools(tools)


async def agent(state: MessagesState):
    return {"messages": [await model.ainvoke(state["messages"])]}


builder = StateGraph(MessagesState)
builder.add_node("agent", agent)
builder.add_node("tools", ToolNode(tools))
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", tools_condition)
builder.add_edge("tools", "agent")
graph = builder.compile()
# In a notebook, await the call instead of asyncio.run.
result = asyncio.run(graph.ainvoke({"messages": [{"role": "user", "content": "Ping hello"}]}))
print(result["messages"][-1].content)
In your graphWhat happens
A fixed sessionId in each node's header, such as "sessionId":"ping-graph.agent"That node's calls share one row on the Sessions page, run after run. On the OpenAI package it is the only way they group
A node whose model binds no toolsIts requests are never trimmed, even after earlier nodes trimmed theirs
Send fan-outBranches that open with the same system prompt and first message share one session until each makes its first tool call
An interrupt and resumeThe resumed run keeps its session
LangGraph Platform (langgraph.json)The model options stay in your graph code; put the key in the deployment's environment
Never send thread_id as the sessionId

A thread_id or any other per-run id changes every run. On OpenAI models sessionId also becomes the prompt-cache key, so a per-run value sends every run to a cold cache. Why, and what to send instead.

Tools and retrieval​

Tool calls go through unchanged, so there is nothing to add. The agent row reads Active · forwarded unchanged, or Active · no tool calls for a chain with no tools (RAG, summarization).

Add the gateway's MCP tools with langchain-mcp-adapters when you want to:

  • call your organization's shared connectors (keep just the anyray-connectors entry below),
  • read back trimmed output, which harnessRetrieval allows by default, or
  • fix an MCP disconnected row after an admin opted tool loops into gateway-served retrieval.

On a served deployment the gateway answers retrieval itself, except in three setups:

Your app usesWhy the gateway stands aside
thinking on ChatAnthropicA thinking block's signature binds to rounds your app never sees
ChatOpenAI with use_responses_api=TrueThat lane is served only with store=False
A forced tool choice, which includes most structured output (with_structured_output, TypeScript responseFormat)The model may not pick the retrieval tool

Replace the tools line in the script with:

Python
# Replace the tools line above with these.
from langchain_mcp_adapters.client import MultiServerMCPClient

mcp = MultiServerMCPClient({
    "anyray": {
        "transport": "streamable_http",
        "url": "https://gateway.example.com/mcp",
        "headers": {
            "x-anyray-api-key": "ark_synthetic-123456",
            "x-anyray-metadata": "{\"tool\":\"langchain\"}"
        }
    },
    "anyray-connectors": {
        "transport": "streamable_http",
        "url": "https://gateway.example.com/mcp/org",
        "headers": {
            "x-anyray-api-key": "ark_synthetic-123456",
            "x-anyray-metadata": "{\"tool\":\"langchain\"}"
        }
    }
})
# In a notebook, await the call instead of asyncio.run.
tools = [ping, *asyncio.run(mcp.get_tools())]

In a graph the header names langgraph. The adapter's Python tools are async-only, which is why the scripts call ainvoke. /mcp reads trimmed content back and /mcp/org serves your organization's shared connectors. Both sit on the gateway origin, not under /v1. Rarer cases are in the full list.

Streaming​

.stream(), .astream() and LangGraph's stream_mode values work unchanged. On a deployment that serves retrieval for tool loops, a reply's tool calls arrive together when the model finishes instead of token by token. Token usage comes once, on the last chunk, even if your client did not ask for it.

Set it up with environment variables
export ANTHROPIC_BASE_URL="https://gateway.example.com"
export ANTHROPIC_API_KEY="ark_synthetic-123456"
export OPENAI_BASE_URL="https://gateway.example.com/v1"
export OPENAI_API_KEY="ark_synthetic-123456"

ChatAnthropic and ChatOpenAI read these when you pass no base URL or key. A variable cannot carry the x-anyray-metadata header, so the console lists this traffic as Shell / SDK rather than LangChain or LangGraph.

Deep Agents

A Deep Agents app takes the model from step 3. A subagent that names its own model needs the same base URL, key and header. It does not need Anyray's built-in skill, which is for coding tools on enrolled machines.

LangSmith tracing

With LANGSMITH_TRACING=true, LangChain sends each run's prompts and responses to LangSmith under your LangSmith key, whether or not you route through Anyray. Leave it unset if prompt content must stay in your network.