Agent tool-loop reference

Per-request optimization

These headers are optional. Omit them to use the organization's configured policy. They work on HTTP POST requests to /v1/chat/completions, /v1/completions, /v1/messages and /v1/responses, including streaming responses. The /v1/responses WebSocket handshake accepts them too and carries them into every turn on the socket, where they apply only to turns the gateway optimizes.

Header	Value	Effect
x-anyray-optimization-profile	interactive	Use configured optimizations and existing safety checks.
x-anyray-optimization-profile	exact	Decline new content, reasoning and output-parameter transformations, and semantic answer reuse. Cache anchoring and diagnostics retain server policy.
x-anyray-optimization-kinds	Comma-separated strategy IDs	Run only the listed kinds, including ones that are off by default. Cache anchoring and diagnostics retain server policy.
x-anyray-optimization-shadow-kinds	Comma-separated strategy IDs	Report whether each listed kind would have saved tokens on this request, without running it on what is sent. For paired experiments.
x-anyray-retrieval-tools	lazy	Hide client-declared Anyray retrieval tools from the upstream wire until the first retrieval handle appears. Send it on a new session's first request.
x-anyray-retrieval-tools	deferred	Declare the Anyray retrieval tools unloaded and load them at the first retrieval handle, without re-writing the cached prompt. Claude Opus 5 models on a Claude seat. Send it on a new session's first request.
x-anyray-harness-retrieval	on or off	Override harnessRetrieval for an app tool loop that declares anyray_retrieve.

For verbatim extraction, start a fresh conversation with exact:

Request options in an existing SDK harness

response = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_headers={"x-anyray-optimization-profile": "exact"},
)

Keep your base URL and client key. For a measured experiment, set x-anyray-optimization-kinds: thinking_trim,command_digest. If both headers are present, their restrictions intersect; exact still declines those transforms. A listed kind that is off by default runs as if an admin had turned it on, so every guard still applies: an admin rule's disable, the session gate, the regression guard, the holdout, and the retrieval, cache-safety, privacy and spend checks. The session gate can hold the kind out for a whole conversation to measure it. The header cannot turn on semantic_cache or change strategy parameters. Admins can make the header narrow-only with requestKindsEnable: false in the optimizer config. x-anyray-optimize: off takes precedence over a valid selection.

When your client declares the Anyray retrieval tools (Claude Code, Codex or OpenCode with Anyray connect), the gateway keeps them on the upstream wire from a session's first request. Hiding them until the first handle would change tools at that handle and re-write the whole cached prompt, so the optimizer usually declines to mint at all. Send x-anyray-retrieval-tools: lazy on the first request to hide them anyway, for example when the tool definitions change how the model works on a task. A session whose first request narrows x-anyray-optimization-kinds to kinds that never mint a handle, or uses the exact profile, is lazy too.

deferred keeps the tools in tools from the first request, marked defer_loading: true, and adds them to the conversation at the first handle with a tool_addition message, so the cached prompt stays warm. It applies to Claude Opus 5 models on a Claude seat when the request also declares a tool that stays loaded, puts no cache marker on a retrieval tool, and runs no tool search of its own. The gateway adds the mid-conversation-tool-changes-2026-07-01 beta. Later requests stay deferred without the header. One that does not qualify (another model or route, its own tool search) is sent as lazy, and so is the rest of a session whose deferred request Anthropic rejected. A Claude Code main thread can use deferred, even if Claude Code counted tokens before its first turn.

The first request decides, and later turns keep that choice; a later header does not change it. A session that already has a recorded request is lazy, and so are a Claude Code subagent and an app tool loop with harnessRetrieval on. The trace of the request that puts a lazy session's tools back names why: a handle minted on it, or a marker already in the conversation.

If the client defers the MCP tools out of tools, the gateway does not add them. If the spend store is unavailable, a later turn keeps the mode this gateway replica last saw for the session; with none, it is lazy, or deferred when the request sends that header.

Choose exact before the conversation starts

Narrowing a later request can preserve earlier settled edits, including thinking trim and deterministic prefix compression. That needs recorded evidence for the same content at the same position; new or edited content stays unchanged. The response lists them in preserved. Older or pinless sessions can lack that evidence; changing their selection can cause a one-time cache rewrite. Use a fresh conversation for strict exact behavior. A kind the header switched on keeps its earlier edits for the rest of the conversation, even on requests that stop naming it, so the cache stays warm. It makes no new edits on those requests.

The x-anyray-optimization-result response header contains JSON:

Selection feedback

{"status":"applied","requested":{"profile":"exact"},"applied":[],"skipped":[{"kind":"tool_schema_compression","reason":"request_selection"}],"preserved":[]}

status: applied acknowledges the selection, even when no strategy changed the request. applied lists kinds that produced decisions. skipped reports policy_disabled, request_selection, safety_gate or no_change for kinds that produced no decision. A safety_gate entry can also carry guard, the guard that held the kind back, for example {"kind":"observation_mask","reason":"safety_gate","guard":"cache_guard"} when a new edit would rewrite history the provider already cached. guard takes the suppressedKinds reasons listed in the optimizer protocol and is omitted for any other.

A request with x-anyray-optimization-shadow-kinds also gets shadow, for example {"requested":["repeat_factor"],"wouldApply":["repeat_factor"],"unsupported":[]}. A requested kind in neither list ran and would not have applied. unsupported lists kinds that can't run this way, because they store content for retrieval, and kinds the optimizer didn't answer. Like the other selection headers, it skips the response cache for that request. preserved lists kinds whose earlier edits were replayed or verified and preserved. Only strategy IDs and fixed status/reason codes appear; prompt content is never included.

bypassed, unavailable and unsupported mean the selection was not applied. A missing header means support is unconfirmed. Upgrade the optimizer before the gateway; an older optimizer cannot acknowledge the selection. Invalid or unknown values, empty kind lists and use on other endpoints return 400 without echoing the supplied text.

The built-in Anyray skill teaches these choices when an agent writes a harness. Source hooks run earlier, before tool output enters the transcript, and keep their own configuration. Profiles do not submit batch jobs; background work still uses the provider's batch API.

Defer rarely used tools

On a /v1/messages request to Anthropic or Bedrock's native Claude InvokeModel lane, the gateway stops you paying for tool schemas the model rarely calls. It marks those tools defer_loading and adds Anthropic's tool search tool, so the model loads one only when it needs it.

It is on by default. An admin turns it off for every client by setting toolDefer to off on PUT /admin/v1/settings. A request's own header wins: x-anyray-tool-defer: off opts that request out, and on opts it in when the setting is off.

A Claude Code session keeps the mode its first request got. Changing the setting or the header later applies to new sessions only, because changing tools mid-session would re-write its whole cached prompt. For the same reason, a session already running before deferral applied to it (for example, when the gateway was upgraded) keeps its full tool list.

It defers Claude Code's rarely used builtins (scheduling, worktree and agent messaging tools) and MCP tools named mcp__*. It keeps the file, shell, web, subagent and skill tools loaded, plus Anyray's own tools, the tool tool_choice forces, and any tool with a cache_control marker. It does nothing when the model has no tool search or the schemas are too small to matter. Bedrock Converse, other providers, inline configs, and custom hosts get the request as sent.

When the client already sends its own tool search tool, as Claude Code does, the gateway forwards its tools as sent, with reason client_tool_search. Deferring into the client's own search (with no second search tool) is built but not yet on. Once it is, it applies only to Claude Code sessions that start after that. A session already running keeps its tool list, and a request with no session never gets it.

Every request with a mode in effect gets an x-anyray-tool-defer-result response header:

Tool deferral feedback

{"applied":true,"deferred":16,"deferredChars":41980,"search":"gateway","modeSource":"request"}

search is gateway when the gateway added its own search tool. It is client only for a session that deferred into the client's own search.

modeSource is session when a Claude Code session's held mode decided and request when the header or setting did. It is fallback when the held mode could not be read in time: that request forwards its tools as sent, with reason session_unavailable.

When the rewrite stands aside, the header gives a named reason, for example {"applied":false,"reason":"bedrock_converse","modeSource":"request"}. The result is also stored in the request trace's toolDefer metadata.

Reason	Why it stood aside
handler_not_reached, target_not_reached	The request did not reach the rewrite or a pending Bedrock target.
inline_config, custom_host, other_provider, bedrock_converse	The resolved lane is not eligible.
invalid_json, no_tools, unsupported_model	The request cannot use tool search.
client_tool_search	The client runs its own tool search and this request is not on a session that defers into it, so its tools went as sent. Also when the client defers tools without a search tool, or its search already covers every rare tool.
no_deferrable_tools, small_catalog	There is nothing worth deferring.
session_unavailable	The session's held mode could not be read in time, so its tools went as sent.

Restore Claude Code's parallel tool calls

Claude Code can end its system prompt with a token budget line, <total_tokens>15000000 tokens left</total_tokens>. Some custom base URL sessions leave it out. Send x-anyray-cc-budget-notice: on to add the line at the end of the last system block when it is missing. Claude Code 2.1.286 often sends the line itself; those requests stand aside as already_present.

An admin can turn it on for every client by setting ccBudgetNotice to on on PUT /admin/v1/settings (default off). A request's own header wins, and x-anyray-cc-budget-notice: off opts that request out.

Measure before you turn it on

It is off by default because it did not lower cost in our paired runs on Claude Code 2.1.286. On a coding task, sessions with the line cost a median 1.21 times as much and none of 8 was cheaper, because the model reads more of each tool result. On a documentation task the parallel tool calls rose and the cost did not fall.

It only touches a Claude Code request (its first system block is Claude Code's billing header line) whose last system block is text, and it keeps that block's cache_control. A request that already carries a <total_tokens> line is sent as is. Other providers and custom hosts get the request as sent.

Every request with a mode in effect gets an x-anyray-cc-budget-notice-result response header, {"applied":true} or a named stand-aside reason such as {"applied":false,"reason":"already_present"}. The result is also stored in the request trace's ccBudgetNotice metadata.

Reason	Why it stood aside
already_present	The client already sent a <total_tokens> notice.
not_claude_code	The first system block has no Claude Code billing line.
last_system_not_text	The last system block cannot receive the notice.
inline_config, custom_host, other_provider	The resolved lane is not eligible.
invalid_json, handler_not_reached	The body cannot be parsed or the rewrite was not reached.

Close the retrieval loop in an agent SDK

When the optimizer elides bulk it leaves a ctx_… handle in the prompt, and the model needs a way to read the original back. The gateway keeps retrieval-dependent strategies off until the request has one of these:

Read path	What proves it
Gateway-served (opt-in)	An API-key request whose class an admin added to servedRetrievalClasses, such as no_discovery: it declares tools but no retrieval tool. Once the conversation carries a marker, the gateway appends anyray_retrieve and anyray_recall, keeps them for the session, and resolves the model's calls itself (up to 3 rounds), removing them from the response. A streaming request still streams; only the tool-call tail waits. The console reports served.
A declared tool	An anyray_retrieve tool the model can call, implemented against POST /connect/retrieve (below).
A registered MCP server	The gateway's POST /mcp, a hosted connector aimed at it, or Connect's stdio server.

No class is opted in by default, so a tool loop with no read path gets no retrieval-dependent trims. Lossless folds and cache optimization can still run; app-state API response schemas may also lose annotations (details). Each such request is recorded as proof, and a runtime whose every request carries it reads parity (Active · forwarded unchanged). On a measured LangGraph tool loop, serving retrieval cost more than forwarding unchanged, so opt a class in only after a paired comparison on your own session shape.

Onboarding requires readPath: "declared" even when its runtime reads parity and problems is 0. absent or partial means the app still needs wiring. A missing readPath needs a gateway upgrade. Probe the app's actual MCP transport too; a separate HTTP probe cannot prove its stdio server started.

Served retrieval stands aside, and the request needs another row, when any of these holds:

• The request's class is not in servedRetrievalClasses (settings). An agent SDK loop is the no_discovery class on every wire.
• The request declares no tools.
• The request uses Codex's WebSocket transport.
• The request goes to /v1/responses without store: false, or with background: true.
• tool_choice is anything other than auto.
• The request already declares a tool named anyray_retrieve or anyray_recall.
• The client already has another read path (a declared tool, a registered MCP server, or a deferred tool catalog), so the gateway leaves retrieval to it. An admin can opt the deferred-catalog and unapproved-Codex classes in through servedRetrievalClasses; both are off by default.
• The key bills a subscription, or the request sends x-anyray-auth-mode: passthrough.
• An OpenAI-wire request sets n above 1.
• A /v1/messages request enables extended thinking.
• An admin set servedRetrieval: false (settings).
• The gateway is older than v1.10.390.

GET /admin/v1/spend/dashboard reports how often each reason applied, under retrieval.servedBlockers: requests per reason, and soleRequests where it was the only one. It counts tool-calling requests whose client has no read path of its own.

The SDK guide has the short version; OpenClaw's is in its guide, and LangChain's in its own.

A workflow of one-shot calls needs none of this. A summarize, extract, write pipeline, a batch job, or an embeddings-only service cannot call tools, so it runs the no-retrieve lane automatically and nothing is elided behind a handle the flow could not resolve.

A source hook is an optional extra: it trims a tool result before it enters the transcript, so the trimmed bytes are what gets cached. For the Claude Agent SDK, use the complete options below. For another tool loop, call the same PostToolUse(output, tool) function before appending each result, and send x-anyray-metadata: {"tool":"sdk-agent"} on its inference requests so the console can identify the lane.

The gateway MCP endpoint

POST /mcp speaks the streamable-HTTP transport: one JSON-RPC message per POST, notifications acknowledged with 202, and a signed Mcp-Session-Id pinning the initial announcement. Instruction and reconnect behavior. Successful initialization and other valid authenticated messages renew the key's 6-hour retrieval lease. Rejected initialization does not claim retrieval capability.

The host that sent the handshake is the process exposing tools to the model, so retrieval-dependent optimization unlocks after initialization without waiting for a retrieval call. The endpoint advertises the retrieval pair only, because the stdio server's local-workspace tools (anyray_search, anyray_read) act on files a remote endpoint does not have.

On a Claude Code or Claude Agent SDK host that also runs anyray-connect, register this endpoint under a name other than anyray in mcpServers. That name is connect's local server, which can read the local transcript; the gateway treats a tool under it as that reader.

Hosted connectors

Zero client-side runtime: the platform calls your gateway from the outside. Two endpoints, one key: /mcp for retrieval, /mcp/org for your organization's connectors. Machines running anyray-connect get both registered for them; an SDK agent registers them here.

Anthropic's Messages API needs all three parts. mcp_servers, a paired mcp_toolset entry in tools for each one (a server entry without one is a validation error), and the beta header:

Request params (header: anthropic-beta: mcp-client-2025-11-20)

"mcp_servers": [
  { "type": "url", "url": "https://<gateway>/mcp", "name": "anyray",
    "authorization_token": "<your-gateway-key>" },
  { "type": "url", "url": "https://<gateway>/mcp/org", "name": "anyray-connectors",
    "authorization_token": "<your-gateway-key>" }
],
"tools": [
  { "type": "mcp_toolset", "mcp_server_name": "anyray" },
  { "type": "mcp_toolset", "mcp_server_name": "anyray-connectors" }
]

Or the OpenAI Responses hosted tools, inside tools:

{ "type": "mcp", "server_label": "anyray", "server_url": "https://<gateway>/mcp",
  "headers": { "x-anyray-api-key": "<your-gateway-key>" }, "require_approval": "never" },
{ "type": "mcp", "server_label": "anyray_connectors", "server_url": "https://<gateway>/mcp/org",
  "headers": { "x-anyray-api-key": "<your-gateway-key>" }, "require_approval": "never" }

In the SDKs, the Anthropic params go on client.beta.messages.create with betas=["mcp-client-2025-11-20"], and the OpenAI tools go on client.responses.create. The console's Add agent dialog prints both calls, and curl, with your gateway and key filled in.

Constraint	Detail
Same public URL	Use the gateway URL the SDK already sends requests to. The gateway credits a connector as the read path only when it targets its own /mcp.
Same key	The connector carries the key the request sends. A pointer opens only for the key that made it, so with no key or another key the gateway leaves that turn's context whole.
Publicly reachable	The platform calls from the outside, so a VPC-private gateway cannot use hosted connectors. Register the MCP server in the agent runtime instead.
First-party APIs only	Per the providers' own compatibility notes, Anthropic's connector is not available on Amazon Bedrock or Google Cloud.
OpenAI: Responses API only	chat.completions has no hosted connector. Its tool loops go through unchanged, or the gateway serves retrieval once an admin opts no_discovery in.

A deployment routing Claude through Bedrock or Vertex loses nothing else. The declared-tool path and the runtime MCP entry are provider-independent: the marker rides the prompt, the tool rides the request, and the agent's own loop calls the gateway directly. The declaration makes the tool selectable, and authenticated MCP activity supplies the live lease.

Frameworks

LangChain and LangGraph have their own guide.

Any other framework built on the OpenAI or Anthropic SDK needs only the base URL and the client key. Most read both from the environment:

export OPENAI_BASE_URL="$ANYRAY_GATEWAY_URL/v1"
export OPENAI_API_KEY="$ANYRAY_CLIENT_KEY"
export ANTHROPIC_BASE_URL="$ANYRAY_GATEWAY_URL"
export ANTHROPIC_API_KEY="$ANYRAY_CLIENT_KEY"

The Claude Agent SDK reads the two ANTHROPIC_* variables. To set the values in code instead:

OpenAI Agents SDK

from agents import set_default_openai_client, set_tracing_disabled
from openai import AsyncOpenAI

set_default_openai_client(AsyncOpenAI(base_url=f"{gateway}/v1", api_key=key))
set_tracing_disabled(True)

LlamaIndex, Vercel AI SDK, LiteLLM and Instructor accept a base URL and key the same way.

Turn off Agents SDK tracing

The OpenAI Agents SDK sends each run's trace, prompts and responses included, to api.openai.com with the key it holds. The base URL does not change that. Call set_tracing_disabled(True), or set OPENAI_AGENTS_DISABLE_TRACING=1 when you use the environment variables.

The Agents SDK calls /v1/responses. The gateway serves retrieval there only when both hold:

• An admin added no_discovery to servedRetrievalClasses.
• The request sends store: false. The Agents SDK leaves store out, so set ModelSettings(store=False) on the agent.

set_default_openai_api("chat_completions") skips the store step; the opt-in still applies (what each lane covers).

Grouping a multi-step run

A graph or multi-step agent makes several calls per run, and each node usually sends its own system prompt. The gateway cannot tell from the wire that those calls belong together, so by default each one becomes its own row on the Sessions page.

Name the step, not the run:

-H 'x-anyray-metadata: {"sessionId":"billing-agent.planner"}'

Every call from that node carries the same value, on every run. One row per step, and each row collects that step's history.

Set it where you build the model, once per node. LangChain, LangGraph (example) and LlamaIndex all pass a default-headers option through to the underlying client.

Do not send a per-run id here

A LangGraph thread_id, a job id, a fresh UUID per invocation: all of these change every run, so each run becomes its own row and the Sessions page stops grouping your steps. A fixed step name keeps one row per step.

On OpenAI models the gateway sets no prompt_cache_key for an app sending tool langgraph, langchain or sdk-agent, because OpenAI caches your shared system prompts without one. Mistral routes cache only with a key, so there, and for every other client on OpenAI models, the key comes from this field and a per-run id sends each run to a cold cache.

Use an opaque id

The value is stored verbatim in the content-free spend and trace stores, so a human-readable title derived from a user's message would put content there. A fixed name you chose is fine; anything built from request data is not.

Sessions are derived without the header too, so a call that sets nothing still lands somewhere sensible. That derivation keys on the conversation prefix, which is why separate nodes do not share a session on their own.

Plain HTTP tools (your own tool loop, no MCP)

For an agent that runs its own tool loop with no MCP support. Three independent things must be true:

1

The declaration

Proves the model can select the tool on this turn. It matches on the tool name (anyray_retrieve, or an mcp__…__anyray_retrieve-style namespaced form), so the name is load-bearing and must not be aliased.

2

The implementation

Against POST /connect/retrieve. This is what makes the handle resolvable once the model calls it.

3

The liveness signal

Proves the implementation is running. Send an authenticated empty POST /connect/mcp-heartbeat when the worker starts, and every 5 minutes while it can execute tool calls. Stop heartbeats before shutting the executor down. The lease expires after 6 hours without verified activity.

A declaration qualifies for that turn. The lease covers later turns whose tool catalog is deferred.

The declaration shape follows whichever dialect you already call. Both satisfy the gateway:

Anthropic dialect

Anthropic dialect

ANYRAY_RETRIEVE_TOOL = {
    "name": "anyray_retrieve",
    "description": (
        "Fetch the full original text behind an Anyray handle. When a message "
        "contains a marker like [anyray: ... retrieve ctx_abc123], pass that "
        "ctx_ handle here to read what it replaced."
    ),
    "input_schema": {
        "type": "object",
        "properties": {"handle": {"type": "string"}},
        "required": ["handle"],
    },
}

OpenAI dialect

OpenAI dialect

ANYRAY_RETRIEVE_TOOL = {
    "type": "function",
    "function": {
        "name": "anyray_retrieve",
        "description": (
            "Fetch the full original text behind an Anyray handle. When a message "
            "contains a marker like [anyray: ... retrieve ctx_abc123], pass that "
            "ctx_ handle here to read what it replaced."
        ),
        "parameters": {
            "type": "object",
            "properties": {"handle": {"type": "string"}},
            "required": ["handle"],
        },
    },
}

The description is doing real work. The gateway unlocks retrieval on the name, but the model only calls the tool if it knows what a ctx_… marker is. Then implement it, in either dialect:

Implement

def anyray_retrieve(handle: str) -> str:
    r = requests.post(
        "http://<gateway>:8787/connect/retrieve",
        headers={"x-anyray-api-key": "<your-gateway-key>"},
        json={"handle": handle},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["content"]

The response is {"status":"ok","content":"…","handle":"ctx_…"}. startLine, endLine, and grep are optional body fields for reading back a slice, and an unknown or expired handle answers 404. The stdio server below wraps this same route, so neither path is more capable than the other.

Stdio MCP (Connect already installed)

Where anyray-connect is installed, its stdio server is equivalent to the remote endpoint, being a wrapper over the same routes, and also serves the local-workspace tools anyray_search and anyray_read. For the Claude Agent SDK's TypeScript query options, the complete server entry is paired with its source hook:

Claude Agent SDK options

settingSources: ['user', 'project', 'local'],
mcpServers: { anyray: { type: 'stdio', command: 'anyray-connect', args: ['__anyray-mcp-server', 'claude'] } },
settings: { hooks: { PostToolUse: [{ matcher: '*', hooks: [{
  type: 'command', command: 'anyray-connect __anyray-hook --managed-source', timeout: 3
}] }] } }

The hook trims before the result enters the transcript. It requires successful retrieval in the current main transcript and checks that the returned handle reads back the whole original. Missing proof, a failed request, or a two-second deadline leaves the original untouched. Restricted subagents stand aside. Managed hosts without a binary receive a native HTTP hook; its event has no callable-tool evidence, so the gateway currently returns a no-op. No key-level latch authorizes that lane.

doctor --json and status --json report sourceHook.installation and sourceHook.retrieval separately. Retrieval reports expire after one minute and never authorize a trim.

The Agents row polls its key record. Active · observing source trim is the grace state; a successful trim changes it to Active once retrieval is connected. Active · source trim not seen means at least seven days between the start of observation (or last trim) and the latest eligible inference request, with at least 100 successful inference requests since then. Idle time alone cannot trigger it. This diagnoses absent evidence, not a promise that a workload must trim. Only Claude Code has an automatically installed source hook today. sdk-agent identifies an explicitly integrated harness; other client tools report source-hook support as unavailable. Older gateways show Needs gateway upgrade instead of diagnosing the host.

This mounts the exact stdio command anyray-connect __anyray-mcp-server claude. The executable must be a durable Connect install on the agent process's PATH, with an enrolled profile using the same key and gateway as the model requests. On MCP initialization the server exposes anyray_retrieve and anyray_recall. Connect renews that evidence every 5 minutes while the initialized stdio transport is alive, and the gateway expires it 6 hours after the last accepted heartbeat or model retrieval call. claude reads this machine's Claude Code settings, so it fits a developer machine that loads them. An app host has no Claude Code settings files: register sdk there, as below.

Custom SDK callback

For an explicitly integrated sdk-agent harness, use this callback with its MCP read path.

Source hook and read-back together

import type { Options } from "@anthropic-ai/claude-agent-sdk";

const gateway = "https://gateway.example"; // deployment's public gateway URL
const headers = { "x-anyray-api-key": process.env.ANYRAY_CLIENT_KEY! };

async function PostToolUse(output: unknown, tool: string): Promise<unknown> {
  if (typeof output !== "string" || !output) return output;
  try {
    const response = await fetch(gateway + "/connect/optimize-output", {
      method: "POST",
      headers: { ...headers, "content-type": "application/json" },
      body: JSON.stringify({ output, tool, reversible: true, retrievalContext: "available" }),
      signal: AbortSignal.timeout(5000),
      redirect: "error",
    });
    if (!response.ok) return output;
    const result = await response.json();
    return result.optimized === true && typeof result.output === "string"
      ? result.output : output;
  } catch { return output; }
}

const anyrayOptions: Options = {
  mcpServers: {
    anyray: { type: "stdio", command: "anyray-connect",
      args: ["__anyray-mcp-server", "sdk"] },
  },
  hooks: {
    PostToolUse: [{ hooks: [async (input) => {
      if (input.hook_event_name !== "PostToolUse") return {};
      const output = await PostToolUse(input.tool_response, input.tool_name);
      return output === input.tool_response ? {} : {
        hookSpecificOutput: { hookEventName: "PostToolUse", updatedToolOutput: output },
      };
    }] }],
  },
};

The sdk source reads the Connect profile, like profile, but lists only anyray_retrieve, anyray_recall and anyray_read_batch. At startup it also tells the model to read located ranges with one anyray_read_batch call, while that tool is listed. Set clientToolPolicies.readBatchInstructions to false to drop that line. An older Connect reads sdk as auto, which on a host with no Claude Code settings files also resolves the profile, but it lists anyray_search and anyray_read too and gives no startup line.

Keep the key in the host's existing secret environment. The authenticated POST /connect/optimize-output contract accepts {"output", "tool", "reversible", "retrievalContext"}. Send reversible: true and retrievalContext: "available" only when the harness mounts anyray_retrieve, as this one does: the response is then {"optimized": true, "output": "…"} with retrieve ctx_… handles that MCP reads back. A request without both returns {"status": "ok", "optimized": false, "output": "…", "handled": true} and records no savings. (Connect's own hook also has a ranged lane that relies on a local tee file; an SDK callback has no tee, so send the reversible flag.) Empty inputs and diagnostic probes may omit output. Preserve the original on a no-op, timeout, HTTP error, or invalid replacement. Remove the hooks entry to stop source trimming.

How the model knows when to call it

You do not need to teach the happy path. Every elision leaves a marker in-band, at the spot the content was, and the marker names the verb:

[anyray: observation masked · 330509 chars · retrieve ctx_9f3a2b via anyray_retrieve]

Do not rewrite the marker or the tool description

This wording is load-bearing and measured. Before the marker named the tool, the fleet resolved 0.4% of emitted handles. Naming anyray_retrieve outright is what made models call it.

Reinforce it with a system-prompt line

This is the only teaching channel that reaches every runtime, and the main way to surface anyray_recall, which nothing in-band points at once a marker has scrolled out of view or been compacted away.

Some earlier content is stored behind [anyray: … retrieve ctx_…] markers.
Call anyray_retrieve with the ctx_ handle to read it back; if you remember
content that is no longer visible, find its handle with anyray_recall.

Keep it byte-identical across turns, because it sits in the cached prefix.

For a judgment task (a review agent, an auditor, a grader) make the line task-directive. An action agent that is missing bytes gets stuck and retrieves. A judge can assess the outline, produce something plausible, and never notice, so the failure mode is a shallow verdict that looks complete. Set the standard, not just the mechanism:

Never assess code you have not read in full. If a file you are commenting on
is behind an [anyray: … retrieve ctx_…] marker, call anyray_retrieve first.

Two mechanisms already reduce how often this fires. Files the live turn names, and files the agent is editing, are kept whole rather than outlined. Files the agent already evaluated are spared on later turns. The directive is the backstop for content the task reaches but did not name.

Reinforce it with an OpenClaw skill

OpenClaw loads markdown skills that teach tool use, so a minimal anyray-retrieval/SKILL.md with the same two sentences does the same job there. Not required, because OpenClaw models see the same in-band markers.

Read-only workflows

Opt-in experiment

For dependent reads in a TypeScript tool loop, import anyray-connect/workflows (Node 24 or later). The managed helper reads the calling key's strategy policy once, then executes a fixed dependency graph and returns every step's exact JSON result, or a count and top-k summary for the step you project. Turn on Read-only workflows in Console → Optimizer first. It does not contact an LLM, start the CLI or add an MCP tool.

Install the optional helper

npm install anyray-connect zod

Synthetic workflow

import { createManagedReadOnlyWorkflow } from 'anyray-connect/workflows';
import { z } from 'zod';

const gateway = process.env.ANTHROPIC_BASE_URL;
const apiKey = process.env.ANTHROPIC_API_KEY;
if (!gateway || !apiKey) throw new Error('Set the Anyray base URL and client key.');

const workflow = await createManagedReadOnlyWorkflow({
  enabled: true,
  name: 'account_context',
  description: 'Look up an account and read its invoices and credits together.',
  inputSchema: z.object({ customerRef: z.string().min(1) }).strict(),
  onEvent: (event) => console.info(event),
  tools: {
    lookup: { readOnly: true, execute: async (args) => ({ id: 'synthetic-7', args }) },
    invoices: { readOnly: true, execute: async () => [{ amount: 120, status: 'open' }] },
    credits: { readOnly: true, execute: async () => [{ amount: 10 }] },
  },
  steps: [
    { id: 'account', tool: 'lookup', arguments: ({ input }) => input },
    { id: 'invoices', tool: 'invoices', dependsOn: ['account'],
      arguments: ({ results }) => results.account },
    { id: 'credits', tool: 'credits', dependsOn: ['account'],
      arguments: ({ results }) => results.account },
  ],
}, { gateway, apiKey });

if (workflow) {
  const tool = {
    name: workflow.definition.name,
    description: workflow.definition.description,
    input_schema: { ...workflow.definition.inputSchema, type: 'object' as const },
  };
  // Register `tool` in your tool list and dispatch it to workflow.execute.
  const result = await workflow.execute({ customerRef: 'synthetic-customer' });
}

Create the workflow once when a new conversation starts. Local enabled: false, an org Off policy or unavailable policy returns undefined; keep your original tools for that session. Keep the same definitions on every turn, including gateway bypass turns. Changes apply to the next session. The calling key needs no admin credential.

Users → Client strategies shows the helper's last availability report and its time. It does not confirm registration, dispatch, completed steps or savings. A failed report does not break a workflow.

Register trusted read handlers

readOnly: true is your application's declaration. The helper cannot prevent a handler from writing. Register only handlers you control, enforce their authorization, and pass the supplied AbortSignal to their I/O. Argument callbacks are trusted application code; never accept code or workflow configuration from the model. No retries run automatically.

Option or behavior	Contract
inputSchema	A Zod schema supplies both runtime validation and the exported JSON Schema. Use an object schema for model tools. Input, arguments and results must be JSON data; invalid values are rejected without coercion. Callbacks and handlers receive detached, deeply frozen snapshots.
steps	At most 16 steps and 32 explicitly read-only handlers. Unknown tools, missing dependencies, duplicate IDs and cycles fail before handler I/O. Callbacks see only their declared dependency results.
maxConcurrency	Defaults to 2, from 1 to 4. Ready steps run together; dependent steps wait. Results follow step declaration order.
timeoutMs	Defaults to 10,000, from 1 to 60,000. execute(input, { signal }) also accepts caller cancellation. Failures abort running handlers and stop queued steps.
maxResultBytes	Defaults to 65,536 UTF-8 bytes for the complete result, including keys; at most 1,048,576. Oversized results fail without truncation.
project	Optional. Returns one step as a { matched, top } summary instead of its rows. See Count and top-k projection below.
onEvent	Optional callback with fixed status, code and projection, and numeric steps, completed, elapsedMs. projection is applied on a successful run with project, failed on projection_failed, otherwise none. No names, inputs, results or raw errors. Observer failures do not affect execution. The managed factory separately reports availability and policy revision.
Errors	ReadOnlyWorkflowError.code: invalid_config, invalid_input, invalid_arguments, invalid_result, result_too_large, step_failed, aborted, timeout, projection_failed. Errors carry no upstream cause or message.

Timeouts bound asynchronous execution after input validation, so synchronous code must yield to the event loop. A handler that ignores cancellation can continue its I/O after the helper rejects.

Check the workflow result and its onEvent outcome in your app, and keep result content out of logs. Measure complete paired sessions, including tool-definition cache writes, before enabling it for a workload. The helper adds no booked savings credit.

Count and top-k projection

Set project to return a summary of one step's array result instead of its rows. Dependent steps still receive the raw rows.

Count failures, return the five slowest

project: {
  kind: 'count_and_top_k',
  step: 'incidents',
  where: { field: 'status', equals: 'failed' },
  sortBy: { field: 'durationMs', order: 'desc' },
  tieBreak: { field: 'id', order: 'asc' },
  limit: 5,
  fields: ['id', 'service', 'durationMs'],
},

Field	Rule
kind	'count_and_top_k'.
step	A declared step ID. Its result must be an array of JSON objects.
where	{ field, equals }. A record matches when its own field strictly equals equals: a string, finite number, boolean or null. A missing field never matches null.
sortBy	{ field, order } with order set to asc or desc. Every match needs its own value, a string or finite number, one type across all matches. Strings compare by UTF-16 code unit.
tieBreak	Optional, shaped like sortBy. Remaining ties keep the step's result order in both directions.
limit	An integer from 1 to 50: the most records in top.
fields	1 to 16 distinct names copied into each top record in this order. Every match needs them as its own properties. Integer-like names such as '0' are refused.

The step's entry becomes { matched, top }, and matched counts every match. The raw rows and the returned result must both fit maxResultBytes. The contract must be plain data: getters, proxies and extra keys fail with invalid_config before any handler runs, and later edits to your object change nothing. A result that breaks these rules rejects with projection_failed; no partial summary is returned.

Offline workflow construction

The original synchronous createReadOnlyWorkflow(options) export remains available. Its application-supplied enabled flag controls it; it does not read console policy or send availability reports. Use the managed factory for central strategy controls.