Skip to main content

Developer FAQ

Quick answers for developers using an Anyray-connected tool or SDK.

Quality and reliability​

Will my answers get worse?

No. That's the core design constraint. The optimizer hook fails open (800 ms timeout): if it's down or slow your request is forwarded unchanged. The worst case is "you paid full price," never "you got a worse answer."

Does streaming still work? Do tool calls still work?

Yes. Streaming, tool/function calls, and the response shape are the gateway's responsibility and are unchanged. The optimizer only rewrites the request (params/messages/tools), or serves a cache hit.

Will it add latency?

Generally no, often less. Cache hits skip the provider. The one added step is the /v1/optimize call, which fails open (800 ms timeout) if the optimizer is slow or down. Spend is recorded by the gateway's own store and never blocks your response.

What happens if the gateway itself is down?

Your tools stop working until it's back. Every request goes through the gateway (that's what makes spend governable) and nothing on the developer's machine routes around it. Run more than one replica if you need the outage window closed.

Getting connected, and removing it​

Do I have to change my code or SDK?

No. You keep your OpenAI / Anthropic SDK and your request shapes. Set the base URL yourself, or run npx anyray-connect@latest to configure Claude Code, Codex, Cursor's OpenAI override, Devin Desktop via its in-IDE ACP agents, OpenCode, OpenClaw, VS Code Copilot Chat, the GitHub Copilot CLI (BYOK env), JetBrains AI Assistant, Claude Desktop third-party inference, and your shell/SDK env. Connect uses only each tool's application-owned seam, never installs a CA or changes machine networking, and leaves hard-coded first-party endpoints untouched.

Is there a CLI to point my tools at the gateway?

Yes. With enterprise SSO, copy the organization command from Users → SSO enrollment:

macOS / Linux
curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- --sso https://app.anyray.ai/sso/<tenantId> --yes

It signs you in, mints an IdP-bound personal ark_… key, and configures every supported tool it detects; the real provider key stays server-side. No SSO? Ask your admin for an enrollment link (enl_…) and run curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- --enroll <link> --yes. The apply is idempotent, previews with --dry-run, and undoes itself with --revert. On an MDM-managed device you may never run a command at all: your first prompt shows a one-time sign-in URL, and enrollment happens in the background.

Full flows: SSO enrollment · Passwordless enrollment · CLI commands · MDM enrollment.

How do I remove Anyray from my machine?
anyray-connect --revert                # installed CLI
npx anyray-connect@latest --revert     # no install needed

# through the installer: the flag goes after `sh -s --`, not on the curl line
curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- --revert

Then restart your editor and your terminal, and check with npx anyray-connect@latest doctor: No problems found means you're done.

The restart is the step people skip

Editors load Anyray's hooks at startup and shells export its settings when they start; neither notices the files disappearing underneath them. VS Code needs a full Cmd+Q, not a Reload Window.

The revert removes every tool's routing and credential config, the hooks and MCP registrations, your stored credential, the managed shell block, and the key-renewal scheduler; anything you set before Anyray is put back, not deleted. Reverting removes your credential, so reconnecting means enrolling again. On an MDM-managed device the org's system-level profile re-enrolls you on your next prompt; that's policy re-applying itself, not a failed revert. Per-tool detail and fleet-managed machines: CLI commands → Reverting.

After reverting I get `401 valid client key required` on a loop

A terminal that was already open when you reverted is still exporting the gateway. Your tools keep routing there, but the credential is gone, so every request 401s while every file on disk reads perfectly clean. Fix that terminal:

unset OPENAI_BASE_URL ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN OPENAI_API_KEY \
  ENABLE_TOOL_SEARCH _CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL

Or open a new terminal, except in VS Code, which needs a full Cmd+Q (it hands every terminal the environment it read at startup). anyray-connect doctor, run in the terminal that's failing, detects this and prints the exact line; its --json report lists variable names only, never values, so it's safe to share.

Desktop apps and coding tools​

Why did Connect change my Claude Code model id and add an env var?

Claude Code only looks a model's real context window up when it talks to api.anthropic.com directly. Behind any custom base URL it budgets a flat 200,000 tokens and auto-compacts at roughly 95% of that, so a seat on a 1M-context model would silently lose 800k of window. Connect restores it two ways:

What Connect writesWhy
env._CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URLLets Claude Code use its own per-model windows again.
[1m] appended to settings.modelCovers models that reach 1M through Claude Code's long-context option rather than natively.

Both are read locally, neither changes how you sign in or where requests route, and anyray-connect --revert removes both. One request detail: the flag also makes Claude Code mark its system prompt with a prompt-cache scope value only a subscription sign-in is entitled to; Anyray strips that mark on every other lane, or an org-API-key seat would get system.2.cache_control.ephemeral.scope: Extra inputs are not permitted as a 400 on every request.

If the context meter looks stuck at 100%

That is the un-restored behavior: a 1M model clamped against a 200k budget. Re-run Connect, then fully restart Claude Code; it reads its environment once at startup.

Desktop apps: what routes through Anyray?

npx anyray-connect@latest covers the local tools that read normal base-URL config, listed above under "Do I have to change my code or SDK?". Setup is configuration-native and never changes system networking. The boundaries that surprise people:

  • Claude desktop app. The first-party subscription endpoint is not configurable and stays untouched. On verified macOS builds, Connect configures the third-party gateway mode automatically (on the Mac it can ride your Claude Code subscription credential through a Keychain-backed helper); any unverified build gets the manual, API-billed Developer → Configure third-party inference flow instead, never a silent fallback. Windows is API-billed and fails closed when subscription is selected. Detail: Claude Desktop.
  • Codex. The app, CLI, and IDE extension share ~/.codex/config.toml, so one adapter covers all three. A model picker missing custom-provider models: set ANYRAY_CODEX_MODEL and re-run Connect. Codex app.
  • OpenCode. Its built-in ChatGPT OAuth has no endpoint override; use Codex for ChatGPT subscription pass-through (OpenCode).
  • GitHub Copilot CLI. Routed via GitHub's BYOK env contract at the gateway's Copilot carrier route; billing stays on the Copilot plan, and a seat with no readable sign-in is held rather than silently moved to org billing (Copilot).
  • JetBrains / Devin. Connect registers pinned Claude and Codex ACP carriers; the native AI/Copilot/Devin clients remain separate.
  • Cursor Team. Native inference stays Cursor-hosted; Connect installs user-level hooks and retrieval MCP only. Only explicit --org selects the API-key BYOK lane (Cursor).

Fleet admins can emit credential-free enterprise artifacts for Claude Desktop and Codex; see Managed fleets (MDM). Cloud and remote sessions (Claude web, Claude Slack, Codex cloud tasks) run on provider infrastructure and cannot route through a local gateway.

Can it route apps with no base-URL setting?

No. anyray-connect routes only through a supported application seam. Apps that hard-code their endpoint (the ChatGPT desktop app, Claude Desktop's first-party subscription lane, native Copilot/Devin/Cursor surfaces) stay untouched. No CA, DNS rewrite, or machine-level proxy is involved.

For optimization, use a supported carrier: Claude Code, the Codex app/CLI/IDE, the Anyray VS Code provider, or the pinned ACP agents in JetBrains/Devin. Cursor's native subscription mode still optimizes its local Shell/MCP context through hooks, but not the hidden model request; for visibility into native usage, a subscription spend connector imports the vendor's own metrics.

Does Anyray routing apply to my subscription seat?

For Claude Code and Codex, yes: subscription pass-through keeps the client's provider OAuth token and request shape, so the seat keeps working while the request crosses the gateway. That traffic gets enrollment, attribution, policy checks, and supported optimizer savings, but not routing to a different provider. Cursor Team exposes no Anthropic override, so its native requests never cross the gateway; Connect installs local hooks and retrieval only (Cursor).

Provider routing applies to API-key traffic: server-held keys, anyray-default, and SDK requests that ask Anyray to pick the model. Orgs where each developer holds a personal key for their own upstream use the BYO-upstream lane (anyray-connect --upstream <url> --upstream-key <personal-key>): the key passes through verbatim on both the Anthropic and the OpenAI lane, to an upstream that must be allowlisted via ANYRAY_CUSTOM_HOST_ALLOWLIST (LiteLLM). Routing headers you set on a tool's config yourself are left alone by anyray-connect, credential included.

Metering has two lanes: included usage is billingMode: subscription; recognized provider response metadata refines paid credits/overage to subscription-extra (full real cost and savings). Missing quota metadata stays in the conservative included lane.

API and SDK usage​

What base URL do I point my SDK at, and which endpoints work?

Point your SDK at the gateway, http://localhost:8787/v1/... (or your gateway's host), exactly as you would at OpenAI. It's OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, …) and also speaks Anthropic natively at /v1/messages (plus /v1/messages/count_tokens). The auxiliary endpoints (/v1/images/*, /v1/audio/*, /v1/responses, /v1/files, and /v1/batches, the async Batch API) are served only for the openai and azure-openai providers (audio also works on groq). See the API reference and the gateway.

What API key do I put in my SDK?

A minted personal Anyray client key (e.g. ark_…), never a real provider key. Your provider keys live server-side in the gateway and are never exposed to clients; the gateway attaches the real key when it forwards your call. See server-held provider keys.

How do I attribute requests to a user or team?

Send the x-anyray-metadata header with fields like user, team, and session. It drives spend attribution, per-user caps, and rule-based strategy overrides, and never carries prompt or response content. See Configure.

Which providers can I call through it?

It's multi-provider out of the box (openai, anthropic, bedrock, vertex-ai, azure-openai, groq, deepseek, mistral-ai, nebius, tensormesh, openrouter, x-ai (Grok), oracle (OCI Generative AI), litellm and truefoundry (your own gateway), and more), with Anthropic, Vertex, and Bedrock spoken natively. You reach any of them through the one OpenAI-compatible base URL; an OpenRouter key fronts its 250+ models (anthropic/claude-3.5-sonnet, openai/gpt-4o, …). See the gateway.

Optimization and savings​

Is response caching on by default?

Yes, semantic_cache is default-on. On a duplicate request it serves the stored response and skips the provider (non-streaming, for callers that can short-circuit), which is faster and free. The volatile-normalized cache tier is measurement-only: it records would-have-hit precision but never serves a response. See the strategy menu.

Will tool pruning drop my MCP tools?

No. Namespaced MCP tools (mcp__…) and the tool-search surface are never pruned. tool_pruning only trims other tools whose name and description both share nothing with the conversation; on prompt-cached and subscription traffic it runs via a decide-once/replay session pin. A separate default-on strategy, tool_schema_compression, shrinks the descriptions of kept tools (MCP included) without dropping any; it is lossless with default params and rewrites to the same bytes every turn, so it is cache-neutral.

When Anyray trims a long tool output, can the model get the detail back?

Usually yes. A trim leaves a · retrieve ctx_… marker, and the model pulls the original back through the anyray_retrieve MCP tool that Connect registers (with anyray_recall to find a marker that scrolled away by meaning, and anyray_history when an imported archive exists). Retrieval accepts startLine/endLine and grep so "find the error in the 4,000-line log" costs ~30 lines instead of 4,000; retrieved content goes straight back to the asking model and is never logged. Humans get the same read path: anyray-connect retrieve ctx_… and anyray-connect recall "what to find" (CLI commands).

In an agent you build yourself, register the gateway itself as a remote MCP server (POST /mcp) or declare a tool named anyray_retrieve against POST /connect/retrieve. The declaration proves the tool is callable on that turn; authenticated MCP, heartbeat, or model retrieval activity also supplies the lease used when later turns defer the tool catalog. The name is load-bearing, so an alias turns retrieval off. See Retrieval reference.

Capability follows the developer's machine, not the credential: a freshly-minted key inherits the same user's still-fresh evidence from the same device, another machine must prove its own loop, and evidence decays 6 hours after the last verified activity. Connect sends a heartbeat after MCP initialization and every 5 minutes while the stdio transport is alive; valid MCP and model retrieval calls renew the same lease. Recall needs the durable stash, so it returns no matches when the content mode is off.

How are the token-savings numbers estimated?

Two numbers, two sources. The cost of each request is exact, priced from your provider's own reported usage at the official per-model rate. Only the saved-token delta is estimated, because the optimizer computes it before the provider responds, on text it's removing:

  • OpenAI models. Exact local BPE counts: GPT-4o / 4.1 / 5 and the o-series use o200k_base; GPT-4 / 3.5 use cl100k_base.
  • Everything else (Claude, Gemini, …). A chars→tokens heuristic calibrated per request: scaled by the ratio of the provider's reported prompt tokens to the heuristic count of that same prompt, so the estimate is anchored to how the model actually tokenized this request. When the ratio can't be trusted (no usage returned, or a large inline image/audio payload), it falls back to the plain heuristic.

Both paths run once, post-response, off the request path: no provider token-count call, no added latency. The same number feeds both the billing spend record and the dashboard, so the two always agree. One exception: window_budget uses the shared CJK-aware estimator with a configurable charsPerToken calibration (default 3.3) for cropping safety, since that calibration decides which whole messages get evicted; it stays operator-tunable and separate.

Privacy and admin​

Does my prompt data leave the company?

Anyray is self-hosted. Content travels only on the customer-selected inference path: through your gateway/optimizer to the selected model provider. Locally captured content follows the org content mode (encrypted by default, off, or deploy-gated plaintext); the spend store and logs carry no bodies, and the console follows the mode plus RBAC. The only automatic egress to Anyray is content-free usage metering. A subscription spend connector, if an admin enables one, adds customer-authorized HTTPS calls to that vendor's official admin API. Full map: Data boundary.

What does the admin see in the user roster?

The console's Users page shows one row per user (keyed by email + deployment):

FieldMeaning
StatusEnrolled (personal key issued), Pending (link minted, not used), Revoked, or Expired (link TTL passed).
SourceIdentity provenance: manual, provisioning, sso, scim, email, mdm, device, or service. device is the per-Mac certificate path.
Typeperson or service (machine account).

Regenerate mints a new link and discards the old one (shown once); Revoke stops an active link, though issued keys run to expiry; Delete removes the row and offboards no one, because the machine's DevCert keeps minting keys. To cut access off, revoke the user (POST /admin/revoked-users), then their live key (DELETE /admin/client-keys/:id). See Passwordless enrollment.