Troubleshooting

Start on the console Health page: account menu → Health (/console/health). It lists open problems and failing checks, each with what to do about it.

Check deployment health

/console/health is not on the sidebar; open it by URL. When something needs an admin, an Errors item appears in the sidebar. It lists each problem on the left, grouped by where the fix is. Pick one to see what is wrong and what to do. Open goes to the page with the fix. For a person, that is their Machine status tab on Users. A dismissed problem stays hidden for 24 hours, or until the problem changes. The page's checks come from /admin/health, which probes every leg live and tells not configured apart from configured but unreachable:

Check deployment health

curl -fsS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN"

200 means every required leg is up. 503 means a required leg is down, named in the body. A down optimizer or a lagging schema never fails the check.

Leg	What it checks	Required	If it's the failing leg
gateway	The process is serving requests	No	A response at all means it's up
observability	Trace store reachable, or intentionally not configured. cause carries the error code when it fails	When configured	Check ANYRAY_OBSERVABILITY_DB_URL and the Langfuse stack
spend	Spend store (Postgres) answers a live SELECT 1. spend.auth: password or rds-iam; spend.cause carries the error code when it fails	Yes	Check ANYRAY_SPEND_DB_URL and that Postgres is reachable
pools	Per-process connection budget and max, total, idle, waiting for pools already built. Missing pools have not opened yet	No · informational	Check waiting when Postgres connections are exhausted
schema	Boot migrations applied. state places the database relative to this image; jsonbChecks counts JSON column constraints	No · informational	See the schema leg reports drift
optimizer	Optimizer /health reachable, plus configStore (settings) and retrievalStore (durable handle writes)	No · fails open	Check the optimizer service and ANYRAY_OPTIMIZER_URL
inference	A working way to serve: a provider key (configuredProviders), an upstream you run (byoUpstream), or inference already served (servedRequests). defaultRoute says whether header-free requests have a route	No · informational	See inference.ok is false
portal	Billing App metering, entitlement lease status, lease provenance (leaseSource, leaseTrust, leaseAcquire)	No	Check the Billing App connection (configure); see a replica answers 503 on every request
secrets	How the admin token, provider-key store, stored routing credentials and content key are configured. Names a state, never a value	No · informational	See Console sign-in fails and stored provider keys are gone

If the optimizer or gateway is down

If only the optimizer is down, leave clients pointed at the gateway. It starts without waiting for optimizer health and forwards requests without optimization when the bounded optimizer hook fails. Spend attribution and provider routing remain on the gateway. Check the optimizer health leg and restart or repair that service when convenient.

If the gateway itself is down, an enrolled developer can switch supported tools on their own machine, provided each has its own provider credential or native subscription:

anyray-connect fallback
anyray-connect fallback status
# Once the gateway answers again:
anyray-connect fallback --revert

This currently covers machines whose managed tools are Claude Code and/or Codex. It changes local configuration once; no local agent or proxy monitors requests. It disables Connect-owned hooks and MCP registrations, including in an already-running session's hook entrypoints, before removing gateway routing. Restart editors and terminals after switching. Start a new conversation if the old one contains ctx_ handles, which need the gateway retrieval service. A managed setting that still pins the gateway needs an administrator to remove it. Direct calls are absent from live Anyray spend and policy reports; reconcile their cost from provider records. The plain anyray-connect --revert command is full offboarding and clears enrollment, so use the fallback --revert form to return after an outage.

inference.ok is false

No server-side provider key, no upstream you run, and no successful inference since the replica booted. Everything else can be green while every request is refused. Org-key developers get:

424  no provider key configured for anthropic: the request authenticated with
     a gateway client key, but the gateway holds no server-side anthropic API
     key to forward; add one in the console Providers page

The developer's credential is fine, so re-running anyray-connect won't help. Add a provider key in the console → Providers. If only some developers fail, those are the org-key ones; same fix.

• A subscription-only deployment (Copilot or Claude subscription seats) is healthy: once it has served anything, servedRequests keeps ok true.
• servedRequests counts 2xx on inference paths since boot. Zero on a fresh or idle replica is not a problem; it can only make ok true.
• configuredProviders is a count, never vendor names; null when the probe could not open the provider store.
• defaultRoute is separate from ok: two keys and no default route keeps ok true while every header-free client is refused. See Requests are refused for a missing routing header.

Common issues

Gateway and deployment

/admin/health returns 503: a required leg is down

Only spend (always) and observability (when configured) can turn the check red. The failing leg has ok: false and a cause holding the database's error code.

Which leg failed, and why

curl --fail-with-body -sS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" | jq '{spend, observability}'

Use --fail-with-body (curl 7.76+), not -f: the 503 body is the health report, and -f discards it.

cause	Fix
28P01	Password rejected. If the credential rotates, restart the gateway; it reads it at boot
ENOTFOUND / ECONNREFUSED	Host unreachable. Confirm ANYRAY_SPEND_DB_URL (or ANYRAY_OBSERVABILITY_DB_URL) and that Postgres accepts the gateway's connections
25006	The spend store is a read-only standby and refuses every write. Point ANYRAY_SPEND_DB_URL at the primary and restart the gateway
57014	Query timed out. The store is reachable but slow or saturated
unreachable / unclassified	Generic: check the URL, the credential, and that Postgres accepts connections

No cause means the leg is healthy, or the gateway predates the field. Not running observability? Leave it unconfigured; it reports not configured and never gates. Reachable and still failing: send a support bundle.

One gateway replica answers 503 on every /v1 request, and its readiness check fails

After a rollout, one pod is Running but not Ready, /v1/* returns 503, and GET / returns AI Gateway awaiting entitlement lease. A metered gateway serves only while it holds a signed entitlement lease; replicas share it through the spend database, so a new pod normally adopts a peer's lease within seconds. Readiness fails only with no lease at all; an expired or suspended lease stays Ready and answers 402/403.

Check the portal leg on that pod (on Compose, docker compose exec gateway instead of kubectl exec):

Lease provenance on one replica

kubectl -n "$ANYRAY_NAMESPACE" exec <gateway-pod> -- sh -c \
  'wget -qO- --header "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" localhost:8787/admin/health' \
  | jq .portal

Field	Meaning
lease	active, grace, suspended, or null (none held)
leaseSource	db (shared by a peer), file (this pod's cache), cp (the Billing App)
leaseTrust	Last lease read: trusted, bad_signature, or deployment_mismatch
leaseAcquire.lastVia / lastOutcome / lastStatus	Last Billing App attempt: meter or entitlement, ending in lease, throttled, rejected (untrusted lease), http_error (status in lastStatus), or network (never connected)
leaseAcquire.retryInMs	Next attempt after a throttled or leaseless outcome (null on the regular cadence)

You see	Fix
leaseTrust: deployment_mismatch	The lease is bound to another deployment (two gateways on one adt_ token, or a shared spend database). Give each its own token and database
leaseTrust: bad_signature	The lease does not verify against the pinned Billing App key
lastOutcome: throttled, lease: null persists	The Billing App is shedding load; the pod retries at the given Retry-After. Leaseless for more than a few minutes: send a support bundle
lastOutcome: network	No path to the Billing App. Check egress, plus any forward proxy or TLS inspection: Behind a forward proxy
http_error, lastStatus: 401	The deployment token is invalid or revoked

Console sign-in fails and stored provider keys are gone after a deploy

Check secrets on /admin/health. The gateway keeps serving in every state below.

Field	Value	What to do
adminToken	env_blank	ANYRAY_ADMIN_TOKEN reached the pod empty, usually a GitOps sync that stripped its valueFrom. Admin-token sign-in and saving stored secrets are closed; SSO, email and admin API key sign-ins work. Restore the secret reference and check the Secret still holds the key
adminToken	placeholder	No admin token set; same closures. Set ANYRAY_ADMIN_TOKEN; stored secrets move to it as they are next written
providerKeys	file_only	Keys served from this pod's local file; none can reach the shared store without a private admin token. Set ANYRAY_ADMIN_TOKEN; keys move to the shared store on the next restart
providerKeys	unreadable	The current admin token cannot open the stored keys. Restore the previous token. Re-entering a key on the Providers page replaces every stored key for all replicas, so the page asks you to confirm first
routingCredentials	unreadable	The current admin token cannot open a routing target's stored credential, so that target answers 424. Restore the previous token, or replace the target's credential with a provider key on the Routing page
contentKey	malformed	ANYRAY_CONTENT_KEY is not 64 hex characters, so content encryption is off. Generate one with openssl rand -hex 32

Without a private ANYRAY_ADMIN_TOKEN, saving optimizer settings and purging the optimizer's stash also fail for every sign-in. On Kubernetes, kubectl get deploy anyray-gateway -n "$ANYRAY_NAMESPACE" -o yaml shows a stripped entry as a bare name. ArgoCD: Install with ArgoCD / GitOps.

IAM database auth is not taking effect

spend.auth is derived from ANYRAY_SPEND_DB_URL; use it to confirm RDS IAM authentication landed:

Check the database auth mode

curl --fail-with-body -sS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" | jq '.spend.auth'

Value	Meaning and fix
password	A password was found, and it always wins. The gateway looks where Postgres looks: the URL, ?password=, PGPASSWORD, and the password file (PGPASSFILE, else ~/.pgpass). Remove it and keep the user (postgresql://appuser@mydb.abc123.us-east-1.rds.amazonaws.com:5432/anyray). A URL with no user also reads password
rds-iam	Tokens mint from the pod's AWS identity. If connections fail, the role needs rds-db:connect on arn:aws:rds-db:<region>:<account>:dbuser:<db-resource-id>/<db-user>, and the database needs GRANT rds_iam TO <db-user>
null	Not backed by Postgres

An IAM URL that cannot be honoured stops the gateway at boot with a FATAL line:

Code	Fix
RDS_IAM_NO_REGION	The endpoint carries no region AWS recognizes. Set AWS_REGION on the service
RDS_IAM_SSL_DISABLED	Remove sslmode=disable; RDS IAM requires TLS, and the gateway configures it
RDS_IAM_BAD_PORT	The port in the URL is not a valid port number
RDS_IAM_EMPTY_TOKEN	AWS returned no token. Check the pod's AWS credentials

The schema leg reports drift after an update

The gateway migrates on boot. schema reports version (newest applied), expected (newest this image ships), and state:

Assert the schema is current

curl -fsS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" | jq -e '.schema.state == "converged"'

state	Meaning	Do
converged	Applied schema matches this image	Nothing
ahead	A replica on a newer image already migrated; normal mid-rollout or after a rollback	Nothing; clears once every replica runs the new image
behind	This replica's migration is pending or failed	Re-check after a moment; if it persists, read the gateway logs for the migration error
unknown	Database unreadable	Confirm spend.ok
none	No Postgres	Nothing

No state fails the overall check. schema.converged (boolean, for older scripts) is true for both converged and ahead.

schema.jsonbChecks counts the JSON column constraints in the spend store; every count is healthy. enforced: checked on write and proven over stored rows. pending: checked on write, older rows not yet proven (normal right after the release that adds one). missing: the migration is not applied yet. null means unknown (no Postgres, or unreadable), not zero.

Console shows an old version after an upgrade

GET /admin/update-status → version comes from the running gateway image, so a stale number means the new image never rolled. helm upgrade or docker compose up -d only restarts the gateway when the rendered spec changes. (schema.version on /admin/health is the newest migration applied.)

Confirm and force the running image (Kubernetes)

# What image is live right now?
kubectl -n "$ANYRAY_NAMESPACE" get deployment anyray-gateway \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
# Pin the build and roll it out
helm upgrade anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE" \
  --set image.tag=vX.Y.Z
kubectl -n "$ANYRAY_NAMESPACE" rollout status deployment/anyray-gateway

On a moving tag (image.tag: latest), set pullPolicy: Always and run kubectl -n "$ANYRAY_NAMESPACE" rollout restart deployment/anyray-gateway. Full flow: Kubernetes → Upgrade. The console catches up within a couple of minutes of the pod turning Ready.

Optimizer settings revert after a deploy

A saved Optimizer change reverts after the next image bump, or two tabs disagree. Check optimizer.configStore on /admin/health (the Optimizer page shows the same state as a banner):

Value	Meaning	Do
shared	Settings in Postgres, read by every replica	Nothing
per-pod	Settings on the replica that served the save, lost on the next roll	Set ANYRAY_SPEND_DB_URL on the optimizer service, or check the gateway can reach the optimizer
refused	A store was supplied but the optimizer cannot read it; saves are rejected	Check the optimizer can reach the gateway's database (security groups, network policy, credentials)
probing	Store just handed over	Re-check in a few seconds
absent	Optimizer unreachable, or image predates the field	Check the optimizer leg, then the image tag

The gateway pushes its database URL to the optimizer every 60 seconds. per-pod that persists means those pushes are not arriving or not accepted.

Health says open problems could not be read

The page could not read its own store, so it shows this replica only. The line under that heading says which failure it hit:

It says	What happened	Where to look
The database is too busy to answer	Postgres is up, but the health read timed out	Long-running queries in the Database section. A query running 5 minutes or more opens a problem
The database could not be reached	The connection was refused or failed to authenticate	The spend leg of /admin/health and its cause

To see what is running long, ask Postgres directly:

Active queries, oldest first

SELECT pid, now() - query_start AS running, wait_event_type, left(query, 60) AS query
  FROM pg_stat_activity
 WHERE state = 'active' AND pid <> pg_backend_pid()
 ORDER BY query_start;

SELECT pg_cancel_backend(<pid>) stops one query and leaves its connection open.

Sessions show “—” in Saved: optimization is paused

Requests are served, Sessions shows — in Saved, and Errors shows “Optimizations are paused”. This is billing: a trial ended with no subscription, so the entitlement lease pauses the optimizer (optimizerEnabled: false) and the gateway proxies unchanged. An image bump can surface it; older gateways kept optimizing while lapsed.

Fix: add a payment method in the Anyray Billing App. Optimization resumes on the next lease refresh (usually minutes), no redeploy. On a contract? Contact Anyray.

Requests and upstream errors

“Connection closed mid-response” or an API error naming your gateway host

The host in the message is only the client's ANTHROPIC_BASE_URL (or OPENAI_BASE_URL) in a fixed template, not evidence the gateway failed. The wording names the origin:

The client says	What happened	Look at
Server error mid-response	The provider dropped the stream; the gateway relayed and labelled it	The provider's status page
Connection closed mid-response	The TCP connection between tool and gateway died	Your own edge (below)
Response stalled mid-stream	The client's idle watchdog fired	The provider, then latency
request failed, nothing else	Upstream refused and its message was lost in translation (Claude-format client to OpenAI-format upstream, builds through v1.10.319)	Upgrade past v1.10.319; until then read the upstream's logs

On a provider-side close the gateway appends a final SSE error frame. A client with nothing rendered yet shows it in full; otherwise it shows Server error mid-response:

What the client renders on a provider-side close

Anthropic (api.anthropic.com) closed the response stream before it finished. This is an
upstream provider failure, not an Anyray gateway failure: the gateway relayed the stream as
it arrived and the connection was closed from the provider side. Check https://status.claude.ai/
for provider status. Retry the request.

Fix: retry. If failures cluster, check the provider's status page. The gateway log confirms the origin:

Gateway log: provider closed the stream

passthrough stream closed by upstream: upstream=Anthropic class=TypeError code=UND_ERR_SOCKET

No first byte at all returns a 502 with a status-page hint and logs passthrough fetch failed; also upstream-origin.

No log line, and it said Connection closed mid-response? The stream died on your side:

• Match timestamps to a rollout or restart (kubectl get pods restart counts; lastState.terminated.reason for OOMKilled).
• Every hop in front of the gateway needs read and idle timeouts longer than one completion (the chart's Ingress defaults are streaming-safe).
• The gateway drains for ANYRAY_SHUTDOWN_DRAIN_MS (default 90s). Give each edge a longer stop budget: stop_grace_period (Compose), terminationGracePeriodSeconds (chart), StopTimeout plus deregistration delay (ECS), RAILWAY_DEPLOYMENT_DRAINING_SECONDS (Railway). The optimizer reads the same variable with a 15s default.

Non-streaming requests still truncate

Only streaming responses get the error frame. A non-streaming body that dies mid-way is a bare truncation; use the log line above to attribute it.

Requests are refused for a missing routing header

Send either x-anyray-config or x-anyray-provider, or ask the gateway operator to set a
default route for headerless requests.

anyray-connect tools send the header, so this is a hand-written request or a header-free deployment missing its default route. Check inference.defaultRoute on /admin/health:

Value	Meaning	Do
configured	Header-free requests have a route	Nothing
no-configured-providers	Nothing to route to	Add a provider in the console → Providers
multiple-providers-no-default-route	Several providers, none is default	Set the default on the console → Routing page
provider-store-unreadable	This replica could not read the provider store	If secrets.providerKeys is unreadable, this replica's admin token differs from the one that sealed the keys. Otherwise check the spend leg; the store is shared

A single provider is the default on its own. Adding a second removes it, so header-free requests can break after an unrelated-looking change.

An upstream error says Not Found, or names a status without a cause

Relayed upstream errors carry the status and the URL Anyray called (query string removed):

litellm error (HTTP 404 from https://llm.example.internal/v1/messages): Not Found

The URL shows	Fix
The right path, not served by your upstream	The route is missing on that build. LiteLLM serves the Claude Messages API at /v1/messages on current versions only
A doubled path or unexpected prefix	custom_host is being appended to. Set it to the full endpoint URL; it is used as-is
Your upstream serves it elsewhere, or serves two endpoints at unrelated paths	Set custom_endpoints (Endpoint paths on the provider key). See Naming the paths yourself

Replies come back blank, or an error says the body has no choices

Your gateway answered 200 with a body that is not an OpenAI chat completion:

upstream returned 200 with no choices; the body is not an OpenAI chat completion (keys=[data,request_id])

keys lists the top-level keys your gateway returned (never values). Earlier builds showed an empty reply with zero tokens, recorded as a success. A native Claude response is already forwarded untouched, so the usual cause is a third shape, such as a wrapper.

Fix: if your gateway serves the Claude Messages API, set the provider to litellm and point custom_host at it; both directions then pass through. Otherwise make sure custom_host names the chat-completions endpoint itself.

A SCIM-provisioned user's key returns user_deactivated

The key is valid but its email is inactive in the SCIM identity store. Inspect with GET /scim/v2/Users?filter=userName eq "dev@example.com" (SCIM bearer), then reactivate the user in the IdP so it sends active:true. Don't mint a new key; every key bound to that email is blocked.

Many users failing at once: check the spend leg. SCIM reads fail closed when Postgres is down; restoring ANYRAY_SPEND_DB_URL connectivity restores verification.

Developer machines and Connect

One user's tools fail, but health is green

In the console, open Users, click the person, and read Diagnose. Every refused request is reason-coded with its fix (expired or revoked key, reached cap, deactivated SCIM user, provider-refused upstream credential). Events are kept 30 days.

Nothing recent there means the fault is on the machine (stale base URL, expired enrollment, missing key-renewal schedule). Run doctor in the user's failing terminal:

Generate a client report (in the user's failing terminal)

npx anyray-connect@latest doctor --json > anyray-client-report.json

Run it in the terminal that's failing

doctor reads that shell's environment. A shell open during a revert keeps exporting the gateway with no credential, so its tools return 401 valid client key required while files on disk read clean. doctor reports this as a Shell problem with the exact unset to run.

The auth smoke probe runs by default and tells key rejected, enrollment required, and provider rejected apart. With --no-verify, an all-clear only means nothing looks misconfigured. The report carries versions, gateway origin, per-tool state, cert validity window, exported variable names and key presence; never the key, the cert signature, or prompt content.

Savings look low and the hook lane may be dark

Connect's PostToolUse hook trims recognized command output and large reads (what it trims). A dead lane errors nowhere (the hook exits 0), so anyray-connect doctor probes it and prints Hook lane:

State	Meaning and fix
live	The gateway accepted this machine's key and answered the probe
live / no_recent_use	Healthy. The hook sits in a tool nobody used here this week (e.g. Claude Code installed, work done in Codex)
live / usage_unknown	Healthy. Recent use could not be established within one scan's limit; the fleet sees hook_usage_unknown on a passing check
key_rejected	Usually a lapsed hand-pasted key. On an enrolled machine an expired key self-heals at the next refresh; doctor --repair does it now and mints a credential (plain doctor never does; ANYRAY_REFRESH_DISABLE=true forbids it). Only missing or expired keys re-mint, and only when the gateway names the class. A revoked key, unknown key, or deactivated user never re-mints: re-enroll. An older gateway sends no class: re-run anyray-connect
target_null	No gateway target resolves, so the hook sends nothing. Run anyray-connect once
never_invoked	Used this week (per session-file mtimes for Claude Code, Cursor, OpenCode, Codex) but the hook never ran. Restart the tool. On OpenCode, check the binary path in the generated plugin; a failed launch there is silent
undetermined	Unknown (old gateway protocol, proxy or WAF, unreachable host). Informational; never repaired or reported to the fleet

Claude Code's machine-wide managed-settings.json can also kill the lane: disableAllHooks: true skips every hook; allowManagedHooksOnly: true drops user hooks unless the managed hooks block carries Anyray's PostToolUse hook. Not repairable from user scope: have the MDM owner drop the key, or deploy the block anyray-connect managed --print --sso <link> --bin <path> prints. Fleet check hook_policy (disabled or restricted_dropping_ours); an absent, clean, unreadable, or malformed file is never a fault. The probe carries the x-anyray-retrieve-probe marker and never books a saving.

Externalized output can't be pulled back (retrieval is dark)

Trimmed output leaves a · retrieve ctx_… marker; the model fetches the original through the anyray_retrieve MCP tool. New handles are made only while the gateway holds a fresh retrieval lease and the optimizer confirmed the durable write; otherwise the span stays whole. A settled decision pin may still replay an older marker byte-for-byte for provider-cache safety.

On the affected machine:

Check the whole retrieval loop

anyray-connect doctor --verify

• Tools: Not registered usually means Connect last ran from a one-off npx cache (registration needs a durable install); binary is gone means the executable moved. Re-run anyray-connect. If doctor names another Connect install, keep one durable install per machine or bring both to the same version.
• Retrieval loop sends one probe with a synthetic handle; healthy is the gateway answering "unknown handle". Failures: auth (enroll or set a personal key), unsupported (update the gateway image), unavailable (optimizer down or ANYRAY_OPTIMIZER_URL unset), unreachable (network). The probe reads nothing, and its x-anyray-retrieve-probe header never counts as retrieve capability.

Open a marker by hand with anyray-connect retrieve ctx_…; find one that scrolled away with anyray-connect recall "describe it".

Operators can read optimizer.retrievalStore on /admin/health:

Value	Meaning
ready	New durable handles allowed
cooldown	Backend circuit open; retries automatically after five minutes
unconfigured	No store, or no key for it (neither ANYRAY_CONTENT_KEY nor ANYRAY_ADMIN_TOKEN on the optimizer)
policy_pending	This replica has not read the console content mode yet. It holds new stash writes until the read settles.
content_disabled	Content mode is off
placeholder_key	No ANYRAY_CONTENT_KEY and ANYRAY_ADMIN_TOKEN is the public example token. Set a private token or a content key
absent	Optimizer unreachable or predates the field

Fix drift in one step

Run anyray-connect repair to restore managed configuration. It only touches tools a previous apply recorded, never one you reverted. The key refresher also repairs routing drift when it notices it.

Certificate errors, or could not reach the control plane, behind a TLS-inspecting proxy

npx anyray-connect@latest … fails with UNABLE_TO_GET_ISSUER_CERT_LOCALLY (or SELF_SIGNED_CERT_IN_CHAIN) against registry.npmjs.org while the browser opens it fine. A corporate proxy re-signs TLS with a private root CA in the OS trust store, and Node never reads that store.

Unblock now with the standalone binary (no Node; reads the OS trust store; every flag passes through):

macOS / Linux

macOS / Linux

curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- --sso https://app.anyray.ai/sso/<tenantId> --yes

Windows PowerShell

Windows PowerShell

& ([scriptblock]::Create((irm https://app.anyray.ai/connect.ps1))) "--sso" "https://app.anyray.ai/sso/<tenantId>" --yes

Older standalone binaries used a bundled CA list: download worked, then could not reach the control plane at https://app.anyray.ai. Reinstall with the line above; the self-updater fails the same way.

Then fix Node. Claude Code, the Codex CLI, npm, and the Anthropic and OpenAI SDKs fail the same way once inspection covers your gateway host. Point NODE_EXTRA_CA_CERTS at the proxy's root CA (it adds to Node's built-in list):

macOS / Linux

macOS / Linux

# macOS: MDM-installed roots live in the System keychain
security find-certificate -a -p /Library/Keychains/System.keychain > ~/corp-ca.pem
echo 'export NODE_EXTRA_CA_CERTS="$HOME/corp-ca.pem"' >> ~/.zshrc
export NODE_EXTRA_CA_CERTS="$HOME/corp-ca.pem"

Windows PowerShell

Windows PowerShell

Get-ChildItem Cert:\LocalMachine\Root |
  Where-Object { $_.Subject -match 'YourProxyVendor' } |
  ForEach-Object {
    "-----BEGIN CERTIFICATE-----`n" +
    [Convert]::ToBase64String($_.RawData, 'InsertLineBreaks') +
    "`n-----END CERTIFICATE-----"
  } | Set-Content "$env:USERPROFILE\corp-ca.pem"
[Environment]::SetEnvironmentVariable('NODE_EXTRA_CA_CERTS', "$env:USERPROFILE\corp-ca.pem", 'User')

Claude Code reaching your gateway does not need this step: see A certificate from your company's CA.

Connect never installs a CA or changes proxy settings. Re-run anyray-connect doctor in a new shell; open terminals and running background services keep the old environment.

Never disable certificate verification to get past this

npm config set strict-ssl false, NODE_TLS_REJECT_UNAUTHORIZED=0, and per-tool equivalents turn off verification for every connection the process makes, including package downloads.

The other half of the fix is an allowlist

If the proxy blocks hosts outright, allowlist registry.npmjs.org, github.com plus release-assets.githubusercontent.com (the binary redirects from one to the other), app.anyray.ai (enrollment and metering), and your gateway origin. anyray-connect doctor --security-review prints the exact URLs.

The installer stops mid-download with a connection error

The one-liner downloads a 100 MB binary; an unreliable link can drop it before enrollment:

Windows PowerShell

anyray-connect: downloading anyray-connect-windows-x64.exe...
Invoke-WebRequest : The request was aborted: The connection was closed unexpectedly.

On macOS and Linux: curl: (18) transfer closed with N bytes remaining to read.

Re-run the same command. Current installers resume and print transfer interrupted at … resuming (attempt 2 of 4). The binary is always checked against the release SHA-256.

Dies at the same point every time? A proxy is cutting it. Allowlist github.com and release-assets.githubusercontent.com, or use npm:

npm fallback

npx anyray-connect@latest --enroll <your-enrollment-link> --yes

anyray-connect login cannot start or expires

Run the current version against the gateway origin your tools use:

Start a fresh self-service SSO login

npx anyray-connect@latest login --gateway https://<your-anyray-gateway>

What you see	Cause and fix
Could not start SSO login	Gateway not connected to the Billing App, SSO not configured for the tenant, or the spend database is down. An operator checks /admin/health, the adt_ connection, and SSO setup
Login expired	The browser flow exceeded ten minutes. Start again
Multiple teams are available (non-interactive shell)	Pass --team <name>; only teams from the verified identity policy are accepted
The gateway rejected this login session	Poll secret mismatch or session already used. Start again; never put the poll secret in a URL or support report

anyray-connect finishes but the verify step returns HTTP 404

An enrollment link was used as a gateway URL, so tools point at …/sso/<tenantId>/v1/…, which serves no inference. Either Connect is pinned to a version without --sso, or the link went to --gateway. /sso/… links go to --sso, /enroll/… links to --enroll. Re-run current Connect with the link in the right slot; verify then returns 200:

macOS / Linux

macOS / Linux

curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- --sso https://app.anyray.ai/sso/<tenantId> --yes --force

Windows PowerShell

Windows PowerShell

& ([scriptblock]::Create((irm https://app.anyray.ai/connect.ps1))) "--sso" "https://app.anyray.ai/sso/<tenantId>" --yes --force

Endpoint security quarantined the key-renewal schedule

doctor reports the key-renewal schedule enabled but missing while gateway, enrollment and key read clean; tools may be unconfigured from a killed install. An EDR persistence heuristic (SentinelOne's persistence_deception, or CrowdStrike and Defender equivalents) quarantined the login item. Recent doctor builds list the files under Endpoint security. Re-installing or signing doesn't help.

Unblock now, then allow-list

cd ~ && anyray-connect --no-key-refresh   # no background service at all
anyray-connect doctor --security-review   # the sheet for your security team

--no-key-refresh changes nothing about optimization, metering or attribution; you re-run when the short-lived key lapses. Once allowlisted, anyray-connect --yes restores renewal where needed (else doctor reports Retired). Allowlist by path and service name, not hash; the hash changes each release. Full footprint: Key renewal.

Tool routing reads hooks arrive, requests do not

The developer's hooks reached the gateway in the last day, but none of their Claude Code requests did. ~/.claude/settings.json points at the gateway, but the running Claude Code reads another base URL: a project .claude/settings.json, the desktop app's Code tab, claude --settings, or CLAUDE_CONFIG_DIR.

Have the developer run /status in Claude Code. It names the base URL in use and where it came from. Remove that override. Rerunning anyray-connect does not help, because the file it writes is already correct.

The console drops the reading as soon as a Claude Code request from that person arrives after it, without waiting for the next Connect report. It is judged per person, so traffic from one of their machines also clears it for another.

Anyray skill loaded reads on disk, not loaded

The skill file is on the machine, but for two days none of the developer's Claude Code requests listed it to the model. A session started before the skill landed may not see it until it restarts. A project .claude/settings.json, claude --settings, or CLAUDE_CONFIG_DIR can also point Claude Code at settings that leave the skill out.

Have the developer restart Claude Code. If the result stays, remove the override. Rerunning anyray-connect does not help, because the file it writes is already in place.

A Claude Code session sends more requests than it takes turns

Expected when Claude Code delegates. Each subagent it starts (the Agent tool) runs its own loop, so a session of 3 to 11 turns can send 80 to 180 requests. Traces list them under the parent session. Such sessions cost 1.5 to 2.5 times as much per main-loop request as a similar session that does not delegate. Nothing on the gateway needs fixing.

Enrollment succeeded but Claude Code still talks to Bedrock or Vertex directly

CLAUDE_CODE_USE_BEDROCK (or CLAUDE_CODE_USE_VERTEX) makes Claude Code sign to AWS/GCP itself and ignore ANTHROPIC_BASE_URL. Connect (at apply and on doctor --repair) pins the switch to '' in the ~/.claude/settings.json env block, which outranks a shell export. Restart Claude Code. If the switch is in managed-settings.json, apply prints WARNING:, doctor reports routing bypassed, and the admin who deploys that file must remove it.

To keep serving from AWS, use the gateway's bedrock provider.

Enrollment succeeded but a managed-settings.json overrides the route

managed-settings.json outranks every other Claude Code scope, so an MDM file with its own ANTHROPIC_BASE_URL wins while user scope reads healthy. anyray-connect doctor names the overridden key, the file path, and the origin requests actually reach (scheme, host, port only). doctor --repair won't touch it. Either the admin drops the key, or deploys Anyray's block with anyray-connect managed.

Claude Desktop will not route, and setup says a machine policy owns it

Once a managed profile sets any Claude policy key other than an app-behavior-only one (such as disableAutoUpdates, relaunchEnforcementHours, or egressProxyUrl), Claude Desktop ignores local config, so Connect won't write it.

1

Re-run for Claude Desktop alone

npx anyray-connect@latest --tools claude-desktop

Connect before 0.11.147 could falsely report a policy on a Mac with no Claude profile.

2

Install the profile Connect generated

Still declined? Install ~/.anyray/claude-desktop-anyray.mobileconfig (macOS; double-click, approve under System Settings → General → Device Management) or %USERPROFILE%\.anyray\claude-desktop-anyray.reg (Windows). It holds no credential; install the shared helpers once:

sudo anyray-connect desktop helper --write --platform posix --bin /usr/local/bin/anyray-connect

Quit Claude Desktop fully (Cmd+Q) and reopen.

3

Or hand the same file to whoever manages the devices

Deploy it through Jamf, Intune, or Kandji, or generate an editable bootstrap profile with anyray-connect desktop fleet bootstrap <platform> <https-url>. See Claude Desktop on managed devices.

Claude Code, Codex, and Cursor are per-user and route immediately; don't wait on IT for them.

Claude Desktop lost its MCP servers, skills, or plugins after switching to Anyray

Third-party inference uses a separate Desktop profile; nothing is deleted. Connect copies local MCP servers, user skills and user plugins across on every apply (MDM included); skills and plugins land after Claude Desktop is fully quit. anyray-connect doctor → Claude Desktop profile: pending (copy queued), not_routed (Desktop not routed). All states: Connect reference. For history, fully quit Desktop and run anyray-connect migrate. Fleet check: desktop_config_carry (never counts against compliance).

Send Anyray a support bundle

Generate a support bundle

curl -fsS "https://<your-gateway>/admin/support/bundle" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -o anyray-support-bundle.json

Section	Carries
gateway	Gateway version, product release, Node runtime, platform, uptime, memory use
health	The /admin/health probes at bundle time
config	Every set ANYRAY_* variable: plain values verbatim, URLs as scheme://host, secrets as [set]
optimizer	Whether an optimizer is wired, and its live strategy config
hotPathMemos	Hit and miss counts for the gateway's transcript-hash memo and the optimizer's command_digest memo. Many evictions, or any sessionsOversize, means long sessions recompute every turn. conversation and prefix count the two memos a turn that misses the optimizer replays: entries and chars now, peaks since start, limits, and evictions. Any evictedRecentlyUsed means the memo is smaller than this replica's active sessions, so a missed turn can re-write a session's prompt cache
telemetry	Request counts by status class and route since boot, plus the last 100 error events (route, status, error class, duration)
manifest	Section names, generation time, approximate size

Errors are recorded as class names, never messages.

1

Reproduce

Trigger the failing request once so it lands in the telemetry ring.

2

Generate and review

Account menu → Diagnostics → Generate & review (needs config:read; sends nothing). The curl above produces the same file.

3

Send it yourself

Select Copy and paste the JSON into your support thread. The console never sends it.