Troubleshooting

Start at /admin/health — one call names the failing leg.

Check deployment health

/admin/health is the first thing to run. It probes every leg of the deployment live and names the one that's failing — telling not configured apart from configured but unreachable.

1

Run the health check

Check deployment health

curl -fsS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN"

2

Read the status code

200 — every required leg is up. 503 — a required leg is down, and its name is in the response body.

3

Find the failing leg

Match the leg to the table below for what it checks and the first move.

Every leg reports on the same JSON. Required legs are the only ones that can turn the check red — a down optimizer or a lagging schema never fails it.

Leg	What it checks	Required	If it's the failing leg
gateway	The process is serving requests	—	A response at all means it's up
observability	Trace store reachable — or intentionally not configured	When configured	Check ANYRAY_OBSERVABILITY_DB_URL and the Langfuse stack
spend	Spend store (Postgres) answers a live SELECT 1	Yes	Check ANYRAY_SPEND_DB_URL and that Postgres is reachable
schema	Boot migrations converged — version vs expected	No · informational	See schema.converged is false below
optimizer	Optimizer /health reachable — or not wired up	No · fails open	Check the optimizer service and ANYRAY_OPTIMIZER_URL
portal	Billing app metering + entitlement lease status	No	Check the Billing app connection (configure)

Common issues

Find the symptom that matches, then expand it for the cause and the fix. This list grows over time — scan the titles first.

“Connection closed mid-response” or an API error naming your gateway host

A coding assistant reports that the response was cut off part-way, or shows an API error that tells the reader to check your gateway's host. The host in that message is just the client's ANTHROPIC_BASE_URL (or OPENAI_BASE_URL) — clients render a fixed template around it and name whatever they were pointed at. It is not evidence that the gateway caused the failure.

The common cause is the upstream provider closing the connection after it started streaming. The gateway relays the provider's stream as it arrives, so a provider-side close lands mid-response. When that happens on a streaming request the gateway now appends a final SSE error frame naming the real origin, so the client shows something like:

What the client renders on a provider-side close

Anthropic (api.anthropic.com) closed the response stream before it finished. This is an
upstream provider failure, not an Anyray gateway failure — the gateway relayed the stream as
it arrived and the connection was closed from the provider side. Check https://status.claude.ai/
for provider status. Retry the request.

Fix: retry the request — these are transient. If they cluster, check the provider's status page before opening a gateway ticket.

Confirm it upstream-side in your gateway logs. A provider-origin stream close logs a single content-free line; there is no matching line when the gateway itself is the problem:

Gateway log — provider closed the stream

passthrough stream closed by upstream: upstream=Anthropic class=TypeError code=UND_ERR_SOCKET

A failure that never got a first byte fails differently — the gateway returns a 502 with a status-page hint, and logs passthrough fetch failed. Both are upstream-origin. A restart or deploy during an in-flight request can also cut a stream; check whether the timestamps line up with a rollout before assuming a provider incident.

Non-streaming requests still truncate

The error frame only exists for streaming responses — that's the only place the protocol has a slot for it. A non-streaming request that dies mid-body still surfaces as a bare truncation, and the log line above is the way to attribute it.

Traces show “—” in Tokens Saved — optimization is paused

The gateway is serving every request, but no strategies are running, so the console's Traces list shows — in the Tokens Saved column and savings sit at zero. The console also shows a persistent “Optimizations are paused” banner on every page.

This is a billing state, not a bug — and not caused by an upgrade. When an Anyray trial ends with no active subscription, the signed entitlement lease pauses the optimizer (optimizerEnabled: false): the gateway keeps proxying every request unchanged, only the optimize hook is skipped. (A gateway old enough to predate this behavior kept optimizing while lapsed, so the pause can first appear right after an image bump — the upgrade only started honoring a state that was already true.)

Fix: add a payment method in the Anyray billing portal. Optimization resumes automatically on the next lease refresh, usually within a few minutes — no redeploy. If you believe your workspace is on a contract and shouldn't be on the self-serve trial clock, contact Anyray.

/admin/health returns 503 — a required leg is down

Only the required legs turn the check red: spend (always) and observability (when it's configured). The response body names which one, and its ok field is false.

• spend is down — the gateway can't reach its spend store. Confirm ANYRAY_SPEND_DB_URL points at a reachable Postgres and the database is accepting connections.
• observability is down — a trace store is configured but unreachable. Confirm ANYRAY_OBSERVABILITY_DB_URL and that the Langfuse stack is up. If you don't run observability, leave it unconfigured — the leg reports not configured and never gates.

Re-run the health check once the leg is reachable; it flips back to 200. If the leg looks configured and reachable but still fails, send a support bundle.

schema.converged is false after an update

The gateway self-migrates its database schema on boot, so an image bump — a helm upgrade, a one-click Update now, or the auto-updater — is all that's needed. The schema leg reports version (the newest migration applied to your database), expected (the newest the running image ships), and converged — true once they match. After an update, confirm it caught up:

Assert the schema converged

curl -fsS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" | jq -e '.schema.converged == true'

• false — a migration is still pending, or the database is unreachable. Give a fresh deploy a moment to finish migrating, then re-check; if it stays false, confirm the database is reachable.
• null — the gateway isn't backed by Postgres, so there's nothing to migrate.

converged is informational: a mid-rolling-deploy replica that hasn't caught up won't fail the overall health check.

Externalized output can't be pulled back (retrieval is dark)

When Anyray trims an oversized tool output it leaves a · retrieve ctx_… marker and stashes the original — the model pulls it back through the anyray_retrieve MCP tool. If that read path is broken, nothing errors loudly: trims silently become lossy and the model re-reads files instead. Diagnose it in one command on the affected developer's machine:

Check the whole retrieval loop

anyray-connect doctor --verify

Two sections cover the loop end to end:

• Tools shows each tool's retrieval MCP-server registration. Not registered usually means connect last ran from a one-off npx cache (registration requires a durable install) — re-run anyray-connect. Binary is gone means the registered command's executable moved since registration — same fix.
• Retrieval loop (with --verify) sends one probe-marked /connect/retrieve request with a synthetic handle. The healthy verdict is the gateway answering "unknown handle" — that proves gateway reachability, key auth, and the optimizer's retrieve wiring in one shot. Failures name the broken leg: auth (enroll / set a personal key), unsupported (gateway image predates the loop — update it), unavailable (optimizer down or ANYRAY_OPTIMIZER_URL unset on the gateway), unreachable (network).

The probe is content-free and side-effect-free: the synthetic handle reads nothing, and the x-anyray-retrieve-probe header keeps a diagnostic run from counting as model retrieve capability. You can also open a marker by hand: anyray-connect retrieve ctx_… prints the stashed original, and anyray-connect recall "describe it" finds a handle whose marker scrolled away.

Fix drift in one step

anyray-connect doctor --repair re-applies any tool whose gateway routing drifted (a tool update resetting its config is the usual cause). It only touches tools a previous apply recorded — a tool you explicitly reverted is never resurrected. Everything else about doctor stays read-only, and the same repair runs automatically in the background whenever Connect's key refresher notices a recorded tool has drifted.

Console shows an old version after an upgrade

The deployment version in the console (and the schema.version line on GET /admin/health) is read from the running gateway image — baked in at build and reported on each heartbeat. A stale number therefore means the new image never rolled, not that reporting is behind: a helm upgrade (or docker compose up -d) only restarts the gateway when the rendered spec actually changes, so re-running it without a new chart or image tag is a no-op.

Check what the pod is actually running, then pin the build you want and force the roll:

Confirm and force the running image (Kubernetes)

# What image is live right now?
kubectl -n "$ANYRAY_NAMESPACE" get deployment anyray-gateway \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
# Pin the build and roll it out
helm upgrade anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE" \
  --set image.tag=vX.Y.Z
kubectl -n "$ANYRAY_NAMESPACE" rollout status deployment/anyray-gateway

On a moving tag (image.tag: latest) pair it with pullPolicy: Always and kubectl -n "$ANYRAY_NAMESPACE" rollout restart deployment/anyray-gateway, or the node keeps its cached image. See Kubernetes → Upgrade for the full flow — the console catches up within a couple of minutes once the new pod is Ready.

One user's tools fail, but health is green

When the gateway is healthy but a single user's tool misbehaves, the failure usually lives on that machine — a stale base URL, an expired enrollment, a key-refresh daemon that's down. None of that is visible to the gateway, so a support bundle can't see it. Run the client doctor on the affected machine — it diagnoses each finding with a fix, and --json prints the same findings as a shareable report:

Generate a client report (in the user's failing terminal)

npx anyray-connect doctor --json > anyray-client-report.json

Run it in the terminal that's failing

doctor inspects that shell's own environment, so a report from a different terminal can miss the problem. A shell that was open when the user reverted keeps exporting the gateway even after the config is removed — every tool launched from it then reaches the gateway with no credential and returns 401 valid client key required, while every file on disk reads perfectly clean. doctor reports that as a Shell problem with the exact unset to run.

Add --verify to also run the auth smoke probe — one tiny request through the gateway whose verdict distinguishes key rejected from enrollment required from provider rejected, the three failures that all look like "my requests don't work" from the tool.

The report carries machine and config metadata only: connect version, gateway origin and reachability, per-tool pointing state, enrollment-cert validity window, the names of any Anyray variables the shell is still exporting, and key presence — never the key, the cert signature, or any prompt content.

Endpoint security quarantined the key-refresh services

anyray-connect doctor reports the key-refresh daemon enabled but not responding and the key-refresh schedule missing at the same time, on a machine whose gateway, enrollment, and personal key all read clean. Tools may also be left unconfigured, because the install was killed partway through.

That pattern is an EDR product acting on a persistence heuristic (SentinelOne's persistence_deception, and equivalents in CrowdStrike and Defender): a login item that restarts itself is the shape a persistence implant uses, so the scanner kills the process and quarantines the unit file. Recent doctor builds name it directly — under Endpoint security — and list the files that are gone.

Re-running the installer does not fix it: it writes the same unsigned login item straight back into quarantine. Unblock the seat first, then fix it properly:

Unblock now, then allow-list

cd ~ && anyray-connect --no-key-refresh   # no background service at all
anyray-connect doctor --security-review   # the sheet for your security team

--no-key-refresh points tools straight at the gateway, so optimization, metering, and spend attribution are unchanged — those all happen at the gateway. The only cost is a re-run when the short-lived key lapses. Once the paths in the sheet are allow-listed, anyray-connect --yes restores automatic refresh. Full footprint: Key-refresh daemon → Endpoint security allow-list.

Check for a cached provider key at the same time

A seat that ran the retired anyray-connect --fallback lane holds the org's encrypted provider key on disk, and the purge rides a normal apply — which is exactly what a quarantine prevents. doctor reports it under Retired fallback lane, and anyray-connect doctor --repair deletes it without needing an apply to succeed.

anyray-connect login cannot start or expires

Run the current Connect version and name the same public gateway origin your tools use:

Start a fresh self-service SSO login

npx anyray-connect@latest login --gateway https://<your-anyray-gateway>

• Could not start SSO login — the gateway is not connected to the Billing app, enterprise SSO is not configured for that tenant, or the shared spend database is unavailable. Ask an operator to check /admin/health, the deployment's adt_ connection, and the SSO setup.
• Login expired — the browser flow exceeded ten minutes. Start again; the code and poll secret are one-session capabilities and cannot be reused.
• Multiple teams are available in a non-interactive shell — pass --team <name>. The gateway accepts only a team returned by the verified identity policy.
• The gateway rejected this login session — the poll secret did not match or the session was already consumed. Start again; never put the poll secret in a URL or support report.

anyray-connect finishes but the verify step returns HTTP 404

connect wrote the config, then its own smoke test came back 404. This almost always means an enrollment link was used as a gateway URL, so the tools now point at …/sso/<tenantId>/v1/… — an enrollment path that doesn't serve inference. Two common causes:

• A pinned, outdated connect. Releases before the --sso flag existed don't recognize it, so the link falls through to the gateway slot (and older builds also predate newer tool adapters like OpenCode). Always run the current version — the connect.sh installer or npx anyray-connect@latest — never a pinned older anyray-connect@<old>.
• The link passed to --gateway. An /sso/… or /enroll/… link is an enrollment link, not a gateway origin — pass it to --sso (SSO) or --enroll (invite link). connect resolves the real gateway URL from the link itself.

Re-run with the link in the right slot; enrollment runs, mints the key, and the verify step returns 200:

Re-enroll with the current connect

npx anyray-connect@latest --sso https://app.anyray.ai/sso/<tenantId> --yes --force

A SCIM-provisioned user's key returns user_deactivated

The key is valid, but its bound email is inactive in the durable SCIM identity store. Confirm the IdP user is active, then inspect it with GET /scim/v2/Users?filter=userName eq "dev@example.com" using the SCIM bearer. Reactivate the user in the IdP so it sends active:true; do not mint a new key, because verification blocks every key bound to that inactive email.

If multiple users fail at once, check the spend leg on /admin/health. SCIM access reads fail closed when the shared Postgres store is unavailable, so restoring ANYRAY_SPEND_DB_URL connectivity restores verification without changing keys.

Send Anyray a support bundle

When health and the checks above don't explain the problem, /admin/support/bundle assembles a one-shot diagnostic snapshot you can send to Anyray support — it answers the questions we'd otherwise have to ask one at a time:

Generate a support bundle

curl -fsS "https://<your-gateway>/admin/support/bundle" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" \
  -o anyray-support-bundle.json

Section	What it carries
gateway	Gateway version, product release, Node runtime, platform, uptime, memory use
health	The same per-leg probes as /admin/health, captured at bundle time
config	Every set ANYRAY_* variable — plain values verbatim, URLs stripped to scheme://host, secrets shown as [set]
optimizer	Whether an optimizer is wired up, and its live strategy configuration
telemetry	Request counts by status class and route since boot, plus the last 100 error events (route, status, error class, duration)
manifest	The section names, generation time, and approximate serialized size shown before send

The bundle contains no prompt or response content

The bundle is metadata only, by construction: secrets are redacted to presence ([set]), connection URLs lose their credentials, and errors are recorded as class names — never messages, which can embed request text. No prompt or response content, no key material, no message bodies. It's plain JSON, and nothing leaves until you review it and explicitly select Send to Anyray support.

1

Reproduce

Trigger the failing request once if you can, so the error lands in the telemetry ring.

2

Generate and review

Open Privacy → Share diagnostics with Anyray support and select Generate & review. The console shows the manifest, approximate size, and complete pretty-printed JSON. Generating needs config:read and sends nothing. The curl above produces the same local review artifact.

3

Send explicitly

After reviewing the JSON, select Send to Anyray support. This separate action requires content:manage (security admin or owner), rebuilds and redacts the snapshot, and records a metadata-only entry in your deployment audit log. Nothing is sent before this click.

4

Keep the receipt

Copy the returned receipt id into your support thread. The bundle lands in the Billing app's support store, where Anyray operators see the analyzer findings and may open the bundle you explicitly sent. The store keeps at most 20 bundles per deployment and drops bundles older than 90 days.