Security

Content stays on your inference path; Anyray receives metadata only.

Access control

Two separate surfaces:

Surface	Reaches	Gate
Admin (console + /admin/*)	Config, spend, traces	Break-glass admin key (ANYRAY_ADMIN_TOKEN), optionally SSO and RBAC, or a capability-scoped aak_… admin API key.
Inference (/v1/*)	LLM traffic	Per-user ark_… or non-human ark_svc_… client key, always required, no opt-out.

Every /v1/* request needs a valid ark_… client key. A missing, unknown, expired, or revoked key gets a 401 with a content-free reason code and a fix. The key's identity is authoritative for attribution, and x-anyray-metadata cannot override it. Health and enrollment routes are never key-gated.

A fresh gateway blocks traffic until users are enrolled

Connect Billing, start the gateway, then enroll users before routing production traffic.

Condition	Status
Invalid user key	401
Missing lease, during first-boot grace	503 (entitlement_inactive)
Missing lease, after grace, or expired	402
Suspended	403

Console access and RBAC

Credential	How it works
Break-glass admin key (ANYRAY_ADMIN_TOKEN)	Always works and is always owner, with no Billing round-trip, so it stays your recovery path even if Billing is unreachable. Treat it like a root password.
Console SSO Optional	Operators sign in with your corporate IdP through WorkOS. The gateway verifies the signed session offline, so it survives restarts and works across replicas.
Scoped admin API key (aak_…)	An automation credential carrying an explicit subset of the capabilities below, for a budget bot or a rotation job that must never hold the root key. See API access.

Console SSO signs operators into the admin console. User SSO enrollment signs users in to get a /v1/* key. Same WorkOS broker, different audience.

Roles and capabilities

Every /admin/* route checks a capability, so being signed in is not being fully privileged.

Role	Can do
viewer	Read metadata only: spend dashboard, usage, trace lists and detail with bodies redacted.
auditor	viewer, plus audit-log reads.
operator	viewer, plus day-to-day ops: routing, user caps, model aliases, the playground. No secrets, no updates, no audit log.
security_admin	Everything: trace content, provider-key rotation, content mode, minting and revoking user keys, running updates, SSO and RBAC config.
owner	Same capabilities as security_admin. The break-glass role, deliberately not assignable from the IdP.

Map an email or IdP group to a role under Users → Identity → SSO configuration. Unmapped operators default to viewer, and someone in several mapped groups gets the most privileged of them. An email mapping overrides their groups either way, so it also holds one person below what their groups would grant.

RBAC never shows more content than the org content mode already allows. Automation uses scoped admin API keys, whose grants are subsets of these.

Capability bundles

Some access is not a rung of the ladder. Reading prompt and response bodies is observability:read-content, held only by security_admin, which also rotates provider keys and configures SSO. A capability bundle is a named set of capabilities you define, then map to an IdP group the way you would a role.

1

Add a bundle

Under Users → Identity → SSO configuration → Capability bundles, click Add bundle, name it (say compliance), and tick the capabilities it grants. Only capabilities you hold yourself are listed.

2

Map a group to it

Under Group → bundle mapping, enter the IdP group and pick the bundle. A group can also map straight to a built-in role.

3

Save

Bundles save with the rest of the SSO configuration. Each operator picks up the change on their next sign-in.

A compliance bundle granting Read traces, Read prompt and response bodies, and Read the audit log, mapped to sec-team, gives that group exactly those three.

Rule	Detail
A bundle grants exactly what it lists	Nothing is implied.
You can only grant what you hold	Every bundle is checked against the saving operator's own capabilities, so an operator cannot write themselves a bundle carrying providerkeys:manage. The break-glass admin key holds everything.
Several groups combine	Someone in two mapped groups gets everything both grant, unlike the role ladder.
owner is never a bundle	The built-in role names are reserved.
A group mapped to a deleted bundle grants nothing	It does not fall back to a default.
An email → role mapping still wins outright	It remains the way to hold one person below what their groups grant.

Finish the gateway rollout before relying on a bundle

A signed session carries both the resolved capabilities and a role, and a gateway that predates bundles reads only the role. When a group maps solely to custom bundles that role is viewer, which is not empty: it grants read configuration and spend and read trace metadata. So on an old replica, a bundle narrower than viewer grants more than you asked for, not less.

Per-user gateway keys (client keys)

Each user gets a personal ark_… key, so you attribute and revoke per person without sharing the admin token or a provider key. A personal key is shown once at mint, and only a SHA-256 hash is stored. A revoked key stays listed, marked revoked.

Agent keys (ark_svc_…) are the exception: an unattended host has no one to re-enroll it, so the gateway also keeps the key encrypted (AES-256-GCM, under a key derived from ANYRAY_ADMIN_TOKEN) and shows it back on the agent's Setup tab. Reading it needs its own capability (clientkeys:secret, scope keys:secret) that no scoped admin API key can carry, and both the reads and the refusals are written to the audit log; a read the trail cannot record is refused. Only the SHA-256 hash ever authenticates a request, so a database dump on its own yields no working credential, and rotating the admin token makes stored agent keys unreadable — the console then offers a new key instead.

Mint, list, and revoke from the console Users page or /admin/client-keys. ANYRAY_VERIFIED_DEV_KEY_TTL_DAYS (default 90) caps a minted key's lifetime, never past the enrollment certificate's expiry.

What configured SSO changes on the server

Connecting SSO changes the enrollment mode on the server, and is not only a console hint. Billing reads the tenant's durable IdP configuration before signing a certificate and carries the result in the control-plane-signed lease. A failed policy read denies enrollment or human-key minting.

Once SSO is active, shared email links and direct human-key mint cannot produce a standalone working credential, and a manual-kind provisioning token's mint, rotation, and redemption are refused the same way. An admin-issued, email-bound personal link stays available as a recovery path.

An MDM-kind provisioning token (kind: mdm) is a separate lane. It mints, rotates, and redeems regardless of SSO state, requires email identity mode, and still refuses machine mode. A macOS token with proofMode: managed-preference enrolls each Mac as the email its MDM writes into a managed preference, with no endpoint agent, fleetd, MDM vendor CA, or MDM API. Existing MDM tokens without that proof mode keep the legacy endpoint-evidence check at mint and heartbeat. See Bulk enrollment with your MDM.

An MDM-kind provisioning token (kind: mdm) is a separate lane. It mints, rotates, and redeems regardless of SSO state, requires email identity mode, and still refuses machine mode. A macOS token with proofMode: managed-preference enrolls each Mac as the email its MDM writes into a managed preference, the same trust model as the Windows and Linux enrollment command, with no endpoint agent, fleetd, MDM vendor CA, or MDM API. Existing MDM tokens without that proof mode keep the legacy endpoint-evidence check at mint and heartbeat. See Bulk enrollment with your MDM.

Fleets without an MDM lane have MDM distribute the SSO handoff instead. Non-human agents and CI use separately managed service keys. A header claiming SSO or another identity is ignored for authorization and audit attribution.

Passwordless enrollment

Without SSO, an admin mints a personal or shared link, the user runs one command, and the gateway verifies it offline with public-key crypto and no shared secret, then mints their key (default 90-day TTL, auto-refreshed). Once SSO is active, a shared link returns sso_required and signs no certificate. An admin can still issue an email-bound personal link for one developer.

Full flow: Passwordless enrollment. For IdP-verified identity, SSO enrollment.

Abuse and DoS controls

Per-key and per-IP rate limits are opt-in, set in .env (Rate limits), and the rest is always on.

Control	Detail
Rate limits	Requests per minute per client key and per source IP.
Concurrency cap	In-flight /v1/* requests per key.
Body size cap	ANYRAY_MAX_BODY_BYTES, default 32 MiB, answering 413.
Failed-auth throttle	ANYRAY_AUTH_FAIL_LIMIT, default 20 401s per IP per 10 min.
Enrollment throttle	ANYRAY_RATE_LIMIT_UNAUTH_RPM, default 10/min per IP, on the public on-ramp.

Only 401s that look like key guessing count toward the throttle, meaning the request presented a credential and it matched no known key. An expired or revoked key is not a guess, so a whole team behind one NAT cannot trip the lockout. Past the limit, further failing guesses get a 429 with a generic message and Retry-After.

Connection timeouts close slow-drip sockets. L3 and L4 floods should never reach the app, so front a public gateway with the bundled Caddy TLS edge plus a CDN, and bind the gateway to localhost (Gateway TLS).

The container runs as a non-root user

The gateway image runs as the unprivileged node user (uid 1000). It drops privileges at boot rather than through a USER directive, because the image ran as root for its whole life, so volumes created before that are root-owned.

The entrypoint starts as root, takes ownership of the runtime state directories, then drops to node and hands over. An existing deployment is adopted on the first boot after the upgrade, with no manual step. The gateway stays PID 1, so docker stop still delivers SIGTERM straight to it.

Verify

docker compose exec gateway id
# uid=1000(node) gid=1000(node)

Three cases where it stays root, all deliberate

• You already pin a user. A Kubernetes securityContext.runAsUser, an ECS task user, or docker run --user means the container never starts as root. The entrypoint execs straight through and the volume permissions are yours, so point the mount at uid 1000, or at whichever uid you pinned.
• The handover cannot be completed. A read-only mount, or an NFS/EFS export with root_squash, cannot be chowned. If the runtime user cannot write each state directory, the gateway stays root and logs the reason, so an upgrade can never leave a working deployment unable to write.
• A custom runtime user. ANYRAY_RUNTIME_USER selects a different account. If it does not exist in the image, the gateway logs the mismatch and continues as root.

Bind-mounting a host directory is the case most likely to need attention. The directory is chowned to uid 1000 on the host as well, so pick a path not shared with something expecting different ownership.

How traffic reaches Anyray

Apps point their SDK base URL at the gateway (OPENAI_BASE_URL or ANTHROPIC_BASE_URL to :8787). There is no org CA, no TLS interception, and no HTTPS_PROXY. The SDK just talks to a different URL.

Updates are locked down

On Docker and Compose, a bundled updater pulls newer Anyray images and recreates the three app containers. It runs from Update now in the console's sidebar Updates panel, or automatically for soft updates on by default, which are image-only releases whose local preflight passes with no breaking flag.

Because that needs the Docker daemon, the path is tightly scoped:

Scope	Detail
Socket proxy only	The updater reaches Docker through a socket proxy, socket mounted read-only, for just the pull and recreate calls. Neither the proxy nor the updater publishes a host port.
Capability-gated	The button, and changing the auto-apply policy, needs update:run (security_admin and above). The updater token is injected server-side and never reaches the browser.
Anyray images only	Your datastores are never touched.
The decision is always local	The vendor publishes a signed advisory and schema-validated manifest. The gateway compares version strings and inspects its own environment. Nothing received from outside is executed.

Hard updates, meaning a new required env var or a release flagged breaking, are never applied unattended. The update notice names exactly what is missing, so the operator reconciles infrastructure first. Kubernetes, AWS, and Railway have no updater, so upgrades roll through the platform.

What an update contains, and how to verify it

Stage	Guarantee
Before merge	Static analysis (CodeQL, Semgrep), secret scanning, the full test suites, and drift guards that force any required-setting or stack-topology change into the release manifest your preflight classifies against.
At publish: scanning	Every image is vulnerability-scanned by digest, per architecture, before the public tag exists. A fixable HIGH or CRITICAL finding blocks the release. Reproduce it anytime with trivy image public.ecr.aws/anyray/gateway:<tag>.
At publish: signing	Every customer image and the Helm chart is signed with Sigstore cosign, by digest, before any channel tag moves onto it.
In flight	The update recommendation is Ed25519-signed by the vendor key your image pins, and the manifest is schema-validated over TLS from the pinned Billing host.
On your host	Every unattended apply is recorded in anyray_audit_log (actor system:auto-update, from and to version, result).

The signature is keyless. It binds the artifact to the GitHub Actions workflow that built it, and records that in the public Rekor transparency log. Verify anything you pull:

cosign verify \
  --certificate-identity-regexp '^https://github\.com/anyrayHQ/monorepo/\.github/workflows/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  public.ecr.aws/anyray/gateway:<tag>

The same command verifies the chart (public.ecr.aws/anyray/anyray:<chart-version>). It answers one question: did Anyray's CI build these exact bytes? That is what a substituted or typosquatted image fails.

What the signature does and does not prove

It authenticates the build, not a company name. The developer-machine binaries are the ones signed to a vetted legal entity: anyray-connect is Developer ID-signed and notarized on macOS, and Authenticode-signed on Windows as Othentic Labs LTD, Anyray's legal entity.

To hold a build, pin ANYRAY_IMAGE_TAG=vX.Y.Z, or pin by digest through the per-image overrides (ANYRAY_GATEWAY_IMAGE=public.ecr.aws/anyray/gateway@sha256:…), which no updater or channel move can advance.

This is not a proof that no vulnerability exists, because no vendor can prove absence. It guarantees that known fixable HIGH and CRITICAL CVEs cannot ship, that what you pull is what was scanned, and that every automatic change on your host is attributable and reversible.

Server-held provider keys

Provider keys live server-side, never on the client. Apps authenticate with a personal ark_… client key, and the real provider key is never returned to a caller.

Source	Detail
Env vars	ANYRAY_PROVIDER_KEY_* at deploy time.
Runtime admin	The console Providers page or /admin/provider-keys, to rotate without a redeploy. This wins over env vars.
AWS workload role	Bedrock only, using the gateway's ECS, EC2, or EKS role, so no long-lived key is stored.

In every case the key stays server-side and is audit-logged by provider name only, never the value. The gateway does the provider signing, and the optimizer never touches keys or content. See Provider keys.

Custom upstream hosts (SSRF protection)

How a request reaches an OpenAI-compatible upstream depends on who named the host.

Who named it	How it is authorized
You configured it (console routing config, PUT /admin/routing-config, the ANYRAY_UPSTREAM_URL seed)	The write itself. It is admin-gated (routing:write) and audited, so it needs no allowlist entry and no env var, even when the host resolves to a private address. An in-VPC proxy or on-prem vLLM is the ordinary case.
A caller supplied it (x-anyray-custom-host, or an x-anyray-config body)	Two gates, below. Without them it could point the org's server-held provider keys at an unapproved upstream.

A configured upstream is scoped to the exact origin, meaning scheme, host, and port. So http://llm.internal:4000 does not make llm.internal:9200 reachable.

The two gates on a caller-supplied host:

Gate	Detail
Host allowlist	Set ANYRAY_CUSTOM_HOST_ALLOWLIST. With none configured, every caller-supplied host is refused. Entries match the exact hostname or a dotted suffix, and a miss is a 400. The first-party subscription backends Connect routes seats to (chatgpt.com, for Codex) are built in and always allowed.
Private-IP block	Caller-supplied hosts may not resolve to a private or reserved address, which also catches DNS rebinding. A block returns 502 naming the hostname and logs a content-free ssrf_blocked event.

Cloud metadata is never reachable

169.254.169.254, metadata.google.internal, and the rest of the instance-metadata surface are refused on every path, configured or not. Reaching one would hand back the gateway's own cloud credentials, so no configuration authorizes it.

TRUSTED_CUSTOM_HOSTS and a private upstream

It previously had to name any inference upstream resolving to a private address, on top of the allowlist and a redeploy. Configuring the upstream is now sufficient. The variable still applies to the caller-supplied path, and to an organization MCP connector you deliberately run on a private address. Existing values still work. If you set it only to reach your configured inference upstream, you can drop it.

Managed tenant isolation

On the Anyray-hosted plan (ANYRAY_MULTI_TENANT=true), every /v1/* request resolves the caller's ark_ key, stored only as a SHA-256 hash, to a tenant, and fails closed. Per-tenant provider keys are encrypted at rest (AES-256-GCM under a KMS-backed key) and decrypted in-process. If the control plane is unreachable or a key cannot be decrypted, the request is rejected rather than served with shared credentials.

Revocation propagates within a ~60 s cache window, and the signed kill-switch is the immediate backstop. The tenant scopes billing and limits, and each key's optional { user, team } adds per-user breakdown within the org.

The Anyray-operated surface (metering)

The only Anyray-run system is the content-free metering and Billing backend: counts, aggregates, and pseudonymous seat hashes, never prompt or response content. It runs on AWS in the EU (Frankfurt): private-subnet compute behind a WAF, TLS 1.2 and 1.3 in transit, AES-256 (AWS KMS) at rest, least-privilege IAM, and continuous threat detection.

A SOC 2 Type I audit is in progress. An external penetration test of this surface completed in July 2026, with no high or critical findings and remediation complete.