Security

Content stays on your inference path; Anyray receives metadata only.

Access is locked down as follows:

• Admin access — a break-glass admin key, with optional SSO and role-based access.
• User access — every /v1/* request needs a per-user key. There is no open mode.
• Provider keys stay server-side.
• Traffic uses supported app settings — no org CA or TLS interception.
• Built-in rate limits protect a public gateway.

Access control

Two separate surfaces:

Surface	Reaches	Gate
Admin — console + /admin/*	Config, spend, traces	Break-glass admin key (ANYRAY_ADMIN_TOKEN), optionally SSO + RBAC.
Inference — /v1/*	LLM traffic	Per-user ark_… or non-human ark_svc_… client key — always required, no opt-out.

Every /v1/* request needs a valid ark_… client key. A missing, unknown, expired, or revoked key is rejected with 401 and a bounded, content-free reason and remediation. Health and enrollment routes are never key-gated.

A fresh gateway blocks traffic until users are enrolled

Connect Billing, start the gateway, then enroll users before routing production traffic. Invalid user keys return 401. A missing lease returns 503 during first-boot grace and 402 afterward; expired leases return 402, and suspension returns 403. Health and enrollment routes remain open.

Console access and RBAC

Credential	How it works
Break-glass admin key (ANYRAY_ADMIN_TOKEN)	Always works and is always owner — no Billing round-trip, so it stays your recovery path even if Billing is unreachable. Treat it like a root password.
Console SSO Optional	Operators sign in with your corporate IdP through WorkOS. The gateway verifies the signed session offline, so it survives restarts and works across replicas.

Two SSO flows — don't confuse them

Console SSO signs operators into the admin console; user SSO enrollment signs users in to get a /v1/* key. Same WorkOS broker, different audience.

Roles and capabilities

Being signed in isn't being fully privileged — every /admin/* route checks a capability. From least to most privileged:

Role	Can do
viewer	Read metadata only — spend dashboard, usage, trace lists/detail with bodies redacted.
auditor	viewer + audit-log reads.
operator	viewer + day-to-day ops — routing, user caps, model aliases, the playground. No secrets, no updates, no audit log.
security_admin	Everything: trace content, provider-key rotation, content mode, minting/revoking user keys, running updates, SSO/RBAC config.
owner	Same capabilities as security_admin — the break-glass role, deliberately not assignable from the IdP.

Map an email or IdP group to a role under Users → Single sign-on; unmapped operators default to viewer. RBAC never shows more content than the org content mode already allows.

Per-user gateway keys (client keys)

Each user gets a personal ark_… key — per-user attribution and revocation without sharing the admin token or a provider key. Raw keys are shown once at mint; only a SHA-256 hash is stored. A revoked key stays listed, marked revoked. Mint, list, and revoke from the console Users page or /admin/client-keys.

Configured SSO enrollment

Connecting SSO changes the enrollment mode on the server; it is not only a console hint. Billing reads the tenant's durable IdP configuration before signing a certificate and carries the result in the control-plane-signed lease. A failed policy read denies enrollment or human-key minting.

Once SSO is configured, regular email links, both MDM provisioning modes, and direct human-key mint cannot produce a standalone working credential. New link/token mints and token rotation return 409; redemption of an already-issued token returns sso_required. MDM must distribute the SSO handoff. Non-human agents and CI use separately managed service keys. A header claiming SSO or another identity is ignored for authorization and audit attribution.

Passwordless enrollment

Without SSO, an admin can mint a personal or shared link, the user runs one command, and the gateway verifies it offline — public-key crypto, no shared secret — then mints their key (default 90-day TTL, auto-refreshed). Once SSO is configured, that regular link returns sso_required and signs no certificate. Full flow: Passwordless enrollment; for IdP-verified identity, SSO enrollment.

Abuse and DoS controls

Per-key and per-IP rate limits are opt-in (set them in .env — see Rate limits); the rest is always on:

• Rate limits — requests per minute per client key and per source IP.
• Concurrency cap — in-flight /v1/* requests per key.
• Body size cap — ANYRAY_MAX_BODY_BYTES (default 32 MiB) → 413.
• Failed-auth throttle — ANYRAY_AUTH_FAIL_LIMIT (default 20 401s per IP per 10 min). Only 401s that look like key guessing count: the request presented a credential and it matched no known key. A recognized key that has merely expired or been revoked isn't a guess (its hash matched a real record) — it gets a plain 401 and never counts, so a whole team behind one NAT riding a just-expired enrollment key can't trip the lockout. Past the limit, only further failing credentialed guesses are answered 429; requests that authenticate (and requests presenting no credential) from the same IP always pass — one misconfigured tool behind a shared NAT can't lock out the valid keys on that IP.
• Enrollment throttle — ANYRAY_RATE_LIMIT_UNAUTH_RPM (default 10/min per IP) on the public on-ramp.

Over-limit requests get a generic 429 + Retry-After — never detail an attacker could tune against. Connection timeouts close slow-drip sockets; L3/L4 floods should never reach the app — front a public gateway with the bundled Caddy TLS edge plus a CDN / anti-DDoS layer, and bind the gateway to localhost (Deploy gateway TLS).

How traffic reaches Anyray

Apps point their SDK base URL at the gateway (OPENAI_BASE_URL / ANTHROPIC_BASE_URL → :8787). There is no org CA, no TLS interception, and no HTTPS_PROXY — the SDK just talks to a different URL.

Updates are locked down

On Docker / Compose, a bundled updater pulls newer Anyray images and recreates the three app containers — from the Update now button in the console's sidebar Updates panel, or automatically for soft updates (image-only releases whose local preflight passes, no breaking flag) on by default, typically within ~15 minutes of a release. Because that needs the Docker daemon, the path is tightly scoped:

• The updater reaches Docker only through a socket proxy, socket mounted read-only — just the pull/recreate calls. Neither the proxy nor the updater publishes a host port.
• Capability-gated. The button — and changing the auto-apply policy — needs the update:run capability (security_admin+); the updater token is injected server-side and never reaches the browser.
• Only Anyray's own images are touched — never your datastores.
• The decision is always local. The vendor publishes a signed advisory + schema-validated manifest; the gateway compares version strings and inspects its own environment. Nothing received from outside is executed.

Hard updates — a new required env var, or a release flagged breaking — are never applied unattended: the update notice in the sidebar Updates panel names exactly what's missing and the operator reconciles infrastructure first. Non-Docker platforms (Kubernetes, AWS, Railway) have no updater; upgrades roll through the platform.

What an update contains, and how to verify it

• Before merge — static analysis (CodeQL, Semgrep), secret scanning, the full test suites, and drift guards that force any required-setting or stack-topology change to be declared in the release manifest your preflight classifies against.
• At publish — every image is vulnerability-scanned by digest, per architecture, before the public tag exists; a fixable HIGH or CRITICAL finding blocks the release. Reproduce it anytime: trivy image public.ecr.aws/anyray/gateway:<tag>.
• In flight — the update recommendation is Ed25519-signed by the vendor key your image pins; the manifest is schema-validated over TLS from the pinned Billing host.
• On your host — every unattended apply is recorded in anyray_audit_log (actor system:auto-update, from/to version, result). Pin ANYRAY_IMAGE_TAG=vX.Y.Z to hold a build, or pin by digest via the per-image overrides (ANYRAY_GATEWAY_IMAGE=public.ecr.aws/anyray/gateway@sha256:…) — which no updater or channel move can advance.

This is not a proof that no vulnerability exists — no vendor can prove absence. It guarantees that known fixable HIGH/CRITICAL CVEs can't ship, that what you pull is what was scanned, and that every automatic change on your host is attributable and reversible.

Server-held provider keys

Provider keys live server-side, never on the client — apps authenticate with a personal ark_… client key, and the real provider key is never returned to a caller. Supply provider keys three ways:

• Env vars (ANYRAY_PROVIDER_KEY_*) at deploy time.
• Runtime admin — the console Providers page or /admin/provider-keys, to rotate without a redeploy. This wins over env vars.
• AWS workload role — Bedrock only, using the gateway's ECS / EC2 / EKS role, so no long-lived key is stored.

In every case the key stays server-side and is audit-logged by provider name only — never the value. The gateway does the provider signing; the optimizer never touches keys or content. See Configure → Providers.

Custom upstream hosts (SSRF protection)

Requests to a custom / OpenAI-compatible upstream pass two gates:

• Host allowlist — is this hostname allowed at all? Set ANYRAY_CUSTOM_HOST_ALLOWLIST or TRUSTED_CUSTOM_HOSTS; a miss is rejected with 400.
• Private-IP block — does the hostname resolve to a private / reserved address (RFC-1918, loopback, cloud metadata)? This catches DNS rebinding, and only TRUSTED_CUSTOM_HOSTS overrides it. A block returns 502 naming the hostname, and logs a content-free ssrf_blocked event.

Private or VPC-internal upstream? Set TRUSTED_CUSTOM_HOSTS

An upstream inside your network (an in-VPC proxy, an on-prem vLLM) resolves to a private IP, so the allowlist alone isn't enough. Set TRUSTED_CUSTOM_HOSTS=your-upstream.internal.example (exact hostnames, comma-separated) and restart the gateway.

Managed tenant isolation

On the Anyray-hosted plan (ANYRAY_MULTI_TENANT=true), every /v1/* request resolves the caller's ark_ key — stored only as a SHA-256 hash — to a tenant, and fails closed: per-tenant provider keys are encrypted at rest (AES-256-GCM under a KMS-backed key) and decrypted in-process; if the control plane is unreachable or a key can't be decrypted, the request is rejected — never served with shared credentials. Revocation propagates within a ~60 s cache window; the signed kill-switch is the immediate backstop. The tenant scopes billing and limits; each key's optional { user, team } adds per-user breakdown within the org.

The Anyray-operated surface (metering)

The only Anyray-run system is the content-free metering / Billing backend — counts, aggregates, and pseudonymous seat hashes, never prompt or response content. It runs on AWS in the EU (Frankfurt): private-subnet compute behind a WAF, TLS 1.2/1.3 in transit, AES-256 (AWS KMS) at rest, least-privilege IAM, continuous threat detection.

A SOC 2 Type I audit is in progress; an external penetration test of this surface completed in July 2026 (no high or critical findings; remediation complete).