Security

Content stays on your inference path; Anyray receives metadata only.

Access control

Surface	Reaches	Gate
Admin (console + /admin/*)	Config, spend, traces	ANYRAY_ADMIN_TOKEN, SSO + RBAC, or an aak_… key
Inference (/v1/*)	LLM traffic	ark_… or ark_svc_… client key, always required

The key's identity wins over x-anyray-metadata for attribution. Health and enrollment routes are never key-gated. A fresh gateway blocks traffic until users are enrolled.

Condition	Status
Missing, unknown, expired, or revoked key	401, content-free reason
No lease, during first-boot grace	503 entitlement_inactive
No lease after grace, or expired	402
Suspended	403

Console access and RBAC

Credential	How it works
Admin key (ANYRAY_ADMIN_TOKEN)	Always owner, works with Billing down. Treat as a root password.
Console SSO Optional	Your IdP via WorkOS; session verified offline, works across replicas.
Work email	Emailed code. Add under Users → Members; DNS-verified domains get viewer. Max role operator. 12 h sessions.
aak_… key	Capability subset for automation. See API access.

Console SSO signs in operators; user SSO enrollment signs in users for a /v1/* key. A failed IdP policy read denies enrollment and human-key minting. Enforcement detail: SSO reference.

CLI admin sign-in uses /sso/cli with purpose: admin to return a role-scoped console session. It follows the console's sign-in lane: IdP role mappings with SSO, or an emailed code otherwise. The email lane requires a verified address, admits invited members or DNS-proven domains, and refuses revoked or unlisted addresses. Roles stop at operator; sessions last 12 hours. CLI admin sign-in creates no client key, Billing account, or membership. Sessions are handed out once, last up to 12 hours, and are verified offline against the deployment's tenant and deployment id. Successful sign-ins appear in the admin audit trail as cli-sso.

Roles and capabilities

Role	Can do
viewer	Own trace metadata and own agents', and pages colleagues shared with them.
auditor	Everyone's trace metadata, spend, usage, AI activity, audit log.
operator	auditor minus audit log, plus routing, user caps, aliases, playground, Diagnose, publishing client test results.
security_admin	Everything: trace content, key rotation, content mode, user keys, updates, SSO, removing anyone's shared page.
owner	Same as security_admin. Admin key only, never from the IdP.

A new node-written outbound origin needs providerkeys:manage (providers:write for API keys), in addition to routing:write, even on a node without a provider. A new pairing of a credential with a node-written base URL (including its path) or custom_endpoints also needs that grant. A host injected from the node's own provider key is key-sourced and needs no extra grant. Copying that key's Base URL into routing is node-written, even when the URLs match. An inherited host is node-sourced for the child credential.

The comparison covers roots, groups and all branches, regardless of strategy. Credential selectors distinguish named pins, inline bags and unpinned resolution. Inline bags and headers use separate keyed HMACs. Every inline credential field counts, including beside a pin; changing or removing one needs a new grant. Header changes on node-sourced hosts need a new grant. The MACs stay inside the save comparison and need no private admin token. Save checks read provider keys from authority; an unavailable authority returns 503 provider_keys_authority_unavailable. Operators can reorder, move, duplicate or remove existing grants. A new grant without permission records the actor, code and route in the admin audit and returns 403 outbound_host_requires_providerkeys_manage.

Trace bodies stay redacted below security_admin, and never exceed the org content mode. A shared page opens only for the people it is shared with, whatever their role: security_admin can remove one but not open it. Map emails or IdP groups to roles under Users → Identity → SSO configuration: unmapped means viewer, several groups give the highest, an email mapping overrides groups. A viewer sees an agent's traffic once it is assigned under Agents → Belongs to (console-minted agents default to their creator; unassigned ones are visible to auditor and up). The assignment also lets that person read back, by handle, the originals the agent's optimizations stored. A shared bot with delegated attribution never grants this.

Every trace-body read is audit-logged. The admin key logs as principal kind token, or local with actor admin from a console session only it can create.

Upgrading from a release before self-scoped viewers

viewer used to be org-wide. Remap those groups to auditor after the gateway rollout finishes; a mixed fleet grants more than this table, never less.

Capability bundles

A bundle is a named capability set mapped to an IdP group, for access no role expresses (for example observability:read for everyone's traces, or observability:read-content for bodies, without the rest of security_admin). observability:read-own is the viewer floor.

1

Add a bundle

Users → Identity → SSO configuration → Capability bundles → Add bundle. Tick capabilities (only ones you hold are listed).

2

Map a group

Under Group → bundle mapping, pick the group and bundle.

3

Save

Applies at each operator's next sign-in.

Rule	Detail
Exact grant	Only what it lists
Only what you hold	An operator cannot grant providerkeys:manage
Groups combine	Union of all mapped bundles
Reserved names	owner and other role names
Deleted bundle	Grants nothing
Email → role	Still wins

Finish the gateway rollout before relying on a bundle

An older gateway reads only the role, which is viewer for bundle-only groups, so a narrow bundle can grant more than asked.

Per-user gateway keys (client keys)

Key	Storage	Shown
ark_… (person)	SHA-256 hash only	Once, at mint
ark_svc_… (agent)	Hash, plus AES-256-GCM copy keyed from ANYRAY_ADMIN_TOKEN	Agent Setup tab, needs clientkeys:secret (scope keys:secret, never on an aak_ key)

Only the hash authenticates. Agent-key reads and refusals are audited; an unrecordable read is refused. Mint and revoke under Users or /admin/client-keys; revoked keys stay listed. ANYRAY_VERIFIED_DEV_KEY_TTL_DAYS (default 90) caps lifetime at the enrollment certificate's expiry.

Rotating the admin token

Stored agent keys become unreadable (the console offers new ones). Without ANYRAY_CONTENT_KEY, optimizer context-stash rows (at most 48 h old) are orphaned too.

Enrollment: Passwordless · MDM. Identity headers are ignored for authorization and audit.

Abuse and DoS controls

Control	Setting
Rate limits (opt-in)	Per key and per IP: Rate limits
Concurrency cap	In-flight /v1/* per key
Body size	ANYRAY_MAX_BODY_BYTES, 32 MiB, 413
Failed auth	ANYRAY_AUTH_FAIL_LIMIT: after 20 wrong keys or admin tokens per IP in 10 min, further failures get 429 + Retry-After. A wrong admin token sent beside a valid session or key counts too, once per value. A valid credential on its own still passes
Unauthenticated	ANYRAY_RATE_LIMIT_UNAUTH_RPM, 10/min per IP (on-ramp, keyless health reports, console sign-in)
Slow clients	Connection timeouts close slow-drip sockets

Expired or revoked keys never count as failed auth. Put public gateways behind the bundled Caddy edge plus a CDN, bound to localhost (Gateway TLS).

The container runs as a non-root user

The entrypoint chowns the state directories, then drops to node (uid 1000) and stays PID 1.

docker compose exec gateway id
# uid=1000(node) gid=1000(node)

When it stays root

• You pinned a user (securityContext.runAsUser, ECS user, docker run --user): point the mount at that uid.
• The chown fails (read-only mount, NFS/EFS root_squash): stays root and logs why.
• ANYRAY_RUNTIME_USER names a missing account: logs and stays root.

Bind-mounted host directories are chowned to uid 1000 on the host too.

How traffic reaches Anyray

SDKs point OPENAI_BASE_URL or ANTHROPIC_BASE_URL at :8787. Clients need no org CA, TLS interception, or HTTPS_PROXY.

Going out, the gateway can send its provider, metering, and MCP calls through your corporate forward proxy: Behind a forward proxy.

Updates are locked down

The gateway dispatches updates through the existing Compose updater or AWS quicklaunch Lambda. Soft updates on by default apply image-only releases that pass preflight. Hard updates are never applied unattended; they require operator reconciliation.

POST /admin/v1/settings/updates/run (legacy /admin/update/run) requires settings:update, the existing update:run capability and deployment ownership. No aak_ key can trigger it, even with a stored grant. Cross-site browser mutations are refused. Role grants are unchanged.

A shared pending claim precedes dispatch and suppresses duplicate manual and automatic attempts when the applier may have received the request. The response contains attempt metadata; updater credentials, response bodies and raw errors stay private. Redirects are refused. See updates for status and platform limits.

Scope	Detail
Socket proxy	Read-only socket, pull and recreate only, no host port
Capability-gated	update:run (security_admin+), plus deployment ownership; token never reaches the browser
Anyray images only	Datastores untouched
Local decision	Signed advisory, validated manifest; nothing received is executed

What an update contains, and how to verify it

Stage	Guarantee
Before merge	CodeQL, Semgrep, secret scanning, tests, manifest drift guards
Scanning	Per digest and arch before tagging; fixable HIGH/CRITICAL blocks. trivy image public.ecr.aws/anyray/gateway:<tag>
Signing	Images and Helm chart, Sigstore cosign by digest
In flight	Ed25519-signed by the pinned vendor key; manifest over TLS from the pinned Billing host
On your host	anyray_audit_log, actor system:auto-update

Verify an image (keyless, Rekor-logged)

cosign verify \
  --certificate-identity-regexp '^https://github\.com/anyrayHQ/monorepo/\.github/workflows/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  public.ecr.aws/anyray/gateway:<tag>

The same command verifies the chart (public.ecr.aws/anyray/anyray:<chart-version>).

Pinning a build, and signing scope

Pin ANYRAY_IMAGE_TAG=vX.Y.Z or a digest (ANYRAY_GATEWAY_IMAGE=public.ecr.aws/anyray/gateway@sha256:…).

Image signatures prove the CI build, not a company. anyray-connect is signed to Othentic Labs LTD (Developer ID + notarized on macOS, Authenticode on Windows).

Server-held provider keys

Provider keys stay server-side and are never returned to callers; changes are audit-logged by provider name only. The optimizer never touches keys or content.

Source	Detail
Env vars	ANYRAY_PROVIDER_KEY_*
Runtime admin	Providers page or /admin/provider-keys; wins over env
AWS workload role	Bedrock only, ECS/EC2/EKS role

See Provider keys.

Custom upstream hosts (SSRF protection)

Host named by	Authorized by
You (routing config, PUT /admin/routing-config, ANYRAY_UPSTREAM_URL)	A routing save with the origin permissions above, or deployment env; private addresses OK; exact origin only
You (a provider key's Base URL on the Providers page)	The admin write itself; private addresses OK; exact origin only
A caller (x-anyray-custom-host, x-anyray-config)	ANYRAY_CUSTOM_HOST_ALLOWLIST (unset refuses all, miss is 400; chatgpt.com built in for pass-through seat requests only) and a private-IP block (502, ssrf_blocked event)

Cloud metadata (169.254.169.254, metadata.google.internal) is refused on every path. TRUSTED_CUSTOM_HOSTS is no longer needed for a configured upstream; it still applies to caller hosts and private-address MCP connectors.

Managed tenant isolation

With ANYRAY_MULTI_TENANT=true, each ark_ key hash resolves to a tenant and fails closed. Per-tenant provider keys are AES-256-GCM encrypted (KMS-backed); an unreachable control plane or failed decrypt rejects the request. Revocation propagates within ~60 s; the signed kill-switch is immediate. Optional { user, team } per key adds per-user breakdown.

The Anyray-operated surface (metering)

Anyray runs only the metering and Billing backend: counts, aggregates, pseudonymous seat hashes, never prompt or response content. AWS EU (Frankfurt), private subnets behind a WAF, TLS 1.2/1.3, AES-256 (KMS) at rest, least-privilege IAM, threat detection. SOC 2 Type I (6 July 2026), Type II in progress (Compliance). July 2026 external pentest: no high or critical findings.