Deploy on Kubernetes

Deploy the full stack on Kubernetes with the official Helm chart.

Prerequisites

Kubernetes 1.24+
Helm 3.10+
A default StorageClass (or set *.storageClass in your values file)
A node with capacity for the four pods (gateway, optimizer, proxy, Postgres) — node sizing per cloud under System requirements
An Anyray deployment token (adt_…) from app.anyray.ai (setup wizard, or Settings → Deployments → New deployment)
Only for the scripted quickstart: git + openssl on your PATH and the install repo cloned (git clone https://github.com/anyrayHQ/install anyray && cd anyray) — the OCI install below needs neither.

Amazon EKS: install the EBS CSI driver and set a default StorageClass

A new EKS cluster ships neither the EBS CSI driver nor a default StorageClass, so the chart's PersistentVolumeClaims stay Pending and no pods start. One-time setup:

Install the EBS CSI driver (IRSA) on EKS
eksctl create iamserviceaccount --cluster <cluster> --region <region> \
  --namespace kube-system --name ebs-csi-controller-sa \
  --role-name AmazonEKS_EBS_CSI_DriverRole \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve

eksctl create addon --cluster <cluster> --region <region> \
  --name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::<account-id>:role/AmazonEKS_EBS_CSI_DriverRole --force

Then mark a default StorageClass backed by the driver:

gp3-default-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"

(eksctl can install the driver at cluster creation — iam.withOIDC: true plus the aws-ebs-csi-driver add-on — but you still apply the default StorageClass yourself.)

Install

Anyray ships as a published Helm chart — pull it straight from its OCI registry (oci://public.ecr.aws/anyray/anyray, anonymous pulls, no credentials), the same versioned artifact your GitOps pipeline pins. No repo clone, no scripts. The steps below install the OCI chart directly; if you'd rather a script mint your secrets and install in one command, jump to the scripted quickstart. Pulling images from a private registry while keeping the required Billing app egress? See Private-registry install.

Choose a namespace

export ANYRAY_NAMESPACE="team-ai"   # replace with your target namespace
kubectl create namespace "$ANYRAY_NAMESPACE"

Create the anyray-secrets Secret

The chart reads its secrets from a Kubernetes Secret named anyray-secrets. Manage it with your own tooling — External Secrets Operator, Sealed Secrets, SOPS — or apply a reviewed manifest once. It holds the keys shown under Generated file examples: ANYRAY_ADMIN_TOKEN, the ANYRAY_CONTENT_KEY (required by the default encrypted content mode), ANYRAY_DEPLOYMENT_TOKEN (your adt_…), and an ANYRAY_PSEUDONYM_SALT you generate once and keep in-cluster.

Let setup.sh mint the keys

Don't want to hand-roll keys? Run setup.sh --k8s from a clone of the install repo — it emits a ready anyray-secrets.yaml (and my-values.yaml) you can apply with any install method, including this one. See the scripted quickstart.

kubectl apply -n "$ANYRAY_NAMESPACE" -f anyray-secrets.yaml

Write your values file

Non-secret config goes in my-values.yaml (secrets stay in the Secret above):

my-values.yaml
host: "<gateway-ingress-hostname>"                 # gateway ingress hostname
gateway:
  publicUrl: https://<gateway-ingress-hostname>
  consolePublicUrl: https://<gateway-ingress-hostname>
  metering:
    enabled: true                        # required — Billing app metering, metadata-only
# image.tag omitted → pinned to the chart's appVersion (recommended for production).

Full examples — Ingress + TLS, managed Postgres, cluster policy — under Generated file examples and Configuration.

Install the chart

helm install anyray oci://public.ecr.aws/anyray/anyray \
  --version <x.y.z> -f my-values.yaml --namespace "$ANYRAY_NAMESPACE"

List released versions with helm show chart oci://public.ecr.aws/anyray/anyray. Each chart version pins its image tag via appVersion, so every install is a fixed, auditable build. Prefer a classic HTTP repo? helm repo add anyray https://charts.anyray.ai serves the same chart — also published on Artifact Hub.

Wait for pods

kubectl rollout status -n "$ANYRAY_NAMESPACE" deployment/anyray-gateway
kubectl rollout status -n "$ANYRAY_NAMESPACE" deployment/anyray-proxy

Expose the gateway and console

Services default to ClusterIP. Expose them at your host endpoint — to your org network / VPN only, never 0.0.0.0/0. Recommended is Ingress; the chart routes one host: / and /admin → console proxy, /v1 → gateway:

ingress:
  enabled: true
  className: nginx        # your ingress controller
  # annotations / tls: see the commented example in values.yaml
gateway:
  publicUrl: "https://<gateway-ingress-hostname>"
  consolePublicUrl: "https://<gateway-ingress-hostname>"

then re-run the helm upgrade (see Upgrade) to apply it.

The console's /admin/* calls must go through the proxy (it injects the admin key) — routing /admin straight to the gateway makes console login 401 behind the Ingress. For LoadBalancer and the dev-only NodePort fallback, see Exposing services.

Verify the install is healthy

With Ingress / LoadBalancer, the console is at your host endpoint and the gateway base URL for tools is …/v1:

curl -fsI https://<gateway-ingress-hostname>/ && echo "console ok"

Open https://<gateway-ingress-hostname>/ and sign in with the admin key (ANYRAY_ADMIN_TOKEN in the anyray-secrets Secret). With NodePort, use the node IP: curl -fs http://<node-ip>:30787/ && echo "gateway ok".

Install with ArgoCD / GitOps

Point an ArgoCD Application straight at the OCI chart — the only out-of-band piece is the anyray-secrets Secret (managed by your secret tooling, above):

anyray-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: anyray
  namespace: argocd
spec:
  project: default
  source:
    repoURL: public.ecr.aws/anyray     # registry, no chart name
    chart: anyray
    targetRevision: 0.4.40             # a released chart version
    helm:
      valuesObject:
        host: "<gateway-ingress-hostname>"
        gateway:
          publicUrl: https://<gateway-ingress-hostname>
          consolePublicUrl: https://<gateway-ingress-hostname>
  destination:
    server: https://kubernetes.default.svc
    namespace: team-ai

kubectl apply -n argocd -f anyray-application.yaml

Pin targetRevision to a released chart version (list them with helm show chart oci://public.ecr.aws/anyray/anyray). The image tag needs no pinning — each chart version ships a fixed appVersion and images default to it, so every sync deploys an exact, auditable build. Only override image.tag to deliberately track a moving channel.

Scripted quickstart with `setup.sh`

Prefer one command that generates your secrets and installs in a single pass? Clone the install repo and run setup.sh --k8s — it mints the admin key, content key, and pseudonym salt, wires the deployment token, and installs the bundled chart. Needs git + openssl.

Generate secrets and a values file

git clone https://github.com/anyrayHQ/install anyray && cd anyray
./setup.sh --k8s --connect <adt_token> --host <gateway-ingress-hostname> --namespace "$ANYRAY_NAMESPACE"
# Emits: anyray-secrets.yaml  my-values.yaml

--connect wires the deployment to the Anyray Billing app for usage metering (required; the rollups are metadata-only — content is never sent to Anyray).

Use the cluster's ingress endpoint, not a node IP

--host is the external DNS hostname clients reach the gateway/console at — your Ingress or LoadBalancer endpoint (e.g. anyray.example.com). The chart writes it into the Ingress host: rule, which matches on the HTTP Host header, so a raw node IP does not work — and never the machine you run setup.sh from.

Apply the Secret and install

kubectl apply -n "$ANYRAY_NAMESPACE" -f anyray-secrets.yaml
helm install anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE"

The generated files are install-method-agnostic — feed anyray-secrets.yaml / my-values.yaml into the OCI install above instead of ./helm if you'd rather pull the published chart than the bundled one. Don't commit anyray-secrets.yaml — it's gitignored.

Then expose and verify as in the steps above.

After deployment — roll out to your users

Users reach the gateway at your --host Ingress / LoadBalancer endpoint (tools use the …/v1 base URL).

Add a provider key

Provider keys stay server-side — callers never hold them. Add at least one from the console Providers page; multi-field providers (Bedrock, Vertex, Azure) are configured there too. To set a key declaratively instead, add it under gateway.extraEnv in my-values.yaml and helm upgrade. See Configure → Providers.

Send a test request

On the Providers page use Send test request, then confirm a row under Spend and a trace under Traces — that exercises routing, attribution, and the trace store end to end.

Enroll your users

/v1 is always gated — each user enrolls before their traffic is accepted. Mint an enrollment link (enl_…) on the console Users page; each user runs it on their own machine:

curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- --enroll https://…/enroll/enl_…

That points their coding tools (Claude Code, Cursor, Codex, …) at the gateway and mints a personal key bound to their email — no provider keys ever touch a user's machine. For a whole fleet, push one provisioning token via your MDM — see Bulk enrollment with your MDM.

Track the migration and confirm the cutover

Watch the Users roster move from Pending to Enrolled; unenrolled /v1 traffic is always rejected (401). See Configure → Access control.

You're live

Your users' everyday AI tools now run through Anyray — spend attributed by user/team/model, content handled per your privacy mode, the optimizer cutting cost on the hot path.

Generated file examples

The installer emits two files with different jobs:

File	Purpose	Safe to commit?
`anyray-secrets.yaml`	Kubernetes Secret: admin key, content-encryption key (required by the default `encrypted` mode), optimizer shared secret, Postgres password, Billing app metering token.	No
`my-values.yaml`	Helm values: host, image tag, exposure, scheduling, other non-secret settings.	Yes, if it stays secret-free

Treat anyray-secrets.yaml as a generated artifact — review, apply, delete/regenerate to rotate. Put cluster-specific edits in my-values.yaml so upgrades stay simple.

anyray-secrets.yaml

Values shown as base64 placeholders (Kubernetes Secret.data expects base64):

anyray-secrets.yaml
# Generated by setup.sh --k8s - do not commit. Delete and re-run to rotate.
apiVersion: v1
kind: Secret
metadata:
  name: anyray-secrets
  namespace: team-ai # only present when you pass --namespace
type: Opaque
data:
  ANYRAY_ADMIN_TOKEN: <base64-admin-token>
  ANYRAY_OPTIMIZER_TOKEN: <base64-optimizer-shared-secret>
  ANYRAY_CONTENT_KEY: <base64-aes-256-gcm-key>
  POSTGRES_PASSWORD: <base64-postgres-password>

  # Billing app metering — written by setup.sh --k8s --connect <adt_token>.
  ANYRAY_DEPLOYMENT_TOKEN: <base64-billing-app-deployment-token>
  ANYRAY_PSEUDONYM_SALT: <base64-pseudonym-salt>

To reconnect the deployment to Anyray Billing app later, re-run:

./setup.sh --k8s --connect <adt_token> --host <gateway-ingress-hostname> --namespace "$ANYRAY_NAMESPACE"

It rewrites only the Billing app token keys, keeps the existing pseudonym salt, and leaves the rest of the Secret alone.

my-values.yaml

The generated values file is intentionally small and contains no secrets:

my-values.yaml
# Anyray Helm values - safe to commit (no secrets here).
# Namespace: team-ai (set with kubectl -n / helm --namespace; not a Helm value).
host: "<gateway-ingress-hostname>"

# Image tag is pinned to this chart version's appVersion by default
# (recommended for production — every deploy is a fixed, auditable build).
# To test the newest build instead, uncomment and pair with pullPolicy: Always:
# image:
#   tag: latest
#   pullPolicy: Always

# Added by setup.sh --k8s --connect <adt_token> for Billing app metering.
gateway:
  metering:
    enabled: true

For an Ingress-based production install, keep the same file and add the networking values:

my-values.yaml
host: "<gateway-ingress-hostname>"

# Image tag omitted → pinned to the chart's appVersion (recommended for production).

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  tls:
    - secretName: anyray-tls
      hosts:
        - "<gateway-ingress-hostname>"

gateway:
  publicUrl: "https://<gateway-ingress-hostname>"
  consolePublicUrl: "https://<gateway-ingress-hostname>"
  metering:
    enabled: true

Keep LoadBalancer / NodePort service settings in my-values.yaml too. Don't edit the chart's helm/values.yaml — layer your install-specific values on top.

Configuration

Keep secrets in Kubernetes Secrets and reference them from values; don't put provider keys or database passwords directly in Git.

Cluster policy

For clusters that require a service account, private registry, scheduling rules, or pod security context:

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/anyray

image:
  pullSecrets:
    - name: ghcr-pull-secret

nodeSelector:
  workload: ai-platform
tolerations:
  - key: dedicated
    operator: Equal
    value: ai-platform
    effect: NoSchedule
podSecurityContext:
  runAsNonRoot: true
containerSecurityContext:
  allowPrivilegeEscalation: false

nodeSelector, affinity, tolerations, topologySpreadConstraints, and priorityClassName apply to every pod, but can also be set per component on the gateway, optimizer, proxy, or postgres blocks. A component-level value replaces the global for that field (no merge); unset inherits. Example — Postgres on its own node pool:

postgres:
  nodeSelector:
    node.kubernetes.io/instance-type: t3.medium
  tolerations:
    - key: dedicated
      operator: Equal
      value: data-plane
      effect: NoSchedule

Each image can also be mirrored. Leave the app tag empty to keep the chart's pinned appVersion, or set a concrete vX.Y.Z:

images:
  gateway:
    repository: registry.example.com/anyray/gateway
    tag: ""            # inherits the chart appVersion
  optimizer:
    repository: registry.example.com/anyray/optimizer
    tag: ""

Gateway hardening

Set these when exposing Anyray behind an Ingress, reverse proxy, or org-wide gateway policy:

gateway:
  hsts: "true"
  trustProxy: "true"
  rateLimitRpm: "600"
  rateLimitIpRpm: "1200"
  rateLimitUnauthRpm: "60"
  maxConcurrentRequests: "20"
  maxBodyBytes: "33554432"

For uncommon environment variables, use gateway.extraEnv, optimizer.extraEnv, or proxy.extraEnv.

Managed Postgres

The bundled Postgres is a single-replica StatefulSet backing the spend + trace store. For a managed database, disable it and point the gateway at your endpoint — the chart reads it as ANYRAY_OBSERVABILITY_DB_URL. Prefer a Secret reference over an inline URL:

postgres:
  enabled: false
  external:
    databaseUrlSecretKeyRef:
      name: anyray-external-postgres
      key: DATABASE_URL
    # Or, less safely, an inline URL:
    # databaseUrl: postgresql://user:password@db.example.com:5432/postgres

Managed databases usually require SSL — set it in the URL

Most managed Postgres (RDS, Cloud SQL, Azure Database) reject unencrypted connections, and the gateway takes SSL settings only from the connection string: add ?sslmode=require (or ?sslmode=no-verify to skip cert verification without the provider's CA bundle). Also URL-encode reserved characters (@ : / ? # %) in the password — a password p@ss?1 becomes:

postgresql://user:p%40ss%3F1@db.example.com:5432/postgres?sslmode=require

The gateway auto-creates the anyray_traces / anyray_observations tables on first use; trace content follows ANYRAY_CONTENT_MODE — AES-256-GCM ciphertext by default, omitted in off, readable only in deploy-gated plaintext.

Exposing services

Services default to ClusterIP. Pick one way to expose the proxy (console) and gateway — to your org network / VPN only, never 0.0.0.0/0:

Method	How
Ingress (recommended)	Set `ingress.enabled: true`, fill in `ingress.className` and cert-manager annotations (see the commented example in `values.yaml`). Resolves at your `--host` endpoint — `/` console, `/v1` gateway
Gateway API	Set `httpRoute.enabled: true` and `httpRoute.parentRefs` (the Gateway your platform team runs). Generates an `HTTPRoute` with the same routing as Ingress — the modern successor to Ingress. Needs the Gateway API CRDs + chart 0.4.41+
LoadBalancer	Set `proxy.service.type` and `gateway.service.type` to `LoadBalancer`, point your `--host` DNS at the load balancer address, and lock it down with `loadBalancerSourceRanges` or your firewall
NodePort (dev/quick)	Set `proxy.service.type: NodePort`, `proxy.service.nodePort: 30000`, `gateway.service.type: NodePort`, `gateway.service.nodePort: 30787`. Reachable by node IP; local/dev clusters only

Service name constraints

The gateway and optimizer Services use bare (unprefixed) names. The proxy dials them by in-cluster FQDN — gateway.<namespace>.svc.<clusterDomain> — because nginx's resolver does not apply Kubernetes DNS search domains. Override clusterDomain (default cluster.local) only if your cluster uses a non-default DNS domain, and install the chart in its own namespace to avoid name collisions.

Using Gateway API? Attach the route to a Gateway your platform team runs — the chart generates only the HTTPRoute (your app's routing), not the Gateway itself (that's platform infra, like the IngressClass):

my-values.yaml
httpRoute:
  enabled: true
  parentRefs:
    - name: my-gateway
      namespace: gateway-system   # omit if the Gateway is in this namespace
      sectionName: https          # a specific listener on the Gateway (optional)
  # hostnames default to `host`

Enable either ingress or httpRoute, not both.

Upgrade

The image tag is pinned to the chart's appVersion, so moving to a newer chart is what advances the version. Bump the chart version from the OCI registry:

helm upgrade anyray oci://public.ecr.aws/anyray/anyray \
  --version <chart-version> -f my-values.yaml --namespace "$ANYRAY_NAMESPACE"

Installed from a clone (./helm) instead? git pull to refresh the chart, then helm upgrade anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE".

A helm upgrade with no version change rolls nothing

helm upgrade only restarts pods when the rendered manifest changes, so re-running it against an unchanged chart is a no-op — the gateway keeps serving the old image and the console's version won't move. To pin a specific build without pulling a new chart, set the tag explicitly (the top-level image.tag from my-values.yaml):

helm upgrade anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE" \
  --set image.tag=vX.Y.Z

If you pin a moving tag (image.tag: latest + pullPolicy: Always), a same-tag upgrade still won't re-pull a cached image — force a fresh pull with kubectl -n "$ANYRAY_NAMESPACE" rollout restart deployment/anyray-gateway.

Confirm the roll landed — the console's deployment version reflects the running image, so it catches up within a couple of minutes once the new pod is Ready:

Verify the running version
kubectl -n "$ANYRAY_NAMESPACE" rollout status deployment/anyray-gateway
kubectl -n "$ANYRAY_NAMESPACE" get deployment anyray-gateway \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

There's no separate migration step — the gateway self-migrates its database schema on boot (serialized across replicas). Durable state, including the user access keys minted by anyray-connect, lives in Postgres and is shared by every replica, so a rolling upgrade or scaling gateway.replicaCount up or down never drops keys and no user has to re-run anyray-connect. See Configure → Schema and data migrations.

System requirements

The chart's pods request ~1.5 vCPU / ~3 GB in total (limits ~3.5 vCPU / ~4 GB). The optimizer is the largest single pod — 1 vCPU / 2 GB — because it runs onnxruntime re-rank/OCR synchronously on the Node event loop and starves at fractional vCPU. A 2-vCPU / 4-GB node fits it; size up only for high trace/spend volume (Postgres).

Managed Kubernetes	Node type
AWS EKS	`t3.medium` (2 / 4)
GCP GKE	`e2-medium` (2 / 4)
Azure AKS	`Standard_B2ms` (2 / 4)

Troubleshooting

These come up on a first install backed by a cloud block store (AWS EBS, GCP PD, Azure Disk) or on mixed-architecture clusters. The bundled chart (≥ 0.4.9) handles all three out of the box — the notes are for forks and custom value overrides.

PVCs stuck in Pending

kubectl get pvc -n "$ANYRAY_NAMESPACE"
# data-anyray-postgres-0   Pending   ...   (no STORAGECLASS)

The chart's volumes bind the cluster's default StorageClass; a fresh EKS cluster often has neither the EBS CSI driver nor a default class, so the claims never provision. Install the driver and mark a default class — see Prerequisites — and the PVCs bind on the next reconcile (no reinstall needed).

Postgres won't start — lost+found

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.

An ext4 block-store volume always has a lost+found directory at its root, and Postgres's initdb refuses any non-empty data directory. The chart avoids this by initializing into a subdirectory of the mount (PGDATA=/var/lib/postgresql/data/pgdata). If you mount your own Postgres volume, point PGDATA (or a subPath) at a subdirectory — never the mount root.

Reset a stuck Postgres volume

If the bundled Postgres failed to initialize on a first install (e.g. it crash-looped on the lost+found error before the chart fix), its PVC may be half-provisioned. After upgrading the chart, reset just that volume. Safe only when Postgres never finished initializing (no real data to lose):

Clean-reset the bundled Postgres
# Inspect first
kubectl get pvc -n "$ANYRAY_NAMESPACE"

# Stop Postgres so it releases its volume (gentler than deleting the StatefulSet)
kubectl scale statefulset -n "$ANYRAY_NAMESPACE" anyray-postgres --replicas=0

# Delete the empty PVC (StatefulSet volumeClaimTemplate claim)
kubectl delete pvc -n "$ANYRAY_NAMESPACE" data-anyray-postgres-0

# Re-apply — scales the StatefulSet back to 1 and provisions a fresh PVC
helm upgrade anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE"

Never delete the gateway or optimizer PVCs

anyray-gateway-data and anyray-optimizer-data hold your routing config, provider keys, and spend/audit logs. They carry a helm.sh/resource-policy: keep annotation for exactly this reason. Only reset the Postgres PVC above.

Mixed-architecture clusters (arm64 + amd64 nodes)

No architecture nodeSelector needed

Every image the chart ships is a multi-arch manifest list (linux/amd64 + linux/arm64) — Kubernetes schedules each pod onto any node and pulls the matching architecture.

If you mirror images into a private registry, mirror both architectures (use docker buildx imagetools create, not a single-arch docker pull/push), or pods fail to schedule on the architecture you dropped. To pin a workload to a node pool, set nodeSelector / affinity / tolerations — globally or per component — as under Cluster policy.

Prerequisites​

Install​

Install with ArgoCD / GitOps​

Scripted quickstart with setup.sh​

After deployment — roll out to your users​

Generated file examples​

Configuration​

Exposing services​

Upgrade​

System requirements​

Troubleshooting​