Kubernetes reference

Depth behind the Kubernetes guide.

Install with ArgoCD / GitOps

Point an ArgoCD Application at the chart repo. The only out-of-band piece is the anyray-secrets Secret, managed by your secret tooling as in the install steps.

anyray-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: anyray
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.anyray.ai
    chart: anyray
    targetRevision: <x.y.z>            # a released chart version
    helm:
      valuesObject:
        host: "<gateway-ingress-hostname>"
        gateway:
          publicUrl: https://<gateway-ingress-hostname>
          consolePublicUrl: https://<gateway-ingress-hostname>
        postgres:
          storage: 150Gi               # immutable after the first sync
  destination:
    server: https://kubernetes.default.svc
    namespace: team-ai

kubectl apply -n argocd -f anyray-application.yaml

Pin targetRevision to a released chart version (list them with helm search repo anyray/anyray --versions). With no image.tag, each chart version deploys its fixed appVersion and the nightly roll stays off, so Git decides what runs. To upgrade, raise targetRevision; the changelog lists what each release changed.

Using the policy-stable channel under ArgoCD instead

The nightly roll pins each Deployment to that night's digest (<app>:policy-stable@sha256:…), which ArgoCD reads as drift. Setting image.tag: policy-stable and ignoring the image field keeps the pin:

  ignoreDifferences:
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - .spec.template.spec.containers[].image
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

Not with ServerSideApply

ArgoCD 2.9 with ServerSideApply=true has applied this as gateway env entries with names and no values. Use this only with client-side apply, and check a synced Deployment still carries its valueFrom entries. A stripped ANYRAY_ADMIN_TOKEN shows on /admin/health as secrets.adminToken: env_blank.

Already syncing a deployment below 150Gi? Add one line first

ArgoCD and Flux render with helm template, which reports an install on every sync, so the chart's fresh-install storage floor applies to each one. A deployment already running a smaller volume keeps rendering only with:

        postgres:
          acknowledgeSmallVolume: true

The live volume is untouched either way. Its volumeClaimTemplate cannot be resized in place.

Scripted quickstart with setup.sh

setup.sh --k8s mints the admin key, content key, and pseudonym salt, wires the deployment token, and installs the bundled chart in one pass. It needs git and openssl on PATH.

--host is the external DNS hostname clients reach the gateway and console at, meaning your Ingress or LoadBalancer endpoint. The chart writes it into the Ingress host: rule, which matches on the HTTP Host header, so a raw node IP does not work, and neither does the machine you run setup.sh from.

Generate secrets and a values file

git clone https://github.com/anyrayHQ/install anyray && cd anyray
./setup.sh --k8s --connect <adt_token> --host <gateway-ingress-hostname> --namespace "$ANYRAY_NAMESPACE"
# Emits: anyray-secrets.yaml  my-values.yaml

Apply the Secret and install

kubectl apply -n "$ANYRAY_NAMESPACE" -f anyray-secrets.yaml
helm install anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE"

The generated files are install-method-agnostic. Feed them into the OCI install instead of ./helm to pull the published chart. Do not commit anyray-secrets.yaml, which is gitignored. Then expose and verify as in the install steps.

Re-running the same command reconnects the deployment to the Billing App later. It rewrites only the Billing App token keys, keeps any legacy pseudonym seed, and leaves the rest of the Secret alone.

Default StorageClass on EKS and OKE

Amazon EKS: install the EBS CSI driver and mark a default class

A new cluster ships neither the EBS CSI driver nor a default StorageClass, so the chart's PersistentVolumeClaims stay Pending and no pods start. One-time setup:

Install the EBS CSI driver (IRSA) on EKS

eksctl create iamserviceaccount --cluster <cluster> --region <region> \
  --namespace kube-system --name ebs-csi-controller-sa \
  --role-name AmazonEKS_EBS_CSI_DriverRole \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve

eksctl create addon --cluster <cluster> --region <region> \
  --name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::<account-id>:role/AmazonEKS_EBS_CSI_DriverRole --force

gp3-default-storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"

Oracle OKE: confirm oci-bv is still default

New clusters ship the block-volume CSI add-on with oci-bv already marked default. Confirm:

kubectl get storageclass
# oci-bv (default)   blockvolume.csi.oraclecloud.com   Delete   WaitForFirstConsumer   true

Some cluster templates and Terraform modules strip the (default) marker. Restore it:

kubectl patch storageclass oci-bv \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

OCI block volumes have a 50 GB minimum, so a smaller postgres.storage rounds up and bills at 50 GB regardless. Ampere (A1.Flex) node pools work as-is, because every chart image is a multi-arch manifest. No nodeSelector is needed.

Exposing services

Pick one way to expose the proxy (console) and gateway, to your org network or VPN only, never 0.0.0.0/0. Enable either ingress or httpRoute, not both.

Method	How
Ingress (recommended)	Set ingress.enabled: true, fill in ingress.className and cert-manager annotations. Resolves at your --host endpoint: / console, /v1 gateway.
Gateway API	Set httpRoute.enabled: true and httpRoute.parentRefs. Generates an HTTPRoute with the same routing as Ingress. Needs the Gateway API CRDs and chart 0.4.41+.
LoadBalancer	Set proxy.service.type and gateway.service.type to LoadBalancer, point your --host DNS at the load balancer address, and lock it down with loadBalancerSourceRanges or your firewall.
NodePort (dev only)	Set proxy.service.type: NodePort, proxy.service.nodePort: 30000, gateway.service.type: NodePort, gateway.service.nodePort: 30787. Verify by node IP: curl -fs http://<node-ip>:30787/.

Keep LoadBalancer and NodePort service settings in my-values.yaml too. Do not edit the chart's helm/values.yaml.

my-values.yaml: Ingress with TLS via cert-manager

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  tls:
    - secretName: anyray-tls
      hosts:
        - "<gateway-ingress-hostname>"

gateway:
  publicUrl: "https://<gateway-ingress-hostname>"
  consolePublicUrl: "https://<gateway-ingress-hostname>"

The chart generates only the HTTPRoute, meaning your app's routing, not the Gateway itself. That is platform infra, like the IngressClass:

my-values.yaml

httpRoute:
  enabled: true
  parentRefs:
    - name: my-gateway
      namespace: gateway-system   # omit if the Gateway is in this namespace
      sectionName: https          # a specific listener on the Gateway (optional)
  # hostnames default to `host`

The chart pre-tunes the Ingress for streaming

/v1 streams responses for minutes and carries megabyte request bodies, which ingress-nginx's stock 60s read timeout and 1 MB body cap cut short. ingress.streamingDefaults is on by default and sets proxy-read-timeout and proxy-send-timeout to 3600, proxy-body-size to 32m, and proxy-buffering off. ingress.annotations overrides key by key, and httpRoute.streamingDefaults does the same on the Gateway API path.

A 60s idle or read timeout anywhere in front, such as a load balancer or mesh, shows up as API Error: Connection closed mid-response.

ingress-nginx 1.12 and later: use chart 0.7.39 or newer

From 1.12, ingress-nginx turns on strict-validate-path-type by default. Its admission webhook then rejects any path containing a . unless its pathType is ImplementationSpecific, and it rejects the whole Ingress, / and /v1 included. Charts before 0.7.39 route the two /.well-known/oauth-* paths as Prefix, so they fail that check. From 0.7.39 they are ImplementationSpecific and the Ingress is admitted.

Service name constraints

The gateway and optimizer Services use bare, unprefixed names. The proxy dials them by in-cluster FQDN (gateway.<namespace>.svc.<clusterDomain>), because nginx's resolver does not apply Kubernetes DNS search domains. Override clusterDomain (default cluster.local) only for a non-default DNS domain, and install the chart in its own namespace to avoid name collisions.

Cluster policy values

Service account, private registry, scheduling

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/anyray

image:
  pullSecrets:
    - name: ghcr-pull-secret

nodeSelector:
  workload: ai-platform
tolerations:
  - key: dedicated
    operator: Equal
    value: ai-platform
    effect: NoSchedule
podSecurityContext:
  runAsNonRoot: true
containerSecurityContext:
  allowPrivilegeEscalation: false

nodeSelector, affinity, tolerations, topologySpreadConstraints, and priorityClassName apply to every pod. They can also be set per component on the gateway, optimizer, proxy, or postgres blocks. A component-level value replaces the global for that field with no merge, and unset inherits.

Each image can be mirrored. Leave the app tag empty to inherit image.tag:

images:
  gateway:
    repository: registry.example.com/anyray/gateway
    tag: ""            # inherits image.tag (the chart's appVersion by default)
  optimizer:
    repository: registry.example.com/anyray/optimizer
    tag: ""

Keep secrets in Kubernetes Secrets and reference them from values. Never put provider keys or database passwords in Git.

Gateway hardening

Set these when exposing Anyray behind an Ingress, reverse proxy, or org-wide gateway policy:

gateway:
  hsts: "true"                       # HSTS headers; set only when serving HTTPS
  trustProxy: "true"
  rateLimitRpm: "600"                # per-identity /v1 requests per minute
  rateLimitIpRpm: "1200"             # per-source-IP /v1 requests per minute
  rateLimitUnauthRpm: "60"           # unauthenticated requests per minute
  maxConcurrentRequests: "20"        # simultaneous /v1 requests per identity or IP
  maxBodyBytes: "33554432"           # max request body in bytes
  optimizerTimeoutMs: "1200"         # normal optimizer timeout (the default)
  optimizerVisionTimeoutMs: "10000"  # vision optimizer timeout (the default)

gateway.allowPlaintext is the deploy gate for plaintext content capture. The content mode itself is set in the console, not in values. gateway.contentMode now feeds only the optimizer's gateway-less paths (BYO /v1/record, attach mode). For uncommon environment variables, use gateway.extraEnv, optimizer.extraEnv, or proxy.extraEnv. A corporate forward proxy goes in gateway.extraEnv: Behind a forward proxy.

Automatic updates

Value	Default	What it does
image.tag	"" (the chart's appVersion)	The tag every app image resolves to. policy-stable is the moving channel and the whole auto-update opt-in (chart 0.6.1+); a vX.Y.Z pins one specific build.
image.pullPolicy	IfNotPresent	Forced to Always whenever the effective tag is a moving channel. A pinned tag is immutable and needs no re-pull.
autoUpdate.enabled	true	Arms the roll, but renders nothing unless image.tag can actually move. Each run pins every Deployment to one digest, so a pod that restarts between rolls stays on its siblings' build. Set false to follow a moving channel without a scheduled roll.
autoUpdate.schedule	"30 3 * * *"	Standard cron. Daily, outside working hours.
autoUpdate.timeZone	""	IANA name (Europe/Berlin). Empty uses the cluster's zone. Kubernetes 1.27+.
autoUpdate.image.repository / .tag	registry.k8s.io/kubectl / v1.33.0	Any kubectl within one minor of your cluster. Mirrored by global.imageRegistry like every other image.
images.<component>.tag	"" (inherits image.tag)	Per-component override for gateway, optimizer, proxy. Most specific wins.
global.imageRegistry	""	Swaps the registry host on every image at once, for a private mirror. The path after the host is preserved, so a crane cp mirror resolves.

Render the image refs without installing

helm template anyray anyray/anyray \
  --version <chart-version> -f my-values.yaml \
  | grep -E 'image:|imagePullPolicy:'

What the update CronJob is allowed to touch

It runs under a namespaced Role, never a ClusterRole:

rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get"]

patch pins each Deployment's image to the digest resolved that night, then stamps the restart. The pod read is the Job reading its own resolved digests.

Deployments only, so the bundled Postgres StatefulSet is never restarted, and only this release's app.kubernetes.io/instance label, so a second release in the namespace is untouched.

Let the gateway drive the roll instead (the applier endpoint)

The CronJob is the cluster deciding when to roll. Alternatively the gateway decides, as on the Docker and AWS installs: it applies only releases that need nothing from you, on a ~15 minute cycle, and holds hard releases for you (Configure → Updates). Point ANYRAY_UPDATER_URL at an in-cluster endpoint that can pull images and restart workloads:

my-values.yaml

autoUpdate:
  enabled: false        # the gateway drives the roll instead of the schedule
gateway:
  extraEnv:
    - name: ANYRAY_UPDATER_URL
      value: http://my-applier.anyray-system.svc.cluster.local/v1/update

The endpoint accepts POST with an Authorization: Bearer header carrying ANYRAY_UPDATER_TOKEN (which falls back to ANYRAY_ADMIN_TOKEN) and a JSON body naming the version the gateway settled on: {"target": "vX.Y.Z"}. Applying it is the endpoint's job. The rollout restart under Upgrade is enough while the tag is policy-stable.

The URL is validated before any token is sent: https anywhere, and plain http only to a host that cannot route off the cluster (a bare Service name, a .svc.cluster.local or other internal suffix, a private or loopback address). Anything else is ignored with a warning, so a typo cannot put the admin token on the wire in cleartext. Until one is set, GET /admin/update-status reports updateMethod: "kubernetes" and hasApplier: false, and the console shows a helm upgrade command rather than a one-click button.

Postgres volume size and the install-time floor

Three values control the bundled Postgres volume. Only storage is normally set:

Value	Default	What it does
postgres.storage	10Gi	The volume size. Lands in a StatefulSet volumeClaimTemplate, immutable after the first install.
postgres.minStorageGi	150	Floor enforced on a fresh install. Below it the chart refuses to render rather than provisioning a volume that fills before the 90-day retention window closes.
postgres.acknowledgeSmallVolume	false	Opt out of the floor for a deliberately small volume.

The default is deliberately below the floor, so a fresh install that never sets storage is refused instead of silently sizing itself too small. Growing later means expanding the PVC by hand, and only on a StorageClass with allowVolumeExpansion.

Run a smaller volume only when you have shortened the trace-retention window on the console Privacy page, or turned content off with ANYRAY_CONTENT_MODE=off. The trace tables are what grow, and spend rows are a fraction of the size.

postgres:
  storage: 10Gi
  acknowledgeSmallVolume: true

helm upgrade never triggers the floor. ArgoCD and Flux render with helm template, which reports an install, so a deployment already running below the floor needs acknowledgeSmallVolume: true to keep syncing. Its live volume is untouched either way. Measurement queries: Measure datastore growth.

Managed Postgres

The values block is in the production checklist, and the chart reads the URL as ANYRAY_OBSERVABILITY_DB_URL. The storage floor does not apply: with postgres.enabled=false the chart provisions no volume. An inline databaseUrl: postgresql://user:password@db.example.com:5432/postgres works in place of the Secret reference, less safely.

Most managed Postgres (RDS, Cloud SQL, Azure Database) rejects unencrypted connections, and the gateway takes SSL settings only from the connection string. Add ?sslmode=require, or ?sslmode=no-verify to skip cert verification without the provider's CA bundle. URL-encode reserved characters (@ : / ? # %) in the password, so a password p@ss?1 becomes:

postgresql://user:p%40ss%3F1@db.example.com:5432/postgres?sslmode=require

On RDS, ?sslmode=verify-full works with no CA file, because the images ship Amazon's RDS root certificates. Another provider's CA goes in the URL as sslrootcert=/path/to/ca.pem. RDS with IAM authentication needs neither setting.

Use Postgres 16 or 17. The database user needs no superuser role and no extensions; owning its database is enough. The gateway migrates the schema into public on boot and auto-creates the anyray_traces and anyray_observations tables on first use, and trace content follows the deployment's content mode: AES-256-GCM ciphertext by default, omitted in off, and readable only in deploy-gated plaintext.

RDS IAM database authentication

On Amazon RDS or Aurora you can drop the stored password entirely. The gateway mints a short-lived token from the pod's own AWS identity, so the credential rotates on every connection.

Turn it on by leaving the password out of the URL. An RDS endpoint with a user and no password anywhere is read as IAM auth, and there is no separate flag. The gateway looks everywhere Postgres itself looks, so a password still reachable through PGPASSWORD, a ?password= parameter, or a matching line in the password file (PGPASSFILE, otherwise ~/.pgpass) keeps password auth. An existing deployment cannot flip modes by accident.

postgres:
  enabled: false
  external:
    databaseUrlSecretKeyRef:
      name: anyray-external-postgres
      key: DATABASE_URL
    # DATABASE_URL, with a user and no password:
    #   postgresql://appuser@mydb.abc123.us-east-1.rds.amazonaws.com:5432/anyray

Three things have to be true on the AWS side:

1

IAM auth is enabled on the instance or cluster

--enable-iam-database-authentication.

2

The database user can use it

As the master user: CREATE USER appuser; GRANT rds_iam TO appuser;, plus the schema privileges the gateway needs (CREATE on the database, because it creates and migrates its own tables).

3

The pods' role is allowed to connect

Attach a policy granting rds-db:connect on arn:aws:rds-db:<region>:<account>:dbuser:<db-resource-id>/appuser and annotate the release's ServiceAccount (serviceAccount.annotations) with the role. EKS IRSA, a node instance profile, or environment credentials all work. One ServiceAccount covers every Anyray pod, which is what you want: the gateway and the optimizer each open their own connections.

The region comes from the endpoint hostname. If yours does not carry one, set AWS_REGION on the gateway and the optimizer.

TLS is configured for you. The gateway ships Amazon's RDS root certificates and verifies against them, so you need no sslmode and no mounted CA bundle. sslrootcert= overrides which CA is trusted, and sslmode=disable is rejected because RDS IAM requires an encrypted connection. On the IAM path, sslmode=no-verify and the uselibpqcompat modes do not disable certificate verification the way they do for password auth, so drop sslmode=no-verify when moving an existing external URL over.

Confirm it took effect

GET /admin/health reports the mode it resolved.

curl -fsS "https://<your-gateway>/admin/health" \
  -H "Authorization: Bearer $ANYRAY_ADMIN_TOKEN" | jq '.spend.auth'

"rds-iam" means tokens are being minted. "password" means the URL still carries one. "invalid" means the URL asks for IAM but something blocks it (troubleshooting).

The same rule applies to the optimizer and to ANYRAY_OBSERVABILITY_DB_URL when the trace store lives on its own RDS instance. Every service ships the same Amazon roots and mints its own token.

The optimizer needs the grant too, and fails quietly without it

The services depend on the same role but connect independently, and only the gateway refuses to start when its database is unreachable. The optimizer degrades silently: it keeps serving traffic while its spend writes and decision pins go nowhere, which under-reports savings and can misprice cache-heavy rows. After enabling IAM, confirm the optimizer leg on /admin/health still reports configStore: shared, not per-pod or refused.

Built-in availability

Behavior	Detail
Liveness and startup probes (from v1.10.224)	Liveness recovers a wedged process that a readiness probe would only de-list. The startup probe allows 30 minutes: a gateway applying the migration ledger opens no port until it finishes.
Readiness follows the entitlement lease	A metered gateway serves /v1/* only while it holds a signed lease; GET / answers 503 (AI Gateway awaiting entitlement lease) until then. Replicas share the lease through the spend database, so a new pod adopts its peer's lease and turns Ready in seconds. Diagnose via the portal leg of /admin/health (troubleshooting).
Drain-aware shutdown	preStopDrainSeconds keeps a pod serving while its de-listing propagates, then in-flight requests finish, up to ANYRAY_SHUTDOWN_DRAIN_MS: 90s gateway, 15s optimizer. Both are ceilings, not delays. The chart refuses to render if terminationGracePeriodSeconds no longer covers pause plus drain.
Soft pod anti-affinity	Replicas prefer separate nodes, then separate zones.
maxUnavailable: 0 rollouts	The Service endpoint list never empties mid-rollout, even at one replica.
Two replicas of every service (from chart 0.8.0)	The optimizer drops to one with persistence.enabled: true or an image below v1.10.403. Autoscaling is available per service (<component>.autoscaling.enabled) but off: scaling in drops in-flight spend writes and busts the prompt cache on warm sessions. The bundled Postgres allows 300 connections (postgres.maxConnections); raise it before adding pods.
PodDisruptionBudgets	Caps what a node drain or autoscaler scale-down may take at once, for any workload running two or more pods.

Existing installs: two-step migrations

Optimizer, moving to persistence.enabled: false

The optimizer copies its runtime config from the volume into the database on boot, only while the volume is mounted. Upgrade once with persistence.enabled: true still set, confirm the console still shows your optimizer settings, then remove the override and upgrade again. The old PVC is retained.

The pre-upgrade settings audit file is not migrated. New events land in anyray_audit_log from the first boot on the new image, but optimizer-settings-audit.log stays on the retained PVC. Copy it off before unmounting if you need that history:

kubectl -n "$ANYRAY_NAMESPACE" exec deploy/anyray-optimizer -- \
  cat /data/optimizer-settings-audit.log > optimizer-settings-audit.log

System requirements

The default Deployments and StatefulSet request 2.8 vCPU and 5.5 GiB in total, including bundled Postgres. The optimizer is the largest single pod: it requests 1 vCPU and 2 GiB and may burst to 2 vCPU and 4 GiB, because it runs onnxruntime re-rank and OCR on the Node event loop and holds every in-flight request body. Use the three-node production baseline for the default replicas and rolling upgrades. For long agent sessions, use larger nodes so the optimizer can burst. Size Postgres for high trace or spend volume.

Managed Kubernetes	Node type
AWS EKS	t3.medium (2 / 4)
GCP GKE	e2-medium (2 / 4)
Azure AKS	Standard_B2ms (2 / 4)
Oracle OKE	VM.Standard.E4.Flex 1 OCPU / 4 GB (1 OCPU = 2 vCPU), or Ampere VM.Standard.A1.Flex 2 OCPU / 4 GB

setup.sh --k8s and the GKE and AKS scripts write postgres.storage: 150Gi, which holds the gateway's default 90-day trace retention window for a typical team. The size is immutable after the first install (the floor). Measure your own growth rate in the first week: Measure datastore growth.

Troubleshooting

The bundled chart (0.4.9 and later) handles all of these out of the box. The notes are for forks and custom value overrides.

PVCs stuck in Pending

kubectl get pvc -n "$ANYRAY_NAMESPACE"
# data-anyray-postgres-0   Pending   ...   (no STORAGECLASS)

The chart's volumes bind the cluster's default StorageClass, and a fresh EKS cluster often has neither the EBS CSI driver nor a default class. Install the driver and mark a default class (EKS setup). The PVCs bind on the next reconcile, with no reinstall needed.

Postgres will not start on lost+found

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.

An ext4 block-store volume always has a lost+found directory at its root, and Postgres's initdb refuses any non-empty data directory. The chart initializes into a subdirectory of the mount (PGDATA=/var/lib/postgresql/data/pgdata). If you mount your own Postgres volume, point PGDATA or a subPath at a subdirectory, never the mount root.

Reset a stuck Postgres volume

If the bundled Postgres failed to initialize on a first install, its PVC may be half-provisioned. After upgrading the chart, reset just that volume. This is safe only when Postgres never finished initializing, so there is no real data to lose.

Clean-reset the bundled Postgres

# Inspect first
kubectl get pvc -n "$ANYRAY_NAMESPACE"

# Stop Postgres so it releases its volume (gentler than deleting the StatefulSet)
kubectl scale statefulset -n "$ANYRAY_NAMESPACE" anyray-postgres --replicas=0

# Delete the empty PVC (StatefulSet volumeClaimTemplate claim)
kubectl delete pvc -n "$ANYRAY_NAMESPACE" data-anyray-postgres-0

# Re-apply: scales the StatefulSet back to 1 and provisions a fresh PVC
helm upgrade anyray ./helm -f my-values.yaml --namespace "$ANYRAY_NAMESPACE"

Never delete the gateway or optimizer PVCs. anyray-gateway-data and anyray-optimizer-data hold your routing config, provider keys, and spend and audit logs. They carry a helm.sh/resource-policy: keep annotation for exactly this reason. Only reset the Postgres PVC above.

Mixed-architecture clusters (arm64 + amd64)

Every image the chart ships is a multi-arch manifest list (linux/amd64 and linux/arm64), so Kubernetes schedules each pod onto any node and pulls the matching architecture. No architecture nodeSelector is needed.

If you mirror images into a private registry, mirror both architectures with docker buildx imagetools create, not a single-arch docker pull and push, or pods fail to schedule on the architecture you dropped. To pin a workload to a node pool, set nodeSelector, affinity, or tolerations, globally or per component (cluster policy).