Deploy on a Compute Engine VM

Run the whole stack with docker compose on one Compute Engine VM.

The deploy script creates the VM with a persistent data disk, generates every secret on the box, and brings up the gateway, optimizer, console, and Postgres, with :3000 and :8787 firewall-scoped to the CIDR you choose.

Why a VM and not Cloud Run

The gateway keeps provider keys and user enrollments as files on a persistent /data volume and runs as a single instance — it wants durable local disk, not an ephemeral serverless filesystem.

Prerequisites

• A Google Cloud project with billing enabled and permission to create Compute Engine instances, disks, and firewall rules.
• The Compute Engine and IAP APIs — the deploy script enables them for you. IAP backs the SSH used to read your admin key; no SSH port is exposed to the internet.
• A narrow ALLOWED_CIDR — your office or VPN range. For a single workstation: curl -fsS https://api.ipify.org, then <that-ip>/32.
• An Anyray deployment token (adt_…) from app.anyray.ai (setup wizard, or Settings → Deployments → New deployment) — required for usage metering.

Cloud Shell already has gcloud — nothing to install locally.

Install

1

Open in Cloud Shell

Click to clone the install repo into Cloud Shell and open the guided walkthrough.

Open in Cloud Shell

Prefer your own terminal?

Any machine with the gcloud SDK authenticated to your project works — see Install from your own terminal below.

2

What you're deploying

One VM running the full stack with docker compose, fronted by two firewall rules:

Service	Where	What it is
gateway	host :8787 (CIDR-scoped)	OpenAI-compatible multi-provider API
proxy (console)	host :3000 (CIDR-scoped)	Admin console (Spend, Traces, Optimizer, Privacy)
optimizer	in-VM only	Request/response optimization hook
Postgres	in-VM only	Spend + trace store — spend rows are metadata only; trace content per ANYRAY_CONTENT_MODE (encrypted default)

All container state — the /data volumes and Postgres — lives on a dedicated persistent disk (Docker's data-root) that is not deleted with the VM, so provider keys and enrollments survive a VM rebuild. Every secret (ANYRAY_ADMIN_TOKEN, ANYRAY_CONTENT_KEY, the Postgres password, the pseudonym salt) is generated on the VM by setup.sh and never leaves it. SSH is reachable only via Google IAP.

3

Set your inputs

Export your two required values, then run the deploy script:

Cloud Shell

export ALLOWED_CIDR="203.0.113.0/24"     # your office / VPN range
export DEPLOYMENT_TOKEN="adt_..."        # from app.anyray.ai
# Optional overrides:
# export ZONE="us-central1-a"
# export INSTANCE="anyray"
# export MACHINE_TYPE="e2-standard-2"    # 2 vCPU / 8 GB
# export DISK_SIZE="20GB"
# export IMAGE_TAG="latest"              # or pin vX.Y.Z

./gcp/gce/deploy.sh

The script enables the APIs and creates the firewall rules and the VM; the VM's first-boot script installs Docker, generates the secrets, connects to Anyray Billing app, and starts the stack.

4

Wait for first boot

First boot installs Docker and pulls the images — allow ~3–6 min. deploy.sh polls over IAP SSH and prints your URLs and admin key when the stack is up.

5

Get your URLs and admin key

deploy.sh prints the Console URL, Gateway URL, and admin key. Read the admin key again any time over IAP SSH:

gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap \
  --command "sudo sed -n 's/^ANYRAY_ADMIN_TOKEN=//p' /opt/anyray/.env | head -n1"

6

Open the console

From a machine inside your ALLOWED_CIDR, open the Console URL and sign in with that admin key, then follow After deployment below.

7

Verify the deployment

Run the bundled per-leg check on the VM:

gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap \
  --command "cd /opt/anyray && sudo ./scripts/verify-deploy.sh"

It polls liveness, queries the admin-gated /admin/health, and prints a per-leg verdict (gateway / observability / spend / optimizer / portal) with a remedy for any failing leg.

What runs where

Everything above runs in your Google Cloud project. To point your local coding tools (Claude Code, Cursor, Codex) at the gateway, run npx anyray-connect --gateway <GatewayURL> on your laptop. See the developer FAQ.

Install from your own terminal

From any machine with the gcloud SDK authenticated to your project:

git clone https://github.com/anyrayHQ/install anyray && cd anyray

export PROJECT_ID="my-gcp-project"
export ALLOWED_CIDR="203.0.113.0/24"     # your office / VPN range
export DEPLOYMENT_TOKEN="adt_..."        # from app.anyray.ai

./gcp/gce/deploy.sh

The script is idempotent: re-running it reuses an existing VM (it never recreates the instance, so data and secrets are preserved) and leaves existing firewall rules in place. The inputs:

Variable	Required	Default	What it is
PROJECT_ID	yes	gcloud's active project	GCP project to deploy into.
ALLOWED_CIDR	yes	—	CIDR allowed to reach the console/gateway; 0.0.0.0/0 is rejected.
DEPLOYMENT_TOKEN	yes	—	Anyray Billing app deployment token (adt_…).
ZONE	no	us-central1-a	VM zone.
INSTANCE	no	anyray	VM name (reused if present).
MACHINE_TYPE	no	e2-standard-2	VM size (2 vCPU / 8 GB).
DISK_SIZE	no	20GB	Persistent data-disk size.
IMAGE_TAG	no	latest	Anyray image tag; pin vX.Y.Z for a reproducible deploy.
DEFAULT_MODEL	no	anthropic/claude-sonnet-4-5	Target for the anyray-default model alias.
NETWORK	no	default	VPC network for the VM + firewall rules.

Do not use 0.0.0.0/0

The console and gateway carry your org's spend data and admin access. deploy.sh rejects 0.0.0.0/0 — still choose the narrowest CIDR that works for your team.

After deployment — roll out to your users

Users reach the gateway at the Gateway URL from the deploy output.

1

Add a provider key

Provider keys stay server-side — callers never hold them. Add at least one from the console Providers page; multi-field providers (Bedrock, Vertex AI, Azure) are configured there too. See Configure → Providers.

2

Send a test request

On the Providers page use Send test request, then confirm a row under Spend and a trace under Traces — that exercises routing, attribution, and the trace store end to end.

3

Enroll your users

/v1 is always gated — each user enrolls before their traffic is accepted. Mint an enrollment link (enl_…) on the console Users page; each user runs it on their own machine:

npx anyray-connect --enroll https://…/enroll/enl_…

That points their coding tools (Claude Code, Cursor, Codex, …) at the gateway and mints a personal key bound to their email — no provider keys ever touch a user's machine. For a whole fleet, push one provisioning token via your MDM — see Bulk enrollment with your MDM.

4

Track the migration and confirm the cutover

Watch the Users roster move from Pending to Enrolled; unenrolled /v1 traffic is always rejected. See Configure → Access control.

You're live

Your users' everyday AI tools now run through Anyray — spend attributed by user/team/model, content handled per your privacy mode, the optimizer cutting cost on the hot path.

Configuration

The stack is configured through /opt/anyray/.env on the VM — SSH in, edit, re-apply:

gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap
cd /opt/anyray
sudo nano .env
sudo docker compose up -d        # recreate changed services

Common hardening and tuning knobs:

.env variable	What it controls
ANYRAY_HSTS	Emits HSTS headers (set only when serving over HTTPS).
ANYRAY_CONTENT_MODE / ANYRAY_ALLOW_PLAINTEXT	Content-privacy mode (encrypted by default) and the deploy gate for plaintext.
ANYRAY_RATE_LIMIT_RPM	Per-identity /v1 request limit per minute.
ANYRAY_RATE_LIMIT_IP_RPM	Per-source-IP /v1 request limit per minute.
ANYRAY_RATE_LIMIT_UNAUTH_RPM	Unauthenticated endpoint request limit per minute.
ANYRAY_MAX_CONCURRENT_REQUESTS	Max simultaneous /v1 requests per identity or IP.
ANYRAY_MAX_BODY_BYTES	Max request body size in bytes.
ANYRAY_OPTIMIZER_TIMEOUT_MS	Normal optimizer timeout; defaults to 800.
ANYRAY_OPTIMIZER_VISION_TIMEOUT_MS	Vision optimizer timeout; defaults to 10000.

To re-scope the console/gateway to a different CIDR later:

gcloud compute firewall-rules update anyray-allow-web --source-ranges <your-cidr>

A stable address

The VM's external IP is ephemeral — it can change if the VM is recreated. For a fixed address, reserve a static external IP, assign it to the VM, then re-run setup.sh --host <static-ip> on the box so the gateway reports the stable URL.

Troubleshoot

The deploy finished but printed no admin key

First boot was still running, or IAP SSH wasn't reachable yet:

gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap \
  --command "sudo tail -n 50 /var/log/anyray-startup.log"
gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap \
  --command "sudo sed -n 's/^ANYRAY_ADMIN_TOKEN=//p' /opt/anyray/.env | head -n1"

The console or gateway is unreachable from your network

The firewall only admits traffic from ALLOWED_CIDR. Confirm your current public IP is inside that range (curl -fsS https://api.ipify.org), and re-scope if needed:

gcloud compute firewall-rules update anyray-allow-web --source-ranges <your-cidr>

A container is unhealthy or restarting

gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap
cd /opt/anyray
sudo docker compose ps
sudo docker compose logs --tail 100 gateway

A gateway that keeps restarting is most often still waiting on Postgres. Re-run the per-leg check: sudo ./scripts/verify-deploy.sh.

Upgrade

IMAGE_TAG=latest follows the moving release channel; soft updates (image-only releases) apply automatically by default (turn off in the console under Settings → Updates), and the one-click Update now button covers the rest. To converge manually, or to pin a tag:

gcloud compute ssh anyray --zone us-central1-a --tunnel-through-iap
cd /opt/anyray
# (optional) pin a tag: set ANYRAY_IMAGE_TAG=vX.Y.Z in .env
sudo docker compose pull && sudo docker compose up -d

Outgrowing a single box?

High trace/spend volume, or want a managed control plane and rolling deploys? Move to Google Cloud (GKE) — same stack, managed Kubernetes.