Deploy on AWS

Launch the whole stack on ECS/Fargate with one CloudFormation template.

You pick a VPC, two public subnets, and the CIDR allowed in; CloudFormation runs the gateway, optimizer, and console behind an Application Load Balancer, with managed RDS PostgreSQL and every secret generated into Secrets Manager.

Prerequisites

• AWS account with permissions to create ECS, Fargate, RDS, EFS, Elastic Load Balancing, Secrets Manager, IAM, Lambda, and CloudFormation resources.
• VPC with two public subnets in two different Availability Zones that auto-assign public IPv4 (the ALB and RDS each require two AZs; Fargate tasks pull images over their public IP, so no NAT gateway is needed).
• A narrow AllowedCidr — your office or VPN range. For a single workstation: curl -fsS https://checkip.amazonaws.com, then <that-ip>/32. Comma-separate to allow several ranges (e.g. 203.0.113.0/24,198.51.100.0/24).
• (Optional) an ACM certificate in the same region for HTTPS listeners; without it the listeners serve HTTP, scoped to your CIDR.
• An Anyray deployment token (adt_…) from app.anyray.ai (setup wizard, or Settings → Deployments → New deployment) — required for usage metering.

Find the AWS values

Not sure which VPC or subnets to use? List them first:

export AWS_REGION=eu-central-1

aws sts get-caller-identity

aws ec2 describe-vpcs \
  --region "$AWS_REGION" \
  --filters Name=is-default,Values=true \
  --query 'Vpcs[].{VpcId:VpcId,CidrBlock:CidrBlock,State:State}' \
  --output table

aws ec2 describe-subnets \
  --region "$AWS_REGION" \
  --filters Name=default-for-az,Values=true \
  --query 'Subnets[].{SubnetId:SubnetId,Az:AvailabilityZone,AutoPublicIp:MapPublicIpOnLaunch,AvailableIps:AvailableIpAddressCount}' \
  --output table

Pick two subnets where AutoPublicIp is True and the Az values differ. No public subnets? Create them, or use Remote docker on an instance you manage.

Install

1

Deploy on AWS

Click to open the CloudFormation quick-create console with the Anyray template pre-filled.

Deploy on AWS

Deploying outside eu-central-1?

The button opens in eu-central-1 (Frankfurt) — switch region top-right before creating the stack. Your VPC and subnets must be in that region.

2

What you're deploying

An ECS/Fargate cluster behind an Application Load Balancer:

Service	Where	What it is
gateway	Fargate, ALB :8787	OpenAI-compatible multi-provider API
proxy (console)	Fargate, ALB :3000	Admin console (Spend, Traces, Optimizer, Privacy)
optimizer	Fargate, internal	Request/response optimization hook
RDS PostgreSQL	managed, internal	Spend + trace store — spend rows are metadata only; trace content per ANYRAY_CONTENT_MODE (encrypted default)

The gateway and optimizer mount EFS at /data, so runtime config (routing, optimizer edits, the entitlement lease) survives task restarts; services find each other via Cloud Map service discovery (gateway-dns.anyray.internal:8787, optimizer-dns.anyray.internal:8088). Every secret (ANYRAY_ADMIN_TOKEN, ANYRAY_CONTENT_KEY, ANYRAY_OPTIMIZER_TOKEN, the pseudonym salt, the composed database URL) is generated into Secrets Manager and injected at runtime — never printed in the template or outputs.

3

Set your stack parameters

Parameter	What to enter
VpcId	The VPC to launch into.
SubnetA / SubnetB	Two public subnets in different AZs that auto-assign public IPv4.
AllowedCidr	CIDR range(s) allowed to reach :3000 and :8787 — your office / VPN range. Comma-separate to allow several (e.g. 203.0.113.0/24,198.51.100.0/24).
ImageTag	latest follows the release channel; set vX.Y.Z to pin.
DefaultModel	What the anyray-default routing sentinel resolves to.
DeploymentToken	Your adt_… token from app.anyray.ai — required for usage metering.
CertificateArn	(Optional) ACM cert ARN for HTTPS on both listeners.
GatewayPublicUrl / ConsolePublicUrl	(Optional) externally reachable URLs; empty = ALB DNS.
DbInstanceClass / DbAllocatedStorage	RDS sizing; defaults (db.t4g.micro, 20 GiB) suit a small team.

Hardening and optimizer-tuning parameters: see Configuration.

4

Create the stack

Set VpcId, SubnetA, SubnetB, AllowedCidr, and DeploymentToken, then choose Create stack.

5

Wait for first boot

RDS is the long pole — allow ~5–10 min. The gateway restarts until the database is reachable, then goes healthy.

6

Get your URLs and admin token

On the stack's Outputs tab: ConsoleURL, GatewayURL, and AdminTokenCmd — a CLI one-liner that prints the generated ANYRAY_ADMIN_TOKEN from Secrets Manager.

7

Open the console

Open ConsoleURL and sign in with the admin token, then follow After deployment below.

8

Verify the deployment

export GATEWAY_URL="http://your-alb-dns:8787"
export ADMIN_TOKEN="..."   # from the AdminTokenCmd output

curl -fsS "$GATEWAY_URL/" && echo "gateway ok"
# Deployment health — gateway / observability / spend / optimizer / portal:
curl -fsS "$GATEWAY_URL/admin/health" -H "Authorization: Bearer ${ADMIN_TOKEN}"

/admin/health returns 503 and names the failing leg if any required service is down; a healthy stack returns "ok": true with every leg green.

What runs where

Everything above runs in your AWS account. To point your local coding tools (Claude Code, Cursor, Codex) at the gateway, run npx anyray-connect --gateway <GatewayURL> on your laptop. See the developer FAQ.

Install with AWS CLI

1

Validate the template

aws cloudformation validate-template \
  --region "$AWS_REGION" \
  --template-url https://anyray-quicklaunch.s3.us-east-1.amazonaws.com/anyray-quicklaunch.template.yaml

2

Create the stack

The template creates IAM roles and uses the AWS::LanguageExtensions transform, so pass both CAPABILITY_IAM and CAPABILITY_AUTO_EXPAND:

export VPC_ID="vpc-..."
export SUBNET_A="subnet-..."          # AZ 1
export SUBNET_B="subnet-..."          # AZ 2 (different AZ)
export ALLOWED_CIDR="$(curl -fsS https://checkip.amazonaws.com)/32"
export DEPLOYMENT_TOKEN="adt_..."

aws cloudformation create-stack \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --template-url https://anyray-quicklaunch.s3.us-east-1.amazonaws.com/anyray-quicklaunch.template.yaml \
  --parameters \
    ParameterKey=VpcId,ParameterValue="$VPC_ID" \
    ParameterKey=SubnetA,ParameterValue="$SUBNET_A" \
    ParameterKey=SubnetB,ParameterValue="$SUBNET_B" \
    ParameterKey=AllowedCidr,ParameterValue="$ALLOWED_CIDR" \
    ParameterKey=DeploymentToken,ParameterValue="$DEPLOYMENT_TOKEN"

aws cloudformation wait stack-create-complete \
  --region "$AWS_REGION" \
  --stack-name anyray

aws cloudformation describe-stacks \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --query 'Stacks[0].Outputs' \
  --output table

3

Read the outputs and fetch the admin token

export GATEWAY_URL="$(
  aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
    --query "Stacks[0].Outputs[?OutputKey=='GatewayURL'].OutputValue | [0]" --output text
)"
export CONSOLE_URL="$(
  aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
    --query "Stacks[0].Outputs[?OutputKey=='ConsoleURL'].OutputValue | [0]" --output text
)"

# AdminTokenCmd output is the exact CLI call that prints the admin token:
aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
  --query "Stacks[0].Outputs[?OutputKey=='AdminTokenCmd'].OutputValue | [0]" --output text

curl -fsS "$GATEWAY_URL/" && echo "gateway ok"
printf 'Console: %s\n' "$CONSOLE_URL"

Do not use 0.0.0.0/0

The console and gateway carry your org's spend data and admin access. The template rejects 0.0.0.0/0 — still choose the narrowest CIDR that works for your team.

After deployment — roll out to your users

Users reach the gateway at the GatewayURL stack output.

1

Add a provider key

Provider keys stay server-side — callers never hold them. Add at least one from the console Providers page (no stack update needed); multi-field providers (Bedrock, Vertex, Azure) are configured there too. See Configure → Providers.

2

Send a test request

On the Providers page use Send test request, then confirm a row under Spend and a trace under Traces — that exercises routing, attribution, and the trace store end to end.

3

Enroll your users

/v1 is always gated — each user enrolls before their traffic is accepted. Mint an enrollment link (enl_…) on the console Users page; each user runs it on their own machine:

npx anyray-connect --enroll https://…/enroll/enl_…

That points their coding tools (Claude Code, Cursor, Codex, …) at the gateway (GatewayURL) and mints a personal key bound to their email — no provider keys ever touch a user's machine. For a whole fleet, push one provisioning token via your MDM — see Bulk enrollment with your MDM.

4

Track the migration and confirm the cutover

Watch the Users roster move from Pending to Enrolled; unenrolled /v1 traffic is always rejected (401). See Configure → Access control.

You're live

Your users' everyday AI tools now run through Anyray — spend attributed by user/team/model, content handled per your privacy mode, the optimizer cutting cost on the hot path.

Configuration

Gateway hardening and optimizer tuning are stack parameters — change one later with aws cloudformation update-stack … (or Update in the console) and ECS rolls the affected service:

Parameter	What it controls
Hsts	Emits HSTS headers (set true only when serving HTTPS via CertificateArn).
ContentMode / AllowPlaintext	Content-privacy mode (encrypted by default) and the deploy gate for plaintext.
RateLimitRpm	Per-identity /v1 request limit per minute.
RateLimitIpRpm	Per-source-IP /v1 request limit per minute.
RateLimitUnauthRpm	Unauthenticated endpoint request limit per minute.
MaxConcurrentRequests	Max simultaneous /v1 requests per identity or IP.
MaxBodyBytes	Max request body size in bytes.
OptimizerTimeoutMs	Normal optimizer timeout; defaults to 800.
OptimizerVisionTimeoutMs	Vision optimizer timeout; defaults to 10000.

ANYRAY_TRUST_PROXY is always on — the gateway sits behind the load balancer. Other controls match Local / VM.

Troubleshoot

CloudFormation fails or rolls back

aws cloudformation describe-stack-events \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --query 'StackEvents[0:15].[Timestamp,ResourceStatus,LogicalResourceId,ResourceStatusReason]' \
  --output table

The stack completes but the console does not open

CloudFormation finishes once resources exist, but the Fargate services need RDS reachable before they go healthy:

aws ecs describe-services \
  --region "$AWS_REGION" \
  --cluster anyray-anyray \
  --services anyray-gateway anyray-proxy \
  --query 'services[].{name:serviceName,running:runningCount,desired:desiredCount,events:events[0].message}'

A service stuck at running: 0 is usually crash-looping — read its CloudWatch logs under /anyray/<stack-name>/gateway (or /optimizer, /proxy).

The gateway or console target is unhealthy

The ALB health-checks the gateway on / and the console on /anyray-login:

aws elbv2 describe-target-health \
  --region "$AWS_REGION" \
  --target-group-arn "$(aws elbv2 describe-target-groups --region "$AWS_REGION" \
    --query "TargetGroups[?contains(TargetGroupName, 'anyray')].TargetGroupArn" --output text | head -1)"

A gateway that keeps failing is most often still waiting on RDS — check its CloudWatch log group for the database connection.

The console shows a bare nginx 500

Stacks created before July 2026 wired services with ECS Service Connect, whose names the console proxy cannot resolve — every console request returned 500 Internal Server Error (nginx). Update the stack with the current template to pick up the fix (settings, database, and secrets are untouched; the ECS services are replaced):

aws cloudformation update-stack \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --template-url https://anyray-quicklaunch.s3.us-east-1.amazonaws.com/anyray-quicklaunch.template.yaml \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --parameters "$(aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
      --query 'Stacks[0].Parameters[].{ParameterKey:ParameterKey,UsePreviousValue:`true`}' --output json)"

Upgrade

ImageTag=latest follows the moving release channel. To pin or roll back, update the stack with a specific tag; ECS rolls each service:

aws cloudformation update-stack \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --use-previous-template \
  --parameters \
    ParameterKey=ImageTag,ParameterValue=vX.Y.Z \
    ParameterKey=VpcId,UsePreviousValue=true \
    ParameterKey=SubnetA,UsePreviousValue=true \
    ParameterKey=SubnetB,UsePreviousValue=true \
    ParameterKey=AllowedCidr,UsePreviousValue=true \
    ParameterKey=DeploymentToken,UsePreviousValue=true

A pinned :vX.Y.Z keeps rollbacks reproducible and auditable; return to latest to follow the moving channel again.