Deploy on AWS

One CloudFormation template runs the stack on ECS/Fargate behind an ALB, with managed RDS and every secret generated into Secrets Manager.

You pick a VPC, two public subnets, and the CIDR allowed in. Prefer the terminal? The same template deploys via aws cloudformation create-stack.

Prerequisites

• AWS account with permissions to create ECS, Fargate, RDS, EFS, Elastic Load Balancing, Secrets Manager, IAM, Lambda, and CloudFormation resources.
• A VPC with two public subnets in different AZs that auto-assign public IPv4. You normally supply nothing: leave VpcId, SubnetA and SubnetB blank and the stack uses the account's default VPC, which qualifies. Choosing a specific VPC instead: find the values.
• VPC DNS resolution and DNS hostnames enabled (the tasks mount EFS by DNS name). Default VPCs have both on; a VPC created via CLI or Terraform has DNS hostnames off unless set, and the stack rolls back (troubleshoot).
• A narrow AllowedCidr: your office or VPN range. For one workstation: curl -fsS https://checkip.amazonaws.com, then <that-ip>/32. Comma-separate several ranges (e.g. 203.0.113.0/24,198.51.100.0/24).
• (Optional) an ACM certificate in the same region for HTTPS listeners; without it the listeners serve HTTP, scoped to your CIDR.
• An Anyray deployment token (adt_…) from app.anyray.ai (setup wizard, or Settings → Deployments → New deployment).

Install

1

Deploy on AWS

Opens the CloudFormation quick-create console with the Anyray template pre-filled.

Deploy on AWS

The button opens in eu-central-1 (Frankfurt); switch region top-right before creating the stack. Your VPC and subnets must be in that region.

2

What you're deploying

Service	Where	What it is
gateway	Fargate, ALB :8787	OpenAI-compatible multi-provider API
proxy (console)	Fargate, ALB :3000	Admin console (Spend, Traces, Optimizer, Privacy)
optimizer	Fargate, internal	Request/response optimization hook
RDS PostgreSQL	managed, internal	Spend + trace store

The gateway and optimizer mount EFS at /data; services find each other via Cloud Map (gateway-dns.anyray.internal:8787, optimizer-dns.anyray.internal:8088). Every secret (ANYRAY_ADMIN_TOKEN, ANYRAY_CONTENT_KEY, ANYRAY_OPTIMIZER_TOKEN, the pseudonym salt, the composed database URL) is generated into Secrets Manager and injected at runtime, never printed in the template or outputs.

3

Set your stack parameters

Parameter	What to enter
VpcId	Leave blank for the account's default VPC. Set it (with both subnets) only to choose a specific VPC.
SubnetA / SubnetB	Leave blank with VpcId. If set: two public subnets in different AZs that auto-assign public IPv4.
AllowedCidr	CIDR range(s) allowed to reach :3000 and :8787: your office / VPN range. Comma-separate several.
ImageTag	Immutable release tag (vX.Y.Z); the default is stamped to the latest release when the template is published. Mutable latest/stable channels are rejected.
DefaultModel	What the anyray-default routing sentinel resolves to.
DeploymentToken	Your adt_… token from app.anyray.ai.
EnableAutoUpdate	Default true: compatible ("soft") releases apply on their own; see Updates.
EnableRdsDeletionProtection	Default false. Set true in production to block an accidental database delete; the stack cannot be deleted while it is on.
HighAvailability	Default false: one task per service, single-AZ database, so any task or AZ loss is an outage until AWS replaces it. true runs two tasks per service across both AZs and a Multi-AZ database; roughly doubles compute and database cost. Safe to turn on later (rolling change).
LoadBalancerScheme	Default internet-facing. internal keeps Anyray inside your VPC (VPN / Direct Connect / peered access only); tasks keep their own egress to providers either way.
CertificateArn	(Optional) ACM cert ARN for HTTPS on both listeners.
GatewayPublicUrl / ConsolePublicUrl	(Optional) externally reachable URLs; empty = ALB DNS.
DbInstanceClass / DbAllocatedStorage	RDS sizing; defaults (db.t4g.micro, 50 GiB) suit a small team and hold the gateway's 90-day trace retention window.
DbMaxAllocatedStorage	Ceiling for RDS storage autoscaling, default 100 GiB; keeps a busy deployment off a storage-full stall.

Hardening and optimizer-tuning parameters: Configuration.

4

Create the stack

Fill in AllowedCidr and DeploymentToken, the only two required fields, then choose Create stack. Everything else has a working default.

5

Wait for first boot

RDS is the long pole, so allow ~5–10 min. The gateway restarts until the database is reachable, then goes healthy.

6

Get your URLs and admin token

On the stack's Outputs tab: ConsoleURL, GatewayURL, and AdminTokenCmd, a CLI one-liner that prints the generated ANYRAY_ADMIN_TOKEN from Secrets Manager.

7

Open the console

Open ConsoleURL and sign in with the admin token.

8

Verify the deployment

export GATEWAY_URL="http://your-alb-dns:8787"
export ADMIN_TOKEN="..."   # from the AdminTokenCmd output

curl -fsS "$GATEWAY_URL/" && echo "gateway ok"
# Deployment health: gateway / observability / spend / optimizer / portal:
curl -fsS "$GATEWAY_URL/admin/health" -H "Authorization: Bearer ${ADMIN_TOKEN}"

/admin/health returns 503 and names the failing leg if any required service is down; a healthy stack returns "ok": true with every leg green.

Do not use 0.0.0.0/0

The console and gateway carry your org's spend data and admin access. The template rejects 0.0.0.0/0; still choose the narrowest CIDR that works for your team.

Everything above runs in your AWS account. Point your local coding tools at the gateway with npx anyray-connect@latest --gateway <GatewayURL> (developer FAQ).

Configuration

Gateway hardening and optimizer tuning are stack parameters; change one later with aws cloudformation update-stack … (or Update in the console) and ECS rolls the affected service:

Parameter	What it controls
Hsts	Emits HSTS headers (set true only when serving HTTPS via CertificateArn).
ContentMode / AllowPlaintext	Content-privacy mode (encrypted by default) and the deploy gate for plaintext.
RateLimitRpm	Per-identity /v1 request limit per minute.
RateLimitIpRpm	Per-source-IP /v1 request limit per minute.
RateLimitUnauthRpm	Unauthenticated endpoint request limit per minute.
MaxConcurrentRequests	Max simultaneous /v1 requests per identity or IP.
MaxBodyBytes	Max request body size in bytes.
OptimizerTimeoutMs	Normal optimizer timeout; defaults to 800.
OptimizerVisionTimeoutMs	Vision optimizer timeout; defaults to 10000.

ANYRAY_TRUST_PROXY is always on, because the gateway sits behind the load balancer. Other controls match Local / VM.

RDS grows for its first three months before the first trace prune reclaims anything; measure your own rate in the first week with Measure datastore growth.

Updates

The stack keeps itself current by default: the gateway checks for new releases and applies the compatible ones on its own, an image roll across all three services together and nothing else.

Release class	What happens
Soft, an image swap only	Applied automatically. Schema migrations run on boot, so a schema change alone is still soft.
Hard, needing an infra change (a new required setting, or a release flagged breaking)	Never applied automatically. The console banner tells you what to do.

All three services always move to the same version. If one fails its health checks, the deployment circuit breaker rolls it back. A roll never interrupts requests: services deploy at MinimumHealthyPercent: 100 / MaximumPercent: 200, so replacements pass their health checks before any old task retires, and a deregistering gateway task gets 120s (covering its 90s stream drain) before SIGKILL. At HighAvailability=false a deploy still never drops the only task; an unplanned task or AZ loss does.

Two off switches, either is enough:

• EnableAutoUpdate=false on the stack removes the updater entirely
• Console → PUT /admin/update/settings {"autoSoftUpdates": false} turns the policy off at runtime without touching the stack

Upgrade or roll back by hand

The stack always runs a pinned, immutable vX.Y.Z tag (mutable channels are rejected because independently rolled ECS tasks must stay compatible). Set the target tag and ECS rolls each service; rolling back is the same command with the previous tag:

aws cloudformation update-stack \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --use-previous-template \
  --parameters \
    ParameterKey=ImageTag,ParameterValue=vX.Y.Z \
    ParameterKey=AllowedCidr,UsePreviousValue=true \
    ParameterKey=DeploymentToken,UsePreviousValue=true

A stack update re-pins the version

ImageTag is the floor the template describes, so a stack update sets the services back to it. If automatic updates had moved you ahead, the next check (within 15 minutes) moves you forward again. Pass the version you actually want when updating for any other reason.

Troubleshoot

Symptom-by-symptom runbooks, including the fixes that require a template update (bare nginx 500, vanished usage / SQLSTATE 28P01, 60-second timeouts): AWS troubleshooting.