Skip to main content

AWS reference

Depth behind the AWS guide: the CLI install path, VPC lookups, and troubleshooting.

Find the AWS values​

Only needed when placing Anyray in a specific VPC; leaving VpcId, SubnetA and SubnetB blank lets the stack use the account's default VPC.

export AWS_REGION=eu-central-1

aws sts get-caller-identity

aws ec2 describe-vpcs \
  --region "$AWS_REGION" \
  --filters Name=is-default,Values=true \
  --query 'Vpcs[].{VpcId:VpcId,CidrBlock:CidrBlock,State:State}' \
  --output table

aws ec2 describe-subnets \
  --region "$AWS_REGION" \
  --filters Name=default-for-az,Values=true \
  --query 'Subnets[].{SubnetId:SubnetId,Az:AvailabilityZone,AutoPublicIp:MapPublicIpOnLaunch,AvailableIps:AvailableIpAddressCount}' \
  --output table

Pick two subnets where AutoPublicIp is True and the Az values differ. No public subnets? Create them, or use Remote docker on an instance you manage.

Install with the AWS CLI​

Validate the template
aws cloudformation validate-template \
  --region "$AWS_REGION" \
  --template-url https://anyray-quicklaunch.s3.us-east-1.amazonaws.com/anyray-quicklaunch.template.yaml

The template creates IAM roles and uses the AWS::LanguageExtensions transform, so pass both CAPABILITY_IAM and CAPABILITY_AUTO_EXPAND:

Create the stack
export VPC_ID="vpc-..."
export SUBNET_A="subnet-..."          # AZ 1
export SUBNET_B="subnet-..."          # AZ 2 (different AZ)
export ALLOWED_CIDR="$(curl -fsS https://checkip.amazonaws.com)/32"
export DEPLOYMENT_TOKEN="adt_..."

aws cloudformation create-stack \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --template-url https://anyray-quicklaunch.s3.us-east-1.amazonaws.com/anyray-quicklaunch.template.yaml \
  --parameters \
    ParameterKey=AllowedCidr,ParameterValue="$ALLOWED_CIDR" \
    ParameterKey=DeploymentToken,ParameterValue="$DEPLOYMENT_TOKEN"

aws cloudformation wait stack-create-complete \
  --region "$AWS_REGION" \
  --stack-name anyray

aws cloudformation describe-stacks \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --query 'Stacks[0].Outputs' \
  --output table
Read the outputs and fetch the admin token
export GATEWAY_URL="$(
  aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
    --query "Stacks[0].Outputs[?OutputKey=='GatewayURL'].OutputValue | [0]" --output text
)"
export CONSOLE_URL="$(
  aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
    --query "Stacks[0].Outputs[?OutputKey=='ConsoleURL'].OutputValue | [0]" --output text
)"

# AdminTokenCmd output is the exact CLI call that prints the admin token:
aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
  --query "Stacks[0].Outputs[?OutputKey=='AdminTokenCmd'].OutputValue | [0]" --output text

curl -fsS "$GATEWAY_URL/" && echo "gateway ok"
printf 'Console: %s\n' "$CONSOLE_URL"

Troubleshooting​

CloudFormation fails or rolls back​

aws cloudformation describe-stack-events \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --query 'StackEvents[0:15].[Timestamp,ResourceStatus,LogicalResourceId,ResourceStatusReason]' \
  --output table

A ResourceInitializationError … Failed to resolve "fs-….efs.<region>.amazonaws.com" on GatewayServiceV2 / OptimizerServiceV2 means the VPC has DNS hostnames disabled: the tasks cannot mount EFS, so the ECS deployment circuit breaker rolls the stack back. Enable both DNS attributes on the VPC, then delete the stack and create it again:

aws ec2 modify-vpc-attribute --vpc-id <vpc-id> --enable-dns-support
aws ec2 modify-vpc-attribute --vpc-id <vpc-id> --enable-dns-hostnames

A failed create can end ROLLBACK_FAILED: the template enables deletion protection on the load balancer and the RDS instance, which blocks the rollback's deletes. Disable both, then delete the stack and create it again:

aws elbv2 modify-load-balancer-attributes --load-balancer-arn <alb-arn> \
  --attributes Key=deletion_protection.enabled,Value=false
aws rds modify-db-instance --db-instance-identifier <db-id> \
  --no-deletion-protection --apply-immediately

The stack completes but the console does not open​

CloudFormation finishes once resources exist, but the Fargate services need RDS reachable before they go healthy:

aws ecs describe-services \
  --region "$AWS_REGION" \
  --cluster anyray-anyray \
  --services anyray-gateway anyray-proxy \
  --query 'services[].{name:serviceName,running:runningCount,desired:desiredCount,events:events[0].message}'

A service stuck at running: 0 is usually crash-looping; read its CloudWatch logs under /anyray/<stack-name>/gateway (or /optimizer, /proxy).

The gateway or console target is unhealthy​

The ALB health-checks the gateway on / and the console on /anyray-login:

aws elbv2 describe-target-health \
  --region "$AWS_REGION" \
  --target-group-arn "$(aws elbv2 describe-target-groups --region "$AWS_REGION" \
    --query "TargetGroups[?contains(TargetGroupName, 'anyray')].TargetGroupArn" --output text | head -1)"

A gateway that keeps failing is most often still waiting on RDS; check its CloudWatch log group for the database connection.

The console shows a bare nginx 500​

Stacks created before July 2026 wired services with ECS Service Connect, whose names the console proxy cannot resolve, so every console request returned 500 Internal Server Error (nginx). Update the stack with the current template to pick up the fix (settings, database, and secrets are untouched; the ECS services are replaced):

aws cloudformation update-stack \
  --region "$AWS_REGION" \
  --stack-name anyray \
  --template-url https://anyray-quicklaunch.s3.us-east-1.amazonaws.com/anyray-quicklaunch.template.yaml \
  --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
  --parameters "$(aws cloudformation describe-stacks --region "$AWS_REGION" --stack-name anyray \
      --query 'Stacks[0].Parameters[].{ParameterKey:ParameterKey,UsePreviousValue:`true`}' --output json)"

Usage stops being recorded, or a service crash-loops on the database​

The symptom is a gateway that serves every request normally while its dashboards read empty, or an ECS service that crash-loops right after a roll. Both point at the same thing: the stored database connection URL no longer matches the database password.

Stacks created before August 2026 let RDS rotate the master password on its own schedule, and nothing in the stack followed that rotation. Already-running tasks keep working off their open connections, so the mismatch stays invisible until something opens a new one (an image roll, a scale event, an automatic update), which is why the failure often appears long after the rotation that caused it. In the logs it is password authentication failed for user "postgres" (SQLSTATE 28P01).

Check the current state with the health endpoint; spend.ok is the live database probe (GATEWAY_URL / ADMIN_TOKEN as in Install with the AWS CLI):

curl -s "$GATEWAY_URL/admin/health" -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  | python3 -m json.tool

The current template fixes this permanently: the master password is generated once into <stack-name>/anyray/db-master and no longer rotates, and the stack re-derives the connection URL before every automatic update. Updating the stack is the fix, and it is the only way to get it: automatic updates roll container images and never change infrastructure. Run the update-stack command from the previous section, and always pass --template-url; --use-previous-template re-applies the template already stored on the stack, so it cannot deliver this fix.

The update sets a new master password and re-derives the connection URL. Tasks running with the old credential are not migrated in place, so roll the services once the update completes:

for svc in $(aws ecs list-services --region "$AWS_REGION" --cluster anyray-anyray \
    --query 'serviceArns[]' --output text); do
  aws ecs update-service --region "$AWS_REGION" --cluster anyray-anyray \
    --service "${svc##*/}" --force-new-deployment >/dev/null
done

Usage recorded while the database was unreachable is not recoverable (the gateway does not queue spend rows), so the gap stays visible in historical dashboards. Optimizer savings are reported separately and are unaffected, as is billing.

What removes this class of failure entirely

A stored password is the thing that can drift out of sync. RDS IAM database authentication stores none: the gateway mints a short-lived token from its own task role on each connection. The quick-launch template does not wire it up yet, so it is not a fix you can apply to this stack. It is available wherever you supply ANYRAY_SPEND_DB_URL yourself, including Kubernetes and Local / VM.

Long requests fail after about a minute​

The symptom is a request that dies at roughly 60 seconds every time, or a coding assistant reporting ECONNRESET / a closed socket on the first request after a pause, while short requests are fine.

Both are the load balancer's idle timeout, not the gateway. Stacks created before August 2026 left it at the AWS default of 60s, and that one value cut two different things:

  • A completion in flight. A model is silent between the request and its first token; on a large prompt with extended thinking that gap runs past a minute, so the ALB closed a connection the gateway was still legitimately holding.
  • A pooled connection between turns. Coding tools keep a connection open across turns, and a real session's think-gap routinely exceeds a minute. The gateway holds those sockets for ten minutes on purpose, but on AWS the client pools against the load balancer, so its 60s was the limit that actually applied and the next reuse hit a closed socket.

The current template sets idle_timeout.timeout_seconds to 300. Updating the stack is the fix (pass --template-url, as in the section above); automatic updates roll container images and never change infrastructure, so this cannot arrive on its own. To confirm what your stack has today:

aws elbv2 describe-load-balancer-attributes \
  --region "$AWS_REGION" \
  --load-balancer-arn "$(aws elbv2 describe-load-balancers --region "$AWS_REGION" \
    --query "LoadBalancers[?contains(LoadBalancerName, 'anyray')].LoadBalancerArn" \
    --output text | head -1)" \
  --query "Attributes[?Key=='idle_timeout.timeout_seconds']"