Vertex AI
The gateway speaks Vertex AI natively — Gemini, Claude-on-Vertex, Llama.
What's specific to Vertex AI
| Property | Value |
|---|---|
| Provider id | vertex-ai |
| Gateway speaks | Vertex REST ({region}-aiplatform.googleapis.com) |
| Credential | Service-account JSON + project id + region, server-held |
| Signing | OAuth2 access token minted from the service account |
| Client base URL | OPENAI_BASE_URL (Gemini/Llama) or ANTHROPIC_BASE_URL (Claude) → :8787 |
Everything shared — self-hosting, the config-based redirect, how credentials are stored — is documented once in the gateway.
Configure the provider
Console → Providers → add vertex-ai:
- Project ID — your GCP project id (or read from the JSON's
project_id) - Region — the region serving the model, e.g.
us-east5(availability is region-specific) - Service account JSON — paste a key with the Vertex AI User role
The host doesn't need to be a GCE VM — the service account is the credential, so any Docker host in your project works (in-project is still the right place for data-boundary reasons).
Running it in GKE
The same stack runs unchanged in a GKE cluster — point workloads' base-URL env at the gateway's in-cluster Service. See Get started → Install.
Route requests to it
# Gemini / Llama via OpenAI SDKs:
export OPENAI_BASE_URL=http://<gateway>:8787/v1
# Claude-on-Vertex via Anthropic SDKs (Claude Code):
export ANTHROPIC_BASE_URL=http://<gateway>:8787
# requests carry: x-anyray-provider: vertex-ai
Models use Vertex ids, e.g. gemini-1.5-pro-002, claude-3-5-sonnet-v2,
llama-3.1-405b-instruct-maas.