Vertex AI
The gateway speaks Vertex AI natively: Gemini, Claude-on-Vertex, Llama. It mints the OAuth2 access token from a server-held service account.
| Property | Value |
|---|---|
| Provider id | vertex-ai |
| Gateway speaks | Vertex REST ({region}-aiplatform.googleapis.com) |
| Credential | Service-account JSON + project id + region, server-held |
| Client base URL | OPENAI_BASE_URL → :8787/v1 (OpenAI wire, every model family) |
Configure the provider
Console → Providers → add vertex-ai:
- Project ID. Your GCP project id (or read from the JSON's
project_id) - Region. The region serving the model, e.g.
us-east5(availability is region-specific) - Service account JSON. Paste a key with the Vertex AI User role
The host doesn't need to be a GCE VM; the service account is the credential, so any Docker host in your project works (in-project is still the right place for data-boundary reasons).
The same stack runs unchanged in a GKE cluster; point workloads' base-URL env at the gateway's in-cluster Service. See Get started → Install.
Route requests to it
# Gemini, Claude-on-Vertex and Llama, all via OpenAI SDKs:
export OPENAI_BASE_URL=http://<gateway>:8787/v1
# requests carry: x-anyray-provider: vertex-ai
Every model goes through /v1/chat/completions. Gemini ids are plain (gemini-3.5-flash);
Claude and Llama ids carry their publisher: anthropic.claude-sonnet-4@20250514,
meta.llama-3.1-405b-instruct-maas.
/v1/messages is not served for vertex-ai, so Anthropic SDK clients such as Claude Code cannot
use it: they get a 400 that points to /v1/chat/completions. Route those clients to the
anthropic or bedrock provider instead.
/v1/files, /v1/batches, /v1/fine_tuning/jobs and GET/DELETE /v1/responses/{id} are not
served for vertex-ai: every request gets a 400, so the gateway never reads Cloud Storage or the
project's jobs with the stored service account.
On a key the gateway holds, /v1/proxy/* and other unrouted /v1/ paths also get a 400, so the
stored service account never reaches the Vertex AI control plane. A caller's own token keeps them.
GET /v1/models still answers.
A custom host on a key the gateway holds must be a Vertex AI host when it is a Google domain;
other Google APIs get a 400.
The held-credential host rule covers googleapis.com, google.com and googleusercontent.com,
so admins should not allowlist other Google domains that accept the token (pkg.dev, gcr.io,
firebaseio.com) as custom hosts.