Vertex AI
The gateway speaks Vertex AI natively: Gemini, Claude-on-Vertex, Llama. It mints the OAuth2 access token from a server-held service account.
| Property | Value |
|---|---|
| Provider id | vertex-ai |
| Gateway speaks | Vertex REST ({region}-aiplatform.googleapis.com) |
| Credential | Service-account JSON + project id + region, server-held |
| Client base URL | OPENAI_BASE_URL (Gemini/Llama) or ANTHROPIC_BASE_URL (Claude) → :8787 |
Configure the provider
Console → Providers → add vertex-ai:
- Project ID. Your GCP project id (or read from the JSON's
project_id) - Region. The region serving the model, e.g.
us-east5(availability is region-specific) - Service account JSON. Paste a key with the Vertex AI User role
The host doesn't need to be a GCE VM; the service account is the credential, so any Docker host in your project works (in-project is still the right place for data-boundary reasons).
Running it in GKE
The same stack runs unchanged in a GKE cluster; point workloads' base-URL env at the gateway's in-cluster Service. See Get started → Install.
Route requests to it
# Gemini / Llama via OpenAI SDKs:
export OPENAI_BASE_URL=http://<gateway>:8787/v1
# Claude-on-Vertex via Anthropic SDKs (Claude Code):
export ANTHROPIC_BASE_URL=http://<gateway>:8787
# requests carry: x-anyray-provider: vertex-ai
Models use Vertex ids, e.g. gemini-1.5-pro-002, claude-3-5-sonnet-v2,
llama-3.1-405b-instruct-maas.