TrueFoundry

The gateway sits in front of your TrueFoundry AI Gateway, which keeps owning upstream routing and credentials.

Same front-door arrangement as LiteLLM. Requests flow client tool → Anyray gateway → TrueFoundry → provider, each dialect forwarded in its own wire format, so client tools stay on their native managed lanes. What differs here:

Property	Value
Provider id	truefoundry, never the generic custom upstream (why)
Gateway speaks	Anthropic /v1/messages for Claude, OpenAI /v1/chat/completions otherwise
Credential	A TrueFoundry Personal Access Token or Virtual Account Token
Base URL	Your control plane including /api/llm, or TrueFoundry's SaaS host
Model ids	Namespaced by provider account, such as anthropic-main/claude-opus-4-8

Set it up

1

Allow the TrueFoundry hostname

Add it to ANYRAY_CUSTOM_HOST_ALLOWLIST, or every request fails with "Invalid custom host". Skip this when the control plane is already the gateway's configured upstream, a private address included.

2

Add the provider in the console

Field	Value
Base URL	https://acme.truefoundry.cloud/api/llm, keeping the /api/llm prefix
API key	Your PAT or VAT, sent as a Bearer token
Custom headers	Optional JSON, for TrueFoundry's x-tfy-* routing headers

A trailing /v1 is fine. TrueFoundry's own docs publish the base URL with that suffix, and the gateway appends its own /v1/… path, so both spellings resolve to the same upstream URL.

3

Onboard each developer

One command per machine. Your admin supplies the enrollment link (onboard users), and the upstream key is that developer's own TrueFoundry token:

curl -fsSL https://app.anyray.ai/connect.sh | sh -s -- \
  --enroll https://app.anyray.ai/enroll/enl_… \
  --user dev@example.com \
  --upstream https://acme.truefoundry.cloud/api/llm \
  --upstream-key tfy-synthetic1234 \
  --yes

This mints a personal ark_… key, so spend is attributed to that developer. It points every detected tool at the gateway on its managed lane, and writes the pins that forward the token verbatim on both the Anthropic and the OpenAI lane, so every model the control plane serves is paid for by that developer's own token. --upstream and --upstream-key go together (CLI commands).

4

Run one session and check the console

The request appears under the truefoundry provider with the developer's name attached. Savings show within a few turns.

No per-developer tokens? Omit both flags, and set one org token on the TrueFoundry provider in the console instead.

Keep the provider-account prefix on your model ids

TrueFoundry addresses models as provider_account/model_name, and an org can also define short aliases. Keep the prefixed form where you can.

Two things read the model id, and a house alias is opaque to both. Pricing resolves the rate from the id, and an unrecognized one falls back to a flat rate that misprices every row. The optimizer's cache guard classifies the lane from it. A prefixed id such as anthropic-main/claude-opus-4-8 resolves correctly on both.

Want short names? Define them as model aliases on the Anyray side, where the gateway resolves them to a concrete provider and model before either check runs.

Request prompt caching with block-level cache_control

TrueFoundry accepts a cache request three ways: block-level cache_control, an automatic top-level placement, and an x-tfy-cache-control header. Prefer the block-level form, which is what client tools send natively.

The optimizer detects a cached request by scanning the body for cache_control breakpoints. It cannot see the header form, so a request cached that way reads as uncached, and the cache-safety guard will not know there is a warm prefix to protect.

The Responses API needs a routing header

TrueFoundry's /responses endpoint requires an x-tfy-provider-name header naming the provider integration. On the org-key lane you can set it in the provider's Custom headers field.

The per-developer passthrough lane has no way to attach it, so Codex-style clients on that lane should use the chat or messages endpoints.