Observability
Usage, cost, savings, and optimizer decisions in the console.
How it works
The gateway writes usage to your Postgres, in-process: no extra service, no project keys.
The console (:3000) is a read-only view behind the gateway's admin gate
(ANYRAY_ADMIN_TOKEN, or an optional
role-scoped SSO session). It never touches
Postgres directly, only the gateway's /admin/* endpoints, so the browser never sees the
database, and stored rows carry metadata, not content.
Console pages
With SSO on, any role can read the dashboards and traces. Changing settings needs a higher role,
and request/response bodies in a trace open only for security_admin/owner
(observability:read-content); every other
role sees the same trace with the bodies hidden, metadata and optimizer decisions intact.
| Page | Shows |
|---|---|
| Home | Spend, savings, and traffic across the org: the headline KPIs and token charts. |
| Usage | Per-user token use over time, with each user's monthly cap and a split of their traffic by upstream provider. |
| Users | Enroll users, set USD budgets, manage SSO/MDM, and diagnose auth failures. |
| Traces | Request list (including the provider and reasoning effort of each request) + per-request drill-down (optimizer decisions, provider attempts). |
Spend and savings
The Home dashboard reads GET /admin/spend/dashboard. Its headline numbers:
| KPI | What it means |
|---|---|
| Tokens saved | Input tokens the optimizer trimmed. |
| Total saving | Trimmed tokens as a share of the cost-weighted total (trimmed plus kept), with tokens served from the provider's cache counted at roughly a tenth of a fresh token, the share of a fresh token they cost. |
| Value created | The list-price value of what the optimizer saved (grossSavingsUsd, before over-allowance scaling), split into what a subscription covers vs. what's billable. Each trim is priced on its cache evidence: proven already-cached is worth a tenth of fresh input; no cache evidence counts at the full list input rate. On a warm session the token count can climb while the dollar value moves little. |
| Extra capacity | How much further the same subscription usage limits stretch: trimmed tokens against what the seat actually consumes, with provider-cached tokens weighted at roughly a tenth (shown only when you have seats). |
The same response also splits cost and tokens by reasoning effort, so you can see what the high-effort share of the bill is; rows that named no effort are collected under Default. The split spans models, and a level does not mean the same thing on all of them, so treat a shift in it as a prompt to look at the model mix rather than as a like-for-like comparison.
The same response splits cost and tokens by model, by team (each with its distinct active-member
count), by user (byUser: kept tokens, cost, and the client tools each user drove), and by
originating client tool (byClient, with a per-day clientTrend): Claude Code, Codex,
Copilot, Cursor, OpenCode, Windsurf, and raw SDK traffic each roll up separately. Rows recorded before client attribution
existed fold into unknown; rows with no attributed user into unattributed.
Each spend row carries a billing lane, taken from quota response metadata, that sets how its cost and savings count:
| Lane | Seat? | Cost & savings |
|---|---|---|
api-key | No | Full cost, full savings. |
subscription (included) | Yes | Configured over-allowance fraction. |
subscription-extra (provider-confirmed) | Same seat | Full real cost and savings. |
When a provider holds multiple named keys, each row
also records which server-held key served it (providerKeyId: the content-free key id only,
never the credential), so spend can be reconciled against each upstream account. It is empty
wherever no server-held key paid: BYO-key rows, a seat serving on its own token, and rows written
before the field existed. A seat that overflows onto the org key does carry the id, since that
row bills subscription-extra at full real cost.
To check reported savings against real measurements, sample a shadow baseline or run an audited holdout.
Developers on their own subscription
When a developer runs a tool on their own provider seat (GitHub Copilot in VS Code, a Claude or Codex subscription), Anyray optimizes the request and the tool then calls the model on that subscription. The inference never passes through the gateway, so there is nothing for it to meter:
| What the console shows for that developer | Why |
|---|---|
| Tokens saved, exact | The optimizer measured the trim before the call left the machine. |
Tokens used, cost, and spend: 0 | The model call ran on the developer's own plan; Anyray never saw the usage. |
| Not counted as an active seat | A seat is a user with billed tokens, so this usage is never billed either. |
Total saving reads — | The share of tokens saved needs the tokens that were sent; those were billed elsewhere. |
The per-user, per-team and per-model panels plot used + saved tokens, the work a developer's requests demanded, whichever side of the gateway the model call landed on. The two are disjoint, so a developer on their own seat shows up as their saved tokens beside colleagues whose traffic was billed here, and the Usage page marks how many of a developer's requests ran off-gateway. To see used-token counts for that traffic too, point the tool through the gateway or connect the provider's own usage report under spend sources.
A flat-fee seat can't produce a smaller invoice, so the value lands as capacity: the same seat does more work before it hits its usage limits. That's the Extra capacity KPI.
Custom attribution dimensions
Beyond the fixed user/team/session set, a request can carry arbitrary named dimensions
(per-feature, per-deployment, per-tenant), one header per dimension, modelled on Helicone's
Helicone-Property-<Name>:
x-anyray-property-feature: checkout
x-anyray-property-deployment: prod-eu-1
Each dimension lands on the spend row under attribution.properties and is groupable at
GET /admin/spend/usage/properties?name=<dimension>, which returns per-value token, spend, and
savings totals. The recording seam bounds each dimension:
| Bound | Rule |
|---|---|
| Dimensions per request | 16; extras are dropped. |
| Names | Lower-cased; must match [A-Za-z0-9][A-Za-z0-9_.-]{0,63} (≤ 64 chars). A failing name is skipped, never rejected. |
| Values | Truncated at 1024 characters, matching the other attribution ids. |
| Collisions | A custom name matching a fixed dimension (e.g. cohort) is namespaced under properties and cannot overwrite the enforced value. |
A malformed custom-attribution header is dropped, not fatal: it never fails the request, and the raw header value is never logged.
Usage and caps
The Usage page (GET /admin/spend/usage) shows a live cap gauge (used / cap / percent) for
capped users. Set USD budgets on the Users page; token caps are set through
PUT /admin/user-caps. When a capped user crosses a share of their cap (default 80% and
100%), the gateway posts one alert to a Slack-compatible webhook and logs a security event,
once per user, threshold, and month across the whole fleet.
Subscription spend connectors
When first-party traffic bypasses the gateway, connectors can import usage from Cursor, Devin,
and GitHub Copilot into a separate local table. Imported rows stay in each vendor's units and do
not affect gateway spend, token caps, Billing rollups, or savings. Configure sources under
Settings → Providers or /admin/spend-connectors; credentials are encrypted and write-only.
Permissions, retention, GDPR behavior, and outbound hosts:
Subscription spend connectors.
Traces
A trace records one request end-to-end; bodies follow the content mode:
| Row | What it captures |
|---|---|
| Trace | The request: session, user, status, cost, token totals, the provider that served it, and the reasoning effort it was sent with. |
| Span | Each optimizer decision: kind, summary, tokens/cost saved. |
| Generation | Each provider attempt: model, usage, timing (one per retry). |
The provider is the resolved slug (anthropic, bedrock, vertex-ai), which the model id
cannot tell you: the same claude-sonnet-4-5 runs on Anthropic direct, Bedrock, or Vertex.
Effort is what the request was sent with, after any optimizer downshift. Traces shows it as a
column and a filter; a request that named no effort shows a dash, because the provider's own
default ran. The levels are the union of what the providers accept rather than one ranked scale —
low means Anthropic's floor and OpenAI's second rung — so read a level against the model beside
it.
Requests served before your gateway recorded a slug fall back to the upstream host. Token counts
include the cache split, and promptTokens is the full billed input (fresh + cache read + cache
write), so cost stays correct for prompt-caching traffic. Full field list:
Gateway → Spend and metering.
Traces are the only store that can hold content, and only when
the content mode keeps it. Their retention is set
independently of spend retention on the console Privacy page; unset, they're pruned after
90 days (or ANYRAY_SPEND_RETENTION_DAYS). A GDPR delete erases a user's traces. Auth-event
rows keep a reason code and 30 days of history for the Users → Diagnose panel.
Where it's stored
Everything derives from Postgres; the gateway is the only writer.
| Store | Table(s) | Read by |
|---|---|---|
| Spend | anyray_spend | /admin/spend/dashboard · /admin/spend/usage |
| Trace | anyray_traces + anyray_observations | trace drill-down (/admin/observability/*) |
| Auth events | anyray_auth_events | /admin/auth-events |