Measure on your traffic
Have Claude Code signed in and Anyray Connect installed. Claude checks the kit and gateway credentials before planning. Keep credentials in the environment, never in chat or scenario files.
Scenario files, patches, the plan lock and run records stay on your machine. Paid runs send your task to the model provider, and the Anyray arm goes through your gateway, which records it like any other traffic under your content mode (encrypted by default). Publishing sends only a redacted result to your frozen gateway, and only when you ask: no task content, repository URLs, transcripts, free text or the plan lock's nonce.
In Claude Code, run:
/plugin marketplace add anyrayHQ/benchmarks
/plugin install anyray-test@anyray-benchmarks
From your repository, ask Claude to measure Anyray on the task, script or test description you already planned to run. Include the test itself in your message.
Claude records your words unchanged as source: client under anyray-test/client-1/.
It proposes extra scenarios separately. Choose which proposals to include; your test stays.
Supply your budget and the typical cost of one direct session when asked.
Before any spend, the plan shows each scenario's source and prediction, pairs needed and
planned, what saving your budget can detect, and the spend ceiling (spendCeilingUsd).
resolvable: false means the budget cannot establish the difference being tested.
Short tasks are predicted at parity. Treat that as a hypothesis until the plan confirms it.
A plan can also show prediction: none with a reason. A prediction is not a measured saving.
Review the scenarios, pinned versions and ceiling, then approve freezing the plan. Claude freezes and verifies it and shows its hash. Any later change needs a new plan.
Claude shows the spend ceiling again. Choose Start explicitly for that frozen plan. Approving the plan does not start paid work.
Claude measures direct-versus-direct noise first, then runs the paired tests. It reports progress, exclusions and costs when available. A larger scope or ceiling needs a new plan and another Start.
Read the kit's exact verdict and pooled Rule 0 result, including exclusions and predicted
versus actual results. INCONCLUSIVE stays INCONCLUSIVE; a median ratio alone is not a
savings claim.
Claude creates a local HTML report from the validated, redacted result. Keep raw transcripts local.
Ask Claude to publish this result to your console. It sends the validated, redacted result only to the frozen gateway and returns a result id. If the console is unavailable, Claude reports the blocker.
In the console, open Optimizer and click the Measurement card. Your result shows as a "Your test" line with its plan hash, pairs, cost ratio and Rule 0 verdict, beside the holdout reading.
See the command reference for manual commands, credentials and the optional per-release workflow.