Measurement reference

Follow Measure on your traffic for the Claude Code plugin workflow. The commands below run from a benchmark kit checkout with dependencies installed.

Prerequisites

Requirement	Value
Runtime	Node.js 22.13+, npm and git
Connect	Executable at ANYRAY_CONNECT_BIN or ~/.anyray/bin/anyray-connect
Claude Code	Signed in; resolve one absolute binary and pin its version for both arms
Gateway	ANYRAY_GATEWAY_URL, matching the frozen gateway
Benchmark key	ANYRAY_BENCH_CLIENT_KEY, a dedicated benchmark service key for the intended tenant
Admin credential	ANYRAY_TRACE_TOKEN, then ANYRAY_TRACE_KEY, then ANYRAY_ADMIN_KEY; needs session-gate and spend read access
Direct credential	Anthropic: ANYRAY_UPSTREAM_TOKEN or a valid Claude Code subscription token accessible through the macOS keychain. Bedrock: configured AWS profile and region
Extra headers	Clear ANYRAY_BENCH_EXTRA_HEADERS and ANYRAY_BENCH_SHARED_HEADERS; overrides are refused

Keep credentials in the environment. Kit execution supports Claude Code; plan --client codex classifies a Codex seat but does not enable this workflow to execute Codex runs.

Commands

Use --scenarios-dir ./client-scenarios on every command below. Add --json for structured output; omit it for human output. Set each scenario's source to client or proposed. Complete repo.git, pin repo.ref to a 40-character commit, and supply exactly one grading key: check, citations or keyFacts. Fill caps and any relevant patch, paste files or mcpTools. Keep client tasks verbatim.

Command	Key flags	Result
npm run plan --	--budget-usd <budget> --session-cost-usd <direct-session-cost> --gateway <url>; optional --tenant <id>, --client codex	Splits the budget evenly; shows predictions, pairs needed, detectable saving and spend ceiling. No model calls or file writes. Refuses an unreadable or all-shadow tenant gate
npm run freeze --	--model <model> --provider <provider> --client claude-code --client-version <version> --gateway <url>; --replace only for an approved replacement plan	Freezes predictions and kit Git HEAD with the plan's pins
npm run verify --	Common flags above	Checks the frozen plan; retains its predictions even if the catalog changes

Use your own typical direct-session cost for --session-cost-usd. An insufficient budget prints resolvable: false. Review and approve freeze separately from an explicit Start for paid work. Run verify again before steps that depend on an existing lock.

Paid commands

noise and run spend money. Use them only after an explicit Start for the frozen plan and stated ceiling. A larger scope or ceiling needs a new plan and Start.

Command	Key flags	Result
npm run noise --	--claude-bin <absolute-path> --rounds 6	At least six interleaved direct-versus-direct pairs per scenario; records measured log-ratio SD and reprints pairs needed
npm run run --	--claude-bin <absolute-path> --rounds <approved-rounds> --redraw-holdout 2	Paired sessions using tenant defaults. Two holdout redraws by default; the final holdout is excluded even with --redraw-holdout 0
npm run verdict --	Common flags above	Verdicts and actual medians against frozen prediction bands
npm run report --	Required --redact; add --json for the validated publishable body	Human output includes scenario ids; JSON contains aggregate shapes and statistics. Creates no HTML file
npm run publish-result --	Common flags above	Sends that validated body only to the frozen gateway; prints the result id. Missing or mixed gateway versions prevent sharing
npm run release-check --	Common flags above	Compares the gateway version with this plan's newest published result; an unreadable gateway fails the check

Run noise before paired tests. Both arms use the resolved Claude Code binary checked against the lock's version. Repeating noise or run adds rounds; monitor an existing run before resuming. Never duplicate a running command, repeat completed work or retry a losing pair.

Plan lock

Item	Contract
plan.lock.json	Keyed plan hash covers every scenario file, pins, kit commit and prediction
Frozen changes	Any change needs a new plan, freeze approval and Start. Never edit frozen scenarios or hand-write the lock
freeze --replace	New plan with a fresh nonce and current prediction catalog
Noise block	Signed noise floor added without changing the plan hash
Results	<scenarios-dir>/.kit-results/<planHash>/, with an ignoring .gitignore; a replacement plan gets a separate directory
Record contents	Allowlisted hashes, counts, costs, outcomes and versions after grading; no tasks, repositories, patches, transcripts, tool I/O, diffs or check output

Result publication excludes the nonce. See the guide's local data boundary before publishing a result.

Per-release template

Install this separately when you choose to run future paid tests in CI. This path deliberately stores scenarios and the lock in your private repository.

Setup	Value
Template	Copy kit/workflows/anyray-regression.yml to .github/workflows/anyray-regression.yml in your private scenarios repository
Inputs	Commit scenarios and plan.lock.json at the repository root after measuring noise
Repository variable	ANYRAY_GATEWAY_URL
Secrets	ANYRAY_ADMIN_KEY, ANYRAY_BENCH_CLIENT_KEY, ANYRAY_UPSTREAM_TOKEN (provider API credentials for CI)
Admin scopes	settings:read, client-tests:read, client-tests:write, optimizer:read, traces:read, spend:read
Schedule	Daily release-check; only a changed gateway version starts paid rounds. Manual dispatch also runs them
Pins	Kit commit and Claude Code version from the lock; CI keeps the frozen plan and measured noise floor
Runner	Optional ANYRAY_TEST_RUNNER, default ubuntu-latest. Bedrock needs a self-hosted runner whose instance role reaches Bedrock
OIDC	If added, include id-token: write in your workflow copy
Supported client	Claude Code plans

Workflow installation and result publication are separate choices. Retain freeze and Start gates for future paid work.