Strategy
A strategy is the unit of optimization. It inspects one request, transforms it or leaves it
untouched, and emits a decision recording what it did (kind, short summary, token/cost
estimates). It applies only when its own condition matches, and one that throws is skipped, so a
broken strategy can never break inference.
Each strategy is registered in the optimizer's REGISTRY
(optimizer/src/strategies/index.ts); a new optimization is a new strategy plus a registry entry.
Managed Claude source hooks run before tool results enter the transcript. They stand aside unless the current lane can retrieve the original. Source-hook behavior.
The Library
| Strategy | What it does |
|---|---|
observation_mask | Retires stale tool observations behind a retrievable marker. |
command_digest | Digests recognized dev-command output (test runs, grep, logs). |
code_graph | Keeps relevant symbols across multi-file reads. |
cache_optimizer | Stabilizes the request's start so providers reuse their cache. |
relevance_filter | Keeps only the relevant lines of a tool output. |
thinking_trim | Removes re-sent copies of past reasoning. The current turn's thinking is untouched. |
context_dedupe | Collapses repeated identical or near-identical tool outputs to the first copy plus changed lines. |
context_compression | Shrinks tool outputs, logs, and RAG chunks. |
semantic_cache | Serves a cached response for duplicate requests. |
prompt_compression | Shortens long prompts and system messages. |
tool_schema_compression | Shrinks tool definitions while keeping all tools. |
provider_context_trim | Asks Anthropic to clear aged tool results provider-side, before billing. |
Off by default
Enable these on a deployment you have measured. Each one either needs a client capability, or changes what the model sees in a way worth checking against a direct connection first.
| Strategy | What it does |
|---|---|
output_externalize | Moves a bulky tool output off the request, retrievable on demand. |
columnar_fold | Sends a record list's field names once instead of once per record. |
repeat_factor | Points a repeated block at its first copy, which stays inline. |
tool_pruning | Drops tools unlikely to be called. |
client_prefix_compression | Swaps a known client's own boilerplate for curated short forms. |
param_tuning | Caps an over-large max_tokens. |
reasoning_budget | Downshifts reasoning effort on routine tool-resume turns. |
output_shaping | Asks for concise replies on routine tool-resume turns. |
window_budget | Crops low-relevance middle turns to fit the model's context window. |
Provider prompt caches bill only the new part of a request at full price; everything before
the cache_control breakpoints is a cheap cache read. Cache guards suppress unstable prefix edits. Replay-capable strategies may make a bounded,
pinned rewrite and replay it on later turns, refreshing any retrieval handle. The session gate measures whether that cost pays
back on the tenant's traffic.
Reference
What each strategy edits, when it runs, and which parameters it takes. Observe-only kinds, meaning meters and audits, are not listed.
observation_mask
Replaces an old, large tool result with a one-line marker, so you stop paying for it every turn. The original stays retrievable, and failures, the current turn, and files being edited are always kept.
Masking reserves room for the requested output, estimator uncertainty, and the largest completed
exchange observed so far. The latest completed exchange and unresolved tool calls stay intact.
Once a turn nears its context limit (the client's own ceiling when it declares one, otherwise the
model window), masking runs freely. Below that, only cached-prefix masks that pay on the current
turn run, using the input-cost gate below. Explicit keepRecentTurns, minWindowFill, and
minFillTokens retain the older policy; setting both fill bars to 0 permits masking at any size.
Existing pins still replay.
New cached-prefix masks, command digests, and graph outlines share an estimated input-cost gate.
The retained prefix must cost no more to rewrite now than the original would cost to read from
cache. This includes system/tool definitions and the cache TTL, using the shared provider pricing.
Unknown routes or prices, media, and first-time retrieval-tool activation decline automatic mints.
An explicit mintPaybackRatio selects the legacy size-ratio policy. These estimates do not prove
whole-session savings or answer quality; the session gate
remains the shipping requirement.
A client with no read path gets a short
head-and-tail excerpt instead (noRetrieveKeepChars, default 800), and an api-key lane with no
read path is skipped. Content storage turned off does the same: the marker would point at nothing.
turn 2 [tool] Read src/checkout.ts → 380 lines (≈ 9 KB)
turn 5 [tool] npm test → 2 failed, 126 passed (≈ 6 KB)
…
turn 12 [tool] Read src/tax.ts ← fresh input
turn 2 [tool] [anyray: observation masked · 9214 chars · retrieve ctx_9f… via anyray_retrieve]
turn 5 [tool] npm test → 2 failed, 126 passed ← kept: error
…
turn 12 [tool] Read src/tax.ts ← kept verbatim
command_digest
Recognizes known dev-command output and keeps what matters for that command. A test run keeps its
failures and its count line. grep output is bucketed by file, and repetitive logs collapse to
[×N]. The full output stays retrievable.
Grep hits that differ only in a timestamp, duration, or client IP fold into one line with a count.
Hits that differ by path, identifier, or status code stay separate, so 500 and 200 never merge.
It touches history only, so the current turn's results always pass through. On a cached session, a settled digest replays to identical bytes.
============ test session starts ============
platform linux -- Python 3.11.8, pytest-8.1.1 · collected 128 items
tests/test_auth.py ......F............... [ 22%]
tests/test_billing.py .................... [ 71%]
tests/test_api.py ..........F......... [100%]
================== FAILURES ==================
FAIL: test_login_lockout - assert False is True (tests/test_auth.py:42)
FAIL: test_refund_rounding - assert 999 == 1000 (tests/test_billing.py:88)
====== 2 failed, 126 passed in 3.41s ======
FAIL test_login_lockout tests/test_auth.py:42 assert False is True
FAIL test_refund_rounding tests/test_billing.py:88 assert 999 == 1000
2 failed, 126 passed in 3.41s
… digest · full output retrievable (retrieve ctx_a4f1)
code_graph
When an agent re-reads several source files in one request, code_graph keeps the functions in play
and the ones they call. Other bodies become a short marker, and the signatures stay. Hidden bodies
stay retrievable.
It picks what to keep from a reference graph across the files, matched against the current turn, not against which file was read last. A file the agent is editing, or that the turn names by path, is preserved during new selection, so edits use real bytes. Fresh tool results and retrieved originals stay intact. New outlines apply only beyond the provider's cached prefix; existing pinned outlines replay unchanged. If the cache covers all eligible history, the strategy leaves it alone. On an api-key lane whose route caches without markers (for example OpenAI or xAI, including the Messages dialect translated onto xAI) it stands aside: there each new outline is billed as a fresh rewrite of the cached prefix, and that cost more than it saved.
Markdown, JSON, shell output and unrecognized source stay intact. The retired jsonSkeleton,
jsonDepth and mintPaybackRatio settings remain accepted for compatibility but do not enable
fallback outlines or cached-prefix rewrites. No LLM or embeddings run.
# 6 files re-read this session · ≈ 48,000 chars
checkout.ts (asked about) charge(), refund()
pricing.ts (asked about) price()
tax.ts (imported) calcTax() ← called by price()
audit.ts (imported) log(), flush()
loyalty.ts (imported) award()
email.ts (imported) send()
# working set kept in full · ≈ 11,000 chars
checkout.ts charge(), refund() full body
pricing.ts price() full body
tax.ts calcTax() ← neighbor of price() full body
audit.ts log() → "[8 lines · retrieve ctx_7c]"
loyalty.ts, email.ts → outlined, bodies elided
relevance_filter
Builds a query from the latest message, ranks every line of the tool output, and keeps the top scorers plus a few leading lines for orientation. The user's own words are never ranked or trimmed. Removed lines stay retrievable.
It touches history only, and on cached traffic it ranks just the uncached tail. It stands aside for
a source file that code_graph will outline. Ranking that file line by line would
break the structure code_graph reads.
Hybrid mode adds a local embedding re-rank that catches paraphrases and synonyms. It runs in-network and falls back to lexical ranking.
find the 500 errors on /checkout
10.0.2.14 GET /health 200 2ms
10.0.2.14 GET /assets/app.js 200 5ms
10.0.9.3 POST /checkout 500 812ms
10.0.2.14 GET /home 200 8ms
10.0.9.7 POST /checkout 500 903ms
… 1,195 more lines … (≈ 52,000 chars)
find the 500 errors on /checkout ← kept verbatim
… [312 line(s) omitted by anyray · retrieve ctx_5a]
10.0.9.3 POST /checkout 500 812ms
10.0.9.7 POST /checkout 500 903ms
10.0.9.7 POST /checkout 500 774ms
… [180 line(s) omitted by anyray · retrieve ctx_5a]
(≈ 3,800 chars)
semantic_cache
When a request exactly repeats one already seen, Anyray serves the stored response and skips the
provider. A hit needs an exact match on the caller and on every field that can change the answer:
model, system and messages, tools, and all sampling params. Only stream and a top-level
prompt-cache directive are exempt, and headers never count. Any other body field is part of the key.
Entries are partitioned per caller: the tenant plus the attributed user, else the team, else the
session id. A request that resolves none of those is neither served nor stored. On a miss it also
tries a shadow key that ignores timestamps, UUIDs, and temp paths, then records whether that key
would have hit. That tier only measures, and its counters sit on GET /admin/optimizer/settings.
The deprecated similarityThreshold parameter is accepted but ignored; it never enables fuzzy serving.
request #1 "What is our refund policy?" (gpt-4o) → provider call (~2.3 s)
request #2 "What is our refund policy?" (gpt-4o) → provider call (~2.3 s)
• billed twice
request #1 "What is our refund policy?" (gpt-4o) → MISS → provider call (~2.3 s)
request #2 "What is our refund policy?" (gpt-4o) → HIT → served from cache
• provider skipped → 0 tokens billed
• ~0 ms vs ~2.3 s upstream
context_dedupe
Collapses a tool output the agent has already seen. A re-read file or a re-run test suite becomes a
one-line marker, and a near-repeat keeps only the changed lines. The first copy stays verbatim, and
every collapsed copy stays retrievable. A near-repeat must reconstruct exactly from its anchor and
bounded diff, with positive estimated token savings. There is no default similarity percentage or
half-size savings requirement. An explicit nearSimilarity remains supported.
It rules on a copy once, when that copy arrives, then replays the same verdict on later turns, so
the request keeps the same bytes for the provider cache. Content the agent pulled back with
anyray_retrieve is never collapsed again. On an api-key lane it stands aside, read path or not,
because that lane bills every byte and the agent may be checking a fix against the re-run output.
Collapses made before a session reached this rule keep replaying, so nothing reverts mid-session.
turn 3 [tool] Read src/billing.ts → 412 lines (≈ 11 KB)
…
turn 19 [tool] Read src/billing.ts → the same 412 lines, barely changed (≈ 11 KB)
turn 3 [tool] Read src/billing.ts → 412 lines (≈ 11 KB)
…
turn 19 [tool] [anyray: near-duplicate delta-collapsed — 98% identical to the
412-line output above; 3 changed line(s) below · retrieve ctx_8k2p…]
@@ -41,1 +41,1 @@ -const RATE = 0.07; +const RATE = 0.075;
@@ -388,0 +389,1 @@ +export const auditRate = () => RATE;
context_compression
The catch-all for bulky tool output. It minifies JSON and compresses known output shapes. Under context pressure it budgets whole paragraphs, log lines, diff hunks, or stack frames. A unit it cannot interpret safely stays complete; unparsed code and JSON are never byte-sliced. What it removes stays retrievable.
Long arrays and string values stay whole by default (maxArrayItems: 0, maxFieldChars: 0).
Explicit maxChars, maxFrames, or maxDiffLines set structural limits. An explicit
maxFieldChars replaces an oversized JSON string as a whole value, not a partial identifier. A cut list still looks complete to the model, so a
value past the cut becomes a confident wrong answer. Set maxArrayItems only for data that is safe
to cut from the end. It runs after the specialists, so it cannot mangle their input, and it never
touches the current turn. A client that cannot call POST /v1/retrieve still gets its history
compressed, but nothing is stashed and no handles are emitted.
On prompt-cached traffic, it leaves cached originals untouched. Where replay remains enabled,
existing compression pins recreate the same bytes and retrieval handles, but settlement alone never
permits a new edit inside the cached prefix. New compression can run on eligible uncached history;
clients that cache through the newest message may have none. Stale pins are dropped without
recompressing cached originals. On an api-key route that caches without honored markers (for example
OpenAI or xAI), fresh compression stands aside under billed_lane. Legacy recipes without retrieval
handles can still replay; recipes with handles are withheld by the pipeline.
[
{ "id": 1, "name": "alice", "email": "a@ex.com", "active": true },
{ "id": 2, "name": "bob", "email": "b@ex.com", "active": true },
{ "id": 3, "name": "carol", "email": "c@ex.com", "active": false },
… 797 more rows …
] (≈ 41 KB)
[{"id":1,"name":"alice","email":"a@ex.com","active":true},
{"id":2,"name":"bob","email":"b@ex.com","active":true},
{"id":3,"name":"carol","email":"c@ex.com","active":false},…] (≈ 17 KB)
output_externalize
Replaces a whole bulky message with a compact handle and stashes the original. The agent pulls the
exact bytes back with POST /v1/retrieve. It applies to tool outputs at or above minChars
(default 4000), keeps the copy for ttlSeconds (default 86400), and skips the current turn.
It stands down in two cases. It runs only when the durable tier can persist and the client can call
POST /v1/retrieve. It is also registered as cache-busting, so it is suppressed on prompt-cached
traffic.
turn 4 [tool] psql → 1,842 rows (≈ 180 KB)
turn 4 [tool] [anyray:externalized 184320 chars · retrieve ctx_9f3a]
columnar_fold
A tool that returns a list of records repeats the same field names on every record.
columnar_fold rewrites the list as one column header plus one row per record, and lifts out any
field with the same value on every row.
Nothing is dropped, shortened, or stashed. The folded payload is unfolded and compared against the original first, and the tool result passes through untouched if anything differs.
| Parameter | Default | Meaning |
|---|---|---|
| roles | tool, function | Message roles it applies to. |
minChars | 400 | Minimum payload size. |
minRecords | 4 | Minimum records in the array. |
minSavedChars | 120 | Skip a fold that would not save at least this much. |
Records with different key sets are left alone, and a payload with prose after the JSON is declined. The same tool result folds to the same bytes every turn, so a cached prefix holding it does not change.
[{"object":"order","id":"ord_1N00","currency":"usd","amount":1995,"status":"paid"},
{"object":"order","id":"ord_1N01","currency":"usd","amount":4200,"status":"failed"},
… 498 more records …]
{"__anyray_columns__":["id","currency","amount","status"],
"__anyray_constant__":{"object":"order"},
"__anyray_rows__":[["ord_1N00","usd",1995,"paid"],
["ord_1N01","usd",4200,"failed"],
… 498 more rows …]}
repeat_factor
Long blocks of text repeat across tool results that are not duplicates of each other: the same
licence header on nine file reads, the same schema echoed by three query results. repeat_factor
finds the longest repeated blocks, keeps the first copy as it is, and replaces each later copy with
one line.
Nothing is stashed. The block a marker points at is still in the request, so the model resolves the reference by reading, not by calling a tool. The first copy is never rewritten, and neither is a span another strategy already claimed.
| Parameter | Default | Meaning |
|---|---|---|
| roles | tool, function | Message roles it applies to. |
minLines | 8 | Minimum lines in a repeated block. |
minRunChars | 400 | Minimum characters in a repeated block. |
maxLinesScanned | 20000 | Scan cap per request, counted from the first message. |
It removes only redundancy, so a cached prefix holding a marked block does not change.
[tool] read src/billing.ts 62-line licence header + imports, then the code
[tool] read src/checkout.ts the same 62 lines, then different code
[tool] read src/pricing.ts the same 62 lines, then different code
[tool] read src/billing.ts the 62 lines, kept verbatim
[tool] read src/checkout.ts [anyray: 62 identical lines elided — repeat of the
62-line block starting "// Copyright (c) 2026 …"
earlier in this conversation]
[tool] read src/pricing.ts the same one-line marker
tool_pruning
Drops tools the model is unlikely to call, which trims schema overhead on every request, and it errs
toward keeping. A tool survives if its name or description shares a topic word with the
conversation. Generic words like "get" or "find" do not count, and neither does wording most
descriptions share. MCP tools (mcp__…) are never pruned.
Deciding again every turn would shift the tools prefix and lose the cache. On a pinned session it
decides once and replays the same set byte-for-byte. It decides again when the tool list changes,
when tool_choice forces a pruned tool, or after a tool-miss. Without pins it stands aside on warm
cached traffic.
task: "comment on the P0 bugs in the PAY project"
tools (3): (≈ 2,400 schema tokens)
jira_search_issues "Search issues in a Jira project by JQL."
slack_post_message "Post a message to a Slack channel."
datadog_query_metric "Query a Datadog metric over a range."
tools (1): (≈ 800 schema tokens)
jira_search_issues "Search issues in a Jira project by JQL."
prompt_compression
Collapses repeated whitespace and drops duplicate sentences and paragraphs in system and user
messages over 400 characters. It keeps the first copy and preserves line and list structure. Code
passes through byte-for-byte, and tool result blocks are never touched.
Each message is judged on its own bytes. One is rewritten only when the pass saves at least
minSavedChars (24) and minSavedRatio (3%) of it, because the system prompt and the first user
turn sit in the cached prefix, where a trivial rewrite is re-billed in full on a bypassed turn. On
an api-key lane it stands aside, retrieval path or not, since a user turn there can carry a pasted
tool report and a thinner report changed how the agent worked in a paired benchmark.
You are a senior support engineer.
Read the ticket below and reply. Be concise. Be concise. Be concise.
Then suggest next steps. Then suggest next steps.
You are a senior support engineer.
Read the ticket below and reply. Be concise.
Then suggest next steps.
tool_schema_compression
Shrinks each tool's free-text description. It collapses whitespace, strips boilerplate like "This tool allows you to…", and rewrites filler. Types, enums, and required fields stay byte-for-byte intact, so every tool stays callable. It also shrinks parameter descriptions, and it keeps the details that tell the model what to pass, such as "for example PAY or CORE".
It pairs with tool_pruning: pruning drops the irrelevant tools, and this shrinks
the ones that stay. It reads only each tool's own bytes, so the same schema always gives the same
output. On ten verbose MCP schemas the default pass trims about 8% of the request with no loss, and
the optional maxDescriptionChars cap takes that to about 32%.
search_kb: "This tool allows you to search the knowledge base. Use it
whenever you need a citation." (~110 chars)
create_ticket: "This tool allows you to create a support ticket. Please be
sure to fill every field." (~102 chars)
search_kb: "search the knowledge base. Use it whenever you need
a citation." (~76 chars)
create_ticket: "create a support ticket. Fill every field." (~44 chars)
client_prefix_compression
Some clients ship a large fixed prefix on every request. Claude Code's system prompt and built-in tool descriptions run to about 12k tokens, re-read on every call. This strategy carries a curated table: for each known passage, the exact bytes a shipped version sends and a hand-written short form that keeps every instruction. A match is byte-for-byte, so anything that does not match is forwarded untouched. The table covers Claude Code 2.1.257 and 2.1.260 on Sonnet 5.
| Parameter | Default | Meaning |
|---|---|---|
tables | ["claude_code_system"] | Curated tables to apply. claude_code is both halves; claude_code_tools the built-in tool descriptions only. |
Per turn it saves about 20% of the input bill, but on a real repository the curated prefix made sessions longer, and a long session costs four to five short ones. Measure your own sessions against a direct connection before you turn it on.
Static system prompt passages: 24,570 chars
Static system prompt passages: 15,065 chars
param_tuning
Caps an over-large max_tokens. The default cap is 65536. This is a guardrail against runaway
completions, not a spend saver. You are billed for the tokens actually generated, so a lower ceiling
does not lower the bill. A request already under the cap passes through unchanged.
max_tokens: 100000
max_tokens: 65536
provider_context_trim
On a Claude request over a size threshold, in any billing mode, it adds Anthropic's native
context-management edit (clear_tool_uses_20250919) and its beta header. Anthropic then clears aged
tool results before billing. API-billed requests stop paying for those tokens, and subscription
seats stop burning usage limit on them.
It uses the same pressure calculation as masking. The keep count follows
active exchanges, and the clear amount restores headroom for another observed exchange. No known
capacity or insufficient clearable tool history means no annotation. Explicit triggerTokens,
keepToolUses, and clearAtLeastTokens remain overrides. Messages are not rewritten, and a request
with its own context_management block passes through untouched.
observation_mask takes precedence, and Anyray reports no savings here.
POST /v1/messages claude-* (metered API key)
long agent trajectory ≈ 72,000 input tokens
aged tool results are re-billed every turn
same message content + injected context_management:
trigger: usable input capacity minus observed exchange growth
keep: tool uses in the active exchanges
clear_at_least: estimated tokens needed to restore headroom
→ Anthropic clears aged tool results before billing
reasoning_budget
On a routine tool-resume turn, where the fresh input is tool results and nothing failed, it lowers reasoning effort for metered reasoning models. A client-set value only ever moves down:
- Anthropic thinking budget. Caps
thinking.budget_tokens, and never adds or removesthinking. - Anthropic effort. Sets or clamps
output_config.efforton allow-listed Claude models. - OpenAI output controls. Sets or clamps GPT-5
verbosityandreasoning_effort.
On a pinned session the first decision mints a pin, so later turns replay the same value. Values back off after a tool error, and a model switch drops the pin. It claims zero estimated savings, so run it as a canary and measure the output-token and session-length delta.
[assistant] run tests with high reasoning
[tool] 128 passed (fresh input)
thinking.budget_tokens: 31999
[assistant] run tests with high reasoning
[tool] 128 passed (fresh input)
thinking.budget_tokens: 8192
output_shaping
The one output-side strategy. On a routine tool-resume turn it appends one short advisory to the end of the request. The advisory asks the model to reference file paths and diffs instead of restating context. It edits nothing already in the request, so cached prefixes stay byte-identical.
| Parameter | Default | Meaning |
|---|---|---|
minAssistantTurns | 2 | Assistant turns required before a tool-resume counts as routine. |
skipOnErrorTurn | true | Skip fresh tool results the provider flagged with is_error. |
It reports zero estimated savings. The advisory steers the model rather than capping it, so only the measured output-token delta is honest. The advisory is visible text in the request, and some coding agents flag in-content instructions as prompt injection. Try it on one lane first.
[assistant] called a tool
[tool] routine tool result (fresh input)
[assistant] called a tool
[tool] routine tool result (fresh input)
+ [anyray:output-guidance] Be concise: reference file paths and diffs
instead of restating context or re-printing unchanged files.
thinking_trim
Shown in the console as Thinking replay trim, it removes replayed reasoning, not the model's
reasoning on the current turn (that is reasoning_budget). Agentic clients
resend finished thinking blocks with every request, and you pay input tokens for them every time.
It keeps the opening plan of each tool loop (keepLoopHead, default 1), then trims a block once
everything it names also appears in the same assistant message. Opaque blocks still cost you on
replay, so cutOpaqueChain (default true) drops them past the loop head. Three kinds are always
kept: encrypted redacted_thinking, reasoning after a tool error (skipOnErrorTurn), and reasoning
on a turn that called an Edit or Write tool (protectMutationToolCalls). The first cut is sticky.
Claude Fable 5.1 and later validate replayed reasoning against the conversation it was produced
from. On those models thinking_trim only ever removes reasoning from the start of the history, and
holds any other trim, so a later block never loses the context it was written against. With the
default keepLoopHead of 1 the opening plan stays, which leaves nothing in front of it to remove,
so the strategy usually makes no change there.
[assistant] (thinking: plan, 1.2k chars) (tool_use run_tests)
[tool] 128 passed
[assistant] (thinking: step, 800 chars) (tool_use read_file)
[tool] contents…
[assistant] (thinking: plan, 1.2k chars) (tool_use run_tests)
[tool] 128 passed
[assistant] (tool_use read_file)
[tool] contents…
window_budget
When a session's history outgrows the model's context window, window_budget evicts middle messages
and replaces each with a retrievable placeholder. It scores messages by relevance to the live turn
plus recency. Task setup (keepLeading), the live turn (keepRecent), and pinned roles are never
evicted, and every evicted original comes back through POST /v1/retrieve.
The input budget subtracts requested output and a 5% estimator-uncertainty reserve from the model
window, then honors any narrower trusted client ceiling. Thinking already included in the output
limit is not counted twice. Unknown capacity means no automatic crop; maxTokens can set a fixed
ceiling instead (maxTokens: 0 disables cropping). The latest completed exchange and unresolved
work stay protected unless keepRecent explicitly selects the older message-count policy. It is the one strategy that drops whole messages, and
settled crops replay identically, so they never re-bust the provider cache.
[user] task setup (kept: leading)
[tool] 12k-token exploration dump (low relevance → evicted)
[tool] shorter, on-topic result (kept: relevant)
[user] the live question (kept: recent)
[user] task setup
[tool] [anyray: 12k tokens cropped · retrieve ctx_…]
[tool] shorter, on-topic result
[user] the live question
cache_optimizer
Runs last and keeps the provider's cache discount alive. It puts the tools block in a
byte-identical order every turn. On Anthropic it inserts cache breakpoints for a cached-read
discount. On selected OpenAI models it stamps a deterministic prompt_cache_key when the client set
none.
It is lossless. It only reorders tools and adds cache hints, and it never touches prompt content.
It backs off when the client already manages caching. The conversation anchor follows append-only
batches of complete exchanges, sized by the provider's cache minimum. Missing parallel results
prevent settlement. Explicit conversationStableTurns and anchorQuantumMessages retain the older
count-based policy; existing EDIT pins and valid legacy KEEP watermarks survive the transition.
turn 1 tools: [search, book_hotel, get_weather]
turn 2 tools: [get_weather, search, book_hotel] ← order drifts
turn 3 tools: [book_hotel, search, get_weather]
→ prefix changes → provider cache misses → full price
every turn tools: [book_hotel, get_weather, search] (fixed)
+ cache_control on the last tool, system, and settled history
→ prefix identical → provider cache hits → cached-read rate