Use cases
The workloads that waste the most tokens — and the strategy that cuts each.
Logs, JSON dumps, test runs, diffs, code search, agent sessions, tool schemas.
Working with dataAnalysts, BI, finance ops — exports, query results, pipeline runs, recurring reports.
Knowledge & operationsHR, ops, support, research & writing — policies, handbooks, repeat questions, recalled notes.
Agent automationsMulti-step pipelines that fetch, process, and write back across your tools and data.
Engineering
| Workload | What you ask | Strategy | Reduction |
|---|---|---|---|
| Access log | "find the failing 5xx requests" | relevance_filter | ~99% |
| Incident debugging | "why did p99 spike at 10:05?" | relevance_filter | ~98% |
| GitHub issues | "which open issues are P0 auth bugs?" | relevance_filter | ~83% |
| JSON dump | "which of these 500 orders failed?" | context_compression | ~97% |
| Test-suite output | "which tests failed and why?" | command_digest | ~77% |
| Code search | "where is the retry policy configured?" | relevance_filter | ~72% |
| Git diff review | "any change that weakens an auth check?" | relevance_filter | ~72% |
| Codebase exploration | "map the architecture; where do retries live?" | code_graph | ~15% |
| Multi-file code trace | "how does submitOrder capture a payment?" | code_graph | ~33% |
| Pasted screenshot | "what is this error in my screenshot?" | vision_ocr | ~84% |
| Long agent session | "given all the above, where should X live?" | window_budget keeps relevant older turns | ~72% |
| MCP tool-schema bloat | any request from a tool-loaded assistant | tool_pruning | ~91% |
| Verbose tool descriptions | a tool-loaded assistant on cached traffic | tool_schema_compression | ~8% lossless |
| Catch me up on a branch | "where does the payments-retry branch stand?" | relevance_filter | ~86% |
| Decision recall | "what did we decide about the job queue, and why?" | relevance_filter | ~74% |
Access log
You paste a few hundred lines of an nginx access log and ask the model to find the
failures. Almost every line is 2xx/3xx noise; the answer lives in the handful of 5xx
lines in the incident window.
Prompt: "Checkout is failing intermittently in production. Here is the nginx access log. Find the failing requests: which endpoint and method are returning 5xx, how many, the time window they happened in, and the client IPs — so I can narrow it down." — followed by ~500 log lines.
relevance_filter scores every line against the question — by keyword
match, no extra model call — and keeps the top lines plus a few for context; the rest collapse to a
placeholder it can expand back if needed. The 5xx POST /api/checkout lines survive; the
200 OK noise doesn't.
Incident debugging
During an incident you dump the merged logs, metrics, and traces for the affected window and ask what caused the spike. The spike is a tiny slice of a large window.
Prompt: "Checkout p99 latency spiked around 10:05 and we got paged. Here are the merged logs, metrics and traces for the window. Find what caused the spike."
relevance_filter keeps the lines that match the incident question (the error logs, the
slow spans, the saturated-pool metrics) and drops the steady-state noise around them.
GitHub issues
You feed in dozens of open issues with their comment threads and ask which are urgent. Most issues are low-priority chatter; a few are the real fires.
Prompt: "Here are our 60 open issues with their comment threads. Which ones are P0 authentication bugs that should be fixed this week? List the issue numbers."
relevance_filter surfaces the issues whose text matches the question and drops the
rest. (If your issues arrive as raw API JSON instead of text, this becomes a
context_compression job.)
JSON dump
You paste the output of an API call or kubectl get … -o json and ask one question about
it. Pretty-printed JSON is mostly structural whitespace and repeated keys.
Prompt: "Here is the JSON from
GET /api/orders?limit=500. Which orders have status 'failed', and what is their retry count? Ignore the settled ones."
context_compression shrinks the JSON — stripping the whitespace,
repeated keys, and over-long lists — without reading what any of it means, and keeps the original so
nothing is lost.
Test-suite output
The most common loop in agentic coding: run the suite, read the wall of output, fix, repeat. A
user (or a coding agent) pastes the full pytest/jest/go test/cargo test run — mostly
passing tests, a session banner, plugin lists — and asks what broke. The answer is the handful of
failing blocks and the count line.
Prompt: "I ran our test suite and a few tests are failing. Here is the full pytest output. Which tests failed, and what is the root cause of each? Ignore the ones that passed." — followed by an 80-test run.
command_digest spots a test run by its shape (not from the
question), keeps each failure's error and traceback plus the 3 failed, 77 passed summary, and drops
the banner and every passing line — and can restore them. Because it works off the structure, not the
question, it keeps cutting even on cached traffic. On by default.
Code search
You run grep/ripgrep or a semantic search, get a hundred hits, and ask which one is the
real match. Most hits are incidental keyword matches.
Prompt: "I ran
rg -n -C1 retryand got 100 hits across the codebase. Where is the retry policy / max-attempts actually defined (not just used)? Point me to the file and line."
relevance_filter ranks the matches against the question and keeps the ones that
actually answer it — the policy definition, not the incidental usages and comments.
Git diff review
You ask the model to review a sizable PR diff for one specific risk. Most of the changed files are unrelated to the question.
Prompt: "Review this PR diff. Is there any change that weakens an auth/permission check? Point to the exact file and line."
relevance_filter keeps the part of the diff that matches the review question (the weakened
requireAdmin check) and drops the unrelated file changes, so the reviewer pays only for the
diff that matters.
Codebase exploration
A coding agent reads several whole source files back (as tool results) to understand the architecture, but only needs the shape and the parts on the path the question asks about.
Prompt: "Map the architecture of these services and tell me where retry logic is handled." — after the agent has read the core source files back as tool messages.
code_graph maps how the files reference each other, keeps the bodies
the question targets (and their close collaborators) at full fidelity, and elides the rest —
reversibly. A broad "map everything" prompt keeps more bodies than a targeted trace, so it saves
less; savings depend on how much of the codebase is off the question's path.
Multi-file code trace
A coding agent traces one flow across a service — every read_file it runs piles another
whole file into the context window, but only a few of those files are on the path the
question asks about.
Prompt: "How does
Checkout.submitOrderend up capturing a payment? Trace the call path and tell me where a failed capture is retried." — after the agent has read 11 source files back as separate tool messages.
code_graph maps how the files reference each other, finds the symbols
the question actually touches (submitOrder → capturePayment → sendCharge, and their
collaborators), keeps those bodies whole, and trims the 7 off-path files — reversibly. It's the
code path for source reads; with no targeted symbol it outlines the whole file.
Pasted screenshot
A user pastes a screenshot of a terminal stack trace, an error dialog, or log output instead of the text. A high-detail image input costs hundreds-to-thousands of provider vision tokens — far more than the equivalent text.
Prompt: "My Node service crashes on checkout. Here is a screenshot of the stack trace from the terminal — what is the bug and how do I fix it?" — with a PNG attached.
vision_ocr runs a cheap local OCR pass and, only when the image is
clearly text, swaps in the extracted text before the request reaches the model — for a fraction of the
vision-token cost. It is gated and off by default: anything that reads as a diagram, chart, UI, or
photo passes through untouched. The swap is reversible — the original image is kept.
Long agent session
A long coding session accumulates dozens of stale tool turns. Every new step re-sends the entire history, so the oldest, least-relevant turns are paid for over and over.
Prompt (turn N of a long session): "Given everything above, where should the shared retry logic live? Be specific."
window_budget trims low-scoring middle turns to fit a token limit,
keeping the system prompt, the task setup, the most recent turns, and older turns that still match the
live question — and saving what it trims so it can be restored.
MCP tool-schema bloat
Your users connected MCP servers for GitHub, Slack, Jira, a database, and a browser — genuinely useful, but now every request ships all 41 tool schemas, used or not. Industry numbers put schema overhead at roughly 1k tokens per tool.
Prompt: "Move ticket PLAT-281 to In Review and assign it to me." — a request that needs two Jira tools, sent with 41 schemas attached.
tool_pruning matches each tool's name and description against the
conversation and drops the schemas the task neither names nor shares a domain term with — erring
toward keeping when in doubt — before the request reaches the provider.
Verbose tool descriptions
tool_pruning drops the schemas a request won't use — but the tools it keeps still carry long,
sometimes multi-paragraph descriptions. And on prompt-cached traffic (every billing mode)
tool_pruning is held back entirely: dropping a tool from one turn to the next would break the cache.
Prompt: "Summarize the open incidents and assign the database one to me." — sent on a warm, cached turn with ten verbose MCP tool schemas attached.
tool_schema_compression is the always-on complement. It rewrites only
the free-text descriptions in each tool — collapsing extra whitespace and cutting boilerplate
openings like "This tool allows you to…" — while keeping every tool and leaving the actual schema
(types, enums, required fields) untouched. Because it only ever looks at each tool's own text, the
tools block comes out the same every turn, so it keeps working even on cached traffic, where
tool_pruning can't.
On ten verbose MCP schemas it trims ~8% of the request with no loss; the opt-in
maxDescriptionChars cap takes that to ~32% by cutting over-long descriptions at a sentence
boundary (falling back to a word boundary).
Catch me up on a branch
You return to a branch after weeks away. An agent recalls your past sessions — hundreds of summaries across every branch you have touched — and you ask where this one stands. Only a few of those sessions are about it.
Prompt: "Catch me up on the payments-retry branch: where does it stand, what changed and why, and what was the next step I had planned?" — answered against ~200 recalled session summaries.
relevance_filter ranks each recalled record against the question and keeps the handful about
that branch — its state, the decision behind it, the planned next step — dropping the rest. It
ignores the boilerplate every record shares and locks onto the one rare branch name. The recalled
history can be restored if needed.
Decision recall
You are about to re-open a past decision. The agent recalls your decision log and you ask what you chose and why. One record holds the answer.
Prompt: "We are revisiting our background job/task queue. What did we decide for it, and why — and which alternatives did we rule out?" — answered against a recalled decision log.
relevance_filter keeps the one on-topic decision — the choice, the reasoning, and the
alternatives that were ruled out — and drops the unrelated ones. This is the smallest store
here, so the saving is modest; it grows as the log does.
Working with data
| Workload | What you ask | Strategy | Reduction |
|---|---|---|---|
| Spreadsheet export | "which orders are EMEA refunds?" | relevance_filter | ~90% |
| Database schema for text-to-SQL | "write the query for revenue by region" | relevance_filter | ~88% |
| Query result set | "which accounts are past due?" | context_compression | ~96% |
| Data-pipeline run | "which models failed, and why?" | command_digest | ~75% |
| Recurring report | nightly summary over N datasets | prompt_compression | ~80% |
Spreadsheet export
An analyst pastes a large CSV export — a sales transactions table, a subscriber list, an expense report — and asks one question about a subset of the rows. Most rows are noise; the answer lives in the handful that match.
Prompt: "Here is our quarterly sales export (about 8,000 rows: order date, rep, region, product, amount, status). Which orders are refunds in the EMEA region? I only need those — ignore the completed ones." — followed by the full CSV.
relevance_filter scores every row against the question (the same
ranking as the Access log row) and keeps the top rows plus the header for context; the
rest collapse to a placeholder it can expand back if needed. The refunded EMEA rows survive; the
thousands of completed orders don't.
Database schema for text-to-SQL
A "chat with your data" tool or text-to-SQL agent ships the entire database schema —
hundreds of CREATE TABLE definitions, every column and type — on every question, just so
the model can write one query. Any single question touches a handful of tables; the rest of
the schema is dead weight billed on each turn.
Prompt: "Using this schema, write the SQL for monthly revenue by region for the last quarter." — preceded by the full 200-table schema dump.
relevance_filter scores each table against the question (the same
ranking as the Spreadsheet export row) and keeps the few the query needs —
orders, regions, their keys — dropping the rest. The dropped tables stay retrievable, so the
model can pull one back if it needs it.
Query result set
You paste the output of a SQL query or a BI/analytics API call — a few hundred rows of JSON — and ask one question about it. Pretty-printed result sets are mostly structural whitespace and repeated keys.
Prompt: "Here is the JSON result of our accounts-receivable query (500 rows). Which accounts are past due, and by how many days? Ignore the ones in good standing."
context_compression does the same job as the engineering
JSON dump row — shrinking the JSON and keeping the original so it can be restored. If
your result arrives as a text table rather than JSON, this becomes a
relevance_filter job instead.
Data-pipeline run
A data engineer or analytics-engineering agent runs a transformation pipeline — dbt,
Airflow, an ETL job — and pastes the full run output to ask what broke. The output is mostly
per-model PASS lines, a config banner, and timing noise; the answer is the few failing
models and the summary count.
Prompt: "I ran our nightly dbt build and a few models failed. Here is the full run output. Which models failed, and what is the root cause of each? Ignore the ones that passed." — followed by a 120-model run.
command_digest is the same recognizer behind the engineering
Test-suite output row — it keeps each failure's error block and the
3 failed, 117 passed summary line and drops the banner and every passing line, reversibly.
On by default.
Recurring report
A scheduled job re-appends the same instruction block — metric definitions, the output format, the tone — before each of N datasets, every night, so the instructions bill N times.
Prompt: the same six-sentence "summarize this dataset against these KPI definitions, in this format" template pasted before each of 30 regional datasets.
prompt_compression removes the repeated sentences and trims the
filler; the first copy stays and the model's output is unchanged. The saving grows with how often the
job runs and how many datasets it covers.
Knowledge & operations
| Workload | What you ask | Strategy | Reduction |
|---|---|---|---|
| Policy lookup | "what's the parental-leave policy?" | relevance_filter | ~80% |
| RAG over-retrieval | "what does the refund policy say about X?" | relevance_filter | ~67% |
| Repeat question | the same support question, asked again | semantic_cache | provider call skipped |
| Templated batch prompt | nightly classification over N tickets | prompt_compression | ~82% |
| Research brief | "everything I found on vector DBs, in one brief" | relevance_filter | ~85% |
| Content memory | "what have I published on onboarding?" | relevance_filter | ~81% |
| Open loops | "what is still open and waiting on me?" | relevance_filter | ~79% |
Policy lookup
An HR or operations assistant answers an employee question against a long internal document — the employee handbook, a benefits summary, an SOP — pasted whole into the prompt. The answer is one section.
Prompt: "Here is our employee handbook. What is the parental-leave policy — how many weeks, paid or unpaid, and what's the notice requirement?" — followed by the full handbook.
relevance_filter keeps the sections whose text matches the question
and drops the rest. (If your handbook is retrieved as chunks from a knowledge base rather than
pasted whole, this is the same mechanic pointed at the retrieved set — the
RAG over-retrieval case.)
RAG over-retrieval
A support copilot retrieves the top-20 chunks by embedding similarity for every customer question and stuffs them all into the prompt. Two contain the answer; most pipelines over-fetch 3–5×.
Prompt: "What does the refund policy say about annual plans cancelled mid-term?" — with 20 retrieved chunks attached.
relevance_filter re-ranks the chunks against the live question on the
gateway and keeps only what matters.
Repeat question
High-volume support and FAQ traffic asks the same questions in different words, and every rephrasing pays for a fresh provider call.
Prompt: "How many days do I have to return something?" — semantically equal to a prior "What's your refund window?".
semantic_cache spots the repeat (when it's close enough to a previous
question) and serves the stored answer — the provider is never called, so the call is ~free and
faster. It is default-on.
Templated batch prompt
A nightly batch script re-appends the same six-sentence instruction block before each of 40 tickets — billing the instructions 40 times every night, and nobody reviews the prompts a cron job sends.
Prompt: the same classification template pasted before each of 40 support tickets.
The same prompt_compression job as the data
Recurring report row: the repeated sentences are removed, only the first copy
stays, and the model's output is unchanged.
Research brief
Your research notes pile up across many topics. The agent recalls them all and you ask for a brief on one topic, plus which sources you already vetted.
Prompt: "Pull together everything I found on vector-database options for our RAG — the findings and which sources I had already vetted (and why) — into one brief." — answered against weeks of recalled notes.
relevance_filter keeps the on-topic notes and their source tags — so "which sources did I already
vet?" is answered from the notes themselves — and drops everything off-topic, reversibly.
Content memory
You are about to write another piece on a theme you have covered before. The agent recalls your back catalogue so the new piece builds on prior work instead of repeating it.
Prompt: "I am about to write on developer onboarding. What have I already published on onboarding, so the next piece builds on it instead of repeating it?" — answered against a recalled back catalogue.
relevance_filter keeps the pieces on that theme — their angles and takes — and drops the rest.
The same recall mechanic as the engineering rows, pointed at a content library.
Open loops
Across many projects, most of the week is already shipped; a few items are blocked on you. The agent recalls the week's activity and you ask what is still open.
Prompt: "Across all my projects, what is still open — blocked or waiting on my decision or input? Leave out anything already finished." — answered against the week's recalled activity.
relevance_filter keeps the open/blocked items waiting on a decision and drops the finished work.
It sorts by wording: blocked, waiting, open, decision rank the loose ends, while
shipped/merged do not.
Agent automations
These are multi-step pipelines — fetch → process → write back — that call the model once per item or per source, each call dragging the full fetched payload into context. Anyray sits on every call and cuts the token cost of those bulky data legs; it does not parallelize or remove steps — that orchestration lives in your agent runtime — so the saving here is per-call cost, not wall-clock.
| Workload | What you ask | Strategy | Reduction |
|---|---|---|---|
| Email filtering | "which are assigned to sarah@company.com?" | context_compression | ~95% |
| Article summary pipeline | summarize each of 20 fetched articles | context_compression | ~70% |
| Customer data enrichment | enrich 150 records, write back | context_compression | ~93% |
| Multi-source aggregation | weekly report from three source dumps | relevance_filter | ~88% |
Email filtering by assignee
An agent pulls a batch of support emails through a mail or helpdesk tool and is asked to return only the ones routed to a single person. The tool returns all 200; the answer is the dozen assigned to that address.
Prompt: "Filter these 200 support emails and return only the ones assigned to
sarah@company.com— sender, subject, and received date." — followed by the full tool result.
When the emails come back as JSON (the usual tool shape),
context_compression shrinks the structure and trims the list, keeping
the original so it can be restored — the model still does the assignee match, on a fraction of the
bytes. If the tool returns a text list instead, this becomes a
relevance_filter job that ranks each line against sarah@company.com and
keeps only the matches.
Article summary pipeline
A content or marketing automation fetches the latest N articles, summarizes each, and posts the digest to Slack. Each article's full body — often raw HTML or boilerplate-heavy text — is fed to the model just to produce a few sentences.
Prompt: "Summarize each of these 20 tech articles in three sentences for our Slack digest." — one fetched article body per call.
context_compression strips the markup and repeated structure from
each fetched body before it reaches the model, and semantic_cache
skips re-summarizing an article the pipeline has already seen on an earlier run. Anyray cuts the
per-article token cost; the fan-out across the 20 articles stays in your runtime.
Customer data enrichment
A RevOps agent fetches a batch of CRM records, enriches each against a third-party provider (Clearbit, Apollo), and writes the result back. Both the record batch and each enrichment response are bulky JSON the model only skims.
Prompt: "For each of these 150 customer records, attach the firmographic fields from the enrichment response and flag any with no match."
context_compression shrinks the record batch and each enrichment
payload, and semantic_cache skips re-enriching a company already
looked up recently. The token cost of every leg drops; the per-record loop itself runs in your
runtime.
Multi-source data aggregation
A scheduled agent pulls a week of data from several systems — sales from Stripe, tickets from Zendesk, activity from Mixpanel — and rolls them into one report. Each source returns a large dump, but the report needs only a few metrics from each.
Prompt: "From these three exports — Stripe charges, Zendesk tickets, and Mixpanel events for the week — produce the weekly summary: revenue, ticket volume and top issues, and active users." — followed by all three payloads.
relevance_filter ranks each source dump against the report's
questions and keeps only the lines that feed a metric; context_compression
shrinks whatever arrives as JSON; and window_budget caps the merged
context if the three sources together overrun the window. The report is built from the slices that
matter, not three full exports.