Guardrail checks
Guardrails are the checks a routing config runs on each request and its answer.
Which guardrails run
The guardrails that run are those of the config that routes the request. A request's
x-anyray-config, its x-anyray-provider, or the signing person's or agent's own routing replaces the org
config, guardrails included: the org config's guardrails do not run on that request. On a
passthrough request, x-anyray-provider only names the provider the caller's key belongs to, so
it replaces nothing (pass-through traffic).
In a config, a target's hooks and guardrails run in addition to the ones it inherits, never instead of them, so a fallback, load-balance or conditional target cannot skip a rule set above it.
What checks read
| Request | Input checks read | Output checks read |
|---|---|---|
| Chat completions, messages | The latest turn | The answer's text |
Responses (/v1/responses) | The latest turn | The answer's output_text and refusals |
| Completions | prompt | The answer's text |
| Embeddings | input | Nothing to check |
| Token counts | The latest turn | Nothing to check |
| Image generations and edits | prompt | Refused, as below |
The latest turn is every message after the last assistant message, plus an assistant prefill that
ends the request. That covers text blocks, tool results, text documents, search results, OpenAI
tool messages, and a system message that follows the user's. On the Responses API it is what the
client added after the model's last response (its reasoning, message and tool calls): tool outputs
(function_call_output, shell stdout and stderr, and the rest) and new messages, plus the
prompt template's variables. The system prompt, instructions and earlier turns are not read,
except by a redaction (see Redaction). Partner guardrails get the same text, tool
output included.
Any other body is one the checks cannot read: a file upload, audio, a batch, a request through the
/v1/* proxy, or an image, a file or any answer that is not JSON. Under a guardrail with deny,
or a check that can rewrite it, it gets a 446 and is not sent on. A
GET or DELETE sends no body, so input checks have nothing to read on it. Checks that read only
the request type, the metadata or a header (allowedRequestTypes, requiredMetadataKeys, jwt)
run on every request, and a hook made only of them never refuses a body for being unreadable.
Input checks judge the request as the client sent it. Optimizing an allowed request never changes
their result: a wordCount minimum the client's text meets still passes when the optimized text is
shorter, and a maximum it exceeds still refuses the request, however short the optimized text would be.
On chat completions, completions, messages and the Responses API, the first target's input checks
run before the optimizer. The optimizer sees only what they let through, and a refused request is
never optimized. The first attempt reuses their result. If a check rewrote a value the target's
override_params sets, that attempt runs the checks again. A fallback target runs its own input
checks when the request reaches it.
When a check fails
A check that throws fails its hook. With deny, or a check that can rewrite, the request gets a
446; otherwise the check is recorded as failed and the request goes on.
These checks can rewrite what they read: regexReplace, addPrefix, webhook, anyray.pii,
azure.pii, bedrock.guard, pangea.pii, promptfoo.pii, patronus.pii, patronus.phi,
f5-guardrails.scan, acuvity.scan, crowdstrike-aidr.guardChatCompletions and exa.online.
Every other check only passes or fails.
A check that finds no text to read (an answer that only calls tools, an empty prompt) passes with no
error, whatever failOnError says. An answer whose only text is a refusal, or whose text sits past
its first choice, does not count as having no text. The checks do not read that text, so such an
answer gets the same result it always did.
A check that returns an error (a partner it cannot reach, missing credentials) fails if the check
sets failOnError: true. A guardrail with deny then blocks the request with a 446, and the
check's error in hook_results reads the check returned an error. Without failOnError, the
error is recorded and the request goes on.
With failOnError, a partner outage blocks requests until the partner is back. Without it, a
request can slip past: one that makes the partner call time out causes the error itself.
Under failOnError, a check that can rewrite blocks with a 446 when it returns an error, even in
a mutator or a guardrail without deny. An async hook runs too late to block. Any other check in
a hook without deny records the failure, and the request or answer goes on.
failOnError takes true or false. Any other value (the string "false", 1) gets a 400, in
an x-anyray-config, in the default-guardrail headers and when an admin saves the routing config.
A config saved before this check that holds such a value still loads, and the value counts as
false.
A config saved before these limits still loads. Each rule runs on as much text as it can finish in
time: about 100,000 characters for a rule saved under these limits, with the turn's text blocks
counted together. Past that, when a text block is not a string (a token-id prompt, say), or when a
jsonSchema check can't validate its JSON in time (nested too deeply, an array too long for
uniqueItems, more JSON than the request's validation work allows, or a saved schema these limits
now refuse), a guardrail with deny blocks the request with a 446. A regexReplace always does,
and so does one that fails to run, so unchecked or unredacted text never goes through. A guardrail
without deny records the check as failed.
The hook_results the gateway adds to a response list each hook's id, type, verdict, deny,
transformed and skipped, and each check's id, verdict and an error with a fixed reason.
They never include the text or JSON a check matched, or its rule.
Regex rules
A regexMatch or regexReplace rule has the limits of a $regex condition
(regex limits), and a client
x-anyray-config may carry one. The flags are g, i and s, each at most once:
| Check | Rule | Flags |
|---|---|---|
regexMatch | the pattern as written (/api/v1/ matches the slashes) | a flags parameter: { "rule": "secret", "flags": "i" } |
regexReplace | /pattern/flags, or a bare pattern; redactText up to 1,024 characters. Replaces every match, with or without g | in the rule: /secret/i (a flags parameter gets a 400) |
Each rule is judged on a 100,000-character prompt, so one that slows on long text (a.*b,
grok.*fast) gets a 400 on the request and is refused on save, while \d{3}-\d{2}-\d{4}, \d+
and sk-[A-Za-z0-9]{48} pass. A config may carry at most 32 regexMatch and regexReplace
checks. The x-anyray-default-input-guardrails and x-anyray-default-output-guardrails headers
count toward the 32 of the x-anyray-config sent with them, and get 32 of their own without one.
One request's regex checks share a fixed amount of matching work, across every fallback or
load-balance target that reruns them and every config they come from. A request that would go past
it gets a 446 and is not sent on, so text never goes through unchecked or unredacted. The same
446 stops a regexReplace whose redactText would grow the text by more than its own length
plus 64 KiB.
A jsonSchema guardrail's pattern and patternProperties keys have the same limits, read with
the u flag the validator uses, and share the same matching work. A schema can't use the url
format (it can take too long on long text): use uri. A schema that could apply too many
subschemas to one JSON value (allOf of two $refs to the next level, repeated), or is too large
to load, gets a 400 and is refused on save. A recursive schema, such as a tree node whose
children refer back to it, is fine. One request's jsonSchema checks also share a fixed amount
of validation work.
Redaction
A regexReplace redaction covers more than the latest turn: the system prompt and every message (on
the Responses API, instructions and every input item), rewritten the same way every turn. A client
sends its own unredacted copy of the history back each turn, so this keeps a secret out of later
requests, and it keeps the bytes the provider caches identical turn to turn. Signed thinking blocks
and tool-call inputs are left as they are; an OpenAI refusal is redacted like any text. A redacted
answer loses the log probabilities of the text that changed. A redaction writes its result back to
the places it read, an empty result included, and a failOnDetection one fails only on a match in
the latest turn.
Redactions add up, across hooks and whether or not a hook is sequential. A regexReplace that
read text another redaction has since changed runs again on the new text. A partner redaction is
never called twice: when it lands on changed text, the regexReplace redactions run again on its
result, and the request gets a 446 if one of them can't. An embeddings request is checked and
redacted too, its inputs counted as one text (a card-number rule reaches about 106,000 characters
across the batch).
For a request from a verified user, each gateway replica remembers where a rule matched each text
(never the text), for that user only and until it goes unused for an hour, so a session pays for
its history's regex work once. A long session that reaches a replica that has not seen it is checked in full
there and can pass the rule's limit: that request gets a 446, but the texts that fit are
remembered, so sending it again gets further. A request with no verified user is checked in full
every time.
A partner redaction (anyray.pii, azure.pii, pangea.pii, promptfoo.pii, patronus.pii,
patronus.phi, bedrock.guard, f5-guardrails.scan and acuvity.scan, with redaction on) covers
the same places for a request from a verified user. The latest turn goes to the partner, and only it
decides the check's verdict. Each replica remembers, for that user and until it goes
unused for an hour, where the partner changed each text and what it put there, never the text
itself. A hash or format-preserving replacement is kept as the partner returned it, which is what
the provider gets anyway. History the replica has seen is rewritten the same way with no partner
call. History it has not seen goes to the partner in separate calls, oldest first: for each check,
at most 512 texts, 2,000,000 characters and 10 seconds a request, with 8 calls at a time.
Past those limits, or when the partner fails on the history, the check returns an error. With
failOnError, the request gets a 446 and the next one picks up where it stopped, so a long
session on a replica that has not seen it is refused until enough of its history is remembered. A
1,500-text session takes 3 requests when each partner call takes 120 ms, and 21 at 1.2 seconds for a
partner that checks one text a call; after that, the session is not refused again on that replica
while it stays in use. Without failOnError, history the partner did not check goes on as the
client sent it. A request with no verified user has only its latest turn redacted: its history
reaches the provider as the client sent it, and each redacted turn costs one prompt-cache miss when
it becomes history. A crowdstrike-aidr check already sends the whole request and replaces it with
the partner's answer.
Azure's PII service takes at most 5 texts of 5,120 characters a call, counting an emoji as one. A longer text is cut at its last newline before the limit, else its last space, else its last punctuation mark, else at the limit, and each piece is checked; an entity that spans a cut, such as a two-word name, can be missed. A text counts as checked only when every piece comes back. A latest turn of more than 512 pieces is not checked.
Azure and Bedrock checks
An Azure check (azure.contentSafety, azure.pii, azure.shieldPrompt, azure.protectedMaterial)
in a request's x-anyray-config or default-guardrails header runs on the caller's own API key or
Entra client credentials. One that sets azureAuthMode to managed or workload gets a 400,
because those modes use the gateway's own Azure identity. In the org routing config, managed
works (workload identity is not supported by these checks). The gateway mints that identity's token
for Azure AI services only and sends it only to an Azure AI services host
(*.openai.azure.com, *.cognitiveservices.azure.com or *.services.ai.azure.com). A save whose check
names another customHost, or a resourceName that moves the host off Azure, gets a 400, and the
check refuses it with an error before fetching the token. The gateway verifies the TLS certificate of
every Azure host, custom or not.
A bedrock.guard check reads awsAccessKeyId, awsSecretAccessKey, awsRegion and an optional
awsSessionToken from its credentials, or awsAuthType: assumedRole with awsRoleArn and
awsExternalId. A check saved with the older names accessKeyId, accessKeySecret and region
works too. When a check has both, the aws name wins.
A caller's bedrock.guard check with awsAuthType: assumedRole gets a 400: it would assume the
role with the gateway's own AWS identity. Send the caller's own AWS access key pair.
Streamed answers
On chat completions, completions, messages and the Responses API, a streamed response is held until
it ends and checked whole when an output guardrail has deny or an output check can rewrite the
answer. A denial is the same 446 JSON a non-streamed one returns, and a redacted answer is sent as a stream carrying the redacted text. A
held stream the gateway cannot read, or cannot rebuild after a redaction, gets a 446, and so
does a Responses stream with an event whose text the checks do not read. A held stream past 64 MB
gets a 446 too, and the gateway stops reading it. When a held stream ends before the checks get
an answer to read, the 446 error type says why: upstream_stream_failed when the upstream broke
it, held_stream_cut when the gateway stopped reading (the size cap, or a client that left). On
chat completions and messages, a stream the upstream drops after the answer began stays a 200,
held or not, and ends with one error event in the request's format, with a fixed message. Other
streams are sent as they arrive. A single stream event longer than 128 MB (as many characters, on a
text stream), held or not, ends the stream before that event, the way a dropped upstream connection
ends it, and the gateway stops reading it. The spend row of a refused answer carries what the
provider billed for it, and the refusal adds no savings. When a fallback or retry then moves on, the
row adds that cost, at the refused target's price, to the answer's.
Codex gets a guardrail denial and a held_stream_cut as a 400 with the same body, on an API key
and on a subscription seat. Codex retries a 446 up to five times, and each retry is a new answer
the provider bills and the gateway refuses again; a 400 it shows at once. An upstream_stream_failed stays a 446, so Codex retries
a broken upstream. To tell a denial apart, the gateway reads at most 64 KB of a 446 body, for at
most two seconds; a longer or slower body reaches Codex as sent.
Pass-through traffic
A passthrough request keeps its own token and provider and is never routed through the config:
a Claude Code, Codex or Copilot seat, or any request sent with x-anyray-auth-mode: passthrough
and the caller's own key. It runs the config's guardrails and nothing else of it (no key, retry,
cache or override): the hooks the org default config, or the signing agent's own routing, gives
that request. A caller's x-anyray-config replaces the org's, guardrails included.
| Check | On pass-through |
|---|---|
| Input deny | 446; the request never reaches the upstream |
| Input redaction | The request is rewritten before it leaves, retries included |
| Output deny or redaction | A successful answer is held until it ends and checked whole, as on API-key traffic, then sent, sent redacted, or refused with a 446. An upstream error is sent as it came |
| A body the checks cannot read | 446; on Copilot, also a request to a path that is not chat completions, Messages or Responses while an output deny or redaction applies |
As on an API key, Codex gets a guardrail denial or a held_stream_cut as a 400.
When the config's router picks no target for a seat request (no condition matches and there is no
default, or every load-balance weight is 0), the seat runs the guardrails set on that router and
above it, and none of its targets'. With none there, the request goes as if no guardrail were set.
A route that fails for another reason (a condition or the default names a target that doesn't
exist, say) still gets a 446, as it fails on API-key traffic.
Every check runs, so a webhook, log or partner check in the config gets the request's text, as
it does for API-key traffic. With no guardrails, pass-through traffic is forwarded with no added work.
On Claude Code and Codex, a caller's x-anyray-config that breaks a config rule gets the same
400 as on any other route. Copilot reads a caller's config only for its guardrails, so only one
that carries guardrails gets that 400; one without them is ignored.
Checks that only record (a log hook, a guardrail without deny) run on the answer as they do for
API-key traffic: a streamed answer is not held for them, and a failed one turns a JSON answer into
a 246. They don't run on Codex's WebSocket.
Codex's WebSocket cannot hold an answer, so when an output deny or redaction applies, the gateway
answers the upgrade with 426 and Codex sends the session over HTTPS instead. One added
while a socket is open closes it once the turn in flight ends, and Codex reopens it over HTTPS. On
the socket an input denial reaches Codex as a 400 error event, as over HTTPS. While the config
has a deny or redaction anywhere, a socket frame that is neither a request nor Codex's interrupt of
its own response is refused the same way, since the checks cannot read it.