Skip to main content

Guardrail checks

Guardrails are the checks a routing config runs on each request and its answer.

Which guardrails run​

The guardrails that run are those of the config that routes the request. A request's x-anyray-config, its x-anyray-provider, or the signing person's or agent's own routing replaces the org config, guardrails included: the org config's guardrails do not run on that request. On a passthrough request, x-anyray-provider only names the provider the caller's key belongs to, so it replaces nothing (pass-through traffic).

In a config, a target's hooks and guardrails run in addition to the ones it inherits, never instead of them, so a fallback, load-balance or conditional target cannot skip a rule set above it.

What checks read​

RequestInput checks readOutput checks read
Chat completions, messagesThe latest turnThe answer's text
Responses (/v1/responses)The latest turnThe answer's output_text and refusals
CompletionspromptThe answer's text
EmbeddingsinputNothing to check
Token countsThe latest turnNothing to check
Image generations and editspromptRefused, as below

The latest turn is every message after the last assistant message, plus an assistant prefill that ends the request. That covers text blocks, tool results, text documents, search results, OpenAI tool messages, and a system message that follows the user's. On the Responses API it is what the client added after the model's last response (its reasoning, message and tool calls): tool outputs (function_call_output, shell stdout and stderr, and the rest) and new messages, plus the prompt template's variables. The system prompt, instructions and earlier turns are not read, except by a redaction (see Redaction). Partner guardrails get the same text, tool output included.

Any other body is one the checks cannot read: a file upload, audio, a batch, a request through the /v1/* proxy, or an image, a file or any answer that is not JSON. Under a guardrail with deny, or a check that can rewrite it, it gets a 446 and is not sent on. A GET or DELETE sends no body, so input checks have nothing to read on it. Checks that read only the request type, the metadata or a header (allowedRequestTypes, requiredMetadataKeys, jwt) run on every request, and a hook made only of them never refuses a body for being unreadable.

Input checks judge the request as the client sent it. Optimizing an allowed request never changes their result: a wordCount minimum the client's text meets still passes when the optimized text is shorter, and a maximum it exceeds still refuses the request, however short the optimized text would be.

On chat completions, completions, messages and the Responses API, the first target's input checks run before the optimizer. The optimizer sees only what they let through, and a refused request is never optimized. The first attempt reuses their result. If a check rewrote a value the target's override_params sets, that attempt runs the checks again. A fallback target runs its own input checks when the request reaches it.

When a check fails​

A check that throws fails its hook. With deny, or a check that can rewrite, the request gets a 446; otherwise the check is recorded as failed and the request goes on.

These checks can rewrite what they read: regexReplace, addPrefix, webhook, anyray.pii, azure.pii, bedrock.guard, pangea.pii, promptfoo.pii, patronus.pii, patronus.phi, f5-guardrails.scan, acuvity.scan, crowdstrike-aidr.guardChatCompletions and exa.online. Every other check only passes or fails.

A check that finds no text to read (an answer that only calls tools, an empty prompt) passes with no error, whatever failOnError says. An answer whose only text is a refusal, or whose text sits past its first choice, does not count as having no text. The checks do not read that text, so such an answer gets the same result it always did.

A check that returns an error (a partner it cannot reach, missing credentials) fails if the check sets failOnError: true. A guardrail with deny then blocks the request with a 446, and the check's error in hook_results reads the check returned an error. Without failOnError, the error is recorded and the request goes on.

Set failOnError on a deny guardrail you rely on

With failOnError, a partner outage blocks requests until the partner is back. Without it, a request can slip past: one that makes the partner call time out causes the error itself.

Under failOnError, a check that can rewrite blocks with a 446 when it returns an error, even in a mutator or a guardrail without deny. An async hook runs too late to block. Any other check in a hook without deny records the failure, and the request or answer goes on.

failOnError takes true or false. Any other value (the string "false", 1) gets a 400, in an x-anyray-config, in the default-guardrail headers and when an admin saves the routing config. A config saved before this check that holds such a value still loads, and the value counts as false.

A config saved before these limits still loads. Each rule runs on as much text as it can finish in time: about 100,000 characters for a rule saved under these limits, with the turn's text blocks counted together. Past that, when a text block is not a string (a token-id prompt, say), or when a jsonSchema check can't validate its JSON in time (nested too deeply, an array too long for uniqueItems, more JSON than the request's validation work allows, or a saved schema these limits now refuse), a guardrail with deny blocks the request with a 446. A regexReplace always does, and so does one that fails to run, so unchecked or unredacted text never goes through. A guardrail without deny records the check as failed.

The hook_results the gateway adds to a response list each hook's id, type, verdict, deny, transformed and skipped, and each check's id, verdict and an error with a fixed reason. They never include the text or JSON a check matched, or its rule.

Regex rules​

A regexMatch or regexReplace rule has the limits of a $regex condition (regex limits), and a client x-anyray-config may carry one. The flags are g, i and s, each at most once:

CheckRuleFlags
regexMatchthe pattern as written (/api/v1/ matches the slashes)a flags parameter: { "rule": "secret", "flags": "i" }
regexReplace/pattern/flags, or a bare pattern; redactText up to 1,024 characters. Replaces every match, with or without gin the rule: /secret/i (a flags parameter gets a 400)

Each rule is judged on a 100,000-character prompt, so one that slows on long text (a.*b, grok.*fast) gets a 400 on the request and is refused on save, while \d{3}-\d{2}-\d{4}, \d+ and sk-[A-Za-z0-9]{48} pass. A config may carry at most 32 regexMatch and regexReplace checks. The x-anyray-default-input-guardrails and x-anyray-default-output-guardrails headers count toward the 32 of the x-anyray-config sent with them, and get 32 of their own without one.

One request's regex checks share a fixed amount of matching work, across every fallback or load-balance target that reruns them and every config they come from. A request that would go past it gets a 446 and is not sent on, so text never goes through unchecked or unredacted. The same 446 stops a regexReplace whose redactText would grow the text by more than its own length plus 64 KiB.

A jsonSchema guardrail's pattern and patternProperties keys have the same limits, read with the u flag the validator uses, and share the same matching work. A schema can't use the url format (it can take too long on long text): use uri. A schema that could apply too many subschemas to one JSON value (allOf of two $refs to the next level, repeated), or is too large to load, gets a 400 and is refused on save. A recursive schema, such as a tree node whose children refer back to it, is fine. One request's jsonSchema checks also share a fixed amount of validation work.

Redaction​

A regexReplace redaction covers more than the latest turn: the system prompt and every message (on the Responses API, instructions and every input item), rewritten the same way every turn. A client sends its own unredacted copy of the history back each turn, so this keeps a secret out of later requests, and it keeps the bytes the provider caches identical turn to turn. Signed thinking blocks and tool-call inputs are left as they are; an OpenAI refusal is redacted like any text. A redacted answer loses the log probabilities of the text that changed. A redaction writes its result back to the places it read, an empty result included, and a failOnDetection one fails only on a match in the latest turn.

Redactions add up, across hooks and whether or not a hook is sequential. A regexReplace that read text another redaction has since changed runs again on the new text. A partner redaction is never called twice: when it lands on changed text, the regexReplace redactions run again on its result, and the request gets a 446 if one of them can't. An embeddings request is checked and redacted too, its inputs counted as one text (a card-number rule reaches about 106,000 characters across the batch).

For a request from a verified user, each gateway replica remembers where a rule matched each text (never the text), for that user only and until it goes unused for an hour, so a session pays for its history's regex work once. A long session that reaches a replica that has not seen it is checked in full there and can pass the rule's limit: that request gets a 446, but the texts that fit are remembered, so sending it again gets further. A request with no verified user is checked in full every time.

A partner redaction (anyray.pii, azure.pii, pangea.pii, promptfoo.pii, patronus.pii, patronus.phi, bedrock.guard, f5-guardrails.scan and acuvity.scan, with redaction on) covers the same places for a request from a verified user. The latest turn goes to the partner, and only it decides the check's verdict. Each replica remembers, for that user and until it goes unused for an hour, where the partner changed each text and what it put there, never the text itself. A hash or format-preserving replacement is kept as the partner returned it, which is what the provider gets anyway. History the replica has seen is rewritten the same way with no partner call. History it has not seen goes to the partner in separate calls, oldest first: for each check, at most 512 texts, 2,000,000 characters and 10 seconds a request, with 8 calls at a time.

Past those limits, or when the partner fails on the history, the check returns an error. With failOnError, the request gets a 446 and the next one picks up where it stopped, so a long session on a replica that has not seen it is refused until enough of its history is remembered. A 1,500-text session takes 3 requests when each partner call takes 120 ms, and 21 at 1.2 seconds for a partner that checks one text a call; after that, the session is not refused again on that replica while it stays in use. Without failOnError, history the partner did not check goes on as the client sent it. A request with no verified user has only its latest turn redacted: its history reaches the provider as the client sent it, and each redacted turn costs one prompt-cache miss when it becomes history. A crowdstrike-aidr check already sends the whole request and replaces it with the partner's answer.

Azure's PII service takes at most 5 texts of 5,120 characters a call, counting an emoji as one. A longer text is cut at its last newline before the limit, else its last space, else its last punctuation mark, else at the limit, and each piece is checked; an entity that spans a cut, such as a two-word name, can be missed. A text counts as checked only when every piece comes back. A latest turn of more than 512 pieces is not checked.

Azure and Bedrock checks​

An Azure check (azure.contentSafety, azure.pii, azure.shieldPrompt, azure.protectedMaterial) in a request's x-anyray-config or default-guardrails header runs on the caller's own API key or Entra client credentials. One that sets azureAuthMode to managed or workload gets a 400, because those modes use the gateway's own Azure identity. In the org routing config, managed works (workload identity is not supported by these checks). The gateway mints that identity's token for Azure AI services only and sends it only to an Azure AI services host (*.openai.azure.com, *.cognitiveservices.azure.com or *.services.ai.azure.com). A save whose check names another customHost, or a resourceName that moves the host off Azure, gets a 400, and the check refuses it with an error before fetching the token. The gateway verifies the TLS certificate of every Azure host, custom or not.

A bedrock.guard check reads awsAccessKeyId, awsSecretAccessKey, awsRegion and an optional awsSessionToken from its credentials, or awsAuthType: assumedRole with awsRoleArn and awsExternalId. A check saved with the older names accessKeyId, accessKeySecret and region works too. When a check has both, the aws name wins.

A caller's bedrock.guard check with awsAuthType: assumedRole gets a 400: it would assume the role with the gateway's own AWS identity. Send the caller's own AWS access key pair.

Streamed answers​

On chat completions, completions, messages and the Responses API, a streamed response is held until it ends and checked whole when an output guardrail has deny or an output check can rewrite the answer. A denial is the same 446 JSON a non-streamed one returns, and a redacted answer is sent as a stream carrying the redacted text. A held stream the gateway cannot read, or cannot rebuild after a redaction, gets a 446, and so does a Responses stream with an event whose text the checks do not read. A held stream past 64 MB gets a 446 too, and the gateway stops reading it. When a held stream ends before the checks get an answer to read, the 446 error type says why: upstream_stream_failed when the upstream broke it, held_stream_cut when the gateway stopped reading (the size cap, or a client that left). On chat completions and messages, a stream the upstream drops after the answer began stays a 200, held or not, and ends with one error event in the request's format, with a fixed message. Other streams are sent as they arrive. A single stream event longer than 128 MB (as many characters, on a text stream), held or not, ends the stream before that event, the way a dropped upstream connection ends it, and the gateway stops reading it. The spend row of a refused answer carries what the provider billed for it, and the refusal adds no savings. When a fallback or retry then moves on, the row adds that cost, at the refused target's price, to the answer's.

Codex gets a guardrail denial and a held_stream_cut as a 400 with the same body, on an API key and on a subscription seat. Codex retries a 446 up to five times, and each retry is a new answer the provider bills and the gateway refuses again; a 400 it shows at once. An upstream_stream_failed stays a 446, so Codex retries a broken upstream. To tell a denial apart, the gateway reads at most 64 KB of a 446 body, for at most two seconds; a longer or slower body reaches Codex as sent.

Pass-through traffic​

A passthrough request keeps its own token and provider and is never routed through the config: a Claude Code, Codex or Copilot seat, or any request sent with x-anyray-auth-mode: passthrough and the caller's own key. It runs the config's guardrails and nothing else of it (no key, retry, cache or override): the hooks the org default config, or the signing agent's own routing, gives that request. A caller's x-anyray-config replaces the org's, guardrails included.

CheckOn pass-through
Input deny446; the request never reaches the upstream
Input redactionThe request is rewritten before it leaves, retries included
Output deny or redactionA successful answer is held until it ends and checked whole, as on API-key traffic, then sent, sent redacted, or refused with a 446. An upstream error is sent as it came
A body the checks cannot read446; on Copilot, also a request to a path that is not chat completions, Messages or Responses while an output deny or redaction applies

As on an API key, Codex gets a guardrail denial or a held_stream_cut as a 400.

When the config's router picks no target for a seat request (no condition matches and there is no default, or every load-balance weight is 0), the seat runs the guardrails set on that router and above it, and none of its targets'. With none there, the request goes as if no guardrail were set. A route that fails for another reason (a condition or the default names a target that doesn't exist, say) still gets a 446, as it fails on API-key traffic.

Every check runs, so a webhook, log or partner check in the config gets the request's text, as it does for API-key traffic. With no guardrails, pass-through traffic is forwarded with no added work.

On Claude Code and Codex, a caller's x-anyray-config that breaks a config rule gets the same 400 as on any other route. Copilot reads a caller's config only for its guardrails, so only one that carries guardrails gets that 400; one without them is ignored.

Checks that only record (a log hook, a guardrail without deny) run on the answer as they do for API-key traffic: a streamed answer is not held for them, and a failed one turns a JSON answer into a 246. They don't run on Codex's WebSocket.

Codex's WebSocket cannot hold an answer, so when an output deny or redaction applies, the gateway answers the upgrade with 426 and Codex sends the session over HTTPS instead. One added while a socket is open closes it once the turn in flight ends, and Codex reopens it over HTTPS. On the socket an input denial reaches Codex as a 400 error event, as over HTTPS. While the config has a deny or redaction anywhere, a socket frame that is neither a request nor Codex's interrupt of its own response is refused the same way, since the checks cannot read it.