Guardrail · Benchmarks

Benchmarks

Two independent measurements. First, every system scored on one fixed sample of our own held-out test split, so the comparison is apples-to-apples. Second, public safety benchmarks that were never part of training at all — the harder and more honest test.

How these were produced

Both tables come from scripts in the open-source repository — scripts/benchmark_all.py and scripts/eval_public_benchmarks.py — and the page itself is generated from their JSON output, so nothing here is transcribed by hand.

Part 1 — head-to-head on held-out rows

• Benchmark set: 20,000 test rows (10,993 Turkish / 9,007 English), sampled at natural prevalence from the held-out split
• Positives: 7,441 moderation, 1,177 prompt shield
• Thresholds: fitted per system on a separate 8,000-row calibration sample, never on test
• Input: raw text for every system. Guardrail normalizes internally; the external ones receive exactly what a real integration would hand them.

None of the external systems share our nine categories, so the comparable axis is the two gates. A system with no signal for a gate is absent from that table rather than scored zero — a toxicity classifier is not a failed prompt shield, it is simply not one.

Moderation gate

Insulting, hateful, sexual, violent, self-harm or illegal-activity content.

System	AP	AUC	F1	Precision	Recall	FPR	AP (real rows only)	F1 (TR)	F1 (EN)
Frontier LLM judge (zero-shot)	0.946	0.976	0.918	0.913	0.922	0.0532	—	0.953	0.872
Guardrail (this library)	0.837	0.900	0.730	0.804	0.669	0.0965	0.703	0.790	0.645
Qwen/Qwen3Guard-Gen-0.6B	0.816	0.883	0.743	0.663	0.845	0.2544	0.731	0.737	0.753
textdetox/xlmr-large-toxicity-classifier	0.695	0.818	0.672	0.707	0.641	0.1573	0.693	0.678	0.664
unitary/multilingual-toxic-xlm-roberta	0.627	0.744	0.590	0.598	0.583	0.2327	0.740	0.526	0.683
s-nlp/roberta_toxicity_classifier	0.598	0.712	0.610	0.464	0.889	0.6075	0.754	0.576	0.686
unitary/toxic-bert	0.579	0.696	0.598	0.453	0.877	0.6266	0.728	0.569	0.662
unitary/unbiased-toxic-roberta	0.574	0.691	0.580	0.471	0.755	0.5029	0.735	0.536	0.649
citizenlab/distilbert-…-toxicity	0.552	0.689	0.592	0.495	0.735	0.4442	0.612	0.587	0.603
martin-ha/toxic-comment-model	0.530	0.666	0.599	0.458	0.866	0.6066	0.673	0.577	0.651
KoalaAI/Text-Moderation	0.488	0.588	0.551	0.408	0.848	0.7283	0.644	0.551	0.551

Prompt-shield gate

Attacks on the assistant itself: jailbreaks, prompt injection, and attempts to extract system prompts or data.

System	AP	AUC	F1	Precision	Recall	FPR	AP (real rows only)	F1 (TR)	F1 (EN)
Frontier LLM judge (zero-shot)	0.887	0.974	0.888	0.837	0.946	0.0101	—	0.939	0.638
Guardrail (this library)	0.797	0.978	0.730	0.677	0.793	0.0236	0.519	0.763	0.574
Qwen/Qwen3Guard-Gen-0.6B	0.414	0.859	0.451	0.412	0.497	0.0443	0.370	0.495	0.327
protectai/deberta-v3-base-prompt-injection-v2	0.268	0.808	0.209	0.570	0.128	0.0061	0.375	0.061	0.615
deepset/deberta-v3-base-injection	0.154	0.781	0.226	0.157	0.408	0.1373	0.044	0.239	0.169
protectai/deberta-v3-base-prompt-injection	0.103	0.546	0.126	0.110	0.148	0.0750	0.207	0.084	0.253
madhurjindal/Jailbreak-Detector	0.096	0.674	0.158	0.095	0.469	0.2789	0.074	0.180	0.112
jackhhao/jailbreak-classifier	0.073	0.442	0.120	0.129	0.111	0.0469	0.226	0.020	0.207

Obfuscation robustness — prompt-shield gate

Recall on known-unsafe text after an attacker perturbs the surface form, at each system's own threshold.

An attack that raises recall is not robustness

Read each number against that system's unperturbed recall in the gate table above. unitary/toxic-bert recovers 0.98 under masked vowels against 0.56 on clean text — censored profanity (f*ck) is itself a strong signal in its Jigsaw training data, so the perturbation hands it a feature rather than hiding one. Guardrail strips the mask characters first and then has to recognise the word underneath, which is strictly harder and scores lower here.

Attack	protectai/deberta-v3-base-prompt-injection-v2	protectai/deberta-v3-base-prompt-injection	deepset/deberta-v3-base-injection	jackhhao/jailbreak-classifier	madhurjindal/Jailbreak-Detector	Qwen/Qwen3Guard-Gen-0.6B	Guardrail (this library)
leetspeak	0.081	0.150	0.136	0.034	0.451	0.550	0.771
Cyrillic homoglyphs	0.115	0.166	0.314	0.056	0.640	0.520	0.782
zero-width characters	0.404	0.235	0.001	0.107	0.464	0.470	0.782
spaced letters	0.212	0.527	0.328	0.058	0.672	0.525	0.750
masked vowels	0.109	0.547	0.025	0.106	0.495	0.490	0.595
character repeats	0.119	0.206	0.062	0.041	0.810	0.510	0.634
Turkish de-ASCII	0.147	0.229	0.285	0.107	0.505	0.510	0.784
uppercase	0.105	0.403	0.160	0.107	0.879	0.595	0.782
punctuation noise	0.089	0.059	0.045	0.071	0.500	0.495	0.624
combined attack	0.134	0.266	0.009	0.039	0.629	0.535	0.774
base64 wrapping	1.000	1.000	0.000	0.004	0.000	0.000	0.935
reversed text	0.422	0.924	0.001	0.020	0.072	0.015	0.882

Obfuscation robustness — moderation gate

Recall on known-unsafe text after an attacker perturbs the surface form, at each system's own threshold.

An attack that raises recall is not robustness

Read each number against that system's unperturbed recall in the gate table above. unitary/toxic-bert recovers 0.98 under masked vowels against 0.56 on clean text — censored profanity (f*ck) is itself a strong signal in its Jigsaw training data, so the perturbation hands it a feature rather than hiding one. Guardrail strips the mask characters first and then has to recognise the word underneath, which is strictly harder and scores lower here.

Attack	unitary/multilingual-toxic-xlm-roberta	unitary/toxic-bert	unitary/unbiased-toxic-roberta	martin-ha/toxic-comment-model	s-nlp/roberta_toxicity_classifier	citizenlab/distilbert-…-toxicity	textdetox/xlmr-large-toxicity-classifier	KoalaAI/Text-Moderation	Qwen/Qwen3Guard-Gen-0.6B	Guardrail (this library)
leetspeak	0.251	0.993	0.421	0.225	0.716	0.989	0.890	0.519	0.795	0.684
Cyrillic homoglyphs	0.669	0.988	0.385	0.854	0.961	0.990	0.858	0.921	0.805	0.694
zero-width characters	0.955	0.885	0.868	0.873	0.989	0.726	0.920	0.865	0.840	0.694
spaced letters	0.955	0.995	0.969	0.801	0.995	0.929	0.713	0.875	0.845	0.676
masked vowels	1.000	1.000	1.000	0.999	1.000	1.000	0.998	0.889	0.855	0.552
character repeats	0.945	0.999	0.911	0.954	0.990	0.991	0.963	0.945	0.880	0.591
Turkish de-ASCII	0.797	0.899	0.915	0.823	0.700	0.814	0.715	0.771	0.845	0.694
uppercase	0.970	0.899	0.984	0.823	0.969	1.000	0.830	0.395	0.805	0.694
punctuation noise	0.879	0.976	0.820	0.320	0.958	0.995	0.889	0.800	0.715	0.550
combined attack	0.367	0.993	0.372	0.343	0.830	0.995	0.960	0.784	0.755	0.688

Latency

Single-message latency, which is what a request path actually pays. Same machine for every system in this table; every neural model here, including the Guardrail row, ran on this machine's Apple-Silicon GPU (MPS) via the training repo's best_device(), through the PyTorch checkpoint rather than the shipped ONNX package.

The shipped Python package measures separately, CPU-only: onnxruntime 1.30 on an Apple M-series (14 cores), 1,000 texts of mean 150 characters — scan() p50 0.45 ms, 1.25 ms p50 single-threaded, 667 texts/s batched on a single thread.

System	p50 (ms)	p95 (ms)	Throughput (texts/s, batched)
Guardrail (this library)	1.3	1.6	3838
citizenlab/distilbert-…-toxicity	4.3	5.5	362
madhurjindal/Jailbreak-Detector	5.5	9.5	372
s-nlp/roberta_toxicity_classifier	7.4	10.0	97
textdetox/xlmr-large-toxicity-classifier	7.4	8.5	122
martin-ha/toxic-comment-model	10.5	15.8	170
KoalaAI/Text-Moderation	15.9	57.6	51
unitary/toxic-bert	16.3	29.1	86
jackhhao/jailbreak-classifier	18.4	29.2	87
unitary/unbiased-toxic-roberta	18.8	27.6	83
protectai/deberta-v3-base-prompt-injection-v2	19.5	28.3	67
protectai/deberta-v3-base-prompt-injection	21.3	35.1	71
unitary/multilingual-toxic-xlm-roberta	22.2	34.6	84
deepset/deberta-v3-base-injection	40.5	52.3	32
Qwen/Qwen3Guard-Gen-0.6B	83.1	130.3	12

Part 2 — public safety benchmarks

Our corpus already contains AdvBench, DoNotAnswer, ToxiGen, Jigsaw, RTP-LX and MultiJail, so quoting those would be quoting the training set. These are benchmarks the model has never seen — except the two marked otherwise, which are in the corpus but were reserved for test from the start by construction.

detect is the share of should-flag prompts caught; FP is the share of must-not-flag prompts wrongly flagged. Both use each system's own calibrated thresholds — no per-benchmark tuning.

XSTest

XSTest (Röttger et al., 2024) — exaggerated-safety test suite
450 prompts · 200 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	12.5%	4.4%	0.563	0.617

HarmBench

HarmBench (Mazeika et al., 2024) — harmful behaviours, all textual
400 prompts · 400 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	21.5%	—	—	—

StrongREJECT

StrongREJECT (Souly et al., 2024) — forbidden prompts
313 prompts · 313 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	26.5%	—	—	—

JailbreakBench

JailbreakBench JBB-Behaviors (Chao et al., 2024) — 100 harmful + 100 matched benign
200 prompts · 100 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	30.0%	11.0%	0.679	0.707

deepset prompt-injections

deepset/prompt-injections — EN+DE injections with benign controls (label includes scope violations, not only instruction overrides; never trained on, but its failure analysis motivated later training data)
546 prompts · 203 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	9.9%	0.3%	0.716	0.760

OR-Bench-Hard-1K

OR-Bench-Hard-1K (Cui et al., 2024) — benign prompts that look harmful
1,319 prompts · 0 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	—	8.6%	—	—

HateCheck

HateCheck (Röttger et al., 2021) — functional tests; test-only in this corpus
3,728 prompts · 2,563 should-flag · in corpus, test-only split

System	detect	FP	AP	AUC
Guardrail (this library)	57.4%	39.5%	0.766	0.616

Turkish over-refusal twins

Turkish over-refusal twins (Yurtsevenler) — matched safe/harmful pairs, test-only in this corpus
240 prompts · 120 should-flag · in corpus, test-only split

System	detect	FP	AP	AUC
Guardrail (this library)	49.2%	5.0%	0.812	0.849

tr_advbench

Turkish AdvBench translation (MultiJail-style 'cogoverload' data) -- machine-translated AdvBench harmful-behavior goals; never used in training, eval-only
520 prompts · 520 should-flag · never used in training

System	detect	FP	AP	AUC
Guardrail (this library)	23.1%	—	—	—

---

Every number regenerates with bash scripts/run_all.sh in the repository. If a number here looks wrong, the harness that produced it is open — check it.