pii · Guide

Model & benchmarks

Every number on this page is copied from the project's own reports/ and docs/MODEL.md. Regressions are reported alongside the gains, and the two genuinely external benchmarks (never trained on) are called out separately from the package's own evaluation suites.

Read the contamination notes before quoting a number

Several of the suites below share data with the model's own training set in ways that are documented but easy to miss — most importantly the homonym suite (below) and the identifier-probe suite. The two external, never-trained-on checks are the third-party CC0 Turkish PII benchmark and the Text Anonymization Benchmark (TAB) / ECHR court judgments.

Architecture

The shipped model (v8, 2026-09-13) is a purpose-built token tagger trained from scratch — not a shrunk BERT — specifically because quantising a BERT-based predecessor addressed size, not shape: it would still have been roughly 100 MB and tens of milliseconds per call, an order of magnitude away from the pattern tier it sits beside.

per token   char embedding (24d) -> Conv1d widths 2/3/4, 32 filters -> max-pool  (96d)
            FNV-1a hashed word embedding, 16384 buckets                          (32d)
            16 shape features (casing, digits, punctuation, position, length,
                                4 identifier-shape features)                     (16d)
                                                                     concat -> 144d
sentence    Linear -> 128d, mask applied BEFORE the first conv block too
            4x [Conv1d k=3, dilation 1/2/4/8] + GELU + residual + LayerNorm
per token   Linear -> 15 BIO logits (O, B/I- x {PER, ORG, LOC, PHONE, EMAIL, ID, FIN})

753,703 parameters, 2.91 MB as fp32, 1.45 MB after int8 dynamic quantisation — applied only to the embedding (Gather) tables, which are most of the size. Quantising the convolution layers too was tried and measured: it produced a smaller file (0.79 MB) but ran slower (0.42 ms/text vs. the Gather-only build's 0.29 ms/text on one thread), because a dynamically-quantised convolution is slower than plain float convolution at this size on CPU. Gather-only quantisation keeps fp32 speed and label-agrees with the fp32 graph 99.97% of the time on real text (the export gate requires ≥99.5% on held-out data before it ships int8 at all).

Key design choices, each tied to a measured failure it fixes:

• A character CNN, because Turkish is agglutinative (Yılmaz, Yılmaz'ın, Yılmaz'dan, Yılmazlar are one stem with different endings) and no fixed word vocabulary handles that gracefully.
• A hashed word embedding (FNV-1a into 16,384 buckets — the Bloom-embedding trick spaCy uses), because the char CNN alone cannot easily memorise that Yılmaz is usually a surname.
• 16 explicit shape features, four of them added specifically when PHONE/EMAIL/ID/FIN became real labels (7 ≤ length ≤ 11, all-digit starting with 0, all-digit length in {10, 11}, has both a digit and a letter). Without these, an 11-digit run and a 0-prefixed 11-digit phone differed only inside a max-pooled character window with no notion of overall length — measurably causing the judge to read checksum-invalid TCKN-shaped numbers as phones.
• Dilated convolutions (1/2/4/8), not a BiLSTM — a wide receptive field without an RNN, chosen because it exports cleanly to ONNX and is several times faster than an LSTM on CPU at this size.
• No CRF — the model outputs plain per-token logits; a BIO-constrained Viterbi decoder (not a learned transition matrix) removes structurally invalid spans at inference time, so both language ports decode identically without reimplementing a CRF in JavaScript.
• Overlapping inference windows. A single call covers at most 160 tokens; longer text is split into windows that overlap by 32 tokens and stitched into one decode, so an entity that used to straddle a hard window boundary is no longer split or dropped. Measured on 300 long documents: entities straddling the old boundary went from 0/31 found correctly to 29/31, and F1 on those documents from 0.826 to 0.838 — at a cost of roughly 24% more tokens processed on long inputs. Text under 160 tokens (the common case) is byte-identical to the pre-stitching output.

Both language ports run the same ONNX graph, and tests/parity/ asserts they return identical spans — FNV-1a hashing (not either language's native hash()), an explicit character vocabulary shipped in tagger.json (not ord()/Unicode category tables), and apostrophe-based tokenisation are all chosen specifically because they produce identical results in both languages. One divergence is deliberate: JavaScript strings are UTF-16 and Python strings are code points, so an emoji or other non-BMP character shifts integer offsets by one between languages — redact() output text is still byte-identical in both.

Training data

Public Turkish NER corpora exist but do not describe the text this package actually sees — measured, not assumed. Across the entire 17,967-sentence Turkish-WikiNER corpus, A.Ş. appears 8 times, Ltd/Şti 6/1 times, and tokens of 10+ digits (a TCKN/phone/IBAN shape) appear once; its most frequent PERSON values are Buffy, Madonna, Harry, and Hitler. Five sources close that gap:

Source	Licence	Contributes
Wismut/nym-pii-multilingual-data	MIT	Form- and record-shaped PII text in TR and EN.
BTX24/turkish-privacy-pii-ner	CC BY 4.0	Turkish PII sentences with character-level person and address spans.
Turkish-WikiNER	CC BY-SA 4.0	Encyclopaedic name diversity: 10,038 distinct PERSON surface forms, 91% appearing exactly once.
Teacher-labelled silver	Source-dependent	Ordinary running Turkish (SQuAD-TR contexts, UD treebanks, reviews), labelled by a 421 MB BERT teacher with a confidence floor.
LLM-generated	Generated in-project	The business/support domain nothing public covers — fillers, templates, minimal pairs, and hard-context sentences.

The ShareAlike question. Turkish-WikiNER is CC BY-SA 4.0, and whether trained model weights count as "Adapted Material" under ShareAlike is legally unsettled — it is never vendored into the repository, and a build script can exclude it entirely so a release can be trained on MIT/CC-BY/CC0-only data if that ambiguity matters for a given use.

Contamination discipline

Read each of these before treating the corresponding number as a clean generalisation result:

• The homonym suite is not an unseen-word test. Its words (Deniz, Umut, Can, …) appear as PERSON names in the model's own generated training data — llm_direct.json alone uses them as names 2,497 times. What the suite measures is telling a known name from the same everyday word, not handling a word never seen as a name. BERT's pretraining has seen these names too, so the head-to-head comparison is fair, but neither system's number is a clean out-of-vocabulary result.
• data/eval/id_probes.json (350 hand-written identifier probes) is never trained on, but it is not third-party or human-annotated data either — it was written inside this project by a separate Claude (Sonnet) agent, the same day it started being used, specifically to avoid sharing the LLM data generator's own blind spots. Treat it as an independent check on the project's own generator, not as evidence of external validation.
• The external CC0 Turkish PII benchmark and TAB/ECHR are the real external checks — built by someone else, for a different purpose, never touched by any generator or training step in this project.
• Nemotron-PII is no longer a clean external check for this package specifically. An earlier round measured it cleanly; this package now trains on its TRAIN split, so its own score on the TEST split measures in-distribution fit, not generalisation. It remains a clean comparison point for BERT, Presidio, and spaCy, whose training was unaffected.

Results

All numbers below are for the shipped model (v8, 2026-09-13) unless a table says otherwise. P/R/F1 to three decimals.

Identifier probes (data/eval/id_probes.json, 350 hand-written rows)

Mode	F1	Precision	Recall
pattern	0.795	—	—
pattern+ner-verified	0.922–0.934*	0.897–0.916	0.948–0.959

* F1 moved across checkpoints on the same day: 0.937 (v4) → 0.932 (v4b) → 0.922 (v5, after a retrain that traded some identifier precision for a homonym-precision gain) → 0.923 (v6) → 0.934 (v8, the shipped checkpoint). See Regressions below for what moved and why.

Clean third-party CC0 Turkish PII benchmark (never trained on)

Recall by tag presence, 1,000 sentences, cagrigungor/turkish-pii-masking-benchmark (CC0):

System	All PII types	PERSON/ORG/LOC only	ms/call	Resident memory
pii (pattern+ner-verified)	0.887–0.888	0.850–0.858	~0.9 ms	~25 MB
pii (pattern+dictionary+ner)	0.835	0.860	~0.8 ms	~24 MB
bert (akdeniz27, 441 MB)	0.291	0.871	~18–21 ms	944 MB
presidio (+ en_core_web_lg)	0.470	0.683	~2 ms	1,437 MB
spacy (en_core_web_lg)	0.228	0.683	~2.5 ms	1,285 MB
pii-dictionary (gazetteer only, no model)	0.742**	0.782**	0.23 ms	0.5 MB
pii-pattern (regex/checksum only)	0.504**	0.070**	0.05 ms	0.6 MB

** From the first head-to-head round (not re-measured in the 2026-09-13 re-run) — the layered design's own point holds either way: regex alone knows no names, the gazetteer adds them, and the model generalises past the gazetteer's fixed list.

On the three types BERT can emit, a 1.45 MB model comes within roughly one point of a 441 MB one — a ~300x size difference and a ~29–38x memory difference for a little over a point of recall. On the whole PII task (which needs email/IBAN/national-ID detection BERT and spaCy have no concept of at all) BERT falls to 0.291 and spaCy to 0.228 — this is the argument for the layered pattern+dictionary+model design generally, not only for this specific model.

Turkish homonym minimal pairs (Can/can, Deniz/deniz, …)

Contamination caveat applies here — read it above

System	Strict F1
pii (pattern+ner-verified)	0.793
bert	0.773
pii (own eval_suite homonym suite, a different, non-identical methodology)	0.775
pii-dictionary (gazetteer only)	0.464
presidio / spacy	0.025–0.031

The gap to BERT closed from 0.434 (first round) to roughly 0.02–0.05 across later rounds, then crossed over in the most recent head-to-head re-run — driven by a span-boundary fix and training on a word-disjoint homonym pool, not a bigger network.

Text Anonymization Benchmark (TAB / ECHR court judgments, MIT, human-annotated)

TAB ships its own evaluator, and its Table 6 publishes baselines produced by that same script — this is the only benchmark on this page scored entirely by someone else's tooling. This package's official-evaluator numbers below use pattern+dictionary+ner with every category enabled (the methodology EXTERNAL_BENCHMARKS.md establishes); the separate strict-span comparison against BERT/Presidio/spaCy further down uses pattern+ner-verified specifically.

Official TAB evaluator, all categories enabled:

System	Token recall	Token precision
Fine-tuned RoBERTa (OntoNotes 5, TAB's own baseline)	0.906	0.515
pii (pattern+dictionary+ner, v8)	0.669	0.631

Per-entity-type token recall (measured on the original 2026-09-11 baseline model — not re-run per-tag against v8, though the aggregate numbers above are v8's): the categories this package actually models score 0.53–0.82 (LOC 0.822, PERSON 0.755, ORG 0.526); categories it does not model at all score near zero (CODE — case/application numbers — 0.005, DEM — nationality/ethnicity/religion — 0.183, QUANTITY 0.075). The aggregate score is low mainly because TAB's definition of "identifying" is much wider than this package's category list — a case number, the word "Turkish", and every date in a judgment are all things TAB expects masked and this package does not touch by default. That is a category gap for anonymisation use cases, not a quality gap on what it does model. Part of the v8 TAB improvement over earlier checkpoints comes from birthDate's month-name date forms matching ECHR dates directly — a regex-tier change, not a model change, and birthDate is off by default.

benchmark_all.py's own strict-span F1, across systems (v8, pattern+ner-verified):

System	Strict span F1
pii (pattern+ner-verified, v8)	0.325
bert	0.262
spacy	0.210
presidio	0.209

Head-to-head speed (p50 ms/call, 40 iterations, one machine)

System	TR short (41 ch)	TR medium (257 ch)	EN medium (232 ch)
pii-pattern	0.017	0.076	0.059
pii-dictionary	0.151	0.309	0.258
pii-ner	0.509	1.247	1.146
spacy	0.789	1.893	4.692
presidio	0.941	3.059	6.141
bert	6.933	17.554	22.114

The neural tier is 13–19x faster than the 441 MB BERT and 1.5–5x faster than spaCy/Presidio, on a model roughly 300–600x smaller by file size and ~29–38x smaller resident.

Regressions, recorded not hidden

Model and regex-tier changes on 2026-09-13 traded some numbers down for a homonym-precision gain. Named in the project's own version-history report rather than smoothed over:

• TAB token recall: 0.291 (7-label baseline) → 0.280 → 0.266 at the low point, before the later checkpoints' front-tier and register-data work brought TAB token recall back up to 0.669 in the shipped v8 model (a different axis — precision/recall trade-off within TAB, not a straight regression against the very first baseline).
• Held-out Turkish strict F1: 0.820 (v4) → 0.809 (v5) → 0.816 (v8).
• Identifier probe F1: 0.932 (v4b) → 0.922 (v5), before recovering to 0.934 by v8.
• Gazetteer-disjoint names F1 (the measurement the whole gazetteer-generalisation argument stands on): 0.967 (7-label baseline) → 0.946 (v5) — a 2-point move worth watching on the next retrain, though still far above the gazetteer tier's own recall on the same set.
• CC0 KONUM (bare Turkish location mentions) recall: 0.316 (baseline) → 0.228 in a same-scope re-measurement — the single weakest per-tag CC0 result, and almost certainly the same PER/ORG/LOC trade-off family as the other regressions above.

Known gaps

• username/password are keyword-anchored and off by default — a value with no introducing keyword is never found, and roughly half of raw keyword hits on a 3,000-row sample were ordinary prose, not real values.
• CNN-only identifier spans are deliberately down-weighted (nerOnlyNumberWeight 0.35 in pattern+ner-verified) — the regex front stays in charge of whether a number-shaped entity exists at all.
• birthDate, when enabled, matches every calendar-valid date, not only birth dates — it has no way to distinguish them without a keyword.
• English is secondary. TAB strict span F1 (0.325) now leads the systems compared here, but TAB's own token-recall bar (0.906, a model trained specifically for anonymising court text) is far off, and English training data is a fraction of the Turkish volume.
• Turkish names on real third-party text are still very slightly behind BERT: CC0 PER/ORG/LOC recall 0.850–0.858 against BERT's 0.871.

Reproducing these numbers

The project ships the scripts behind every table above (scripts/eval_suite.py, scripts/eval_identifiers.py, scripts/eval_global.py, scripts/benchmark_all.py, scripts/bench_ner.py, scripts/run_tab_official.py, scripts/run_multiconer.py) — see the repository's reports/ directory for full methodology notes, including which runs shared a machine with a training job and should be read as directional rather than exact.

Where to go next

• Detection modes — the judge's fusion math that produces the pattern+ner-verified numbers above.
• Turkish specifics — the morphology and homonym handling these benchmarks measure.
• pii overview — headline numbers with the same caveats, condensed.