pii · Guide

Turkish specifics

Turkish is why this package exists, so its failure modes get dedicated handling rather than a generic NER pass with a language flag. This page covers the checksums, the case-folding rules, and the morphology-driven edge cases the pattern and dictionary tiers are built around.

The Turkish-I problem

"İstanbul".toLowerCase() is "i̇stanbul" (with an extra combining dot above) in a locale-independent lowercase, but "I".toLowerCase() is "ı" (dotless) under a Turkish locale and "i" almost everywhere else. Gazetteer lookups need a Turkish-aware fold; generic text scanning (context-word matching, spelled-digit tokenising) needs an ASCII-safe fold that does not corrupt English content. This package keeps both as two deliberately separate functions in each language:

• Turkish-locale fold (toLocaleLowerCase('tr') in JS, its Python equivalent), used only for matching against the Turkish gazetteer, where İ → i and I → ı is the correct behaviour.
• General fold (foldGeneral/fold_general), used for context-word windows and spelled-digit matching on text whose language is not known: it replaces İ (U+0130) with plain i before a locale-independent lowercase, so a dotted capital folds correctly without leaving a combining mark, while a plain ASCII I still folds the English way.

Neither is "more correct" — they solve different problems, and using the wrong one for a task silently breaks matching in one language or the other.

Real, checksummed Turkish ID validation

tc_kimlik (national ID) validates the official two-step algorithm: 11 digits, first digit non-zero, a mod-10 check on the odd/even digit sums for digit 10, and a mod-10 sum check for digit 11. A random 11-digit run does not false-positive just because it happens to be 11 digits long.

tr_vkn (tax ID) also computes a checksum, but — stated plainly in the source — that algorithm has never been checked against an official test vector. It ships off by default with a deliberately low base score (0.25), so a tr_vkn finding needs a nearby context word ("vergi no", "VKN") to clear a non-zero minConfidence. tr_iban, by contrast, has no checksum or even a length validator at all — it is a regex-shape match only (the cross-language iban category validates length but not the mod-97 checksum either).

Suffix trimming: Ayşe'nin

Turkish attaches case suffixes to proper nouns after an apostrophe — Ahmet'in, İstanbul'dan, EDA KURT'UN. The entity is the part before the apostrophe. Both the dictionary pass and the NER span reconstruction trim the tail when it is short and its case matches the suffix pattern: lowercase after a normally-cased name, or uppercase after an all-uppercase one (ALL-CAPS Turkish text capitalises the suffix too). No suffix list is needed for this — the rule reads case and length only, so it stays inert for genuine apostrophes in English names: O'Brien continues with a capital right after a non-capital, and O'BRIEN has a 5-character tail, so neither trims.

Before this rule existed, the dictionary tier reported spans like Turan'a and Sayın Ahmet Yılmaz'ın — on the Turkish homonym set, 48% of everything the gazetteer found had the wrong edges.

Homonyms: Deniz/deniz, Can/can

Several everyday Turkish words are also common first names: Can (life/soul), Deniz (sea), Barış (peace), Gül (rose), and others. The dictionary pass leans entirely on capitalisation to tell them apart — a lowercase deniz never matches the gazetteer at all (its base score is 0 unless capitalized), so ordinary prose is safe by construction. The harder case is a capitalised common noun that is not a name: Turkish (like English) capitalises the first word of a sentence regardless of what it is, so "Deniz'e gittik" at the start of a sentence looks identical, capitalisation-wise, to "Deniz'e telefon ettim" about a person named Deniz. The dictionary still reports the span in that case (the alternative is silently missing every sentence-initial name), but at a reduced confidence a caller's minConfidence can filter — sentence-initial capitalisation is explicitly not treated as name evidence on its own.

This is exactly the gap the bundled neural tagger is trained to close: a second, disjoint homonym training pool (in addition to the model's regular training data) was added specifically to push Turkish homonym precision closer to a much larger BERT model's — see Model & benchmarks for the measured numbers and an important caveat about what that suite does and does not prove.

ALL-CAPS text

In ALL-CAPS text, every token starts with an uppercase letter, so "is this word capitalised" carries no information at all. The dictionary pass detects this explicitly (a whole-text check for the presence of any lowercase letter) and disables its riskiest heuristic — extending a known name by one adjacent capitalised word — specifically in that mode, because it was measured to swallow unrelated neighbouring words (İMAR BAŞVURUSUNDA TOPRAK misread as containing the name TOPRAK). A bare, single-token gazetteer hit still fires in ALL-CAPS text, since every token there is technically "capitalised" by the same test.

ASCII-folded text

Logs and call-centre transcripts routinely lose Turkish diacritics (ı→i, ş→s, ğ→g, ü→u, ö→o, ç→c). The bundled neural tagger's training data includes a 10% ASCII-folded augmentation of Turkish rows specifically to make this behaviour a trained property rather than an accidental one — the un-augmented model already generalised to some folded text, but the augmentation closed cases like "hatali cikti" (folded "hatalı çıktı") being misread as a name. The month-name date pattern (birthDate) also matches ASCII-folded month names directly in its regex (Subat/Şubat, Agustos/Ağustos, and so on), independent of the model.

Spelled-out digits

Call-centre and speech-to-text style records spell digits out as words instead of writing them: "tc sıfır sekiz iki beş üç dört beş dokuz yedi altı dokuz", "kart no dört iki yedi dokuz yedi bir dört iki sıfır altı sıfır", even ALL-CAPS "KART NO ÜÇ BİR DOKUZ SIFIR". No regex sees any of this on its own. The pattern tier finds runs of 7 to 32 consecutive digit-word tokens (Turkish and English word lists, case-folded), reconstructs the digit string they spell, and hands it to the same validators the digit-shaped categories already use — a spelled-out number becomes a finding only when it would have been a real one written as digits.

Turkish word	Digit	Turkish word	Digit
sıfır / sifir	0	beş / bes	5
bir	1	altı / alti	6
iki	2	yedi	7
üç / uc	3	sekiz	8
dört / dort	4	dokuz	9

A run only continues across whitespace, commas, or hyphens — a full stop ends it, so two unrelated sentences are never joined into one spelled number. A spelled phone candidate additionally needs either a Turkish national phone shape (optional 0/90/0090 prefix, then a 2–5 area/mobile code and nine digits) or a nearby phone keyword, since a phone has no checksum to gate on and Turkish prose genuinely does count out loud. Every spelled-digit candidate is front-tier (L1) only: pattern+ner-verified passes it through unjudged, because the model has never seen a spelled number to agree or disagree with.

Record-dump text

CRM-style field dumps (ad: {PERSON}, tel: {TCKN}, ...) turned out to be a common register in the text this package actually sees, and are represented directly in the model's training data (--register record templates) rather than only prose sentences. A related, narrower fix: password and pin extraction explicitly excludes a trailing comma or bare sentence-ending period from the captured value, because a field: value, field: value record dump otherwise left a trailing comma glued onto the password (pwd e^!+8L#r^W, capturing the comma) while a value containing an internal dot (Marmara.9553) still needs to be captured whole.

Where to go next

• Categories — the full category table, including every Turkish-specific id.
• Detection modes — how the dictionary and NER tiers combine, and the judge's fusion math.
• Model & benchmarks — the homonym suite's numbers, and the contamination caveat that applies to them.