pii · API Reference

Python API

The cognipeer-pii distribution, imported as cognipeer_pii (Python 3.9+). See JavaScript API for the mirrored surface; the two are independent ports of the same algorithm, checked against shared parity fixtures.

Entry points

from cognipeer_pii import (
    detect, redact, mask, tokenize_text, detokenize_text,
    detect_async, redact_async, mask_async, tokenize_async,
    PII_CATEGORIES, PII_CATEGORIES_BY_ID,
    category_label, filter_categories_by_languages,
    build_default_categories,
)

Synchronous functions (pattern tier only)

These never touch the dictionary or NER tiers, regardless of detection's mode — they always run the pattern (L1) tier.

def detect(text: str, categories: dict | None = None, custom_patterns=None,
           languages=None, locale: str = "en", detection: dict | None = None) -> dict: ...
def redact(text: str, ...) -> dict: ...
def mask(text: str, ...) -> dict: ...
def tokenize_text(text: str, ...) -> dict: ...        # adds a "vault" key
def detokenize_text(text: str, vault: dict) -> dict: ...

import cognipeer_pii as pii

pii.detect("e-postam ahmet@example.com, IBAN: TR330006100519786457841326")
pii.redact("...")   # findings removed, replaced with [REDACTED_<CATEGORY>]
pii.mask("...")     # findings replaced with a partial mask ([CATEGORY] or j***@gmail.com-style)

result = pii.tokenize_text("Kart no: 4111111111111111")
pii.detokenize_text(result["outputText"], result["vault"])  # restores the original

Async functions (opt into dictionary and NER tiers)

Identical keyword arguments, coroutine functions, and the only way to reach pattern+dictionary, pattern+dictionary+ner, and pattern+ner-verified:

async def detect_async(text: str, categories=None, custom_patterns=None,
                        languages=None, locale: str = "en", detection: dict | None = None) -> dict: ...
async def redact_async(text: str, ...) -> dict: ...
async def mask_async(text: str, ...) -> dict: ...
async def tokenize_async(text: str, ...) -> dict: ...   # adds a "vault" key

async def with_ner(text):
    return await pii.detect_async(
        text,
        detection={
            "mode": "pattern+dictionary+ner",
            "ner": {"minScore": 0.5, "maxConcurrent": 2, "timeoutS": 1.5},
        },
    )

async def with_verification(text):
    return await pii.detect_async(text, detection={"mode": "pattern+ner-verified"})

Keyword arguments

Shared by every function above:

Argument	Default	What it does
categories	Every default_enabled category on	{"email": True, "tc_kimlik": False, ...} — explicitly enable/disable per category. Passing this replaces the default set entirely rather than merging with it.
custom_patterns	None	Your own regex rules, run under a SIGALRM-based timeout. POSIX and main-thread only — see regex_safe.py's docstring; a custom pattern's protection does not apply on Windows or off the main thread. See the shape below.
languages	None (all)	Restrict to categories tagged for these languages, e.g. ["tr"]. "global"-tagged categories always run regardless.
locale	"en"	Passed through to category label/message lookups ("en" or "tr").
detection	None (pattern tier)	{"mode": "pattern" | "pattern+dictionary" | "pattern+dictionary+ner" | "pattern+ner-verified", "minConfidence"?, "contextBoost"?, "ner"?: {...}, "verify"?: {...}}.
detection["ner"]	—	{"minScore", "maxChars", "timeoutS", "maxConcurrent", "failMode", "decode"} — note timeoutS (seconds), not timeoutMs as in JavaScript. See Detection modes.
detection["verify"]	see Detection modes	pattern+ner-verified-only: {"agreeWeight": 0.5, "floor": 0.25, "checksumFloor": 0.7, "contextFloor": 0.8, "dictionaryFloor": 0.5, "nerOnlyWeight": 0.6, "nerOnlyNumberWeight": 0.35} — key names stay camelCase, matching JavaScript exactly.

custom_patterns entries

Plain dicts (or any mapping with these keys):

{
    "categoryId": "employee_id",
    "pattern": r"EMP-\d{6}",   # regex source, max 512 characters
    "flags": "i",
    "enabled": True,
    "severity": "medium",     # optional
    "label": "Employee ID",   # optional
    "languages": ["en"],      # optional — applies to all requested languages if omitted
}

Return shape

Both a scan result and each finding are plain dicts, and — deliberately, for cross-language consistency — the top-level result and finding keys stay camelCase, matching JavaScript exactly, even though Python's own function arguments (custom_patterns, min_confidence inside the category dataclass, etc.) are idiomatic snake_case:

{
    "inputLength": int,
    "findings": [
        {
            "category": str, "value": str,
            "start": int, "end": int,        # Unicode code point offsets — see the offset note below
            "label": str, "message": str,
            "severity": "low" | "medium" | "high",
            "confidence": float,
            "action": str, "block": bool,
            "replacement": str,
            "detector": "pattern" | "dictionary",
            "evidence": [str, ...],
        },
        # ...
    ],
    "outputText": str,        # unchanged for "detect"; replacements applied otherwise
    "hasBlocking": bool,      # True if any finding has block: True
    "action": "detect" | "redact" | "mask" | "tokenize",
    "languages": [str, ...],  # ["global"] if none were requested
    "degraded": [...],        # present only when non-empty
}

tokenize_text/tokenize_async additionally return "vault": {token: {"value": str, "category": str}}, keyed by the exact token strings ([EMAIL_1], [TC_KIMLIK_1], …) that appear in "outputText".

Categories catalog

PII_CATEGORIES is a list of PiiCategory dataclass instances (attribute access, not dict access):

from cognipeer_pii import PII_CATEGORIES, PII_CATEGORIES_BY_ID, build_default_categories

len(PII_CATEGORIES)                          # 31
PII_CATEGORIES_BY_ID["tc_kimlik"].default_enabled   # True
build_default_categories()                   # {"email": True, "phone": True, ..., "tr_vkn": False, ...}

See Categories for the full table with languages, defaults, and validators.

JS/Python parity, and where it is not 1:1

The two ports are independent implementations of the same algorithm, checked against shared JSON fixtures (tests/parity/fixtures.json, generated once from the JS engine) so both return byte-identical spans and categories. A separate suite (tests/parity/ner_fixtures.json) checks the tokenizer, the FNV-1a word hashes, and the final NER spans independently. Known, intentional differences:

	JavaScript	Python
Function/argument naming	camelCase (tokenizeText, customPatterns)	snake_case (tokenize_text, custom_patterns)
Result/finding dict keys	camelCase	Also camelCase — kept for interop, not converted to snake_case
detection.ner timeout key	timeoutMs (milliseconds)	timeoutS (seconds)
Custom-pattern timeout mechanism	vm context with a wall-clock timeout	SIGALRM — POSIX and main-thread only
Category catalog shape	Array of plain objects (categoriesById.email.defaultEnabled)	List of PiiCategory dataclass instances (PII_CATEGORIES_BY_ID["email"].default_enabled)
categoryDescription helper	Exported	Not exported — only category_label is
String offsets	UTF-16 code units	Unicode code points

The UTF-16 vs. code-point offset note

Python strings are Unicode code points; the JavaScript port's strings are UTF-16. For text containing a non-BMP character (most emoji), a single such character counts as 1 unit in a Python string but 2 in a JS string, so start/end offsets after that point differ by one per character between the two ports. Neither is wrong — each is correct for slicing the string in its own language — and redact()/mask()/tokenize_text() all produce byte-identical output text in both languages regardless. A caller moving raw offsets between the Python and JS ports (rather than just using each port's own outputText) needs to convert them.

Where to go next

• JavaScript API — the mirrored surface, same option names in JavaScript's casing.
• Detection modes — the full meaning of every detection option.
• Redaction & tokens — mask strategies and the finding shape in depth.