Guardrail · Guide

Self-hosting

There is no model server to run, no weights endpoint to reach and no licence check to phone home to. Deployment is the ordinary problem of getting a package into an image — plus one instance held for the life of the process.

Nothing to download

The 1.4 MB ONNX graph, the character vocabulary, the manifest and the calibrated thresholds are all package data. They are inside the wheel and inside the npm tarball.

That has three consequences worth stating plainly:

• No runtime network access is required. Not at first scan, not at load, not ever. A container with no egress and no DNS behaves identically to one on the open internet.
• No cold-start download. The first scan() after load() is the same speed as the millionth.
• The install is the pin. Version the package and you have versioned the model. There is no separate artifact whose drift you have to track, and no cache directory to warm or mount.

What "offline" means here

It is the strong form. load() opens a handful of files under the installed package directory and creates an ONNX Runtime session. There is no telemetry, no licence handshake, no model registry lookup and no optional "check for updates" path to disable. You can verify this the blunt way: run the container with networking off.

Docker

Python

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Fail the build, not the first request, if the model cannot load.
RUN python -c "from cognipeer_guardrail import Guardrail; Guardrail.load()"

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Node

FROM node:20-slim

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci --omit=dev

COPY . .

# Fail the build, not the first request, if the model cannot load.
RUN node -e "require('@cognipeer/guardrail').Guardrail.load().then(()=>console.log('ok'))"

CMD ["node", "server.js"]

The RUN line that loads the model once at build time is worth keeping. It costs a few seconds and it turns "the native binary is for the wrong architecture" or "the model directory is incomplete" from a 3 a.m. page into a red build.

Install inside the image, for the image's platform

Both ecosystems resolve platform-specific binaries at install time — onnxruntime as a wheel, onnxruntime-node as a native addon. Copying a node_modules tree or a site-packages directory from a developer laptop into a linux/amd64 image gives you the wrong binary and a load-time crash.

Keep node_modules, .venv and __pycache__ in .dockerignore so the COPY . . cannot undo the install that just ran correctly. When cross-building, use docker buildx build --platform linux/amd64 so the install step itself executes for the target platform.

Trimming the Node image

onnxruntime-node is what sets your image size, not the guardrail package. It ships prebuilt binaries for every supported platform in one tarball — in the 1.29 build, around 287 MB installed, of which linux/x64 is 43 MB and linux/arm64 is 25 MB. A container needs exactly one of those.

RUN npm ci --omit=dev \
 && find node_modules/onnxruntime-node/bin/napi-v6 -mindepth 1 -maxdepth 1 \
      ! -name linux -exec rm -rf {} + \
 && find node_modules/onnxruntime-node/bin/napi-v6/linux -mindepth 1 -maxdepth 1 \
      ! -name x64 -exec rm -rf {} +

That reclaims roughly 245 MB. It reaches into another package's internal directory layout, so treat it as an optimization to re-verify after an onnxruntime-node upgrade — which is exactly what the build-time load check above will do for you.

The Python side needs no equivalent trick: pip fetches only the wheel matching the image, roughly 80 MB installed for onnxruntime plus 34 MB for numpy.

Pre-warming

load() builds an ONNX Runtime session and reads a handful of small files. Call it once, at process start, and hold the instance.

Python

from contextlib import asynccontextmanager

from fastapi import FastAPI

from cognipeer_guardrail import Guardrail


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.guard = Guardrail.load()
    app.state.guard.scan("warmup")      # first inference allocates ORT's arenas
    yield


app = FastAPI(lifespan=lifespan)

Node

const { Guardrail } = require("@cognipeer/guardrail");

// Module-scope promise: every caller awaits the same single load.
const ready = (async () => {
  const g = await Guardrail.load();
  await g.scan("warmup");             // first inference allocates ORT's arenas
  return g;
})();

module.exports = { ready };

The throwaway scan("warmup") matters more than it looks. load() creates the session, but ONNX Runtime allocates its memory arenas and picks its kernels on the first run(), so the first real inference is measurably slower than the steady state. Spending that once at startup keeps it out of a user's request.

Do not load per request

This is the single most common deployment mistake with this package. load() is milliseconds and scan() is a fraction of one; loading per request inverts that ratio and multiplies your memory footprint by your concurrency. Under a pre-forking server, load after the fork — in a lifespan handler or a post_fork hook — so each worker owns its session rather than inheriting one across fork().

One loaded instance serves a whole thread pool: ONNX Runtime sessions support concurrent run() calls, and nothing in the encode path holds mutable per-call state. In Node the instance is shared across the event loop by construction.

Readiness

Make the pre-warm the thing your readiness probe waits for, so traffic never lands on a process whose session is not built yet:

Python

@app.get("/readyz")
async def readyz():
    return {"ready": hasattr(app.state, "guard")}

Node

let warm = false;
ready.then(() => { warm = true; });

app.get("/readyz", (req, res) => res.status(warm ? 200 : 503).json({ ready: warm }));

Memory and latency

Measured on a quiet Apple M-series machine (14 cores, macOS, onnxruntime 1.30.0, CPU execution provider) with the shipped Python package, scanning 1,000 benchmark test texts (mean length ~150 characters) end to end via scan(), with nothing else competing for cores. Numbers vary by CPU — this is one representative machine, not a guarantee — and a busy host (including one sharing cores with another workload) will push the tail (p95/p99) out further than these figures show.

scan(), end to end, default thread pool	p50 0.45 ms · p95 1.64 ms · p99 3.85 ms
scan(), end to end, intra_op_num_threads=1	p50 1.25 ms · p95 2.63 ms · p99 11.23 ms
predict_proba, batched, single thread	667 texts/s
Process RSS after warm-up	~120 MB (Python) · ~85 MB (Node)

How to read those:

• Latency. With ONNX Runtime's default thread pool a scan lands around p50 0.45 ms, p95 1.64 ms, p99 3.85 ms on this machine. Pinning the session to one thread (intra_op_num_threads=1, see the tip below) is slower here — p50 1.25 ms, p99 11.23 ms — because the model can no longer parallelize its own convolutions across cores; the benefit of pinning is predictable resource usage under a worker-per-core layout, not raw speed on an otherwise-idle machine. Budget in the low single-digit milliseconds per scan on comparable hardware, more on a busy shared vCPU — there is no batching queue or GPU scheduling to reason about either way.
• Memory. The model is 1.4 MB. The RSS figures above are almost entirely ONNX Runtime and the language runtime, and they do not grow with traffic. Adding the guardrail to an existing service adds tens of megabytes, not hundreds. Budget per process, not per request.
• load(). Node's native-addon initialization is the slower of the two, paid once. It is a reason to pre-warm, not a reason to prefer Python.

For capacity: a single core sustains roughly 667 texts/second through batched predict_proba calls with the session pinned to one thread. Multi-threaded and multi-core deployments push that higher, but the exact number is workload- and CPU-dependent — benchmark on your own hardware rather than treating this figure as a ceiling. For almost any chat workload the guardrail is not the thing you scale. If you are scanning long documents rather than chat turns, the windowing helper in the Python and Node guides turns one document into one batched inference.

Constrain the thread pool in small containers

ONNX Runtime's default intra-op thread pool sizes itself to the machine's cores, which oversubscribes badly in a container limited to a fraction of a CPU or in a worker-per-core layout. The Python guide shows how to build the session with your own SessionOptions and pass it to the constructor. Pinning to one thread costs some raw latency on an unconstrained machine (see the table above), but avoids the far worse oversubscription tax in a fractional-CPU container — measure both on your target hardware before choosing.

Air-gapped installs

Two independent problems: getting the package into the environment, and telling it where the model lives. Only the first one is real work.

Getting the package in

Python

# On a connected machine, for the TARGET platform:
pip download cognipeer-guardrail \
  --platform manylinux2014_x86_64 --python-version 3.12 \
  --only-binary=:all: -d ./wheelhouse

# Transfer ./wheelhouse, then, on the air-gapped host:
pip install --no-index --find-links=./wheelhouse cognipeer-guardrail

Node

# On a connected machine:
npm pack @cognipeer/guardrail        # → cognipeer-guardrail-0.1.0.tgz
npm pack onnxruntime-node

# Transfer the tarballs, then, on the air-gapped host:
npm install ./cognipeer-guardrail-0.1.0.tgz ./onnxruntime-node-*.tgz

The model rides along inside those artifacts. Once the package is installed there is nothing further to fetch — Guardrail.load() with no arguments works on a host that has never had a route to the internet.

--platform is not optional for a cross-platform pip download

Without it, pip resolves the onnxruntime wheel for the machine doing the downloading. A wheelhouse built on a Mac and carried to a Linux host installs nothing usable. The same reasoning applies to npm pack — pack onnxruntime-node from an environment matching the target, or let the target host install it from an internal registry mirror.

Pointing at a mounted model directory

When policy says model artifacts live outside the application package — a read-only volume, a directory your build pins by digest, a mount your security team controls separately — pass its path:

Python

g = Guardrail.load(model_dir="/opt/models/guardrail")

Node

const g = await Guardrail.load({ modelDir: "/opt/models/guardrail" });

The directory must contain model.onnx, vocab.json, manifest.json and thresholds.json — plus threshold_profiles.json if you intend to use profile="balanced" or "sensitive" — copied together from one export. Category order, max_len and the thresholds are read from those JSON files rather than hard-coded, so a directory assembled from two different exports produces silently wrong scores instead of an error. Copy the directory as a unit and check it into your artifact store as a unit.

FROM python:3.12-slim
# … install as above …
ENV GUARDRAIL_MODEL_DIR=/opt/models/guardrail
VOLUME /opt/models/guardrail

import os

g = Guardrail.load(model_dir=os.environ.get("GUARDRAIL_MODEL_DIR") or None)

Passing None falls back to the copy bundled in the package, so the same image runs both ways: bundled by default, mounted when the environment says so.

SaaS versus on-prem

There is no difference in behaviour, and that is the useful property.

The same wheel and the same npm tarball run in a multi-tenant SaaS process and in a customer's air-gapped datacentre. Same weights, same thresholds, same scores, bit for bit — because nothing about the classification path depends on where it runs. There is no hosted tier with a better model behind it, no server-side threshold tuning, and no request that leaves the process.

Practically, that means:

Question	Answer
Does scanned text leave the process?	No. Normalization and inference are in-process; nothing is logged or transmitted by the package.
Do the two environments score identically?	Yes, for the same package version. Pin the version and a SaaS score reproduces on-prem.
Can an on-prem install fall behind?	Only by version. Upgrading the package is the only way to change the model.
What has to be allowlisted in the firewall?	Nothing at runtime. Only your package registry, at install time.
Where does per-customer policy live?	In your threshold overrides, not in the model — see Python and Node.

The last row is the one to design around. Because the calibrated thresholds ship with the model and the raw scores are always returned, a per-tenant policy is a configuration object in your application, not a fine-tune. A stricter customer gets a different comparison against the same nine numbers — which means you can change their policy without redeploying a model, and re-evaluate it against scores you already logged.

Compliance questions have a short answer

For a review that asks where user text goes: it stays in the process, the model is a file inside the installed package, and the container needs no egress. If that needs demonstrating rather than asserting, run the test suite in a container with networking disabled.

Next

• Python — the full API, batch scoring, threshold overrides, FastAPI
• JavaScript and Node — CommonJS and ESM, Express, native binaries
• Getting started — install and first scan
• The model — architecture, normalization, calibration
• Limitations — what it does not catch