← Back to blog

Recall first: picking a PII model that runs on-device

For a graduate project I built a system that detects and censors personally identifiable information — names, addresses, SSNs, dates of birth, 54 entity types in all — entirely on the user's machine. No text leaves the device. That one constraint shaped almost every decision that followed.

The metric comes before the model

The two ways this system can fail are not equally bad. A false positive blacks out a word that was harmless; mildly annoying. A false negative leaks someone's SSN. So before comparing any models I decided to weight recall heavily: missing PII is worse than over-censoring.

Four models, one held-out set

I benchmarked BERT, RoBERTa, DistilBERT, and DeBERTa on the same held-out set, so the comparison was about the architectures and nothing else. DeBERTa won with a 98.1% F1 and 97.9% recall. Its disentangled attention encodes content and position separately, which is a real advantage when the hard part of the task is finding exactly where an entity starts and ends.

Making it run locally

A model that wins the benchmark but needs a server defeats the purpose. I exported the model through ONNX and got inference under 100ms on-device. Around it sits a Streamlit demo, Tesseract OCR so PDFs and images can be processed too, and SHAP/LIME explanations so a user can see entity-level confidence instead of trusting a black box with their private data.

The lesson

Privacy protection at scale is an engineering problem, not just a legal one. A policy can say personal data must not leave the machine; only the architecture can make that true. And the order matters: decide which error you cannot afford, pick the metric that punishes it, and only then pick the model.