PII redaction for privacy-aware document intelligence

Executive view

Redaction is a data-flow property, not a black rectangle

A document is not protected merely because its visible preview is masked. If the original name remains in OCR text, Markdown, JSON, a Lucene index, an embedding payload, a prompt, or an exception log, the system still exposes personal information.

CV and document-intelligence systems multiply data. One uploaded PDF can become extracted text, structured segments, metadata, keywords, summaries, vector chunks, search snippets, cached responses, benchmark records, and downloadable exports. Each representation has a different purpose and a different audience. Redaction must therefore be applied before the first unnecessary copy is created, then verified at every boundary where data are stored or disclosed.

The governing idea is data minimisation: retain information that is necessary for the stated task and restrict everything else. For recruitment retrieval, professional experience, skills, education, certifications, profile evidence, and language ability may support a search. A candidate's name, private contact details, exact home address, photograph, date of birth, or unrelated family information generally contributes little to matching capability with a role. Removing those fields early reduces the payload, the re-identification surface, and the number of values propagated downstream.

Threat model

Protect identities without destroying relevant evidence

The first design decision is whether the output must be anonymous, pseudonymous, or simply access-restricted. These states are not interchangeable. An anonymous publication must resist reasonable re-identification attempts and should not depend on a secret lookup table. A pseudonymous internal record may use a stable candidate token, but the identity mapping remains separately protected personal data. An access-restricted original can retain identity because an authorised operational task requires it, yet it should remain outside general search and model-processing paths.

Reference architecture

Build a layered redaction pipeline

A robust pipeline separates the restricted original from the privacy-filtered analytical record. The original can remain in an encrypted repository with narrow, logged access. A working copy passes through deterministic detection, contextual entity recognition, policy decisions, transformation, and validation before it reaches search or AI components. This placement matters: redacting only the final answer leaves sensitive content inside indexes and model infrastructure.

Local execution on Intel Arc Pro B70

In a workstation deployment, an Intel Arc Pro B70 can host OCR post-processing, entity detection, and local model inference before a derivative reaches retrieval or generative AI. The deployment keeps source documents inside protected storage and process boundaries, omits raw text from telemetry, fails closed on unavailable components, and passes only validated redacted records into downstream systems.

1. Inventory every representation

Map the full lifecycle: upload, parser output, OCR, intermediate Markdown, structured JSON, database fields, Lucene documents, vector chunks, prompt bodies, caches, telemetry, downloads, backups, and deletion queues. Record the purpose, owner, retention period, access group, and redaction state for each representation. This inventory prevents a common failure in which the user-facing copy is clean but derived stores retain the original content indefinitely.

2. Detect with complementary methods

Use exact rules for high-precision patterns such as email addresses, phone numbers, account numbers, and well-formed identifiers. Add multilingual named-entity recognition for people, addresses, organisations, and locations. Then apply contextual rules for CV-specific ambiguity: a personal name near the document header differs from a company name in employment history, while a location may be essential at country level but excessive at street level. No single detector is sufficient for narrative documents.

3. Decide through a versioned policy

Detection should produce candidates, not irreversible edits. A policy engine decides what to remove, replace, generalise, retain, or escalate according to document type, purpose, user role, destination, and confidence. For example, a hiring-search view might retain broad region and work-authorisation facts while suppressing private contact details. A support workflow may reveal contact information only after a user performs an authorised action.

4. Transform source-aligned text

Apply edits from the end of the text toward the beginning so character offsets remain stable. Preserve a provenance record containing entity type, source offsets, detector, confidence, policy version, transformation, and a keyed digest of the original value—not the value itself. If stable pseudonyms are required, derive them with a scoped keyed function so tokens cannot be correlated across tenants or unrelated purposes.

5. Validate before fan-out

Run a second detection pass on the transformed text. Block downstream publication when a high-risk pattern remains, when the parser failed, or when offset reconstruction is inconsistent. Only a validated record should be indexed, embedded, reranked, sent to a language model, rendered in a snippet, or exported. Prompts that tell an extractor to omit names can be useful, but they are not a security boundary: generated and heuristic systems can both leak source text.

Evaluation

Measure privacy and retrieval utility separately

A redaction system can fail in two directions. Under-redaction leaks information; over-redaction removes evidence needed for retrieval and review. Report both. Entity-level recall is central for privacy because every missed high-risk identifier matters. Precision captures unnecessary masking. Exact-match tests should be supplemented with semantic and linkage tests for indirect identifiers.

Build a manually reviewed test set that reflects languages, layouts, occupations, scanned documents, tables, headers, and malformed files found in production. Partition results by entity type. A single pooled score can hide weak recall for addresses or identity numbers behind easy email detection. Include adversarial cases such as spaced phone numbers, obfuscated email addresses, names embedded in URLs, image-only signatures, and identifying facts spread across several sections.

For a CV retrieval system, the best privacy configuration is not necessarily the one that removes the most text. It is the least identifying representation that still supports the declared task. Broad professional evidence can often remain useful after direct and indirect identifiers are removed. Where a quasi-identifier is relevant, reduce its precision rather than retaining the full value by default.

Production operations

Make redaction observable, versioned, and recoverable

Production controls determine whether the design survives contact with real users. Store originals and re-identification keys separately from analytical data. Use role-based access with purpose-specific permissions, short-lived credentials, encryption in transit and at rest, and immutable audit events for reveal, export, policy override, and bulk processing. Never place raw PII in audit messages; log document and transformation identifiers instead.

Version policies and detectors so every derived record can be traced to the rules that produced it. When a policy changes, queue affected records for reprocessing rather than silently mixing protection levels in the same index. Deletion must propagate to extracted JSON, search indexes, vector stores, caches, previews, temporary files, and backups according to an explicit retention design.

Route uncertain detections to a review queue. Reviewers can inspect detector spans, correct false redactions, and rerun the transformation. The interface should distinguish “not present,” “redacted,” and “extraction failed” so downstream code and operators receive an accurate processing state.

Operating checklist

• Define target fields, destinations, retention, and transformation mode before implementation.
• Keep originals outside general retrieval and AI-processing paths.
• Redact before indexing, embedding, prompting, caching, analytics, and export.
• Fail closed on parser errors, detector outages, invalid offsets, and unresolved high-risk entities.
• Test multilingual, scanned, malformed, and adversarial documents against reviewed ground truth.
• Measure retrieval utility and privacy leakage as separate outcomes.
• Test access control, correction, deletion propagation, and incident-response procedures.

PII redaction is most effective when it becomes part of the document contract: every stage knows whether content is original, pseudonymised, generalised, or cleared for a defined destination. That contract lets a privacy-aware system retain useful professional evidence without allowing identity data to spread invisibly through the retrieval stack.

Primary sources

Standards and regulatory context