Using public AI models on sensitive material without disclosing the values

The problem

Public AI models are useful for reading a firewall export, a spreadsheet of records or a set of interview notes. The useful parts are what must not leave the machine: PHI, card data, personal data, CUI and export-controlled detail, and infrastructure secrets.

Whether any form of it may reach a third-party model is a contractual question first. Where tokenized material may, the rest is mechanical: get the analysis back without the values in the prompt, and show afterwards what was kept and destroyed.

What was built

An offline redaction tool for a Windows 11 desktop, built around four steps:

1. Sanitize. Each detected value in pasted text or a dropped file becomes a consistent token such as ⟦PERSON_004⟧, the same across every document in a project, so the model can reason across them.
2. Review. Every detection is highlighted with its type and confidence. Copy-out stays disabled until the review list has been scrolled to the end or the reviewer confirms having reviewed every detection.
3. Copy out. Approved text goes to the clipboard, marked for exclusion from Windows clipboard history and cloud sync, and is cleared after 60 seconds by default.
4. Rehydrate. The model’s reply is pasted back and real values return. Altered tokens are restored and marked; unresolved and possibly invented tokens are listed and left in place, never silently dropped.

Detection covers the HIPAA Safe Harbor identifiers that can appear in text, MBI, NPI and DEA numbers, check-digit-validated card numbers, and infrastructure data: addresses, host names, directory accounts, cloud identifiers, keys and passwords. Each project adds its own dictionary and allow-list. Inputs include plain text, network device exports, Word, Excel and text-based PDF.

Design decisions

• One encrypted vault per project. Values are encrypted individually with AES-256-GCM. Opening one needs an Argon2id-stretched passphrase and the Windows account’s data protection. Tokens never cross vaults.
• Tokens are sequential, not derived. The mapping exists only in the vault. This follows the HIPAA condition on re-identification codes: not derived from the individual’s information, and not disclosed.
• Offline is designed in and tested. A runtime guard, armed before any library loads, refuses and records connection attempts, name lookups, networking-library loads and process creation; one fixed, permit-gated launch of the file manager is the only exception. The guard sees Python-level operations, so tests and the self-test also check the process’s network endpoints and loaded libraries.
• No original value at rest outside the vault. Logs use fixed templates, errors are reduced to types and code locations, and documents are processed in memory. Two exceptions are stated: the approved, tokenized output, kept in plaintext for upload and still holding anything the review missed, and project names, protected only by the Windows account.
• Lock early. Idle timeout, Windows session lock, suspend, logoff and exit all drop the keys and clear every pane.
• Close-out produces evidence. Secure destroy overwrites, deletes and verifies the absence of every file the tool knows about, then writes a certificate of destruction: a PDF, a canonical JSON record and a detached Ed25519 signature checkable with standard OpenSSL. The record holds counts and digests, never a value. If any file survives, the report is titled incomplete. The certificate does not claim cryptographic erasure; its limitations name the copies the destroy cannot reach.

Evidence and limits

Accuracy is measured on synthetic corpora only, every set written by the author of the recognizers.

Set	Run	Protected recall	Precision
Template-generated	after tuning	1.000 (1,382 values)	1.000
Hand-written	first, blind	0.875 (91 of 104)	0.929
Blind, written after tuning	first, blind	0.878 (65 of 74)	0.985
Review probe set	first, before fixes	0.290 (49 of 169)	0.915

The template figure is optimistic: templates and recognizers were written together. The first blind score is the estimate to rely on for ordinary text. The probe set holds formats a code review found the detector missing; its 0.290 shows how far recall falls on unfamiliar material, which is why review is part of the control. After fixes the blind set scores 0.986 and the probe set 1.000, both now tuning numbers.

Other evidence:

• An end-to-end test drives the whole application with networking blocked. It fails on any recorded attempt or process-owned network endpoint at five checkpoints, with positive controls, and scans every file written and the process output for planted values.
• The installer runs a 17-check self-test; more than 690 automated tests cover the rest.

Stated limits:

• Tokenization protects identifiers, not context. A specific enough narrative can still identify a person.
• It is not a defence against a compromised user session.
• Destroy does not reach copies outside the tool’s knowledge (backup-software or hand-made copies, shadow copies, worn SSD cells); full-disk encryption is recommended.
• Images inside documents are not read.
• A certificate’s signature proves integrity; the signer’s identity rests on a key fingerprint exchanged separately.

Where it applies

Payers and CMS contractors. An AI-use control that can be described in a system security plan and tested: a review gate, an offline enforcement test, protection at rest and session lock, with the vault’s audit log, test results and signed destruction record as evidence.

Semiconductor and advanced-manufacturing suppliers. When a customer security questionnaire asks how AI tools handle customer data, the method answers for the identifiers in it: names, hosts, addresses, accounts and customer-specific terms are tokenized, reviewed and rehydrated per project, with a destruction record at close-out. The technical content of CUI or export-controlled material is not tokenized; whether it may reach a third-party model stays a contract and export-control decision.

Fractional security leadership. An AI acceptable-use policy needs a mechanism behind it: define what counts as sensitive, gate release on review, measure detection blind and report the weak number, and keep evidence of what was destroyed.

Independent work by Wade Blackwell. Each write-up is written from scratch for this site, without client, employer or personal data. Names, hostnames, addresses and configuration detail are left out on purpose.