Private retrieval-augmented AI designed so no-egress is proved by test

The problem

Compliance work draws on public guidance such as DISA STIGs, CVE records and NIST catalogs, and often on an organization’s own documents. Sending those to an outside inference service creates a data flow that must be justified.

Local models remove the provider, but “it runs locally” is an assertion. An authorization package or a customer questionnaire needs evidence of what can leave and what stops it.

What was built

The first system is a working retrieval-augmented service. Ingest filters, chunks and embeds documents locally into vector and keyword indexes. A query passes a relevance gate, hybrid ranking and grounded synthesis at temperature zero; answers carry numbered citations. The only inference endpoint in the source is local. No cloud provider SDK is installed, and library telemetry is off.

The successor is a rebuild designed around proof; what follows is design, and its services are stubs. It is built for public STIG, CVE and NIST content. The design also defines non-public source classes, which ingest is to refuse until the egress gate passes. In the design, every service is to sit on an internal-only container network with no published ports or default route, reached only through a host loopback forwarder confined by address deny lists.

Design decisions

Authentication even on loopback

Any local process can reach a loopback port, so binding to localhost is not access control. The first system requires an authenticated session on its routes. The successor’s design uses scoped keys, refuses weak ones at start-up, and rejects requests with an unexpected Host header, which defeats DNS rebinding.

Keyword search ranks; it never decides

In one test on the first system, the answer was one exact string present in 53 chunks. Dense retrieval put one answer-bearing chunk in the top 10, keyword search put five and the fused ranking three. Embeddings match topic; identifiers are lexical tokens, so the tokenizer keeps dotted and hyphenated identifiers whole, and the successor’s keyword spike confirmed that control IDs such as AC-2(3) come back whole. Recall on control identifiers is not yet measured.

Keyword search matches any shared token, so it never opens the gate. When nothing clears the dense threshold, a fixed refusal returns without a generation call. Because fusion scored below keyword search alone, the successor adds an exact-identifier lookup ranked ahead of fused results.

Model answers are not sources by default

Indexed answers would later be cited as sources while every signal still said “grounded”. Generated content is stamped with its provenance and excluded from the default search set.

Secrets filtered before embedding

Files matching secret patterns are excluded at ingest, and free text is redacted for secret formats before embedding. The index is treated as being as sensitive as its sources.

Egress proved first, and the proof expires

The build order puts a full egress test ahead of any non-public text. It is to capture packets through start-up, a fixture ingest and queries, and probe each container for DNS and outbound connections. The same probes from a throwaway container on a normal network must succeed, or the test is void. A canary must appear in the API response and nowhere in the captures.

Each run deletes the gate file. Only a separate acceptance step, after a pass plus a passing restore round trip and log canary on the same commit, writes it again, with hashes of the deployment files and image lockfile. Ingest is specified to refuse non-public sources unless those hashes match the running tree.

AI coding agents kept away from non-public text

As a general practice, AI coding agents that help build a system like this are to see counts, IDs, hashes and pass/fail lines only. Their key is to be scoped on the server to exclude non-public classes, failing closed, and a canary test is to check that logs hold no chunk text. The key scope and log canary are the enforcing mechanisms.

Evidence and limits

• The first system’s evidence is a source review of every network call. The successor’s design adds a packet-level test as its first gate.
• Two adversarial review rounds on the successor’s design raised 26 blocking findings, all answered before build. At the scaffold milestone 109 unit tests passed. A network spike found no default route or IPv6 address in the containers.
• In the scaffold review, the seven must-fix findings were fixed in code, among them a probe that would have reported failure when containment worked; the full egress test is to exercise that fix.
• The full egress test is the next milestone, and there are no answer-quality numbers yet.

Where it applies

For a payer or CMS contractor, an AI tool over sensitive documents is a new information flow in the system security plan. The method is designed to turn boundary, least-privilege and integrity claims into dated test output tied to a commit, which supports audit readiness.

For a manufacturing supplier, sending export-controlled data to an outside inference service needs controls and justification that a local system avoids. When a customer questionnaire asks where AI tools send data, the answer can be a test result.

For fractional security leadership, the agent-separation pattern fits any team using AI near regulated data: decide what the agent may see, enforce it on the server, and test it with a check that can fail.

Independent work by Wade Blackwell. Each write-up is written from scratch for this site, without client, employer or personal data. Names, hostnames, addresses and configuration detail are left out on purpose.