Building a HIPAA Compliant Private LLM for Healthcare Data

Most HIPAA guidance written for AI buyers argues that private deployment is a good idea. This piece assumes that argument is settled and moves to the build. The HIPAA Security Rule gives you three groups of safeguards — Administrative, Physical, and Technical — and each one maps onto specific decisions about model weights, GPU hosts, key custody, log destinations, and retrieval scope. What follows is a specification a technical lead can hand to a compliance committee without translation.

Two numbers set the stakes. Healthcare remained the costliest industry for breaches in 2025 at an average of $7.42 million, the fourteenth consecutive year it topped IBM's ranking. And the Tier 4 civil monetary penalty ceiling reached $2,190,294 per violation for willful neglect not corrected. An LLM that leaks protected health information is not a product problem. It is a reportable event.

Scope the BAA Before You Scope the Model

HIPAA liability travels with the data, not the software. Any vendor that creates, receives, maintains, or transmits ePHI on behalf of a covered entity is a business associate and needs a signed BAA. That rule has two consequences for the stack.

First, the model provider is in scope only if the model provider sees prompts or completions. A locally hosted open-weight model does not. The compliance perimeter shrinks back to the hospital's existing audited perimeter because there is no model provider in the inference loop. That is the structural reason regulated buyers prefer open weights.

Second, every component that touches ePHI needs its own BAA or needs to be owned. Inventory the retrieval index, the embedding model, the vector store, the inference server, the log pipeline, the key manager, the identity provider, the backup target, and anything a RAG agent can call. If a component ships telemetry to a vendor cloud, either turn it off or get a BAA. The LLM Appliance approach collapses most of this list into one audited enclosure, which is why appliances are winning the compliance review even when they lose on raw spec sheets.

Pick the Weights, Then Pick the GPUs

The 2026 default stack for on-prem healthcare serving is Llama 3 70B (or a healthcare fine-tune of it), served through vLLM, with Mixtral 8x22B as the mixture-of-experts option for high-volume routing. Llama and Mistral have been evaluated as viable open-source alternatives to GPT-4o for user-facing healthcare assistant tasks, with the practical advantages of transparency and customization. Domain-specialized checkpoints exist — MedGemma, BioMistral, OpenBioLLM — but independent evaluation has found medical fine-tunes can hallucinate more frequently than general models, with Llama 3 showing the lowest hallucination rate in one cancer-communication study. Treat clinical fine-tunes as candidates for narrow tasks, not drop-in replacements for a general base model.

GPU sizing follows context length and concurrency, not parameter count alone. A 70B model at FP8 fits on a pair of H100s or a single H200 for low-concurrency clinical note workloads; 200 clinicians hitting a RAG assistant simultaneously is a different cluster. The cost math behind that distinction is covered in a separate GPU cluster cost breakdown, which is worth reading before specifying hardware.

Technical Safeguards: Encryption, Audit, Authentication

45 CFR 164.312 lists five technical safeguards: access control, audit controls, integrity, person or entity authentication, and transmission security. The current rule marks some implementation specifications as "addressable" rather than required. The proposed 2025 amendment removes that distinction.

Encryption. AES-256 is table stakes; the compliance artifact is the module, not the algorithm. The requirement is not "use AES-256" but "use a cryptographic module validated by NIST to correctly implement AES-256", which means every component in the inference pipeline that handles PHI — API gateway, model host, vector database, output cache, log store — needs to call FIPS 140-3 validated libraries. An AI data path that uses unvalidated AES-256 is not FIPS-compliant even though the ciphertext looks identical.

Audit controls under 164.312(b). Logs are not optional telemetry. Capture, at minimum: prompt text, retrieved document IDs, model and version, user identity, completion hash, tool calls, and timestamp. Ship them to tamper-evident storage with object lock or append-only semantics. Time-synchronize across hosts. Restrict administrative access to the log store from the people who can access the model. Review on a schedule, not on incident. Shared accounts break every one of these controls at once, which is why unique user identification is a required specification, not an addressable one.

Transmission. TLS 1.3 for every hop, including the internal hop from the retrieval service to the inference server. mTLS between services. No ePHI over SMS, no ePHI in email bodies, no ePHI in third-party monitoring webhooks.

De-Identification and the Minimum Necessary Rule in RAG

The HIPAA Privacy Rule's minimum necessary standard applies to retrieval the same way it applies to a chart pull. A RAG system that returns a full patient record when the clinician asked about a medication interaction has over-disclosed by default. Build the retrieval filter against the role, not just the query.

For analytics, training corpora, and non-clinical workflows, de-identify first. The Safe Harbor method under 45 CFR 164.514(b)(2) requires removal of 18 categories of identifiers, plus a positive attestation that the covered entity has no actual knowledge the residual data could re-identify an individual. The 18 categories are not a suggestion:

• Names, geographic subdivisions smaller than a state, and all date elements other than year for anyone over 89
• Phone, fax, email, URL, and IP address
• Social Security, medical record, health plan, and account numbers
• Certificate, license, vehicle, and device identifiers
• Biometric identifiers and full-face photographs
• Any other unique identifying number, characteristic, or code

Expert Determination is the alternative route when Safe Harbor strips too much signal, but it requires a qualified statistician's documented analysis. For training data in particular, remember that LLMs can memorize and reproduce sensitive training content including personally identifiable information, which is why de-identification happens before fine-tuning, not after.

The 2025 Security Rule Rewrite Changes the Timeline

On January 6, 2025, HHS OCR published a Notice of Proposed Rulemaking that would make encryption at rest and in transit, plus MFA, mandatory rather than addressable — the first major Security Rule update since 2013. If finalized, regulated entities get 240 days to come into compliance: 60 days until effective plus an additional 180 days. Any private LLM program with a longer runway than that should assume the new baseline is already in scope and build to it.

Practically, that means designing the stack with MFA on administrative access, FIPS 140-3 validated encryption for ePHI at rest and in transit, documented asset inventories for every component that touches ePHI, and vulnerability scanning cadence on the inference hosts. These are the same controls that appear in a SOC 2 audit; the overlap across SOC 2, HIPAA, and GDPR is substantial and worth exploiting during policy drafting.

Administrative and Physical Safeguards Are Still the Hard Part

Technical controls fail quietly. Administrative controls fail loudly, and most Office for Civil Rights findings sit in this category. The administrative requirements under 164.308 — risk analysis, workforce training, sanction policy, access authorization, contingency planning — apply to the LLM as they apply to the EHR.

Risk analysis is the specific artifact that regulators ask for first. Document the data flows: where PHI enters the model, where it is cached, where it is logged, how long it persists, who can read it, and what happens on key rotation. Workforce training should cover prompt hygiene, which is a new failure mode — a clinician pasting a full chart into a general-purpose assistant on a personal device is an incident even if the on-prem model is clean. The two feed each other: training reduces risk, risk analysis directs training.

Physical safeguards under 164.310 extend to the rack. GPU hosts belong in facilities with controlled access, environmental monitoring, media disposal procedures, and workstation controls. Air-gapped deployments add a stricter media handling procedure because every model update and every retrieval index refresh is a physical event.

Operate It Like an EHR, Because That Is What It Is

A private LLM that reads clinical notes is a system of record for compliance purposes. It deserves the same change management, the same quarterly access review, the same incident response playbook, and the same restore test as the EHR. Healthcare breaches already take an average of 279 days to identify and contain, roughly five weeks longer than the global average, which is a decent argument for building detection into the LLM pipeline from day one rather than bolting it on after the first audit.

The build spec is not where most programs fail. The first ninety days are, and the pattern of failures is consistent enough to plan around. The checklist above is the floor. What distinguishes a HIPAA-compliant private LLM from a HIPAA-defensible one is whether the organization still runs the controls in month twelve.