Passing an EU AI Act Conformity Assessment on a Self-Hosted LLM

Most CISOs still treat the EU AI Act as a policy exercise. That was defensible a year ago. It is not defensible now that the Digital Omnibus has locked in application dates for high-risk systems and national market surveillance authorities are staffing up to enforce them. The uncomfortable question is no longer whether your self-hosted LLM falls into scope. It is whether the evidence file on your shared drive would survive a knock on the door.

This piece walks through what a real EU AI Act conformity assessment looks like for a privately deployed model in 2026, and which artifacts a self-hosted architecture actually produces on its own. If you already run inference inside your VPC or on an LLM appliance, you are closer than most SaaS buyers. But closer is not the same as compliant.

What the Act Actually Requires You to Produce

Article 43 is the gate. It says a high-risk AI system cannot be placed on the Union market or put into service until conformity is demonstrated through one of two procedures. For systems in Annex III points 2 through 8 (employment, credit, essential services, critical infrastructure, education, law enforcement, migration, justice), the provider follows internal control under Annex VI. No notified body is involved unless you have skipped or diverged from harmonised standards. For Annex III point 1 biometric systems, or any case where harmonised standards are not fully applied, the third-party route under Annex VII kicks in.

Most enterprise LLM deployments land in the internal control lane. That sounds lenient. It is not. The self-assessment path shifts the entire evidentiary burden onto you: the quality management system under Article 17, the technical documentation under Annex IV, the risk management file under Article 9, the post-market monitoring plan under Article 72, the incident reporting workflow under Article 73, and a signed EU declaration of conformity that a national authority can demand at any time.

Two dates matter. The Digital Omnibus on AI, published 24 July 2026, pushed standalone Annex III high-risk obligations from 2 August 2026 to 2 December 2027, and Annex I product-embedded systems to 2 August 2028. That is breathing room, not a reprieve. Notified bodies remain scarce, harmonised standards are still being finalised, and the queue for external assessment when you need one is already lengthening.

The Nine Sections of Annex IV, Translated Into Files

Annex IV lists nine categories of technical documentation: general system description, design specifications, monitoring and control instructions, risk management, changes over the lifecycle, standards applied, EU declaration of conformity, performance evaluation, and the post-market monitoring plan. For a self-hosted LLM this is not abstract. Each section maps to artifacts your platform team already generates, provided you are logging them intentionally.

• System description: model card with weights hash, tokenizer version, quantization scheme, container image digest, GPU topology, and the inference runtime commit. This is a build manifest, not marketing copy.
• Design and development: base model provenance, fine-tuning dataset lineage, RAG index composition, retrieval policy, and any guardrail model in the pipeline. If you fine-tuned on customer data, the DPIA lives here.
• Monitoring, functioning and control: prompt and response schemas, rate limits, refusal categories, and the human-in-the-loop trigger conditions.
• Risk management: the Article 9 file, refreshed on a defined cadence, with red-team findings attached. Our own notes on private LLM red teaming cover the format auditors expect.
• Lifecycle changes: signed changelog of weight updates, prompt template revisions, and index rebuilds, with rollback evidence.
• Standards applied: mapping to ISO/IEC 42001, ISO/IEC 23894, and any harmonised standard once published.
• Declaration of conformity: the signed one-pager, kept current with the actual system state.
• Performance metrics: accuracy, calibration, robustness, and bias evaluations against the population you actually serve.
• Post-market monitoring plan: the living document under Article 72.

None of this is boilerplate. Auditors read the design section against the changelog against the monitoring logs, and they notice when the three do not agree.

Why Self-Hosted Architecture Is a Genuine Advantage Here

Public API deployments create a documentation gap the provider cannot close on their own. You do not control the base model card. You do not see the training data. You cannot attest to weights that a vendor may silently swap. The hidden risks of public AI APIs compound when a regulator asks who is the provider under Article 3(3) and both parties point at each other.

A self-hosted deployment collapses that ambiguity. You are the provider. You hold the weights, the inference stack, and the audit surface. The Annex IV evidence lives on your infrastructure, generated by systems you already run for SOC 2 and ISO 27001. Concretely, a private stack gives you five compliance-native artifacts by default:

• Immutable prompt and response logs written to WORM storage, addressable by request ID and user identity.
• Version-pinned model artifacts with cryptographic hashes, correlated to every inference event.
• Network-level evidence that no inference traffic leaves the perimeter, from VPC flow logs or the appliance's egress deny-list.
• Deterministic access control tied to your existing IdP, satisfying the human oversight provisions of Article 14.
• A single tenant of record for data residency claims under Chapter III, Section 2.

The alternative is reconstructing this evidence from a vendor's SOC 2 report and hoping the assessor accepts a third-party attestation as a substitute for your own logs. Some will. Many will not.

	Public API	Self-Hosted / VPC
Weights and version pinning	Vendor-controlled	Provider-controlled hash
Prompt / response logs	Partial, vendor-hosted	Full, WORM-stored
Egress evidence	Trust vendor SOC 2	VPC flow logs
Fine-tune lineage	Limited visibility	Full DPIA and lineage
Article 14 human oversight	Depends on vendor UI	Enforced at IdP
Provider identity (Art. 3(3))	Ambiguous	Unambiguous

Building the Post-Market Monitoring Loop

Article 72 is where most programs will fail their first audit. It requires an actively operated monitoring system, documented in a plan that forms part of the Annex IV file, capable of detecting drift and evaluating continuous compliance across the system's lifetime. A quarterly review meeting does not satisfy this. Neither does an accuracy dashboard that no one is paged on.

What auditors want to see is a closed loop: telemetry collected from the live model, a defined set of drift and safety signals, thresholds that trigger human review, corrective action tracked to closure, and a feedback path into the risk management file. The signals themselves are the interesting engineering problem. For a private LLM they typically include distributional shift in prompt embeddings, refusal-rate deviation, retrieval hit-quality decay, calibration drift on labeled probes, and jailbreak-signature detections. The subject is covered in more depth in our piece on AI drift in production models.

Article 73 layers incident reporting on top. Serious incidents must be reported to the national market surveillance authority on tight clocks. Your monitoring system should classify events against those thresholds automatically, because reconstructing timing after the fact is what regulators use to establish negligence.

The Penalty Structure That Should Shape Your Board Conversation

The economics of compliance under this regime are asymmetric. Fines for prohibited-practice breaches reach €35 million or 7% of worldwide annual turnover, whichever is higher. High-risk violations sit below that ceiling but are still material. For general-purpose model providers, the AI Office can impose fines up to 3% of global annual turnover or €15 million. Providers of fully open-source GPAI models get a narrow exception under Article 53 for openly published weights and architecture, but the exception does not apply once systemic-risk thresholds are crossed.

For a regulated enterprise, the practical implication is that the cost of assembling the conformity file is dwarfed by the cost of a single failed audit. That reframes the build-versus-buy discussion. A custom LLM deployment designed against Annex IV from the beginning is cheaper than retrofitting a public API integration whose evidence surface you do not own.

What to Have Ready Before the Auditor Arrives

Treat the conformity file as a product, not a policy document. It has owners, a release cadence, and a defect tracker. The minimum viable state before you file the declaration of conformity: an Annex IV binder that is current within thirty days of the live system state; a QMS mapped to Article 17 that references your existing ISO 27001 and, ideally, an ISO/IEC 42001 program (certifications run three years with annual surveillance audits); logging pipelines with tamper-evident storage; a named human oversight officer with authority to halt inference; a post-market monitoring plan under active operation, not draft; and an incident response runbook that hits the Article 73 clocks.

Self-hosting does not exempt you from any of this. It gives you the raw material to satisfy it. The gap between raw material and a defensible file is the work of a compliance-aware infrastructure team, and it is where most 2027 deadlines will be won or lost.