Clinical AI Is High-Risk. Now What?

In late April I sat across the boardroom table from the CIO of a European hospital group — about 4,000 beds across three countries, an AI portfolio of around twenty active production systems, and an internal audit that had just landed on the Chair's desk. The audit's conclusion: the group had eight months to bring clinical AI into EU AI Act compliance, no budget allocated for the work, and no inventory of which systems were even in scope. The CIO's question to me was the one I now hear in some form on every healthcare engagement: "How bad is this, and where do we start?" The honest answer is that the EU AI Act layered on top of the MDR creates compounding obligations for hospital AI, and the engineering work is substantially bigger than most legal teams have signalled to clinical leadership. Here is what that work actually looks like, drawn from the gap assessments we've run at hospital groups across the EU, UK and Gulf over the last nine months.

What's actually in scope (and why most clinical AI is)

Annex III, point 5(a) covers AI systems used to evaluate eligibility for essential public services — which captures most patient-triage and benefits-eligibility AI deployed in EU health systems. Annex III, point 1(a) covers biometric identification used for access control to wards or clinical systems. The MDR (Medical Device Regulation) covers any AI that's part of a medical device, which captures diagnostic AI. The combination is that essentially every clinically-relevant AI workload in a EU healthcare setting is high-risk under the AI Act, the MDR, or both. The 2 August 2026 deadline applies to the Annex III categories; the Annex I systems get an extra year. The clock is short. In the hospital group I mentioned, the inventory turned up 11 of 20 systems in scope of Annex III, 3 in scope of MDR with AI Act overlay, and 6 out of scope. The 11 became the focus of a 90-day remediation programme.

The four moves that buy the most compliance per engineering hour

First, classify every clinical AI workload by risk tier and produce the risk-classification artefact. Most hospital AI portfolios contain 20–40 production AI components; the inventory work is non-trivial but it's the prerequisite for everything else. Second, stand up an Annex IV technical-documentation pipeline that generates documentation from source — model cards, training-data provenance, evaluation reports, version history — rather than maintaining it in Word documents that drift. Third, instrument the inference path to log the full input vector, retrieved context (for RAG-based systems), model version, response and any downstream clinical action, with retention aligned to your regulatory minimum (typically 5–7 years for clinical decision support). Fourth, redesign the clinician-review workflow to surface model reasoning and feature contributions rather than just the recommendation, so the reviewer is exercising effective oversight rather than rubber-stamping.

Where most clinical AI programmes are behind

The gap analysis we run on health-system AI portfolios surfaces three recurring issues. Patient-safety review processes typically cover diagnostic accuracy and clinical efficacy, but rarely cover the fundamental-rights risks (bias against protected groups, fairness drift, disparate impact across populations) that Article 9 risk management demands. Training-data lineage is documented at training time but the documentation degrades within 6 months as the system updates; Article 11 needs the documentation to stay current. The override workflows that clinical informaticists build for clinicians typically surface the model's recommendation but not its reasoning — that's what fails Article 14 effective-oversight review.

The case for sovereign deployment in healthcare

The argument we make to CIOs of EU health systems is that the EU AI Act obligations are easier to satisfy on sovereign infrastructure than on hyperscaler-hosted ones. Articles 10 (data governance), 12 (record-keeping), and 14 (human oversight) all involve evidence about data, logs, and oversight workflows. When the data, logs and workflows live inside the hospital's own infrastructure, the supervisor's question "can you demonstrate the controls" gets a 4-hour answer. When they live with a hyperscaler under BAA, the supervisor's question becomes a multi-week document request. Sovereign isn't uniquely possible — it's just less expensive in evidence terms.

The Annex III categories most relevant to hospital operations

Patient triage and admission prioritisation — Annex III 5(a). Eligibility determination for public services or benefits — Annex III 5(a) and 5(d). Biometric identification for hospital access control — Annex III 1(a). HR systems used for staff scheduling, performance management, recruitment — Annex III 4(a) and 4(b). Document-extraction systems used for diagnosis support — typically MDR Annex I rather than EU AI Act Annex III, but check the intended-purpose statement carefully because both can apply. Patient-facing chatbots that produce health-information advice — usually limited-risk under Article 50, but if the bot is making any eligibility-style determinations it becomes Annex III.

Reference architecture that satisfies the obligations

The architecture we deploy at hospital customers: sovereign LLM serving (Llama 3.3 70B or domain-tuned 8B model) on hospital-controlled GPUs; RAG grounded on the hospital's clinical guidelines and policy corpus with citation-injected answers; the agent runtime sits behind a permission gate at every tool boundary; Langfuse in-perimeter for audit; integrations to Epic / Cerner / Meditech via FHIR for clinical data, with all PHI staying inside the hospital network. Annex IV technical documentation is generated from source on every model update. The clinician override workbench surfaces model reasoning, population-comparison context, and similar-case precedent so reviewers exercise effective oversight.

The 90-day path for a hospital CIO

Days 1–30: inventory all clinical AI systems, classify each against Annex III + MDR, identify gap against Articles 9–15 per system, write the remediation plan and submit to clinical governance + ExCo. Days 31–60: stand up Annex IV documentation pipeline for the top 5 most-used clinical AI components. Pilot the audit-layer extension and the clinician-review workbench redesign on a single workload (typically the prior-auth or triage workflow). Days 61–90: extend to remaining systems, run the eval suite against historical clinical data, produce the supervisor-ready report. By 2 August 2026, be able to hand a regulator or internal audit complete answers on demand. The MindMap Digital EU AI Act whitepaper covers the full article-by-article mapping; for hospital-specific architecture see /ai-for-healthcare.