Healthcare has a longer history of governing algorithms than almost any other sector, and it still finds AI difficult. The reason is that two systems of control meet here and neither was designed with the other in mind. Device regulation asks whether a product is safe and effective enough to be placed on the market. Health system governance asks whether this particular tool, in this hospital, with this population, in these hands, improves care. A model can pass the first and fail the second.
Teams get into trouble by treating a regulatory clearance as the end of the governance question. It is the beginning. A cleared device deployed onto a population it was never validated against, with no monitoring and no clinician training, is a governed product used ungovernably.
Is it a device?
This is the first question and it determines almost everything downstream. Software that supports a clinical decision may or may not be a regulated device depending on what it does and how much the clinician can independently review its basis.
The workable distinction: software that provides information a clinician can independently evaluate, alongside the basis for it, tends to sit further from device regulation. Software that produces a specific diagnostic or treatment output the clinician is expected to rely on, without practical means to check the reasoning, sits closer to it or squarely inside. Time criticality matters too, because a clinician cannot meaningfully review a recommendation in an emergency.
The FDA maintains its position on AI and machine learning in software as a medical device on a dedicated hub, which is the right starting point rather than a secondary summary. Two practical notes. Generative AI applied to clinical documentation is where the line is currently least settled, and organisations differ in where they draw it. And an administrative tool can become a clinical one through use: a scheduling model that begins to influence which patients are seen sooner has acquired clinical consequence regardless of how it was procured.
The PCCP: how a cleared model is allowed to change
The hardest structural problem in regulated clinical AI is that models improve, and the traditional device paradigm assumes a fixed product. Retraining on new data is often the right clinical thing to do and historically triggered a new submission.
The FDA's predetermined change control plan addresses this directly. On 3 December 2024 the agency published final guidance titled Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions, and held a webinar on it on 14 January 2025. The mechanism: a manufacturer specifies in advance what changes it intends to make, so those changes can be implemented later without a new marketing submission.
The guidance recommends a PCCP describe three things:
The PCCP mechanism is available across marketing pathways including premarket approval, the traditional 510(k) route, and De Novo. The FDA also maintains a page on the underlying guiding principles.
For a health system rather than a manufacturer, the PCCP is a procurement question. Ask whether the device has one, what it permits, and how you will be told when a change is made under it. A model that can change within an agreed envelope is a model whose behaviour may shift without any action on your side, which has direct implications for your monitoring.
The European overlay
In the EU, clinical AI can sit under both medical device rules and the AI Act, and the two are designed to interlock rather than compete. Software that is a medical device continues to be regulated as one. Where it is also an AI system falling into the AI Act's high-risk category, AI Act requirements apply alongside, and for products already subject to third-party conformity assessment the intention is that the assessments are combined rather than duplicated.
What this means practically for a manufacturer is that the technical documentation, risk management, data governance, logging, human oversight and post-market monitoring expectations of the two regimes need to be satisfied by one coherent set of artefacts. Building two parallel files is a common and expensive mistake. For a deployer, it means AI Act deployer duties apply to your use of the system on top of whatever device rules govern the product.
Serious incidents are the place where the regimes most obviously touch. Device vigilance duties and the AI Act's serious incident reporting can both be triggered by the same event, on different clocks and to different authorities. Map the overlap in advance; we cover the AI Act clocks in AI incident response and reporting.
Transparency to the clinician
A clinician asked to exercise judgement over a model's output needs to know something about the model. In the United States, certification requirements for health IT under the HTI-1 rule moved in this direction by requiring source attribute information to be made available for decision support interventions, so that a clinician can see something about where a recommendation comes from.
Treat that as a floor rather than a target. The information a clinician actually needs at the point of care is narrow and specific:
- What population was this validated on, and how does it compare with the patient in front of me
- What is the performance, expressed in a way that supports a decision rather than a benchmark table
- What are the known failure modes and the situations where it should not be used
- How confident is this particular output, if the system can express that honestly
- When was the model last changed
Delivered as a documentation pack, none of this reaches the bedside. Surfaced in the interface at the moment of the recommendation, it becomes usable. That is a design decision as much as a governance one, and it is where automation bias is either mitigated or reinforced.
Local validation, which does not transfer
A model that performs well at the institution that developed it frequently performs worse elsewhere, and the reasons are mundane rather than mysterious: different case mix, different measurement practices, different documentation habits, different equipment, different coding behaviour. External performance is evidence that the approach can work. It is not evidence that it works here.
A workable local validation sequence:
| Stage | What happens | Gate |
|---|---|---|
| Retrospective | Run the model on your own historical data with known outcomes | Performance overall and by subgroup within an agreed margin of the published claim |
| Silent prospective | Run live with outputs recorded but hidden from clinicians | Performance holds on current data, and workflow effects become visible |
| Limited live | Visible to a defined group with heightened monitoring and an easy route to report concerns | No unexpected harm signal, clinician feedback addressed |
| Full deployment | Broad use with standing monitoring and a defined review cadence | Periodic revalidation, and re-entry to this sequence on material change |
The silent prospective stage is the one most often skipped and the one that most often earns its keep. It surfaces the gap between how a model performs and how it changes what people do.
Subgroup performance is a safety question
In most sectors, unequal performance across groups is framed as fairness. In healthcare it is more precisely framed as differential risk of harm, and that framing changes who cares about it. A model that is less accurate for a subgroup exposes that subgroup to more misdiagnosis. That is a patient safety finding, and it belongs in the safety governance channel rather than only in an equity report.
Practical implications. Measure performance by subgroup, not only in aggregate, and be explicit when a subgroup is too small in your data to support a conclusion, because a wide confidence interval is itself information. Watch for proxies: a variable that encodes access to care rather than clinical need will reproduce existing inequity and look predictive while doing it. Historical utilisation data is particularly prone to this. Our guide to bias auditing and fairness testing covers the measurement methods.
What a clinical AI committee owns
Health systems generally do not need a new governance structure for AI. They need an existing clinical governance structure to acquire specific competence and a defined remit. The remit that works:
- Intake. Every proposed clinical or clinically adjacent AI use, including features arriving inside systems you already own.
- Classification. Device or not, and the risk tier, recorded with reasoning.
- Validation approval. Approving the local validation plan and reviewing its results before live use.
- Deployment authorisation. With conditions, including what clinicians are told and what training precedes access.
- Monitoring review. Performance and subgroup performance on a cadence, plus incident review.
- Change response. A defined reaction when a vendor changes a model under a PCCP or otherwise.
- Retirement. Including what happens to the clinical workflow that grew around the tool.
Membership needs clinical leadership with authority, informatics, quality and safety, privacy, and someone who can read a validation study critically. The last is the role most often missing, and its absence is why some committees approve claims they have not evaluated. Structural guidance on decision rights and gates is in building an AI governance operating model.
Procurement is the strongest control
By the time a clinical AI tool is deployed, most of your options have narrowed. The questions worth asking before signing are specific to this sector: what population was it validated on and what were the subgroup results, does it have a PCCP and what does that permit, how will we be notified of model changes, what does it assume a clinician will verify, will you support local validation with the access it requires, and what happens to our data. The general framework for this is in third-party and AI vendor risk assessment.
The World Health Organization's guidance on large multi-modal models in health is worth reading alongside, particularly for the equity and access dimensions that procurement conversations tend to omit.
Frequently Asked Questions
Is clinical decision support software always a regulated device?
No, and the boundary turns on the nature of the output and the clinician's ability to review its basis. Software that provides information a clinician can independently evaluate, with the basis for the recommendation available, sits further from device regulation. Software producing a specific diagnostic or treatment output the clinician is expected to rely on, without a practical means to check the reasoning, sits closer to or inside it. Time criticality matters too. Because the boundary is genuinely fact specific, record your classification and its reasoning for each tool rather than applying a rule of thumb across a portfolio.
What is a predetermined change control plan, in plain terms?
It is agreement in advance about how a cleared AI device is allowed to change. The manufacturer specifies which modifications it intends to make, the methodology for developing and validating them, and an assessment of their impact. Once that plan is authorised as part of the marketing submission, changes falling inside it can be made without a new submission. For a health system the significant consequence is that a device's behaviour can legitimately change within the agreed envelope, so ask what the envelope permits and how you will be told when it is used.
Do we need to revalidate a model that is already FDA cleared?
Clearance addresses whether the product may be marketed. It does not establish that the model performs adequately on your population, in your workflow, with your documentation practices. Performance degradation across sites is common and well documented, so local validation is a clinical safety step rather than a regulatory duplication. At minimum, run a retrospective evaluation on your own data with subgroup breakdowns before the model influences care, and prefer a silent prospective period before it becomes visible to clinicians.
How should we govern generative AI used for clinical documentation?
Treat it as clinically consequential even where you conclude it is not a device, because an error that enters the record propagates into future decisions, billing and research. The controls that matter are attribution of the final record to the clinician who signs it, a review step that is genuinely performed rather than nominal, measurement of omission and fabrication rates against source audio or notes, and clarity for patients where required. The documentation burden problem it solves is real, which is why the temptation to deploy it without measurement is strong.
Who should chair a clinical AI committee?
A clinician with institutional authority, usually a chief medical officer or chief medical information officer, because the decisions are clinical decisions with technical inputs rather than the reverse. Where these committees underperform, it is typically because they are chaired from IT and become procurement reviews, or because no member can critically appraise a validation study and so published claims are accepted at face value. Fill that appraisal seat deliberately.
What monitoring is realistic for a small health system?
Three things, all achievable without a dedicated platform. A stable retrospective test set that you re-score on a schedule, which detects drift in a bought model without any vendor cooperation. A simple clinician reporting route for outputs that looked wrong, which catches what metrics miss. And a standing agenda item that reviews both, with the authority to suspend use. Sophistication can come later; the absence of any feedback loop is the actual risk.
