Most organisations discover their AI incident process during their first AI incident. The pattern is consistent. The problem surfaces through a customer complaint or a journalist rather than an alert. It reaches a service desk that treats it as a bug. It takes four days to reach anyone who knows a legal deadline exists. By then, on the tightest path, the deadline has passed.
The gap is structural. Incident processes were built for outages and breaches, where something is down or someone got in. A model that quietly produces worse outcomes for one group of applicants is neither. Nothing is unavailable, nothing was breached, no attacker is involved, and no monitor fires. It is still an incident, and under some conditions it is a reportable one.
What counts as an AI incident
Work from consequence rather than cause. An AI incident is any event where an AI system in production produced, or came close to producing, an outcome that harms a person, breaches an obligation, or materially misinforms a decision. Useful families:
| Family | What it looks like | Why classic tooling misses it |
|---|---|---|
| Correctness | Confident, plausible, wrong output relied on by a person or a downstream system | The system is healthy. Latency and error rates look normal |
| Fairness | Outcomes diverge across protected groups after a retrain or a data shift | Requires measurement against protected attributes, which is a deliberate test, not a monitor |
| Disclosure | Training data, another user's content, or internal records surface in output | Looks like normal traffic. No exfiltration path is involved |
| Safety | Harmful instruction, crisis mishandling, unsafe recommendation in a clinical or physical context | Content is the payload. Nothing anomalous at the infrastructure layer |
| Agentic action | An agent takes a real action: a payment, a deletion, an email, a ticket, at scale | Each individual action is authorised. The sequence is the failure |
| Security | Prompt injection, jailbreak, model or tool abuse | This family your security team does own, and should |
The agentic row is the one changing fastest. When the unit of action stops being a prediction and becomes a sequence of tool calls, the blast radius of a single bad decision grows and the audit question changes from what did it say to what did it do. Our agentic AI governance framework covers the control design; here it matters because containment needs a kill switch for the agent's permissions, not just for the model endpoint.
The legal clock
Under the EU AI Act, serious incident reporting is governed by Article 73. Providers of high-risk AI systems placed on the Union market report serious incidents to the market surveillance authorities. Three deadlines apply, and they are graduated by severity.
| Situation | Deadline | Counted from |
|---|---|---|
| General case | No later than 15 days | The provider, or where applicable the deployer, becoming aware of the serious incident |
| Death of a person | Immediately, and no later than 10 days | Establishing, or as soon as it suspects, a causal relationship |
| Widespread infringement or serious incident | Immediately, and no later than 2 days | Becoming aware of the incident |
Two features of this scheme catch teams out.
The clock starts at awareness, not at confirmation. The initial report is due once a causal link is established or reasonably likely. Waiting for a complete root cause analysis before reporting inverts the design: you report, then you investigate, then you update.
Provider and deployer duties differ. The reporting duty to authorities is the provider's. Under Article 26, a deployer that identifies a serious incident informs the provider first, then the importer or distributor and the relevant market surveillance authorities. If you deploy a system someone else built, your first obligation runs to your supplier, and your contract should already say how fast they will respond. We cover that clause in third-party and AI vendor risk assessment.
Sector regimes stack on top rather than replacing this. A clinical AI failure may trigger device vigilance duties, a credit decision failure may trigger consumer protection notification, and a personal data element may trigger breach rules with their own, often shorter, clocks. Map your overlapping obligations once, in advance, because incident time is the wrong time to research them. The EU AI Act timeline sets out when each part of the regime became applicable.
Detect
Most AI incidents are reported by humans, which means your detection strategy is partly a reporting-channel design problem. Three layers, in rough order of how much they actually catch:
- Human channels. One route, publicised, usable by a support agent, a customer, or an engineer, that does not require the reporter to know whether it is an AI problem. Most first signals arrive here.
- Output monitoring. Distribution shift in model outputs, refusal-rate changes, confidence collapse, and disparity metrics recomputed on a schedule rather than at launch. This is where fairness regressions become visible.
- Action monitoring. For agents, the rate and type of real-world actions, with thresholds that trigger review. A tenfold jump in refunds issued is a signal even if every refund was individually authorised.
External corroboration is worth wiring in as well. The OECD AI Incidents Monitor and the AI Incident Database both index publicly reported failures, and a pattern involving a model or vendor you use is a reason to look at your own logs. Our AI incident monitor tracks reported cases as they surface.
Triage, against the clock
Triage has one job that is unlike normal incident triage: decide quickly whether a regulatory deadline has started. Everything else can take a little longer.
| Severity | Test | Internal response | Regulatory posture |
|---|---|---|---|
| S1 | Death, serious injury, serious and irreversible disruption of critical infrastructure, or widespread harm across many people | Immediate suspension, executive notification, named decision owner on call | Assume the 2 or 10 day path applies. Start the notification draft now |
| S2 | Serious harm to health, property, or fundamental rights of identifiable people; a breach of an obligation with a live legal duty | Contain within hours, remediation plan within days | Assess against the 15 day path. Record the awareness timestamp |
| S3 | Material quality, fairness, or disclosure failure with limited or reversible impact | Standard change process with a deadline | Log. Reassess if scope grows |
| S4 | Near miss caught by a control before reaching anyone | Record and feed the control back into testing | Not reportable, and the most useful data you will collect |
Record the awareness timestamp explicitly, on every case, at S3 and above. It is the single fact the whole reporting analysis turns on, and it is the fact nobody can reconstruct later.
Track near misses seriously. They are free information about where your controls are thin, and organisations that only log incidents that reached a person are throwing away the cheap half of their data.
Contain
Containment for AI has options that classic incident response does not, and choosing among them is a judgement about which failure mode you prefer.
| Option | Use when | Cost |
|---|---|---|
| Full disable | Harm is active and severity is S1 or S2 | Loss of the function. Fastest and safest |
| Roll back to a prior version | A specific change introduced the failure and the prior version is known good | Requires versioned models, prompts, and configuration, which is why that discipline matters |
| Human in the loop | The function is needed and a person can check each output at volume | Throughput. Often the right answer for consequential decisions |
| Narrow the scope | Failure is confined to a segment, geography, or use case | Requires the segmentation to exist already |
| Revoke agent permissions | An agent is taking actions and you do not yet know the bound | The specifically agentic control. Removes the ability to act while preserving the ability to answer |
Two things need to be true before an incident for any of this to work: someone can execute containment without waiting for a meeting, and the model, prompt, and configuration are versioned well enough that rollback is a real option. Both are operating-model properties. See building an AI governance operating model for where the suspension authority should sit.
Investigate
AI failures rarely have a single cause, and the useful discipline is to check every layer rather than stopping at the first plausible one.
- Data. Did the input distribution shift? Was a pipeline change deployed? Did an upstream source change schema or quality?
- Model. Was there a retrain, a fine-tune, a base model version change, or a quantisation? Was the provider's hosted model updated underneath you without notice?
- Prompt and configuration. Prompts are code and are frequently changed without change control. Check them.
- Retrieval. For retrieval augmented systems, was a document indexed that should not have been, or removed that should have stayed?
- Integration. Did a downstream system begin treating an advisory output as a decision?
- Human factors. Did operators have the context to catch it, and did the interface encourage deference to the model? Automation bias is a cause, not an excuse.
The provider-side model update is worth calling out. If you build on a hosted model, its behaviour can change without any deployment on your side. Your investigation needs to be able to establish which model version served the output, which means logging it at inference time.
Report
An initial regulatory report is short and honest. What happened, when you became aware, which system and version, the affected population as far as known, what you have done to contain it, and what you are still establishing. Resist the urge to wait for completeness.
Beyond the regulator, work out in advance who else must hear and how fast: affected individuals where rights are engaged, the provider if you are a deployer, insurers under some policies, and customers under contractual notification terms. Write the list once. During an incident you will not have time to derive it.
Keep the record. A dated timeline including the awareness timestamp, the decisions taken and by whom, the containment actions, the notifications sent, the investigation findings, and the remediation with its verification. This is the artefact an auditor or authority will ask for, and it is also the input to the next section.
Learn, and feed it back
Under Article 72, providers of high-risk systems must operate a post-market monitoring system. Incidents are one of its primary inputs, which makes the loop from postmortem back into monitoring a compliance artefact rather than an engineering nicety.
Three outputs from every S1 or S2 postmortem are worth insisting on. A new test that would have caught this, added to the pre-deployment suite. A monitoring signal that would have caught it sooner. And an honest answer to whether an existing gate should have stopped it, which sometimes means the gate needs strengthening and sometimes means it was bypassed. The NIST AI RMF MANAGE function frames the same loop if you are working from that framework; see our NIST AI RMF guide.
A one-page runbook
Where this usually breaks
- No awareness timestamp. Every deadline is measured from it, and it cannot be reconstructed.
- Routing through general IT. A queue that treats a fairness regression as a low priority bug will consume the two day path.
- Waiting for certainty before reporting. The scheme expects an early report and later updates.
- No versioning, so no rollback. Containment collapses to full disable, which raises the cost of every incident.
- Deployers assuming the provider is handling it. Your Article 26 duty is yours, and it starts with informing them.
- Postmortems that end in a document. If nothing changed in the test suite, the monitoring, or the gate, the incident will recur.
Frequently Asked Questions
What is a serious incident under the EU AI Act?
The regime is aimed at incidents involving death or serious harm to a person's health, serious and irreversible disruption of critical infrastructure, breach of obligations under Union law protecting fundamental rights, and serious harm to property or the environment. The graduated deadlines in Article 73 turn on which of those applies, with the two day path reserved for widespread infringement or a serious incident and the ten day path for death. Read the article text against your own use cases rather than working from a summary, including this one.
We deploy a vendor's system. Do we report to the regulator?
In the general case, no. The reporting duty to market surveillance authorities sits with the provider. As a deployer under Article 26 you inform the provider first, then the importer or distributor and the relevant market surveillance authorities. Practically, this means your vendor contract needs a notification clause with a timeframe short enough to let the provider meet a two day deadline, and you need to be able to evidence when you told them.
Can our existing security incident process cover AI?
Partly, and it is the right place to start rather than building in parallel. The lifecycle stages transfer, and NIST SP 800-61 remains a sound base. Three things need adding: severity criteria that recognise correctness and fairness failures with no breach, containment options beyond disable including rollback and permission revocation, and a triage step that establishes whether an AI-specific regulatory clock has started. What does not transfer is the assumption that an incident involves an attacker.
How do we handle an incident caused by a hosted model changing underneath us?
Treat it as an incident in your system, because from your users' perspective it is. This is why logging the model version at inference time matters: without it you cannot establish that the behaviour changed without a deployment on your side. Then treat it as a vendor management finding as well, and check whether your contract entitles you to notice of material model changes. Many do not, and that is worth fixing at renewal.
Should near misses be logged as incidents?
Yes, in the same register with a distinct severity. A near miss is evidence that a control worked, and evidence about where the margin is thin, at no cost in harm. Programs that only record failures that reached a person lose the cheapest data they will ever get, and they also lose the ability to show a regulator that their controls catch things.
Who should own AI incident response?
Operationally, whoever owns incident response generally, because duplicating the function creates a handoff during the worst possible hour. What has to be added is a named AI decision owner who can suspend a system or revoke an agent's permissions without convening a group, and a route to whoever will judge the regulatory question. The judgement call and the technical response do not need to sit with the same person, but both need to be reachable within the hour.
