AI and Data Protection: Three Operations, Not One

Ask a privacy team whether an AI system is covered and the answer arrives as a single question: is there a DPIA. Ask which processing operation the DPIA covers and the conversation usually stops, because the assessment was written against "the AI system" as though that were one thing.

It is three things. Training a model on personal data, running it on personal data, and holding sensitive data in order to test it for bias are three distinct operations with three different legal bases, three different sets of conditions, and, since the Digital Omnibus amended the AI Act, three different legal homes. A single assessment covering all three is approximately right about each and precisely right about none.

This guide separates them. It covers whether the trained model is itself personal data, what the European Data Protection Board actually said about legitimate interest, what happens to a deployment when the training data turns out to have been unlawfully processed, the bias testing permission that moved out of Article 10 into a new article almost nobody's guidance reflects yet, and how the AI Act's data governance duty differs from a DPIA.

Three operations, not one

The separation matters because the answers diverge. A lawful basis that works for one operation does not carry to the next, and a condition attached to one has nothing to say about the others.

The third row is the one that blocks real work most often. Testing a system for disparate outcomes across a protected characteristic requires holding data about that characteristic, and privacy teams correctly resist collecting it. That tension is the single most common reason bias testing does not happen, and it now has a specific legal route through it, which the fifth section covers.

Is the model itself personal data?

This is the question that determines whether anything downstream is a data protection matter at all, and the EDPB's Opinion 28/2024, adopted on 18 December 2024 at the request of the Irish Data Protection Authority, is the reference point. Its full title is "Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models".

The Board declines to give a general answer. Its position is that "whether an AI model is anonymous should be assessed on a case by case basis by the DPAs". That refusal is the finding: there is no category of model that is anonymous by construction, and a claim that a model is anonymous because weights are not records is not an argument the Board accepts on its own.

The threshold it describes has two limbs, and both have to hold.

1. Identification must be highly improbable
   It must be highly improbable that people whose data trained the model can be identified, directly or indirectly, from it.
2. Extraction through queries must also be highly improbable
   And it must be highly improbable that the personal data used in training can be extracted from the model through queries. This is the limb that catches memorisation, and it is an empirical property of a particular model rather than a design intention.

Legitimate interest, and what strictly necessary does to it

Legitimate interest is available in principle for developing and deploying AI models. The Opinion assesses it through a three step test covering the interest, necessity and the balancing against data subjects' rights, and permits it "only if the processing is shown to be strictly necessary and the balancing of rights is respected".

Two things in that formulation do most of the work.

Necessity is doing more than it usually does. The ordinary question is whether the purpose could be achieved another way. For model training the realistic alternatives are a smaller corpus, synthetic data, anonymised data, or not training on this category of data at all. Those have to have been considered and rejected on the record, because the assessment turns on them, and an assessment that reaches necessity without naming what was rejected has skipped the test.

Mitigating measures can change the outcome. The Opinion notes that where the balancing test would otherwise weigh against processing, mitigating measures may reduce the negative impact enough to change the result. This is the practical lever: filtering, deduplication, opt out mechanisms, limits on memorisation and controls on what can be extracted are not just good hygiene, they are inputs to the lawfulness conclusion and should appear in it as such.

Note also what the Opinion does not do, which is bless legitimate interest as a default. It gives considerations for data protection authorities applying the test, case by case. An organisation reading it as a green light for training on whatever it holds has read the headline rather than the test.

When the training data was unlawful

This is the question with the sharpest consequence for anyone deploying a model they did not build, which is most organisations.

The EDPB's position is that where a model was developed using unlawfully processed personal data, this "could have an impact on the lawfulness of its deployment, unless the model has been duly anonymised".

Read that as a supply chain statement. A defect in how a model was trained does not necessarily stay with the developer. It can reach the deployment, which means the lawfulness of your use can depend on a processing operation you did not perform, cannot inspect and were not party to. The escape route offered is anonymisation of the model, which returns to the two limb test above and is therefore not a box anyone can tick on a questionnaire.

The bias testing permission that moved

This is the part most internal guidance has not caught up with, and it is worth checking your own against.

The AI Act used to carry the permission to process special categories of personal data for bias detection in Article 10(5). The Commission's own Article 10 page now marks paragraph 5 as deleted, with the note that it is not present after the amendment. The permission did not disappear. It moved into a new Article 4a, titled "Processing of special categories of personal data for bias detection and correction", and it got wider.

Under Article 4a, providers of high risk systems may "exceptionally process special categories of personal data" for this purpose where strictly necessary. The conditions are cumulative, all of them have to be met, and they apply in addition to the GDPR rather than instead of it.

1. Other data genuinely will not do
   The bias detection "cannot be effectively fulfilled by processing other data", including synthetic or anonymised data. The alternatives have to be considered and ruled out, which mirrors the necessity test for legitimate interest.
2. Reuse is technically limited and the data is protected
   There must be "technical limitations on the re-use of personal data" alongside state of the art security and privacy preserving measures, pseudonymisation among them.
3. Access is controlled and documented
   "strict controls and documentation of the access", limited to authorised persons under confidentiality obligations, to prevent misuse.
4. It goes nowhere else
   The data is "not transmitted, transferred or otherwise accessed by other parties". This is the condition that rules out handing the dataset to an external testing supplier without a very careful arrangement.
5. It is deleted when the job is done
   Deleted once the bias has been corrected or the retention period ends, "whichever comes first". A standing bias testing dataset held indefinitely does not meet this.
6. The reasoning is written down
   Records of processing activities must state "the reasons why the processing of special categories of personal data was strictly necessary" and why other data could not achieve the objective. The justification is itself a required artefact.

The widening is in paragraph 2. The same route is extended to providers of other AI systems and models, and to deployers of high risk systems, where processing is strictly necessary to address biases likely to affect health or safety, negatively affect fundamental rights or lead to prohibited discrimination, "especially where data outputs influence inputs for future operations". All of the paragraph 1 conditions still apply. And the paragraph is explicit that it "does not create any obligation to conduct such bias detection and correction", so it is a permission rather than a mandate.

For a governance programme the practical reading is that the long standing deadlock between "we must test for disparate impact" and "we cannot collect that characteristic" now has a documented route through it, with six conditions and a written justification as the price. Our guide to bias auditing and fairness testing covers the measurement this permits.

Data governance is not a DPIA

Article 10 asks for something a DPIA does not produce, and teams that assume the DPIA discharges it tend to find the gap during conformity work rather than before.

It requires data governance and management practices "appropriate for the intended purpose", covering design choices, preparation steps such as annotation, cleaning and enrichment, stated assumptions about what the data measure and represent, assessment of availability, quantity and suitability, examination of possible biases, "appropriate measures to detect, prevent and mitigate possible biases", and identification of data gaps that prevent compliance.

One item in that list is worth separating out: data collection and data origin, including the original purpose for which the personal data was collected. That is a provenance requirement, and it is the one most likely to be unanswerable for a corpus assembled over several years from several sources. A DPIA asks whether current processing is lawful. Article 10 asks what the data was originally collected for, which is a different record and frequently one nobody kept.

The quality criteria are separate again. Data sets must be relevant and sufficiently representative and, "to the best extent possible, free of errors and complete in view of the intended purpose", must have "appropriate statistical properties" including for the groups the system is intended for, and must reflect the particular geographical, contextual, behavioural or functional setting of use as far as the intended purpose requires. Note also the scope rule: for high risk systems that do not train models, those duties plus Article 4a(1) apply only to testing data sets.

What to do about it

1. Split the assessment by operation
   One document covering training, deployment and bias testing will be vague about all three. Separate sections with separate bases and separate conditions, even where they sit in the same file, make each one reviewable and make the gaps visible.
2. Record what you rejected, not just what you chose
   Both the legitimate interest test and the Article 4a necessity condition turn on alternatives having been considered and found inadequate. Synthetic data, anonymised data, a smaller corpus, a different characteristic proxy. Name them and say why each was insufficient.
3. Test extraction before claiming anonymity
   The second limb of the EDPB threshold is empirical. Put extraction testing in the assurance programme and keep the result with the anonymity position, so the claim has evidence under it.
4. Reconstruct provenance while you still can
   Original purpose of collection is the Article 10 item most likely to be missing, and it gets harder to recover every quarter. For any corpus still in active use, writing down where each component came from and what it was originally collected for is cheap now and may be impossible in two years.
5. Design bias testing data to be deleted
   The Article 4a deletion condition is incompatible with a permanent fairness testing dataset. Build the testing cycle so the special category data has a defined life: acquired for a test, used under access control, deleted when the correction is made. That shapes the pipeline, so decide it before building rather than after.
6. Put the justification where it can be found
   Article 4a requires the reasoning in the records of processing activities, and the AI Act requires data governance evidence in the technical documentation. Our guide to audit ready evidence covers how to keep the two consistent without maintaining them twice.

Frequently asked questions

Where this leaves you

The pattern across all of this is that the law has become more specific while most internal guidance has stayed general. There is now a published position on when a model can be considered anonymous and it has two empirical limbs. There is a stated consequence for deploying a model trained on unlawfully processed data. There is a named article with six cumulative conditions for the special category processing that bias testing needs, and it covers deployers as well as providers.

None of that is satisfied by a document that says the organisation complies with the GDPR and has completed a DPIA. What it needs is three separable assessments, a record of the alternatives that were rejected, evidence rather than assertion on anonymity, and provenance for a corpus while the people who assembled it are still reachable. The last of those is the one with a deadline nobody sets, because it quietly becomes impossible.