How should testing and validation be planned for high-risk systems?

How should metrics and thresholds be set?

Metrics and thresholds should be set before testing begins, in light of the intended purpose, the people affected, and the consequences of error. A metric should measure what matters for the use, not only what is easy to measure, and a threshold should state the level that must be achieved for the system to be relied on for that use. The provider should record who set the metric and threshold, what evidence supported the choice, and what action follows if the threshold is not met. Setting thresholds after seeing results, or choosing a metric that flatters the system, undermines the evidential value of testing. Where a threshold is not met, the record should show the decision taken, whether remediation, a limitation on use, or a decision not to place the system on the market in its current form.

What test data should be used?

Test data should be relevant, sufficiently representative, and as free of errors and complete as possible for the intended purpose, in accordance with Article 10, and should be kept separate from training data where the testing purpose requires independence. The test set should reflect the deployment population, geography, and conditions, including edge cases and groups that may be affected differently. Data leakage between training and testing, or a test set that mirrors only the easiest cases, will overstate performance. The provider should record the test data source, composition, preparation, and any known limitation, and should keep the test data and the results together with the system version tested. A test result without a description of the test data cannot be interpreted by a reviewer.

For the broader regulatory framework, see High-risk requirements overview.

How should functional and robustness testing be organised?

Functional testing should check that the system performs its intended task correctly under the conditions of use, including with realistic inputs, volumes and workflows. Robustness testing should check behaviour under errors, faults, degraded inputs and unexpected conditions, and should confirm that failures are safe and detectable rather than silent. Both kinds of testing should be planned as test cases with expected outcomes, executed in a controlled environment that matches deployment as closely as possible, and recorded with results, deviations and follow-up actions. Testing should involve the human oversight workflow where the system will be overseen, so that the combined human and system performance is understood, not only the model in isolation. A test plan that covers only the model, without the workflow, leaves the deployed performance untested.

What adversarial and security testing is needed?

Adversarial and security testing should be proportionate to the system's exposure and the threats identified in the risk and cybersecurity assessment. It should test, as relevant, data and model poisoning, adversarial examples, evasion, confidentiality attacks, and attempts to alter use or outputs. Testing should be carried out by people with the competence to design and interpret attacks, using methods that are documented and repeatable, and the findings should be recorded with severity, exploitability and remediation. A finding that is recorded but not remediated, or remediated but not retested, should remain open in the test file. Security testing should be repeated when the system, its dependencies, or the threat landscape changes, and the results should feed into the Article 15 evidence and the risk management file.

When must testing be repeated?

Testing must be repeated when a change could affect performance, robustness, security or bias: a model update, a data change, a change in intended purpose, a change in deployment context, or a significant change in the operating environment. The provider should set retesting triggers in advance, linked to the change management process, so that a change does not reach deployment without the testing it requires. Retesting should cover the affected areas and, where the change is material, the full validation suite. The version history should show which version was tested, when, against which thresholds, and with what result, so that a reviewer can tell whether the version on the market is the version that was tested. A system that is tested once and then changed repeatedly without retesting has no current validation evidence.

Continue with Article 9 risk management for high-risk systems.

What does this guide not decide?

This guide does not decide what metrics, thresholds, test data or attack methods are appropriate for a particular system, or whether a test result shows that the system meets Articles 9 and 15. It does not approve a test plan or assess a validation report. Those questions depend on the intended purpose, the affected people, the risks and threats identified, and the consolidated legal text. A provider should have its test plan and results reviewed by competent, independent reviewers, record the decisions taken on any failure, and confirm the position with a competent authority or legal adviser where the performance picture is contested.

Frequently asked questions

Continue through this AI Act cluster

• High-risk requirements overview
• Article 9 risk management for high-risk systems
• Article 15 accuracy, robustness and cybersecurity
• Article 10 data and data governance
• Conformity assessment and CE marking
• What technical documentation does Article 11 require?