What should a sandbox plan and exit report contain?

What should a sandbox plan contain?

A sandbox plan should contain the identification of the provider and the competent authority, the system description and intended purpose, the innovation and learning objectives, the testing description including real world testing where applicable, the duration and phases, the participants and their roles, the data categories, sources, retention and safeguards, the risk assessment with hazards, mitigations and residual risk, the safeguards for participants including information and consent, the supervision arrangements, the monitoring and reporting arrangements, the incident and serious incident arrangements, the exit criteria and arrangements, and the confidentiality and publication arrangements. The plan should be written in a form that the authority can supervise and that the provider can follow, with each element owned by a named person and with dates. The plan should state what will be learned, how the learning will be measured, and what evidence will be produced. It should state the limits of the testing, including the number of participants, the locations, the data volumes and the system versions, so that testing outside the plan is visible as a deviation. The plan should be version controlled, with changes agreed with the authority in advance where the Article requires it, and recorded where they are made. A plan that is vague about risks, safeguards or exit arrangements will not support supervision or learning.

What records should be kept during testing?

During testing, the provider should keep records of the system versions tested, the participants, the data used, the tests performed, the results including failures, the incidents and serious incidents, the authority guidance and directions, the plan changes, the monitoring data, and the action taken on findings. The records should be kept in a form that can be inspected, with dates, authors and versions, and should be available to the authority on request. Test results should include the conditions, the measures, the outcomes and the limitations, not only a statement that the test passed. Failures should be recorded with the cause, the action and the retest, because a record that shows only successes will not be credible to a reviewer. Participant information and consent records should be kept separately from test data where data protection requires it, with access controls. The monitoring record should show the indicators tracked, the thresholds, the alerts and the responses. A testing record that is kept retrospectively, after the testing has finished, will be weaker than a record kept during testing, and the provider should assign record keeping as a named duty in the plan, with time protected for it.

For the broader regulatory framework, see codes, standards and sandboxes overview.

How is an exit report drafted?

An exit report is drafted by the competent authority after sandbox testing, on the basis of the provider evidence, the authority supervision record and the testing records. It describes the system tested, the plan, the testing conducted, the outcomes, the risks identified and mitigated, the compliance lessons, the guidance given, and any further action recommended, including further testing, conformity assessment steps, or changes to the system. The provider should contribute a draft or a submission for the exit report, with the evidence, the analysis and the lessons learned, and should review the authority draft for factual accuracy. The exit report should be written in plain language, so that a person who was not involved in the testing can follow what was done and what was learned. It should distinguish between findings of fact, such as test results, and conclusions, such as compliance lessons, so that a later reviewer can test the conclusions against the facts. It should state the limitations of the testing, including the duration, the participants and the conditions, so that the learning is not overstated. The final exit report should be dated, versioned and kept with the sandbox file and the conformity file.

How should exit report learning be used?

Exit report learning should be used to update the system, the risk assessment, the documentation, the testing and the post market monitoring plan. The provider should create an action list from the exit report, with each action owned, dated and evidenced, and should record the completion of each action. Actions can include design changes, data changes, oversight changes, instruction changes, and further testing. The conformity file should cite the exit report, the actions taken, and the evidence produced, so that a notified body or a market surveillance authority can follow the learning into the file. Learning that is not acted on, or that is acted on without a record, will be lost. The provider should also consider whether the learning affects other systems, other versions or other markets, and should record that consideration. Where the exit report identifies a fundamental rights, health or safety issue, the provider should pause the affected use until the issue is resolved, and should record the pause and the resolution. Where the exit report identifies a guidance need, the provider should consider whether to raise it with the authority or through the standardisation or guidance process, with the record showing the route taken.

What annual reporting applies to sandboxes?

Competent authorities must report annually on sandbox activity, including the number of sandboxes, the participants, the outcomes and the lessons learned, to the Commission and the Board, in accordance with Article 57. That reporting is an authority duty, not a provider duty, but a provider should be aware that its sandbox participation, its outcomes and its lessons may be included in anonymised or aggregated reporting, subject to confidentiality. The provider should agree, in the sandbox plan, the publication and confidentiality arrangements, including what will be published about the sandbox, the system and the outcomes, and what will be kept confidential as a trade secret or as personal data. The provider should not assume that sandbox participation is confidential in full, and should not assume that all sandbox material will be published. The plan should state the arrangements clearly, with the authority agreement recorded. A provider that is concerned about confidentiality should raise it at the plan stage, with legal advice, rather than after the exit report has been drafted. The annual reporting also means that sandbox learning can inform future guidance, standards and supervision, and a provider that has completed a sandbox should follow the published annual reports for lessons that affect its systems.

Continue with regulatory sandboxes.

What does this guide not decide?

This guide does not draft a sandbox plan or an exit report, does not state the eligibility criteria or the plan requirements for any sandbox, and does not decide whether any testing proposal should be accepted into a sandbox. Sandbox plans and exit reports must be agreed with the competent authority concerned, on the basis of the Articles 57 and 58 provisions, any implementing acts, and the sandbox notice for that Member State. A provider with a specific testing proposal should confirm the plan, the records and the exit arrangements with the competent authority before testing begins. This guide gives the contents and the record structure, not the authority decision for a named proposal.

Frequently asked questions

Continue through this AI Act cluster

• codes, standards and sandboxes overview
• regulatory sandboxes
• real world testing
• sandboxes for smaller firms
• conformity assessment and CE marking
• providers