The AI Act does not make it illegal to use AI, but it does regulate a lot of the use cases. As EU acts typically go, it makes it mandatory to do a lot of assessment, documentation and governance – at least for so-called “high-risk” use cases. The EU has published an official short summary here: https://artificialintelligenceact.eu/high-level-summary/.

The main points of the AI Act

• The AI Act classifies AI systems based on risk. There are 4 levels: unacceptable (illegal use cases), high-risk (OK, but with a lot of paperwork and controls), limited risk (chatbots, be transparent), and minimal risk (unregulated, for example spam filters).
• The AI Act has rules for companies using AI, but more rules for companies making AI systems. Your personal hobby use and development is not regulated.
• General purpose AI systems (basically, systems capable of solving many tasks such as AI agents able to execute commands via API’s) has requirements for documentation, instructions for use, respect copyright and publish a summary of the content used for training. Open source: only copyright and summary of training data needed, unless the system is “high-risk”. GPAI systems also need threat modeling, testing, incident reporting and reasonable security controls.

Banned AI systems

The unacceptable ones: these protections are there to protect you against evil, basically systems made for mass surveillance, social credit systems, predictive crime profiling of individuals, manipulation of people’s decisions, etc.

High-risk AI systems

Systems that are safety critical are considered high-risk, including a long list of systems under other EU legislation such as important components in machinery, aircraft, cars and medical systems (Annex I in the EU Act has a long list of systems). There is also an Annex III, listing particular high-risk systems, including using AI for employee management, immigration decisions and safety critical components in critical infrastructure. OK – it is quite important that we can trust all of this, perhaps a bit of governance and oversight is not so bad? At the same time, the important cases are perhaps also the areas where we would expect to see a lot of benefit from using technology to make things better, more efficient, cheaper, etc. So, what are makers and users of high-risk AI systems required to do? Let’s begin with the makers. They need to:

• Create a risk management system
• Perform data governance, to make sure training and validation data sets are appropriate and of good quality
• Create technical documentation to demonstrate compliance (this can be interpreted in many ways)
• Design the system for “record keeping” to identify national level risks(?) and substantial modifcations throughout the system’s lifecycle
• Create instructions for use to downstream deployers
• Design the system so that users can implement human oversight
• Ensure acceptable levels of cybersecurity, robustness and accuracy
• Establish a quality management system

Most of these requirements should be part of any serious software or product development.

Limited risk

For limited risk systems, the main requirement is to be transparent to the user that the system is using artificial intelligence. The transparency requirement is regulated in Article 50 of the AI Act. Content generated by AI systems must be marked as such, including deep-fakes. There is an exception for satirical or artistic content (to avoid making the art less enjoyable, but you still have to be honest about AI being part of the content), and also for “assistive editing functions”, like asking an LLM to help you edit a piece of text you wrote.

Risk management requirements for “high-risk” systems

The first requirement for developers of “high-risk” AI systems is to have a risk management system. The system must ensure that risk management activities follow the lifecycle of the AI system. The key requirements for this system:

• Identify and analyze potential risks to health, safety or fundamental rights
• Estimate and evaluate the risks
• Adopt measures to manage the risks to acceptable levels, following the ALARP principle
• The systems shall be tested to identify the best risk management methods
• The developer must consider whether the AI system can have negative effects for people under the age of 18 years, or other vulnerable groups

In other words, the developer needs to perform risk assessments and follow up on these. Most companies are used to performing risk assessments, but in this case the term “fundamental rights” is perhaps less common, except for in privacy assessments under the GDPR. The fundamental rights requirements are detailed out in Article 27. The EU has a Charter of fundamental rights covering dignity, freedoms, equality, solidarity, citizen’s rights and justice. The AI Office will publish tools to simplify the fundamental rights assessment for AI system developers.

AI based glucose level regulation in diabetes patients (a ficticious example)

Consider the use of an AI system used to optimize blood glucose level regulation in diabetes type I patients. The system works in a closed loop, and automatically adjusts continuous insulin injection using an insulin pump. The system measures blood glucose levels, but also senses activity level, environmental factors such as humidity, temperature, altitude. The system also uses image recognition using a small camera to detect what the patient is eating as early as possible, including interpreting menu items in a restaurant before the food is ordered. Using this system, the developer claims to completely remove the hassle of carbohydrate calculations and manual insulin adjustments, to reduce the amount of time the patient has a too high or low glucose level, and avoid the typical delayed insulin-glucose response in the body through feedforward mechanisms based on predictive powers of the AI.

For a system like this, how could one approach the risk management requirements? Let’s first consider the risk categories and establish acceptance criteria.

Health and safety (for the patient):

• Critical: Death or severe patient injuries: unacceptable
• High severity: Serious symptoms related to errors in glucose level adjustment (such as hyperglycemia with very high glucose levels): should occur very rarely
• Medium: Temporary hypoglycemia (low blood sugar levels) or hyperglycemia (increased blood suger levels): acceptable if the frequency is lower than in manually regulated patients (e.g. once per month)
• Low: annoyances, requiring patient to perform manual adjustments. Should occur less than weekly.

If we compile this into a risk matrix representation, we get:

Critical	Unacceptable	Unacceptable	Unacceptable
High	Unacceptable	Unacceptable	ALARP
Medium	ALARP	Acceptable	Acceptable
Low	Acceptable	Acceptable	Acceptable
	Weekly	Yearly	Decades

Fundamental rights (for the patient and people in the vicinity of the patient). A fundamental rights assessment should be performed at the beginning of the development, and to be updated with major feature or capability changes. Key questions:

• Will use of the system reveal to others your health data?
• Will the sensors in the system process data about others that they have not consented to, or where there is no legal basis for collecting the data?

We are not performing the fundamental rights assessment here, but if there are risks to fundamental rights, mitigations need to be put in place.

Let’s consider some risk factors related to patient safety. We can use the MIT AI Risk repository as as starting point for selecting relevant checklist items in order to trigger identification of relevant risks. The taxonomy of AI risks has 7 main domains:

1. Discrimination and toxicity
2. Privacy and security
3. Misinformation
4. Malicious actors and misuse
5. Human-computer interaction
6. Socioeconomic and environmental harms
7. AI system safety, failures and limitations

In our ficticious glucose regulation system, we consider primarily domain 7 (AI system safety, failures and limitations) and domain 2 (Privacy and security)

• AI system safety failures and limitations (7)
  • AI possessing dangerous capabilities (7.2)
    • Self-proliferation: the AI system changes its operational confines, evades safeguards due to its own internal decisions
  • Lack of capability or robustness (7.3)
    • Lack of capability or skill: the quality of the decisions is not good enough
    • Out-of-distribution inputs: input data is outside the validity for the trained AI model
    • Oversights and undetected bugs: lack of safeguards to catch bugs or prevent unintended use
    • Unusual changes or perturbations in input data (low noise robustness)
  • Lack of transparency and interpretability (7.4)
    • Furstrate achievement of auditing: lack of compliance to relevant standards, cannot be assessed.
• Privacy and security (2)
  • Compromise privacy by obtaining, leaking or correctly inferring personal data (2.1)
    • PII memorization: Models inadvertently memorizing or producing personal data present in training data
    • Prompt injection: Compromise of privacy by prompt based attacks on the AI model
  • AI system vulnerability exploitation (2.2)
    • Physical or network based attack: can lead to manipulation of model weights and system prompts
    • Toolchain and dependency vulnerabilities (vulnerabilities in software)

To assess the AI system for these risks, the typical process would follow typical risk management practices:

• Describe the system and its context
• Break down the system into parts or use cases
• Assess each part or use case, as well as interactions between parts to identify hazards
• Document the finding with cause, consequence, existing safeguards
• Perform evaluation of probability and severity, compare with acceptance criteria
• Identify mitigations

Let’s consider a particular risk for our glucose regulator:

RISK CATEGORY: (7.3) Lack of capability or skill.

• Possible risk: the system makes the wrong decision about insulin injection rate due to lack of capabilities.
• Possible causes: insufficient training data, insufficient testing.

Consequence: over time it can lead to frequent hypo- or hyperglycemia, causing long-term patient complications and injury.

Probability: would require testing or an assessment of the training and testing regime to determine the probability.

Suggested decision: provide extra safeguards based on blood glucose level measurements, and let the patient take over to adjust manually if the glucose regulation is detected as outside of expected performance bounds. Use this while performing testing to to assess the reliability of the model’s inference in order to allow fully automatic regulation.

Key take-aways

1. The AI act puts requirements on developers and users of AI systems.
2. For high-risk systems, a robust risk management system must be put in place
3. AI risks is an active field of research. A good resource for AI risks is the MIT AI Risk taxonomy.

Further reading

AI Risk repository

AI Act Explorer