Key takeaways
Following the political agreement on the EU AI Act from last Friday evening, the fragile agreement was sold to Member States. Meanwhile, daily technical meetings kicked off to lock in the details and develop the final text.
This process yielded a set of known knowns on generative AI, high-risk systems on prohibited use cases, and an understanding of what the known unknowns of technical details and enforcement are. The new roles of the Commission are significant.
The EU AI Act has been celebrated as a historic global precedent by EU policymakers, while the industry’s first reactions range from sceptical to ambivalent.
Our first take is that the cost of the rules on generative AI is acceptable, while the scope will very much depend on the Commission. The focus of any assessment should be the rules on high-risk uses by corporates. These will give some legal certainty at first but stand to be replaced by sector-specific rules.
From here onwards, the EU will seek to retrofit any international governance developments into its framework, diluting international efforts. Any matters of EU national security and many matters of law enforcement will be exempt from the AI Act.
Principles and place in the global process
After 38 hours of final negotiations, two years since the proposal, and over 600 hours of negotiations overall, the EU reached a political consensus over its AI Act (AIA) late Friday evening. This is a world first in terms of horizontal AI legislation.
The hard-fought outcomes of the political agreement have more impact on future civil liberties in the EU than on commercial uses of AI. Most matters of industry impact have been known in the lead-up to last Friday.
There are also a group of known unknowns – to be resolved in technical talks over the next two weeks or so, technical standards which will follow the principle legislative text, and national-level implementation decisions. Member States will not stand still in this process. A number are exploring preparedness and market development strategies.
There is a national security exemption to the AI Act. This is a reminder that, as we have been writing, defence risks, or in our language, US-style concerns, are real regarding AI, but they are a Member State and, in some cases, a NATO competence.
From here onward, the EU will seek to align international standards to this framework or, if necessary, dilute them sufficiently so that frameworks fit with them. This applies to expected G7, G20 and UN initiatives on AI.
Finally, the world’s largest AI markets have not yet indicated intent to develop similar horizontal AI legislation, notably the US, China and the UK. In the EU, we expect 2024 to kick off further vertical rule-making, including financial services and labour rights.
High-risk systems
The requirements of the AI Act should be grouped around generative AI, prohibited use cases, and high-risk systems. The rules on the latter are most extensive.
High-risk AI falls into two categories: systems that are a product or safety component, such as toys and medical devices, and systems used for designated ‘high-risk’ purposes, such as employment, credit scoring, and critical infrastructure. There are eight such categories, and the Commission will be given the power to amend this list.
Ultimately, it was agreed that the use of AI to influence the outcome of elections and voter behaviour would be classified under the high-risk framework.
The key debate here was who is on and off the list. A high-risk system can be self-exempt if it meets one or more of four criteria – if the system performs a narrow procedural task; if the system reviews or confirms human-made decisions (rather than being the sole basis for decision-making itself); if the system is an accessory to human work; and if the system only performs a preparatory task to the decision-making process.
Systems on this list will be subject to rules on risk mitigation, data governance, transparency and documentation, accuracy, robustness and cybersecurity and ought to undergo internal conformity assessment
Explainability and redress have also been prioritised, meaning that individuals can initiate complaints and receive explanations about decisions made using high-risk AI if their rights have been impacted.
At the final trilogue, MEPs successfully argued for including a mandatory Fundamental Rights Impact Assessment (FRIA) for high-risk AI systems before they are put onto the market, on top of other conformity assessments. Member States had originally opposed this.
We ought to be cognisant that after this horizontal legislative effort is over, the sector-specific Directorate Generals of the Commission will want to create specialised rules. This has always been, and is likely to be, the case in financial services. Therefore, the high-risk use cases of credit and insurance will likely be even more impacted by those rules in the mid-term.
Generative AI
As we expected, consensus landed on a two-tier regulatory system.
The upper tier will comprise models with 10~25 FLOPS – unfortunately, this was just the mid-point of Parliament’s 10~24 ask and Council’s 10~26 proposal.
Crucially, the Commission can also designate models as having systemic risk based on factors, such as known or estimated cost of training the model, performance benchmarks, and the number of business users in the EU (set at 10,000).
Models in the general tier will have basic documentation and information-sharing obligations for all general-purpose AI (GPAI) models. An important compromise is that these basic obligations will be subject to stronger enforcement than just voluntary codes of practice – i.e. codes of conduct can be introduced as an interim measure while the benchmarks are being drawn up, but these will supplement rather than replace the final rulebook.
For models in the “systemic risk” tier, there will be advanced model assessment criteria and transparency requirements.
Contrary to the Council’s final compromise proposal ahead of the final trilogue, GPAI systems – i.e. the downstream applications of GPAI models – will not face additional regulations. It seems the centre and centre-right of Parliament felt that these were unnecessary. As a result, providers of downstream GPAI systems will only need to comply with additional technical requirements if their systems are deployed in high-risk use cases.Prohibited usesThe controversial real-time remote biometric identification by law enforcement authorities will be subject to prior authorisation, done within 24 hours if necessary. It will also be limited to 16 specific crimes. In response to civil rights advocates, any use will be preceded by prior fundamental rights impact assessment. Member States may introduce even more restrictions here.
Use of AI for biometric categorisation is prohibited for those systems that categorise persons individually, based on their biometric data, to infer their race, political opinions, trade union membership, religious or philosophical beliefs or sexual orientation. This does not mean searching databases (e.g. images). Law enforcement is exempt from these prohibitions.
It is also prohibited to use Minority Report-style systems for predictive policing, as well as systems for the recognition of emotions.
The controversial real-time remote biometric identification by law enforcement authorities will be subject to prior authorisation, done within 24 hours if necessary. In response to rights advocates, any use will be preceded by prior fundamental rights impact assessment. Member States may introduce even more restrictions here.
Systems offering post-remote biometric identification would be considered as high risk.
Governance
As anticipated, and in light of the new GPAI model rulebook, an AI Office has been set up within the Commission to oversee the regulation of the Act. The Office will help to set standards and benchmarks, particularly around testing and evaluation practices, and to ensure that the Act’s rules are enforced equally across Member States. This had been a priority after the difficulties arising from GDPR, which has sometimes been applied unevenly across Member States.
The AI Office will be advised by a scientific panel of independent experts, to help with setting and adjusting benchmarks and designation criteria for fGPAI models with systemic risk, as well as to advise on the emergence of next-generation foundation models and their associated risks.
Adjacent to the AI Office will be an AI Board made up of Member State representatives. The Board will function as a coordination platform and advisory body to the Commission, as well as allowing Member States a say in how the foundation model regulations are implemented (including, e.g., creating codes of practice).
Finally, industry representatives, SMEs, start-ups, civil society, and academia will be able to provide technical expertise to the AI Board through a new ‘advisory forum’.
Next steps
This week, at least, and possibly next, technical trilogues will continue daily to iron out the details of the political agreement. There is still tension among the co-legislators on some of the details of the tedious consensus reached.
Recitals will likely be the only part of the text left to the incoming Belgian Presidency of the Council of the EU, i.e. to be concluded in Q1. Then, between Q1 and the entry force of the AIA there will be procedural votes, legal and language review – so earliest end H1.
Most of the requirements will apply two years after the Act enters into force, i.e. around mid-2026. However, a few provisions will apply earlier: notably, prohibited AI systems bans will apply from six months after the Act enters into force, while the requirements for GPAI models, high-risk conformity assessment bodies and the AI Office will start applying within one year from entry into force.
Level two requirements will be developed before then: some have exact deadlines (such as the development of codes of practice, which must happen within nine months after the entry into force), but the deadlines for most are open-ended.
Conclusion
Some provisions in the AI Act – like sandboxes and SME relief – aim to foster innovation. However, the key contribution the law can make is to offer a playbook for the division of responsibilities along an AI supply chain, particularly for the developers and deployers of high-risk uses.
The heated European debate on the use of AI in law enforcement points to a strong value system, which may come at odds with other jurisdictions where the civil liberates debate is more secondary.
Finally, on generative AI, the rules seem to mostly allow the Commission a close oversight of the developing market.
