CERNAI’s director, Prof. Federico Faroldi, has published a new article in the European Journal of Risk Regulation, titled “Risk for AI agents“.
The paper asks whether the concept of risk applies to AI agents. The EU AI Act, in force since 2024, rests on a product-based model of risk: it presupposes an intended purpose, a bounded space of reasonably foreseeable misuses, and a tractable distribution of potential harms. The article argues that these three presuppositions fail for sufficiently general agentic systems. Their behavioural space is open-ended by design, their misuse space can include novel sequences discovered by the agent itself, and the probability and severity of harm shift as the agent learns and acts on its environment. What fails, however, is the product-style operationalisation of risk (per system, ex ante, anchored in intended purpose), not risk as such.
The paper develops two complementary paths. The first, conservative, keeps the standard definition of risk and reconstructs it formally for narrow agents, either as expected deviation from an ideal trajectory or as the probability-weighted sum of constraint violations. The second, a governance path, draws an analogy with the normative systems we use to manage biological agents, and argues that sufficiently general AI agents call for something closer to law-following than to product-safety assessment. The first suits operational risk monitoring within bounded deployments, while the second suits the governance architecture that determines which agents may legitimately be deployed at all.