Which AI regulations require security testing?

The major ones now all point the same way. The EU AI Act requires high-risk systems to be robust and cyber-secure under Article 15, and requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing under Article 55. The US NIST AI Risk Management Framework and its Generative AI Profile name red-teaming as a risk-measurement action. The UK Code of Practice for the Cyber Security of AI expects independent security testing before release under Principle 9. ISO/IEC 42001 requires organisations to assess and treat AI risks through controls that include testing. Singapore’s Model AI Governance Framework for Generative AI lists testing and assurance as a core dimension. They differ in force and detail, but they agree that an AI system should be adversarially tested before it goes live.

Does the EU AI Act require AI red teaming?

For the most capable models, effectively yes. Article 55 requires providers of general-purpose AI models classified as carrying systemic risk to perform model evaluation, including conducting and documenting adversarial testing to identify and mitigate systemic risks. For high-risk AI systems more broadly, Article 15 requires an appropriate level of accuracy, robustness and cybersecurity, and names resilience against attempts to alter use or behaviour through third parties exploiting vulnerabilities. The Act does not use the phrase "red teaming" for every system, but adversarial testing is the practical way to evidence these obligations.

What does the NIST AI Risk Management Framework say about testing?

The NIST AI RMF is voluntary and organises AI risk work into four functions: Govern, Map, Measure and Manage. Testing lives in Measure, which calls for AI systems to be evaluated for trustworthiness characteristics using appropriate methods. The Generative AI Profile, published in July 2024, is more specific: it lists structured red-teaming as an action for measuring and managing generative AI risks such as harmful outputs, data leakage and prompt injection. Because so many buyers and regulators cite NIST, its language on red-teaming has become a common reference even outside the US.

We operate in several countries. Which framework do we follow?

Test to the strictest one that binds you, and the rest are covered. The frameworks disagree on paperwork, not on the underlying security work: they all want an AI system probed for the ways it can be manipulated, before it is exposed to real users, by testers independent of the builders. An AI red team mapped to the OWASP LLM Top 10 and MITRE ATLAS produces evidence that satisfies the EU AI Act, NIST, the UK Code and ISO/IEC 42001 in one engagement, because those catalogues are what the frameworks point to when they get specific.

Field Notes / AI Security

Every AI Rulebook Now Asks the Same Question: Did You Test It Adversarially? A Global Map.

The EU has the AI Act. The US has the NIST framework. The UK has its Code of Practice. Singapore, ISO and OWASP have their own. They disagree on plenty. They agree on one thing: before an AI system goes live, someone independent should try to break it. Here is what each framework actually asks for, and the testing that answers all of them at once.

Author: Red Team Partners
Read: 12 MIN READ
Filed: 3 Sep 2026

The one demand they share

The world has not agreed on how to govern AI. The European Union wrote a binding law. The United States published a voluntary framework. The United Kingdom chose a code of practice and declined the EU’s approach. Singapore, ISO and OWASP each added their own. They differ on force, on scope, and on almost every definition. On one point they have quietly converged: before an AI system reaches real users, someone independent should try to break it, and you should be able to show what they found EU AI Act, Art. 15 & 55 .

That convergence is useful, because it means you do not have to build a separate compliance project for every jurisdiction. The paperwork differs. The security work underneath does not. An AI system tested adversarially against the OWASP LLM Top 10 and MITRE ATLAS produces the same evidence a regulator in Brussels, an auditor citing NIST, and a board following the UK Code all want to see. Here is the map, framework by framework.

01 The EU AI Act

he AI Act, Regulation (EU) 2024/1689, is the one with teeth. It is binding law, it reaches any provider placing an AI system on the EU market wherever they are based, and it carries fines. For AI security testing, two articles matter. Article 15 requires high-risk systems to reach an appropriate level of accuracy, robustness and cybersecurity, and names resilience against third parties exploiting the system’s vulnerabilities to alter its use or behaviour AI Act, Article 15 .

Article 55 goes further for the largest models. A provider of a general-purpose AI model classified as carrying systemic risk must perform model evaluation, and the Act spells out that this includes conducting and documenting adversarial testing to identify and mitigate systemic risk. That is a legal requirement to red-team, written into statute. The phrase the Act uses is not always "red teaming", but adversarial testing is how you evidence the obligation.

02 The NIST framework (US)

he United States took the opposite route: no single AI law, but a voluntary framework that has become the common vocabulary anyway. The NIST AI Risk Management Framework, published in January 2023, organises the work into four functions, Govern, Map, Measure and Manage. Security testing lives in Measure, which asks organisations to evaluate AI systems for trustworthiness using methods fit for the risk NIST AI RMF 1.0 .

The Generative AI Profile, added in July 2024, is where NIST gets concrete. It names structured red-teaming as an action for measuring and managing generative AI risks, and it lists the risks by name: harmful output, data leakage, and prompt injection among them NIST AI 600-1 . Alongside it, NIST’s adversarial machine learning taxonomy gives testers a shared language for the attacks themselves, from evasion to poisoning to model extraction. When an auditor anywhere asks whether you follow "recognised practice", this is usually the document they mean.

03 The UK, ISO and Singapore

Three more frameworks matter for a global operator, and each reinforces the same expectation from a different angle.

The UK Code of Practice for the Cyber Security of AI, published by DSIT and the NCSC in January 2025, sets 13 principles. Principle 9 expects AI systems to be security-tested before release, by testers independent of the people who built them. The Code is voluntary, but it became the basis for the first global baseline standard, ETSI TS 104 223, so its expectations are spreading beyond Britain UK Code of Practice .

ISO/IEC 42001:2023 is the international management-system standard for AI, the AI counterpart to ISO 27001. It does not prescribe a single test, but it requires an organisation to assess and treat its AI risks through documented controls, and independent security testing is how you demonstrate the control works. For enterprises already certified to ISO 27001, this is the natural home for AI testing evidence ISO/IEC 42001:2023 .

Singapore’s Model AI Governance Framework for Generative AI, from the IMDA and the AI Verify Foundation in May 2024, lists testing and assurance as one of its core dimensions and points to independent third-party evaluation for higher-risk deployments. It is the reference point across much of Asia-Pacific IMDA GenAI Framework .

04 The technical catalogues everyone points to

one of the frameworks above tells a tester exactly what to try. When they get specific, they point to the same two catalogues, and this is the quiet reason the whole landscape is more unified than it looks. Learn these two and you are testing to the substance behind every rulebook.

The OWASP Top 10 for Large Language Model Applications, refreshed for 2025, names the ten weakness classes an AI red team works through: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. MITRE ATLAS maps real adversary tactics and techniques against AI systems in the same style as the ATT&CK matrix your SOC already uses. Between them they turn a regulator’s abstract demand into a concrete test plan OWASP LLM Top 10 / MITRE ATLAS .

05 The map, in one table

Read across the frameworks and the pattern is hard to miss. The force varies from binding law to voluntary guidance. The demand for adversarial testing does not.

Framework	Force	What it says on testing
EU AI Act (2024/1689)	Binding law	Art. 15 robustness & cybersecurity; Art. 55 requires adversarial testing of systemic-risk GPAI models
NIST AI RMF + GenAI Profile	Voluntary (US)	Names structured red-teaming as a risk-measurement action
UK Code of Practice	Voluntary (UK)	Principle 9: independent security testing before release
ISO/IEC 42001:2023	Certifiable standard	Requires controls to treat AI risk; testing evidences them
Singapore GenAI Framework	Voluntary (SG)	Testing and assurance as a core dimension
OWASP LLM Top 10 / MITRE ATLAS	Technical catalogues	The concrete weakness classes and attack techniques to test

06 What to test, whatever binds you

ick the strictest framework that reaches your business and test to it. For most global teams that is the EU AI Act, because it follows your users into Europe regardless of where you sit. Once you satisfy its demand for adversarial testing, the voluntary frameworks are covered by the same evidence, because they ask for less, not for something different.

What that testing looks like in practice does not change by jurisdiction. Map what your AI can read, the tools it can call, and the permissions it holds. Attack it the way a real adversary would, through the OWASP LLM classes and the ATLAS techniques. Prove which attack paths work, fix them, and confirm the fix holds. Keep the report, because every framework that matters now expects you to show your working.

Key takeaways

The EU AI Act, NIST AI RMF, UK Code of Practice, ISO/IEC 42001 and Singapore’s GenAI framework all converge on one demand: adversarial testing before deployment.
The EU AI Act is binding and extraterritorial. Article 55 requires adversarial testing of systemic-risk general-purpose models; Article 15 requires robustness and cybersecurity for high-risk systems.
NIST names structured red-teaming as a risk-measurement action in its Generative AI Profile, and its language has become a global reference.
When the frameworks get specific, they point to the same two catalogues: the OWASP LLM Top 10 and MITRE ATLAS.
Test to the strictest framework that binds you and the rest are covered. The paperwork differs; the security work does not.
One AI red team engagement, mapped to OWASP and ATLAS, produces evidence for every framework at once.

If you have shipped an AI feature into more than one market and you are not sure which rulebook sets your bar, that uncertainty is the place to start. A free audit tells you where your exposure sits and which frameworks reach you, then gives you a fixed price to test to the highest one.

References

Sources

European Union. Regulation (EU) 2024/1689 (the AI Act). Article 15 (accuracy, robustness and cybersecurity of high-risk systems); Article 55 (obligations for GPAI models with systemic risk). eur-lex.europa.eu
NIST. AI Risk Management Framework (AI RMF 1.0), January 2023. nist.gov
NIST. Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1), July 2024. nvlpubs.nist.gov
NIST. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2e2025). csrc.nist.gov
DSIT and NCSC. Code of Practice for the Cyber Security of AI, United Kingdom. 31 January 2025. gov.uk
ISO/IEC 42001:2023. Information technology, Artificial intelligence, Management system. iso.org
IMDA and AI Verify Foundation, Singapore. Model AI Governance Framework for Generative AI. 30 May 2024. aiverifyfoundation.sg
OWASP. Top 10 for Large Language Model Applications, 2025. genai.owasp.org
MITRE. ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems). atlas.mitre.org