Red-Teaming
Structured adversarial testing of an AI system, either manually or via automated attack pipelines, to find prompt injection, jailbreak, harmful-output, and misuse vulnerabilities before release. Increasingly required by AI regulation for high-risk systems.
Related terms
- Prompt Injection
An attack where adversarial instructions embedded in untrusted content (user input, retrieved documents, tool responses) hijack the model into ignoring its original instructions. Considered the primary security risk for LLM applications; defenses include separation of channels and output validation.
- AI Governance
The framework of policies, procedures, and controls that guide the development, deployment, and use of AI systems within an organization. Ensures AI aligns with organizational values and regulations.
Related on this site
Where this fits
Red-Teaming is part of the Evaluation & Governance vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.