Evaluation & Governance

Model Extraction

An attack where a black-box model's capabilities are approximated by an adversary through systematic querying. Concern for proprietary hosted models. Rate limiting, query-pattern anomaly detection, and watermarking are common defenses.

Related terms

Red-Teaming
Structured adversarial testing of an AI system, either manually or via automated attack pipelines, to find prompt injection, jailbreak, harmful-output, and misuse vulnerabilities before release. Increasingly required by AI regulation for high-risk systems.

Framework dimensions

Free tools

Whitepapers

GenAI Governance Playbook

Comparisons

EU AI Act vs NIST AI RMF vs ISO 42001: Framework Comparison

Next steps

Take the assessment— See where you stand

Where this fits

Model Extraction is part of the Evaluation & Governance vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.