Distillation
Training a smaller "student" model to reproduce the outputs (or intermediate representations) of a larger "teacher" model. Common technique for compressing frontier-model capability into smaller, cheaper models suitable for high-volume or on-device use.
Related terms
- Fine-tuning
Continuing to train a pretrained model on a task-specific dataset to change its behavior, style, or domain knowledge. Techniques range from full fine-tuning (all weights) to parameter-efficient methods like LoRA and QLoRA that update only small adapter layers.
- Small Language Model (SLM)
A compact language model (typically ≤ 10B parameters) optimized for on-device inference, low-latency use cases, or cost-sensitive high-volume workloads. Examples: Phi-4, Gemma 3, Llama 3.2 1B/3B. Increasingly capable — some SLMs match 2024-era mid-tier LLMs on narrow tasks.
Related on this site
Where this fits
Distillation is part of the AI Fundamentals vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.