RAG vs Fine-tuning vs Prompt Engineering: Decision Framework
RAG, fine-tuning, and prompt engineering are not competing approaches — they solve different problems. RAG grounds an LLM in fresh, external facts. Fine-tuning changes behavior or style. Prompt engineering shapes intent within an already-capable model.
Last reviewed:
What we're comparing
- Prompt engineering
Shape output by carefully constructing the input prompt (roles, examples, structure).
- RAG
Fetch relevant documents at query time and inject them into the prompt as grounding context.
- Fine-tuning
Update model weights (full or LoRA) on task-specific data to change behavior or style.
Side-by-side
| Attribute | Prompt engineering | RAG | Fine-tuning |
|---|---|---|---|
| Freshness | Static per prompt | Live (index refresh cadence) | Frozen at training time |
| Factual grounding | Weak | Strong (source-attributable) | Weak (knowledge diffuses) |
| Latency overhead | None | +50-300ms retrieval | None at inference |
| Setup cost | Low | Medium (index + retriever) | High (data curation + compute) |
| Ongoing cost | Per-token only | Per-token + retrieval infra | Higher per-token (custom model) |
| Best for | Task shaping, few-shot behavior | Grounded QA, freshness, citations | Style/format, restricted vocab, low-resource domains |
| Anti-pattern | Trying to encode all knowledge in prompt | Fine-tuning what a retriever should surface | Fine-tuning to add facts (they will drift) |
When to use which
- Use Prompt engineering
Start here. It covers 60-70% of real use cases and costs nothing to iterate.
- Use RAG
You need current, source-attributable answers from a corpus larger than the context window.
- Use Fine-tuning
You need consistent output structure/style, or domain-specific behavior that prompting cannot reliably produce.
FAQs
Can I combine them?
Yes, and top-performing production systems usually do — fine-tune for style/format, RAG for facts, prompt for task shape.
Is fine-tuning obsolete now that context windows are huge?
No. Fine-tuning still wins on cost-per-token at scale and on tight latency budgets. Long context is a substitute for RAG in narrow cases, not a replacement.