In Part 5 of The AI Engineering Playbook, we explore the limits of RAG and dive into the world of fine tuning. A guide to understanding when and why you need to change the model's brain, not just give it an open book.
In our last post, we did a deep dive into Retrieval-Augmented Generation (RAG), the architectural pattern that allows us to build factual AI that can reason over our private data. For 90% of use cases, RAG is the right tool for the job. It's cost effective, relatively simple to implement, and brilliant for preventing hallucinations.
But what about the other 10%? What happens when providing facts isn't enough? What if you need to change the model's fundamental behaviour, its style, or teach it a new, complex skill? This is where RAG's "open book exam" approach falls short, and we need to reach for a more powerful tool: fine tuning.
The Core Distinction: Knowledge vs. Behaviour
To understand when to use fine-tuning, we need to be crystal clear on the difference between it and RAG.
RAG provides Knowledge. It gives a generalist model access to specific information at the moment of a query. The model's core "brain" remains unchanged. It's like giving a smart graduate a textbook on a new subject, they can answer questions from the book, but their way of speaking and reasoning is the same.
Fine Tuning teaches Behaviour. It takes a pre-trained model and continues the training process on a smaller, specialised dataset of your own examples. This process alters the model's internal weights, it actually changes its brain. It's like sending that same graduate on a six month intensive course to become a legal expert. They no longer need the textbook for every question because the knowledge and, more importantly, the style of a lawyer is now embedded in how they think.
A Simple Rule for Automation: Are You Teaching a Skill or Providing Knowledge?
When you want to create a service for a company that automates a task, this distinction is everything.
You should choose RAG if the task simply requires access to internal information.
Example: Automating answers to employee questions about company policies. The AI doesn't need a new skill; it just needs the knowledge contained in the policy documents. RAG is the perfect, cost-effective solution.
You should choose Fine Tuning if the task requires the AI to learn and reliably replicate a specific process or skill that is unique to your company.
Example: Automating the creation of Jira tickets. A project manager writes a plain text request, and the AI must generate a perfectly formatted JSON object for the Jira API. You are teaching the AI a very specific skill (how to structure data for an API), which makes fine tuning an excellent choice.
Three Scenarios Where Fine Tuning is Required
So, when do you need to send your model to "school"? Here are three common scenarios where fine tuning is the superior choice.
1. To Enforce a Specific Style, Voice, or Tone (A Writing Skill)
The Problem: You need your AI to consistently write in a very specific brand voice. Perhaps witty and informal for marketing copy, or deeply formal and academic for technical reports. Instructing it to do so in a RAG prompt on every single API call is inefficient and often produces inconsistent results.
The Fine Tuning Advantage: By fine tuning the model on hundreds or thousands of examples of your brand's writing, you embed that style into its very core. You no longer need to waste precious prompt tokens on style instructions, the model intrinsically knows how to speak like your brand.
2. To Master a Complex, Niche Format (A Formatting Skill)
The Problem: You need the model to reliably output a complex, non-standard format that it has rarely seen in its original training. This could be a proprietary XML schema, a niche query language, or a specific medical report structure. Few shot prompting can work for simple cases, but it's often brittle for complex ones.
The Fine Tuning Advantage: Fine tuning on a large dataset of
(instruction, correct_output)pairs teaches the model the deep structural rules and syntax of that format. It becomes a specialised "format expert," able to generate the required output with much higher reliability.
3. To Improve Reliability on a Nuanced Skill (A Formatting Skill)
The Problem: You need the model to be exceptionally good at a specific reasoning task where subtle nuances are critical, such as classifying legal documents or identifying positive, negative, or neutral sentiment in financial news. RAG can provide the documents, but the model's generalist reasoning might miss the specific cues that an expert would catch.
The Fine Tuning Advantage: Fine tuning on a curated dataset of expert labelled examples (e.g.,
(news_article, 'negative')) makes the model a specialist in that domain. It learns to recognise the specific patterns and jargon that lead to a correct classification, resulting in significantly higher accuracy.
A Word of Warning: Start with RAG
Fine-tuning is a powerful tool, but it's also far more complex and expensive than RAG. It requires a large, high quality dataset, and there's a risk of "catastrophic forgetting," where the model becomes an expert in its niche but gets worse at general reasoning.
Therefore, the rule of thumb is this: always start with RAG. Only consider fine tuning when you have a clear business case and have proven that RAG is insufficient for changing the model's core behaviour in the way you need.
Conclusion: Two Tools in Your Toolbox
RAG and fine-tuning are not competitors; they are two distinct tools for two different jobs. RAG is for providing knowledge. Fine tuning is for teaching behaviour. A professional AI engineer knows which tool to reach for, ensuring they are always using the most effective and efficient solution for the problem at hand.