AI Engineering Playbook Part 4: A Deep Dive into RAG - The Hallucination Killer

The last part of this series, we mastered the art of talking to the model through prompt engineering. Now, we build the system that makes our prompts trustworthy. Every developer who has worked with LLMs has encountered their biggest flaw: hallucination.

The Enemy: Hallucination

A hallucination is when an LLM confidently states something that is incorrect or nonsensical. It's not a bug, it's a core characteristic. LLMs are probabilistic engines designed to generate coherent text, not to state facts. If you ask a generic model a question about your company's proprietary data, it will not say "I don't know." It will invent a plausible sounding answer. For any professional application, this is a critical failure.

So, how do we force a model to be factual? The answer is the most important architectural pattern in applied AI today: Retrieval Augmented Generation (RAG).

The Solution: Grounding the Model with RAG

RAG is the engineering pattern that grounds a model in reality. The strategy is simple: we give the model an "open book exam." We never ask it to recall information from its own memory. Instead, we find the relevant facts from our own trusted data source and provide them in the prompt, instructing the model to answer the user's question based only on that context.

This turns the LLM from an unreliable narrator into a powerful reasoning engine. Here is the practical pipeline to build it.

Step 1: Data Ingestion & Chunking - Preparing the Knowledge

Before an AI can use your knowledge, you must break it down into digestible pieces, or "chunks." A common mistake is to split text arbitrarily. Through experience, I found splitting the information into paragraphs and sections first is much better. This preserves the semantic context. However, I also make sure to check that they are never over a certain fixed chunk size to ensure they are manageable for the model.

Step 2: Vectorising - Turning Words into Maths

Once you have your chunks, you need to convert them into a format the AI can understand: vectors. This process converts text into a numerical representation of its meaning. My pragmatic approach is to choose an embedding model that is compatible with my generation model. If I'm using an OpenAI model for generation, I'll use their embedding models to ensure everything works together seamlessly.

Step 3: Vector Databases - The AI's "Memory"

You need a specialised vector database to store and query these vectors efficiently. The right choice often depends on your existing infrastructure. As the freelance work I have done is on AWS services, I have used AWS OpenSearch with its vector search capabilities to keep everything in the same ecosystem.

Bonus Technique: Consistency with Semantic Caching

As we touched on in Part 1, even with RAG, a model might slightly rephrase an answer to a similar question. For a help bot, we often want the exact same answer for questions with the same meaning.

We can achieve this with semantic caching. The process is simple:

1. When a user question comes in, we vectorise it.

2. We then use cosine similarity to compare this new vector to a cache of vectors from previously answered questions.

3. If the similarity is above a certain threshold (e.g., 99%), we can bypass the LLM entirely and serve the exact same, pre-approved answer from our cache.

This not only guarantees consistency but also dramatically reduces API costs and improves response time.

Conclusion: Engineering Trust

RAG is more than just a clever technique; it is the fundamental engineering pattern for building trust into your AI applications. By grounding the model with your own data, you mitigate the risk of hallucination and create a system that is not only intelligent but also reliable and factual.

In Part 5, we will explore what happens when you need to teach a model a new skill or give it the ability to take action in the real world, a look into Fine Tuning and AI Agents.