LLMs are known for making things up. So how do you build a factual chatbot? A look into the prompt engineering, vectorisation, and caching techniques I used to create a non hallucinating AI that can answer complex, data driven questions using only a trusted knowledge base and fast.
If you've spent any time with tools like ChatGPT, you've seen the magic. You've also likely seen their biggest flaw for any serious business application: hallucination. LLMs are designed to be creative and fill in the gaps, which often means they confidently make things up when they don't know an answer.
This is a non-starter for a website help bot. The last thing you want is your bot inventing a refund policy or giving a user incorrect instructions. For one of my personal projects, I was determined to solve this. I needed a bot that was helpful but, above all, truthful.
The solution wasn't retraining a massive model or using a complex new framework. It came down to one fundamental principle: the LLM is only as good as the prompt you give it.
The Problem with Simple Prompting
If you simply ask a generic LLM a question, it will use its vast training data to give you a generic, plausible sounding answer. It's guessing. This is where most attempts at building a simple help bot fail. They treat the LLM like an all knowing oracle, when it should be treated as a powerful reasoning engine that needs to be given the right materials.
The Solution: The "Open Book Exam" Prompt
The key to preventing hallucination is to stop asking the LLM to recall information and start giving it the information to process. You give it an "open book exam" with every single query.
This is achieved by constructing a very specific, context-rich prompt. Instead of just passing the user's question, I built a system that first retrieves the relevant information from my site's database and then bundles it into a detailed prompt.
The anatomy of my successful prompt looks like this:
The Persona and The Prime Directive: First, I tell the model exactly what it is and what its most important rule is.
You are a helpful assistant for my-website.com. Your primary goal is to answer user questions accurately based ONLY on the context provided below.The Inescapable Rule: I give it an explicit, non negotiable instruction for what to do if the answer isn't in the provided text.
If the answer to the user's question cannot be found in the provided context, you MUST respond with: "I'm sorry, I don't have enough information to answer that question." DO NOT make up an answer.The Context: This is the "open book" part. My backend code finds the relevant documentation or database entries related to the user's query and injects it directly into the prompt.
--- CONTEXT START --- [Relevant data retrieved from the database...] --- CONTEXT END ---The User's Question: Finally, I append the actual question from the user.
User Question: [The user's original query...]
The Engineering Challenge: Making It Fast
This approach is great for accuracy, but it introduces a performance challenge: what happens when your "open book" is a whole library? Searching thousands of documents or database rows for the right context on every single query would be incredibly slow.
This is where a couple of key techniques come into play:
Vectorising the Context: Instead of doing a slow, full text search, we can pre process our entire knowledge base using a technique called vectorising (or creating embeddings). Each piece of content is converted into a numerical representation, a vector, that captures its semantic meaning. When a user asks a question, their question is also converted into a vector.
Similarity Search & Caching: We can then perform an incredibly fast "similarity search" in a specialised vector database to find the context vectors that are mathematically closest to the question vector. This allows us to instantly retrieve only the most relevant paragraphs to inject into the prompt. To make it even faster, this vectorised context can be cached, and we can even cache the final LLM answers for common questions, delivering near-instant responses.
From Simple Facts to Complex Analysis
This efficient, context-driven structure is incredibly powerful. For example, in the race event CMS I wrote about in a previous post. If a user asks, "Who won the last race?" the system vector searches the results, finds the relevant data, and feeds it to the LLM for a factual answer.
But it scales to more complex queries. If a user asks, "Give me my total points for races this last season," my backend doesn't just grab one document. It retrieves all of that logged in user's race result records from the database and formats them as context. The LLM's task is no longer just to find a fact, but to perform an aggregation, to sum up the points from the provided data. Because its world is limited to the data I give it, it calculates the correct total without hallucinating results from races the user didn't participate in.
Similarly, for a question about a competitor like, "How many podium finishes did Team Bravo have last year?," the system fetches all of Team Bravo's results. The LLM then acts as a data analyst, counting the relevant entries and providing a factual answer based only on that dataset.
Conclusion: From Oracle to Engine
By combining disciplined prompt engineering with efficient retrieval techniques like vectorisation, we transform the LLM from an unreliable oracle into a highly effective reasoning engine. It no longer needs to guess. It simply reads the trusted information I provide and synthesizes a helpful answer, whether that involves simple retrieval or complex calculations. It proves that you can build reliable, factual, and fast AI applications, all by mastering the craft of engineering the perfect prompt and the system that delivers it.