In Part 6 of The AI Engineering Playbook, we get practical. A developer's guide to the essential optimisation techniques that make AI applications viable in the real world: semantic caching, response streaming, and model optimisations like quantisation.
So far in this series, we have designed our system and learned how to talk to the model. Now we face the final gate before production: can we make our application fast enough for users and cheap enough to be viable? A brilliant AI feature is useless if it is too slow or too expensive.
This is where performance engineering comes in. These are the techniques that turn a clever demo into a professional, scalable product
.1. Caching: Your First and Best Defence
The easiest way to make an API call faster and cheaper is to not make it at all. Caching is a developer's best friend, and in the world of AI, we can take it a step further than simple exact match caching. We can use semantic caching.
The problem is that users can ask the same question in many different ways. "How do I reset my password?" and "I forgot my password, what do I do?" mean the same thing, but an exact match cache would miss it. My own approach is to use a cosine match of a vectorised question against a base of previously answered questions. My research suggests that if the similarity score is in the high 90% range, I can confidently serve the cached response. This ensures users get a consistent, instant answer and saves a costly API call. I am still researching which similarity threshold is best, but it is a powerful technique.
2. Streaming: The Key to Perceived Performance
Total response time is one metric, but perceived response time is what matters for user experience. This is where streaming is a game changer.
I implemented streaming for the helper bot on my freelance CMS, and the difference was night and day. Before, the UI would show a static "Waiting for response..." message for several seconds, then suddenly output a large block of text. It felt slow and broken. After implementing streaming, the answer appears word by word, as if being typed out. The total time to get the full answer is the same, but the user can read along as the answer is formed. It prevents the user from getting frustrated while waiting and makes the application feel incredibly fast and interactive.
3. Self Hosting Optimisation: The Future on Your Own Hardware
As I plan my home lab for running open source models, the key challenge is how to run these massive models on consumer grade hardware. This is where specific optimisation techniques become critical.
Quantisation: From my research, I think quantisation will be key for self hosting. This is the process of reducing the precision of the model's weights (e.g., from 32 bit numbers down to 8 bit or even 4 bit). This dramatically reduces the model's size on disk and the RAM required to run it, making it feasible to run on a home GPU.
Batching: To get the best use of the GPU, batching will be essential. This involves the server waiting a fraction of a second to group multiple user requests together and processing them simultaneously in a single pass through the model. This maximises hardware utilisation and dramatically increases the throughput of your service.
Ultimately, I think this combination will be able to make some models small enough for smaller devices in the future, moving AI from the data centre to the edge, which is a truly exciting prospect.
Conclusion
These optimisation techniques are not just "nice to haves"; they are essential for building professional AI products. Caching manages your costs, streaming manages your user's experience, and techniques like quantisation will unlock the future of self hosted AI.
In Part 7, we will tackle one of the hardest problems: how do we actually know if our AI is working correctly? A look into evaluation and observability.