Part 1 of my new series, an AI Engineering Playbook. Before you can build, you need to understand the building blocks. A guide to the absolute fundamentals of working with LLMs: tokens, context windows, temperature, and more.
Welcome to the first post in a new series I'm calling an "AI Engineering Playbook." My goal is to move beyond the hype and share the practical, foundational knowledge that developers need to build professional, production ready applications with Large Language Models.
Before we can architect complex systems or debate the merits of different models, we need to understand the core language of AI, the fundamental units and parameters that govern every single interaction.
The Building Blocks: Tokens and Context Windows
The first thing to understand is that an LLM does not see words or sentences the way we do. It sees tokens.
Tokenisation: This is the process of breaking down a piece of text into the smallest units the model can work with. A token is often a word or a part of a word. For example, the phrase "Hello world!" might be tokenised into
["Hello", " world", "!"].
Why does this matter so much? Because tokens are the fundamental currency of the entire system.
Cost: You are billed per token, for both your input and the model's output. Every token literally has a price.
Limits: Every model has a Context Window, which is the maximum number of tokens it can handle in a single request. This is a hard limit that you must design your entire application around.
Speed: The more tokens you send and receive, the longer the response will take.
Controlling Creativity: Temperature and Top_p
Once you send a prompt to a model, you have powerful dials to control the nature of its response. The most important of these is temperature.
Temperature is the "creativity dial." It controls the randomness of the model's output.
A low temperature (e.g.,
0.1) makes the model's responses more deterministic and focused. It will always choose the most statistically likely next token.A high temperature (e.g.,
0.9) encourages the model to be more creative and generate more diverse, unexpected text.
For instance, in the AI help bot I built for my MSLA race event CMS, the temperature is set very low, around 0.1. I need it to be a reliable, factual assistant, not a creative storyteller. This ensures that when it answers a question using the provided context, it sticks to the facts without deviation.
Top_p (Nucleus Sampling) is an alternative to temperature. Instead of affecting the randomness of all possible tokens,
top_ptells the model to only consider a smaller pool of the most probable tokens. A value of0.9tells the model to only choose from the most likely tokens that make up the top 90% of the probability mass. You generally use either temperature ortop_p, but not both.
A Sneak Peek: Ensuring Consistency with Vectors
Even with a low temperature, if two users ask a slightly different question (e.g., "how do I join a team?" versus "can I become part of a team?"), the LLM might generate a slightly different answer. For a help bot, this can be undesirable.
A more advanced technique to ensure true consistency is semantic caching. Instead of just caching exact text matches, we can vectorise a user's question to capture its semantic meaning. We then use cosine similarity to measure the "angle" between the new question's vector and the vectors of questions already in our cache. If the similarity is high enough, we can confidently serve the exact same, pre approved answer. This guarantees that similar questions will always get the same response, while also reducing cost and latency.
Conclusion: More Than Just a Prompt
Understanding these core mechanics is the first step to moving from a simple prompter to an AI engineer. By controlling the tokens, context window, and creativity parameters, you gain a massive amount of control over the model's behaviour, cost, and performance.
In Part 2, we will move from controlling the model's immediate response to designing the entire system around it.