Tokens, context windows and why AI forgets | Master AI Automation in 4 hours Master AI Automation in 4 hours Course About Ayush Modules Sample chapter Toolbox The Microcap Minute Classroom / Module 01: What AI Actually Is / Chapter 2 Tokens, context windows and why AI forgets Watch first, then read. Same lesson, your pace. What you will learn – What tokens are and why they matter – What a context window is, using a desk analogy – The difference between chat history, memory and knowledge The machine reads in puzzle pieces You read words. An LLM reads tokens , chunks of letters. A token might be a whole word (“elephant”), or a piece (“ele”, “phant”), or punctuation. Rule of thumb for English: 1 token ≈ ¾ of a word, so 1,000 words ≈ 1,300-1,500 tokens. Why should you care? Two reasons: Cost and limits are counted in tokens , not words. API bills (Module 4) charge per token. Unusual text becomes many tokens , rare names, code, other languages, which is why some things confuse models more than others. The desk that is only so big Imagine the model is a student answering exam questions at a desk with limited space . Everything it may use, your whole conversation, pasted documents, its own replies, must fit on that desk at once. That desk is the context window : the maximum amount of text (in tokens) the model can consider at one moment. Modern windows are large (hundreds of thousands of tokens), but not infinite. Three things follow: It only knows what’s on the desk. If a fact isn’t in the conversation or in its training data, it will guess rather than say “unknown”. Very old messages can fall off the edge in long chats; the model may “forget” instructions from an hour ago. Bigger is not automatically better , attention across a huge desk gets expensive and diluted. Precise beats long. Practical habit: keep one conversation = one topic. Start fresh chats when you switch tasks. Paste only the relevant pages of a document, not all 200. Three different kinds of “memory” People get burned by mixing these up, so here is the honest separation: Kind What it is Example Conversation memory Everything still on the desk within this chat “As I said above…” works Saved memory / profile Notes some apps store about you between chats ChatGPT remembering you’re left-handed Training knowledge Patterns baked into the model during training, frozen on a cutoff date It knows 1929 happened; it does not know yesterday’s cricket score The third one matters most: the model’s built-in knowledge has a cutoff date. Ask about anything recent and it either knows from training, guesses, or, if the app has web search, looks it up live. If you need facts that changed recently, either search the web yourself, use a search-enabled mode, or paste the current information into the chat. Try it yourself In a new chat, type: My name is Ravi. Then ask What is my name? , it will answer correctly. Now start a brand-new chat and ask again: it has no idea. You just proved memory lives in the conversation, not inside the machine. Finally, paste this into any chat: Estimate roughly how many tokens this sentence is. and see how it reasons about itself. Key takeaways – Models read tokens (word-chunks); costs and limits count in tokens. – The context window is the model’s desk: everything it may use must fit there. – One chat = one topic keeps the desk clean and answers sharp. – Conversation memory, saved memory, and training knowledge are three different things; training knowledge has a cutoff date. Download the exercise sheet (PDF) Module workbook (PDF) ← Prev: The autocomplete that ate the world Next: Hallucinations: confident nonsense and how to catch it → Classroom / Module 01: What AI Actually Is / Chapter 2 Tokens, context windows and why AI forgets What you will learn – What tokens are and why they matter – What a context window is, using a desk analogy – The difference between chat history, memory and knowledge The machine reads in puzzle pieces You read words. An LLM rea
Tokens, context windows and why AI forgets
Written by
in