How AI Chatbots Remember: Context Windows and Memory Explained

You are forty messages into planning a conference talk with your favorite AI chatbot. The conversation has been great. Back at message three, you told it two things: keep the talk under fifteen minutes, and don't use industry jargon. It nodded along — well, it typed something agreeable — and for the next hour the drafts were tight and jargon-free.

Then you type: "Quick check — what were the two constraints I set at the start?"

The chatbot answers instantly, with total confidence: "You asked me to keep it under twenty minutes and include a humorous opening anecdote."

Neither of those things was ever said. Fifteen minutes became twenty. "No jargon" became "humorous anecdote." The bot didn't forget the way a person forgets — it confabulated, filling the gap with something plausible-sounding and wrong.

That failure is not a glitch in your chatbot. It is the direct consequence of how these systems "remember" — and to understand it, you need exactly three ideas: tokens, the context window, and the bag of tricks chat apps use to pretend the window doesn't exist. This post walks through all three, in plain language, with a few habits that will save you from losing real work.

The unit nobody told you about: tokens

Language models do not read words the way you do. Before a model sees your text, a component called a tokenizer chops it into pieces called tokens — short fragments that can be whole words (conference), parts of words, punctuation, or even single characters. Everything a model consumes and produces is measured in tokens: your prompt, the conversation history, a pasted document, and every word of its reply.

The rough conversion to keep in your head, and it is only a rule of thumb: about 4 characters per token, or roughly three-quarters of a word. In practice people say 100 tokens ≈ 75 words. It is approximate — the exact ratio shifts with the language, the topic, and the tokenizer a particular model uses (code and non-English text tend to chew through more tokens per word; a string of numbers with no spaces can split into surprisingly many tokens). But 100 tokens ≈ 75 words is the working estimate the industry uses for back-of-the-envelope math.

Try it on a concrete case: a typical dense paragraph of this post is about 60–80 words, so roughly 80–110 tokens as a rough estimate. A ten-page report at ~500 words a page is about 5,000 words, so roughly 6,500–7,000 tokens, approximately. Keep that scale in mind — you'll need it in a moment.

The context window: a whiteboard, not a library

Here is the idea that explains everything. When you send a message to a chatbot, the model does not sit down and read the entire history of everything it has ever said to you. It reads one thing: a single, self-contained block of text called its context — and that block has a hard maximum size called the context window.

Think of the context window as a whiteboard of fixed size mounted on a wall. Everything the model "knows" about your conversation has to fit on that board at once:

• The system prompt — hidden instructions from the chat app ("You are a helpful assistant…"). You never see it, but it takes up space on the board.
• The conversation history — your messages and the model's replies, going backward in time. This is usually the biggest item on the board.
• Anything you pasted — documents, screenshots-as-text, code.
• The new message you just typed.
• Room for the reply — the model needs space on the board to "write" its answer.

Every token of all of that must fit within the window. The window size is measured in tokens, and it is fixed for each model — there is no negotiating with it, no "read a bit more" button.

Why does the limit exist? Three reasons, roughly in order of importance. First, compute: the model's core mechanism has to weigh every token against every other token (loosely, "read the whole board at once"), and that work grows steeply as the board gets longer — bigger windows cost more time and more money per message. Second, engineering: a longer context means more memory on the servers running the model, and the providers have to pay for it. Third, quality, which gets its own section below: bigger is not automatically better.

What happens when the window fills up

Two different things happen depending on how you hit the limit, and it matters which one you're dealing with.

One giant request. If a single message — say, a pasted 200-page contract — exceeds the model's window on its own, there is no graceful fallback: the API generally rejects the request with an error about exceeding the maximum context length. You have to shrink the input before trying again. This is the hard, honest limit.

A long conversation that gradually fills up. This is the everyday case, and it's handled silently behind the scenes by the chat app. A chat app has to fit your growing history into the same fixed window for every message, so as you approach the limit it trims — in general terms, the oldest turns fall off first, and often the app quietly replaces them with a short summary of what came before. Your view of the conversation still shows every message you ever typed. The model, however, is reading a shortened version of it. Nothing tells you which messages survived and which got summarized away.

That mismatch — you see the full chat, the model sees a trimmed one — is the root cause of the scenario this post opened with. The model's reply that confidently "reminded" you of wrong constraints is not memory failing; it is context no longer containing the original constraints. When the evidence is gone from the window, the model does what language models do: it generates the most plausible continuation, which in this case was a confident fiction.

Why long conversations degrade: "lost in the middle"

Even when everything fits inside the window, long contexts have a subtler problem. Researchers have repeatedly observed a behavior nicknamed "lost in the middle": across several studies, models answered questions most reliably using information placed near the beginning or the end of a long context, and less reliably using information buried in the middle.

Treat this as an observed tendency, not a law of nature. It varies by model, by task, and it improves as models get better at long contexts. But it has shown up often enough to be worth planning around, especially for the "paste a big document and ask questions" workflow:

• If something matters — the budget figure, the deadline, the constraint from the opening scenario — put it early (the system prompt and the first message tend to survive longest) or restate it right before the question that needs it.
• Don't assume a model read your 80-page PDF with uniform care just because it fit in the window. "It fit" and "it was attended to evenly" are different claims.
• If the answer matters, ask the model to quote the passage it's relying on. A correct quote is evidence it actually found the thing; a vague paraphrase is not.

How chatbots fake memory

Here's the part that surprises most people: the model itself remembers nothing between your messages. The underlying API is stateless — each request is a fresh encounter. Everything that feels like memory is scaffolding the chat app builds around the model. Four pieces of scaffolding, roughly in order of how commonly you meet them:

1. The whole conversation gets resent, every time. When you send message #41, the app packages up the system prompt plus the trimmed history plus message #41 and sends the whole bundle as one request. The model reads it fresh, replies, and forgets it all instantly. This is why long chats get slower and (on API plans) more expensive: each message re-pays the token cost of the entire history.

2. Summaries of older turns. To keep long chats affordable and within the window, many apps compress early turns into a running summary: "User is planning a 15-minute conference talk, no jargon, audience is product managers…" The summary preserves the gist but destroys specifics — "under fifteen minutes" can blur into "keep it short," and from there a confident bot can "recall" twenty. Summarization is lossy compression, and the loss is exactly where errors like our opening scenario come from.

3. Saved facts and preferences ("memory" features). Some chat apps offer an explicit memory feature: they extract durable facts — "user prefers concise answers," "user's team uses Python 3.12" — and store them separately, re-injecting the relevant ones into each new conversation. This is the closest thing to genuine long-term memory, but note the mechanics: it's a small database of sentences the app manages, not the model recalling you. It can be wrong, stale, or pull in a "fact" from an unrelated context — worth reviewing in the app's memory settings occasionally.

4. Retrieval over past conversations. A few systems go one step further and treat your old chats like documents: when you ask something, they search your conversation history for relevant past turns and pull the matches into the current context (the same retrieval idea behind RAG, pointed at your own chats). It's clever, and it inherits RAG's weakness: retrieval quality decides everything — a bad match silently injects irrelevant or misleading "memory" into your current work.

The takeaway: "the chatbot remembers" is shorthand for "the app reconstructs enough context for this message." Memory is a feature of the product, not of the model.

Context windows today: what do the numbers actually look like?

The honest answer first: these numbers change every few months, and the product you use may wrap a model with its own (sometimes smaller) effective limits. Treat any figure below as a snapshot verified via web search on 2026-10-17, and check the provider's current docs before making decisions on it.

With that caveat, the rough landscape as of mid-October 2026, based on provider-published figures quoted in current comparisons:

• Google Gemini 3.1 Pro: advertised around 2 million tokens of context — among the largest on the market.
• OpenAI's GPT-5.5: reported at roughly 1 million tokens (about 1,050,000 per OpenAI's own model reference, as quoted in an October 2026 comparison).
• Anthropic's Claude Opus 4.7 / 4.8: 1 million tokens (announced April 2026; standard claude.ai chat plans use a smaller 200,000-token window).

Two things to notice. First, the frontier has converged: a million tokens is roughly the going rate for a flagship model right now, and at ~0.75 words per token that's very approximately 750,000 words — on the order of several long novels. Second, and more important: a bigger window is headroom, not a guarantee. Providers themselves warn that quality can degrade toward the far end of a very long context, and you pay input-token prices for everything you send. Size the window to your longest realistic request, not to the biggest number on the page.

Practical rules that save you real work

Everything above compresses into five habits. They cost nothing and they prevent the exact failure this post opened with:

• Start a fresh chat for a new topic. A new conversation means a clean window: no stale constraints from last week's project leaking into this one, no summarization loss, no slowdown. It's the single highest-leverage habit on this list.
• Pin key facts early — and restate them when they matter. Decide at the start of an important conversation what you can't afford to lose (budget, deadline, constraints, names) and say it in the first message. When you reach the question that depends on it, restate it. "Lost in the middle" is much less scary when the critical fact is at both ends.
• Don't paste 200 pages and expect perfect recall. If a document fits but barely, consider the cheaper, more reliable route: paste the relevant sections, or ask questions about one section at a time, or summarize it first and then work from the summary. Bigger paste ≠ better answers.
• Summarize-and-continue for genuinely long work. When a project chat is getting long, ask the bot to summarize the state of the work — decisions, open items, constraints — then start a new chat with that summary as the first message. You get a clean window and the specifics, in your own verified words, not a lossy auto-summary.
• Verify critical facts outside the chat. Dates, numbers, names, quoted constraints — if the answer will go into something that matters, check it against your source, not against the chatbot's confident tone. Confidence is a style the model applies uniformly to true and false statements alike.

Honesty corner: the one-line version

Treat chatbot memory like a whiteboard: handy while the work is fresh, gone when it's wiped — so whatever you can't afford to lose, write down or say again.

What this post did not claim, and you shouldn't either: that a bigger context window always means better answers, that "lost in the middle" strikes every model equally, or that any chatbot's memory feature is a reliable record of your work. The window is a budget, memory is scaffolding, and the only memory you fully control is the one you keep yourself — a document, a checklist, a fresh chat seeded with your own summary.

The scenario from the top of this post has a boring, effective fix now: before asking "what were my constraints?", the user could have pinned them in the first message and restated them with the question. Fifteen minutes, no jargon — quoted back correctly, because the evidence was on the whiteboard instead of somewhere off its edge.

Continue the series on the AI hub page, where the General AI track collects every post in order.