You’re three messages into a conversation with your company’s AI assistant. You’ve explained the project, provided background, described what you need. Then you ask a follow-up question and the AI responds like it’s never heard of you before.
This is the memory problem. And if you’re building or buying AI tools for your business, understanding it will save you from a lot of frustration and a few expensive mistakes.
How LLMs actually “think” (the short version)
Large language models don’t have memory the way people do. They don’t learn from your conversations, and they don’t remember you from last Tuesday. Every time you start a new chat, you’re talking to something that has the knowledge it was trained on and nothing else.
What feels like memory in tools like ChatGPT or Claude is actually a trick: the entire conversation so far gets fed back into the model with every new message. The AI reads the whole thread, from the beginning, every single time you hit send.
It’s not remembering. It’s re-reading.
The context window: your AI’s working memory
The technical term for how much text an AI can process at once is the “context window.” Think of it like a desk. Everything the AI needs to work with has to fit on that desk at the same time: the conversation history, any documents you’ve provided, the system instructions that tell it how to behave, and its own response.
Modern models have large context windows. Claude can handle around 200,000 tokens (roughly 150,000 words). GPT-4o handles 128,000 tokens. That sounds like a lot, and for a single conversation it usually is.
But for business applications, that desk fills up fast.
Say you’re building a customer service bot that needs access to your product documentation, your FAQ, your return policy, and the customer’s order history. Add the conversation itself and the instructions for how the bot should behave. You can blow through 200,000 tokens before the customer finishes explaining their problem.
When the context window fills up, the AI starts dropping older information. Usually from the beginning of the conversation. The customer explained their issue in message one, but by message fifteen the AI has quietly forgotten it.
Conversation memory vs. long-term memory
This distinction matters for business applications, and most AI vendors blur the line between them.
Conversation memory is what happens within a single session. The re-reading trick described above. As long as the conversation fits in the context window, the AI can reference anything that was said. Close the chat, start a new one, and it’s gone.
Long-term memory is persistent information that survives between sessions. Some AI platforms now offer this: ChatGPT has a memory feature that stores facts about you across conversations. Claude has a similar project-level memory. These features are still primitive, though. They store a handful of bullet points, not a rich understanding of your history.
For individual use, this is a minor inconvenience. For business applications, it’s a design problem you need to solve.
Picture this: a customer contacts your AI support agent on Monday about a billing issue. They get a helpful response. On Wednesday, they follow up. If the system has no long-term memory, the Wednesday conversation starts from scratch. The customer has to re-explain everything. That’s a bad experience, and it’s one of the main reasons AI customer service tools get a bad reputation.
Retrieval-augmented generation: bolting on a memory
The most common workaround for the memory problem is retrieval-augmented generation, or RAG. The name is terrible. The concept is straightforward.
Instead of trying to cram all your company’s knowledge into the AI’s context window, you build a search system alongside it. When someone asks the AI a question, the system first searches your documents for relevant information, pulls back the most useful chunks, and adds them to the AI’s context along with the question.
Think of it like giving someone an open-book exam instead of asking them to memorize the textbook. The AI doesn’t need to “know” your entire product catalog. It needs to find the right page at the right time.
RAG is how most business AI applications work in practice. Your knowledge base lives in a vector database (a specialized search index), and the AI queries it on every interaction. The quality of your RAG system determines the quality of your AI’s answers. Bad search means bad answers, no matter how smart the model is.
The catch: RAG is not memory. It’s search. The AI isn’t remembering past conversations or building a model of the customer over time. It’s looking up relevant documents for each question independently. That’s useful, but it’s a different thing than what most people mean when they say “memory.”
Session persistence: remembering within a workflow
Between conversation memory and long-term memory, there’s a middle layer that matters for business applications: session persistence.
This is the ability for an AI to maintain context across a multi-step workflow. Not across days or weeks, but across a single task that involves multiple interactions or actions.
An AI agent that processes invoices, for example, needs to read the invoice, check it against the purchase order, flag discrepancies, and generate a report. Each step requires context from the previous ones. If the agent “forgets” the invoice details between steps, the whole workflow breaks.
Session persistence is usually handled by passing a state object between steps. The AI doesn’t remember on its own. The application stores the relevant information and feeds it back at each step. It’s engineering, not AI magic.
But it’s important engineering. Most business AI failures we’ve seen aren’t model failures. They’re context management failures. The AI had the right capabilities, but the application didn’t feed it the right information at the right time.
When memory matters (and when it doesn’t)
Not every business AI application needs sophisticated memory.
Customer service is where it matters most. Customers expect continuity. They don’t want to re-explain their issue every time they reach out. A customer service AI needs, at minimum, access to the customer’s recent interaction history. Ideally it also knows their account details, past purchases, and any open tickets. Note: this isn’t model memory. It’s application design. You pull the customer’s record from your CRM and include it in the context.
Ongoing projects are another case where memory pays off. If you’re using AI to help manage a project over weeks or months, the AI needs to know what’s been decided, what’s been tried, and where things stand. This is where long-term memory features or a well-maintained project document (that the AI can access via RAG) become important.
For one-shot tasks, memory doesn’t really matter. Summarize this document. Draft this email. Analyze this spreadsheet. Self-contained requests where the AI gets everything it needs in a single prompt. Same for pure search and retrieval: if your AI’s job is to answer questions about your knowledge base, RAG handles the work. Each question is independent.
The mistake we see most often: teams building sophisticated memory systems for applications that don’t need them. If your use case is “answer questions about our product documentation,” you need good RAG, not AI memory. Overengineering the memory layer adds cost and complexity without improving results.
Practical patterns that work
If your application does need some form of memory, here’s what we actually use with clients.
The most common pattern is what we call customer context injection. Before the AI responds to a customer, pull their recent interactions, account details, and open issues from your CRM or support system. Include this in the context alongside their current message. The AI doesn’t “remember” the customer. Your application reminds it. This covers 80% of what businesses mean when they say they want “AI with memory.”
For long conversations that might exceed the context window, you can periodically summarize the conversation so far and use the summary instead of the full history. You lose some detail, but the AI maintains awareness of the overall arc. Most AI frameworks have built-in support for this.
Multi-step workflows need a structured state object that tracks what’s been done, what’s pending, and what the AI needs to know for the next step. You pass this state with each interaction. This is standard application development, nothing AI-specific about it.
And here’s one that people often overlook: writing important outcomes back to your knowledge base after they happen. A decision gets made in a conversation, a customer issue gets resolved, a process gets updated. If you write that back so it’s available via RAG in future conversations, you get a form of long-term memory without relying on the AI model to remember anything.
The part that’s harder than it sounds
Context management sounds straightforward on paper. In practice, it’s one of the trickier parts of building production AI.
The core challenge is deciding what context the AI needs, and when. Too little context and the AI gives generic or wrong answers. Too much context and you blow through your token budget, slow down response times, and sometimes confuse the model with irrelevant information. Models can get worse at finding the relevant needle when you give them a bigger haystack.
There’s also the cost dimension. Tokens aren’t free. Every piece of context you add to every AI interaction costs money. For a high-volume application like customer service, the difference between sending 2,000 tokens of context versus 20,000 tokens per interaction adds up to real dollars at the end of the month. We’ll cover costs in more detail in a future article, but memory and context are a line item, not a free feature.
What this means for your AI decisions
If you’re evaluating AI tools or building AI applications for your business, ask these questions:
Does this application need to remember things between conversations? If yes, how is that handled? (If the vendor says “our AI remembers everything,” push them on the specifics. That answer is almost certainly incomplete.)
How much context does the AI need for each interaction, and what’s the cost per interaction?
What happens when the context window fills up? Does quality degrade? Is important information dropped silently?
For ongoing projects or customer relationships, where does the persistent context live? In the AI, or in your systems?
These aren’t exotic architecture questions. They’re the basics. Getting them right is the difference between an AI application that works reliably and one that gives great demos but falls apart when real customers start using it.