Stop Over-Engineering Your AI: Use This 5-Question Tree to Pick the Right Memory

Most agent memory strategies are an afterthought, and it shows. Some agents goldfish on users mid-conversation while others lug around vector databases stuffed with trivia they’ll never use. The real fix isn’t a fancier vector store. It’s sitting down and asking a brutally simple question for each piece of information: how long does this actually need to live?

That’s the core of a decision tree that breaks agent memory into four distinct layers. Working memory is the active conversation—ephemeral and token-bound. Semantic memory is your canonical facts: a user’s name, their subscription tier, a product spec that doesn’t change. Episodic memory is the narrative history, the record of complaints and decisions. Procedural memory is learned skill, getting faster at a refund routine the tenth time you do it. They answer fundamentally different questions, which is why a customer support bot might use all four at once for different data categories.

The tree itself runs five questions, and you apply it per category of information, not to the agent as a whole. First: does this persist beyond the current turn? A one-off classification request doesn’t. Second: if it persists, does it describe a stable, reusable fact? That’s the line between semantic memory and everything else. Third: is the value in the sequence of events, or just the current state? That splits episodic from the rest. Fourth: can the system get better at this task through repetition? That’s your procedural memory.

Where things go wrong is predictable. Shoving stable facts into a vector search is slower and less reliable than a structured lookup. Mining a full chat history for a preference that changed three months ago will surface stale noise that a simple profile overwrite would have fixed. Context windows are a scarce resource—don’t waste tokens on retrieval that doesn’t improve the response.

The decisions aren’t architectural grandstanding; they’re about not quietly dropping information the agent still needs or over-fetching noise that confuses it.