Agent Memory Mechanism: How to Design Short-Term and Long-Term Memory

When an agent forgets earlier parts of the conversation, the problem usually isn’t the model — it’s that nobody designed memory for it. Split short-term and long-term memory into separate management, and AI goes from a one-shot tool to a partner you can work with long-term.

Short-term memory: handle the current step

Short-term memory, also called working memory, only serves the current task. The web pages the agent grabs while researching, and the intermediate results computed in the previous step, all belong to it. Its trait is that it comes fast and goes fast; it can be cleared once the task ends, no need to keep it long. Think of it as the sticky note beside a worker’s hand — torn up when the job’s done.

Two design points. First, don’t let the context window blow up; in practice people commonly use a rolling window keeping only the last 5 to 8 turns, or compress each completed step into two or three sentences of summary before passing it on, instead of piling in the raw running log. Second, when multiple tasks run in parallel, short-term memory must be isolated per task; otherwise task A’s intermediate conclusions leak into task B, results drift inexplicably, and debugging becomes extremely hard to pinpoint.

Long-term memory: handle cross-task accumulation

Long-term memory solves “done this time, usable next time.” It suits stable, repeatedly useful information: user preferences, project background knowledge, common business rules, and pitfalls you’ve hit before. A customer-service assistant with long-term memory doesn’t need you to re-explain your company intro on the second conversation.

There are three common media. A vector database excels at semantic retrieval — you don’t need to remember the exact wording; similar meaning can pull it back. A structured database suits storing definite facts, like “customer A’s unit price is X” or “this process must be approved before publishing.” A user-profile file writes a person’s preferences into a fixed document, read once before each task, so the agent knows what tone to use and what landmines to avoid.

Retrieval needs a switch — don’t dump everything

As long-term memory piles up, its volume keeps growing. If you stuff all memory into the model every time, the window fills fast and cost skyrockets. The right approach is adding a retrieval switch: first generate a query from the current task goal, pull the most relevant few items from the memory store, then stitch them into the context. It’s like not carrying the whole bookshelf to the desk — just pulling the few books you need.

On concrete numbers, taking top-3 to top-5 items per retrieval is usually enough. Too few misses key background; too many returns to the old window-blowout problem. You can set a relevance threshold — for example, only items with cosine similarity above 0.75 qualify; those below stay put in the store. This threshold should adjust with task type: casual chat can be looser, serious decisions stricter.

Memory goes stale and needs correcting too

Long-term memory isn’t write-once-forget-forever. User preferences change, project agreements change, and old conclusions can conflict with new facts. Give each memory item a time tag and periodically review expired ones; when new information contradicts, prefer the newer item, or surface the conflict for a human to confirm — rather than letting old memory silently suppress new facts.

Neglect maintenance, and the memory store turns from an asset into a burden. A real pitfall: I once had a customer-service agent remember “the user hates being rushed,” and it stored the whole chat verbatim into long-term memory; next time it wouldn’t even send a normal progress reminder. After changing it to store just a one-line preference label, the problem disappeared instantly. Memory should be distilled — store conclusions, not running logs.

A small implementation case: the competitor weekly report

Take “competitor weekly report” as an example. Step one, grab five competitors’ homepages, getting a large chunk of raw webpage code. If you stuff all five raw files into the next step directly, the context immediately fills with webpage noise. Better: after grabbing each one, first have the model compress “what changed on this page vs last week” into two sentences, pass only those two to the summarization step, and treat the raw pages as temporary evidence used once then discarded.

At summarization, turn on the retrieval switch and pull back long-term memory like “customer A’s unit price” and “last week’s conclusion” for comparison, producing a differential weekly report. The whole chain stays lightweight throughout — neither losing key information nor being dragged down by redundancy. This approach strings together three things — short-term summarization, long-term retrieval, and per-task isolation — and is the minimal viable template for memory design.

Three anti-patterns in memory design

Anti-pattern one: storing a whole conversation verbatim into long-term memory. It looks information-complete but actually treats noise as assets; next time the behavior gets led astray by old running logs. Store only distilled conclusions, like “the user prefers brief answers,” not a few hundred words of chat history.

Anti-pattern two: dumping everything into the vector database regardless of media. Fact-type information gets distorted by vector retrieval; it should go into the structured store. Anti-pattern three: write-only, never delete. A memory store without an expiry mechanism becomes untrustworthy within six months. Memory quality matters more than quantity; periodic subtraction is the core of maintenance.