The assumption behind agentic memory sounds almost too sensible to question: let an agent mine lessons from its own past work, feed those lessons back at inference time, and watch performance climb. But when researchers at ALTK-Evolve scaled their evaluation to eight models — from a 30B dense model to frontier proprietary systems — the results told a messier story. Memory isn’t something you switch on. It’s something you calibrate.
The clearest finding is that dosage depends on capability. DeepSeek-V3.2, a 671B MoE model with plenty of headroom, gained 9.5 percentage points in task completion when given its full self-mined guideline set. Every edge case and rare lesson helped. But gpt-oss-120b, a smaller 117B MoE, actually did worse with the full set. When researchers switched to a selective approach — a compact core of high-confidence guidelines plus a handful retrieved per task — the model jumped 16.1 percentage points while using roughly 50% fewer tokens than the full-set configuration. The larger set wasn’t just unhelpful; it was actively drowning the model.
Then there’s the saturated pattern. GLM-5, a 745B MoE, showed no measurable gain from any memory configuration. The researchers are careful to say this label describes what they observed, not a proven cause. It could be that GLM-5 was already near its ceiling on these tasks, or that the guidelines didn’t address its remaining failure modes, or that the model simply didn’t apply the guidance effectively. Separating those factors is ongoing work.
What makes this notable is how cheap the whole approach is. No weight updates, no human annotation — just a learning loop that distills behavioral guidelines from successful and unsuccessful trajectories, then injects them at inference time. That portability across eight different models is exactly what makes the dosage question so pressing. If you’re deploying agents at scale, the difference between full-set and curated retrieval isn’t just accuracy — it’s token cost. And with prompt caching, even the full guideline set can be affordable in production. The practical takeaway is blunt: before you give your agent more memory, figure out whether it can actually use it.
