ACE vs. ALTK-Evolve: One Playbook, Two Delivery Methods—and a 60% Token Gap

Two new systems, ACE and our own ALTK-Evolve, are attacking the same maddening problem: LLM agents know the APIs but flub the execution. They mis-paginate, resolve the wrong entity, or return values nobody asked for. Both solutions function as agentic memory, distilling past failures into reusable lessons without weight updates or human labels. And on the core philosophy, they’re shockingly aligned. Both refuse to compress. ACE fights “brevity bias” with a rich, itemized playbook where every bullet gets a helpful/harmful counter. We use support counts—how many independent episodes produced a guideline—and never summarize the store down to a tidy handful of rules. A lesson five tasks discovered is a different object from a one-off, and both are worth keeping.

Where they split is delivery, and it’s a rift that shows up directly in the token bill. ACE injects its comprehensive playbook on every single step, a constant regardless of model or task. We treat delivery as a dial. A small fixed core of high-support guidelines always goes in, but we extend it per task with a handful retrieved for the job at hand—using cosine or LLM-guided selection, weighted by priority. The same lessons are available; ACE just sends all of them, while we send however many a given model can actually use.

On the AppWorld benchmark with the same base ReAct agent, running both in-house, the numbers are stark. On a strong model like DeepSeek-V3.2, we outperform ACE on both accuracy metrics while burning only about 40% of the inference tokens. On a weaker model, gpt-oss-120b, it’s a statistical tie—we edge ACE 56.0 to 54.8, a gap well within run-to-run noise—but at roughly one-seventh the cost. The cost story flips depending on which axis you care about: ACE’s efficiency is about building its context cheaply; ours is about serving it.

The accuracy breakdown by task difficulty reveals why this isn’t a one-size-fits-all race. On gpt-oss-120b, ACE’s full playbook leads on Easy and Medium tasks—generic instruction-following still gets a boost from a comprehensive prompt. But on Hard tasks, where the model needs to navigate specific, tricky API interactions, our per-task selection pulls ahead and wins the aggregate. The lesson isn’t that one system is better. It’s that a memory store’s value is entirely mediated by how it’s delivered, and that dial matters more than the playbook’s contents once the lessons are solid.