The gap between what open-weight models can do and what individual developers can actually run keeps widening. Kimi-K3, GLM-5.2, and DeepSeek-V4-Flash are competitive with proprietary systems, but serving them still assumes datacenter hardware. A team from UC Berkeley and UT Austin thinks that assumption is the real bottleneck. Their answer is FreeToken, an Apache-2.0 serving system that treats a personal machine as a unified inference platform rather than a small GPU.

The numbers are hard to ignore. FreeToken runs a 35B model at interactive speed on an 8 GB laptop GPU, a 284B model on a gaming desktop, and the full 753B GLM-5.2 on a single workstation card. On an RTX 5090, it sustains 77–83 tokens per second on Qwen3.6-35B-A3B and 22–25 tok/s on DeepSeek-V4-Flash — 1.5 to 2.3 times faster than the strongest baseline. Worst-case time-to-first-token stays under 44 seconds across every test cell, while llama.cpp hits 232 seconds, Ollama 179, and KTransformers a staggering 946 seconds somewhere in the matrix.

The team identifies three failure modes in existing engines. Prefill destroys sparsity because thousands of tokens route to nearly every expert, forcing the entire pool across PCIe. Static placement misses decode traffic since routing shifts every token while the GPU sits idle. And consumer CPUs simply can’t carry the remainder: dual-channel DDR5 delivers 80–90 GB/s against the 1–1.8 TB/s a modern GPU draws from on-package memory. FreeToken’s bandwidth-adaptive execution splits each step’s cache misses proportionally between GPU and CPU based on profiled bandwidth — measured at 52.7:77.3 on an RTX 5090 server but just 11.8:47.5 on a 4060 laptop, which explains why one-size-fits-all placement fails so badly.

This matters most for solo developers, startups, and SMB teams whose agent token bills already exceed the cost of a GPU they own. The strongest fit is healthcare and legal, where data never leaving the machine is a compliance requirement, not a preference. FreeToken ships as a one-click desktop app, a pip package, and a CLI that exposes OpenAI- and Anthropic-compatible endpoints — so Claude Code, Codex, and OpenCode can all point at your own hardware. Whether this finally makes local frontier inference practical for real work, or just moves the cost from API bills to electricity bills, is the question worth watching.