Stanford PhD's Rust tokenizer hits 24.5 GB/s — 989x faster than HuggingFace

Tokenization is the plumbing of language models, the unglamorous step nobody bothers to optimize. Marcel Rød, a Stanford PhD student, just proved that assumption was colossally wrong. His new library, Gigatoken, encodes text so fast it makes existing tools look like they’re standing still.

On a dual-socket AMD EPYC server with 144 cores, Gigatoken chewed through the 11.9 GB GPT-2 benchmark corpus at 24.53 GB/s. That’s not a typo. OpenAI’s tiktoken managed 36.0 MB/s on the same hardware, while HuggingFace tokenizers clocked 24.8 MB/s. You’re reading that right — speedups of 681x and 989x respectively. On an Apple M4 Max, the gap widens to 1,268x over HuggingFace. Consumer hardware sees similar fireworks: a Ryzen 7 9800X3D hits 6.27 GB/s, or 106x the HuggingFace baseline.

These aren’t benchmark tricks. Rød achieved this by rethinking two problems most libraries treat as trivial. First, he replaced regex-based pretokenization with a hand-written state machine that eventually evolved into a branchless SWAR implementation eating 1,049 MiB/s on a single thread. The final breakthrough was dual-cursor instruction-level parallelism — running two independent cursors from a safe split point so the CPU’s out-of-order engine keeps every execution port busy. Second, he added pretoken caching to skip recomputation of words the tokenizer has already seen. His optimization log is refreshingly honest about what didn’t work: profile-guided optimization was useless, a hot/cold split actually made things slower, and a clever two-pass buffer got murdered by memory traffic.

Gigatoken ships on PyPI under an MIT license, supporting 23 tokenizer families from GPT-2 to Llama 4 to DeepSeek V3. The native API lets Rust read files directly, which is where the headline numbers come from. A Python compatibility mode exists but pays a real penalty — around 200-300x speedups instead of the full 989x — because it still gets bogged down in list creation and string-to-bytes conversion overhead. The lesson here isn’t just that tokenization was slow. It’s that the boring parts of the stack still hide enormous gains for anyone willing to actually profile them.