Perplexity squeezes embedding costs with one trick: reuse your LLM stack

Perplexity just published a rare under-the-hood look at how it serves embedding models, and the headline is refreshingly contrarian: they didn’t build a dedicated embedding engine. The engineering team argues that small Transformer models make batch embedding look like compute-bound prefill, while online embedding of a few tokens mimics memory-bound decode. So ROSE, their runtime-optimized serving engine, simply reuses the prefill and decode kernels already powering Perplexity’s LLM stack. That’s the kind of architectural restraint you don’t see often in AI infrastructure, where the default impulse is to spin up a new subsystem for every workload.

The serving path splits across three services. Ivy, a Rust HTTP gateway, handles tokenization, JSON parsing, and splits large batches into chunks before load-balancing across replicas. Tulip, a gRPC server built with tokio and tonic, manages scheduling and batching. ROSE does the actual inference and exposes a step() function to Tulip. The scheduler is deliberately first-come, first-served — no fancy priority queues. Perplexity justifies this with a measurement: for sub-billion-parameter models at their typical sequence lengths, dense layer cost dominates attention’s quadratic cost, so latency tracks token count, not sequence count. Once a batch hits roughly 512 tokens, the GPU is saturated and packing in more sequences buys nothing.

CUDA graphs do the heavy lifting on small batches, where CPU-side kernel launches can actually outweigh GPU execution. Perplexity captures whole-model graphs to collapse every launch into a single driver call, and they upstreamed changes to FlashInfer to make that possible. The catch is that graphs must be captured per configuration, which means thousands of graphs and minutes of capture per model. Their fix is lazy capture: warm up eagerly on first hit, capture and replay on the second. A LazyTensor abstraction tracks a page-locked host buffer plus cudaMemcpyAsync and a CUDA event, so step() returns immediately while a Rust async task waits on batch N and the CPU enqueues N+1. On benchmarks against vLLM v0.22.0 in BF16, Perplexity verified cosine similarity divergence stays within 0.1%.

One detail worth flagging for anyone running Qwen-based models: Perplexity reports FlashAttention 4 is generally faster, but FlashInfer 3 wins at very long sequence lengths, so backend selection happens case by case. That kind of nuance rarely makes it into vendor benchmarks. The broader implication is that the embedding serving problem is quietly converging — on mature Hopper and Blackwell hardware, the wins aren’t in the kernels anymore. They’re in the runtime, the harness, and the willingness to treat embeddings as just another Transformer workload rather than a special snowflake.