Classifying 1B texts doesn't need a frontier LLM; here's the real cost down to the penny

Running inference on a billion inputs sounds like a job for a hyperscaler’s budget, but the economics shift completely when you drop the LLM and reach for an encoder model. At this scale—think classifying English Wikipedia 144 times over—the architecture you pick is the single biggest lever on cost. I wanted to cut through the guesswork, so I built a reproducible benchmark using Hugging Face Inference Endpoints, the Infinity inference server, and Grafana k6 for load testing. The goal wasn’t to crown a winner in some framework war, but to give you a clear, adaptable methodology to answer two questions: what’s the cheapest batch configuration, and how do you keep latency in check under heavy load?

The benchmarking setup is deliberately modular. You can swap in your own GPUs, your own deployment process, but the core loop stays the same: sweep across batch sizes and virtual users to find the saturation point for a single GPU. Most encoder models fit comfortably on one, which makes the math clean. Once you’ve squeezed maximum throughput from that single unit, scaling becomes a simple horizontal game of adding replicas. I skipped CPU testing entirely after initial runs showed Nvidia T4s were so cost-effective that grinding it out on CPUs didn’t justify the engineering time, though I left the code hooks in for anyone curious enough to prove me wrong.

What surprised me wasn’t just the raw cost numbers that emerged, but how sensitive throughput is to the interaction between batch size and virtual users. Too few concurrent requests and your GPU sits there, streaming multiprocessors half-asleep. Crank the batch size too high without the VRAM headroom and you choke. The sweet spot is an empirical question, not a spec-sheet calculation, and k6’s pre-allocated VUs turn out to be essential for simulating realistic load without the overhead that would skew your results. I also leaned heavily on Infinity for this work because its default model compatibility meant I spent zero time wrestling with transformers library integration—a non-trivial time-saver when you’re iterating fast across model choices.

One undercurrent here that the source doesn’t spell out explicitly: the LLM industry’s obsession with giant generative models has left a blind spot around how much work you can actually offload to focused encoders at a fraction of a cent per thousand inferences. If you’ve been reaching for an API that charges by the token to classify support tickets or tag content, the numbers from this benchmark might make you reconsider your entire pipeline. The full code is on GitHub, and I’d genuinely be interested to see what numbers people get on their own hardware—my feelings won’t be hurt if you find a cheaper config.