That gray in your training batch is burning cash: How a knapsack cut padding waste from 60% to zero

If you’ve trained a multimodal model, you know the pain: GPUs sitting idle while your data loader catches up, and batches bloated with padding tokens that represent nothing but wasted compute. The team behind the nanoVLM project dissected their own pipeline and found a staggering 60% of their batch was useless padding. The culprit wasn’t the model architecture or the hardware—it was a naive data pipeline that padded every sequence to match the longest one in the batch.

Their fix came in stages, each documented in a new dedicated repository called mmdp. The initial “constrained padding” approach simply set a global maximum length and dropped samples that exceeded it. Better, but still fundamentally wasteful—every sequence got stretched to that fixed limit regardless of its actual content. The real breakthrough arrived when they reframed batching as a classic computer science problem: the knapsack.

Think of each training batch as a backpack with a strict token limit. Each sample is an item whose “weight” is its token count. The goal isn’t to make everything uniform—it’s to cram as many useful sequences as possible into that backpack without going over the limit. To pull this off dynamically, they ditched PyTorch’s standard map-style datasets for an iterable one, which lets batches get assembled on the fly. A producer-consumer pattern using Python queues keeps the packing logic from becoming a bottleneck, with strategies like “greedy” and “binpack” determining how sequences get combined.

The result is a pipeline where padding becomes an occasional gap-filler rather than the main event. For anyone training vision-language models on a budget, this isn’t an academic exercise—it’s the difference between paying for idle GPUs and actually using the compute you’re renting. The code is available now, stripped clean of the larger nanoVLM project so you can drop it into your own pipeline.