A constraint-aware GPU allocator outperformed a FIFO scheduler by as much as 33 percentage points in utilization and 105% in priority-weighted output across seven benchmark scenarios — all on identical hardware running identical workloads. The only variable was the order in which allocation decisions were made.
The core problem is two incompatible allocation shapes competing for the same GPUs. Batch-like jobs (training, batch inference, quantization) need contiguous GPU blocks held uninterrupted until completion. Real-time inference is elastic, scaling with traffic demand that shifts every timestep. A FIFO scheduler handles this by reserving each real-time application’s maximum daily demand around the clock, which means an app needing six GPUs at midday but two at 4am holds all six for 24 hours. Those four idle GPUs are neither used nor free. That’s why the baseline sits at 51.6% and 53.6% utilization in the two reservation-dominated scenarios.
Under contention, the ordering problem compounds. FIFO places jobs in arrival order without weighing priority or checking what else needs to fit within the scheduling horizon. High-priority work waits behind whatever asked first, and capacity gets committed in placements that later jobs can’t use. As the piece puts it, order is not a tiebreaker after capacity is settled — order is a capacity decision. The airline analogy is apt: assigning aircraft to whichever charter called first, then finding nothing left for the route that actually pays.
The allocator’s results across five contention-heavy scenarios moved utilization from a 52–85% band to 72–88%, while priority-weighted value rose between 24.6% and 105.1%, averaging 52%. Every scenario improved on both metrics with no tradeoff to explain away. That’s notable because optimization work in scheduling usually forces a choice between throughput and fairness. Here the constrained search over GPU-job-timestep combinations found allocations that were simply better on both axes. The question worth asking next is whether these gains hold as cluster heterogeneity grows — more varied GPU generations, more network topologies, more job types. The combinatorial space gets uglier fast.
