If you’ve tried to fine-tune a model like t5-large on modest hardware, you know the wall: out-of-memory errors before you can even get a useful batch size going. The ZeRO memory optimization paper from late 2019 promised to change that, and now the Hugging Face Trainer in transformers v4.2.0 actually ships with experimental support for both major implementations—DeepSpeed and FairScale.
The benchmark here is refreshingly practical. Using a translation finetuning task on t5-large with two 24GB Titan RTX cards, the baseline DistributedDataParallel setup maxed out at batch size 16 before hitting OOM. Then the author ran the same workload through six configurations: fp16 alone, FairScale’s sharded DDP (with and without fp16), and DeepSpeed (with and without CPU offloading). Both FairScale and DeepSpeed delivered what the baseline couldn’t—bigger batch sizes and meaningfully faster total training and evaluation time.
As the author notes, DeepSpeed currently implements more of the ZeRO paper’s tricks and came out as the short-term winner on speed, but it requires writing a config file and switching your launcher. FairScale is the low-hanging fruit: you literally just add —sharded_ddp to your existing command line. If you’re running a project that will take hours or days, that extra configuration effort on DeepSpeed is likely worth it, but for a quick experiment FairScale gets you most of the benefit with almost no friction.
One detail worth keeping in mind: the evaluation phase in this translation task is actually slower than training on the same number of samples because it does beam search with size 4. So the speedups reported include both phases, not just the forward pass. The author is upfront that these are proof-of-concept numbers from a few hours of benchmarking, not a fully optimized sweep. If you’re about to run a real multi-day job, spending an afternoon tuning hyperparameters for your specific hardware configuration will almost certainly squeeze out more than what these default settings show.
