Research
DeepSpeed was secretly upcasting to FP32 — here's why your FSDP loss diverged
Hugging Face Blog · Jun 13, 2024 · 2 min read
Running the same training pipeline with DeepSpeed and PyTorch FSDP should produce similar results. But when Hugging Fac...