Tagged: fsdp

3 articles

Latest

PyTorch FSDP trains GPT-2 XL on 2 GPUs where DDP crashes with OOM

Llama 2 70B fine-tuning slashes RAM from 2TB to 32GB with FSDP trick

DeepSpeed was secretly upcasting to FP32 — here's why your FSDP loss diverged