Hugging Face and bitsandbytes have teamed up again, this time to bring 4-bit quantization to a huge swath of models on the hub. The practical upshot: you can now run a 20B parameter model like GPT-neo-X on a free Google Colab instance. That’s not a typo. The same notebook also shows how to fine-tune that 20B model downstream without renting a single A100.
The backbone here is QLoRA, introduced in a new paper by Dettmers et al. The method freezes a 4-bit quantized pretrained model and backpropagates gradients into Low Rank Adapters, or LoRA. According to the abstract, QLoRA cuts memory use enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning performance. Their best model family, Guanaco, hits 99.3% of ChatGPT’s performance on the Vicuna benchmark after just 24 hours of finetuning on one GPU.
The paper credits three memory-saving tricks: a new 4-bit NormalFloat (NF4) data type optimized for normally distributed weights, double quantization that shrinks the average memory footprint by quantizing the quantization constants themselves, and paged optimizers to handle memory spikes. The team finetuned more than 1,000 models across 8 instruction datasets, model types like LLaMA and T5, and scales up to 65B parameters. Notably, they argue GPT-4 evaluations are a reasonable, cheap stand-in for human evaluation—but also warn that current chatbot benchmarks are not trustworthy for accurately measuring chatbot performance. They even include a ‘lemon-picked’ analysis showing where Guanaco fails compared to ChatGPT.
The resources are solid: the original paper, two Colab notebooks for inference and finetuning, the replication repo, and a Guanaco 33B playground. FP8 and FP4 data types get a brief explainer, with E4M3 and E5M2 encodings covering ranges from -448 to 448 and -57344 to 57344 respectively. This is the same playbook that made 8-bit inference mainstream last year. 4-bit could do the same for finetuning.
