Hugging Face just open-sourced a complete recipe for turning Meta’s LLaMA into a question-answering model trained with human preferences. The result, StackLLaMA, answers Stack Exchange questions and the entire pipeline — supervised fine-tuning, reward modeling, and RLHF — is now available in the TRL library.
The economics here are the real story. Training a 7B parameter model conventionally would eat roughly 70GB of memory just for weights and optimizer states, which doesn’t even fit on an 80GB A100. The team sidestepped that wall using two tricks: loading the model in 8-bit (cutting weight memory to 7GB) and applying Low-Rank Adaptation, which freezes the original weights and trains small adapter layers instead. Their rule of thumb: allocate 1.2-1.4GB per billion parameters. That puts 7B comfortably within a single GPU’s reach.
For training data, they mined Stack Exchange, where every answer carries upvotes and an acceptance flag. Following the approach from Askell et al. 2021, each answer gets a score: log2 of one plus upvotes, rounded, plus a bonus point if the questioner accepted it. Negative scores land at -1. They sampled at most ten answer pairs per question to keep the reward model’s comparisons manageable and converted HTML to Markdown for cleaner outputs.
StackLLaMA is available on the Hugging Face Hub with a live demo, and the processing notebook is public too. This is the clearest end-to-end RLHF tutorial I’ve seen yet — most teams keep their reward model training details vague. The fact that it runs on a single A100, rather than a distributed cluster, means smaller labs and independent researchers can finally replicate the InstructGPT recipe without a seven-figure compute budget.
