Fine-tuning Stable Diffusion with Dreambooth isn’t a set-it-and-forget-it process. The team behind this deep dive ran a series of controlled experiments on 2x 40GB A100s using the train_dreambooth.py script with AdamW, and the results are clear: aggressive hyperparameters produce garbage, while patience pays off.
Across four datasets—three object categories and one human subject—a learning rate of 2e-6 consistently outperformed 5e-6. The lower rate gave cleaner outputs with fewer color artifacts and noise remnants. For objects, 400 training steps at that lower rate hit the sweet spot. Faces were far more finicky: they needed roughly 1200 steps at 1e-6 to 2e-6 before results stopped looking like melted wax. The Kramer character from Seinfeld was their test subject for the face experiments, and early attempts at higher learning rates produced what they described as ‘not stellar’ results.
Prior preservation helped—but not universally. When training on a person, generating additional class images of other people via Stable Diffusion itself reduced overfitting noticeably. Side-by-side comparisons with and without prior preservation at 1200 steps showed the preservation runs produced fewer noisy blotches. However, the authors note the technique isn’t a magic bullet; artifacts still lingered.
One of the more practical findings involves the inference scheduler. When a model overfits, DDIM dramatically outperformed PNDM and LMSDiscrete during sampling. The team observed that LMSDiscrete produced ‘terrible’ results on the Kramer face, while DDIM was ‘much better.’ Running inference for around 100 steps also helped resolve noise patches into coherent image details. This is a useful diagnostic: if your fine-tuned model is outputting garbage, try switching schedulers before retraining.
Finally, the researchers found that unfreezing the text encoder—contrary to the original Dreambooth paper—improved results. This mirrors what other implementations like those from @nitrosocke have discovered independently. The takeaway for practitioners: don’t trust the original paper’s defaults. Experiment with the text encoder, lower your learning rate, and give faces extra training time. And if you’re on a 16GB GPU, 8-bit Adam and gradient accumulation make these experiments feasible without a data center budget.
