Training a ControlNet from scratch sounds like the kind of project that requires a research lab, a stack of GPUs, and weeks of patience. The reality, according to a new walkthrough from the Hugging Face team, is considerably more pedestrian. The whole process breaks down into three steps: pick your conditioning, build your dataset, and run the diffusers training script. The team behind the Uncanny Faces model did exactly that, renting a single A100 from LambdaLabs at $1.10 an hour.
The condition they settled on was facial landmarks, reasoning that the existing landmark ControlNet already works well and that facial expression imitation could be genuinely useful. Their dataset plan hit a snag almost immediately. Microsoft’s FaceSynthetics dataset, with its 100K synthetic faces, annotates landmarks in the iBUG 68-point format, but no existing model converts regular face photos into that exact format. They pivoted: use FaceSynthetics ground truth images, run the SPIGA model to extract 68 facial landmarks, then write custom code to render those landmarks into an illustrated mask for the conditioning image. Captions came from running the entire dataset through BLIP, producing the Face Synthetics SPIGA with captions dataset on Hugging Face.
The training itself turned out to be the least dramatic step. Three epochs over the 100K-image dataset, batch size of 4, on one rented A100. The diffusers training script handled the heavy lifting. That’s a notable shift from the early days of ControlNet, when reproducing the original paper’s results meant wrangling custom training loops and praying your VRAM held out. The infrastructure has matured to the point where the bottleneck is dataset engineering, not compute.
What’s interesting here is how the hard problems moved upstream. The team spent most of their effort on the unglamorous stuff: finding a landmark extraction model that matched their chosen format, writing conversion code, generating captions. The actual model training was almost an afterthought. If you’ve been putting off training a custom ControlNet because the compute seemed intimidating, the economics have changed. Your real constraint is whether you can assemble a clean dataset with consistent conditioning images. The Uncanny Faces result, by the way, wasn’t quite what they intended, which suggests that even with a solid pipeline, synthetic faces bring their own quirks to the output.
