Energy-based models match GANs without a generator network

Energy-based models (EBMs) are making a quiet but significant comeback, and a new paper suggests they can now go toe-to-toe with GANs on image quality while retaining the mode coverage guarantees of likelihood-based models. The trick isn’t a new architecture—it’s simply spending more compute at test time. By running iterative Langevin dynamics refinement for longer periods, researchers were able to generate samples that look sharp and diverse, matching what we’d expect from a well-tuned GAN at low temperatures.

This matters because EBMs sidestep one of the fundamental constraints that has always bugged me about VAEs and flow-based models. Those approaches force a generator network to learn a continuous map to a potentially disconnected data distribution—think of trying to draw a single unbroken line through separate islands of data. EBMs don’t have that problem. They just assign low energy to valid configurations, whether those regions are connected or not, which makes them naturally better at handling multimodal data.

The team also tested EBMs on out-of-distribution detection and found them outperforming autoregressive and flow-based models—despite never being explicitly trained for classification. Even more interesting: when pressed into service as classifiers against adversarial examples, these energy models held up better than networks purpose-built to resist such attacks. That suggests something structurally robust about how EBMs carve up the input space.

There’s also a neat compositionality trick buried in the findings. By training separate models on different attributes—say, shape size at a fixed position, and position for a fixed shape—and then combining them as a product of experts, you can generate samples that respect both constraints without retraining anything. The researchers are refreshingly honest about what didn’t work too: vanilla HMC was finicky, gradient penalties killed capacity, and ensembles weren’t worth the complexity. Section A.8 of their paper is apparently a goldmine of failed experiments, which is exactly the kind of candor this field needs more of.