OpenAI's Jukebox generates music—vocals and all—from raw audio

OpenAI just dropped a neural network that doesn’t just compose melodies—it generates full songs, complete with rudimentary singing, as raw audio. They’re calling it Jukebox. The model weights, code, and a tool for exploring the generated samples are all publicly available. You give it a genre, an artist style, and some lyrics, and it produces a new music sample from scratch.

This is a fundamentally different beast than the symbolic MIDI-based generators the world has tinkered with for decades. Those piano-roll approaches can churn out Bach chorales or multi-instrument pieces, but they’re deaf to the human voice and the subtle textures that make a track feel alive. Jukebox tackles the problem at the audio level, a domain OpenAI describes as “particularly unforgiving of errors.” The sheer scale is daunting: a 4-minute CD-quality song contains over 10 million timesteps. For context, GPT-2 handled 1,000.

To manage that complexity without losing its mind, Jukebox uses a multi-stage process. It starts with a VQ-VAE autoencoder that compresses raw audio into a discrete, lower-dimensional space across three hierarchical levels—compressing the original 44kHz signal by 8x, 32x, and 128x. OpenAI modified the standard VQ-VAE-2 architecture with random restarts to prevent “codebook collapse” and a spectral loss function to better reconstruct high frequencies. Separate decoders then independently reconstruct the input from each level’s codes, maximizing the use of the compressed representation.

After compression comes generation. Three levels of autoregressive “prior” models—based on a simplified Sparse Transformer with 72 layers of factorized self-attention—learn the distribution of these music codes. The top-level prior handles the long-range song structure on a 24-second context window. The middle and bottom upsamplers then fill in the local details like timbre, boosting the fidelity. The entire system was trained on a new dataset of 1.2 million songs, 600,000 of them in English, scraped from the web. The result isn’t just a clever parlor trick; it’s a signal that raw audio generation is catching up to the kind of high-level semantic control we’re used to seeing in text and images.