PRX team reveals training data recipe: Why 'ugly' photos make smarter AI

The team behind PRX just published their data strategy, and it challenges the prevailing obsession with pristine training sets. Their core argument is refreshingly blunt: for pre-training a 7B-parameter image model, aesthetic perfection is the enemy. They argue that an overly filtered, pretty dataset actually hurts a model’s ability to learn the full, messy distribution of the visual world — things like diverse compositions, lighting conditions, and everyday objects. “Pre-training is for breadth; fine-tuning is for taste,” the post states, drawing a clear line between the two stages of development.

Instead of building a corpus from scratch, the team took a pragmatic shortcut, blending public and internal datasets that were already quality-filtered and deduplicated. The real magic ingredient isn’t the source of the images, but how they are described. The team’s core philosophy is that long, painfully accurate captions effectively neutralize ‘junk’ data. A screenshot, a watermark, or an advertisement stops being noise if it’s meticulously described in the accompanying text. The model learns it as a controllable attribute you can prompt for or away, not as a flaw to be reproduced unconditionally.

The technical architecture reveals some practical engineering trade-offs. The team uses Lance, a columnar format, for the heavy lifting of dataset exploration and curation because it handles feature engineering on billions of rows without forcing a full dataset rewrite. The curated data is then converted to Mosaic Data Shards (MDS) for efficient streaming during distributed training. They also made the call to compute text latents on the fly using their Qwen3-VL text encoder, rather than pre-computing and storing them. The cost was a minor 3-4% hit to throughput — roughly one extra day on a 30-day run — but it kept their shards small enough to fit on a local filesystem and gave them the freedom to swap text encoders later without rebuilding the entire dataset, a flexibility they already used once.