Filtering 1,493 videos down to just 47: The brutal reality of building clean AI video datasets

If you’ve ever wondered why fine-tuned video models often produce glitchy, watermarked, or just plain ugly outputs, the answer almost always comes back to the data. A new open-source pipeline from the community lays bare just how punishing the curation process really is — and provides the scripts to make it less painful.

The tooling mirrors the industrial-scale approaches used by models like Stable Video Diffusion but packages it for smaller teams. Stage one handles acquisition, using yt-dlp to grab videos and a scene-splitting script to chop them into usable clips. Stage two is where the massacre happens: every frame gets run through LAION-5B watermark detection, an aesthetic predictor, and NSFW screening via Falconsai’s model, while OpenCV scores entire clips for motion. Stage three leans on Microsoft’s Florence-2 for multi-layered captioning — everything from simple descriptions to dense region captions and OCR extraction.

The creators put their own pipeline to the test with a crush-effect dataset. Out of 1,493 initial videos, only 47 survived their thresholds of pwatermark below 0.1 and aesthetic score above 5.5. That’s a 3.1% survival rate. But the post-mortem reveals something more interesting than the brutality of the numbers. Watermark detection proved effective at catching text overlays but also flagged a toy car’s license plate — a false positive that nuked perfectly good training data. The fix? Averaging watermark scores across frames rather than requiring every single frame to clear the bar. The aesthetic scorer showed its own quirks: a pink castle that scored 5.5 in one frame dropped to 4.44 when crushed, and colorful round objects consistently rated higher. The authors suggest using only the first frame’s aesthetic score, and lowering the minimum threshold to 4.25-4.5 to avoid filtering out legitimate content.

This is the kind of nitty-gritty insight that dataset papers usually gloss over. What makes this release genuinely useful isn’t just the code — it’s the frank discussion of where the filters fail and how to tune them for your specific use case. The team already fine-tuned CogVideoX-5B on their crush datasets using the finetrainers library, and they’re actively adding more filters to the repo. For anyone trying to replicate the slick “Pika Effects” style transformations with open models, this is the closest thing to a field manual.