The metrics we’ve been using to judge AI object removal are quietly misleading us. A team from MiLM Plus at Xiaomi dropped a paper at ACM MM 2026 that lays out just how broken the old guard — PSNR, SSIM, LPIPS — really is for this task. Their new evaluation harness, PROVE, is now live under Apache 2.0, and it flips the script by scoring spatial and temporal coherence directly in DINOv2 feature space without ever needing a reference video.
Xiaomi’s core argument is structural: erasing an object is a one-to-many problem. There’s no single correct fill, so any metric that demands point-to-point correspondence with a ground truth is doomed. The paper shows that simply cutting diffusion inference steps actually boosts PSNR and SSIM scores even as visual quality craters — a textbook case of regression to the mean. Even no-reference metrics like ReMOVE and CFD have glaring blind spots. When the team progressively blurred the masked region on ROSE-Bench, both metrics stubbornly refused to degrade, and in some cases scored the blurred output higher than the clean original. CFD’s hallucination detector also embarrassingly flagged a legitimately restored bicycle seat as a fabrication.
PROVE ships as two metrics. RC-S handles spatial coherence by sliding a window across DINOv2 features inside the edited region and computing Maximum Mean Discrepancy against local background features. RC-T does the same across adjacent frames, but only in the intersection of the two restored regions to avoid misalignment artifacts. On human preference benchmarks aggregated via Borda count from 20 participants, RC-S hit a 0.59 average Kendall’s τ and 0.66 Spearman’s ρ. That’s more than double ReMOVE’s 0.26/0.29 and CFD’s 0.16/0.18. It also correctly preferred the clean image over blurred and region-swapped variants in 100% of cases on RORD-Val, something neither competitor managed.
The practical angle here is speed and deployability. RC-S runs at 134.6 ms per frame on a single RTX 4090, making it 13.7× faster than CFD. That’s fast enough for nightly CI gates at startups, and the lack of a reference requirement makes it genuinely useful for large editing catalogs where paired ground truth simply doesn’t exist. The repo needs Python 3.10+, PyTorch 2.6+, and DINOv2-giant weights, with masks as the only mandatory input. Xiaomi explicitly positions this as an evaluation harness, not a product feature — think model bake-offs, tuning inference steps, or filtering training data. The main limitation is that large shadows or reflections extending beyond the cropped evaluation window will slip through, and it’s not built for real-time on-device scoring. For anyone shipping object removal in smartphone galleries, short-video apps, or e-commerce pipelines, this is the first metric that actually correlates with what human eyes see.
