Google’s FLAN showed that language models could learn to follow instructions across thousands of tasks, achieving zero-shot generalization to things they’d never seen. InstructPix2Pix applied that same philosophy to image editing, teaching Stable Diffusion to follow general editing instructions. But general instructions only get you so far. Ask InstructPix2Pix to cartoonize an image and you get mediocre results no matter how much you fiddle with the image guidance scale or inference steps. The problem isn’t the approach — it’s the training data.
A new project from the Hugging Face community tackles this directly by building instruction-prompted datasets for specific image processing tasks. The pipeline is clever in its simplicity: ChatGPT generates 50 synonymous variations of the instruction “Cartoonize the image,” then a subset of 5,000 Imagenette samples gets processed through Whitebox CartoonGAN to create input-output pairs. Those cartoonized outputs become the labels the model learns from — effectively transferring Whitebox CartoonGAN’s biases into a fine-tuned InstructPix2Pix model. The researchers extended the same approach to low-level tasks from the MAXIM benchmark: deraining, denoising, low-light enhancement, and deblurring, pulling varying sample counts from public datasets and combining them into a single multi-task training mixture.
The FLAN parallel is explicit here. Just as FLAN V2 mixed thousands of language tasks to get zero-shot generalization on unseen ones, this work mixes multiple image processing tasks into one training run with the hope that a single model can handle them all. Early results from fine-tuning InstructPix2Pix on the cartoonization dataset are described as promising, though the full experimental details remain in the accompanying code repository.
What’s notable is the pragmatic framing. This isn’t a new architecture or a novel training objective — it’s a data engineering play. The insight is that publicly available paired datasets for tasks like deraining and denoising can be repurposed as instruction-tuning data simply by wrapping them in natural language prompts. If the approach generalizes, it suggests that the bottleneck for instruction-tuned image models isn’t compute or model capacity but rather the availability of properly formatted paired data. The open questions section hints at limitations, but the broader implication is clear: instruction-tuning for image models may follow the same trajectory as language models, where data mixture and task diversity matter more than architectural tweaks.
