A distributed team from the TWIML Slack community fine-tuned OpenAI’s CLIP model on remote sensing imagery during Hugging Face’s Flax/JAX Community Week in July, and the results suggest satellite photos needed the adjustment more than anyone assumed. The group trained on roughly 12,700 images pulled from three datasets—RSICD contributed about 10,000 images from Google Earth, Baidu Map, MapABC, and Tianditu, while the UCM and Sydney datasets added another 2,700. Each image carries up to five human-written captions, and that pairing is what CLIP uses to learn a joint embedding space where matching images and text land close together.
The training itself ran on Google Cloud TPUs using JAX and Flax, with compute time provided free as part of the event. Around 100 teams participated overall, producing 170 models and 36 demos. The team’s intuition was straightforward: CLIP performs zero-shot classification on everyday photos without fine-tuning, but overhead satellite imagery is visually alien enough—different angles, resolutions, and feature scales—that a generic model would struggle. Their evaluation results backed that up, though the post doesn’t include specific benchmark numbers.
The practical payoff is a text-based search engine for satellite image collections. A user can query “beach” or “baseball field” or describe specific geographic features and retrieve relevant imagery without manual tagging. That capability cuts both ways. The team points to legitimate uses like tracking climate change impacts and national defense, but also acknowledges the same tool could serve authoritarian surveillance. It’s a rare case of an AI team flagging the dual-use problem in their own project write-up rather than leaving it to outside critics.
What stands out here isn’t just the model—it’s the pattern. The same fine-tuning approach was applied to medical images by another team during the same event, and that domain-adaptation playbook is likely to repeat across industries where off-the-shelf CLIP falls short. Whether the fine-tuned model generalizes beyond the three training datasets remains the open question, and the team says future work will address that. For now, the model and demo are both public on Hugging Face.
