Google just shipped PaliGemma, an open family of vision-language models that pairs the SigLIP-So400m image encoder with the Gemma-2B text decoder. Available immediately on Hugging Face with full transformers integration, the release spans three resolutions — 224x224, 448x448, and 896x896 — and three precisions: bfloat16, float16, and float32. That’s a lot of checkpoints to sort through, but the practical guidance is refreshingly direct: the 224 versions handle most tasks well, and you only need the bigger resolutions for fine-grained work like OCR.
Don’t expect a chatbot. PaliGemma is explicitly single-turn and not designed for conversation. Instead, you prompt it with task prefixes like “detect” or “segment” to switch between captioning, visual question answering, object detection, and referring expression segmentation. The pretrained checkpoints are meant to be fine-tuned for specific downstream tasks, not used out of the box. For quick experimentation, Google also released “mix” checkpoints fine-tuned on a task mixture — those are the ones you can actually play with immediately, though only for research purposes.
The detection output is where things get interesting. Bounding boxes come back as special <loc[value]> tokens representing normalized coordinates — you divide by 1024 and multiply by image dimensions to get real pixel values. Segmentation pushes further, emitting both location and segmentation tokens that you can process into masks. It’s a similar approach to PaLI-3, and the document understanding capabilities look genuinely strong in the benchmarks Google published.
For anyone who’s watched Google’s open model strategy, PaliGemma fits a clear pattern: ship capable research artifacts, keep the truly polished versions behind APIs, and let the community build fine-tuned variants. The three-precision setup is a practical nod to hardware constraints — bfloat16 for modern accelerators, float32 for compatibility. If you’re prototyping a vision app that needs grounded outputs rather than chatty descriptions, this is worth a serious look. Just budget your memory accordingly: those 896x896 checkpoints generate much longer input sequences, and the quality gain is marginal for most use cases.
