Pollen Robotics, the French team behind the open-source Reachy humanoid, just released pollen-vision — a library that strings together three zero-shot vision models so a robot can locate unknown objects in 3D space without a single round of training. That’s the part worth pausing on. No custom datasets, no fine-tuning, no waiting for a model to learn what a coffee mug is. You point a camera at a scene, tell the system what you’re looking for in plain English, and it hands back spatial coordinates.
The pipeline leans on three models working in sequence. Google Research’s OWL-ViT handles text-conditioned 2D object localization, spitting out bounding boxes for whatever you prompt it with — “paper cups,” say, or anything else in its open vocabulary. Meta’s MobileSAM then takes those boxes and produces pixel-level segmentation masks. Finally, OPPO’s RAM model tags images to identify which objects are actually present, which matters more than it sounds: OWL-ViT slows down as you add prompts, running at roughly 75ms per frame with one prompt on an RTX 3070 but ballooning to 650ms with ten. Using RAM to filter prompts first keeps the whole thing responsive.
The robotics payoff is in how Pollen converts that segmentation mask into real-world coordinates. By averaging depth values inside the mask rather than the full bounding box, the system avoids background pixels that would skew the estimate. A centroid calculation plus the camera’s intrinsic matrix yields an (x, y, z) position in meters. It’s not full 6D pose estimation — you won’t get orientation — but for basic grasping tasks, knowing where an object sits in space is often enough to start reaching.
What’s genuinely interesting here is the timing. The robotics industry has spent years debating whether foundation models would actually translate from demos to deployed manipulation. Pollen is betting they can, now, on consumer hardware, with no cloud dependency. Whether that bet pays off depends on how reliably these models perform outside controlled settings — but releasing it as open source means the robotics community gets to find out fast.
