Florence-2 Fine-Tuned on DocVQA Jumps from 0 to 57% Similarity

Florence-2 was supposed to handle visual question answering out of the box. That’s what Microsoft’s paper implied. But the released checkpoints tell a different story — the VQA capability simply isn’t there. When researchers tried unsupported prompts like "" or ”,” the outputs were unusable. The similarity score on DocVQA’s validation set sat at exactly zero.

So they fine-tuned it. Seven epochs on the DocVQA training set later, that Levenshtein similarity score climbed to 57.0. That’s not a marginal tweak — it’s the difference between a model that fails completely and one that actually completes the task. The authors note this happened with a frozen vision encoder, a batch size of just 6 on a single A100 in Colab, and a learning rate of 1e-6. Anything larger and the model overfit the training set quickly.

The architecture itself isn’t the secret sauce. Florence-2 uses a DaViT vision encoder, BERT for text prompts, and a standard encoder-decoder transformer. What sets it apart is the pre-training data: FLD-5B, a dataset with over 5 billion annotations across 126 million images, built largely through automated pipelines using off-the-shelf models and heuristics. The original training ran with batch sizes of 2048 for the base model and 3072 for the large one.

There’s a broader lesson here about the gap between what a paper claims and what a released model can actually do. Florence-2’s authors said VQA was supported. The community found otherwise. But the fine-tuning results suggest the model has real headroom for downstream tasks, even under constrained resources. A second experiment with the full model unfrozen on 8 H100 GPUs finished in 70 minutes — a reminder that you don’t always need industrial-scale compute to get useful results, but it certainly doesn’t hurt.