LayoutLM hits 95% accuracy on document AI benchmarks, leaving OCR-only approaches behind

Document AI is quietly becoming one of the most practical corners of machine learning, and the open-source models available now are dramatically better than what enterprises were stuck with just a few years ago. The Hugging Face team has laid out a clear taxonomy of the field: OCR, document classification, layout analysis, document parsing, table question answering, and visual question answering. Each has its own benchmarks, its own state-of-the-art models, and its own quirks.

The headline number here is hard to ignore. On the RVL-CDIP document classification benchmark, a plain BERT-base model scraping text alone manages 89% accuracy. That sounds decent until you see what multimodal models do with the same task. LayoutLMv3 and Donut—both of which fuse text with visual structure—hit 95%. That six-point gap is the difference between a system that mostly works and one you can actually deploy in a production accounts payable pipeline without a human babysitter.

The LayoutLM family deserves particular attention. When the first version dropped in 2020, it didn’t just nudge the field forward; it jumped it. On the FUNSD form understanding benchmark, BERT scrapes by with an F1 score around 60%. LayoutLM gets to 90%. It remains one of the most downloaded models on the Hub for document tasks, and the v2 and v3 iterations have only widened the lead by incorporating visual features during pre-training.

For practitioners, the real takeaway is that document AI is no longer a bolt-on OCR problem. The old pipeline—run Tesseract, feed text to a classifier, hope for the best—is obsolete for anything beyond basic digitization. Modern approaches treat the document’s layout as signal, not noise. That means table question answering and key-value extraction are finally viable at enterprise scale, which is where the actual money is. The question now is whether the open-source stack can keep pace with the proprietary document models Google and Microsoft have been quietly shipping in their cloud suites. Given the trajectory, betting against the Hub seems unwise.