The document AI playbook just got flipped. For years, the choice was clear: pay for a proprietary API, or wrestle with brittle open-source heuristics that choked on anything more complex than a clean scan. That binary is dead. The blog post from the team details how fine-tuned vision-language models (VLMs) like AllenAI’s OlmOCR, PaddleOCR-VL, and the Docling-powered Chandra are not just matching commercial OCR—they’re offering capabilities that legacy tools never had, with a privacy and cost profile that’s frankly hard to argue against.
What’s actually new here is the model’s relationship with the document, not just reading lines of text. These aren’t your grandfather’s OCR engines that needed a separate layout parser just to figure out reading order. Modern models bake in “grounding” or “anchoring”—essentially, location metadata using bounding boxes—to natively understand multi-column layouts, floating figures, and tables without hallucinating text into the wrong reading order. The output goes far beyond a raw text dump. Depending on the model, you’re getting structured HTML, human-readable Markdown with auto-generated image captions, or XML-like DocTags that preserve exact layout for digital reconstruction.
The real differentiator, though, is how these models treat non-text elements. A chart isn’t just a missed blob anymore. Some models convert a bar chart directly into a Markdown table or a JSON object, effectively doing the data extraction for you. An image can either be left in place with its original coordinates or, more useful for LLM pipelines, replaced with a generated caption. “This is especially useful if you are feeding the machine-readable output into an LLM,” the post notes, highlighting a growing architectural trend where OCR isn’t the final step but a preprocessing module for a more intelligent downstream system.
Choosing a model now becomes a question of your pipeline’s next step. For digital twins of documents, you want a layout-preserving format like DocTags from the Docling models. If you’re stuffing everything into a RAG pipeline for Q&A, Markdown with captions is the native tongue of LLMs. If the destination is a data analysis script, a model that outputs JSON for tables and charts saves you a ton of post-processing code. The guide doesn’t shy away from the fact that fine-tuning might still be on the table for weird, domain-specific forms, but the out-of-the-box capabilities have crossed a threshold where “just download an open model” is now the smart default, not the experimental one.
