olmOCR fine-tune fixes the header-footer blind spot that killed it for invoices

Pipeline-based OCR engines have a dirty secret: they often butcher the reading order, especially on layout-rich pages. Vision Language Models were supposed to fix this, but when we put the open-source olmOCR-7B-0225-preview to work on real invoices, it developed a convenient form of amnesia. Headers and footers—where dates, invoice numbers, and totals often live—consistently vanished. This wasn’t a bug, exactly. The Allen Institute for AI intentionally trained olmOCR to ignore that “extraneous” information for a smoother reading flow during next-token prediction. For a business parsing invoices, though, that’s a dealbreaker.

So we built a fix. Using Qwen2.5-VL-72B-Instruct, we generated a custom dataset of 8,000 documents where every scrap of text, including the previously ignored headers and footers, was faithfully captured. We fine-tuned olmOCR on this new data using its open-source training pipeline on an 8xH100 Nvidia node, running for 2.5 epochs with 4 gradient accumulation steps. The default hyperparameters held up, sparing us an expensive search. The result is a model that now extracts the complete text, from the very top of the page to the very bottom.

Qualitative tests show the difference clearly. The original model would return a clean but incomplete readout, dropping a “Page 1 of 2” or a critical vendor address. Our fine-tuned version pulls it all in, even while still handling simple tables. We did notice one wrinkle: output quality becomes more sensitive to the temperature setting during inference, so you’ll need to dial that in. But the core problem is solved.

The move from pipeline systems to end-to-end VLMs like olmOCR is a genuine leap for linearizing complex documents. But a model’s training priorities define its usefulness. What the Allen Institute saw as noise, a finance department sees as the only data that matters. We’re open-sourcing this fine-tuned model on Hugging Face for anyone else who needs their OCR to have a perfect memory, not just a good one.