The dirty secret of many open-source language models is they learned to read from garbage. When EleutherAI’s Talkie project quantified the damage, the number was stark: training on old OCR text slashes learning efficiency to just 30 percent of what you get from clean human transcriptions. That’s millions of public-domain books effectively going to waste.
Hugging Face and EleutherAI just dropped FineBooks, a benchmark that tested 14 open-weight OCR models on 2,165 historical book pages to find out which ones can actually fix this mess. The results flip the usual “bigger is better” assumption on its head. The top performer, a model called dots.mocr with just 3 billion parameters, hit over 97 percent character accuracy. Meanwhile, Qwen3.5-9B — nearly three times the size — scored worse. The runner-up, OvisOCR2, manages a respectable showing with only 0.9 billion parameters and costs a mere 46 cents per thousand pages. Model size and OCR quality simply don’t correlate for these old typefaces.
The economics make this instantly practical. Reprocessing a collection the size of the Biodiversity Heritage Library — that’s 64 million pages — is feasible at under two dollars per thousand pages. The team plans to do exactly that for about 200,000 BHL documents and release the cleaned text as an open dataset. That’s the kind of unglamorous data plumbing that genuinely moves the needle for open-source AI, far more than chasing benchmark scores.
But don’t hand these models to a historian just yet. They have a subtle, maddening habit of silently modernizing archaic characters — swapping a long “s” for the modern version, merging ligatures into plain letters. For training a language model, who cares? The meaning is preserved. For scholarly transcription, it’s a nonstarter. The models also output Markdown or plain text without word-level coordinates, which means they can’t plug into existing library systems that rely on the ALTO XML format. Fine-tuning could patch the modernization issue, but the infrastructure gap is a harder problem. What works for AI training and what works for digital humanities remain two very different jobs.
