While debugging a handwritten-date defect on shop-floor production cards, we did the thing everyone does first: swapped the model. Three open-weight model families, from two different vendors, spanning 3B to 14B parameters, plus four prompt variants and a reasoning mode.
They all got the same cards wrong, in the same direction.
If model failures were random, diversity would have helped — some models wrong here, others there, an ensemble cleaning up. Instead the errors were correlated. That correlation is the interesting finding, because it points at why vision-language models fail at reading in the first place — and why the failure usually isn't where people look for it.
The models are structurally near-twins
Under the hood, most current VLMs are the same machine: a vision transformer encodes the image into patch tokens, a spatial merge cuts the token count, a small projector maps them into the language model's embedding space, and a decoder — an ordinary LLM — writes the answer.
The lineages are shallow, too. While preparing a training run we worked with one publicly released OCR-specialist model that is literally the vision tower of one lab's general-purpose assistant bolted onto another lab's decoder — our own training logs printed the framework warning that it was instantiating one model type from the other's configuration. Its behaviour still differed sharply from its architectural twin, and the difference was entirely the training objective: one was trained to transcribe pages, the other to follow instructions about images. Architecture is converging; what a model reads like is decided by what it was trained to do.
So when you swap one VLM for another, you are mostly swapping decoders trained on overlapping data with overlapping objectives, behind encoders of shared ancestry. Correlated errors are what that setup should produce — and it's what we measured.
The failure isn't perception
The defect that started this: a writer draws date separators as tall vertical strokes, and a handwritten 1 is the same stroke. 3/8/26 presents as six marks — 3 1 8 1 2 6 — and models return 31/8/26, or 3/18/26, or 18/26 with the year consumed.
The obvious diagnosis is "the encoder can't resolve the strokes." It's wrong, and a one-line probe shows it: ask the model to simply enumerate the marks on the date line, and it answers 3, 1, 8, 1, 2, 6. Every mark, in order. The strokes are seen.
The failure happens after seeing: deciding which marks are digits and which are punctuation is a grouping decision, and grouping happens on the language side. Every model family we tried grouped the same way — fusing the separator into the day — because grouping marks into numbers is exactly the kind of learned prior their decoders share.
Why this failure is expensive
A fused separator almost always makes the day larger, so the wrong date is frequently in the future — and 31 is a valid day, so range checks ratify it. The most dangerous OCR error is not the garbled one your validation catches; it's the plausible one it waves through.
Priors beat pixels
The second failure mode came from a fine-tune, and it's the cleanest demonstration we have that a VLM's decoder is a language model first and a reader second.
We fine-tuned a 3B model on a few thousand labelled cards. On a held-out gold set it scored 92% on the date field. On the real out-of-distribution bench — 204 operator-confirmed cards from the following month — it scored 51.5%, fourteen points below the production baseline it was meant to replace.
The cause was one number in the training data: 95.5% of training cards were from June. On August's cards, the fine-tune predicted June 65 times. Days often right, month wrong — it had learned "the month is June" as a prior, and the prior outvoted the pixels.
Base models do the same thing in softer ways. Let one answer free-form instead of constrained JSON and it drifted to "18th of June 2026" garnished with a knowledge-cutoff disclaimer. When evidence is thin, a decoder emits what is likely. Likely is trained on the past; the document in front of it is not.
Reading is computation, and small decoders can't do it silently
The third failure mode is the one that ultimately fixed the defect, so it earns the numbers.
Given explicit right-to-left grouping rules — the year is the last two digits, each separator's position is forced rather than recognised — a 3B model asked for JSON-only output still answered wrong. The same rules, with the model required to write out its working (3 1 8 1 2 6 → year 26, month 8, day 3), answered right.
date accuracy, base model + a prompt that writes its working — 204 held-out cards
the same bench, production build at the time
Forbidding the working suppressed the procedure the prompt was built around. A multi-step parse is computation, and a small decoder cannot reliably execute it latently — it needs the tokens as scratch space. This is the mundane, unglamorous reason "just prompt it better" sometimes beats fine-tuning by double digits: the prompt wasn't adding knowledge, it was granting the model somewhere to compute.
Two related observations from the same runs: switching on the vendor's reasoning mode made everything 4× slower and fixed nothing, and shrinking images to save latency was a false lever — in a small probe, going from 1024px to 384px collapsed accuracy while buying about one second, because generation, not image encoding, dominates request time.
Why "try another model" keeps failing
Put the three failure modes together and the correlated-errors observation stops being surprising:
- The encoders see fine, and see roughly alike — perception is not the bottleneck.
- The decoders share training objectives and data lineage, so they share grouping priors — and grouping is where reading errors actually happen.
- The decoders share the same architectural limit on latent multi-step computation, so they fail the same parses.
A voting ensemble cannot fix what every voter gets wrong the same way. Swapping vendors buys you a different logo on the same mistake.
What actually moves the number
Everything above is measured on one hard field class — handwritten dates from one plant's production cards, 204 confirmed cards, which puts the 95% interval around ±6 points. Within that scope, what worked was never a bigger model:
- Treat it as structured extraction, not transcription. The hard part of a production card is not reading
70; it's knowing this70is Total Cakes and not Jet Change. We score per-field exact match, never character error rate — a 2% CER answer that swaps two fields is a 100% wrong record. - Give the model room to compute. Constrained output, working shown, parse anchored from the stable end of the field.
- Benchmark out of distribution, on human-confirmed truth. Our ground truth is operator-confirmed cards from the review workflow, and the bench month deliberately differs from the training months. A gold set that shares the training distribution will happily certify a model that fails in production.
- Read the error shapes, not just the score. Impossible future dates are an operator interruption each; two models three points apart can differ tenfold in how many they produce.
The uncomfortable summary: vision-language models fail at OCR for language-model reasons. The encoder already saw your document. What the decoder does with it is a matter of priors, objectives, and scratch space — and those are things you engineer around, not shop around.