Fine-tuned DistilBERT crushes TF-IDF baseline on IMDb reviews — but confident errors persist

The perennial question in applied NLP isn’t whether transformers beat bag-of-words models, but by how much — and what you’re trading in compute, cost, and interpretability to get there. A new, thorough walkthrough from the Stanford NLP IMDb dataset puts hard numbers behind that trade-off, and some of the findings are genuinely useful for practitioners building production sentiment systems.

The setup is a meticulous audit of the 50,000-review movie dataset, flagging common traps like class ordering and review-length skew before a single model is trained. A TF-IDF plus Logistic Regression pipeline establishes a strong, cheap baseline. That’s then set against a DistilBERT model fine-tuned with LoRA via the PEFT library — a parameter-efficient technique that’s becoming the de facto standard for anyone without a rack of A100s. The DistilBERT model, even with just 2 epochs and a modest 5,000-sample training set, pulls ahead significantly on accuracy, macro-F1, and ROC-AUC.

What’s more interesting than the headline metrics is the post-mortem. The analysis digs into where the transformer still stumbles, surfacing a stubborn class of confident errors — reviews the model miscategorizes with high probability. It also probes how truncation at 256 tokens, a necessary evil for training speed, hurts the model on longer, more nuanced reviews. The occlusion-based saliency maps provide a window into which words the model actually weights, offering a sanity check that a black-box accuracy number never can.

A pseudo-labeling experiment rounds out the piece, using the unlabeled IMDb split to squeeze out additional performance. While semi-supervised learning often gets hyped as a silver bullet, the practical gains here are grounded and realistic. The end result is a saved, merged transformer ready for inference — the kind of artifact you can actually ship. The real takeaway isn’t that transformers win; it’s that you can win with them using a fraction of the data and compute you’d expect, provided you watch for the same failure modes that have plagued text classifiers since the 1990s.