Qwen3-4B Fine-Tune Jumps 15 Points on Data Science Tasks by Ditching Bloat
Curated by the Inblix editorial team
If you’ve ever opened a massive, over-engineered agent framework and wondered if all that code is actually helping, the team behind the Jupyter Agent project just gave you some ammunition. They took a tiny Qwen3-4B-Thinking model, a notoriously difficult DABStep benchmark, and a ruthless approach to scaffolding, and watched accuracy on easy data science tasks leap from 44.4% to 59.7%. The culprit behind the initial sluggish performance wasn’t the model itself—it was the scaffolding.
The DABStep benchmark, built with Adyen, is a gauntlet of realistic, multi-step data analysis questions. Even the top closed-source model, Claude 4 Sonnet, can’t crack 20% accuracy on its hard tasks. So seeing a 4B parameter model gain over 15 points on the easy split just by gutting its support code is a genuine eyebrow-raiser. The team stripped their agent loop down to roughly 200 lines of Python with zero external dependencies, inspired by the ‘tiny-agents’ philosophy. They ditched the ReACT framework hardcoded in tools like smolagents and added an explicit final_answer tool, which they found directly improved performance.
That baseline jump is just the starting line. The real heavy lifting is in the training pipeline. The team is attacking a fundamental problem: smaller models are cheap and fast, but they get steamrolled by giants on complex agentic tasks. To fix this, they built a massive data refinery. They’re processing Kaggle’s 2TB notebook archive using Datatrove, a tool from their prior BigCode work. The critical and slightly humbling finding? About 90% of those notebooks are near-identical duplicates. Culling that bloat shrunk the usable dataset to a still-hefty 250GB, ensuring the fine-tuned model learns from genuine analytical variety rather than memorizing the same boilerplate 100,000 times.
The goal isn’t just to climb a leaderboard. It’s to prove that a meticulously curated dataset and a purpose-built, minimal agent loop can close the gap between a model you can run on a modest GPU and the frontier giants that cost a fortune to query. If they succeed, the assumption that you need a massive model for reliable data science agents starts to look very expensive indeed.
💡 Key Takeaways
- Stripping agent scaffolding to ~200 lines of code without external dependencies improved Qwen3-4B accuracy on DABStep easy tasks from 44.4% to 59.7%.
- An explicit `final_answer` tool and removing the ReACT framework from the agent loop contributed directly to the performance gain over default smolagents implementations.
- Roughly 90% of the 2TB Kaggle notebook dataset were near-identical duplicates, posing a major skew risk that required large-scale deduplication before fine-tuning.
- The project targets small model performance to challenge the idea that frontier models like Claude 4 Sonnet—which still only scores under 20% on hard tasks—are the only viable option for agentic data science.
Keep reading: See related articles below for more coverage on this topic.
Get smarter about AI
The sharpest AI news, curated daily. Delivered free to your inbox.