Structured output is the unglamorous backbone of production AI. If a model can’t return valid JSON or YAML in the exact shape your pipeline expects, it doesn’t matter how clever its reasoning is — it’s useless downstream. Liquid AI’s new IFStruct benchmark finally isolates schema compliance as its own metric, and the baseline numbers for their LFM2.5-350M model are worth a close look.
Running the benchmark locally on a MacBook Pro with an M5 Max chip via llama.cpp, the base model passed 452 out of 2,000 samples — 22.6% overall. That’s marginally above the 21.1% reported in the IFStruct release blog, which the authors attribute to setup differences rather than any meaningful gap. JSON performance lagged behind YAML significantly: 18.0% versus 27.2% pass rates. The most common failure by an enormous margin was missing required fields, accounting for 7,228 errors — nearly ten times the next category, wrong item count at 738. Extraneous fields like ‘notes’, ‘path’, and ‘constraints’ also plagued outputs, suggesting the model defaults to verbose, conversational structures rather than strict schema adherence.
Some entity types fared dramatically worse than others. GPU reviews passed just 6.4% of the time, recipes a dismal 4.3%, and camera reviews 7.2%. Meanwhile, event ticket booking managed 45.8% and real estate listings 37.8%. The spread hints that certain schemas align better with the model’s pretraining distribution, while others demand structural discipline the 350M parameter model simply hasn’t internalized.
The fine-tuning approach here uses GRPO — Group Relative Policy Optimization — in just 100 steps, which is remarkably cheap compared to the multi-epoch supervised fine-tuning runs that dominate most structured-output work. This mirrors a broader shift toward RL-based post-training for smaller models, similar to what DeepSeek demonstrated with R1 but applied to far more modest parameter counts. If a 350M model can approach the schema compliance of models ten times its size after a hundred-step GRPO run, the cost calculus for production structured-output systems changes substantially. The notebook deliberately doesn’t chase the IFStruct benchmark score — it’s a demonstration that task-specific RL fine-tuning can close the gap between small and large models for this narrow but critical capability. The real question is whether those gains generalize beyond the specific schemas seen during training, or if you’re just teaching the model to pass a particular test.
