OpenAI Now Lets You Distill o1 Into GPT-4o Mini — All In-House

OpenAI just stitched together its entire model distillation pipeline into a single, integrated workflow, and it’s a move that eliminates the messy, multi-tool ordeal developers have been dealing with for years. Instead of bouncing between disconnected tools to generate datasets, fine-tune, and evaluate, teams can now do it all on OpenAI’s platform.

The centerpiece is a trio of features that actually talk to each other. Stored Completions lets developers automatically capture input-output pairs from frontier models like o1-preview and GPT-4o by simply setting a ‘store:true’ flag in the API — no latency hit, no extra scripts. Those stored pairs become a dataset you can filter and tag. Then there’s Evals, a beta tool for building custom evaluations to measure model performance on specific tasks, which can pull directly from those Stored Completions or your own uploaded data. Both feed straight into OpenAI’s existing fine-tuning service.

The practical goal is straightforward: make a cheap model like GPT-4o mini behave like a much smarter one on your specific use case. You start by evaluating GPT-4o mini to establish a baseline. You capture GPT-4o’s outputs on real-world tasks using Stored Completions. You fine-tune GPT-4o mini on that data. Then you circle back to Evals to see if the distilled model meets your bar. If not, you iterate.

It’s a recognition that distillation is fundamentally iterative and that forcing developers to wrestle with disconnected infrastructure was adding pointless friction. OpenAI’s pricing on this is also worth a close look: Stored Completions is free, fine-tuning comes with 2M free daily training tokens on GPT-4o mini until October 31, and Evals are free for up to seven runs per week through year-end if you opt in to share your evaluations with OpenAI. That last bit is the fine print that matters — your eval data becomes part of how OpenAI improves future models.