Claude taught open models to write GPU kernels, but most flunked the test

Here’s a truth the ‘democratize AI’ crowd doesn’t like to admit: giving a smaller model a cheat sheet written by a genius doesn’t make it a genius. The team behind the new open-source tool upskill just proved that with receipts.

The concept is elegant. Take Claude Opus 4.5, have it struggle through writing a CUDA kernel for a diffusers model, and then capture its hard-won process as a reusable Skill file — basically a markdown playbook. Hand that playbook to a smaller, cheaper model running on a laptop, and in theory, it should punch above its weight. In practice? The results were a mess. Some open models saw their accuracy climb. Others got actively worse with the Skill, burning more tokens to produce inferior code than if you’d just let them rawdog the problem. The blog post includes a plot showing this fragmentation, and it’s the most honest thing you’ll read this week.

The workflow itself is smart. First, you use Claude Code interactively, guiding it through kernel creation and exporting the agent trace. Then upskill generates a Skill file and — critically — test cases derived from that trace. Most agent frameworks stop at skill creation and call it a day. upskill actually runs an eval, comparing model performance with and without the Skill. That’s where the illusions shatter. Claude Opus performed identically both ways, which makes sense — the Skill just encoded what it already knew. The real question was whether the Skill could transfer that capability downward.

Some transfers worked. moonshotai’s Kimi-K2-Thinking saw clear gains in both accuracy and token efficiency. But for Claude Opus 4.5, using its own Skill actually increased token consumption with zero accuracy benefit. The lesson isn’t that skills are useless — it’s that you have to measure. The format is standardized across Codex, Cursor, and OpenCode, so the plumbing works. The pedagogy doesn’t always follow. If you’re serious about cutting costs by swapping SOTA models for smaller ones on domain-specific tasks, upskill eval is the polygraph test that tells you whether the cheaper model actually learned something or is just cosplaying competence.