Unsloth crushes MoE training with 7.3x speedup, Axolotl hits 1.45x on Qwen3.5

The fine-tuning wars aren’t about which library you use anymore. They’re about where each project places its engineering bets, and for kernel-level speed, Unsloth is running away with it — at least on specific hardware. Their latest benchmarks on an NVIDIA B200 show a 7.3x throughput gap over stock Transformers for a 20B MoE model, clocking 712ms per step versus 5,226ms. That isn’t a typo. The gain scales with sequence length, driven by hand-written Triton kernels and Flex Attention. But the magic fades fast on other architectures; Qwen3-30B-A3B on the same B200 sees that lead collapse to a negligible 1.1x at 16K context.

Axolotl isn’t sitting still. After borrowing Unsloth’s kernel inspiration for their own opt-in LoRA modules in February, they’ve now added SonicMoE LoRA support. That delivers a solid 1.45x speedup and a 30% memory cut for Qwen3.5-35B-A3B on a single H100. It’s a smart move that plays to Axolotl’s core strength: composing parallelism strategies. They don’t need to win the raw kernel race; they just need to make sure their YAML-driven wrapper can slot in the fastest parts and scale across more GPUs than the competition.

TRL remains the baseline everyone measures against, and that’s by design. As the reference implementation layer shipping SFTTrainer, DPOTrainer, and the rest, its job is breadth and correctness, not raw single-GPU throughput. The team has been aggressive about integrating speed levers like Liger Kernel and vLLM sleep mode for GRPO, and a first-party Unsloth integration means you don’t have to choose between them. LLaMA-Factory takes this delegation approach to its logical extreme: zero custom kernels, all config flags. Setting use_unsloth: true reportedly nets 170% relative speed, but you’re entirely dependent on someone else’s optimization work.

Memory is where context length becomes the real differentiator. Unsloth’s gradient checkpointing and Apple’s Cut Cross Entropy let it push Llama 3.1 8B QLoRA to 89,389 tokens on an 80 GB A100. FA2 alone gives up at 6,916. That’s the difference between summarizing a document and summarizing a novel. The MoE story is even more dramatic: Unsloth’s gpt-oss-20b fine-tunes inside 12.8 GB of VRAM. If you’re memory-constrained on consumer hardware, that number matters more than any throughput chart.