Llama 3.1 lands: 405B parameters, 128K context, and a license that lets you distill

Meta just dropped Llama 3.1, and the headline number is absurd: 405 billion parameters. That’s the largest dense open-weights model anyone has shipped, and it comes with a 128K token context window — a 16x jump from Llama 3’s 8K. The release spans six models across three sizes (8B, 70B, 405B), each with base and instruct-tuned variants. All of them handle eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

The training numbers tell the story of scale here. Llama 3.1 consumed over 15 trillion tokens across a custom GPU cluster, racking up 39.3 million GPU hours — 30.84 million of those went to the 405B model alone. Meta also threw in 25 million synthetically generated examples during supervised fine-tuning and RLHF, using LLM-based classifiers to filter high-quality prompts. That’s a significant evolution in how the instruct models were built.

But the real shift might be legal, not technical. The Llama 3.1 license now explicitly allows using model outputs to improve other LLMs. Translation: synthetic data generation and distillation from the 405B model are officially permitted, even for competing models. That’s a direct challenge to closed-model providers. The catch: derivative models still need “Llama” in their name, and anything built on it must say “Built with Llama.”

Meta also shipped two safety tools. Llama Guard 3, fine-tuned on Llama 3.1 8B, handles input and output classification across a risk taxonomy with the same 128K context and multilingual support. Prompt Guard is a tiny 279M-parameter BERT classifier aimed at catching prompt injections and jailbreaks. The instruct models also gained built-in tool-calling capabilities, with search and Wolfram Alpha integration plus support for custom JSON functions — positioning Llama 3.1 squarely for agentic workloads. Whether the 405B model’s inference costs make it practical for most teams is another question, but the licensing change alone could reshape how open-source models are trained going forward.