Meta’s Llama 3 is live on Hugging Face, and the numbers tell the story: over 15 trillion training tokens — roughly 8x more data than Llama 2 — pushed through 24,000 GPUs across two clusters. The release includes four models split across two sizes, 8B and 70B parameters, each with base and instruction-tuned variants. A fifth model, Llama Guard 2, arrives as a safety classifier fine-tuned on the 8B architecture. Hugging Face has baked in full ecosystem support from day one: Transformers integration, Hugging Chat access to the 70B instruct version, and deployment paths through Inference Endpoints, Google Cloud, and Amazon SageMaker.
The most consequential architectural shift is the tokenizer. Vocabulary exploded from 32K tokens in Llama 2 to 128,256 in Llama 3 — a 4x jump that Meta says improves encoding efficiency and could strengthen multilingual performance. That expansion isn’t free. The embedding input and output matrices grew substantially, which is why the small model jumped from 7B to 8B parameters even before adding other improvements. The 8B version also picks up Grouped-Query Attention (GQA), an efficiency trick that should make longer contexts more tractable. Context length sits at 8K tokens across all variants.
Meta trained the instruct versions on over 10 million human-annotated samples, combining supervised fine-tuning with rejection sampling, PPO, and DPO. That’s a serious alignment pipeline — the kind of investment you typically see from closed-model labs, now applied to open weights. The licensing terms remain permissive, allowing redistribution, fine-tuning, and derivative works. One new wrinkle: attribution is now mandatory. Derived models must include “Llama 3” at the start of their name, and services built on it need to state “Built with Meta Llama 3.” It’s a small ask, but it signals Meta wants brand recognition as these models proliferate.
What’s notably absent is any detail on the training data mix. Meta says it used “publicly available online data,” but that’s about as specific as the release gets. Given how much of Llama 3’s performance likely stems from careful curation rather than raw scale, that opacity matters. The Open LLM Leaderboard is already filling with scores, though Hugging Face cautions that the leaderboard skews toward evaluating base models rather than conversational fine-tunes. For developers who’ve been waiting to see whether open models could close the gap with GPT-4-class systems, this is the most serious evidence yet — and the fine-tuning example on a single consumer GPU with TRL means the barrier to entry just dropped again.
