Hugging Face ends its Gaudi fork, baking Intel AI chip support directly into TGI

Hugging Face just made life a lot easier for anyone deploying large language models on Intel’s Gaudi hardware. The team has eliminated its separate TGI-Gaudi fork and merged support directly into the main Text Generation Inference (TGI) codebase with PR #3091. No more juggling a custom repository to access the latest features.

This integration is powered by TGI’s new multi-backend architecture and covers the full range of Intel’s AI accelerators, from the first-gen Gaudi on older AWS DL1 instances all the way up to the latest Gaudi3 chips available through IBM Cloud and OEMs like Dell and HP. It’s a significant step toward hardware diversity in a market utterly dominated by Nvidia’s CUDA ecosystem. The promise here is clear: a production-ready stack that brings TGI’s robustness—think dynamic batching and streaming—to a hardware platform known for compelling price-performance on specific inference workloads.

Under the hood, Hugging Face has specifically optimized modeling code for a healthy list of popular models to squeeze maximum speed out of Gaudi’s architecture. The list includes Llama 3.1 (8B and 70B), Mixtral, the vision-language Llama 3.2, and Falcon 180B. The team isn’t stopping there, teasing upcoming support for the massively popular DeepSeek-R1 and Qwen-VL models. Advanced features like FP8 quantization via Intel Neural Compressor are also on the table, which should unlock another tier of performance for memory-bound inference tasks.

Getting started is as simple as pulling a new official Docker image (ghcr.io/huggingface/text-generation-inference:3.2.1-gaudi) and pointing it at a model ID. The fact that the --runtime=habana flag is the key differentiator in the Docker run command signals a maturing software stack that’s genuinely trying to disappear into the background. While Gaudi is still a niche player compared to Nvidia, removing the friction of a separate code fork lowers the barrier to experimentation considerably—and in a market hungry for GPU alternatives, convenience might just be the killer feature.