Hugging Face just made it dramatically easier to run models on AWS’s custom AI silicon. The company announced that over 100,000 public models on the Hub can now be deployed to Inferentia2 instances through Amazon SageMaker, a massive expansion from the handful of architectures previously supported. The update adds 14 new model architectures—including BERT variants, RoBERTa, DistilBERT, ViT, and XLM-RoBERTa—across six ML tasks spanning text classification, token classification, question answering, and feature extraction.
For developers who don’t want to fuss with SageMaker configuration, Hugging Face Inference Endpoints now offers Inf2 instances directly. The pricing is concrete: $0.75 per hour for an Inf2-small with 2 cores and 32GB of memory, which handles Llama 3 8B comfortably, and $12 per hour for the Inf2-xlarge with 24 cores and 384GB, sized for Llama 3 70B. Billing is per-second, and autoscaling can drop costs to zero when traffic disappears.
Under the hood, Inference Endpoints runs Text Generation Inference for Neuron, which supports continuous batching and streaming. The practical hook for teams already building on OpenAI’s ecosystem: deployed models are compatible with the OpenAI SDK Messages API, meaning you can point an existing application at a new endpoint without rewriting integration code.
Hugging Face says Diffusion and Embedding models are next on the roadmap, which would open image generation and semantic search workloads to the same Inferentia2 acceleration. That’s a sensible expansion—embedding models are the workhorses of RAG pipelines, and getting them onto cheaper dedicated hardware could meaningfully shift cost calculations for production AI stacks. The year-long collaboration with AWS engineering teams has clearly moved past the experimental phase into something closer to default infrastructure.
