Tagged: quantization

35 articles

Latest

Nunchux AI cuts video attention time by 6x on B200 with VC-Attention

Hugging Face squeezed 100x faster transformer inference. Here's the split

Nous Research's Hermes Desktop now sets up local AI models in one click

Hugging Face's Optimum toolkit promises to make slow, costly transformers run fast

Intel and Hugging Face team up to slash Transformer latency to milliseconds

LLM.int8() Cuts BLOOM-176B Memory in Half—No Performance Loss

BLOOM hits under 1ms per token on 8 A100s, but setup is everything

Hugging Face and Intel shrink ViT models 4x with one-click OpenVINO quantization

4-bit GPT-OSS 60B beats its bfloat16 original on 7 of 9 benchmarks

SmoothQuant shrinks 7B-parameter LLMs by 2x with no accuracy loss on CPUs

QLoRA squeezes 65B-parameter finetuning onto a single 48GB GPU

Apple's 6-bit Core ML trick makes Stable Diffusion 10-30% faster

GPT2 runs on encrypted data with FHE, preserving 96% of original accuracy

Hugging Face adds GPTQ: run 176B models in 4-bit with near-zero accuracy loss

Hugging Face: bitsandbytes wins on ease, GPTQ wins on speed

Why your LLM needs 350GB of VRAM just to load — and how to fix it

Your laptop can run a full RAG pipeline offline — here's the exact stack

Liquid AI recovers 97% of accuracy lost to 4-bit quantization

Meta drops 30B agentic model Muse Glimmer under Apache 2.0 — runs on a single GPU

Ollama vs LM Studio vs llama.cpp: Which local AI runtime actually fits your workflow?

Your 'Claude' API call probably ran on a different model—and the law has no idea what to do about that

This 27B reasoning model runs on an iPhone and Apple is testing it

Intel CPUs can now run vision models in 3 steps—no GPU needed

OpenAI drops GPT-OSS: a 120B open-source MoE model that fits on a single 80GB GPU

Intel's AutoRound shrinks LLMs to 2-bit with only 37 minutes of GPU time

Hugging Face and Intel squeeze Llama 3.1 into 4-bit for edge devices with OpenVINO

Llama 8B shrank to 1.58 bits per parameter and still beat a full-size model

ggml packs PyTorch power into 1MB: here's why devs are switching

Quanto FP8 Quantization Cuts SD3 Memory Use by 33% on a Single GPU

SetFit Inference Hits 7.8x Speedup on Intel Xeon With One Quantization Step

Binary quantization cuts embedding memory 32x while keeping 96% of retrieval quality

GaLore Cuts Optimizer Memory 82.5%, Fits 7B Models on an RTX 4090

Phi-2 Runs Locally on Intel Meteor Lake Laptops at 4-Bit Precision

Quanto brings flexible PyTorch quantization to Optimum, no tracing required

Intel Xeon CPUs hit 7x faster StarCoder inference via INT8 quantization