Hugging Face Transformers Adds Efficient Support for llama.cpp GGUF Models

What happened

In a post published on 22 September 2026, Hugging Face engineers Marc Sun, Arthur Zucker and Lysandre announced support for running GGUF models efficiently in the Transformers library. A GGUF checkpoint from the Hub can be loaded with from_pretrained by passing its file name as gguf_file, then used through the normal Transformers APIs.

What we know

• GGUF, developed by the llama.cpp team, packages weights, tokenizer information and an optional chat template in one file and supports several quantization levels. Hugging Face says GGUF models have been downloaded millions of times.
• To get performance close to llama.cpp, Transformers reuses its ggml kernels through the kernels library and reduces overhead in generate. When weights stay packed on Metal, the compatible ggml/Metal kernels and the ggml-org/ggml-attn attention implementation load automatically, with a fallback to sdpa and a warning if the kernel cannot be fetched.
• The initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture.
• Requirements at the time of the post: an Apple Silicon Mac, a PyTorch version supported by the published kernel builds, the main branch of Transformers until the next release, and a compatible kernels version.
• The post's size table for Unsloth's Qwen3.5-4B lists 8.42 GB for BF16, 3.53 GB for Q6_K, 3.14 GB for Q5_K_M and 2.74 GB for Q4_K_M, and suggests starting with Q4_K_M.

What is not yet confirmed

No benchmark numbers against llama.cpp itself are given in the sections reviewed, only the stated goal of getting close to it. Support for other architectures and for non-Apple hardware is described as a starting point rather than a schedule.

Why it matters

Local inference has largely happened outside the Transformers ecosystem, in tools built on llama.cpp. Loading the same quantized files through Transformers means one code path for experimentation, evaluation and fine-tuning workflows that already use the library, on the laptops many developers actually have.