GGUF in Transformers: compact weights reach Python on the Mac

A compact file can now stay compact

In an article published on September 22, 2026, Hugging Face describes efficient GGUF execution in Transformers using ggml computing kernels. The initial target is local use on Apple Silicon. For a team with Python tooling, the change is concrete: a quantized checkpoint can run in a familiar environment without switching to a separate model runtime for each experiment.

A GGUF file stores model tensors together with text-processing settings, such as tokenizer data and, where provided, conversation formatting. Quantization reduces weight precision to save space. In the article’s Qwen3.5-4B example, Q4_K_M occupies 2.74 GB, compared with 8.42 GB for BF16. Q4_K_M mostly uses four-bit weights while retaining higher precision for sensitive tensors.

Download size is not memory consumption

A compatible quantization kernel is essential. It performs operations directly on packed weights. Without it, the loader dequantizes the model into a higher-precision representation and memory use increases. Successfully opening a GGUF file therefore does not establish that the economical path is active. This is the central check for a pilot on a resource-limited machine.

The packed path currently works only through MPS, PyTorch’s computing path for Apple GPUs. The described integration covers dense Qwen3.5 models and variants with experts; selected Qwen3.8 checkpoints also work on that path. The article required Transformers main, compatible kernels and a PyTorch version supported by the published kernel builds. These conditions do not extend to every GGUF file, an ordinary CPU server or every GPU.

Familiar Python with a separate conversation interface

Loading uses from_pretrained with gguf_file, followed by standard generate calls. This helps teams evaluating quantized answers through existing Python scripts or applying custom stopping rules. File size still represents only part of memory demand: conversation state and runtime operation also require memory. Quantization’s quality tradeoff should be assessed on the intended task.

For a conversation interface, transformers serve provides an OpenAI-compatible API. The example runs on localhost with a client such as Jan or Pi; interface compatibility does not imply cloud execution. Models with a suitable chat template offer reasoning on, off and auto, with auto following the template default. These control reasoning behavior, rather than enabling model execution itself.

Why the speed comparison needs careful reading

The authors kept an M2 Max MacBook Pro connected to power. It had 32 GB unified memory and ran macOS 26.6; the software versions were PyTorch 2.12.1 and kernels 0.17.0. llama-bench decoded 128 tokens, excluded prompt processing and averaged three repetitions. Transformers generated 128 tokens from a 12-token prompt, included prompt processing and selected the best of three warmed runs. The methods differ.

Hugging Face describes the results as close to llama.cpp, but the measurement conditions are not identical. The authors still recommend llama.cpp when efficient local execution is the priority. This integration’s value is combining GGUF with Python and PyTorch. Experimental conditions and limitations.

A useful pilot on an existing Mac

Start with a supported architecture and confirm that loading retains packed weights, with no dequantization warning. The attention path, which connects the current token with context, matters separately. If its kernel is unavailable, the model can fall back to sdpa. This fallback differs from dequantization: changing attention does not itself mean expanding all weights.

One representative task is sufficient to begin, such as explaining a small patch and answering a follow-up in the same conversation. Observe answer quality, memory and time until text appears. This tests the intended work. Checkpoint size does not promise full-context memory fit, and a short generation experiment does not establish behavior with long histories or several users.

The initial target is a single interactive conversation. Padding and batching, which align and combine requests, still need work; team workloads require separate evaluation. The specific buyer choice is therefore to use an existing Apple Silicon Mac for a GGUF pilot in Transformers when the Python workflow matters. It provides a substantive check before deciding on a dedicated server.