LLM Quantization: GGUF, AWQ, GPTQ and FP8 Explained

Choose the highest precision that fits your workload and passes your own quality checks. A smaller download helps only if the model still answers correctly and leaves enough memory for the conversation.

Documentation and compatibility claims were verified against the projects' live pages on 20 October 2026. The memory figures are calculations from Qwen2.5-7B-Instruct's documented parameter count, not measured GPU usage or benchmark results.

LLM Quantization Explained: Less Precision, the Same Model Shape

Quantization stores the model's learned numbers with fewer bits. A large language model, or LLM, uses learned numerical parameters called weights to generate text. Reducing their precision makes those numbers cheaper to store and move, while introducing approximation error. Hugging Face's quantization overview describes that memory-versus-accuracy tradeoff.

Think of an architectural plan whose dimensions are recorded to fine precision. Rounding the measurements makes the record more compact, but some dimensions change slightly. The building still has the same rooms. Likewise, ordinary weight quantization does not turn a large model into one with fewer layers or fewer learned parameters; it changes how precisely its weights are represented.

The analogy has a limit: quantization is more deliberate than simply deleting decimal places. Values can be grouped, assigned a scale, and mapped to a smaller set of representable values. The scale helps the runtime recover an approximation when it needs to compute. Important parts can remain at higher precision. IBM's explanation of quantization covers these mappings and the resulting error.

Separate three things before choosing a download:

• Weights: the learned values stored with the model.
• Activations: intermediate values produced while processing your input.
• KV cache: saved attention data for the context already processed, which occupies additional memory as the conversation grows.

A weight-only quantization scheme shrinks the first category. It does not automatically shrink the other two. That distinction is why a model can load successfully and still run out of memory on a long request. Ollama's memory and cache documentation treats context capacity and cache precision separately.

Post-Training Quantization vs Quantization-Aware Training

For a builder downloading a model, post-training quantization is usually the relevant starting point. Post-training quantization, or PTQ, converts an already trained checkpoint. Some methods use calibration data, representative inputs that help choose how to approximate the weights. Quantization-aware training, or QAT, accounts for low-precision effects during training or fine-tuning. IBM describes both approaches.

You do not need to train a model to use a published quantized checkpoint. You do need to know which original model it came from, which recipe produced it, and which runtime can load it. A filename alone is a poor substitute for that information.

GGUF, AWQ, GPTQ and FP8: Which Runtime Uses Which?

These names describe different layers of the system, so they are not interchangeable alternatives. GGUF is a container. AWQ and GPTQ are quantization methods. FP8 is a numerical format. A checkpoint is the saved model artifact that combines weights with the information needed to load them.

Hugging Face, the model-hosting platform and developer of Transformers, documents both the file formats and loading methods. The table combines those docs with the runtimes' own guidance; it is a practical selection map, not a claim that every model architecture works in every listed path.

A .safetensors extension does not settle the precision question either. It identifies a tensor-storage format. Read the checkpoint's quantization configuration and model card to establish how those tensors are represented and which loader is intended. Hugging Face's GGUF documentation distinguishes container metadata from tensor data, while its AWQ guide shows the relevant quantization_config fields.

What Q4_K_M Means

Treat Q4_K_M as a mixed-precision preset, not a promise that every parameter occupies exactly four bits. Q4 identifies the nominal four-bit family, K the K-quant block scheme, and M the medium preset. Selected tensors can use more precision according to the model architecture and the quantizer's rules. llama.cpp's quantization implementation makes those selections explicit.

Even a plain Q4_K block includes bookkeeping: the project's tensor-encoding table lists 4.5 bits per weight once its scale and minimum information are included. That is a block-level figure, not the average for every Q4_K_M model.

For a constrained local machine, Q4_K_M is a sensible candidate to evaluate. If a higher-precision variant fits with your intended context, compare that too. Do not trade away answer quality merely because the smaller file is available.

A Worked Memory Estimate: Qwen2.5-7B-Instruct

Budget for total parameters, then add the memory needed to run them. Qwen2.5-7B-Instruct, Qwen's instruction-tuned model used here as a concrete example, has 7.61 billion total parameters. Its model card separately lists 6.53 billion non-embedding parameters; using that smaller number would omit part of the stored model. Qwen model card.

The idealized calculation is:

Weight bytes ≈ parameter count × bits per parameter ÷ 8

At 16-bit, the calculation is 7,610,000,000 × 16 ÷ 8 = 15,220,000,000 bytes. Repeat it at the other precisions:

These are our calculations from the model card's rounded parameter count, assuming every parameter uses the stated width. Decimal GB divides bytes by 1,000,000,000; GiB divides by 1,073,741,824. The unit difference changes the displayed number, not the amount of memory.

The table deliberately excludes quantization scales, higher-precision tensors, file metadata, cache, activations, temporary work buffers and runtime reservations. A Q4_K_M download will not equal the ideal four-bit row. Nor does the four-bit row establish that the model will run on a hypothetical 4 GB GPU.

For a local coding assistant, a short chat is therefore an incomplete capacity test. The useful test includes the source files, instructions and response length you expect to use. For a server, add simultaneous requests. Ollama explicitly documents the memory effect of parallel requests and context allocation.

Read the KV cache guide for the separate context-memory calculation. Weight quantization and cache quantization are different controls; vLLM also documents KV-cache quantization as its own feature.

Choose the Format From Your Runtime

Choose the execution path before the precision label. Compatibility is a hard requirement. Once that is satisfied, decide by available memory, answer quality and measured latency for your workload.

llama.cpp: Start With a Compatible GGUF

llama.cpp, a local inference engine, is the direct GGUF path when you want control over the artifact and device placement. Its project documentation lists CPU execution, Apple silicon support and mixed CPU/GPU execution. That makes it useful when a model's weights cannot all reside on a discrete GPU. llama.cpp project documentation.

Its wall is that a successful load does not prove useful speed. A split across CPU and GPU changes the execution conditions; you still need to measure your actual prompt and response. Choose the model architecture first, a compatible GGUF second, and a quantization preset third.

Use the site's best local LLMs guide to narrow the base model. Quantization cannot make an unsuitable base model understand a task it handles poorly at higher precision.

Ollama: Import the Quant You Intend to Run

Ollama, a local model runner, can import a prepared GGUF, but importing it does not quantize it. Its current import guide says to prepare the quantized file first. This matters when you expect an import command to turn a higher-precision model into Q4_K_M automatically. Ollama's import documentation.

With Ollama installed and a compatible, already quantized file saved as model.gguf in your current directory, the documented flow becomes:

printf 'FROM ./model.gguf\n' > Modelfile
ollama create quant-check
ollama run quant-check

Run ollama ps in another terminal to inspect the loaded model's CPU/GPU placement. This is a check of what loaded, not a quality benchmark. Ollama FAQ.

Ollama's wall is still the workload's memory requirement. A convenient model command does not remove the context and concurrency budget. The Ollama vs llama.cpp comparison explains which level of artifact and runtime control to choose.

vLLM: Match the Checkpoint, Kernel and GPU

vLLM, an inference and serving engine, needs a quantized checkpoint that matches a supported execution path. Start with the model publisher's supported AWQ, GPTQ or FP8 artifact, then check your GPU against the vLLM hardware matrix. A kernel is the low-level compute implementation that makes a format run efficiently on that hardware.

For new AWQ quantization work, the current docs direct you to LLM Compressor. AutoAWQ is deprecated; AWQ as a method and checkpoint family has not disappeared. For GPTQ in Transformers, the maintained backend is GPTQModel, and compatibility can depend on the checkpoint's settings. vLLM AWQ guidance, Hugging Face GPTQ guidance.

FP8 needs a further distinction. W8A8 means eight-bit weights and activations; W8A16 means eight-bit weights with sixteen-bit activations. vLLM documents native FP8 W8A8 for NVIDIA Ada, Hopper and Blackwell, while Turing and Ampere can use a weight-only W8A16 path through Marlin. Seeing “FP8 supported” therefore does not tell you which computation path you will get. vLLM FP8 documentation.

Do not make GGUF your default vLLM deployment artifact merely because it worked locally. The current docs route GGUF through vllm-gguf-plugin and still describe it as experimental and under-optimized, with possible feature incompatibilities. That is a migration constraint, not proof that GGUF itself is inferior. vLLM GGUF documentation.

Hosted API: Choose Only What the Endpoint Exposes

If a hosted API exposes no precision setting, there is no local quantization choice for you to make. Read the endpoint's contract. If it offers named precision variants, evaluate the selected variant; if it does not, evaluate the service's output and latency without guessing its internal representation.

The same model name is not enough to infer the same quantization, runtime or results across providers. Ask for a documented checkpoint or precision guarantee only when your application needs it. Uploading and serving your own weights is a separate deployment decision that returns you to runtime and hardware compatibility.

What Quantization Trades: Memory, Speed and Quality

Memory savings follow from smaller representations; speed and quality need separate evaluation. Keeping less data in memory can reduce the work of moving weights. But the runtime may also unpack or convert those values before computing with them. Hugging Face's GPTQ documentation explains how a fused kernel can restore low-bit weights for computation without expanding the entire model in global GPU memory.

That explains both the opportunity and the limit. A supported kernel can make compact weights useful. An unsuitable execution path can erase the advantage. Measure time to first token, generation speed and throughput at your intended concurrency rather than treating file size as a speed benchmark.

Quality is equally specific. For a coding assistant, compare whether the proposed change passes the relevant tests. For document extraction, compare required fields and valid structured output. For a multilingual assistant, include the languages you will serve. A response that sounds fluent is not sufficient evidence that the quantization preserved your task's behavior.

The useful comparison is the same original checkpoint at higher precision and at the candidate quantization, using the same prompts, chat template and generation settings. If you compare unrelated models at the same time, you cannot isolate what quantization changed.

When Not to Quantize, and What's Overhyped

Keep higher precision when it already fits and the quantized candidate offers no measured operational benefit. Extra compression creates another artifact and another behavior to validate. It should solve a problem you have.

Do not adopt a lower-precision variant when it fails important task checks, when its runtime path is poorly supported, or when your long-context workload still exceeds memory. In the last case, examine context and concurrency before repeatedly shrinking the weights.

Do not treat an inference quant as a universal training checkpoint, either. Bitsandbytes documents low-bit training support for additional parameters, such as adapter weights. That is a specific fine-tuning workflow, not permission to assume every downloaded quant supports ordinary full-weight training. Hugging Face bitsandbytes documentation.

The oversold claims are straightforward:

• “Four-bit means the whole application uses a quarter of the memory.” It describes an ideal weight-width ratio against sixteen-bit. Runtime memory has other components.
• “Eight-bit is one format.” FP8, INT8 and Q8_0 are different representations and execution paths.
• “Lower bits always run faster.” The checkpoint, kernel, hardware and workload decide whether a memory reduction becomes a latency or throughput improvement.
• “A larger quantized model always beats a smaller precise model.” Parameter count and bit width alone cannot establish task quality. Evaluate both candidates if both fit.

For coding-specific candidates, use the best local AI models for coding guide, then compare the chosen checkpoint's quantization variants on the work you expect it to do.

LLM Model Quantization: Your Next Build Step

Start with one supported artifact and a small, repeatable evaluation. Keep the original model identity fixed until you understand the effect of precision.

Frequently Asked Questions

For practical guides to running and shipping AI systems, join the newsletter.