NVFP4 vs AWQ vs GPTQ vs GGUF for Production Inference

NVFP4, AWQ, GPTQ, and GGUF are the four weight-quantization formats worth short-listing before a production inference deployment, and the right one is a function of the GPU on the floor and the accuracy budget the workload can absorb. Pick wrong and either throughput collapses or regulated outputs start drifting.

Each format compresses weights differently, runs on a different set of accelerators, and fails in a different way. A sober side-by-side is the document that belongs in front of a CIO and an infrastructure lead before the hardware order is cut.

The Four Formats in One View

GPTQ is a post-training weight-only method that rounds each weight against a calibration set using second-order error, usually down to 4 or 3 bits. The original GPTQ paper reports quantizing a 175 billion parameter GPT model in roughly four GPU hours with negligible accuracy loss at 4 bits, and inference speedups of about 3.25x over FP16 on A100 and 4.5x on A6000.

AWQ, short for Activation-aware Weight Quantization, keeps the top one percent of salient channels at higher effective precision by scaling them before rounding, which recovers much of the instruction-following accuracy GPTQ loses on instruction-tuned models. AWQ was awarded Best Paper at MLSys 2024 and ships from MIT Han Lab under the MIT License, which matters when a security review is parsing the OSS inventory.

GGUF is not a quantization method but the single-file container llama.cpp uses to ship quantized models, with its own block-wise integer schemes running from Q2 through Q8. Q4_K_M is the recommended default. Q8_0 is the near-lossless fallback used when accuracy is non-negotiable. The ladder between them sets disk and VRAM footprint on the host.

NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell. Each value is an E2M1 float (one sign, two exponent, one mantissa bit) in micro-blocks of 16 values sharing an FP8 scale factor, with a per-tensor FP32 scalar on top. The element encoding is in the lineage of the OCP Microscaling MXFP4 specification, though the MX blocks are 32 elements with a power-of-two scale rather than FP8.

GPTQ and AWQ Still Carry the Hopper Fleet

On H100 and A100 inventories, INT4 weight-only quantization is the default. GPTQ is the older of the two and remains the easiest to produce from a calibration dataset of a few hundred samples. AWQ typically holds accuracy better on chat and instruction-tuned models because salient channels survive unscaled, which is visible on MMLU, GSM8K, and the HumanEval regressions most procurement teams ask for. Both load cleanly into vLLM and TensorRT-LLM kernels, and a runtime decision made earlier in the stack rarely forces a quantization change.

The operational catch with GPTQ is calibration drift. A model quantized against English web text will degrade on a radiology corpus or a loan-origination corpus, and the quantization has to be re-run when the fine-tune is refreshed. AWQ is less sensitive but still benefits from domain-matched calibration data held under the same access controls as the training set.

GGUF Lives on CPUs, Macs, and Isolated Boxes

GGUF is where llama.cpp and ollama meet the enterprise, and it is the format of choice when the inference target is not a datacenter GPU. Field laptops, a radiologist's workstation, a Mac Studio in a solo attorney's office, an air-gapped SIPR terminal. These all run GGUF because llama.cpp compiles everywhere and asks for no CUDA. The tradeoff is throughput. A GGUF Q4_K_M model on an H100 is slower than the same model as an AWQ or NVFP4 build on the same card, and the throughput difference widens at batch sizes above one.

For regulated edge work the format earns its place anyway. A single-file weight artifact with a stable hash is easier to attest, sign, and ship through a formal change-management process than a directory tree of tensors and config JSON. For a buyer evaluating an appliance-style deployment, GGUF is often what ends up inside the box regardless of what runs in the central cluster.

NVFP4 Is Blackwell's Native Path

NVFP4 is the format NVIDIA built for Blackwell's second-generation Transformer Engine, and on B100, B200, and GB200 systems it is the format with hardware tensor cores behind it. The smaller 16-element block and the finer FP8 scale trade a little extra scale overhead for better accuracy on the long-tail distributions LLMs produce, compared with MXFP4's 32-element block.

The headline for a long-context workload is the KV cache. Moving KV from FP8 to NVFP4 on Blackwell cuts cache memory by up to half, which effectively doubles the usable context budget per GPU, with less than one percent reported accuracy loss across MMLU-PRO, MBPP, LiveCodeBench, and Ruler 64K. For retrieval-heavy private deployments this is the single largest capacity lever Blackwell offers.

Hardware and Runtime Compatibility

Compatibility is where buyers make expensive mistakes. INT4 formats (GPTQ, AWQ) run on Ampere, Hopper, and Blackwell, but the fastest kernels are tuned for Hopper. NVFP4 requires Blackwell tensor cores to hit its posted numbers; on Hopper it falls back to emulation and loses most of the speed advantage. GGUF runs anywhere with a CPU and an optional CUDA, Metal, or ROCm backend.

Runtime support tracks the same seams. vLLM and TensorRT-LLM support AWQ and GPTQ broadly and NVFP4 on Blackwell builds. llama.cpp carries GGUF and nothing else. SGLang covers the GPU formats. A deployment pinned to a specific NVIDIA generation should resolve the quantization question at the same time as the GPU SKU, not after.

The Defensible Choice for a Regulated Buyer

For a Hopper fleet already on the floor, AWQ is the defensible default. Accuracy holds on instruction-tuned models, kernels are mature in every major runtime, and the permissive license is clean for a security-audit inventory. GPTQ is a reasonable second choice when the quantization has to be reproduced locally from a controlled calibration set and the calibration data has to live in the same enclave as the fine-tune.

For a Blackwell order placed in the last two quarters, NVFP4 is the format that justifies the capital expenditure. The memory reduction and the KV-cache capacity gain are the reasons the hardware was bought in the first place, and leaving the GPU in FP8 is a choice that will come up in an internal audit. For edge, workstation, and air-gapped boxes where no datacenter GPU is present, GGUF is the format that actually runs.

The format decision belongs in the same document as the model selection, the runtime pick, and the GPU plan. A custom deployment that silently changes quantization formats between proof of concept and production is a configuration drift incident waiting for an auditor to find it. Pick the format with the GPU already committed, pin it in the artifact manifest, and re-run the accuracy regression on every model update. That is how the side-by-side in front of the CIO stops being a procurement puzzle and becomes part of the standard build.