4.5 FP8 and NVFP4/MXFP4

Earlier sections used integer encodings, and the last one brought in FP8 KV Cache. INT8 and FP8 both take 8 bits; the difference is how those bits are read: integer formats are like a ruler with even ticks, floating-point formats are more like a slide rule with a log scale, denser near 0 and sparser farther out. This section starts with what changes when you move from INT to FP, then covers FP8 as it spread from Hopper onward, and NVFP4 / MXFP4 on Blackwell.

⏰ Currency: Format definitions follow the OCP MX spec, the FP8 Formats paper, and NVIDIA’s docs; vLLM hardware support and online quantization schemes are based on the v0.32.0 docs (released 2026-10-18).

📑 Contents

• 1. FP8: how range and precision are split
• 2. Why FP8 still needs Scale
• 3. FP4 and micro-block scaling
• 4. NVFP4 vs MXFP4
• 5. How hardware support turns into model gains
• Summary
• Self-check list
• References

---

1. FP8: how range and precision are split

1.1 Same bit string, two meanings

In uniform INT8 quantization, the integer $q$ is read as $s(q-z)$; once Scale is fixed, neighboring reconstructed values are evenly spaced. FP8 splits the 8 bits into sign, exponent, and mantissa. Take 00111000:

• Read as signed INT8, it is 56;
• Read as E4M3, sign 0, exponent 0111, mantissa 000, it is 1.0.

So if you only change the type tag on an INT8 encoding to FP8, every value changes. Converting between formats means re-encoding from the original values and recomputing Scale. Full E4M3 encodings and special values are in ONNX’s FP8 notes.

Bit width stays the same; what changes is how representable values are laid out, how wide the range is, and which compute path you take. Which error is smaller depends on the data distribution, Scale, and granularity.

1.2 E4M3 and E5M2

How you split 8 bits between exponent and mantissa is the range-vs-precision tradeoff. The FP8 Formats paper proposed two encodings that NVIDIA and other hardware later adopted widely:

E5M2 follows IEEE 754 special-value rules; E4M3 does not represent infinity and keeps only one mantissa pattern for NaN, which extends the range to 448. Inference weights and activations usually use E4M3, and vLLM’s FP8 W8A8 is mainly E4M3 as well; the paper’s suggested split is E4M3 for weights and activations, E5M2 for gradients during training.

Take E4M3 normals as an example: inside $[1,2)$ neighboring values are $1/8=0.125$ apart; inside $[8,16)$ that becomes 1. What it preserves is relative precision, unlike INT8’s fixed step. Near 0 there are separate subnormal rules.

The figure below draws representable values from 0 to 16 to scale: blue ticks are INT8 with $s=0.125$, green ticks are E4M3 with Scale 1, and the yellow band marks $[1,2)$ where both share the same spacing:

When you compare formats, state the Scale. With external Scale 1, E4M3 can represent 1.375 exactly, while INT8 has to round to the nearest integer; but if you pick $s=0.125$ for INT8, it can also represent 1.375 exactly, at the cost of a max value of $127\times0.125=15.875$. One number’s error does not tell you which format is better overall.

📌 Key point: Dividing by Scale and then round() lands on an integer grid; even if you then clip to $[-448,448]$, that is not E4M3. To simulate FP8 error you need a real type cast (for example PyTorch’s torch.float8_e4m3fn).

---

2. Why FP8 still needs Scale

FP8’s exponent range is still limited. A good external Scale cuts how often values fall into the subnormal region, underflow to 0, or overflow the max. Inside the normal range, changing Scale by powers of 2 alone barely improves relative precision; Scale’s main job is to land the distribution inside the representable range.

A common form:

$$q=\operatorname{cast}_{\text{FP8}}(x/s),\qquad \hat{x}=s q$$

When scaling by the max, take $s=\max|x|/F_{\max}$, with $F_{\max}=448$ for E4M3, and handle all-zero tensors separately. Here $s$ is the Scale you multiply by on dequantization; some libraries store the reciprocal, so check the direction when you read the code.

As with integers, FP8 Scale can be set per tensor, channel, Token, or Block. Online FP8 quantization in vLLM v0.32.0 splits into several schemes by granularity (fp8_per_channel appears as a --quantization scheme name in the source, but is missing from the docs’ scheme table):

On hardware, the vLLM v0.32.0 docs note: FP8 compute needs compute capability 8.9 or higher (Ada, Hopper, Blackwell); Turing and Ampere can still run FP8 models, but through FP8 Marlin’s W8A16 path, so only the weights are FP8 and the multiply still uses 16-bit. The docs say FP8 can cut model memory in half and raise throughput by up to 1.6×.

---

3. FP4 and micro-block scaling

3.1 E2M1 has only 8 magnitudes

At 4-bit, E2M1 has 1 sign bit, 2 exponent bits, and 1 mantissa bit, so the non-negative magnitudes are only:

$$0,\ 0.5,\ 1,\ 1.5,\ 2,\ 3,\ 4,\ 6$$

0.5 is a subnormal; the max is 6 rather than 8 because with only 1 mantissa bit the largest normal is $2^2\times1.5$. Full encodings are in ONNX’s FP4 notes.

With only 8 magnitudes, if a few million values in a layer share one Scale, local variation is almost all lost. That leads to Micro-block Scaling: each small group of elements shares one Scale and fits its own range. Think of one group photo with a single exposure versus many small photos you can brighten separately.

3.2 Spell out the shape when you talk about granularity

Micro-block scaling is the same idea as Per-group in section 4.1, but Per-group and Block-wise naming is not consistent: a Block may be 1-D or 2-D. Take a PyTorch layout $W_{\text{torch}}\in\mathbb{R}^{256\times512}$ (one output channel per row):

A 1-D $1\times128$ Block is the same grouping as Per-group above; a 2-D $128\times128$ shares Scale across output channels, which is what the previous section’s fp8_per_block weights use. When you compare granularity, write down the tensor layout, the grouping axis, and the group shape. GPTQ’s update blocks and GEMM compute tiles are about execution; they do not decide the Scale sharing range.

3.3 Scale itself has a format

For FP4, the format spec sets the block size and the Scale type. The figure below draws the same 32 elements under both formats: blue cells are 4-bit E2M1 elements, yellow cells are 8-bit block Scales, and green cells are NVFP4’s extra FP32 tensor-level Scale:

Both formats use E2M1 data elements; they differ in how many elements per block and which format stores the Scale.

Example: the two groups $[0.5,1,2,6]$ and $[5,10,20,60]$, with Scales 1 and 10, can both be represented exactly by the same E2M1 values. But the Scale itself must be encoded: MXFP4’s E8M0 can only store powers of 2, so it cannot store 10; NVFP4’s E4M3 can, since $10=1.25\times2^3$.

NVIDIA’s NVFP4 write-up gives an example: for the same data, Scale encoding MSE is 0.72 for E8M0 and only 0.08 for E4M3. The closer the Scale sits to the true range, the more of the block’s E2M1 ticks get used.

---

4. NVFP4 vs MXFP4

4.1 Two-level Scale vs one-level Scale

NVFP4 dequantization is a two-level multiply:

$$\hat{x}=s_{\text{tensor}}\cdot s_{\text{block}}\cdot q$$

E4M3 block Scale tops out at 448 and cannot cover a whole tensor’s dynamic range, so an FP32 tensor-level Scale first pulls the overall range into place, then the block Scale does local fine-tuning. MXFP4’s E8M0 has an 8-bit exponent and enough range that it does not need a second level; the cost is that Scale can only be a power of 2, so the block max often underuses the E2M1 ticks.

NVFP4 trades smaller blocks and a finer Scale for precision, paying 0.25 bit/element of metadata. NVIDIA’s notes report that after DeepSeek-R1-0528 moved from FP8 to NVFP4, key eval drops stayed within 1%; model memory was about 1/3.5 of FP16 and 1/1.8 of FP8. Those are vendor results on specific models; you still need to evaluate on your own model.

4.2 How to use them in vLLM v0.32.0

Online quantization converts from a BF16/FP16 checkpoint at load time with no calibration data, which is good for a quick try; NVFP4 is usually exported offline with NVIDIA Model Optimizer first, then loaded in vLLM. You can pin the linear-layer Kernel with --linear-backend, values including cutlass, flashinfer_cutlass, flashinfer_trtllm, marlin, and others; it replaces the old VLLM_NVFP4_GEMM_BACKEND environment variable.

⚠️ Note: Online FP8 scheme names are fp8_per_tensor, fp8_per_channel, or fp8_per_block. Writing --quantization fp8 on an unquantized checkpoint errors out immediately, which differs from older versions.

---

5. How hardware support turns into model gains

“This card supports FP4” and “this model runs faster with FP4” are several steps apart. To get a compute win you need all three at once:

1. Tensor Core has the matching instruction: on NVIDIA GPUs, FP8 starts with Ada and Hopper; native NVFP4 / MXFP4 matrix multiply needs Blackwell’s fifth-generation Tensor Core.
2. The framework picked the matching Kernel: vLLM selects by platform; without a native Kernel it falls back to Weight-only paths like Marlin, with a warning in the logs.
3. Activations are quantized too: when only weights are low-bit (W4A16, W8A16), the multiply still runs at 16-bit, so you save bandwidth rather than compute.

Even when all three hold, only the linear layers get faster. Under long context, Attention and KV reads can dominate; those need KV quantization from section 4.4 and the Attention backends from chapter 2.

💡 Tip: To confirm the real path, look for Selected <kernel> for <module> or fallback warnings in the startup logs; that is more reliable than the model name.

---

📝 Summary

• INT and FP use the same bit width; they differ in how representable values are laid out: integers are evenly spaced, floats keep relative precision inside the normal range.
• E4M3 maxes at ±448 with higher precision and is used for inference weights and activations; E5M2 maxes at ±57344 with a wider range, and the paper suggests it for gradients.
• FP8 still needs an external Scale so data lands in the representable range; online FP8 schemes in vLLM v0.32.0 are fp8_per_tensor, fp8_per_channel, and fp8_per_block.
• E2M1 has only 8 magnitudes, so it needs micro-block scaling; when comparing granularity, spell out layout, grouping axis, and group shape.
• MXFP4 is 32 elements + E8M0 Scale; NVFP4 is 16 elements + E4M3 Scale + FP32 tensor Scale, with a finer Scale.
• Hardware gains need instruction support, a selected Kernel, and quantized activations all at once; otherwise you fall back to a Weight-only path.

🎯 Self-check list

• Can explain why the same 8-bit string yields different numbers as INT8 vs E4M3
• Can write the bit split, max value, and special-value differences for E4M3 and E5M2
• Can say why FP8 still needs Scale, and compute $s$ by max scaling
• Can list E2M1’s 8 non-negative magnitudes and explain why the max is 6
• Can compute Scale counts for a given tensor under Per-group and 2-D Block granularity
• Can compare MXFP4 and NVFP4 on block size, Scale format, and average bit width
• Can state the hardware bar for FP8 compute in vLLM v0.32.0, and which path Ampere takes
• Can tell whether an NVFP4 model on a given GPU is really using FP4 compute

📚 References

• FP8 Formats for Deep Learning (Micikevicius et al., 2022)
• OCP Microscaling Formats (MX) Specification v1.0
• NVIDIA — Introducing NVFP4 for Efficient and Accurate Low-Precision Inference
• ONNX — Float stored in 8 bits
• ONNX — Float stored in 4 bits
• vLLM v0.32.0 — FP8 W8A8
• vLLM v0.32.0 — Online Quantization
• vLLM v0.32.0 — NVIDIA Model Optimizer