Researchers have successfully trained an 8-billion-parameter language model using ultra-compressed 4-bit floating-point (FP4) math without relying on the computationally expensive workarounds previously required by industry standards. By introducing a novel “block scaling” technique paired with an unsigned 8-bit format (UE5M3), the team achieved a 21.2% increase in model-body token throughput while simultaneously lowering training loss compared to NVIDIA’s current FP4 recipe.
The Core Discovery
For the past year, the artificial intelligence hardware industry has been racing toward 4-bit quantization—a method of compressing neural network weights and activations to a quarter of the size of standard 16-bit formats. The theoretical payoff is massive: quadrupled memory capacity and drastically accelerated compute times. However, training models natively in 4-bit floating-point (FP4) is notoriously unstable.
To prevent models from collapsing during training, the current state-of-the-art methodology—NVIDIA’s Transformer Engine recipe—relies on a series of mathematical crutches. These include a Randomized Hadamard Transform (RHT) to smooth out data spikes, and keeping the final layers of the neural network in higher 16-bit precision (bfloat16).
A team of researchers has now proven that these crutches are unnecessary. By fundamentally redesigning how numerical scaling is handled at the micro-architectural level, they successfully pretrained a Nemotron-H 8B model on nearly 190 billion tokens using a pure, simplified FP4 pipeline. Their proposed “UE5M3 block scaling” recipe strips out the RHT overhead, applies FP4 to all eligible internal linear layers, and yields a smarter, faster, and more accurate training paradigm.
The Pre-Existing Bottleneck
To understand the bottleneck, one must look at the anatomy of an FP4 number. The standard FP4 payload used in these operations is known as E2M1—meaning it allocates two bits for the exponent and one bit for the mantissa (the precision of the number), plus a sign bit. This format can only represent a highly restricted, narrow range of magnitudes.
Imagine trying to paint a highly detailed, photorealistic landscape using a palette of only 16 colors. If you encounter a bright flash of light (an outlier in neural network activations) or a subtle shadow (a tiny gradient update during backpropagation), your limited palette forces you to snap that value to the nearest available color. In machine learning, this snapping causes catastrophic information loss. Tiny, crucial learning signals are rounded to zero, and large activation spikes overflow the format, causing the model’s learning process to mathematically derail.
NVIDIA’s workaround was to use “current-tensor scaling” combined with the Randomized Hadamard Transform. The RHT acts like a mathematical blender, mathematically rotating the data to distribute the energy of large outliers across the entire tensor before it gets quantized into 4-bit. While effective at preventing collapse, this blending process requires significant computational work outside of the core matrix multiplications. Furthermore, because the final layers of a model are particularly sensitive, NVIDIA’s recipe exempts them from 4-bit compression entirely, forcing them to run in 16-bit (BF16) and creating a persistent memory and compute bottleneck.
How the Mechanism Works
The researchers bypassed this bottleneck by abandoning tensor-wide scaling in favor of block scaling, paired with a custom scaling format called UE5M3.
Instead of forcing an entire massive tensor (a giant grid of numbers) to share a single scaling factor, the data is divided into smaller, manageable chunks—specifically, blocks of 16 elements. Each block is assigned its own unique scaling factor. Returning to the painting analogy: instead of forcing the entire canvas to share one limited 16-color palette, you divide the canvas into small squares, and each square gets its own custom multiplier to adjust the brightness and contrast of its specific 16 colors.
The genius of the paper lies in the format of that scaling factor. The researchers utilized an Unsigned E5M3 (UE5M3) format for the block scales. Because it is unsigned (scaling factors are always positive), it dedicates 5 bits to the exponent and 3 to the mantissa. This provides a vastly wider dynamic range than standard formats, allowing the network to accurately capture both massive activation spikes and microscopic gradient updates without overflowing or underflowing.
With this wider range secured, the team introduced a few critical algorithmic tweaks:
- Periodic Tensor Scaling: Because the block scales handle the heavy lifting, the overarching tensor scales only need to be updated periodically, saving compute cycles.
- Selective Stochastic Rounding: When calculating the backward gradients (how the model learns from its mistakes), the system uses stochastic rounding. Instead of always rounding a tiny number like 0.1 down to 0, it rolls a weighted mathematical die—giving it a 10% chance to round up to 1. Over billions of operations, this preserves the mathematical expectation of the tiny learning signals that E2M1 would normally crush.
- Total FP4 Integration: Because the block scaling is so robust, the RHT “blender” is entirely omitted, and the final layers are safely compressed into FP4 alongside the rest of the model.
Empirical Results & Benchmarks
To prove their architecture, the team didn’t just run theoretical simulations; they pretrained a Nemotron-H 8B model for nearly 190 billion tokens and pitted their UE5M3 block scaling recipe directly against NVIDIA’s Transformer Engine baseline.
The empirical results demonstrated superiority across both training efficiency and model quality. The proposed recipe finished with a lower final-window training loss. When evaluated on held-out data (data the model had never seen), it achieved a lower validation loss, measured as negative log-likelihood. Furthermore, in downstream quantized-inference tasks, the new recipe scored higher on all three reported aggregate benchmarks.
Most critically for hardware engineers, the removal of the RHT and the BF16 final-block exemption resulted in massive throughput gains. In a native ablation study, stripping out these computational crutches increased the measured model-body token throughput by exactly 21.2%.
| Metric | NVIDIA Transformer Engine (Baseline) | Proposed UE5M3 Block Scaling |
|---|---|---|
| Training Loss | Baseline | Lower (Improved) |
| Validation Loss (NLL) | Baseline | Lower (Improved) |
| Model-Body Throughput | Baseline | +21.2% Increase |
| Architecture | Requires RHT & Mixed BF16 | Pure FP4, No RHT |
Practical Constraints & Commercial Horizon
While the mathematical proof and empirical results are highly compelling, there is a significant practical constraint separating this research from immediate data center deployment: hardware support.
The end-to-end pretraining demonstrated in this paper was software-emulated. Current-generation AI accelerators do not possess the native silicon logic gates required to execute UE5M3 block scaling natively at the hardware level. The 21.2% throughput increase was measured via a native execution ablation—essentially proving what the speedup will be once the hardware catches up to the math.
The commercial horizon for this breakthrough is likely tied to the next generation of AI silicon. By proving that a simpler, RHT-free recipe can achieve better loss curves and higher throughput, the authors have provided a clear blueprint for hardware designers. Future AI chips that natively support UE5M3 block scaling will be able to train massive language models significantly faster, using less power, and requiring a fraction of the memory bandwidth currently demanded by state-of-the-art clusters.
Paper & Author Citations
This research, titled “UE5M3 FP4 Block Scaling for Stable Language Model Pretraining,” was authored by Robert Hu, Carlo Luschi, and Paul Balanca. The paper is available as a preprint on arXiv (cs.LG) under the identifier arXiv:2609.02846v1.
Research Paper & Citation Details
Original Title: UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
Authors / Affiliation: Robert Hu, Carlo Luschi, Paul Balanca
Source Repository: arXiv Preprint (cs.LG)
Read Original Preprint / Paper →