Abstract:In recent years fused-multiply-add (FMA) units with lower-precision multiplications and higher-precision accumulation have proven useful in machine learning/artificial intelligence applications, most notably in training deep neural networks due to their extreme computational intensity. Compared to classical IEEE-754 32 bit (FP32) and 64 bit (FP64) arithmetic, these reduced precision arithmetic can naturally be sped up disproportional to their shortened width. The common strategy of all major hardware vendors is to aggressively further enhance their performance disproportionately. One particular FMA operation that multiplies two BF16 numbers while accumulating in FP32 has been found useful in deep learning, where BF16 is the 16-bit floating point datatype with IEEE FP32 numerical range but 8 significant bits of precision. In this paper, we examine the use this FMA unit to implement higher-precision matrix routines in terms of potential performance gain and implications on accuracy. We demonstrate how a decomposition into multiple smaller datatypes can be used to assemble a high-precision result, leveraging the higher precision accumulation of the FMA unit. We first demonstrate that computations of vector inner products and by natural extension, matrix-matrix products can be achieved by decomposing FP32 numbers in several BF16 numbers followed by appropriate computations that can accommodate the dynamic range and preserve accuracy compared to standard FP32 computations, while projecting up to $5.2 imes$ speed-up. Furthermore, we examine solution of linear equations formulated in the residual form that allows for iterative refinement. We demonstrate that the solution obtained to be comparable to those offered by FP64 under a large range of linear system condition numbers.

Floating-floating point: a highly accurate number representation with flexible Counting ranges

On the precision attainable with various floating-point number systems

Approximate Floating-Point FFT Design with Wide Precision-Range and High Energy Efficiency.

EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Models

Floating-Point Numbers with Error Estimates (revised)

Exploring and Exploiting Runtime Reconfigurable Floating Point Precision in Scientific Computing: a Case Study for Solving PDEs

Floating-Point Unit Processing Denormalized Numbers

Leveraging the bfloat16 Artificial Intelligence Datatype For Higher-Precision Computations

Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks

Ascend HiFloat8 Format for Deep Learning

Floating-point to Fixed-point Transformation Using Extreme Value Theory

Application-specific number representation

PAM: A Piecewise-Linearly-Approximated Floating-Point Multiplier with Unbiasedness and Configurability

Secure and Accurate Summation of Many Floating-Point Numbers

Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs

Optimizing FPGA-Based DNN Accelerator with Shared Exponential Floating-Point Format

A two-item floating point fused dot-product unit with latency reduced

Low-Precision Floating-Point Schemes for Neural Network Training

FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding

MiniFloats on RISC-V Cores: ISA Extensions with Mixed-Precision Short Dot Products

Training DNNs with Hybrid Block Floating Point