Fault-Tolerant Dot-Product Engines

Ron M. Roth

DOI: https://doi.org/10.48550/arXiv.1708.06892

2017-08-23

Abstract:Coding schemes are presented that provide the ability to correct and detect computational errors while using dot-product engines for integer vector--matrix multiplication. Both the $L_1$-metric and the Hamming metric are considered.

Information Theory

What problem does this paper attempt to address?

"Short-Dot": Computing Large Linear Transforms Distributedly Using Coded Short Dot Products

Sanghamitra Dutta,Viveck Cadambe,Pulkit Grover

DOI: https://doi.org/10.48550/arXiv.1704.05181

2017-04-18

Abstract:Faced with saturation of Moore's law and increasing dimension of data, system designers have increasingly resorted to parallel and distributed computing. However, distributed computing is often bottle necked by a small fraction of slow processors called "stragglers" that reduce the speed of computation because the fusion node has to wait for all processors to finish. To combat the effect of stragglers, recent literature introduces redundancy in computations across processors, e.g.,~using repetition-based strategies or erasure codes. The fusion node can exploit this redundancy by completing the computation using outputs from only a subset of the processors, ignoring the stragglers. In this paper, we propose a novel technique -- that we call "Short-Dot" -- to introduce redundant computations in a coding theory inspired fashion, for computing linear transforms of long vectors. Instead of computing long dot products as required in the original linear transform, we construct a larger number of redundant and short dot products that can be computed faster and more efficiently at individual processors. In reference to comparable schemes that introduce redundancy to tackle stragglers, Short-Dot reduces the cost of computation, storage and communication since shorter portions are stored and computed at each processor, and also shorter portions of the input is communicated to each processor. We demonstrate through probabilistic analysis as well as experiments that Short-Dot offers significant speed-up compared to existing techniques. We also derive trade-offs between the length of the dot-products and the resilience to stragglers (number of processors to wait for), for any such strategy and compare it to that achieved by our strategy.

Information Theory
Exploiting Data Representation for Fault Tolerance

James Elliott,Mark Hoemmen,Frank Mueller

DOI: https://doi.org/10.1016/j.jocs.2015.12.002

2013-12-09

Abstract:We explore the link between data representation and soft errors in dot products. We present an analytic model for the absolute error introduced should a soft error corrupt a bit in an IEEE-754 floating-point number. We show how this finding relates to the fundamental linear algebra concepts of normalization and matrix equilibration. We present a case study illustrating that the probability of experiencing a large error in a dot product is minimized when both vectors are normalized. Furthermore, when data is normalized we show that the absolute error is less than one or very large, which allows us to detect large errors. We demonstrate how this finding can be used by instrumenting the GMRES iterative solver. We count all possible errors that can be introduced through faults in arithmetic in the computationally intensive orthogonalization phase, and show that when scaling is used the absolute error can be bounded above by one.

Numerical Analysis
Coded Computing for Fault-Tolerant Parallel QR Decomposition

Quang Minh Nguyen,Iain Weissburg,Haewon Jeong

2023-11-21

Abstract:QR decomposition is an essential operation for solving linear equations and obtaining least-squares solutions. In high-performance computing systems, large-scale parallel QR decomposition often faces node faults. We address this issue by proposing a fault-tolerant algorithm that incorporates `coded computing' into the parallel Gram-Schmidt method, commonly used for QR decomposition. Coded computing introduces error-correcting codes into computational processes to enhance resilience against intermediate failures. While traditional coding strategies cannot preserve the orthogonality of $Q$, recent work has proven a post-orthogonalization condition that allows low-cost restoration of the degraded orthogonality. In this paper, we construct a checksum-generator matrix for multiple-node failures that satisfies the post-orthogonalization condition and prove that our code satisfies the maximum-distance separable (MDS) property with high probability. Furthermore, we consider in-node checksum storage setting where checksums are stored in original nodes. We obtain the minimal number of checksums required to be resilient to any $f$ failures under the in-node checksum storage, and also propose an in-node systematic MDS coding strategy that achieves the lower bound. Extensive experiments validate our theories and showcase the negligible overhead of our coded computing framework for fault-tolerant QR decomposition.

Distributed, Parallel, and Cluster Computing,Information Theory,Systems and Control
Algorithmic Based Fault Tolerance Applied to High Performance Computing

George Bosilca,Remi Delmas,Jack Dongarra,Julien Langou

DOI: https://doi.org/10.48550/arXiv.0806.3121

2008-06-19

Distributed, Parallel, and Cluster Computing

Abstract:We present a new approach to fault tolerance for High Performance Computing system. Our approach is based on a careful adaptation of the Algorithmic Based Fault Tolerance technique (Huang and Abraham, 1984) to the need of parallel distributed computation. We obtain a strongly scalable mechanism for fault tolerance. We can also detect and correct errors (bit-flip) on the fly of a computation. To assess the viability of our approach, we have developed a fault tolerant matrix-matrix multiplication subroutine and we propose some models to predict its running time. Our parallel fault-tolerant matrix-matrix multiplication scores 1.4 TFLOPS on 484 processors (cluster jacquard.nersc.gov) and returns a correct result while one process failure has happened. This represents 65% of the machine peak efficiency and less than 12% overhead with respect to the fastest failure-free implementation. We predict (and have observed) that, as we increase the processor count, the overhead of the fault tolerance drops significantly.
Fault Tolerant Parallel FFTs Using Error Correction Codes and Parseval Checks

zhen gao,pedro reviriego,zhan xu,xin su,ming zhao,jing wang,juan antonio maestro

DOI: https://doi.org/10.1109/TVLSI.2015.2408621

2016-01-01

Abstract:Soft errors pose a reliability threat to modern electronic circuits. This makes protection against soft errors a requirement for many applications. Communications and signal processing systems are no exceptions to this trend. For some applications, an interesting option is to use algorithmic-based fault tolerance (ABFT) techniques that try to exploit the algorithmic properties to detect and correct errors. Signal processing and communication applications are well suited for ABFT. One example is fast Fourier transforms (FFTs) that are a key building block in many systems. Several protection schemes have been proposed to detect and correct errors in FFTs. Among those, probably the use of the Parseval or sum of squares check is the most widely known. In modern communication systems, it is increasingly common to find several blocks operating in parallel. Recently, a technique that exploits this fact to implement fault tolerance on parallel filters has been proposed. In this brief, this technique is first applied to protect FFTs. Then, two improved protection schemes that combine the use of error correction codes and Parseval checks are proposed and evaluated. The results show that the proposed schemes can further reduce the implementation cost of protection.
Erasure coding for fault oblivious linear system solvers

David F. Gleich,Ananth Grama,Yao Zhu

DOI: https://doi.org/10.48550/arXiv.1412.7364

2014-12-23

Abstract:Dealing with hardware and software faults is an important problem as parallel and distributed systems scale to millions of processing cores and wide area networks. Traditional methods for dealing with faults include checkpoint-restart, active replicas, and deterministic replay. Each of these techniques has associated resource overheads and constraints. In this paper, we propose an alternate approach to dealing with faults, based on input augmentation. This approach, which is an algorithmic analog of erasure coded storage, applies a minimally modified algorithm on the augmented input to produce an augmented output. The execution of such an algorithm proceeds completely oblivious to faults in the system. In the event of one or more faults, the real solution is recovered using a rapid reconstruction method from the augmented output. We demonstrate this approach on the problem of solving sparse linear systems using a conjugate gradient solver. We present input augmentation and output recovery techniques. Through detailed experiments, we show that our approach can be made oblivious to a large number of faults with low computational overhead. Specifically, we demonstrate cases where a single fault can be corrected with less than 10% overhead in time, and even in extreme cases (fault rates of 20%), our approach is able to compute a solution with reasonable overhead. These results represent a significant improvement over the state of the art.

Numerical Analysis,Distributed, Parallel, and Cluster Computing
Detection of soft errors in LU decomposition with partial pivoting using algorithm-based fault tolerance

Erlin Yao,Jiutian Zhang,Mingyu Chen,Guangming Tan,Ninghui Sun

DOI: https://doi.org/10.1177/1094342015578487

2015-01-01

Abstract:Soft errors in scientific computing applications are becoming inevitable with the ever-increasing system scale and execution time, and new technologies that feature increased transistor density and lower voltage. Soft errors can be mainly classified into two categories: bit-flipping error e.g. 1 becomes −1 in random access memory; and computation error e.g. 1+1 = 3 in floating point units. Traditionally, bit-flipping error is handled by the Error Correcting Code ECC technique, and computation error is dealt with the Triple Modular Redundancy TMR method. Note that, ECC cannot handle computation error, while TMR cannot deal with bit-flipping error and is not efficient on handling computation error. To uniformly and efficiently handle both computation and bit-flipping errors in matrix operations, the Algorithm-Based Fault Tolerance ABFT method is developed. This paper focuses on the detection of soft errors in the LU Decomposition with Partial Pivoting LUPP algorithm, which is widely used in scientific computing applications. First, this paper notes that existing ABFT methods are not adequate to detect soft errors in LUPP in terms of time or space. Then we propose a new ABFT algorithm which can detect soft errors in LUPP both flexible in time and comprehensive in space. Flexible in time means that soft errors can be detected flexibly during the execution instead of only at the end of LUPP, while comprehensive in space indicates that all of the elements in data matrices L and U will be covered for detecting soft errors. To show the feasibility and efficiency of the proposed algorithm, this paper has incorporated it into the implementation of LUPP in the widely used benchmark High Performance Linpack HPL. Experiment results verify the feasibility of this algorithm: for soft errors injected at various timings and to different elements in LUPP, this algorithm has detected most of the injected errors, which have covered all of the errors that cannot pass the residual check of HPL. Both theoretical overhead analysis and experiments demonstrate that this ABFT algorithm is also very efficient at detecting soft errors in LUPP.
Efficient Coding Schemes for Fault-Tolerant Parallel Filters

Zhen Gao,Pedro Reviriego,Zhan Xu,Xin Su,Jing Wang,Juan Antonio Maestro

DOI: https://doi.org/10.1109/tcsii.2015.2404219

2015-01-01

IEEE Transactions on Circuits & Systems II Express Briefs

Abstract:As the complexity of communications and signal processing systems increases, so does the number of blocks or elements that they have. In many cases, some of those elements operate in parallel, performing the same processing on different signals. A typical example of those elements are digital filters. The increase in complexity also poses reliability challenges and creates the need for fault-tolerant implementations. A scheme based on error correction coding has been recently proposed to protect parallel filters. In that scheme, each filter is treated as a bit, and redundant filters that act as parity check bits are introduced to detect and correct errors. In this brief, the idea of applying coding techniques to protect parallel filters is addressed in a more general way. In particular, it is shown that the fact that filter inputs and outputs are not bits but numbers enables a more efficient protection. This reduces the protection overhead and makes the number of redundant filters independent of the number of parallel filters. The proposed scheme is first described and then illustrated with two case studies. Finally, both the effectiveness in protecting against errors and the cost are evaluated for a field-programmable gate array implementation.
Flag Gadgets based on Classical Codes

Benjamin Anker,Milad Marvian

2024-02-21

Abstract:Fault-tolerant syndrome extraction is a key ingredient in implementing fault-tolerant quantum computations. While conventional methods use a number of extra qubits linear in the weight of the syndrome, several improvements have been introduced using flag gadgets. In this work, we develop a framework to design flag gadgets using classical codes. Using this framework we show how to perform fault-tolerant syndrome extraction for any stabilizer code with arbitrary distance using exponentially fewer qubits than conventional methods when qubit measurement and reset are relatively slow compared to a round of error correction. We further take advantage of the saving provided by our construction to fault-tolerantly measure multiple stabilizers using a single gadget, and show that it maintains the same exponential advantage when it is used to fault-tolerantly extract the syndrome of quantum LDPC codes. Using the developed framework we perform computer-assisted search to find several small examples where our constructions reduce the number of qubits required. These small examples may be relevant to near-term experiments on small-scale quantum computers.

Quantum Physics
Comparing structures of two-dimensional error correction codes

Adahil Muniz,Lucas Mazzoco,Wagner Savaris,Eduarda Pissolatto,Tiago Beneditto,Andrew Fritsch,Jarbas Silveira,César Marcon

DOI: https://doi.org/10.1016/j.microrel.2024.115481

IF: 1.6

2024-08-19

Microelectronics Reliability

Abstract:Advances in integrated circuit production technologies have reduced device sizes, leading to corresponding scaling in electrical characteristics, such as threshold voltage. This scaling has increased the susceptibility of devices to electromagnetic radiation, raising the bitflip probability. Systems requiring a certain level of fault tolerance employ techniques like Error Correction Codes (ECC), providing a degree of reliability in mitigating this issue. The error correction and detection efficacies and ECC scalability vary based on the encoding and codestruct employed. This study employs four Hamming and parity code organizations for performing four Two-Dimensional (2D)-ECCs (N × 4p, N × ExHam, N × Ham_p, and N × Ham2_2p). We investigated the scalability, synthesis results, and correction and detection rates employing the same number of check and data bits for the four 2D-ECCs. The results point to the advantages for ECCs that employ cross-checking using radiation-hardened memories for checkbits, especially when ECCs scale to large codestructs.

engineering, electrical & electronic,nanoscience & nanotechnology,physics, applied
Tolerating C Integer Error Via Precision Elevation

Xi Cheng,Min Zhou,Xiaoyu Song,Ming Gu,Jiaguang Sun

DOI: https://doi.org/10.1109/tc.2018.2866388

IF: 3.183

2018-01-01

IEEE Transactions on Computers

Abstract:In C programs, integer error is a common yet important kind of defect due to arithmetic operations that produce unrepresentable values in certain types. Integer errors are harbored in a wide range of applications and possibly lead to serious software failures and exploitable vulnerabilities. Due to the complicated semantics of C, manually preventing integer errors is challenging even for experienced developers. In this paper we propose a novel approach to automate C integer error repair by elevating the precision of arithmetic operations according to a set of code transformation rules. A large portion of integer errors can be repaired by recovering expected results (i.e., tolerance) instead of removing program functionality. Our approach is fully automatic without requiring code specifications. Furthermore, the transformed code is ensured to be well-typed and has conservativeness property with respect to the original code. Our approach is implemented as a prototype CIntFix which succeeds in repairing all the integer errors from 7 categories in NIST's Juliet Test Suite. Furthermore, CIntFix is evaluated on large code bases in SPEC CINT2000, scaling to 366 KLOC within 126 seconds while the transformed code has 10.5 percent slowdown on average. The evaluation results substantiate the potential of our approach in real-world scenarios.
Multiple-Error-Correcting Codes for Analog Computing on Resistive Crossbars

Hengjia Wei,Ron M. Roth

2024-02-21

Abstract:Error-correcting codes over the real field are studied which can locate outlying computational errors when performing approximate computing of real vector--matrix multiplication on resistive crossbars. Prior work has concentrated on locating a single outlying error and, in this work, several classes of codes are presented which can handle multiple errors. It is first shown that one of the known constructions, which is based on spherical codes, can in fact handle multiple outlying errors. A second family of codes is then presented with $\zeroone$~parity-check matrices which are sparse and disjunct; such matrices have been used in other applications as well, especially in combinatorial group testing. In addition, a certain class of the codes that are obtained through this construction is shown to be efficiently decodable. As part of the study of sparse disjunct matrices, this work also contains improved lower and upper bounds on the maximum Hamming weight of the rows in such matrices.

Combinatorics
On the Optimal Recovery Threshold of Coded Matrix Multiplication

Sanghamitra Dutta,Mohammad Fahim,Farzin Haddadpour,Haewon Jeong,Viveck Cadambe,Pulkit Grover

DOI: https://doi.org/10.1109/tit.2019.2929328

IF: 2.5

2020-01-01

IEEE Transactions on Information Theory

Abstract:We provide novel coded computation strategies for distributed matrix–matrix products that outperform the recent "Polynomial code" constructions in recovery threshold, i.e., the required number of successful workers. When a fixed $1/m$ fraction of each matrix can be stored at each worker node, Polynomial codes require $m^{2}$ successful workers, while our MatDot codes only require $2m-1$ successful workers. However, MatDot codes have higher computation cost per worker and higher communication cost from each worker to the fusion node. We also provide a systematic construction of MatDot codes. Furthermore, we propose "PolyDot" coding that interpolates between Polynomial codes and MatDot codes to trade off computation/communication costs and recovery thresholds. Finally, we demonstrate a novel coding technique for multiplying $n$ matrices ( $n geq 3$ ) using ideas from MatDot and PolyDot codes.

computer science, information systems,engineering, electrical & electronic
The Detection and Correction of Silent Errors in Pipelined Krylov Subspace Methods

Erin Claire Carson,Jakub Hercík

2024-09-25

Abstract:As computational machines are becoming larger and more complex, the probability of hardware failure rises. ``Silent errors'', or, bit flips, may not be immediately apparent but can cause detrimental effects to algorithm behavior. In this work, we examine an algorithm-based approach to silent error detection in the context of pipelined Krylov subspace methods, in particular, Pipe-PR-CG, for the solution of linear systems. Our approach is based on using finite precision error analysis to bound the differences between quantities which should be equal in exact arithmetic. Through inexpensive monitoring during the iteration, we can detect when these bounds are violated, which indicates that a silent error has occurred. We use this approach to develop a fault-tolerance variant and also suggest a strategy for dynamically adapting the detection criteria. Our numerical experiments demonstrate the effectiveness of our approach.

Numerical Analysis
2D Parity Product Code for TSV online fault correction and detection

Khanh N. Dang,Akram Ben Ahmed,Abderazek Ben Abdallah,Michael Corad Meyer,Xuan-Tu Tran

DOI: https://doi.org/10.21553/rev-jec.242

2020-08-17

REV Journal on Electronics and Communications

Abstract:Through-Silicon-Via (TSV) is one of the most promising technologies to realize 3D Integrated Circuits (3D-ICs). However, the reliability issues due to the low yield rates and the sensitivity to thermal hotspots and stress issues are preventing TSV-based 3D-ICs from being widely and efficiently used. To enhance the reliability of TSV connections, using error correction code to detect and correct faults automatically has been demonstrated as a viable solution.This paper presents a 2D Parity Product Code (2D-PPC) for TSV fault-tolerance with the ability to correct one fault and detect, at least, two faults. In an implementation of 64-bit data and 81-bit codeword, 2D-PPC can detect over 71 faults, on average. Its encoder and decoder decrease the overall latency by 38.33% when compared to the Single Error Correction Double Error Detection code. In addition to the high detection rates, the encoder can detect 100% of its gate failures, and the decoder can detect and correct around 40% of its individual gate failures. The squared 2D-PPC could be extended using orthogonal Latin square to support extra bit correction.
Resource savings from fault-tolerant circuit design

Andrew K. Tan,Isaac L. Chuang

2023-11-03

Abstract:Using fault-tolerant constructions, computations performed with unreliable components can simulate their noiseless counterparts though the introduction of a modest amount of redundancy. Given the modest overhead required to achieve fault-tolerance, and the fact that increasing the reliability of basic components often comes at a cost, are there situations where fault-tolerance may be more economical? We present a general framework to account for this overhead cost in order to effectively compare fault-tolerant to non-fault-tolerant approaches for computation, in the limit of small logical error rates. Using this detailed accounting, we determine explicit boundaries at which fault-tolerant designs become more efficient than designs that achieve comparable reliability through direct consumption of resources. We find that the fault-tolerant construction is always preferred in the limit of high reliability in cases where the resources required to construct a basic unit grows faster than $\log(1 / \epsilon)$ asymptotically for small $\epsilon$.

Computational Engineering, Finance, and Science,Information Theory
The MacWilliams Identity for the Hermitian Rank Metric

Izzy Friedlander

2024-01-14

Abstract:Error-correcting codes have an important role in data storage and transmission and in cryptography, particularly in the post-quantum era. Hermitian matrices over finite fields and equipped with the rank metric have the potential to offer enhanced security with greater efficiency in encryption and decryption. One crucial tool for evaluating the error-correcting capabilities of a code is its weight distribution and the MacWilliams Theorem has long been used to identify this structure of new codes from their known duals. Earlier papers have developed the MacWilliams Theorem for certain classes of matrices in the form of a functional transformation, developed using $q$-algebra, character theory and Generalised Krawtchouk polynomials, which is easy to apply and also allows for moments of the weight distribution to be found. In this paper, recent work by Kai-Uwe Schmidt on the properties of codes based on Hermitian matrices such as bounds on their size and the eigenvalues of their association scheme is extended by introducing a negative-$q$ algebra to establish a MacWilliams Theorem in this form together with some of its associated moments. The similarities in this approach and in the paper for the Skew-Rank metric by Friedlander et al. have been emphasised to facilitate future generalisation to any translation scheme.

Information Theory
Twofold fast summation

Evgeny Latkin

DOI: https://doi.org/10.48550/arXiv.1401.0248

2014-01-03

Abstract:Debugging accumulation of floating-point errors is hard; ideally, computer should track it automatically. Here we consider twofold approximation of an exact real with value + error pair of floating-point numbers. Normally, value + error sum is more accurate than value alone, so error can estimate deviation between value and its exact target. Fast summation algorithm, that provides twofold sum of x[1]+...+x[N] or dot product x[1]*y[1]+...+x[N]*y[N], can be same fast as direct summation sometimes if leveraging processor underused potential. This way, we can hit three goals: improve precision, track inaccuracy, and do this with little if any loss in performance.

Numerical Analysis
Testing Tensor Products of Algebraic Codes

Sumegha Garg,Madhu Sudan,Gabriel Wu

2024-10-30

Abstract:Motivated by recent advances in locally testable codes and quantum LDPCs based on robust testability of tensor product codes, we explore the local testability of tensor products of (an abstraction of) algebraic geometry codes. Such codes are parameterized by, in addition to standard parameters such as block length $n$ and dimension $k$, their genus $g$. We show that the tensor product of two algebraic geometry codes is robustly locally testable provided $n = \Omega((k+g)^2)$. Apart from Reed-Solomon codes, this seems to be the first explicit family of codes of super-constant dual distance that is robustly locally testable.

Information Theory
Faster arbitrary-precision dot product and matrix multiplication

Fredrik Johansson

DOI: https://doi.org/10.48550/arXiv.1901.04289

2019-05-10

Abstract:We present algorithms for real and complex dot product and matrix multiplication in arbitrary-precision floating-point and ball arithmetic. A low-overhead dot product is implemented on the level of GMP limb arrays; it is about twice as fast as previous code in MPFR and Arb at precision up to several hundred bits. Up to 128 bits, it is 3-4 times as fast, costing 20-30 cycles per term for floating-point evaluation and 40-50 cycles per term for balls. We handle large matrix multiplications even more efficiently via blocks of scaled integer matrices. The new methods are implemented in Arb and significantly speed up polynomial operations and linear algebra.

Mathematical Software,Numerical Analysis

Fault-Tolerant Dot-Product Engines

"Short-Dot": Computing Large Linear Transforms Distributedly Using Coded Short Dot Products

Exploiting Data Representation for Fault Tolerance

Coded Computing for Fault-Tolerant Parallel QR Decomposition

Algorithmic Based Fault Tolerance Applied to High Performance Computing

Fault Tolerant Parallel FFTs Using Error Correction Codes and Parseval Checks

Erasure coding for fault oblivious linear system solvers

Detection of soft errors in LU decomposition with partial pivoting using algorithm-based fault tolerance

Efficient Coding Schemes for Fault-Tolerant Parallel Filters

Flag Gadgets based on Classical Codes

Comparing structures of two-dimensional error correction codes

Tolerating C Integer Error Via Precision Elevation

Multiple-Error-Correcting Codes for Analog Computing on Resistive Crossbars

On the Optimal Recovery Threshold of Coded Matrix Multiplication

The Detection and Correction of Silent Errors in Pipelined Krylov Subspace Methods

2D Parity Product Code for TSV online fault correction and detection

Resource savings from fault-tolerant circuit design

The MacWilliams Identity for the Hermitian Rank Metric

Twofold fast summation

Testing Tensor Products of Algebraic Codes

Faster arbitrary-precision dot product and matrix multiplication