Abstract:Machine learning algorithms are typically run on large scale, distributed compute infrastructure that routinely face a number of unavailabilities such as failures and temporary slowdowns. Adding redundant computations using coding-theoretic tools called "codes" is an emerging technique to alleviate the adverse effects of such unavailabilities. A code consists of an encoding function that proactively introduces redundant computation and a decoding function that reconstructs unavailable outputs using the available ones. Past work focuses on using codes to provide resilience for linear computations and specific iterative optimization algorithms. However, computations performed for a variety of applications including inference on state-of-the-art machine learning algorithms, such as neural networks, typically fall outside this realm. In this paper, we propose taking a learning-based approach to designing codes that can handle non-linear computations. We present carefully designed neural network architectures and a training methodology for learning encoding and decoding functions that produce approximate reconstructions of unavailable computation results. We present extensive experimental results demonstrating the effectiveness of the proposed approach: we show that the our learned codes can accurately reconstruct $64 - 98\%$ of the unavailable predictions from neural-network based image classifiers on the MNIST, Fashion-MNIST, and CIFAR-10 datasets. To the best of our knowledge, this work proposes the first learning-based approach for designing codes, and also presents the first coding-theoretic solution that can provide resilience for any non-linear (differentiable) computation. Our results show that learning can be an effective technique for designing codes, and that learned codes are a highly promising approach for bringing the benefits of coding to non-linear computations.

AI Coding: Learning to Construct Error Correction Codes

An Improved Error-Correcting Output Coding Framework with Kernel-Based Decoding

The design of variable-length coding matrix for improving error correcting output codes

The design of soft recoding-based strategies for improving error-correcting output codes

Learning Rate-Compatible Linear Block Codes: An Auto-Encoder Based Approach

A self-adaptive soft-recoding strategy for performance improvement of error-correcting output codes

Learning Linear Block Error Correction Codes

Overview of Error Correcting Output Coding

Unrestricted Error-Type Codebook Generation for Error Correction Code in DNA Storage Inspired by NLP

Multiple-Error-Correcting Codes for Analog Computing on Resistive Crossbars

A novel soft-coded error-correcting output codes algorithm

Error Correction Performance Analysis of Error Resilient Arithmetic Code

Learning a Code: Machine Learning for Approximate Non-Linear Coded Computation

A Novel Implementation Methodology for Error Correction Codes on a Neuromorphic Architecture

On the Design of Near-Optimal Variable-Length Error-Correcting Codes for Large Source Alphabets

ProductAE: Toward Deep Learning Driven Error-Correction Codes of Large Dimensions

LEARN Codes: Inventing Low-latency Codes via Recurrent Neural Networks

On the Design and Performance of Machine Learning Based Error Correcting Decoders

Fault Secure Encoder and Decoder Designs for Matrix Codes

Construction of EIRA Codes with Enlarged Dual Diagonal Distance by EXIT Charts