Abstract:Recent work in Deep Learning has re-imagined the representation of data as functions mapping from a coordinate space to an underlying continuous signal. When such functions are approximated by neural networks this introduces a compelling alternative to the more common multi-dimensional array representation. Recent work on such Implicit Neural Representations (INRs) has shown that - following careful architecture search - INRs can outperform established compression methods such as JPEG (e.g. Dupont et al., 2021). In this paper, we propose crucial steps towards making such ideas scalable: Firstly, we employ state-of-the-art network sparsification techniques to drastically improve compression. Secondly, introduce the first method allowing for sparsification to be employed in the inner-loop of commonly used Meta-Learning algorithms, drastically improving both compression and the computational cost of learning INRs. The generality of this formalism allows us to present results on diverse data modalities such as images, manifolds, signed distance functions, 3D shapes and scenes, several of which establish new state-of-the-art results.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the trade - off between network size and approximation quality when using Implicit Neural Representations (INRs) in data compression, as well as the high computational cost of learning INRs. Specifically: 1. **Trade - off between network size and approximation quality**: Traditional INR methods need to balance network size and approximation quality through architecture search or strong inductive bias, which is a challenge in practical applications. The paper proposes to significantly improve compression efficiency by introducing state - of - the - art network sparsification techniques, reducing the need for complex architecture search. 2. **High computational cost**: The computational cost required to learn the INR of a single data point is much higher than that of standard compression methods (such as JPEG), especially when additional architecture search is required. The paper uses meta - learning techniques, especially the MAML (Model - Agnostic Meta - Learning) algorithm, to significantly reduce the computational cost of learning INRs. Specifically, the paper proposes the first technique that can use sparsification in the inner loop of the meta - learning algorithm, thereby improving both compression efficiency and computational efficiency at the same time. The main contributions of the paper are: - **Introduction of L0 regularization**: Through the L0 regularization technique, the network parameters are sparsified, thereby reducing the storage and computational requirements of the model while maintaining high quality. - **Sparse adaptation**: Sparsification is introduced in the inner loop of meta - learning, enabling the network to quickly adapt to new tasks and maintain a sparse structure, further reducing the compression cost. - **Multiple sparse forms**: The paper explores two specific sparsification forms, unstructured sparse gradients and structured sparse modulation, which are suitable for different application scenarios. Through these innovations, the paper aims to make INRs a more efficient and practical data compression method, especially when dealing with multi - modal data (such as images, manifolds, signed distance functions, 3D shapes and scenes).

Meta-Learning Sparse Compression Networks

Neural Network Compression Via Sparse Optimization

LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time

Hybrid Network Compression Via Meta-Learning

Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction

Breaking the Barriers of One-to-One Usage of Implicit Neural Representation in Image Compression: A Linear Combination Approach with Performance Guarantees

Efficient Neural Network Compression Inspired by Compressive Sensing.

Spatially-Adaptive Learning-Based Image Compression with Hierarchical Multi-Scale Latent Spaces

Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics

Exploring Structural Sparsity in Neural Image Compression

On Compressing Deep Models by Low Rank and Sparse Decomposition.

Extreme Compression of Adaptive Neural Images

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

Layerwise Sparse Coding for Pruned Deep Neural Networks with Extreme Compression Ratio.

Compressing Neural Networks Using Learnable 1-D Non-Linear Functions

Streaming Neural Images

Learning to Compress: Local Rank and Information Compression in Deep Neural Networks

Resource Constrained Model Compression via Minimax Optimization for Spiking Neural Networks

Enhanced Invertible Encoding for Learned Image Compression

Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training

COIN++: Neural Compression Across Modalities