Abstract:Gradient descent optimizations and backpropagation are the most common methods for training neural networks, but they are computationally expensive for real time applications, need high memory resources, and are difficult to converge for many networks and large datasets. [Pseudo]inverse models for training neural network have emerged as powerful tools to overcome these issues. In order to effectively implement these methods, structured pruning maybe be applied to produce sparse neural networks. Although sparse neural networks are efficient in memory usage, most of their algorithms use the same fully loaded matrix calculation methods which are not efficient for sparse matrices. Tridiagonal matrices are one of the frequently used candidates for structuring neural networks, but they are not flexible enough to handle underfitting and overfitting problems as well as generalization properties. In this paper, we introduce a nonsymmetric, tridiagonal matrix with offdiagonal sparse entries and offset sub and super-diagonals as well algorithms for its [pseudo]inverse and determinant calculations. Traditional algorithms for matrix calculations, specifically inversion and determinant, of these forms are not efficient specially for large matrices, e.g. larger datasets or deeper networks. A decomposition for lower triangular matrices is developed and the original matrix is factorized into a set of matrices where their inverse matrices are calculated. For the cases where the matrix inverse does not exist, a least square type pseudoinverse is provided. The present method is a direct routine, i.e., executes in a predictable number of operations which is tested for randomly generated matrices with varying size. The results show significant improvement in computational costs specially when the size of matrix increases.

Sparse Partial-Tracing

Efficient Distributed-Memory Parallel Matrix-Vector Multiplication with Wide or Tall Unstructured Sparse Matrices

Sparse Matrices in MATLAB: Design and Implementation

Multiresolution kernel matrix algebra

Arrow Matrix Decomposition: A Novel Approach for Communication-Efficient Sparse Matrix Multiplication

Keeping it sparse: Computing Persistent Homology revisited

Near Optimal Frequent Directions for Sketching Dense and Sparse Matrices

TensorTrace: an application to contract tensor networks

MMSparse: 2D Partitioning of Sparse Matrix Based on Mathematical Morphology

Efficient Utilization of Multi-Threading Parallelism on Heterogeneous Systems for Sparse Tensor Contraction

Performance Optimization for Sparse A(T)Ax in Parallel on Multicore Cpu

High-Performance Tensor Contraction without Transposition

Shared Memory Parallelization of MTTKRP for Dense Tensors

Approximating Sparse Matrices and their Functions using Matrix-vector products

Sparse Matrix-Based HPC Tomography

Masked Matrix Multiplication for Emergent Sparsity

A Structured Sparse Neural Network and Its Matrix Calculations Algorithm

Sparsification of Matrices and Compressed Sensing

Cardinality Sparsity: Applications in Matrix-Matrix Multiplications and Machine Learning

Skew-Symmetric Matrix Decompositions on Shared-Memory Architectures

Sparse Signal Processing with Frame Theory