Abstract:Most multilayer least squares (LS)-based neural networks are structured with two separate stages: unsupervised feature encoding and supervised pattern classification. Once the unsupervised learning is finished, the latent encoding would be fixed without supervised fine-tuning. However, in complex tasks such as handling the ImageNet dataset, there are often many more clues that can be directly encoded, while the unsupervised learning, by definition cannot know exactly what is useful for a certain task. This serves as the motivation to retrain the latent space representations to learn some clues that unsupervised learning has not yet learned. In particular, the error matrix from the output layer is pulled back to each hidden layer, and the parameters of the hidden layer are recalculated with Moore-Penrose (MP) inverse for more generalized representations. In this paper, a recomputation-based multilayer network using MP inverse (RML-MP) is developed. A sparse RML-MP (SRML-MP) model to boost the performance of RML-MP is then proposed. The experimental results with varying training samples (from 3 K to 1.8 M) show that the proposed models provide better generalization performance than most representation learning algorithms.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve include: 1. **Poor performance on high - dimensional large - data sets**: Existing least - squares (LS) - based representation learning (RL) algorithms cannot obtain satisfactory results when dealing with high - dimensional large - data sets. Most LS - based RL algorithms focus on handling small and medium - sized data sets, such as MNIST and NORB (with no more than 100,000 samples), and there are few studies using Moore - Penrose (MP) inverse to handle large - data sets (such as ImageNet, which contains more than 1.2 million images). 2. **Lack of supervised adjustment**: Most MP - inverse - based RL frameworks generate loosely - connected representations when dealing with large - data sets. Since these models belong to unsupervised dimensionality reduction (without label information), once the features are extracted, the parameters are fixed and not fine - tuned. Therefore, some important clues may be filtered out before training the final layer, thus affecting the final performance. This problem is particularly evident when dealing with large - scale complex data sets. To address the above problems, this paper proposes a novel LS - based algorithm named RML - MP (Recomputation - based Multilayer Network using MP Inverse) for big - data analysis. Specifically, RML - MP improves existing methods in the following ways: - **Multilayer AE stacking**: Use LS - based AE with ℓ2 penalty for effective encoding and calculate the parameters of the final classification layer through the LS scheme. - **Error back - propagation and weight update**: Propagate the error of the current output layer back to each hidden layer layer by layer, generate the required output \(P\) for each hidden layer, and use the MP - inverse technique to recalculate the weights of each layer to strengthen the connection between the hidden - layer representations and the labels and obtain more robust representations. - **Sparse learning framework SRML - MP**: Introduce the ℓ1/2 - penalty - based learning framework SRML - MP to further improve performance, enabling SRML - MP to obtain sparse weights. Through these improvements, RML - MP and SRML - MP can show better generalization performance on data sets of different scales, especially when dealing with large - scale complex data sets, significantly outperforming existing multilayer LS - based RL algorithms.

Multi-Model Least Squares-Based Recomputation Framework for Large Data Analysis

Multimodal Moore-Penrose Inverse-Based Recomputation Framework for Big Data Analysis

s-LMPNet: a super-lightweight multi-stage progressive network for image super-resolution

ReFiNe: Recursive Field Networks for Cross-modal Multi-scene Representation

A Multilayered-and-Randomized Latent Factor Model for High-Dimensional and Sparse Matrices

RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition

Revisiting Recursive Least Squares for Training Deep Neural Networks

Scene Classification of Remote Sensing Image Based on Multi-Path Reconfigurable Neural Network

Enhancing MRF Reconstruction: A Model-Based Deep Learning Approach Leveraging Learned Sparsity and Physics Priors

Efficient Deep Spiking Multilayer Perceptrons With Multiplication-Free Inference

Brain-inspired Multilayer Perceptron with Spiking Neurons

MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning

Understanding Dynamics of Nonlinear Representation Learning and Its Application

SRP&PASMLP‐Net: Lightweight skin lesion segmentation network based on structural re‐parameterization and parallel axial shift multilayer perceptron

R2-MLP: Round-Roll MLP for Multi-View 3D Object Recognition

MRIo3DS-Net: A Mutually Reinforcing Images to 3D Surface RNN-like framework for model-adaptation indoor 3D reconstruction

Optimizing Large Model Training through Overlapped Activation Recomputation

You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet

Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks

MRFN: Multi-Receptive-Field Network for Fast and Accurate Single Image Super-Resolution

Efficient Single Image Super-Resolution with Entropy Attention and Receptive Field Augmentation