Abstract:We find that different Deep Neural Networks (DNNs) trained with the same dataset share a common principal subspace in latent spaces, no matter in which architectures (e.g., Convolutional Neural Networks (CNNs), Multi-Layer Preceptors (MLPs) and Autoencoders (AEs)) the DNNs were built or even whether labels have been used in training (e.g., supervised, unsupervised, and self-supervised learning). Specifically, we design a new metric Pdocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${mathcal {P}}$$end{document}-vector to represent the principal subspace of deep features learned in a DNN, and propose to measure angles between the principal subspaces using Pdocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${mathcal {P}}$$end{document}-vectors. Small angles (with cosine close to 1.0) have been found in the comparisons between any two DNNs trained with different algorithms/architectures. Furthermore, during the training procedure from random scratch, the angle decrease from a larger one (70°–80° usually) to the small one, which coincides the progress of feature space learning from scratch to convergence. Then, we carry out case studies to measure the angle between the Pdocumentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$${mathcal {P}}$$end{document}-vector and the principal subspace of training dataset, and connect such angle with generalization performance. Extensive experiments with practically-used Multi-Layer Perceptron (MLPs), AEs and CNNs for classification, image reconstruction, and self-supervised learning tasks on MNIST, CIFAR-10 and CIFAR-100 datasets have been done to support our claims with solid evidences.

Condition Identification Using Direction of Principal Component and Neural Network

A novel compound neural network for fault diagnosis

Scene-Based Nonuniformity Correction Method Using Principal Component Analysis for Infrared Focal Plane Arrays

Condition classification and tendency prediction for prognostics using feature extraction and reconstruction

Neural Incremental Attribute Learning Based on Principal Component Analysis

Discrimination Of Varieties Of Paddy Seed Using Vis/Nirs By Principal Component Analysis And Artificial Neural Network Model

Forecasting Approach for Short-term Traffic Flow Based on Principal Component Analysis and Combined Neural Network

Principal component net analysis for face recognition

A novel approach to pattern recognition based on PCA-ANN in spectroscopy

A stepwise method for extracting the characteristic of complex chemical pattern in natural plants

Neural Network Characterization and Entropy Regulated Data Balancing through Principal Component Analysis

ConditionSenseNet: A Deep Interpolatory ConvNet for Bearing Intelligent Diagnosis under Variational Working Conditions

Principal Components-Artificial Neural Networks for Predicting SSC and Firmness of Fruits Based on Near Infrared Spectroscopy

A Modulation Recognition Algorithm via Hybrid Feature Analysis in Aeronautical Wireless Channel

A IQ Correlation and Long-term Features Neural Network Classifier for Automatic Modulation Classification

A Unified Self-Stabilizing Neural Network Algorithm for Principal Takagi Component Extraction

Target Recognition in Naval Battlefield Based on Principal Component Analysis and Neural Networks

Multivariable correlation feature network construction and health condition assessment for unlabeled single-sample data

Principal Component Analysis and Automatic Relevance Determination in Damage Identification

Discrimination of Varieties of Apple Using Near Infrared Spectra Based on Principal Component Analysis and Artificial Neural Network Model

Exploring the common principal subspace of deep features in neural networks