Abstract:Manifold learning techniques have significantly enhanced the comprehension of massive data by exploring the geometric properties of the data manifold in low-dimensional subspaces. However, existing research on manifold learning primarily focuses on understanding the intricate data, overlooking the explosive growth of the scale and complexity of deep neural networks (DNNs), which presents a significant challenge for model optimization. In this work, we propose to explore the intrinsic low-dimensional manifold of network parameters for efficient model optimization. Specifically, we analyze parameter distributions in a deep model and perform sampling to map them onto a low-dimensional parameter manifold using the local tangent space alignment (LTSA). Since our focus is on studying parameter manifolds to guide model optimization, we therefore select dynamic optimal training trajectories for sampling and approximate tangent spaces to obtain low-dimensional representations of DNNs. By applying manifold learning techniques and employing a two-step alternate optimization method, we achieve a fixed subspace that reduces training time and resource costs for commonly used deep networks. The trained low-dimensional network can be mapped back to the original parameter space for further use. We demonstrate the benefits of learning low-dimensional parameterization of DNNs on both noisy label learning and federated learning tasks. Extensive experimental results on various benchmarks show the effectiveness of our method concerning both superior accuracy and reduced resource consumption.

Exploring Low-Dimensional Manifolds of Deep Neural Network Parameters for Improved Model Optimization

Deep Manifold Transformation for Dimension Reduction

Image Feature Optimization Based on Nonlinear Dimensionality Reduction

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

Deep Manifold Computing and Visualization

Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness

Markov-Lipschitz Deep Learning

Manifold Learning for Accelerating Coarse-Grained Optimization

Optimizations on Unknown Low-Dimensional Structures Given by High-Dimensional Data

Manifold Neural Network With Non-Gradient Optimization

Invertible Manifold Learning for Dimension Reduction

A Geometric Modeling of Occam's Razor in Deep Learning

The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold

Low-rank representation with adaptive dimensionality reduction via manifold optimization for clustering

Exploring the Manifold of Neural Networks Using Diffusion Geometry

Manifold Optimization Assisted Gaussian Variational Approximation

Train Deep Neural Networks in 40-D Subspaces

Efficient Pareto Manifold Learning with Low-Rank Structure

Deep Manifold Part 1: Anatomy of Neural Network Manifold

Efficient Representation of Low-Dimensional Manifolds using Deep Networks

Adaptive Neighborhood Selection for Manifold Learning