Manifold Learning for Visualizing and Analyzing High-Dimensional Data

Junping Zhang,Hua Huang,Jue Wang

DOI: https://doi.org/10.1109/mis.2010.8

IF: 6.744

2010-01-01

IEEE Intelligent Systems

Abstract:Assuming that high-dimensional data are generated from intrinsic variables with lower dimensions, several key manifold-learning algorithms can help effectively analyze and visualize such data.

What problem does this paper attempt to address?

Deep Manifold Transformation for Dimension Reduction

Stan Z. Li,Zelin Zang,Lirong Wu

2020-01-01

Abstract:Nonlinear dimensionality reduction (NLDR) plays an important role in feature extraction and visualization of high dimensional data. It transforms patterns of interest, or manifolds, in data to regions in a lower-dimensional latent space. When the manifolds are highly complicated, the NLDR transformation has to provide sufficient nonlinearity for downstream tasks. This paper proposes a novel method, called {\em deep manifold transformation} (DMT), to tackle this problem. As a multi-layer neural network, DMT can support a higher degree of nonlinearity than single-layer methods. Continuation strategy is applied during training to address the local minimum problem common in manifold learning algorithms. Such a learned DMT network can generalize to unseen data whereas traditional manifold learning methods only provide an embedding of the training data. Extensive experiments, comparisons, and ablation studies demonstrate that DMT can deliver results superior to UMAP and t-SNE, and other leading manifold-based NLDR methods.
Deep Manifold Computing and Visualization

Stan Z. Li,Zelin Zang,Lirong Wu

2020-01-01

Abstract: The ability to preserve local geometry of highly nonlinear manifolds in high dimensional spaces and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold computing, nonlinear dimensionality reduction (NLDR) and visualization. This paper proposes a novel method, called elastic locally isometric smoothness (ELIS), to empower deep neural networks with such an ability. ELIS requires that a desired metric between points should be preserved across layers in order to preserve local geometry; such a smoothness constraint effectively regularizes vector-based transformations to become well-behaved local metric-preserving homeomorphisms. Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions. The ELIS method incorporates a class of suitable nonlinear similarity functions into a two-way divergence loss and uses hyperparameter continuation in finding optimal solutions. Extensive experiments, comparisons, and ablation study demonstrate that ELIS can deliver results not only superior to UMAP and t-SNE for and visualization but also better than other leading counterparts of manifold and autoencoder learning for NLDR and manifold data generation.
Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness

Stan Z. Li,Zelin Zang,Lirong Wu

2021-01-01

Abstract:The ability to preserve local geometry of highly nonlinear manifolds in high dimensional spaces and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold computing, nonlinear dimensionality reduction (NLDR) and visualization. This paper proposes a novel method, called elastic locally isometric smoothness (ELIS), to empower deep neural networks with such an ability. ELIS requires that a desired metric between points should be preserved across layers in order to preserve local geometry; such a smoothness constraint effectively regularizes vector-based transformations to become well-behaved local metric-preserving homeomorphisms. Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions. The ELIS method incorporates a class of suitable nonlinear similarity functions into a two-way divergence loss and uses hyperparameter continuation in finding optimal solutions. Extensive experiments, comparisons, and ablation study demonstrate that ELIS can deliver results not only superior to UMAP and t-SNE for and visualization but also better than other leading counterparts of manifold and autoencoder learning for NLDR and manifold data generation.
Deep Manifold Transformation for Nonlinear Dimensionality Reduction

Stan Z. Li,Zelin Zang,Lirong Wu

2020-01-01

Abstract:Manifold learning-based encoders have been playing important roles in nonlinear dimensionality reduction (NLDR) for data exploration. However, existing methods can often fail to preserve geometric, topological and/or distributional structures of data. In this paper, we propose a deep manifold learning framework, called deep manifold transformation (DMT) for unsupervised NLDR and embedding learning. DMT enhances deep neural networks by using cross-layer local geometry-preserving (LGP) constraints. The LGP constraints constitute the loss for deep manifold learning and serve as geometric regularizers for NLDR network training. Extensive experiments on synthetic and real-world data demonstrate that DMT networks outperform existing leading manifold-based NLDR methods in terms of preserving the structures of data.
The Research Of Visualization Data Mining Based On Manifold Learning

daren yu,huixia he

DOI: https://doi.org/10.1115/1.859919.paper173

2012-01-01

Abstract:Based on characteristics of data distribution in the low dimension nature, this paper studies the effect of manifold embedding technology used in typical industrial data set analysis, and manifold learning theory was introduced into the visualization data mining to help get quick effective information in a large number of data. Cognitive process with the help of that information reduces the difficulty of analysis and improves the data utilization efficiency, having a high value in engineering application.
Manifold-based learning and synthesis.

Dong Huang,Zhang Yi,Xiaorong Pu

DOI: https://doi.org/10.1109/TSMCB.2008.2007499

2009-01-01

Abstract:This paper proposes a new approach to analyze high-dimensional data set using low-dimensional manifold. This manifold-based approach provides a unified formulation for both learning from and synthesis back to the input space. The manifold learning method desires to solve two problems in many existing algorithms. The first problem is the local manifold distortion caused by the cost averaging of the global cost optimization during the manifold learning. The second problem results from the unit variance constraint generally used in those spectral embedding methods where global metric information is lost. For the out-of-sample data points, the proposed approach gives simple solutions to transverse between the input space and the feature space. In addition, this method can be used to estimate the underlying dimension and is robust to the number of neighbors. Experiments on both low-dimensional data and real image data are performed to illustrate the theory.
Analyzing Magnification Factors and Principal Spread Directions in Manifold Learning

Junping Zhang,Li He,Zhi-hua Zhou

DOI: https://doi.org/10.1007/3-540-31662-0_49

2006-01-01

Abstract:Great amount of data under varying intrinsic features is thought of as high dimensional nonlinear manifold in the observation space. How to analyze the mapping relationship between the high dimensional manifold and the corresponding intrinsic low dimensional one quantitatively is important to machine learning and cognitive science. In this paper, we propose SVD (singular value decomposition) based magnification factors and spread direction for quantitative analyzing the relationship. The result of conducting experiments on several databases show the advantages of this proposed SVD-based approach in manifold learning.
Constrained Manifold Learning for Hyperspectral Imagery Visualization

Danping Liao,Yuntao Qian,Yuan Yan Tang

DOI: https://doi.org/10.1109/igarss.2013.6723196

IF: 4.715

2018-01-01

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Abstract:Displaying the abundant information contained in a hyperspectral image is a challenging task. Previous visualization approach focused only on preserving the structure in the original images. They ended up with presenting pseudo-color images and stopped short of adjusting the color of the images to retrieve more desirable visual effects. In this paper, a new visualization algorithm is proposed. It can be modeled as a two stage approach. At the first stage, Laplacian Eigenmaps algorithm is applied to reduce the dimension of the hyperspectral image. In this way we obtain a three dimensional image with pseudo-color. At the second stage, we transfer the natural color of a panchromatic image to the image obtained by the first step via manifold alignment. Experimental results show that the visualized image not only retains the structure of the hyperspectral image but also possesses natural colors.
Investigating Manifold Learning Algorithms Based on Magnification Factors and Principal Spread Directions

HE Li,ZHANG Jun-Ping,ZHOU Zhi-Hua

DOI: https://doi.org/10.3321/j.issn:0254-4164.2005.12.007

2005-01-01

Chinese Journal of Computers

Abstract:As a new unsupervised learning technique, manifold learning has captured the attention of many researchers in the field of machine learning and cognitive sciences. The major algorithms include Isometric mapping (ISOMAP) and Locally Linear Embedding (LLE). The approaches can be used for discovering the intrinsic dimensions of nonlinear high-dimensional data effectively and aim researchers to analyze the data better. How to quantitatively analyze the relationship between the intrinsic dimensions and the observation space, however, has fewer reports. And thus further works in manifold learning may have suffered some difficulties. The paper focuses on two kinds of manifold learning algorithms (ISOMAP, LLE), and discusses magnification factors and principal spread directions from the observation space to the intrinsic low-dimensional space. Also the corresponding algorithm is proposed. Experiments show the effectiveness and advantages of the research.
Manifold learning: what, how, and why

Marina Meilă,Hanyu Zhang

2023-11-07

Abstract:Manifold learning (ML), known also as non-linear dimension reduction, is a set of methods to find the low dimensional structure of data. Dimension reduction for large, high dimensional data is not merely a way to reduce the data; the new representations and descriptors obtained by ML reveal the geometric shape of high dimensional point clouds, and allow one to visualize, de-noise and interpret them. This survey presents the principles underlying ML, the representative methods, as well as their statistical foundations from a practicing statistician's perspective. It describes the trade-offs, and what theory tells us about the parameter and algorithmic choices we make in order to obtain reliable conclusions.

Machine Learning
Editorial of the Special Issue on Manifold Learning.

Xiaofei He

2013-01-01

International Journal of Software and Informatics

Abstract:In many information analysis tasks, one is often confronted with thousands to millions dimensional data, such as images, documents, videos, web data, bioinformatics data, etc. Conventional statistical and computational tools are severely inadequate for processing and analysing high-dimensional data due to the curse of dimensionality, where we often need to conduct inference with a limited number of samples. On the other hand, naturally occurring data may be generated by structured systems with possibly much fewer degrees of freedom than the ambient dimension would suggest. Recently, various works have considered the case when the data is sampled from a submanifold embedded in the much higher dimensional Euclidean space. Learning with full consideration of the low dimensional manifold structure, or specifically the intrinsic topological and geometrical properties of the data manifold is referred to as manifold learning, which has been receiving growing attention in our community in recent years. This special issue is to attract articles that (a) address the frontier problems in the scientific principles of manifold learning, and (b) report empirical studies and applications of manifold learning algorithms, including but not limited to pattern recognition, computer vision, web mining, image processing and so on. A total of 13 submissions were received. The papers included in this special issue are selected based on the reviews by experts in the subject area according to the journal’s procedure and quality standard. Each paper is reviewed by at least two reviewers and some of the papers were revised for two rounds according to the reviewers’ comments. The special issue includes 6 papers in total: 3 papers on the foundational theories of manifold learning, 2 papers on graph-based methods, and 1 paper on the application of manifold learning to video compression. The papers on the foundational theories of manifold learning cover the topics about the generalization ability of manifold learning, manifold ranking, and multi-manifold factorization. In the paper entitled “Manifold Learning: Generalizing Ability and Tangential Proximity”, Bernstein and Kuleshov propose a tangential proximity based technique to address the generalized manifold learning problem. The proposed method ensures not only proximity between the points and their reconstructed values but also proximity between the corresponding tangent spaces. The traditional manifold ranking methods are based on the Laplacian
A geometric viewpoint of manifold learning

Binbin Lin,Xiaofei He,Jieping Ye

DOI: https://doi.org/10.1186/s40535-015-0006-6

2015-03-12

Applied Informatics

Abstract:In many data analysis tasks, one is often confronted with very high dimensional data. The manifold assumption, which states that the data is sampled from a submanifold embedded in much higher dimensional Euclidean space, has been widely adopted by many researchers. In the last 15 years, a large number of manifold learning algorithms have been proposed. Many of them rely on the evaluation of the geometrical and topological of the data manifold. In this paper, we present a review of these methods on a novel geometric perspective. We categorize these methods by three main groups: Laplacian-based, Hessian-based, and parallel field-based methods. We show the connection and difference between these three groups on their continuous and discrete counterparts. The discussion is focused on the problem of dimensionality reduction and semi-supervised learning.
Visualization of Hyperspectral Imaging Data Based on Manifold Alignment

Danping Liao,Yuntao Qian,Jun Zhou

DOI: https://doi.org/10.1109/icpr.2014.22

2014-01-01

Abstract:Tristimulus display of the abundant information contained in a hyper spectral image is a challenging task. Previous visualization approaches focused on preserving as much information as possible in the reduced spectral space, but ended up with displaying hyper spectral images as false color images, which contradicts with human experience and expectation. This paper proposes a new framework to tackle this problem. It is based on the fusion of a hyper spectral image and a high-resolution color image via manifold alignment technique. Manifold learning is an important tool for dimension reduction. Manifold alignment projects a pair of two data sets into a common embedding space so that the pairs of corresponding points in these two data sets are pair wise aligned in this new space. Hyper spectral image and high-resolution color image have strong complementary properties due to the high spectral resolution in the former and the high spatial resolution in the latter. The embedding space produced by manifold alignment bridges a gap between the high dimensional spectral space of hyper spectral image and RGB space of color image, making it possible to transfer the natural color and spatial information of a high-resolution color image to a hyper spectral image to generate a visualized image with natural color distribution and finer details.
Overview of Manifold Learning

XU Rong,JIANG Feng,YAO Hong-xun

DOI: https://doi.org/10.3969/j.issn.1673-4785.2006.01.007

2006-01-01

Abstract:As a new unsupervised learning method, manifold learning is capturing increasing interests of researchers in the field of machine learning and cognitive sciences. To understand manifold learning better, the topology concept of manifold learning was presented firstly, and then its development history was traced. Based on different representations of manifold, several major algorithms were introduced, whose advantages and defects were pointed out respectively. After that, two kinds of typical applications of Isomap and LLE were indicated. The results show that compared with traditional linear method, manifold learning can discover the intrinsic dimensions of nonlinear high-dimensional data effectively, helping researchers to reduce dimensionality and analyze data better. Finally the prospect of manifold learning was discussed, so as to extend the application area of manifold learning.
Interactive Pattern Discovery in High-Dimensional, Multimodal Data Using Manifolds

Jinhong K. Guo,Martin O. Hofmann

DOI: https://doi.org/10.1016/j.procs.2017.09.037

2017-01-01

Procedia Computer Science

Abstract:Data mining to discover patterns and aid decisions is the key to utilizing massive data for process automation and optimization. An especially challenging data mining problem is kriging, i.e., prediction of multiple, related variables from latent patterns in the data. We present a manifold based machine learning approach to discover patterns in massive, correlated, high-dimensional data. Dimensionality reduction using a manifold is a type of non-linear principal component analysis (PCA). The manifold captures the underlying data structure of the inputs and corresponding outputs by way of projecting the data onto a set of basis functions defined by the manifold. These bases ensure that any future adjustments affect the model with respect to the natural geometry of the data. We chose the manifold learning technique for its robustness against unbalanced data. Our contribution, described in this paper, enables interactive learning and incremental learning, i.e., incremental adjustment of the manifold (and its predictions) based on new observations and also user corrections to the predicted values, rerun the analysis on the full data set. Our experiments demonstrate that prediction performance remains equivalent to Multi-kernel Gaussian Processes on standard data sets despite these practically useful enhancements.
Adaptive Manifold Learning

Zhenyue Zhang,Jing Wang,Hongyuan Zha

DOI: https://doi.org/10.1109/tpami.2011.115

IF: 23.6

2012-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Manifold learning algorithms seek to find a low-dimensional parameterization of high-dimensional data. They heavily rely on the notion of what can be considered as local, how accurately the manifold can be approximated locally, and, last but not least, how the local structures can be patched together to produce the global parameterization. In this paper, we develop algorithms that address two key issues in manifold learning: 1) the adaptive selection of the local neighborhood sizes when imposing a connectivity structure on the given set of high-dimensional data points and 2) the adaptive bias reduction in the local low-dimensional embedding by accounting for the variations in the curvature of the manifold as well as its interplay with the sampling density of the data set. We demonstrate the effectiveness of our methods for improving the performance of manifold learning algorithms using both synthetic and real-world data sets.
Nonlinear Principal Manifolds --- Adaptive Hybrid Learning Approaches

Hujun Yin

DOI: https://doi.org/10.1007/978-3-540-87656-4_4

2008-01-01

Abstract:Dimension reduction has long been associated with retinotopic mapping for understanding cortical maps. Multisensory information is processed, fused, fed and mapped to a 2-D cortex in a near-optimal information preserving manner. Data projection and visualization, inspired by this mechanism, are playing an increasingly important role in many computational applications such as cluster analysis, classification, data mining, knowledge management and retrieval, decision support, marketing, image processing and analysis. Such tasks involving either visual and spatial analysis or reduction of features or volume of the data are essential in many fields from biology, neuroscience, decision support, to management science. The topic has recently attracted a great deal of attention. There have been considerable advances in methodology and techniques for extracting nonlinear manifold as to reduce data dimensionality and a number of novel methods have been proposed from statistics, geometry theory and adaptive neural networks. Typical approaches include multidimensional scaling, nonlinear PCA and principal curve/surface. This paper provides an overview on this challenging and emerging topic. It discusses various recent methods such as self-organizing maps, kernel PCA, principal manifold, isomap, local linear embedding, Laplacian eigenmap and spectral clustering, and many of them can be seen as a combined, adaptive learning framework. Their usefulness and potentials will be presented and illustrated in various applications.
Exploring the Manifold of Neural Networks Using Diffusion Geometry

Elliott Abel,Peyton Crevasse,Yvan Grinspan,Selma Mazioud,Folu Ogundipe,Kristof Reimann,Ellie Schueler,Andrew J. Steindl,Ellen Zhang,Dhananjay Bhaskar,Siddharth Viswanath,Yanlei Zhang,Tim G. J. Rudner,Ian Adelstein,Smita Krishnaswamy

2024-11-20

Abstract:Drawing motivation from the manifold hypothesis, which posits that most high-dimensional data lies on or near low-dimensional manifolds, we apply manifold learning to the space of neural networks. We learn manifolds where datapoints are neural networks by introducing a distance between the hidden layer representations of the neural networks. These distances are then fed to the non-linear dimensionality reduction algorithm PHATE to create a manifold of neural networks. We characterize this manifold using features of the representation, including class separation, hierarchical cluster structure, spectral entropy, and topological structure. Our analysis reveals that high-performing networks cluster together in the manifold, displaying consistent embedding patterns across all these features. Finally, we demonstrate the utility of this approach for guiding hyperparameter optimization and neural architecture search by sampling from the manifold.

Machine Learning
Learning High-Dimensional Correspondence Via Manifold Learning and Local Approximation

Chenping Hou,Feiping Nie,Hua Wang,Dongyun Yi,Changshui Zhang

DOI: https://doi.org/10.1007/s00521-013-1369-z

2013-01-01

Abstract:The recent years have witnessed a surge of interests of learning high-dimensional correspondence, which is important for both machine learning and neural computation community. Manifold learning–based researches have been considered as one of the most promising directions. In this paper, by analyzing traditional methods, we summarized a new framework for high-dimensional correspondence learning. Within this framework, we also presented a new approach, Local Approximation Maximum Variance Unfolding. Compared with other machine learning–based methods, it could achieve higher accuracy. Besides, we also introduce how to use the proposed framework and methods in a concrete application, cross-system personalization (CSP). Promising experimental results on image alignment and CSP applications are proposed for demonstration.
The Research of Visualization Data Mining in Industrial Data

Daren Yu,Huixin He,Huixia He,Xiaoxian Zhang,Jinfu Liu

DOI: https://doi.org/10.4028/www.scientific.net/amm.274.545

2013-01-01

Abstract:Based on characteristics of data distribution in the low dimension nature, this paper studies the effect of manifold embedding technology used in typical industrial data set analysis, and manifold learning theory was introduced into the visualization data mining to help get quick effective information in a large number of data. Cognitive process with the help of that information reduces the difficulty of analysis and improves the data utilization efficiency, having a high value in engineering application.

Manifold Learning for Visualizing and Analyzing High-Dimensional Data

Deep Manifold Transformation for Dimension Reduction

Deep Manifold Computing and Visualization

Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness

Deep Manifold Transformation for Nonlinear Dimensionality Reduction

The Research Of Visualization Data Mining Based On Manifold Learning

Manifold-based learning and synthesis.

Analyzing Magnification Factors and Principal Spread Directions in Manifold Learning

Constrained Manifold Learning for Hyperspectral Imagery Visualization

Investigating Manifold Learning Algorithms Based on Magnification Factors and Principal Spread Directions

Manifold learning: what, how, and why

Editorial of the Special Issue on Manifold Learning.

A geometric viewpoint of manifold learning

Visualization of Hyperspectral Imaging Data Based on Manifold Alignment

Overview of Manifold Learning

Interactive Pattern Discovery in High-Dimensional, Multimodal Data Using Manifolds

Adaptive Manifold Learning

Nonlinear Principal Manifolds --- Adaptive Hybrid Learning Approaches

Exploring the Manifold of Neural Networks Using Diffusion Geometry

Learning High-Dimensional Correspondence Via Manifold Learning and Local Approximation

The Research of Visualization Data Mining in Industrial Data