A survey on spectral embeddings and their applications in data analysis

David Tewodrose

2020-11-19

Abstract:The aim of this survey is to present some aspects of the Bérard-Besson-Gallot spectral embeddings of a closed Riemannian manifold from their origins in Riemannian geometry to more recent applications in data analysis.

Differential Geometry

What problem does this paper attempt to address?

Imagerank : Spectral Techniques for Structural Analysis of Image Database

XF He,WY Ma,HJ Zhang

DOI: https://doi.org/10.1109/icme.2003.1220845

2003-01-01

Abstract:Drawing on the correspondence between spectral clustering, spectral dimensionality reduction, and the connections to the Markov chain theory, we present a novel unified framework for structural analysis of image database using spectral techniques. The framework provides a computationally efficient approach to both clustering and dimensionality reduction, or 2-D visualization. Within this framework, we can also infer the semantic degrees of the images, i.e. imagerank, which characterize the richness of semantics contained in the images. Some illustrative examples are discussed.
Geometrical embeddings for distributions into algebras of generalized functions

Shantanu Dave

DOI: https://doi.org/10.48550/arXiv.0709.2039

2007-09-13

Abstract:We use spectral theory to produce embeddings of distributions in the algebras of generalized functions on a closed Riemannian manifold. These embeddings are invariant under isometries and preserve the singularity structure of the distributions.

Analysis of PDEs
A Survey on Spectral Graph Neural Networks

Deyu Bo,Xiao Wang,Yang Liu,Yuan Fang,Yawen Li,Chuan Shi

DOI: https://doi.org/10.48550/arXiv.2302.05631

IF: 5.414

2023-02-11

Machine Learning

Abstract:Graph neural networks (GNNs) have attracted considerable attention from the research community. It is well established that GNNs are usually roughly divided into spatial and spectral methods. Despite that spectral GNNs play an important role in both graph signal processing and graph representation learning, existing studies are biased toward spatial approaches, and there is no comprehensive review on spectral GNNs so far. In this paper, we summarize the recent development of spectral GNNs, including model, theory, and application. Specifically, we first discuss the connection between spatial GNNs and spectral GNNs, which shows that spectral GNNs can capture global information and have better expressiveness and interpretability. Next, we categorize existing spectral GNNs according to the spectrum information they use, \ie, eigenvalues or eigenvectors. In addition, we review major theoretical results and applications of spectral GNNs, followed by a quantitative experiment to benchmark some popular spectral GNNs. Finally, we conclude the paper with some future directions.
From Varadhan's Limit to Eigenmaps: A Guide to the Geometric Analysis behind Manifold Learning

Chen-Yun Lin,Christina Sormani

DOI: https://doi.org/10.48550/arXiv.2210.10405

2022-10-19

Differential Geometry

Abstract:We present an overview of the history of the heat kernel and eigenfunctions on Riemannian manifolds and how the theory has lead to modern methods of analyzing high dimensional data via eigenmaps and other spectral embeddings. We begin with Varadhan's Theorem relating the heat kernel to the distance function on a Riemannian manifold. We then review various theorems which bound the heat kernel on classes of Riemannian manifolds. Next we turn to eigenfunctions, the Sturm-Liouville Decomposition of the heat kernel using eigenfunctions, and various theorems which bound eigenfunctions on classes of Riemannian manifolds. We review various notions of convergence of Riemannian manifolds and which classes of Riemannian manifolds are compact with respect to which notions of convergence. We then present B\'erard-Besson-Gallot's heat kernel embeddings of Riemannian manifolds and the truncation of those embeddings. Finally we turn to Applications of Spectral embeddings to the Dimension Reduction of data sets lying in high dimensional spaces reviewing, in particular, the work of Belkin-Niyogi and Coifman-Lafon. We also review the Spectral Theory of Graphs and the work of Dodziuk and Chung and others. We close with recent theorems of Portegies and of the first author controlling truncated spectral embeddings uniformly on key classes of Riemannian manifolds. Throughout we provide many explicitly computed examples and graphics and attempt to provide as complete a set of references as possible. We hope that this article is accessible to both pure and applied mathematicians working in Geometric Analysis and their doctoral students.
Spectral Methods for Data Science: A Statistical Perspective

Yuxin Chen,Yuejie Chi,Jianqing Fan,Cong Ma

DOI: https://doi.org/10.1561/2200000079

2021-01-01

Foundations and Trends® in Machine Learning

Abstract:Spectral methods have emerged as a simple yet surprisingly effective approach for extracting information from massive, noisy and incomplete data. In a nutshell, spectral methods refer to a collection of algorithms built upon the eigenvalues (resp. singular values) and eigenvectors (resp. singular vectors) of some properly designed matrices constructed from data. A diverse array of applications have been found in machine learning, imaging science, financial and econometric modeling, and signal processing, including recommendation systems, community detection, ranking, structured matrix recovery, tensor data estimation, joint shape matching, blind deconvolution, financial investments, risk managements, treatment evaluations, causal inference, amongst others. Due to their simplicity and effectiveness, spectral methods are not only used as a stand-alone estimator, but also frequently employed to facilitate other more sophisticated algorithms to enhance performance. While the studies of spectral methods can be traced back to classical matrix perturbation theory and the method of moments, the past decade has witnessed tremendous theoretical advances in demystifying their efficacy through the lens of statistical modeling, with the aid of concentration inequalities and non-asymptotic random matrix theory. This monograph aims to present a systematic, comprehensive, yet accessible introduction to spectral methods from a modern statistical perspective, highlighting their algorithmic implications in diverse large-scale applications. In particular, our exposition gravitates around several central questions that span various applications: how to characterize the sample efficiency of spectral methods in reaching a target level of statistical accuracy, and how to assess their stability in the face of random noise, missing data, and adversarial corruptions? In addition to conventional ℓ2 perturbation analysis, we present a systematic ℓ∞ and ℓ2,∞ perturbation theory for eigenspace and singular subspaces, which has only recently become available owing to a powerful “leave-one-out” analysis framework.
On the Schoenberg Transformations in Data Analysis: Theory and Illustrations

François Bavaud

DOI: https://doi.org/10.48550/arXiv.1004.0089

2010-04-02

Abstract:The class of Schoenberg transformations, embedding Euclidean distances into higher dimensional Euclidean spaces, is presented, and derived from theorems on positive definite and conditionally negative definite matrices. Original results on the arc lengths, angles and curvature of the transformations are proposed, and visualized on artificial data sets by classical multidimensional scaling. A simple distance-based discriminant algorithm illustrates the theory, intimately connected to the Gaussian kernels of Machine Learning.

Machine Learning
Geometric Methods for Spherical Data, with Applications to Cosmology

Javier Carrón Duque,Domenico Marinucci

2023-03-27

Abstract:This survey is devoted to recent developments in the statistical analysis of spherical data, with a view to applications in Cosmology. We will start from a brief discussion of Cosmological questions and motivations, arguing that most Cosmological observables are spherical random fields. Then, we will introduce some mathematical background on spherical random fields, including spectral representations and the construction of needlet and wavelet frames. We will then focus on some specific issues, including tools and algorithms for map reconstruction (\textit{i.e.}, separating the different physical components which contribute to the observed field), geometric tools for testing the assumptions of Gaussianity and isotropy, and multiple testing methods to detect contamination in the field due to point sources. Although these tools are introduced in the Cosmological context, they can be applied to other situations dealing with spherical data. Finally, we will discuss more recent and challenging issues such as the analysis of polarization data, which can be viewed as realizations of random fields taking values in spin fiber bundles.

Cosmology and Nongalactic Astrophysics,Statistics Theory
Spectral regression: a regression framework for efficient regularized subspace learning

Jiawei Han,Deng Cai

2009-01-01

Abstract:Spectral methods have recently emerged as a powerful tool for dimensionality reduction and manifold learning. These methods use information contained in the eigenvectors of a data affinity (i.e., item-item similarity) matrix to reveal the low dimensional structure in the high dimensional data. The most popular manifold learning algorithms include Locally Linear Embedding, ISOMAP, and Laplacian Eigenmap. However, these algorithms only provide the embedding results of training samples. There are many extensions of these approaches which try to solve the out-of-sample extension problem by seeking an embedding function in reproducing kernel Hilbert space. However, a disadvantage of all these approaches is that their computations usually involve eigen-decomposition of dense matrices which is expensive in both time and memory. In this thesis, we introduce a novel dimensionality reduction framework, called Spectral Regression (SR). SR casts the problem of learning an embedding function into a regression framework, which avoids eigen-decomposition of dense matrices. Also, with the regression as a building block, different kinds of regularizers can be naturally incorporated into our framework which makes it more flexible. SR can be performed in supervised, unsupervised and semi-supervised situation. It can make efficient use of both labeled and unlabeled points to discover the intrinsic discriminant structure in the data. We have applied our algorithms to several real world applications, e.g. face analysis, document representation and content-based image retrieval.
A tutorial on the dynamic Laplacian

Gary Froyland

2024-08-08

Abstract:Spectral techniques are popular and robust approaches to data analysis. A prominent example is the use of eigenvectors of a Laplacian, constructed from data affinities, to identify natural data groupings or clusters, or to produce a simplified representation of data lying on a manifold. This tutorial concerns the dynamic Laplacian, which is a natural generalisation of the Laplacian to handle data that has a time component and lies on a time-evolving manifold. In this dynamic setting, clusters correspond to long-lived ``coherent'' collections. We begin with a gentle recap of spectral geometry before describing the dynamic generalisations. We also discuss computational methods and the automatic separation of many distinct features through the SEBA algorithm. The purpose of this tutorial is to bring together many results from the dynamic Laplacian literature into a single short document, written in an accessible style.

Dynamical Systems,Spectral Theory,Machine Learning
Statistical embedding: Beyond principal components

Dag Tjøstheim,Martin Jullum,Anders Løland

DOI: https://doi.org/10.48550/arXiv.2106.01858

2022-08-31

Abstract:There has been an intense recent activity in embedding of very high dimensional and nonlinear data structures, much of it in the data science and machine learning literature. We survey this activity in four parts. In the first part we cover nonlinear methods such as principal curves, multidimensional scaling, local linear methods, ISOMAP, graph based methods and diffusion mapping, kernel based methods and random projections. The second part is concerned with topological embedding methods, in particular mapping topological properties into persistence diagrams and the Mapper algorithm. Another type of data sets with a tremendous growth is very high-dimensional network data. The task considered in part three is how to embed such data in a vector space of moderate dimension to make the data amenable to traditional techniques such as cluster and classification techniques. Arguably this is the part where the contrast between algorithmic machine learning methods and statistical modeling, the so-called stochastic block modeling, is at its greatest. In the paper, we discuss the pros and cons for the two approaches. The final part of the survey deals with embedding in $\mathbb{R}^ 2$, i.e. visualization. Three methods are presented: $t$-SNE, UMAP and LargeVis based on methods in parts one, two and three, respectively. The methods are illustrated and compared on two simulated data sets; one consisting of a triplet of noisy Ranunculoid curves, and one consisting of networks of increasing complexity generated with stochastic block models and with two types of nodes.

Machine Learning,Methodology
Spectral embedding of directed networks

Q. Zheng,D. B. Skillicorn

DOI: https://doi.org/10.1007/s13278-016-0387-0

2016-09-10

Social Network Analysis and Mining

Abstract:Most relationships in a social network are asymmetric: The strength of A’s relationship to B is not the same as the strength of B’s relationship to A. Such relationships can reflect asymmetric emotional bonds, influence or power. It is natural to model such social networks by directed graphs, with a node for each participant, and a weighted directed edge for each relationship. Spectral embeddings for directed graphs are known, but they have significant weaknesses. We design a new directed graph embedding, prove that it has desirable mathematical properties, and demonstrate its application to both synthetic and real-world networks. The advantages of the new technique are that it avoids the weaknesses of existing techniques, it models the net flow across each node (the extent to which its upstream community is distinct from its downstream community), and it enables directed edge prediction (which so-far-unobserved edges are most likely to exist and which direction and intensity they will have).
The spectral geometry of hyperbolic and spherical manifolds: analogies and open problems

Emilio A. Lauret,Benjamin Linowitz

2024-05-12

Abstract:The spectral geometry of negatively curved manifolds has received more attention than its positive curvature counterpart. In this paper we will survey a variety of spectral geometry results that are known to hold in the context of hyperbolic manifolds and discuss the extent to which analogous results hold in the setting of spherical manifolds. We conclude with a number of open problems.

Differential Geometry
Survey on graph embeddings and their applications to machine learning problems on graphs

Ilya Makarov,Dmitrii Kiselev,Nikita Nikitinsky,Lovro Subelj

DOI: https://doi.org/10.7717/peerj-cs.357

2021-02-04

Abstract:Dealing with relational data always required significant computational resources, domain expertise and task-dependent feature engineering to incorporate structural information into a predictive model. Nowadays, a family of automated graph feature engineering techniques has been proposed in different streams of literature. So-called graph embeddings provide a powerful tool to construct vectorized feature spaces for graphs and their components, such as nodes, edges and subgraphs under preserving inner graph properties. Using the constructed feature spaces, many machine learning problems on graphs can be solved via standard frameworks suitable for vectorized feature representation. Our survey aims to describe the core concepts of graph embeddings and provide several taxonomies for their description. First, we start with the methodological approach and extract three types of graph embedding models based on matrix factorization, random-walks and deep learning approaches. Next, we describe how different types of networks impact the ability of models to incorporate structural and attributed data into a unified embedding. Going further, we perform a thorough evaluation of graph embedding applications to machine learning problems on graphs, among which are node classification, link prediction, clustering, visualization, compression, and a family of the whole graph embedding algorithms suitable for graph classification, similarity and alignment problems. Finally, we overview the existing applications of graph embeddings to computer science domains, formulate open problems and provide experiment results, explaining how different networks properties result in graph embeddings quality in the four classic machine learning problems on graphs, such as node classification, link prediction, clustering and graph visualization. As a result, our survey covers a new rapidly growing field of network feature engineering, presents an in-depth analysis of models based on network types, and overviews a wide range of applications to machine learning problems on graphs.
Spectral 3D Computer Vision -- A Review

Yajie Sun,Ali Zia,Vivien Rolland,Charissa Yu,Jun Zhou

DOI: https://doi.org/10.48550/arXiv.2302.08054

2023-02-16

Abstract:Spectral 3D computer vision examines both the geometric and spectral properties of objects. It provides a deeper understanding of an object's physical properties by providing information from narrow bands in various regions of the electromagnetic spectrum. Mapping the spectral information onto the 3D model reveals changes in the spectra-structure space or enhances 3D representations with properties such as reflectance, chromatic aberration, and varying defocus blur. This emerging paradigm advances traditional computer vision and opens new avenues of research in 3D structure, depth estimation, motion analysis, and more. It has found applications in areas such as smart agriculture, environment monitoring, building inspection, geological exploration, and digital cultural heritage records. This survey offers a comprehensive overview of spectral 3D computer vision, including a unified taxonomy of methods, key application areas, and future challenges and prospects.

Computer Vision and Pattern Recognition,Artificial Intelligence
A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications

Hongyun Cai,Vincent W. Zheng,Kevin Chen-Chuan Chang

2018-02-02

Abstract:Graph is an important data representation which appears in a wide diversity of real-world scenarios. Effective graph analytics provides users a deeper understanding of what is behind the data, and thus can benefit a lot of useful applications such as node classification, node recommendation, link prediction, etc. However, most graph analytics methods suffer the high computation and space cost. Graph embedding is an effective yet efficient way to solve the graph analytics problem. It converts the graph data into a low dimensional space in which the graph structural information and graph properties are maximally preserved. In this survey, we conduct a comprehensive review of the literature in graph embedding. We first introduce the formal definition of graph embedding as well as the related concepts. After that, we propose two taxonomies of graph embedding which correspond to what challenges exist in different graph embedding problem settings and how the existing work address these challenges in their solutions. Finally, we summarize the applications that graph embedding enables and suggest four promising future research directions in terms of computation efficiency, problem settings, techniques and application scenarios.

Artificial Intelligence
A Survey on Graph Embedding Techniques for Biomedical Data: Methods and Applications

Yaozu Wu,Yankai Chen,Zhishuai Yin,Weiping Ding,Irwin King

DOI: https://doi.org/10.1016/j.inffus.2023.101909

IF: 18.6

2023-01-01

Information Fusion

Abstract:As a result of the expeditious advancement of biomedical technologies, a plethora of relational data linking biomedical entities such as genes, proteins, and drugs have been collected for modern biomedical research. Biomedical graphs, one of the most popular ways to represent relational data, can easily describe different complex biomedical systems, including molecular-level, multi-omics-level, therapeutics-level, and healthcare-level interactions. However, traditional graph analysis methods still suffer from two difficulties (i.e., heterogeneities and dynamic properties of biomedical graphs) when handling high-dimensional, multi-modal, and sparsely interconnected biomedical data. To address these issues, graph embedding methods that can effectively analyze biomedical graphs have received a significant amount of attention recently. Generally, graph-based data is converted into a low-dimensional vector space with its structural properties and well-reserved information. These vectorized representations are used for further computation in various downstream biomedical tasks, such as gene function prediction and drug–target interaction prediction. In this article, we focus on the application of graph data in the biomedical domain and mainly introduce recent developments of graph embedding techniques (homogeneous, heterogeneous, and dynamic graph embedding techniques), including methodologies and related biomedical tasks. We also summarize relevant biomedical datasets and open-source implementations. We further discuss existing limitations and potential solutions. We hope this survey can provide useful directions for researchers who are interested in using graph embedding methods to solve problems in the biomedical field.
Spectral Inference Methods on Sparse Graphs: Theory and Applications

Alaa Saade

DOI: https://doi.org/10.48550/arXiv.1610.04337

2016-10-14

Abstract:In an era of unprecedented deluge of (mostly unstructured) data, graphs are proving more and more useful, across the sciences, as a flexible abstraction to capture complex relationships between complex objects. One of the main challenges arising in the study of such networks is the inference of macroscopic, large-scale properties affecting a large number of objects, based solely on the microscopic interactions between their elementary constituents. Statistical physics, precisely created to recover the macroscopic laws of thermodynamics from an idealized model of interacting particles, provides significant insight to tackle such complex networks. In this dissertation, we use methods derived from the statistical physics of disordered systems to design and study new algorithms for inference on graphs. Our focus is on spectral methods, based on certain eigenvectors of carefully chosen matrices, and sparse graphs, containing only a small amount of information. We develop an original theory of spectral inference based on a relaxation of various mean-field free energy optimizations. Our approach is therefore fully probabilistic, and contrasts with more traditional motivations based on the optimization of a cost function. We illustrate the efficiency of our approach on various problems, including community detection, randomized similarity-based clustering, and matrix completion.

Disordered Systems and Neural Networks,Information Theory,Machine Learning
Spectral Algorithms on Manifolds through Diffusion

Weichun Xia,Lei Shi

2024-03-07

Abstract:The existing research on spectral algorithms, applied within a Reproducing Kernel Hilbert Space (RKHS), has primarily focused on general kernel functions, often neglecting the inherent structure of the input feature space. Our paper introduces a new perspective, asserting that input data are situated within a low-dimensional manifold embedded in a higher-dimensional Euclidean space. We study the convergence performance of spectral algorithms in the RKHSs, specifically those generated by the heat kernels, known as diffusion spaces. Incorporating the manifold structure of the input, we employ integral operator techniques to derive tight convergence upper bounds concerning generalized norms, which indicates that the estimators converge to the target function in strong sense, entailing the simultaneous convergence of the function itself and its derivatives. These bounds offer two significant advantages: firstly, they are exclusively contingent on the intrinsic dimension of the input manifolds, thereby providing a more focused analysis. Secondly, they enable the efficient derivation of convergence rates for derivatives of any k-th order, all of which can be accomplished within the ambit of the same spectral algorithms. Furthermore, we establish minimax lower bounds to demonstrate the asymptotic optimality of these conclusions in specific contexts. Our study confirms that the spectral algorithms are practically significant in the broader context of high-dimensional approximation.

Machine Learning
Diffusion-based Semi-supervised Spectral Algorithm for Regression on Manifolds

Weichun Xia,Jiaxin Jiang,Lei Shi

2024-10-18

Abstract:We introduce a novel diffusion-based spectral algorithm to tackle regression analysis on high-dimensional data, particularly data embedded within lower-dimensional manifolds. Traditional spectral algorithms often fall short in such contexts, primarily due to the reliance on predetermined kernel functions, which inadequately address the complex structures inherent in manifold-based data. By employing graph Laplacian approximation, our method uses the local estimation property of heat kernel, offering an adaptive, data-driven approach to overcome this obstacle. Another distinct advantage of our algorithm lies in its semi-supervised learning framework, enabling it to fully use the additional unlabeled data. This ability enhances the performance by allowing the algorithm to dig the spectrum and curvature of the data manifold, providing a more comprehensive understanding of the dataset. Moreover, our algorithm performs in an entirely data-driven manner, operating directly within the intrinsic manifold structure of the data, without requiring any predefined manifold information. We provide a convergence analysis of our algorithm. Our findings reveal that the algorithm achieves a convergence rate that depends solely on the intrinsic dimension of the underlying manifold, thereby avoiding the curse of dimensionality associated with the higher ambient dimension.

Machine Learning
Spatial-Spectral Manifold Embedding of Hyperspectral Data

Danfeng Hong,Jing Yao,Xin Wu,Jocelyn Chanussot,Xiao Xiang Zhu

DOI: https://doi.org/10.48550/arXiv.2007.08767

2020-07-17

Abstract:In recent years, hyperspectral imaging, also known as imaging spectroscopy, has been paid an increasing interest in geoscience and remote sensing community. Hyperspectral imagery is characterized by very rich spectral information, which enables us to recognize the materials of interest lying on the surface of the Earth more easier. We have to admit, however, that high spectral dimension inevitably brings some drawbacks, such as expensive data storage and transmission, information redundancy, etc. Therefore, to reduce the spectral dimensionality effectively and learn more discriminative spectral low-dimensional embedding, in this paper we propose a novel hyperspectral embedding approach by simultaneously considering spatial and spectral information, called spatial-spectral manifold embedding (SSME). Beyond the pixel-wise spectral embedding approaches, SSME models the spatial and spectral information jointly in a patch-based fashion. SSME not only learns the spectral embedding by using the adjacency matrix obtained by similarity measurement between spectral signatures, but also models the spatial neighbours of a target pixel in hyperspectral scene by sharing the same weights (or edges) in the process of learning embedding. Classification is explored as a potential strategy to quantitatively evaluate the performance of learned embedding representations. Classification is explored as a potential application for quantitatively evaluating the performance of these hyperspectral embedding algorithms. Extensive experiments conducted on the widely-used hyperspectral datasets demonstrate the superiority and effectiveness of the proposed SSME as compared to several state-of-the-art embedding methods.

Computer Vision and Pattern Recognition

A survey on spectral embeddings and their applications in data analysis

Imagerank : Spectral Techniques for Structural Analysis of Image Database

Geometrical embeddings for distributions into algebras of generalized functions

A Survey on Spectral Graph Neural Networks

From Varadhan's Limit to Eigenmaps: A Guide to the Geometric Analysis behind Manifold Learning

Spectral Methods for Data Science: A Statistical Perspective

On the Schoenberg Transformations in Data Analysis: Theory and Illustrations

Geometric Methods for Spherical Data, with Applications to Cosmology

Spectral regression: a regression framework for efficient regularized subspace learning

A tutorial on the dynamic Laplacian

Statistical embedding: Beyond principal components

Spectral embedding of directed networks

The spectral geometry of hyperbolic and spherical manifolds: analogies and open problems

Survey on graph embeddings and their applications to machine learning problems on graphs

Spectral 3D Computer Vision -- A Review

A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications

A Survey on Graph Embedding Techniques for Biomedical Data: Methods and Applications

Spectral Inference Methods on Sparse Graphs: Theory and Applications

Spectral Algorithms on Manifolds through Diffusion

Diffusion-based Semi-supervised Spectral Algorithm for Regression on Manifolds

Spatial-Spectral Manifold Embedding of Hyperspectral Data