Abstract:In data science, individual observations are often assumed to come independently from an underlying probability space. Kernel matrices formed from large sets of such observations arise frequently, for example during classification tasks. It is desirable to know the eigenvalue decay properties of these matrices without explicitly forming them, such as when determining if a low-rank approximation is feasible. In this work, we introduce a new eigenvalue quantile estimation framework for some kernel matrices. This framework gives meaningful bounds for all the eigenvalues of a kernel matrix while avoiding the cost of constructing the full matrix. The kernel matrices under consideration come from a kernel with quick decay away from the diagonal applied to uniformly-distributed sets of points in Euclidean space of any dimension. We prove the efficacy of this framework given certain bounds on the kernel function, and we provide empirical evidence for its accuracy. In the process, we also prove a very general interlacing-type theorem for finite sets of numbers. Additionally, we indicate an application of this framework to the study of the intrinsic dimension of data, as well as several other directions in which to generalize this work.
What problem does this paper attempt to address?
### The problems the paper attempts to solve
The paper "Fast Spectrum Estimation of Some Kernel Matrices" aims to solve the problem of how to quickly estimate the eigenvalue distribution of some kernel matrices without explicitly constructing the kernel matrices. Specifically, the author proposes a new eigenvalue quantile estimation framework, which can provide effective bounds for all eigenvalues of the kernel matrix in sub - quadratic time (i.e., the time complexity is lower than \(O(n^2)\)).
### Background and motivation
In data science, it is usually assumed that the observed data independently come from some underlying probability space. When dealing with a large amount of such observed data, kernel matrices are often formed, for example, in classification tasks. Understanding the eigenvalue decay properties of these kernel matrices is very important, especially when determining whether a low - rank approximation can be used. However, directly constructing and computing the eigenvalues of large - scale kernel matrices is usually infeasible because it requires a large amount of computing resources.
### Limitations of existing methods
1. **Asymptotic analysis**: Previous studies mainly focused on the asymptotic behavior of the eigenvalues of kernel matrices when the number of samples tends to infinity. These methods rely on assumptions about the distribution and the kernel function, and require truncated eigenvalue decomposition, which is often impractical in practical applications.
2. **Matrix sketching techniques**: Some empirical methods, such as matrix sketching techniques, can estimate eigenvalues through random sampling. But these methods usually need to construct the complete kernel matrix first, and the computational complexity is at least \(O(n^2)\).
3. **Nyström method**: The Nyström method is a matrix sketching technique that does not require constructing a complete kernel matrix and obtains a low - rank decomposition by randomly sampling points. However, this method does not work well when the kernel matrix has a high numerical rank.
### Advantages of the new method
The new method proposed by the author estimates the eigenvalue distribution of a large - scale kernel matrix by matching the moments of a small matrix. Specifically, this method:
1. **Select a small matrix**: Randomly select \(k\) points from the original data points to form a \(k\times k\) small - scale kernel matrix \(B\).
2. **Match moments**: Estimate the eigenvalue distribution of \(A\) by matching the first \(k\) moments of \(B\) and the large - scale kernel matrix \(A\).
3. **Fast computation**: The time complexity of this method is \(O(mk^2)\), where \(m\) is a constant depending on the required approximation accuracy.
### Main contributions
1. **New eigenvalue estimation framework**: Propose a new framework based on eigenvalue quantile estimation, which is suitable for kernel functions with fast decay characteristics.
2. **Generalized interlacing theorem**: Prove a generalized interlacing theorem about a finite set of real numbers, which is a completely new result.
3. **Application prospects**: Propose an application of this method to the estimation of the intrinsic dimension of data.
### Conclusion
The paper solves the problem of quickly estimating the eigenvalue distribution without explicitly constructing the kernel matrix by proposing a new eigenvalue estimation framework. This method not only provides new insights in theory, but also has potential value in practical applications.