Abstract:Nonparametric classification models, such as K-Nearest Neighbor (KNN), have become particularly powerful tools in machine learning and data mining, due to their simplicity and flexibility. However, the testing time of the KNN classifier becomes unacceptable and the KNN's performance deteriorates significantly when applied to data sets with millions of dimensions. We observe that state-of-the-art approximate nearest neighbor (ANN) methods aim to either reduce the number of distance comparisons based on tree structure or decrease the cost of distance computation by dimension reduction methods. In this paper, we propose a doubly approximate nearest neighbor classification strategy, which marries the two branches which compress the dimensions for decreasing distance computation cost as well as reduce the number of distance comparison instead of full scan. Under this strategy, we build a compressed dimensional tree (CD-Tree) to avoid unnecessary distance calculations. In each decision node, we propose a novel feature selection paradigm by optimizing the feature selection vector as well as the separator (indicator variables for splitting instances) with the maximum margin. An efficient algorithm is then developed to find the globally optimal solution with convergence guarantee. Furthermore, we also provide a data-dependent generalization error bound for our model, which reveals a new insight for the design of ANN classification algorithms. Our empirical studies show that our algorithm consistently obtains competitive or better classification results on all data sets, yet we can also achieve three orders of magnitude faster than state-of-the-art libraries on very high dimensions.

K-Nearest Neighbors Classifier over Manifolds

A Graph-Based Semi-Supervised K Nearest-Neighbor Method for Nonlinear Manifold Distributed Data Classification.

K-Ns: A Classifier by the Distance to the Nearest Subspace.

Nearest Linear Manifold Classification

Adaptive Neighborhood Selection for Manifold Learning

Doubly Approximate Nearest Neighbor Classification

Locally Centralizing Samples For Nearest Neighbors

Nonparametric Discriminant Multi-Manifold Learning for Dimensionality Reduction

Nearest Neighbor Method Based on Local Distribution for Classification.

Reweighting Recognition Using Modified Kernel Principal Component Analysis Via Manifold Learning

Relative Nearest Neighbors for Classification

Adaptive Neighborhood MinMax Projections

Bayesian Low-Rank and Sparse Nonlinear Representation for Manifold Clustering

Coarse to Fine K Nearest Neighbor Classifier

Local Distribution in Neighborhood for Classification.

Adaptive Neighboring Selection Algorithm Based on Curvature Prediction in Manifold Learning

Discovering the Latent Similarities of the KNN Graph by Metric Transformation

Nonparametric Discriminant Multi-Manifold Learning

A neighborhood selection algorithm for manifold learning

Intrinsic K-means clustering over homogeneous manifolds

Sparse Manifold Learning and Its Applications in Image Classification