Abstract:Nonparametric classification models, such as K-Nearest Neighbor (KNN), have become particularly powerful tools in machine learning and data mining, due to their simplicity and flexibility. However, the testing time of the KNN classifier becomes unacceptable and the KNN's performance deteriorates significantly when applied to data sets with millions of dimensions. We observe that state-of-the-art approximate nearest neighbor (ANN) methods aim to either reduce the number of distance comparisons based on tree structure or decrease the cost of distance computation by dimension reduction methods. In this paper, we propose a doubly approximate nearest neighbor classification strategy, which marries the two branches which compress the dimensions for decreasing distance computation cost as well as reduce the number of distance comparison instead of full scan. Under this strategy, we build a compressed dimensional tree (CD-Tree) to avoid unnecessary distance calculations. In each decision node, we propose a novel feature selection paradigm by optimizing the feature selection vector as well as the separator (indicator variables for splitting instances) with the maximum margin. An efficient algorithm is then developed to find the globally optimal solution with convergence guarantee. Furthermore, we also provide a data-dependent generalization error bound for our model, which reveals a new insight for the design of ANN classification algorithms. Our empirical studies show that our algorithm consistently obtains competitive or better classification results on all data sets, yet we can also achieve three orders of magnitude faster than state-of-the-art libraries on very high dimensions.

Theoretical Analysis of the Confidence Metrics for Nearest Neighbor Classifier

An Improvement To The Nearest Neighbor Classifier And Face Recognition Experiments

A theoretical analysis on nearest feature space classifier

Evaluation and application of recognition confidence in OCR

AN IMPROVEMENT TO THE NEAREST NEIGHBOR CLASSIFIER AND F ACE RECOGNITION EXPERIMENTS

Researches on Algorithm for Confidence Evaluation and Decision Modification of Svm

The Nearest Neighbor Algorithm of Local Probability Centers

Multi-hypothesis nearest-neighbor classifier based on class-conditional weighted distance metric.

Confidence Analysis of Target Recognition

In defense of Nearest-Neighbor based image classification

Nearest-neighbor Classifier Motivated Marginal Discriminant Projections for Face Recognition

Face Recognition Using Two Novel Nearest Neighbor Classifiers

Perceptual Nearest Neighbors for Classification

Interpreting Neural Networks With Nearest Neighbors

On the Consistency of Exact and Approximate Nearest Neighbor with Noisy Data.

Doubly Approximate Nearest Neighbor Classification

A Local Mean Representation-based K-Nearest Neighbor Classifier

An Efficient Nearest Neighbor Classifier Algorithm Based on Pre-classify

ICNN: an Inter-class Nearest Neighbor Method for Face Recognition

On high-dimensional modifications of the nearest neighbor classifier

Nearest Feature Space Analysis for Classification