Abstract:While the classical KNN (k nearest neighbor) shares its avoidance of the consistent distribution assumption between training and testing samples to achieve fast prediction, it still faces two challenges: (a) its generalization ability heavily depends on an appropriate number k of nearest neighbors; (b) its prediction behavior lacks interpretability. In order to address the two challenges, a novel Bayes-decisive linear KNN with adaptive nearest neighbors (i.e., BLA-KNN) is proposed to obtain the following three merits: (a) a diagonal matrix is introduced to adaptively select the nearest neighbors and simultaneously improve the generalization capability of the proposed BLA-KNN method; (b) the proposed BLA-KNN method owns the group effect, which inherits and extends the group property of the sum of squares for total deviations by reflecting the training sample class-aware information in the group effect regularization term; (c) the prediction behavior of the proposed BLA-KNN method can be interpreted from the Bayes-decision-rule perspective. In order to do so, we first use a diagonal matrix to weigh each training sample so as to obtain the importance of the sample, while constraining the importance weights to ensure that the adaptive k value is carried out efficiently. Second, we introduce a class-aware information regularization term in the objective function to obtain the nearest neighbor group effect of the samples. Finally, we introduce linear expression weights related to the distance measure between the testing and training samples in the regularization term to ensure that the interpretation of Bayes-decision-rule can be performed smoothly. We also optimize the proposed objective function using an alternating optimization strategy. We experimentally demonstrate the effectiveness of the proposed BLA-KNN method by comparing it with 7 comparative methods on 15 benchmark datasets.

A New k-Nearest Neighbors Algorithm for Learning from Multiple Experts' Uncertain Decisions

RW.KNN: a proposed random walk KNN algorithm for multi-label classification.

Introduction to Machine Learning: K-Nearest Neighbors

Distributionally Robust Weighted $k$-Nearest Neighbors

K-Ns: A Classifier by the Distance to the Nearest Subspace.

Efficient Estimation of k for the Nearest Neighbors Class of Methods

Bayes-Decisive Linear KNN with Adaptive Nearest Neighbors

Ensemble Learning with Extremely Randomized k-Nearest Neighbors for Accurate and Efficient Classification

Under-bagging Nearest Neighbors for Imbalanced Classification

A K-Nearest Neighbor Based Algorithm for Multi-Label Classification.

A Theory-Based Evaluation of Nearest Neighbor Models Put Into Practice

Active and Passive Nearest Neighbor Algorithm: A Newly-Developed Supervised Classifier

Minimax Optimal Algorithms with Fixed-$k$-Nearest Neighbors

On Efficiently Finding Reverse K-Nearest Neighbors over Uncertain Graphs

An Optimal Random Projection k Nearest Neighbors Ensemble via Extended Neighborhood Rule for Binary Classification

Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks

Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning

A New Hashing based Nearest Neighbors Selection Technique for Big Datasets

Knowledge-based system for three-way decision-making under uncertainty

Advanced kNN: A Mature Machine Learning Series

Improved KNN algorithms of spherical regions based on clustering and region division