Abstract:Very few K-nearest-neighbor (KNN) ensembles exist, despite the efficacy of this approach in regression, classification, and outlier detection. Those that do exist focus on bagging features, rather than varying k or bagging observations; it is unknown whether varying k or bagging observations can improve prediction. Given recent studies from topological data analysis, varying k may function like multiscale topological methods, providing stability and better prediction, as well as increased ensemble diversity. This paper explores 7 KNN ensemble algorithms combining bagged features, bagged observations, and varied k to understand how each of these contribute to model fit. Specifically, these algorithms are tested on Tweedie regression problems through simulations and 6 real datasets; results are compared to state-of-the-art machine learning models including extreme learning machines, random forest, boosted regression, and Morse-Smale regression. Results on simulations suggest gains from varying k above and beyond bagging features or samples, as well as the robustness of KNN ensembles to the curse of dimensionality. KNN regression ensembles perform favorably against state-of-the-art algorithms and dramatically improve performance over KNN regression. Further, real dataset results suggest varying k is a good strategy in general (particularly for difficult Tweedie regression problems) and that KNN regression ensembles often outperform state-of-the-art methods. These results for k-varying ensembles echo recent theoretical results in topological data analysis, where multidimensional filter functions and multiscale coverings provide stability and performance gains over single-dimensional filters and single-scale covering. This opens up the possibility of leveraging multiscale neighborhoods and multiple measures of local geometry in ensemble methods.

Tittle : A novel ensemble method for k-nearest neighbor Author order :

A novel ensemble method for k-nearest neighbor.

An Optimal Random Projection k Nearest Neighbors Ensemble via Extended Neighborhood Rule for Binary Classification

New Method of Optimizing K Nearest Neighbor Ensemble for Text-Independent Speaker Recognition

Under-bagging Nearest Neighbors for Imbalanced Classification

Improved KNN Algorithm based on Probability and Adaptive K Value.

Ensemble Learning with Extremely Randomized k-Nearest Neighbors for Accurate and Efficient Classification

K-Ns: A Classifier by the Distance to the Nearest Subspace.

Ensemble Learning and Optimizing KNN Method for Speaker Recognition.

Joint Evidential K-Nearest Neighbor Classification

Ensemble learning based on random super-reduct and resampling

Distributionally Robust Weighted $k$-Nearest Neighbors

KNN Ensembles for Tweedie Regression: The Power of Multiscale Neighborhoods

Adapt Bagging to Nearest Neighbor Classifiers

Adaptive evidential K-NN classification: Integrating neighborhood search and feature weighting

Ensembling Local Learners Through Multimodal Perturbation

A New Evidential K-Nearest Neighbors Data Classification Method

A Representation Coefficient-Based K-Nearest Centroid Neighbor Classifier

A Distributed Rough Evidential K -NN Classifier: Integrating Feature Reduction and Classification

A Distributed Rough Evidential K-NN Classifier: Integrating Feature Reduction and Classification

Eeknn: K-Nearest Neighbor Classifier With An Evidential Editing Procedure For Training Samples