Abstract:Feature selection (FS) is an important technique in machine learning and data mining, especially for large-scale high-dimensional data. Most existing studies have been restricted to batch learning, which is often inefficient and poorly scalable when handling big data in real world. As real data may arrive sequentially and continuously, batch learning has to retrain the model for the new coming data, which is very computationally intensive. Online feature selection (OFS) is a promising new paradigm that is more efficient and scalable than batch learning algorithms. However, existing online algorithms usually fall short in their inferior efficacy. In this article, we present a novel second-order OFS algorithm that is simple yet effective, very fast and extremely scalable to deal with large-scale ultra-high dimensional sparse data streams. The basic idea is to exploit the second-order information to choose the subset of important features with high confidence weights. Unlike existing OFS methods that often suffer from extra high computational cost, we devise a novel algorithm with a MaxHeap-based approach, which is not only more effective than the existing first-order algorithms, but also significantly more efficient and scalable. Our extensive experiments validated that the proposed technique achieves highly competitive accuracy as compared with state-of-the-art batch FS methods, meanwhile it consumes significantly less computational cost that is orders of magnitude lower. Impressively, on a billion-scale synthetic dataset (1-billion dimensions, 1-billion non-zero features, and 1-million samples), the proposed algorithm takes less than 3 minutes to run on a single PC.

Orthogonal Floating Search Algorithms: From The Perspective of Nonlinear System Identification

Regularized orthogonal forward feature selection for spectral data

To obtain orthogonal feature extraction using training data selection.

A Modified Sequential Deep Floating Search Algorithm For Feature Selection

Nonlinear System Identification Using Orthogonal Wavelet Networks

Kernel-OPBS Algorithm: A Nonlinear Feature Selection Method for Hyperspectral Imagery

Two-Dimensional (2D) Particle Swarms for Structure Selection of Nonlinear Systems

A NOVEL METHOD FOR IDENTIFYING NONLINEAR SYSTEMS USING ORTHOGONAL WAVELET NETWORKS

LOFS: Library of Online Streaming Feature Selection

Orthogonal variance decomposition based feature selection

Large-Scale Online Feature Selection for Ultra-High Dimensional Sparse Data.

Unsupervised feature selection via dual space-based low redundancy scores and extended OLSDA

A Forward Selection Reverse Path Method for Spatial Location Identification of Nonlinearities in MDOF Systems

Supervised Feature Selection With Orthogonal Regression and Mountain Range Area

Unsupervised feature selection using orthogonal encoder-decoder factorization

Multi-objective evolutionary framework for non-linear system identification: A comprehensive investigation

Low-Redundant Unsupervised Feature Selection based on Data Structure Learning and Feature Orthogonalization

Boosting Open-Set RF Signal Recognition under Low SNR Condition in Feature Space

Online Feature Selection with Group Structure Analysis

Unsupervised Feature Selection via Nonnegative Orthogonal Constrained Regularized Minimization

An Efficient Parallel Reptile Search Algorithm and Snake Optimizer Approach for Feature Selection