Abstract:PURPOSE: The small number of samples available for training and testing is often the limiting factor in finding the most effective features and designing an optimal computer-aided diagnosis (CAD) system. Training on a limited set of samples introduces bias and variance in the performance of a CAD system relative to that trained with an infinite sample size. In this work, the authors conducted a simulation study to evaluate the performances of various combinations of classifiers and feature selection techniques and their dependence on the class distribution, dimensionality, and the training sample size. The understanding of these relationships will facilitate development of effective CAD systems under the constraint of limited available samples.METHODS: Three feature selection techniques, the stepwise feature selection (SFS), sequential floating forward search (SFFS), and principal component analysis (PCA), and two commonly used classifiers, Fisher's linear discriminant analysis (LDA) and support vector machine (SVM), were investigated. Samples were drawn from multidimensional feature spaces of multivariate Gaussian distributions with equal or unequal covariance matrices and unequal means, and with equal covariance matrices and unequal means estimated from a clinical data set. Classifier performance was quantified by the area under the receiver operating characteristic curve Az. The mean Az values obtained by resubstitution and hold-out methods were evaluated for training sample sizes ranging from 15 to 100 per class. The number of simulated features available for selection was chosen to be 50, 100, and 200.RESULTS: It was found that the relative performance of the different combinations of classifier and feature selection method depends on the feature space distributions, the dimensionality, and the available training sample sizes. The LDA and SVM with radial kernel performed similarly for most of the conditions evaluated in this study, although the SVM classifier showed a slightly higher hold-out performance than LDA for some conditions and vice versa for other conditions. PCA was comparable to or better than SFS and SFFS for LDA at small samples sizes, but inferior for SVM with polynomial kernel. For the class distributions simulated from clinical data, PCA did not show advantages over the other two feature selection methods. Under this condition, the SVM with radial kernel performed better than the LDA when few training samples were available, while LDA performed better when a large number of training samples were available.CONCLUSIONS: None of the investigated feature selection-classifier combinations provided consistently superior performance under the studied conditions for different sample sizes and feature space distributions. In general, the SFFS method was comparable to the SFS method while PCA may have an advantage for Gaussian feature spaces with unequal covariance matrices. The performance of the SVM with radial kernel was better than, or comparable to, that of the SVM with polynomial kernel under most conditions studied.

The effect of unlabeled samples in reducing the small sample size problem and mitigating the Hughes phenomenon

Mitigating Hughes Phenomenon: Improving Hyperspectral Imaging Classification Through Active Learning for Generalization Enhancement

Blind Spectral Unmixing Considering the Adjacent Effect

Undersampling is a Minimax Optimal Robustness Intervention in Nonparametric Classification

Small sample classification of hyperspectral image using model-agnostic meta-learning algorithm and convolutional neural network

Hyperspectral Image Classification Based on Semisupervised Self-Learning and Multiview Information Fusion

Enhancing Semi-Supervised Few-Shot Hyperspectral Image Classification via Progressive Sample Selection

Revisiting Sample Size Determination in Natural Language Understanding

Dimension Reduction Aided Hyperspectral Image Classification with a Small-sized Training Dataset: Experimental Comparisons

Identifying Mislabeled Data using the Area Under the Margin Ranking

Importance of Disjoint Sampling in Conventional and Transformer Models for Hyperspectral Image Classification

Addressing the multi-label imbalance for neural networks: An approach based on stratified mini-batches

The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration

Advances in Hyperspectral Image Classification Methods with Small Samples: A Review

Light-Weight Self-Supervised Contrastive Learning Network for Small Sample Hyperspectral Image Classification

A Low-Measurement-Cost-Based Multi-Strategy Hyperspectral Image Classification Scheme

Depthwise Separable Relation Network for Small Sample Hyperspectral Image Classification

Small Sample Classification of Hyperspectral Remote Sensing Images Based on Sequential Joint Deeping Learning Model

Online Dictionary Self-Taught Learning for Hyperspectral Image Classification.

Effect of finite sample size on feature selection and classification: A simulation study

Iterative Training Sample Augmentation for Enhancing Land Cover Change Detection Performance With Deep Learning Neural Network