Abstract:In this paper, we propose the use of a black-box optimization method called deterministic Mesh Adaptive Direct Search (MADS) algorithm with orthogonal directions (Ortho-MADS) for the selection of hyperparameters of Support Vector Machines with a Gaussian kernel. Different from most of the methods in the literature that exploit the properties of the data or attempt to minimize the accuracy of a validation dataset over the first quadrant of (C, gamma), the Ortho-MADS provides convergence proof. We present the MADS, followed by the Ortho-MADS, the dynamic stopping criterion defined by the MADS mesh size and two different search strategies (Nelder-Mead and Variable Neighborhood Search) that contribute to a competitive convergence rate as well as a mechanism to escape from undesired local minima. We have investigated the practical selection of hyperparameters for the Support Vector Machine with a Gaussian kernel, i.e., properly choose the hyperparameters gamma (bandwidth) and C (trade-off) on several benchmark datasets. The experimental results have shown that the proposed approach for hyperparameter tuning consistently finds comparable or better solutions, when using a common configuration, than other methods. We have also evaluated the accuracy and the number of function evaluations of the Ortho-MADS with the Nelder-Mead search strategy and the Variable Neighborhood Search strategy using the mesh size as a stopping criterion, and we have achieved accuracy that no other method for hyperparameters optimization could reach.

Meta-parameter Selection of Support Vector Machines Based on Orthogonal Experiment Design

Novel Linear Search for Support Vector Machine Parameter Selection

SVM KERNEL PARAMETER SELECTION BASED ON IMPROVED GSO

Feature Selection and Parameter Optimization for Support Vector Machines: A New Approach Based on Genetic Algorithm with Feature Chromosomes.

Training support vector data descriptors using converging linear particle swarm optimization

Modified Optimal Foraging Algorithm for Parameters Optimization of Support Vector Machine

A Novel Orthogonal Direction Mesh Adaptive Direct Search Approach for SVM Hyperparameter Tuning

Hybrid optimization of feature selection and SVM training model

Enhancing Support Vector Machine Performance: A Hybrid Approach with Davidon-Fletcher-Powell Algorithm and Elephant Herding Optimization (EHO-DFP) for Parameter Optimization

Parameter Selection for an SVM Based on a Differential Evolution Algorithm

Parameter Selection for Support Vector Machine and Its Application in Structural Optimization

Efficient Model Selection for Support Vector Machine with Gaussian Kernel Function

A Parameter Selection Method of Support Vector Machine Based on Improved FOA

Parameter Optimization of SVM Based on Particle Swarm Optimization Algorithm and GPU Acceleration

Parameter Selection For Gaussian Radial Basis Function In Support Vector Machine Classification

Fast and Efficient Strategies for Model Selection of Gaussian Support Vector Machine

Parameter Selection Method for Svm with Pso

SVM Parameter Selection Algorithm Based on Maximum Kernel Similarity Diversity

Variational Sequential Optimal Experimental Design using Reinforcement Learning

Chaos Optimization Method of SVM Parameters Selection for Short-term Load Forecasting

Parameter Selection of SVM with Gaussian kernel