Abstract:This paper considers the hyperparameter optimization problem of mathematical techniques that arise in the numerical solution of differential and integral equations. The well-known approaches grid and random search, in a parallel algorithm manner, are developed to find the optimal set of hyperparameters. Employing rational Jacobi functions, we ran these algorithms on two nonlinear benchmark differential equations on the semi-infinite domain. The configurations contain different rational mappings along with their length scale parameter and the Jacobi functions parameters. These trials are configured on the collocation Least-Squares Support Vector Regression (CLS-SVR), a novel numerical simulation approach based on spectral methods. In addition, we have addressed the sensitivity of these hyperparameters on the numerical stability and convergence of the CLS-SVR model. The experiments show that this technique can effectively improve state-of-the-art results.

What problem does this paper attempt to address?

This paper attempts to address the problem of optimizing the hyperparameters of orthogonal functions (such as Jacobi functions) when solving differential equations on semi-infinite domains. Specifically, the authors developed two algorithms (grid search and random search) to find the optimal combination of hyperparameters, thereby improving the accuracy and stability of numerical simulations. These hyperparameters include basis functions, nonlinear rational mappings, and scale parameters. ### Main Research Content 1. **Background and Motivation**: - Differential equations are widely used in physics and engineering, but analytical methods often fail to solve nonlinear problems. - Numerical methods (such as finite element methods, spectral methods, etc.) are used to solve these problems, but selecting appropriate hyperparameters is crucial for the accuracy of the results. - The authors focus on how to optimize these hyperparameters using machine learning techniques, especially in solving differential equations on semi-infinite domains. 2. **Methods**: - **Grid Search**: Finds the optimal set of hyperparameters by traversing all possible combinations. - **Random Search**: Finds the optimal set of hyperparameters by randomly sampling from defined distributions. - Uses **rational Jacobi functions** as basis functions and considers different nonlinear mappings and scale parameters. - Applies the **Constrained Least Squares Support Vector Regression (CLS-SVR)** method to approximate the solution of differential equations. 3. **Experiments and Results**: - The authors selected two nonlinear benchmark differential equations for experiments: the Volterra population model and the Kidder equation. - The effectiveness of the algorithms was evaluated by comparing the absolute errors under different hyperparameter combinations. - The experimental results show that the proposed methods can significantly improve the accuracy of existing methods. ### Key Contributions - **Optimizing Hyperparameters**: Proposed two effective algorithms (grid search and random search) to optimize the hyperparameters in spectral methods. - **Improving Accuracy**: Improved the accuracy and stability of numerical solutions by optimizing hyperparameters. - **Applicability**: The proposed methods are not only applicable to the Volterra population model and the Kidder equation but can also be extended to other mathematical models and problems. ### Conclusion - This study demonstrates the importance and effectiveness of optimizing hyperparameters through machine learning techniques in the numerical solution of differential equations. - The proposed algorithms have advantages in computational complexity and parallel processing, making the method easy for other researchers to use. - Future research can further explore the application of modern gradient-free global optimization techniques (such as Bayesian optimization and tree-structured Parzen estimators) to achieve better approximation results.

Hyperparameter optimization of orthogonal functions in the numerical solution of differential equations

Optimization of Approximate Solution of Integral Equations of Several Variables

Numerical Solution of Higher-Order Linear and Nonlinear Ordinary Differential Equations with Orthogonal Rational Legendre Functions

Global Versus Local Orthogonal Collocation in Simultaneous Approach

A Parallel Function Evaluation Approach for Solution to Large-Scale Equation-Oriented Models.

Random sampling-based automatic parameter tuning for nonlinear programming solvers

Tailored Parameter Optimization Methods for Ordinary Differential Equation Models with Steady-State Constraints

Parallel iterative solvers for discretized reduced optimality systems

Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints

Efficient Hyper-parameter Optimization with Cubic Regularization.

New Methods for Parametric Optimization via Differential Equations

New orthogonal hybrid function based numerical method to solve system of fractional order differential equations

The numerical solution of fractional integral equations via orthogonal polynomials in fractional powers

Piecewise linear approximate solution of fractional order non-stiff and stiff differential-algebraic equations by orthogonal hybrid functions

Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Parallel-in-time Multilevel Krylov Methods: A Prototype

Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization

Hyperopt: a Python library for model selection and hyperparameter optimization

Solve Partial Differential Equations by Two or More Radial Basis Functions