Abstract:Separable nonlinear least squares (SNLLS) problems have attracted interest in a wide range of research fields such as machine learning, computer vision, and signal processing. During the past few decades, several algorithms, including the joint optimization algorithm, alternated least squares (ALS) algorithm, embedded point iterations (EPI) algorithm, and variable projection (VP) algorithms, have been employed for solving SNLLS problems in the literature. The VP approach has been proven to be quite valuable for SNLLS problems and the EPI method has been successful in solving many computer vision tasks. However, no clear explanations about the intrinsic relationships of these algorithms have been provided in the literature. In this paper, we give some insights into these algorithms for SNLLS problems. We derive the relationships among different forms of the VP algorithms, EPI algorithm and ALS algorithm. In addition, the convergence and robustness of some algorithms are investigated. Moreover, the analysis of the VP algorithm generates a negative answer to Kaufman's conjecture. Numerical experiments on the image restoration task, fitting the time series data using the radial basis function network based autoregressive (RBF-AR) model, and bundle adjustment are given to compare the performance of different algorithms.

A novel expectation–maximization-based separable algorithm for parameter identification of RBF-AR model

Least squares support vector machine based partially linear model identification

A nonlinear recursive instrumental variables identification method of Hammerstein ARMAX system

Model selection for RBF-ARX models

A Novel Nonlinear Dynamic System Modeling Approach Using Radial Basis Function Neural Networks

Parameter estimation for a class of radial basis function-based nonlinear time-series models with moving average noises

Parameter Identification of the RBF-ARX Model Based On the Hybrid Whale Optimization Algorithm

Pls Algorithm For Radial Basis Function Networks

Identifiability Implies Robust, Globally Exponentially Convergent On-line Parameter Estimation: Application to Model Reference Adaptive Control

Efficient Parameters Estimation Method for the Separable Nonlinear Least Squares Problem

A NEW IDENTIFICATION APPROACH FOR ARX MODELS

Exploiting the interpretability and forecasting ability of the RBF-AR model for nonlinear time series

Parameter-Covariance Maximum Likelihood Estimation

Regularized EM algorithm for sparse parameter estimation in nonlinear dynamic systems with application to gene regulatory network inference

A novel coupled recursive multivariate nonlinear time-series modelling method by using interactive identification

Recursive Variable Projection Algorithm for a Class of Separable Nonlinear Models

Online Identification of Nonlinear Systems With Separable Structure

Auxiliary model maximum likelihood least squares-based iterative algorithm for multivariable autoregressive output-error autoregressive moving average systems

Variable projection algorithms with sparse constraint for separable nonlinear models

EM Based p-norm-like Constraint RLS Algorithm for Sparse System Identification

Insights Into Algorithms for Separable Nonlinear Least Squares Problems