Abstract:Complex-valued neural networks (CVNNs) bring in nonholomorphic functions in two ways: (i) through their loss functions and (ii) the widely used activation functions. The derivatives of such functions are defined in Wirtinger calculus. In this paper, we derive two popular algorithms—the gradient descent and the Levenberg-Marquardt (LM) algorithm—for parameter optimization in the feedforward CVNNs using the Wirtinger calculus, which is simpler than the conventional derivation that considers the problem in real domain. While deriving the LM algorithm, we solve and use the result of a least squares problem in the complex domain,$\|\mathbf{b-(Az+Bz^*)}\|_{\underset{\mathbf{z}}{\min}}$, which is more general than the $\|\mathbf{b-Az}\|_{\underset{\mathbf{z}}{\min}}$. Computer simulation results exhibit that as with the real-valued case, the complex-LM algorithm provides much faster learning with higher accuracy than the complex gradient descent algorithm. $|\mathbf{b-(Az+Bz^*)}\|_{\underset{\mathbf{z}}{\min}}$, which is more general than the $\|\mathbf{b-Az}\|_{\underset{\mathbf{z}}{\min}}$. Computer simulation results exhibit that as with the real-valued case, the complex-LM algorithm provides much faster learning with higher accuracy than the complex gradient descent algorithm. $|\mathbf{b-(Az+Bz^*)}\|_{\underset{\mathbf{z}}{\min}}$, which is more general than the $\|\mathbf{b-Az}\|_{\underset{\mathbf{z}}{\min}}$. Computer simulation results exhibit that as with the real-valued case, the complex-LM algorithm provides much faster learning with higher accuracy than the complex gradient descent algorithm. $|\mathbf{b-(Az+Bz^*)}\|_{\underset{\mathbf{z}}{\min}}$, which is more general than the $\|\mathbf{b-Az}\|_{\underset{\mathbf{z}}{\min}}$. Computer simulation results exhibit that as with the real-valued case, the complex-LM algorithm provides much faster learning with higher accuracy than the complex gradient descent algorithm.

Is a Complex-Valued Stepsize Advantageous in Complex-Valued Gradient Learning Algorithms?

Adaptive Complex-Valued Stepsize Based Fast Learning of Complex-Valued Neural Networks.

Adaptive Orthogonal Gradient Descent Algorithm for Fully Complex-Valued Neural Networks.

Adaptive Stepsize Estimation Based Accelerated Gradient Descent Algorithm for Fully Complex-Valued Neural Networks

A complex conjugate gradient training algorithm with Barzilai-Borwein stepsize for complex-valued neural networks

A Hybrid Complex Spectral Conjugate Gradient Learning Algorithm for Complex-Valued Data Processing

Gradient Methods with Adaptive Step-Sizes

Provably Faster Gradient Descent via Long Steps

Learning complexity of gradient descent and conjugate gradient algorithms

New Gradient Methods with Adaptive Stepsizes by Approximate Models

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Complex-valued Neurons Can Learn More but Slower Than Real-valued Neurons Via Gradient Descent.

New Stepsizes for the Gradient Method.

Wirtinger calculus based gradient descent and levenberg-marquardt learning algorithms in complex-valued neural networks

Accelerated Gradient Descent via Long Steps

A Levenberg-Marquardt Algorithm Based Incremental Scheme for Complex-Valued Neural Networks

Towards Statistical and Computational Complexities of Polyak Step Size Gradient Descent

Understanding the unstable convergence of gradient descent.

Accelerated Gradient Descent by Concatenation of Stepsize Schedules

Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency