Abstract:We introduce a new workflow for unconstrained optimization whereby objective functions are mapped onto a physical domain to more easily design algorithms that are robust to hyperparameters and achieve fast convergence rates. Specifically, we represent optimization problems as an equivalent circuit that are then solved solely as nonlinear circuits using robust solution methods. The equivalent circuit models the trajectory of component-wise scaled gradient flow problem as the transient response of the circuit for which the steady-state coincides with a critical point of the objective function. The equivalent circuit model leverages circuit domain knowledge to methodically design new optimization algorithms that would likely not be developed without a physical model. We incorporate circuit knowledge into optimization methods by 1) enhancing the underlying circuit model for fast numerical analysis, 2) controlling the optimization trajectory by designing the nonlinear circuit components, and 3) solving for step sizes using well-known methods from the circuit simulation. We first establish the necessary conditions that the controls must fulfill for convergence. We show that existing descent algorithms can be re-derived as special cases of this approach and derive new optimization algorithms that are developed with insights from a circuit-based model. The new algorithms can be designed to be robust to hyperparameters, achieve convergence rates comparable or faster than state of the art methods, and are applicable to optimizing a variety of both convex and nonconvex problems.

Circuit Implementation of Proximal Projection Neural Networks for Composite Optimization Problems.

Convergence Analysis and Digital Implementation of a Discrete-Time Neural Network for Model Predictive Control

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Fixed-Time Proximal Gradient Neurodynamic Network With Time-Varying Coefficients for Composite Optimization Problems and Sparse Optimization Problems With Log-Sum Function

Fixed-Time Neurodynamic Optimization Algorithms and Application to Circuits Design

Deep Neural Network-Based Surrogate-Assisted Inverse Optimization for High-Speed Interconnects

A Unified Algorithmic Framework for Distributed Composite Optimization.

An Equivalent Circuit Workflow for Unconstrained Optimization

Performance improvement for the CMOS rail-to-rail amplifier via APSO-based design and SNN's training

CMOS Implementation of ANNs Based on Analog Optimization of N-Dimensional Objective Functions

Method for implementing feedforward neural network by using digital and analog circuits

Analog Circuits for High-Speed Complex Neural-Fuzzy Network

An Inexact Preconditioned Zeroth-order Proximal Method for Composite Optimization

Harnessing Intrinsic Noise in Memristor Hopfield Neural Networks for Combinatorial Optimization

Distributed Proximal Alternating Direction Method of Multipliers for Constrained Composite Optimization Over Directed Networks

Optimization Algorithm Design via Electric Circuits

An Adaptive Regularized Proximal Newton-Type Methods for Composite Optimization over the Stiefel Manifold

Analogue Circuit Realization of a Programmable Sigmoidal Function and Its Derivative for On-Chip BP Learning

Construction and Application of a Neuromorphic Circuit With Excitatory and Inhibitory Post-Synaptic Conduction Channels Implemented Using Dual-Gate Thin-Film Transistors

A Robust Batch Bayesian Optimization for Analog Circuit Synthesis via Local Penalization