Abstract:Programs involving discontinuities introduced by control flow constructs such as conditional branches pose challenges to mathematical optimization methods that assume a degree of smoothness in the objective function's response surface. Smooth interpretation (SI) is a form of abstract interpretation that approximates the convolution of a program's output with a Gaussian kernel, thus smoothing its output in a principled manner. Here, we combine SI with automatic differentiation (AD) to efficiently compute gradients of smoothed programs. In contrast to AD across a regular program execution, these gradients also capture the effects of alternative control flow paths. The combination of SI with AD enables the direct gradient-based parameter synthesis for branching programs, allowing for instance the calibration of simulation models or their combination with neural network models in machine learning pipelines. We detail the effects of the approximations made for tractability in SI and propose a novel Monte Carlo estimator that avoids the underlying assumptions by estimating the smoothed programs' gradients through a combination of AD and sampling. Using DiscoGrad, our tool for automatically translating simple C++ programs to a smooth differentiable form, we perform an extensive evaluation. We compare the combination of SI with AD and our Monte Carlo estimator to existing gradient-free and stochastic methods on four non-trivial and originally discontinuous problems ranging from classical simulation-based optimization to neural network-driven control. While the optimization progress with the SI-based estimator depends on the complexity of the program's control flow, our Monte Carlo estimator is competitive in all problems, exhibiting the fastest convergence by a substantial margin in our highest-dimensional problem.

Smoothing the Nonsmoothness

Characterization of Smoothness of Multivariate Refinable Functions and Convergence of Cascade Algorithms of Nonhomogeneous Refinement Equations

Surface Interpolation Scheme by Distance Blending over Convex Sets

A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems

Coordinate descent methods beyond smoothness and separability

Abstract convex approximations of nonsmooth functions

A Smoothing-Type Algorithm for Solving System of Inequalities

Approximation and Smoothing of Functions Based on Godunov Regularization

An accurate and efficient scheme for function extensions on smooth domains

A UNIFORM ENHANCEMENT APPROACH FOR OPTIMIZATION ALGORITHMS: SMOOTHING FUNCTION METHOD

On the role of semismoothness in nonsmooth numerical analysis: Theory

Newton Type Methods for Solving Nonsmooth Equations

Smoothing Methods for Automatic Differentiation Across Conditional Branches

Fast inertial dynamic algorithm with smoothing method for nonsmooth convex optimization

Nonsmooth, Nonconvex Optimization Using Functional Encoding and Component Transition Information

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

Numerical Differentiation from A Viewpoint of Regularization Theory

A Smoothing Stochastic Gradient Method for Composite Optimization

Constrained Global Optimization by Smoothing

Constructing Approximations to Bivariate Piecewise-Smooth Functions