Abstract: In this paper, we consider the problem of stochastic optimization, where the objective function is in terms of the expectation of a (possibly non-convex) cost function that is parametrized by a random variable. While the convergence speed is critical for many emerging applications, most existing stochastic optimization methods suffer from slow convergence. Furthermore, the emerging technology of parallel computing has motivated an increasing demand for designing new stochastic optimization schemes that can handle parallel optimization for implementation in distributed systems. We propose a fast parallel stochastic optimization framework that can solve a large class of possibly non-convex stochastic optimization problems that may arise in applications with multi-agent systems. In the proposed method, each agent updates its control variable in parallel, by solving a convex quadratic subproblem independently. The convergence of the proposed method to the optimal solution for convex problems and to a stationary point for general non-convex problems is established. The proposed algorithm can be applied to solve a large class of optimization problems arising in important applications from various fields, such as machine learning and wireless networks. As a representative application of our proposed stochastic optimization framework, we focus on large-scale support vector machines and demonstrate how our algorithm can efficiently solve this problem, especially in modern applications with huge datasets. Using popular real-world datasets, we present experimental results to demonstrate the merits of our proposed framework by comparing its performance to the state-of-the-art in the literature. Numerical results show that the proposed method can significantly outperform the state-of-the-art methods in terms of the convergence speed while having the same or lower complexity and storage requirement.

Accelerating Stochastic Computation for Binary Classification Applications

Parallel Stochastic Optimization Framework for Large-Scale Non-Convex Stochastic Problems

A novel high performance discrete cosine transform algorithm based on the stochastic computation

Accelerated Information Processing Based on Deep Photonic Time-Delay Reservoir Computing

Design of a Stochastic Computing Architecture for the Phansalkar Algorithm

Parallel Convolutional Neural Network (CNN) Accelerators Based on Stochastic Computing

A Parallel Bitstream Generator for Stochastic Computing

Hybrid Stochastic-Binary Computing for Low-Latency and High-Precision Inference of CNNs

Parallel Hybrid Stochastic-Binary-Based Neural Network Accelerators

Stochastic Computing for Sub-0.2V Applications-Technology Evaluation and Design Optimization

Stochastic Computing Hardware Design and Optimization for Convolutional Neutral Networks

Stochastic-Binary Hybrid Spatial Coding Multiplier for Convolutional Neural Network Accelerator

Optimizing Stochastic Computing for Low Latency Inference of Convolutional Neural Networks

When Sorting Network Meets Parallel Bitstreams: A Fault-Tolerant Parallel Ternary Neural Network Accelerator Based on Stochastic Computing.

An Accurate and Time-Efficient Subtractor by Cross Format Coding in Stochastic Computing

Efficient Parallel Stochastic Computing Multiply-Accumulate (MAC) Technique Using Pseudo-Sobol Bit-Streams

A Scalable Approach to Performing Multiplication and Matrix Dot-Products in Unary

From Stochastic to Bit Stream Computing: Accurate Implementation of Arithmetic Circuits and Applications in Neural Networks

Efficient Non-Linear Adder for Stochastic Computing with Approximate Spatial-Temporal Sorting Network

Towards Budget-Driven Hardware Optimization for Deep Convolutional Neural Networks Using Stochastic Computing

Not Your Father’s Stochastic Computing (SC)! Efficient Yet Accurate End-to-End SC Accelerator Design