Abstract:A device‐system end‐to‐end design through reinforcement learning is demonstrated to optimize a free‐space optical general matrix multiplication hardware accelerator by engineering a spatially reconfigurable array made from chalcogenide phase change materials without human supervision and domain knowledge. Learning results show a clear device‐system correlation, and optimized hardware can be deployed into a variety of machine learning tasks through physics‐aware training. The incorporation of high‐performance optoelectronic devices into photonic neuromorphic processors can substantially accelerate computationally intensive matrix multiplication operations in machine learning (ML) algorithms. However, the conventional designs of individual devices and system are largely disconnected, and the system optimization is limited to the manual exploration of a small design space. Here, a device‐system end‐to‐end design methodology is reported to optimize a free‐space optical general matrix multiplication (GEMM) hardware accelerator by engineering a spatially reconfigurable array made from chalcogenide phase change materials. With a highly parallelized integrated hardware emulator with experimental information, the design of unit device to directly optimize GEMM calculation accuracy is achieved by exploring a large parameter space through reinforcement learning algorithms, including deep Q‐learning neural network, Bayesian optimization, and their cascaded approach. The algorithm‐generated physical quantities show a clear correlation between system performance metrics and device specifications. Furthermore, physics‐aware training approaches are employed to deploy optimized hardware to the tasks of image classification, materials discovery, and a closed‐loop design of optical ML accelerators. The demonstrated framework offers insights into the end‐to‐end and co‐design of optoelectronic devices and systems with reduced human supervision and domain knowledge barriers.

Reinforcement Learning in a large scale photonic Recurrent Neural Network

Large-scale photonic computing with nonlinear disordered media

Emergent self-adaptation in an integrated photonic neural network for backpropagation-free learning

A complete, parallel and autonomous photonic neural network in a semiconductor multimode laser

Experimentally realized in situ backpropagation for deep learning in photonic neural networks

Photonic neuromorphic architecture for tens-of-task lifelong learning

Photonic reinforcement learning based on optoelectronic reservoir computing

Deep Learning with Photonic Neural Cellular Automata

All-optical spiking neurosynaptic networks with self-learning capabilities

Device‐System End‐to‐End Design of Photonic Neuromorphic Processor Using Reinforcement Learning

Optical training of large-scale Transformers and deep neural networks with direct feedback alignment

Massively scalable wavelength diverse integrated photonic linear neuron

Tutorial: Photonic Neural Networks in Delay Systems

Photonic extreme learning machine by free-space optical propagation

Nonlinear unitary circuits for photonic neural networks

Event-driven adaptive optical neural network

Training Deep Photonic Convolutional Neural Networks With Sinusoidal Activations

Reinforcement learning enabled the design of compact and efficient integrated photonic devices

Photonic neural field on a silicon chip: large-scale, high-speed neuro-inspired computing and sensing

Spatially Varying Nanophotonic Neural Networks

Optical Rectifying Linear Units for Back-Propagation Learning in a Deep Holographic Convolutional Neural Network