Abstract:A device‐system end‐to‐end design through reinforcement learning is demonstrated to optimize a free‐space optical general matrix multiplication hardware accelerator by engineering a spatially reconfigurable array made from chalcogenide phase change materials without human supervision and domain knowledge. Learning results show a clear device‐system correlation, and optimized hardware can be deployed into a variety of machine learning tasks through physics‐aware training. The incorporation of high‐performance optoelectronic devices into photonic neuromorphic processors can substantially accelerate computationally intensive matrix multiplication operations in machine learning (ML) algorithms. However, the conventional designs of individual devices and system are largely disconnected, and the system optimization is limited to the manual exploration of a small design space. Here, a device‐system end‐to‐end design methodology is reported to optimize a free‐space optical general matrix multiplication (GEMM) hardware accelerator by engineering a spatially reconfigurable array made from chalcogenide phase change materials. With a highly parallelized integrated hardware emulator with experimental information, the design of unit device to directly optimize GEMM calculation accuracy is achieved by exploring a large parameter space through reinforcement learning algorithms, including deep Q‐learning neural network, Bayesian optimization, and their cascaded approach. The algorithm‐generated physical quantities show a clear correlation between system performance metrics and device specifications. Furthermore, physics‐aware training approaches are employed to deploy optimized hardware to the tasks of image classification, materials discovery, and a closed‐loop design of optical ML accelerators. The demonstrated framework offers insights into the end‐to‐end and co‐design of optoelectronic devices and systems with reduced human supervision and domain knowledge barriers.

Perfectly Perform Machine Learning Task with Imperfect Optical Hardware Accelerator

Countering Variations and Thermal Effects for Accurate Optical Neural Networks.

Device-system Co-design of Photonic Neuromorphic Processor using Reinforcement Learning

Artificial Intelligence Accelerators based on Graphene Optoelectronic Devices

Optical training of large-scale Transformers and deep neural networks with direct feedback alignment

Parallel Photonic Acceleration Processor for Matrix-Matrix Multiplication

Device‐System End‐to‐End Design of Photonic Neuromorphic Processor Using Reinforcement Learning

Improving Scalability of Mach-Zehnder Interferometer Based Photonic Computers via Algorithm-Hardware Codesign

Nonlinear Schrödinger Kernel for Hardware Acceleration of Machine Learning

PIXEL: Photonic Neural Network Accelerator

Deep Learning with Passive Optical Nonlinear Mapping

OPIMA: Optical Processing-In-Memory for Convolutional Neural Network Acceleration

Optical Multi-Imaging-Casting Accelerator for Fully Parallel Universal Convolution Computing

11 Tera-OPs/s Photonic Convolutional Accelerator and Deep Optical Neural Network Based on an Integrated Kerr Soliton Crystal Microcomb

High Throughput Multi-Channel Parallelized Diffraction Convolutional Neural Network Accelerator

Diffraction‐Driven Parallel Convolution Processing with Integrated Photonics

High‐Throughput Multichannel Parallelized Diffraction Convolutional Neural Network Accelerator

11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks

Compact optical convolution processing unit based on multimode interference

M3ICRO: Machine Learning-Enabled Compact Photonic Tensor Core based on PRogrammable Multi-Operand Multimode Interference

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms