Complex Tensor Convolution in Photonic Frequency Synthetic Dimensions

Jiayuan Guo,Wenjia Zhang,Yue Jiang,Zuyuan He

DOI: https://doi.org/10.1364/cleo_at.2024.jth2a.79

2024-01-01

Abstract:We propose a complex tensor convolution accelerator in photonic frequency synthetic dimensions, which is capable of extracting coupling information from dual-frame images of size 52×42 at a rate of 4.94M frame-per-second.

What problem does this paper attempt to address?

Fully-integrated photonic tensor core for image convolutions

Bicky A Marquez,Jagmeet Singh,Hugh Morison,Zhimu Guo,Lukas Chrostowski,Sudip Shekhar,Paul Prucnal,Bhavin J Shastri

DOI: https://doi.org/10.1088/1361-6528/acde83

IF: 3.5

2023-07-13

Nanotechnology

Abstract:Convolutions are one of the most critical signal and image processing operations. From spectral analysis to computer vision, convolutional filtering is often related to spatial information processing involving neighbourhood operations. As convolution operations are based around the product of two functions, vectors or matrices, dot products play a key role in the performance of such operations; for example, advanced image processing techniques require fast, dense matrix multiplications that typically take more than 90% of the computational capacity dedicated to solving convolutional neural networks. Silicon photonics has been demonstrated to be an ideal candidate to accelerate information processing involving parallel matrix multiplications. In this work, we experimentally demonstrate a multiwavelength approach with fully integrated modulators, tunable filters as microring resonator weight banks, and a balanced detector to perform matrix multiplications for image convolution operations. We develop a scattering matrix model that matches the experiment to simulate large-scale versions of these photonic systems with which we predict performance and physical constraints, including inter-channel cross-talk and bit resolution.
Experimentally Realizing Convolution Processing in the Photonic Synthetic Frequency Dimension

Lingling Fan,Kai Wang,Heming Wang,Avik Dutt,Shanhui Fan

2023-08-11

Abstract:Convolution is an essential operation in signal and image processing and consumes most of the computing power in convolutional neural networks. Photonic convolution has the promise of addressing computational bottlenecks and outperforming electronic implementations. Performing photonic convolution in the synthetic frequency dimension, which harnesses the dynamics of light in the spectral degrees of freedom for photons, can lead to highly compact devices. Here we experimentally realize convolution operations in the synthetic frequency dimension. Using a modulated ring resonator, we synthesize arbitrary convolution kernels using a pre-determined modulation waveform with high accuracy. We demonstrate the convolution computation between input frequency combs and synthesized kernels. We also introduce the idea of an additive offset to broaden the kinds of kernels that can be implemented experimentally when the modulation strength is limited. Our work demonstrate the use of synthetic frequency dimension to efficiently encode data and implement computation tasks, leading to a compact and scalable photonic computation architecture.

Optics,Signal Processing
Parallel Photonic Acceleration Processor for Matrix-Matrix Multiplication

Ying Huang,Hengsong Yue,Wei Ma,Yiyuan Zhang,Yao Xiao,Yong Tang,He Tang,Tao Chu

DOI: https://doi.org/10.1364/ol.488464

IF: 3.6

2023-01-01

Optics Letters

Abstract:We propose and experimentally demonstrate a highly parallel photonic acceleration processor based on a wavelength division multiplexing (WDM) system and a non-coherent Mach–Zehnder interferometer (MZI) array for matrix–matrix multiplication. The dimensional expansion is achieved by WDM devices, which play a crucial role in realizing matrix–matrix multiplication together with the broadband characteristics of an MZI. We implemented a 2 × 2 arbitrary nonnegative valued matrix using a reconfigurable 8 × 8 MZI array structure. Through experimentation, we verified that this structure could achieve 90.5% inference accuracy in a classification task for the Modified National Institute of Standards and Technology (MNIST) handwritten dataset. This provides a new effective solution for large-scale integrated optical computing systems based on convolution acceleration processors.
PhotoFourier: A Photonic Joint Transform Correlator-Based Neural Network Accelerator

Shurui Li,Hangbo Yang,Chee Wei Wong,Volker J. Sorger,Puneet Gupta

DOI: https://doi.org/10.48550/arXiv.2211.05276

2022-11-10

Hardware Architecture

Abstract:The last few years have seen a lot of work to address the challenge of low-latency and high-throughput convolutional neural network inference. Integrated photonics has the potential to dramatically accelerate neural networks because of its low-latency nature. Combined with the concept of Joint Transform Correlator (JTC), the computationally expensive convolution functions can be computed instantaneously (time of flight of light) with almost no cost. This 'free' convolution computation provides the theoretical basis of the proposed PhotoFourier JTC-based CNN accelerator. PhotoFourier addresses a myriad of challenges posed by on-chip photonic computing in the Fourier domain including 1D lenses and high-cost optoelectronic conversions. The proposed PhotoFourier accelerator achieves more than 28X better energy-delay product compared to state-of-art photonic neural network accelerators.
Parallel convolutional processing using an integrated photonic tensor core

J. Feldmann,N. Youngblood,M. Karpov,H. Gehring,X. Li,M. Stappers,M. Le Gallo,X. Fu,A. Lukashchuk,A. S. Raja,J. Liu,C. D. Wright,A. Sebastian,T. J. Kippenberg,W. H. P. Pernice,H. Bhaskaran

DOI: https://doi.org/10.1038/s41586-020-03070-1

IF: 64.8

2021-01-06

Nature

Abstract:With the proliferation of ultrahigh-speed mobile networks and internet-connected devices, along with the rise of artificial intelligence (AI)<a href="/articles/s41586-020-03070-1#ref-CR1">1</a>, the world is generating exponentially increasing amounts of data that need to be processed in a fast and efficient way. Highly parallelized, fast and scalable hardware is therefore becoming progressively more important<a href="/articles/s41586-020-03070-1#ref-CR2">2</a>. Here we demonstrate a computationally specific integrated photonic hardware accelerator (tensor core) that is capable of operating at speeds of trillions of multiply-accumulate operations per second (1012 MAC operations per second or tera-MACs per second). The tensor core can be considered as the optical analogue of an application-specific integrated circuit (ASIC). It achieves parallelized photonic in-memory computing using phase-change-material memory arrays and photonic chip-based optical frequency combs (soliton microcombs<a href="/articles/s41586-020-03070-1#ref-CR3">3</a>). The computation is reduced to measuring the optical transmission of reconfigurable and non-resonant passive components and can operate at a bandwidth exceeding 14 gigahertz, limited only by the speed of the modulators and photodetectors. Given recent advances in hybrid integration of soliton microcombs at microwave line rates<a href="#ref-CR3">3</a>,<a href="#ref-CR4">4</a>,<a href="/articles/s41586-020-03070-1#ref-CR5">5</a>, ultralow-loss silicon nitride waveguides<a href="/articles/s41586-020-03070-1#ref-CR6">6</a>,<a href="/articles/s41586-020-03070-1#ref-CR7">7</a>, and high-speed on-chip detectors and modulators, our approach provides a path towards full complementary metal–oxide–semiconductor (CMOS) wafer-scale integration of the photonic tensor core. Although we focus on convolutional processing, more generally our results indicate the potential of integrated photonics for parallel, fast, and efficient computational hardware in data-heavy AI applications such as autonomous driving, live video processing, and next-generation cloud computing services.

multidisciplinary sciences
High-order Tensor Flow Processing Using Integrated Photonic Circuits

Shaofu Xu,Jing Wang,Sicheng Yi,Weiwen Zou

DOI: https://doi.org/10.1038/s41467-022-35723-2

IF: 16.6

2022-01-01

Nature Communications

Abstract:Tensor analytics lays the mathematical basis for the prosperous promotion of multiway signal processing. To increase computing throughput, mainstream processors transform tensor convolutions into matrix multiplications to enhance the parallelism of computing. However, such order-reducing transformation produces data duplicates and consumes additional memory. Here, we propose an integrated photonic tensor flow processor (PTFP) without digitally duplicating the input data. It outputs the convolved tensor as the input tensor ‘flows’ through the processor. The hybrid manipulation of optical wavelengths, space dimensions, and time delay steps, enables the direct representation and processing of high-order tensors in the optical domain. In the proof-of-concept experiment, an integrated processor manipulating wavelengths and delay steps is implemented for demonstrating the key functionalities of PTFP. The multi-channel images and videos are processed at the modulation rate of 20 Gbaud. A convolutional neural network for video action recognition is demonstrated on the processor, which achieves an accuracy of 97.9%.
Photonic tensor processing unit with single dataflow and programmable high-precision weighting control

Kaifei Tang,Jiantao Wang,Wenqu Xu,Xiang Ji,Jiahui Liu,Xiaobin Huang,Yu Xin,Pan Dai,Guozhu Sun,Zhaobang Zeng,Rulei Xiao,Xiangfei Chen,Wei Jiang

DOI: https://doi.org/10.1109/jlt.2023.3317090

IF: 4.7

2023-01-01

Journal of Lightwave Technology

Abstract:With the explosive growth of data, tensor processing has emerged as a pivotal component of the next generation of artificial intelligence (AI) algorithms. Current photonic convolutional processors transform tensor convolutions into multi-channel general matrix multiplication (GeMM), which follows the path of electronic counterparts, leading to data replication and hardware complexity. In this study, we experimentally and theoretically demonstrate a photonic tensor processing unit (PTPU) with a single modulator, which offers a more concise approach for multi-channel standard tensor convolution processing, different from the channel-wise convolution method. By executing multi-tensor parallel computing instead of multi-channel parallel computing, PTPU can directly produce feature tensors without clock synchronization and delay compensation between multiple channels and allows more release of physical hardware resources. Furthermore, an integrated array of semiconductor optical amplifiers (SOAs) are used to be photonic synapses for programmable weight bank, demonstrating record-high precision of 9.2 bits for weights. In the proof-of-concept experiment, we extracted features from a 3-channel (RGB) image in horizontal and vertical directions, using integrated multi-wavelength photonic tensor kernels. We then built a 3D convolutional neural network to predict the presence of COVID-19 based on computer tomography (CT) scan data consisting of 64-channel tensors.

engineering, electrical & electronic,optics,telecommunications
Higher-dimensional processing using a photonic tensor core with continuous-time data

Bowei Dong,Samarth Aggarwal,Wen Zhou,Utku Emre Ali,Nikolaos Farmakidis,June Sang Lee,Yuhan He,Xuan Li,Dim-Lee Kwong,C. D. Wright,Wolfram H. P. Pernice,H. Bhaskaran

DOI: https://doi.org/10.1038/s41566-023-01313-x

IF: 35

2023-10-20

Nature Photonics

Abstract:New developments in hardware-based 'accelerators' range from electronic tensor cores and memristor-based arrays to photonic implementations. The goal of these approaches is to handle the exponentially growing computational load of machine learning, which currently requires the doubling of hardware capability approximately every 3.5 months. One solution is increasing the data dimensionality that is processable by such hardware. Although two-dimensional data processing by multiplexing space and wavelength has been previously reported, the use of three-dimensional processing has not yet been implemented in hardware. In this paper, we introduce the radio-frequency modulation of photonic signals to increase parallelization, adding an additional dimension to the data alongside spatially distributed non-volatile memories and wavelength multiplexing. We leverage higher-dimensional processing to configure such a system to an architecture compatible with edge computing frameworks. Our system achieves a parallelism of 100, two orders higher than implementations using only the spatial and wavelength degrees of freedom. We demonstrate this by performing a synchronous convolution of 100 clinical electrocardiogram signals from patients with cardiovascular diseases, and constructing a convolutional neural network capable of identifying patients at sudden death risk with 93.5% accuracy.

optics,physics, applied
Integrated Photonic Fractional Convolution Accelerator

Kevin Zelaya,Mohammad-Ali Miri

2024-01-02

Abstract:An integrated photonic circuit architecture to perform a modified-convolution operation based on the Discrete Fractional Fourier Transform (DFrFT) is introduced. This is accomplished by utilizing two nonuniformly-coupled waveguide lattices with equally-spaced eigenmode spectra and with different lengths that perform DFrDT operations of complementary orders sandwiching a modulator array. Numerical simulations show that smoothing and edge detection tasks are indeed performed even for noisy input signals.

Optics,Quantum Physics
Direct Optical Convolution Computing Based on Arrayed Waveguide Grating Router

Jialin Cheng,Chong Li,Jun Dai,Yayan Chu,Xinxiang Niu,Xiaowen Dong,Jian-Jun He

DOI: https://doi.org/10.1002/lpor.202301221

2024-01-01

LASER & PHOTONICS REVIEWS

Abstract:Optical convolution computing is gaining traction owing to its inherent parallelism, multi-dimensional processing, and energy efficiency. To handle input dimensions of N, conventional implementations necessitate N2 optical elements, such as Mach-Zehnder interferometers or micro-ring resonators, to process multiply-accumulate (MAC) operations, limiting scalability and resulting in elevated power consumption. Here, a direct convolution computing method based on wavelength routing, utilizing the unique sliding property of an arrayed waveguide grating router (AWGR) to perform the sliding window operation of the convolution in the wavelength-space domains is proposed. With two input vectors directly loaded onto two modulator arrays, the convolution result is instantaneously produced at a photodetector array. The entire convolution computation is executed within a single clock cycle without the need for preprocessing or decomposition into elementary MAC operations. The number of active elements is minimal, only needed for input/output. The proposed optical convolution unit has striking advantages of high scalability, high speed, and processing simplicity compared to those based on optical matrix-vector multipliers. In the first experimental demonstration, a remarkable classification accuracy of up to 98.2% in handwritten digit recognition tasks using a LeNet-5 neural network is achieved. A direct convolution computing method based on wavelength routing, utilizing the unique sliding property of an arrayed waveguide grating router (AWGR) to perform the sliding window operation of the convolution in the wavelength-space domains is proposed and demonstrated. With two inputs vectors directly loaded onto modulator arrays, the convolution result is instantaneously produced at a photodetector array. image
Time-Wavelength Interleaved Photonic Convolution Accelerator on TFLN Platform

Lin Wang,Yang Gao,Lei Zhang,Kun Yin,Hui Yu

DOI: https://doi.org/10.1364/fio.2024.jd4a.74

2024-01-01

Abstract:We demonstrate a time and wavelength interleaving convolutional accelerator based on TFLN platform, which demonstrates the potential for optimal computational speeds through parallelism and high-speed modulation.
All-optical complex field imaging using diffractive processors

Jingxi Li,Yuhang Li,Tianyi Gan,Che-Yung Shen,Mona Jarrahi,Aydogan Ozcan

2024-01-30

Abstract:Complex field imaging, which captures both the amplitude and phase information of input optical fields or objects, can offer rich structural insights into samples, such as their absorption and refractive index distributions. However, conventional image sensors are intensity-based and inherently lack the capability to directly measure the phase distribution of a field. This limitation can be overcome using interferometric or holographic methods, often supplemented by iterative phase retrieval algorithms, leading to a considerable increase in hardware complexity and computational demand. Here, we present a complex field imager design that enables snapshot imaging of both the amplitude and quantitative phase information of input fields using an intensity-based sensor array without any digital processing. Our design utilizes successive deep learning-optimized diffractive surfaces that are structured to collectively modulate the input complex field, forming two independent imaging channels that perform amplitude-to-amplitude and phase-to-intensity transformations between the input and output planes within a compact optical design, axially spanning ~100 wavelengths. The intensity distributions of the output fields at these two channels on the sensor plane directly correspond to the amplitude and quantitative phase profiles of the input complex field, eliminating the need for any digital image reconstruction algorithms. We experimentally validated the efficacy of our complex field diffractive imager designs through 3D-printed prototypes operating at the terahertz spectrum, with the output amplitude and phase channel images closely aligning with our numerical simulations. We envision that this complex field imager will have various applications in security, biomedical imaging, sensing and material science, among others.

Optics,Computer Vision and Pattern Recognition,Applied Physics
Microcomb-driven Photonic Convolutional Engine

Bowen Bai,Qipeng Yang,Haowen Shu,Lin Chang,Yichen Wu,Fenghe Yang,Bitao Shen,Zihan Tao,Weiqiang Xie,Weiwei Hu,John E. Bowers,Xingjun Wang

DOI: https://doi.org/10.1364/cleo_si.2023.sf1o.2

2023-01-01

Abstract:We demonstrate a microcomb-driven chip-based photonic convolutional engine utilizing time-wavelength plane stretching approach. A preeminent photonic-core compute density of 1.04 TOPS/mm 2 and a record high weight control precision of 9 bits are achieved.
Integrated Photonic FFT for Optical Convolutions towards Efficient and High-Speed Neural Networks

Moustafa Ahmed,Yas Al-Hadeethi,Ahmed Bakry,Hamed Dalir,Volker J. Sorger

DOI: https://doi.org/10.48550/arXiv.2002.01308

2020-02-01

Optics

Abstract:The technologically-relevant task of feature extraction from data performed in deep-learning systems is routinely accomplished as repeated fast Fourier transforms (FFT) electronically in prevalent domain-specific architectures such as in graphics processing units (GPUs). However, electronics systems are limited with respect to power dissipation and delay, both, due to wire-charging challenges related to interconnect capacitance. Here we present a silicon photonics-based architecture for convolutional neural networks that harnesses the phase property of light to perform FFTs efficiently by executing the convolution as a multiplication in the Fourier-domain. The algorithmic executing time is determined by the time-of-flight of the signal through this photonic reconfigurable passive FFT filter circuit and is on the order of 10s of picosecond. A sensitivity analysis shows that this optical processor must be thermally phase stabilized corresponding to a few degrees. Furthermore, we find that for a small sample number, the obtainable number of convolutions per {time-power-chip area) outperforms GPUs by about 2 orders of magnitude. Lastly, we show that, conceptually, the optical FFT and convolution-processing performance is indeed directly linked to optoelectronic device-level, and improvements in plasmonics, metamaterials or nanophotonics are fueling next generation densely interconnected intelligent photonic circuits with relevance for edge-computing 5G networks.
Photonic convolution accelerator based on a hybrid integrated multi-wavelength laser array by photonic wire bonding for real-time image classification

Yuxin Ma,Kaifei Tang,Jun Lu,Zhenxing Sun,Lu Yao,Yujiang Wang,Zhenzhen Xu,Mi Li,Xin Wang,Rulei Xiao,Yu Xin,Jiahui Liu,Xiang Ji,Wei Jiang,Yuechun Shi,Xiangfei Chen

DOI: https://doi.org/10.1364/ol.518837

IF: 3.6

2024-05-08

Optics Letters

Abstract:Yuxin Ma, Kaifei Tang, Jun Lu, Zhenxing Sun, Lu Yao, Yujiang Wang, Zhenzhen Xu, Mi Li, Xin Wang, Rulei Xiao, Yu Xin, Jiahui Liu, Xiang Ji, Wei Jiang, Yuechun Shi, Xiangfei Chen We propose and experimentally demonstrate a compact and efficient photonic convolution accelerator based on a hybrid integrated ... [Opt. Lett. 49, 2629-2632 (2024)]

optics
Exploiting k-space - frequency duality in Fourier optics towards real-time compression-less terahertz imaging

Hichem Guerboukha,Kathirvel Nallappan,Maksim Skorobogatiy

DOI: https://doi.org/10.48550/arXiv.1708.07528

2017-08-25

Abstract:We present theoretical formulation and experimental demonstration of a novel technique for the fast compression-less terahertz imaging based on the broadband Fourier optics. The technique exploits k-vector/frequency duality in Fourier optics which allows using a single-pixel detector to perform angular scan along a circular path, while the broadband spectrum is used to scan along the radial dimension in Fourier domain. The proposed compression-less image reconstruction technique (hybrid inverse transform) requires only a small number of measurements that scales linearly with the image linear size, thus promising real-time acquisition of high-resolution THz images. Additionally, our imaging technique handles equally well and on the equal theoretical footing the amplitude contrast and the phase contrast images, which makes this technique useful for many practical applications. A detailed analysis of the novel technique advantages and limitations is presented, as well as its place among other existing THz imaging techniques is clearly identified

Optics
All-optical complex-valued convolution based on four-wave mixing

Wentao Gu,Xiaoyan Gao,Wenchan Dong,Yilun Wang,Hailong Zhou,Jing Xu,Xinliang Zhang

DOI: https://doi.org/10.1364/optica.495053

IF: 10.4

2024-01-11

Optica

Abstract:Wentao Gu, Xiaoyan Gao, Wenchan Dong, Yilun Wang, Hailong Zhou, Jing Xu, Xinliang Zhang Optical complex-valued convolution can extract the feature of complex-valued data by processing both amplitude and phase information, ... [Optica 11, 64-72 (2024)]

optics
Photonic Convolution Neural Network Based on Interleaved Time-Wavelength Modulation

Yue Jiang,Wenjia Zhang,Fan Yang,Zuyuan He

DOI: https://doi.org/10.1109/jlt.2021.3076070

IF: 4.7

2021-07-01

Journal of Lightwave Technology

Abstract:Convolution neural network (CNN), as one of the most powerful and popular technologies, has achieved remarkable progress for image and video classification since its invention. However, with the high definition video-data explosion, convolution layers in the CNN architecture will occupy a great amount of computing time and memory resources due to high computation complexity of matrix multiply accumulate operation. In this paper, a novel integrated photonic CNN is proposed based on double correlation operations through interleaved time-wavelength modulation. Micro-ring based multi-wavelength manipulation and single dispersion medium are utilized to realize convolution operation and replace the conventional optical delay lines. 200 images are tested in MNIST datasets with accuracy of 85.5$\%$ in our photonic CNN versus 86.5$\%$ in 64-bit computer. We also analyze the computing error of photonic CNN caused by various micro-ring parameters, operation baud rates and the characteristics of micro-ring weighting bank. Furthermore, a tensor processing unit based on 44 mesh with 1.2 Tera operation per second (TOPS) computing capability at 20G baud rate is proposed and analyzed to form a paralleled photonic CNN.

engineering, electrical & electronic,optics,telecommunications
High Throughput Multi-Channel Parallelized Diffraction Convolutional Neural Network Accelerator

Zibo Hu,Shurui Li,Russell L.T. Schwartz,Maria Solyanik-Gorgone,Mario Miscuglio,Puneet Gupta,Volker J. Sorger

DOI: https://doi.org/10.48550/arXiv.2112.12297

2022-07-08

Abstract:Convolutional neural networks are paramount in image and signal processing including the relevant classification and training tasks alike and constitute for the majority of machine learning compute demand today. With convolution operations being computationally intensive, next generation hardware accelerators need to offer parallelization and algorithmic-hardware homomorphism. Fortunately, diffractive display optics is capable of million-channel parallel data processing at low latency, however, thus far only showed tens of Hertz slow single image and kernel capability, thereby significantly underdelivering from its performance potential. Here, we demonstrate an operation-parallelized high-throughput Fourier optic convolutional neural network accelerator. For the first time simultaneously processing of multiple kernels in Fourier domain enabled by optical diffraction has been achieved alongside with already conventional in the field input parallelism. Additionally, we show an about one hundred times system speed up over existing optical diffraction-based processors and this demonstration rivals performance of modern electronic solutions. Therefore, this system is capable of processing large-scale matrices about ten times faster than state of art electronic systems.

Machine Learning,Optics

Complex Tensor Convolution in Photonic Frequency Synthetic Dimensions

Fully-integrated photonic tensor core for image convolutions

Experimentally Realizing Convolution Processing in the Photonic Synthetic Frequency Dimension

Parallel Photonic Acceleration Processor for Matrix-Matrix Multiplication

PhotoFourier: A Photonic Joint Transform Correlator-Based Neural Network Accelerator

Parallel convolutional processing using an integrated photonic tensor core

High-order Tensor Flow Processing Using Integrated Photonic Circuits

Photonic tensor processing unit with single dataflow and programmable high-precision weighting control

Higher-dimensional processing using a photonic tensor core with continuous-time data

Integrated Photonic Fractional Convolution Accelerator

Direct Optical Convolution Computing Based on Arrayed Waveguide Grating Router

Time-Wavelength Interleaved Photonic Convolution Accelerator on TFLN Platform

All-optical complex field imaging using diffractive processors

Microcomb-driven Photonic Convolutional Engine

Integrated Photonic FFT for Optical Convolutions towards Efficient and High-Speed Neural Networks

Photonic convolution accelerator based on a hybrid integrated multi-wavelength laser array by photonic wire bonding for real-time image classification

Exploiting k-space - frequency duality in Fourier optics towards real-time compression-less terahertz imaging

All-optical complex-valued convolution based on four-wave mixing

Photonic Convolution Neural Network Based on Interleaved Time-Wavelength Modulation

High Throughput Multi-Channel Parallelized Diffraction Convolutional Neural Network Accelerator