Abstract:Edge computing of reliable multimodal (2D RGB/3D RGB-Depth) data has a wide range of applications. However, many of currently reported visual processors cannot flexibly handle multimodal data, e.g., the visual streams of RGB-Depth data. The key challenge exists that these prior visual processors do not come with efficient and unified instruction set architecture (ISA) for both conventional and intelligent cognition on the 2D/3D multimodal sensory data. To fill such a gap, this paper proposes a programmable intelligent visual vector processor compatible with multimodal 2D/3D visual data processing ( $1920\times 1080$ -pixel resolution). The processor consists of a reconfigurable processing element (PE) array, a memory access network flexibly configurable to be fine-or coarse-grained, and a high throughput I/O interface. The vectorial PE array with neighbor PE access increases the data reuse rate and parallel computation efficiency, and can implement both convolutional neural networks (CNNs) and conventional image processing algorithms. The proposed ISA is customized and optimally tailored targeting 2D/3D image processing from RGB/Time-of-Flight(ToF) raw data to intelligent inference results. The chip is fabricated in a 55-nm CMOS process. The experimental results showed that the area efficiency, peak performance, and peak throughput of our chip attained as high as 14.41 GOPS/mm $^2$ , 409.6 GOPS, and 9.6 Gbps at 200 MHz, respectively. The measured processing speeds of this chip on ToF depth reconstruction is 87 fps ( $480\times 270$ ) or 31 fps( $1920\times 1080$ ),on 3D object classification is 219 fps ( $256\times 256$ ), and on CNN-based 2D object tracking is 36 fps ( $256\times 256$ ).

A high-speed vision processor based on pixel-parallel PE array and its applications

A high speed multi-level-parallel array processor for vision chips

A 1000 Fps Vision Chip Based on a Dynamically Reconfigurable Hybrid Architecture Comprising a PE Array Processor and Self-Organizing Map Neural Network.

A high speed vision processor for chip package visual inspection

7.3 A 1000fps Vision Chip Based on a Dynamically Reconfigurable Hybrid Architecture Comprising a PE Array and Self-Organizing Map Neural Network

A Real-Time 2D/3D Perception Visual Vector Processor for 1920 × 1080 High-Resolution High-Speed Intelligent Vision Chips.

Simd Massively Parallel Processing System For Real-Time Image Processing

Ultrahigh Framerate Vision Chip Featuring Central-Based Edge Detection Processed by All-Digital In-Imager Global-Parallel Processing Architecture

Sensor-level computer vision with pixel processor arrays for agile robots

A Compact PE Memory for Vision Chips

Mapping Image Transformations Onto Pixel Processor Arrays

A dynamically reconfigurable architecture for vision pre-processor

A Massively Parallel Keypoint Detection and Description (MP-KDD) Algorithm for High-Speed Vision Chip

Hierarchical and Parallel Pipelined Heterogeneous SoC for Embedded Vision Processing

Coarse-Grained Dynamically Reconfigurable Processor for Vision Pre-Processing

Design and Performance of a 1 ms High-Speed Vision Chip with 3D-Stacked 140 GOPS Column-Parallel PEs †

A Pixel-Parallel Array Processor Without Computational Logic for Computational Image Sensors.

Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays

A signed pulse-train based image processor-array for parallel kernel convolution in vision sensors

A Hierarchical and Parallel SoC Architecture for Vision Procesor.

A 182 mW 94.3 f/s in Full HD Pattern-Matching Based Image Recognition Accelerator for an Embedded Vision System in 0.13-$\mu{\rm m}$ CMOS Technology