Abstract:Edge computing of reliable multimodal (2D RGB/3D RGB-Depth) data has a wide range of applications. However, many of currently reported visual processors cannot flexibly handle multimodal data, e.g., the visual streams of RGB-Depth data. The key challenge exists that these prior visual processors do not come with efficient and unified instruction set architecture (ISA) for both conventional and intelligent cognition on the 2D/3D multimodal sensory data. To fill such a gap, this paper proposes a programmable intelligent visual vector processor compatible with multimodal 2D/3D visual data processing ( $1920\times 1080$ -pixel resolution). The processor consists of a reconfigurable processing element (PE) array, a memory access network flexibly configurable to be fine-or coarse-grained, and a high throughput I/O interface. The vectorial PE array with neighbor PE access increases the data reuse rate and parallel computation efficiency, and can implement both convolutional neural networks (CNNs) and conventional image processing algorithms. The proposed ISA is customized and optimally tailored targeting 2D/3D image processing from RGB/Time-of-Flight(ToF) raw data to intelligent inference results. The chip is fabricated in a 55-nm CMOS process. The experimental results showed that the area efficiency, peak performance, and peak throughput of our chip attained as high as 14.41 GOPS/mm $^2$ , 409.6 GOPS, and 9.6 Gbps at 200 MHz, respectively. The measured processing speeds of this chip on ToF depth reconstruction is 87 fps ( $480\times 270$ ) or 31 fps( $1920\times 1080$ ),on 3D object classification is 219 fps ( $256\times 256$ ), and on CNN-based 2D object tracking is 36 fps ( $256\times 256$ ).

A1920 × 1080 25FPS, 2.4TOPS/W Unified Optical Flow and Depth 6D Vision Processor for Energy-Efficient, Low Power Autonomous Navigation

A 1920 <inline-formula> <tex-math notation="LaTeX">$\times$ </tex-math></inline-formula> 1080 25-Frames/s 2.4-TOPS/W Low-Power 6-D Vision Processor for Unified Optical Flow and Stereo Depth With Semi-Global Matching

Full Hd Real-Time Depth Estimation Algorithm and Hardware Implementation

Full High-Definition Real-Time Depth Estimation for Three-Dimensional Video System

Hardware Solution of Real-Time Depth Estimation Based on Stereo Vision

FPGA implementation of full HD real-time depth estimation

A Real-Time 2D/3D Perception Visual Vector Processor for 1920 × 1080 High-Resolution High-Speed Intelligent Vision Chips.

A Fast and Accurate Optical Flow Camera for Resource-Constrained Edge Applications

A Real-Time Full HD 2D-to-3d Video Conversion System Based on FPGA

A 4k×2k@60fps Multi-format Multi-function Display Processor for High Perceptual Quality

A Universal Optical Flow Based Real-Time Low-Latency Omnidirectional Stereo Video System

High-Speed Scene Flow on Embedded Commercial Off-the-Shelf Systems.

A Configurable and Real-Time Multi-Frequency 3D Image Signal Processor for Indirect Time-of-Flight Sensors

High-Speed Optical 3D Measurement Sensor for Industrial Application

A 4Kx2K@60fps Multifunctional Video Display Processor for High Perceptual Image Quality

A 530 Mpixels/s 4096x2160@60fps H.264/AVC High Profile Video Decoder Chip

Stereo Matching Accelerator With Re-Computation Scheme and Data-Reused Pipeline for Autonomous Vehicles

FPGA based memory efficient high resolution stereo vision system for video tolling

A 40nm 2TOPS/W Depth-Completion Neural Network Accelerator SoC With Efficient Depth Engine for Realtime LiDAR Systems

FPGA Architecture for Real-Time Ultra-High Definition Glasses-Free 3D System.

Ultrahigh Framerate Vision Chip Featuring Central-Based Edge Detection Processed by All-Digital In-Imager Global-Parallel Processing Architecture