Abstract:Optical Flow (OF) is the movement pattern of pixels or edges that is caused in a visual scene by the relative motion between an agent and a scene. OF is used in a wide range of computer vision algorithms and robotics applications. While the calculation of OF is a resource-demanding task in terms of computational load and memory footprint, it needs to be executed at low latency, especially in robotics applications. Therefore, OF estimation is today performed on powerful CPUs or GPUs to satisfy the stringent requirements in terms of execution speed for control and actuation. On-sensor hardware acceleration is a promising approach to enable low latency OF calculations and fast execution even on resource-constrained devices such as nano drones and AR/VR glasses and headsets. This paper analyzes the achievable accuracy, frame rate, and power consumption when using a novel optical flow sensor consisting of a global shutter camera with an Application Specific Integrated Circuit (ASIC) for optical flow computation. The paper characterizes the optical flow sensor in high frame-rate, low-latency settings, with a frame rate of up to 88 fps at the full resolution of 1124 by 1364 pixels and up to 240 fps at a reduced camera resolution of 280 by 336, for both classical camera images and optical flow data.

What problem does this paper attempt to address?

This paper attempts to address the issue of achieving low-latency, high-accuracy optical flow computation on resource-constrained edge devices (such as nano drones and AR/VR glasses). Specifically, the paper analyzes the performance of a novel optical sensor (including an ASIC for optical flow computation) in high frame rate, low latency settings, including its accuracy, frame rate, and power consumption. Through experimental evaluation, the paper demonstrates the potential of this sensor in low-power, low-latency applications, particularly for scenarios requiring strict energy consumption control and rapid motion perception. ### Main Research Questions 1. **Low-latency optical flow computation**: How to achieve low-latency optical flow computation on resource-constrained edge devices. 2. **High-accuracy optical flow estimation**: How to maintain high-accuracy optical flow estimation on these devices. 3. **Low-power design**: How to design low-power optical flow computation hardware accelerators to meet the needs of resource-constrained devices. ### Solutions - **Hardware Accelerator**: The paper introduces an optical sensor (VD56G3) with an ASIC specifically designed for optical flow computation. - **Performance Evaluation**: The sensor's frame rate, accuracy, and power consumption were evaluated under different settings through experiments. - **Application Scenarios**: The potential applications of this sensor in fields such as high-speed autonomous vehicles and drones were explored. ### Experimental Results - **Accuracy**: The sensor's optical flow estimation accuracy is high at high resolution and appropriate frame rates. However, accuracy decreases when the frame rate is too high or the resolution is too low. - **Feature Tracking Length**: In static images, the feature tracking length generated by the sensor is comparable to reference implementations (such as ORB descriptors), but performs slightly worse in dynamic scenes. - **Power Consumption**: Enabling optical flow computation increases the sensor's power consumption by approximately 20%-24%, but it remains at a low level suitable for resource-constrained devices. ### Conclusion The paper demonstrates the potential of this novel optical sensor in low-latency, low-power optical flow computation, particularly for applications on resource-constrained edge devices. With reasonable configuration and optimization, this sensor can meet strict energy consumption requirements and provide reliable optical flow estimation in fast-motion scenarios.

A Fast and Accurate Optical Flow Camera for Resource-Constrained Edge Applications

Low Latency Visual Inertial Odometry with On-Sensor Accelerated Optical Flow for Resource-Constrained UAVs

NeuFlow: Real-time, High-accuracy Optical Flow Estimation on Robots Using Edge Devices

Dynamic Illumination Optical Flow Computing for Sensing Multiple Mobile Robots from a Drone.

Optical Convolution Based Computational Method for Low-Power Image Processing

An Efficient Gyro-Aided Optical Flow Estimation in Fast Rotations with Auto-Calibration

NeuFlow v2: High-Efficiency Optical Flow Estimation on Edge Devices

EdgeFlowNet: 100FPS@1W Dense Optical Flow For Tiny Mobile Robots

Parallelizing Optical Flow Estimation on an Ultra-Low Power RISC-V Cluster for Nano-UAV Navigation

Real-time high speed motion prediction using fast aperture-robust event-driven visual flow

An FPGA-Optimized Architecture of Real-time Farneback Optical Flow

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras

Time-of-Travel Methods for Measuring Optical Flow on Board a Micro Flying Robot

FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation

Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras

Ultrafast vision perception by neuromorphic optical flow

DIFT: Dynamic Iterative Field Transforms for Memory Efficient Optical Flow

Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream

RAPIDFlow: Recurrent Adaptable Pyramids with Iterative Decoding for Efficient Optical Flow Estimation

Real-time FPGA Implementation of an Optic Flow Algorithm Suitable for Background Oriented Schlieren Technique

SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations