Abstract:Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) FlowLens includes a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time. 2) It integrates a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. To facilitate training and evaluation, we derive the KITTI360 dataset with various FoV mask, which covers both outer- and inner FoV expansion scenarios. We also conduct both quantitative assessments and qualitative comparisons of beyond-FoV semantics and beyond-FoV object detection across different models. We illustrate that employing FlowLens to reconstruct unseen scenes even enhances perception within the field of view by providing reliable semantic context. Extensive experiments and user studies involving offline and online video inpainting, as well as beyond-FoV perception tasks, demonstrate that FlowLens achieves state-of-the-art performance. The source code and dataset are made publicly available at <a class="link-external link-https" href="https://github.com/MasterHow/FlowLens" rel="external noopener nofollow">this https URL</a>.

VisFlow: Adaptive Content-Aware Video Analytics on Collaborative Cameras

Scaling Video Analytics Systems to Large Camera Deployments

CrossVision: Real-time On-Camera Video Analysis via Common RoI Load Balancing

Adaptive Monitoring for Video Surveillance

Focus! Provisioning Attention-aware Detection for Real-time On-device Video Analytics.

Edge-Assisted Real-Time Video Analytics with Spatial–Temporal Redundancy Suppression

Beyond the Field-of-View: Enhancing Scene Visibility and Perception with Clip-Recurrent Transformer

ViTrack: Efficient Tracking on the Edge for Commodity Video Surveillance Systems

Low-Latency Edge Video Analytics for On-Road Perception of Autonomous Ground Vehicles

Enhancing UAV Detection in Surveillance Camera Videos through Spatiotemporal Information and Optical Flow

StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences

A Novel Efficient Multi-view Traffic-Related Object Detection Framework

Spatial and Temporal Detection With Attention for Real-Time Video Analytics At Edges

StreamYOLO: Real-time Object Detection for Streaming Perception

Scheduling Massive Camera Streams to Optimize Large-Scale Live Video Analytics

Toward Efficient and Adaptive Design of Video Detection System with Deep Neural Networks

VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows

AxiomVision: Accuracy-Guaranteed Adaptive Visual Model Selection for Perspective-Aware Video Analytics

Spatial Visibility and Temporal Dynamics: Revolutionizing Field of View Prediction in Adaptive Point Cloud Video Streaming

Adaptive Network Configuration for Efficient and Accurate Neural Video Inference

Collaborative Multi-Agent Video Fast-Forwarding