Abstract:Event cameras are novel bio-inspired sensors, where individual pixels operate independently and asynchronously, generating intensity changes as events. Leveraging the microsecond resolution (no motion blur) and high dynamic range (compatible with extreme light conditions) of events, there is considerable promise in directly segmenting objects from sparse and asynchronous event streams in various applications. However, different from the rich cues in video object segmentation, it is challenging to segment complete objects from the sparse event stream. In this paper, we present the first framework for continuous-time object segmentation from event stream. Given the object mask at the initial time, our task aims to segment the complete object at any subsequent time in event streams. Specifically, our framework consists of a Recurrent Temporal Embedding Extraction (RTEE) module based on a novel ResLSTM, a Cross-time Spatiotemporal Feature Modeling (CSFM) module which is a transformer architecture with long-term and short-term matching modules, and a segmentation head. The historical events and masks (reference sets) are recurrently fed into our framework along with current-time events. The temporal embedding is updated as new events are input, enabling our framework to continuously process the event stream. To train and test our model, we construct both real-world and simulated event-based object segmentation datasets, each comprising event streams, APS images, and object annotations. Extensive experiments on our datasets demonstrate the effectiveness of the proposed recurrent architecture. Our code and dataset are available at https://sites.google.com/view/ecos-net/.

FusionSeg: Motion Segmentation by Jointly Exploiting Frames and Events

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

ObjectFusion: an Object Detection and Segmentation Framework with RGB-D SLAM and Convolutional Neural Networks

Detection and Segmentation of Moving Objects Using Temporal and Spatial Cues

Motion Objects Segmentation Using a New Level Set Based Method

Event-Based Motion Segmentation with Spatio-Temporal Graph Cuts.

Automatic Segmentation of Moving Objects by Fusing Multi-Information

An Algorithm of Video Object Segmentation Based on Spatial-temporal Information Fusion

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Dual-Branch Dynamic Object Segmentation Network Based on Spatio-Temporal Information Fusion

Multi-Frame Image Fusion Method Combining Spatial-Temporal Saliency Detection and Nsct

Enhancing Moving Object Segmentation with Spatio-Temporal Information Fusion

Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects

FusionSeg: Learning to Combine Motion and Appearance for Fully Automatic Segmentation of Generic Objects in Videos

Detecting Line Segments in Motion-Blurred Images With Events

Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes

Object Tracking by Jointly Exploiting Frame and Event Domain

Multi-Motion Segmentation: Combining Geometric Model-Fitting and Optical Flow for RGB Sensors

CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation

Continuous-time Object Segmentation using High Temporal Resolution Event Camera

Studies on Moving Object Segmentation Based on Information Fusion in Video Sequences