Abstract:Event cameras are novel bio-inspired sensors, where individual pixels operate independently and asynchronously, generating intensity changes as events. Leveraging the microsecond resolution (no motion blur) and high dynamic range (compatible with extreme light conditions) of events, there is considerable promise in directly segmenting objects from sparse and asynchronous event streams in various applications. However, different from the rich cues in video object segmentation, it is challenging to segment complete objects from the sparse event stream. In this paper, we present the first framework for continuous-time object segmentation from event stream. Given the object mask at the initial time, our task aims to segment the complete object at any subsequent time in event streams. Specifically, our framework consists of a Recurrent Temporal Embedding Extraction (RTEE) module based on a novel ResLSTM, a Cross-time Spatiotemporal Feature Modeling (CSFM) module which is a transformer architecture with long-term and short-term matching modules, and a segmentation head. The historical events and masks (reference sets) are recurrently fed into our framework along with current-time events. The temporal embedding is updated as new events are input, enabling our framework to continuously process the event stream. To train and test our model, we construct both real-world and simulated event-based object segmentation datasets, each comprising event streams, APS images, and object annotations. Extensive experiments on our datasets demonstrate the effectiveness of the proposed recurrent architecture. Our code and dataset are available at https://sites.google.com/view/ecos-net/.

EISNet: A Multi-Modal Fusion Network for Semantic Segmentation with Events and Images

A Multi-Scale Recurrent Framework for Motion Segmentation With Event Camera

Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation

Modality Translation and Fusion for Event-Based Semantic Segmentation

EFCNet: Ensemble Full Convolutional Network for Semantic Segmentation of High-Resolution Remote Sensing Images

CIMFNet: Cross-layer Interaction and Multiscale Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Bimodal SegNet: Instance Segmentation Fusing Events and RGB Frames for Robotic Grasping

LMFNet: An Efficient Multimodal Fusion Approach for Semantic Segmentation in High-Resolution Remote Sensing

Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

FDGSNet: A Multi-modal Gated Segmentation Network for Remote Sensing Image Based on Frequency Decomposition

EFRNet: A Lightweight Network with Efficient Feature Fusion and Refinement for Real-Time Semantic Segmentation

AMIANet: Asymmetric Multimodal Interactive Augmentation Network for Semantic Segmentation of Remote Sensing Imagery

Enhanced Feature Pyramid Network for Semantic Segmentation.

Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings

Continuous-time Object Segmentation using High Temporal Resolution Event Camera

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

AEDN: Encoder-Decoder Network with Attention for Semantic Image Segmentation

MCFNet: Multi-scale Covariance Feature Fusion Network for Real-time Semantic Segmentation

PIF-Net: A Deep Point-Image Fusion Network for Multimodality Semantic Segmentation of Very High-Resolution Imagery and Aerial Point Cloud

Remote Sensing Image Semantic Segmentation Network Based on Multi-Scale Feature Enhancement Fusion

Multi-source collaborative enhanced for remote sensing images semantic segmentation