Abstract:To guarantee vessel traffic safety in inland waterways, the automatic identification system (AIS) and shore-based cameras have been widely adopted to monitor moving vessels. The AIS data could provide the unique maritime mobile service identity (MMSI), position coordinates (i.e., latitude and longitude), course over ground, and speed over ground for the vessels of interest. In contrast, the cameras could directly display the visual appearance of vessels but fail to accurately grasp the vessels’ identity information and motion parameters. In this paper, we propose to improve the maritime traffic surveillance in inland waterways using the robust fusion of AIS and visual data. It is able to obtain more accurate vessel tracking results and kinematic characteristics. In particular, to robustly track the visual vessels under complex scenarios, we first propose an anti-occlusion vessel tracking method based on the simple online and real-time tracking with a deep association metric (DeepSORT) method. We then preprocess and predict the vessel positions to obtain synchronous AIS and visual data. Before the implementation of AIS and visual data fusion, the AIS position coordinates in the geodetic coordinate system will be projected into the image coordinate system via the coordinate transformation. A multi-feature similarity measurement-based Hungarian algorithm is finally proposed to robustly and accurately fuse the AIS and visual data in the image coordinate system. For the sake of repeating fusion experiments, we have also presented a new multi-sensor dataset containing AIS data and shore-based camera imagery. The quantitative and qualitative experiments show that our fusion method is capable of improving the maritime traffic surveillance in inland waterways. It can overcome the vessel occlusion problem and fully utilizes the advantages of multi-source data to promote the maritime surveillance, resulting in enhanced vessel traffic safety and efficiency. In this work, the presented multi-sensor dataset and source code are available at https://github.com/QuJX/AIS-Visual-Fusion.

More Modalities Mean Better: Vessel Target Recognition and Localization Through Symbiotic Transformer and Multiview Regression

More Modalities Mean Better: Vessel Target Recognition and Localization Through Symbiotic Transformer and Multi-View Regression

More Perspectives Mean Better: Underwater Target Recognition and Localization with Multimodal Data via Symbiotic Transformer and Multiview Regression

Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways

A parallel convolutional neural network-transformer model for underwater target recognition based on multimodal feature learning

Vessel Type Recognition Using a Multi-Graph Fusion Method Integrating Vessel Trajectory Sequence and Dependency Relations

Mobile_ViT: Underwater Acoustic Target Recognition Method Based on Local–Global Feature Fusion

A Viewpoint Adaptation Ensemble Contrastive Learning framework for vessel type recognition with limited data

A Third-Modality Collaborative Learning Approach for Visible-Infrared Vessel Reidentification

A Third-Modality Collaborative Learning Approach for Visible-Infrared Vessel Re-Identification

A Hybrid Method for Inland Ship Recognition Using Marine Radar and Closed-Circuit Television

Underwater Shipwreck Target Detection Based on Audio-Visual Heterogeneous Data

Underwater Acoustic Target Recognition Method Based on Local–Global Feature Fusion

VLP2MSA: Expanding Vision-Language Pre-Training to Multimodal Sentiment Analysis

MMMViT: Multiscale multimodal vision transformer for brain tumor segmentation with missing modalities

Research on Active Intelligent Perception Technology of Vessel Situation Based on Multisensor Fusion

Visual Detection Algorithm for Enhanced Environmental Perception of Unmanned Surface Vehicles in Complex Marine Environments

A Multi-Modal Approach Based on Large Vision Model for Close-Range Underwater Target Localization

Fine-grained ship image classification and detection based on a vision transformer and multi-grain feature vector FPN model

Improving Maritime Traffic Surveillance in Inland Waterways Using the Robust Fusion of AIS and Visual Data

Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation