Abstract:To guarantee vessel traffic safety in inland waterways, the automatic identification system (AIS) and shore-based cameras have been widely adopted to monitor moving vessels. The AIS data could provide the unique maritime mobile service identity (MMSI), position coordinates (i.e., latitude and longitude), course over ground, and speed over ground for the vessels of interest. In contrast, the cameras could directly display the visual appearance of vessels but fail to accurately grasp the vessels’ identity information and motion parameters. In this paper, we propose to improve the maritime traffic surveillance in inland waterways using the robust fusion of AIS and visual data. It is able to obtain more accurate vessel tracking results and kinematic characteristics. In particular, to robustly track the visual vessels under complex scenarios, we first propose an anti-occlusion vessel tracking method based on the simple online and real-time tracking with a deep association metric (DeepSORT) method. We then preprocess and predict the vessel positions to obtain synchronous AIS and visual data. Before the implementation of AIS and visual data fusion, the AIS position coordinates in the geodetic coordinate system will be projected into the image coordinate system via the coordinate transformation. A multi-feature similarity measurement-based Hungarian algorithm is finally proposed to robustly and accurately fuse the AIS and visual data in the image coordinate system. For the sake of repeating fusion experiments, we have also presented a new multi-sensor dataset containing AIS data and shore-based camera imagery. The quantitative and qualitative experiments show that our fusion method is capable of improving the maritime traffic surveillance in inland waterways. It can overcome the vessel occlusion problem and fully utilizes the advantages of multi-source data to promote the maritime surveillance, resulting in enhanced vessel traffic safety and efficiency. In this work, the presented multi-sensor dataset and source code are available at https://github.com/QuJX/AIS-Visual-Fusion.

Intelligent Edge-Enabled Efficient Multi-Source Data Fusion for Autonomous Surface Vehicles in Maritime Internet of Things

Edge Computing-Enabled Multi-Sensor Data Fusion for Intelligent Surveillance in Maritime Transportation Systems

Intelligent Maritime Surveillance Framework Driven by Fusion of Camera-Based Vessel Detection and AIS Data

Improving Maritime Traffic Surveillance in Inland Waterways Using the Robust Fusion of AIS and Visual Data

Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways

Research on Surface Environment Perception Via Camera-LiDAR Sensor Fusion

A New Multi-Sensor Fusion Approach for Integrated Ship Motion Perception in Inland Waterways

Research on Active Intelligent Perception Technology of Vessel Situation Based on Multisensor Fusion

Infrared and Visible Image Fusion Methods for Unmanned Surface Vessels with Marine Applications

Adaptive Multi-Source Data Fusion Vessel Trajectory Prediction Model for Intelligent Maritime Traffic

Video-Based Detection Infrastructure Enhancement for Automated Ship Recognition and Behavior Analysis

Enhanced YOLOv8 Ship Detection Empower Unmanned Surface Vehicles for Advanced Maritime Surveillance

Reliable LiDAR-based ship detection and tracking for Autonomous Surface Vehicles in busy maritime environments

An Underwater Multisensor Fusion Simultaneous Localization and Mapping System Based on Image Enhancement

Multi-modal Information Fusion for LiDAR-based 3D Object Detection Framework

A Multi-modality Sensor System for Unmanned Surface Vehicle

Optical and Synthetic Aperture Radar Image Fusion for Ship Detection and Recognition: Current State, Challenges, and Future Prospects

Deep Learning Based Multi-Modal Fusion Architectures for Maritime Vessel Detection

Characteristic evaluation via multi-sensor information fusion strategy for spherical underwater robots

Visual Temporal Fusion Based Free Space Segmentation for Autonomous Surface Vessels

Moving Ship Optimal Association for Maritime Surveillance: Fusing AIS and Sentinel-2 Data