Abstract:To guarantee vessel traffic safety in inland waterways, the automatic identification system (AIS) and shore-based cameras have been widely adopted to monitor moving vessels. The AIS data could provide the unique maritime mobile service identity (MMSI), position coordinates (i.e., latitude and longitude), course over ground, and speed over ground for the vessels of interest. In contrast, the cameras could directly display the visual appearance of vessels but fail to accurately grasp the vessels’ identity information and motion parameters. In this paper, we propose to improve the maritime traffic surveillance in inland waterways using the robust fusion of AIS and visual data. It is able to obtain more accurate vessel tracking results and kinematic characteristics. In particular, to robustly track the visual vessels under complex scenarios, we first propose an anti-occlusion vessel tracking method based on the simple online and real-time tracking with a deep association metric (DeepSORT) method. We then preprocess and predict the vessel positions to obtain synchronous AIS and visual data. Before the implementation of AIS and visual data fusion, the AIS position coordinates in the geodetic coordinate system will be projected into the image coordinate system via the coordinate transformation. A multi-feature similarity measurement-based Hungarian algorithm is finally proposed to robustly and accurately fuse the AIS and visual data in the image coordinate system. For the sake of repeating fusion experiments, we have also presented a new multi-sensor dataset containing AIS data and shore-based camera imagery. The quantitative and qualitative experiments show that our fusion method is capable of improving the maritime traffic surveillance in inland waterways. It can overcome the vessel occlusion problem and fully utilizes the advantages of multi-source data to promote the maritime surveillance, resulting in enhanced vessel traffic safety and efficiency. In this work, the presented multi-sensor dataset and source code are available at https://github.com/QuJX/AIS-Visual-Fusion.

A New Multi-Sensor Fusion Approach for Integrated Ship Motion Perception in Inland Waterways

Research on Active Intelligent Perception Technology of Vessel Situation Based on Multisensor Fusion

Improving Maritime Traffic Surveillance in Inland Waterways Using the Robust Fusion of AIS and Visual Data

Intelligent Edge-Enabled Efficient Multi-Source Data Fusion for Autonomous Surface Vehicles in Maritime Internet of Things

Multimodal Perception Integrating Point Cloud and Light Field for Ship Autonomous Driving

Practical Moving Target Detection in Maritime Environments Using Fuzzy Multi-sensor Data Fusion

Edge Computing-Enabled Multi-Sensor Data Fusion for Intelligent Surveillance in Maritime Transportation Systems

Intelligent Maritime Surveillance Framework Driven by Fusion of Camera-Based Vessel Detection and AIS Data

Asynchronous Trajectory Matching-Based Multimodal Maritime Data Fusion for Vessel Traffic Surveillance in Inland Waterways

Multi-modal Information Fusion for LiDAR-based 3D Object Detection Framework

Applied prospects of multisensor information fusion in ship

Estimation of Ship Berthing Parameters Based on Multi-LiDAR and MMW Radar Data Fusion

Identity Recognition on Waterways: a Novel Ship Information Tracking Method Based on Multimodal Data

A Multi-modality Sensor System for Unmanned Surface Vehicle

A Multi-sensor Fusion Method Based on Strict Velocity for Underwater Navigation System

Reliable LiDAR-based ship detection and tracking for Autonomous Surface Vehicles in busy maritime environments

Ship trajectory anomaly detection based on multi-feature fusion

Intelligent marine area supervision based on AIS and radar fusion

Research on Surface Environment Perception Via Camera-LiDAR Sensor Fusion

Association of AIS and Radar Data in Intelligent Navigation in Inland Waterways Based on Trajectory Characteristics

A Scheme on Indoor Tracking of Ship Dynamic Positioning Based on Distributed Multi-Sensor Data Fusion