Image and AIS Data Fusion Technique for Maritime Computer Vision Applications

Emre Gülsoylu,Paul Koch,Mert Yıldız,Manfred Constapel,André Peter Kelm

DOI: https://doi.org/10.48550/arXiv.2312.05270

2023-12-08

Abstract:Deep learning object detection methods, like YOLOv5, are effective in identifying maritime vessels but often lack detailed information important for practical applications. In this paper, we addressed this problem by developing a technique that fuses Automatic Identification System (AIS) data with vessels detected in images to create datasets. This fusion enriches ship images with vessel-related data, such as type, size, speed, and direction. Our approach associates detected ships to their corresponding AIS messages by estimating distance and azimuth using a homography-based method suitable for both fixed and periodically panning cameras. This technique is useful for creating datasets for waterway traffic management, encounter detection, and surveillance. We introduce a novel dataset comprising of images taken in various weather conditions and their corresponding AIS messages. This dataset offers a stable baseline for refining vessel detection algorithms and trajectory prediction models. To assess our method's performance, we manually annotated a portion of this dataset. The results are showing an overall association accuracy of 74.76 %, with the association accuracy for fixed cameras reaching 85.06 %. This demonstrates the potential of our approach in creating datasets for vessel detection, pose estimation and auto-labelling pipelines.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in marine computer vision applications, although existing deep - learning object detection methods (such as YOLOv5) can effectively identify ships, they lack detailed ship information, such as type, size, speed and direction, etc. This information is crucial for practical applications. To solve this problem, the author has developed a technique to fuse Automatic Identification System (AIS) data with ship detection results in images to create a data set rich in ship information. This fusion not only enhances the information content of ship images, but also provides strong support for waterway traffic management, encounter detection and monitoring.

Specifically, the method proposed in the paper associates the detected ships with their corresponding AIS messages by estimating distance and azimuth, and is applicable to fixed cameras and periodically rotating cameras. This method utilizes a homography - based method and can establish a connection between the image space and the world coordinates, thereby achieving accurate ship information matching. In addition, the paper also introduces a new data set, which contains images taken under different weather conditions and their corresponding AIS messages, providing a stable baseline for improving ship detection algorithms and trajectory prediction models. By manually annotating part of the data set, the author evaluated the performance of the method, and the results showed that the overall association accuracy rate was 74.76%, among which the association accuracy rate of the fixed camera reached 85.06%. This indicates that this method has potential application value in creating data sets for ship detection, pose estimation and automatic annotation pipelines.