NWPU-Captions Dataset and MLCA-Net for Remote Sensing Image Captioning
Qimin Cheng,Haiyan Huang,Yuan Xu,Yuzhuo Zhou,Huanying Li,Zhongyuan Wang
DOI: https://doi.org/10.1109/tgrs.2022.3201474
IF: 8.2
2022-09-16
IEEE Transactions on Geoscience and Remote Sensing
Abstract:Recently, the burgeoning demands for captioning-related applications have inspired great endeavors in the remote sensing community. However, current benchmark datasets are deficient in data volume, category variety, and description richness, which hinders the advancement of new remote sensing image captioning approaches, especially those based on deep learning. To overcome this limitation, we present a larger and more challenging benchmark dataset termed NWPU-Captions is available at https://github.com/HaiyanHuang98/NWPU-Captions. NWPU-Captions contains 157 500 sentences, with all 31 500 images annotated manually by seven experienced volunteers. The superiority of NWPU-Captions over current publicly available benchmark datasets not only lies in its much larger scale but also in its wider coverage of complex scenes and the richness and variety of describing vocabularies. Furthermore, a novel encoder–decoder architecture, multilevel and contextual attention network (MLCA-Net), is proposed. MLCA-Net employs a multilevel attention module to adaptively aggregate image features of specific spatial regions and scales and introduces a contextual attention module to explore the latent context hidden in remote sensing images. MLCA-Net improves the flexibility and diversity of the generated captions while keeping their accuracy and conciseness by exploring the properties of scale variations and semantic ambiguity. Finally, the effectiveness, robustness, and generalization of MLCA-Net are proved through extensive experiments on existing datasets and NWPU-Captions.
imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics