Abstract:Change captioning aims to describe the semantic change between two similar images. In this process, as the most typical distractor, viewpoint change leads to the pseudo changes about appearance and position of objects, thereby overwhelming the real change. Besides, since the visual signal of change appears in a local region with weak feature, it is difficult for the model to directly translate the learned change features into the sentence. In this paper, we propose a syntax-calibrated multi-aspect relation transformer to learn effective change features under different scenes, and build reliable cross-modal alignment between the change features and linguistic words during caption generation. Specifically, a multi-aspect relation learning network is designed to 1) explore the fine-grained changes under irrelevant distractors (e.g., viewpoint change) by embedding the relations of semantics and relative position into the features of each image; 2) learn two view-invariant image representations by strengthening their global contrastive alignment relation, so as to help capture a stable difference representation; 3) provide the model with the prior knowledge about whether and where the semantic change happened by measuring the relation between the representations of captured difference and the image pair. Through the above manner, the model can learn effective change features for caption generation. Further, we introduce the syntax knowledge of Part-of-Speech (POS) and devise a POS-based visual switch to calibrate the transformer decoder. The POS-based visual switch dynamically utilizes visual information during different word generation based on the POS of words. This enables the decoder to build reliable cross-modal alignment, so as to generate a high-level linguistic sentence about change. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the three public datasets.

Describing and Localizing Multiple Changes with Transformers

Neighborhood Contrastive Transformer for Change Captioning

Changes to Captions: An Attentive Network for Remote Sensing Change Captioning

SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning.

Remote Sensing Image Change Captioning With Dual-Branch Transformers: A New Method and a Large Scale Dataset

Detection Assisted Change Captioning for Remote Sensing Image

A Multi-Task Network and Two Large Scale Datasets for Change Detection and Captioning in Remote Sensing Images

Multi-grained Representation Aggregating Transformer with Gating Cycle for Change Captioning

Robust Change Captioning

Enhancing Perception of Key Changes in Remote Sensing Image Change Captioning

Relation-aware Multi-pass Comparison Deconfounded Network for Change Captioning

Context-aware Difference Distilling for Multi-change Captioning

Multi-scale Change-Aware Transformer for Remote Sensing Image Change Detection

Multimodal Transformer With Multi-View Visual Representation for Image Captioning

Bidirectional difference locating and semantic consistency reasoning for change captioning

A Lightweight Sparse Focus Transformer for Remote Sensing Image Change Captioning

A Patch-Level Region-Aware Module with a Multi-Label Framework for Remote Sensing Image Captioning

TypeFormer: Multiscale Transformer With Type Controller for Remote Sensing Image Caption

Relation Changes Matter: Cross-Temporal Difference Transformer for Change Detection in Remote Sensing Images

TChange: A Hybrid Transformer-CNN Change Detection Network

Multimodal attention-based transformer for video captioning