Abstract:The image captioning task is among the most important tasks in computer vision. Most existing methods mine more useful contextual information from image features. Similarly, to mine more contextual information, this paper proposes a visual contextual relationship augmented transformer (VRAT) method for improving the correctness of image description statements. In VRAT, visual contextual features are enhanced by using a pre-trained visual contextual relationship augmented module (VRAM). In VRAM, we classify images into three categories: globe, object, and grid, and use encoders of CLIP and ResNext to encode images and text to supplement the original image descriptions with visual and textual features. Finally, a similarity retrieval model is constructed to match global features, object features, and grid features for contextual relationships. During model training, our model supplements the original image captioning model with global, object, and grid visual features and textual features. In addition, to improve the quality of the attention-focused image features, we propose an attention augmented module (AAM) that adds a compensated attention module to the original multi-head attention, which allows a large number of image features in the model to focus more on important information and filter out some unimportant attention information. To alleviate the imbalance of positive and negative samples during training, we propose a multi-label focal loss in the model and combine it with the original cross-entropy loss function to improve the performance of the model. Experiments on the MSCOCO image description benchmark dataset show that the proposed method can perform well and outperform many existing state-of-the-art methods. The improvement in the CIDEr score and BLEU-1 score over the baseline model was 7.7 and 1.5, respectively.

Interaction augmented transformer with decoupled decoding for video captioning

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

I2Transformer: Intra- and Inter-relation Embedding Transformer for TV Show Captioning

Memory-enhanced Hierarchical Transformer for Video Paragraph Captioning

Cross on Cross Attention: Deep Fusion Transformer for Image Captioning

Dual-Level Decoupled Transformer for Video Captioning

Visual contextual relationship augmented transformer for image captioning

Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

End-to-End Dual-Stream Transformer with a Parallel Encoder for Video Captioning

Dual visual align-cross attention-based image captioning transformer

Image Captioning Based on An Improved Transformer with IoU Position Encoding

Multimodal attention-based transformer for video captioning

Layer-wise enhanced transformer with multi-modal fusion for image caption

Multi-Granularity Aggregation Transformer for Joint Video-Audio-Text Representation Learning

Captioning Transformer with Stacked Attention Modules

Multimodal Transformer With Multi-View Visual Representation for Image Captioning

Multi-Modal interpretable automatic video captioning

Multimodal-enhanced hierarchical attention network for video captioning

Multi-scale features with temporal information guidance for video captioning