Abstract:In several real-world scenarios like autonomous navigation and mobility, to obtain a better visual understanding of the surroundings, image captioning and object detection play a crucial role. This work introduces a novel multitask learning framework that combines image captioning and object detection into a joint model. We propose TICOD, Transformer-based Image Captioning and Object detection model for jointly training both tasks by combining the losses obtained from image captioning and object detection networks. By leveraging joint training, the model benefits from the complementary information shared between the two tasks, leading to improved performance for image captioning. Our approach utilizes a transformer-based architecture that enables end-to-end network integration for image captioning and object detection and performs both tasks jointly. We evaluate the effectiveness of our approach through comprehensive experiments on the MS-COCO dataset. Our model outperforms the baselines from image captioning literature by achieving a 3.65% improvement in BERTScore.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to obtain a better visual understanding of the surrounding environment through Image Captioning and Object Detection in autonomous navigation and human mobility assistance systems. Specifically, the author proposes a Transformer - based multi - task learning framework (TICOD), which can perform image captioning and object detection tasks simultaneously. This method aims to overcome several major problems in existing methods: 1. **Existing image captioning models usually adopt a two - stage training method**: First, use a pre - trained object detector (such as Faster R - CNN) to extract image region features, and then input these features into an encoder - decoder framework for image captioning. This method has the following disadvantages: - The object detection model is trained on a specific visual dataset (such as Visual Genome) in the first step, and its visual representation is not optimized for the image captioning dataset (such as MS - COCO), which may lead to error propagation. - The process of extracting regional features is time - consuming, causing the state - of - - the - art models to rely on cached visual features for training and evaluation, which limits the model design and introduces the problem of low runtime inference efficiency during prediction. 2. **Existing methods are difficult to achieve end - to - end training**: Generating descriptions directly from image pixels limits the application of these models in practical scenarios. To solve these problems, the author proposes a Transformer - based multi - task learning framework (TICOD), which combines image captioning and object detection tasks and improves the overall performance of the model through shared feature learning. Specifically, the model contains three main components: - **Backbone Network**: Used to extract image features. In this paper, Swin Transformer is used as the backbone network. - **Object Detection Network**: Use the Cascade R - CNN framework for object detection. - **Caption Generation Network**: Use the GPT - 2 architecture to generate image captions. By jointly training these two tasks, the model can enhance the overall learning ability with the knowledge obtained from each task, thus achieving a significant performance improvement in the image captioning task and also maintaining considerable performance in the object detection task. Specific improvements include: - Experiments on the MS - COCO dataset show that the BERTScore of the TICOD model in the image captioning task has increased by 3.65%. - In the object detection task, the TICOD model outperforms the performance of Swin Transformer under the Cascade R - CNN framework in metrics such as mAP, AP@0.75, AP@small, and AP@medium. In summary, the main contribution of this paper is to propose a multi - task learning model that can perform image captioning and object detection simultaneously. By optimizing the model parameters through the joint loss function, complementary learning between the two tasks is achieved, thereby improving the overall performance.

Transformer based Multitask Learning for Image Captioning and Object Detection

Transformer Union Convolution Network for Visual Object Tracking

Unified deep learning model for multitask representation and transfer learning: image classification, object detection, and image captioning

Multimodal Transformer With Multi-View Visual Representation for Image Captioning

Dual-level Collaborative Transformer for Image Captioning

Context-Aware Transformer for image captioning

Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentation

BENet: bi-directional enhanced network for image captioning

MulT: An End-to-End Multitask Learning Transformer

Transformer with multi-level grid features and depth pooling for image captioning

Multitask Learning for Cross-Domain Image Captioning

TOD-Net: An end-to-end transformer-based object detection network

End-to-End Transformer Based Model for Image Captioning

Show, Deconfound and Tell: Image Captioning with Causal Inference

CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation

Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning

A Multi-task Learning Approach for Image Captioning.

Dependent Multi-Task Learning with Causal Intervention for Image Captioning.

Transformer-Based Optimized Multimodal Fusion for 3D Object Detection in Autonomous Driving

ViTOC: Vision Transformer and Object-aware Captioner

Caption TLSTMs: Combining Transformer with LSTMs for Image Captioning