Shifted Window Fourier Transform And Retention For Image Captioning

Jia Cheng Hu,Roberto Cavicchioli,Alessandro Capotondi
2024-08-26
Abstract:Image Captioning is an important Language and Vision task that finds application in a variety of contexts, ranging from healthcare to autonomous vehicles. As many real-world applications rely on devices with limited resources, much effort in the field was put into the development of lighter and faster models. However, much of the current optimizations focus on the Transformer architecture in contrast to the existence of more efficient methods. In this work, we introduce SwiFTeR, an architecture almost entirely based on Fourier Transform and Retention, to tackle the main efficiency bottlenecks of current light image captioning models, being the visual backbone's onerosity, and the decoder's quadratic cost. SwiFTeR is made of only 20M parameters, and requires 3.1 GFLOPs for a single forward pass. Additionally, it showcases superior scalability to the caption length and its small memory requirements enable more images to be processed in parallel, compared to the traditional transformer-based architectures. For instance, it can generate 400 captions in one second. Although, for the time being, the caption quality is lower (110.2 CIDEr-D), most of the decrease is not attributed to the architecture but rather an incomplete training practice which currently leaves much room for improvements. Overall, SwiFTeR points toward a promising direction to new efficient architectural design. The implementation code will be released in the future.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The paper attempts to address the problem of implementing efficient and lightweight image captioning models on resource-constrained devices. Specifically: - **Current Model Limitations**: Existing image captioning models (such as those based on the Transformer architecture) perform well in terms of performance but have a large number of parameters, requiring substantial memory and computational resources, making them difficult to deploy on resource-constrained devices. - **Research Objective**: The paper proposes a new architecture, SwiFTeR, which combines the Fourier Transform and Retention Layer, aiming to address the efficiency bottlenecks of existing models, particularly in terms of the overhead of the visual backbone network and the secondary cost of the decoder. SwiFTeR improves model efficiency by reducing the number of parameters and computational complexity, and demonstrates superior scalability when handling long sequences. - **Experimental Results**: Although SwiFTeR slightly underperforms some existing efficient models in terms of caption quality (CIDEr-D score of 110.2), it shows significant advantages in model size (only 20M parameters) and computational efficiency (3.1 GFLOPs per forward pass). Additionally, the authors note that the current lower quality score is mainly due to weaker training configurations, indicating substantial room for improvement in the future.