Abstract:Video captioning aims to automatically generate a textual description for a video, which is a challenging task and has drawn attention recently. Despite existing methods have achieved impressive performance, two challenging problems are remaining to be solved. (1) In the feature encoding stage, existing methods only focus on local features or global features to improve the accuracy or readability of sentences generated, resulting in the underutilization of useful information for the given video. (2) In the decoder stage, vanilla Transformer is usually used to reason about visual relations to generate the textual captions, which is not making good use of the inter-frame temporal information, leads to the relation ambiguity and bad readability for generated captions. To solve these problems, we propose a method of video captioning based on multi-scale feature with temporal information guidance for video captioning. Firstly, the pre-training model CLIP is employed to extract video features. Secondly, the global and local features are encoded separately to learn the overall and detailed information of the video and construct multi-scale features. Finally, the gating unit is used to alleviate the problem which cannot make good use of contextual temporal information in existing decoder module base Transformer. Extensive experiments on two publicly available datasets show that the proposed model improves 4.7%, 2.2%, 0.6%, 2.0% on the MSR-VTT dataset, and 5.1%, 9.0%, 5.8%, 6.7% on the MSVD dataset compared to the best model in the comparison method in the BLEU, METEOR, ROUGE-L, and CIDEr metrics, which demonstrates the ability of our method to achieve more competitive performance.

DCMSTRD: End-to-end Dense Captioning via Multi-Scale Transformer Decoding

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

End-to-End 3D Dense Captioning with Vote2Cap-DETR

Textual Context-Aware Dense Captioning with Diverse Words

Multi-scale features with temporal information guidance for video captioning

Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning

Lightweight dense video captioning with cross-modal attention and knowledge-enhanced unbiased scene graph

EVC-MF: End-to-end Video Captioning Network with Multi-scale Features

Dual-Level Decoupled Transformer for Video Captioning

Captioning Transformer with Stacked Attention Modules

Delving Deeper into the Decoder for Video Captioning

Joint Multi-Scale Information and Long-Range Dependence for Video Captioning

Multi-scale Hierarchical Residual Network for Dense Captioning.

End-to-End Dual-Stream Transformer with a Parallel Encoder for Video Captioning

Multi-Level Visual Representation with Semantic-Reinforced Learning for Video Captioning.

Multi-Task Video Captioning with a Stepwise Multimodal Encoder

PIC 4th Challenge: Semantic-Assisted Multi-Feature Encoding and Multi-Head Decoding for Dense Video Captioning

End-to-end Image Captioning Via Visual Region Aggregation and Dual-level Collaboration

MPP-net: Multi-perspective Perception Network for Dense Video Captioning

X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning

Remote-Sensing Image Captioning Based on Multilayer Aggregated Transformer