Abstract:Capturing a video's meaning and critical concepts by analyzing the subtle details is a fundamental yet challenging task in video captioning. Identifying the dominant emotional tone in a video significantly enhances the perception of its context. Despite a strong emphasis on video captioning, existing models often need to adequately address emotional themes, resulting in suboptimal captioning results. To address these limitations, this paper proposes a novel Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities (SPECTRUM) framework to empower the generation of emotionally and semantically credible captions. Leveraging our pioneering structure, SPECTRUM discerns multimodal semantics and emotional themes using Visual Text Attribute Investigation (VTAI) and determines the orientation of descriptive captions through a Holistic Concept-Oriented Theme (HCOT), expressing emotionally-informed and field-acquainted references. They exploit video-to-text retrieval capabilities and the multifaceted nature of video content to estimate the emotional probabilities of candidate captions. Then, the dominant theme of the video is determined by appropriately weighting embedded attribute vectors and applying coarse- and fine-grained emotional concepts, which define the video's contextual alignment. Furthermore, using two loss functions, SPECTRUM is optimized to integrate emotional information and minimize prediction errors. Extensive experiments on the EmVidCap, MSVD, and MSRVTT video captioning datasets demonstrate that our model significantly surpasses state-of-the-art methods. Quantitative and qualitative evaluations highlight the model's ability to accurately capture and convey video emotions and multimodal attributes.
What problem does this paper attempt to address?
This paper attempts to solve the problem of insufficient emotional information in video caption generation. Specifically, existing video caption generation models often fail to fully consider the emotional themes in videos, resulting in deficiencies in the generated captions in terms of emotional expression and semantic accuracy. To solve this problem, this paper proposes a new framework - **SPECTRUM (Semantic Processing and Emotion - informed video - Captioning Through Retrieval and Understanding Modalities)**, aiming to generate captions that contain both emotional information and are semantically accurate.
### The main contributions of the paper include:
1. **Proposing the SPECTRUM framework**: This framework enhances the accuracy of video semantic description by fusing multi - modal features and combining emotional information and domain knowledge.
2. **Designing the Video - Text Attribute Investigation (VTAI) module**: This module is used to predict the probabilities of attribute cue words (including objects, scenes, actions, and emotions) in each caption, thereby generating meaningful captions.
3. **Developing the Holistic Concept - Oriented Theme (HCOT) module**: This module guides the emotional direction of captions through coarse - grained to fine - grained concept exploration, reducing semantically irrelevant descriptions.
4. **Introducing the pre - trained model CLIP**: It is used to encode text and images and perform video - to - text retrieval to obtain more accurate emotional information and domain knowledge.
5. **Extensive experimental verification**: Experiments on three datasets, EmVidCap, MSVD, and MSR - VTT, show that the proposed network achieves performance comparable to the existing state - of - the - art methods in the video caption generation task, especially outstanding in integrating emotions, facts, and overall concepts.
### Working principle of the SPECTRUM framework:
- **Feature Encoding Unit (FEU)**: Fuses visual, audio, and retrieved text features through a multi - modal method to generate a unified video feature representation.
- **Concept Investigation Unit (CIU)**: Includes two stages, VTAI and HCOT, which are respectively responsible for multi - modal semantic analysis and emotion - oriented concept generation.
- **Knowledge acquisition and caption generation**: Based on the previously generated features and concepts, uses a Transformer decoder to generate the final caption.
### Formula summary:
1. Visual feature extraction formula:
\[
F(a)=\Psi_a(V(a)) = \Gamma(\phi_a(V(a)).W_{\Psi_a}+b_{\Psi_a})
\]
\[
F(m)=\Psi_m(V(m))=\Gamma(\phi_m(V(m)).W_{\Psi_m}+b_{\Psi_m})
\]
2. Audio feature extraction formula:
\[
F(A)=\Psi_A(V(A))=\Gamma(\phi_A(V(A)).W_{\Psi_A}+b_{\Psi_A})
\]
3. Text feature extraction formula:
\[
F(T)=\Psi_T(X(T))=[f_{T_1}, f_{T_2},..., f_{T_{N_T}}]
\]
4. Attribute cue word probability prediction formula:
\[
Pr(ATT)=\Lambda(xW_{ATT}+b_{ATT})
\]
5. Multi - modal semantic extraction loss function:
\[
L(M2S)=-\frac{1}{N_{ATT}}\sum_{i = 1}^{N_{ATT}}\left(v_i\log(p_{ri})+(1 - v_i)\log(1 - p_{ri})\right)
\]
6. Potential overall attribute vector calculation formula:
\[
l_{H(FCT)}=\rho(FCT)