Abstract:Video captioning can understand videos, provide decision-makers with user-friendly natural language narration, alleviate the gap between man and machine, and promote human-machine interaction. Therefore, it has good application prospects in emergency response scenarios, such as aerial refueling and assisted driving. However, there are two problems with the current video captioning methods: 1) they are mainly oriented to general domains, and there are few studies on industrial applications; 2) the methods only interact with the semantics of video and text from a single view (tokens or sentences). For the above problems, this paper proposes a multi-view end-to-end video caption (MVVC) method for human-machine fusion. Compared with the previous video captioning methods, 1) the MVVC model is an end-to-end model which directly takes video frames as input without object detection for each frame; 2) we perform cross-modal interaction of video and text from both local and global views. So the model can simultaneously understand video content and generate text at two granularities(tokens to sentences). In order to verify the performance of the new model, we conducted a series of comparative and ablation experiments on MVVC on the two data sets of aerial refueling and automatic driving. The experiments show that our new method has a stronger video understanding ability and can generate more accurate video descriptions. At the same time, it also verified that the video captioning task could promote human-machine fusion and assist decision-making in emergency scenarios. Note to Practitioners —The motivation of this paper is to convert the video into natural language so that the autonomous system can automatically understand the observed scene, describe it to relevant stakeholders, and promote human-machine fusion. However, the traditional method needs to process the video offline and has an insufficient understanding of the video content. Therefore, this paper proposes an end-to-end video capture method based on multi-view semantic alignment, which can understand the video content directly from the original video pixels in real time and improve captioning accuracy. It can meet the application requirements of the industrial field and has practical application value.

End-to-End Subtitle Detection and Recognition for Videos in East Asian Languages via CNN Ensemble with Near-Human-Level Performance

A new video text detection method.

Video Content Aware Dynamic Subtitles

A Multi-stage Method for Chinese Text Detection in News Videos

End-To-End Chinese Text Recognition

Character-aware audio-visual subtitling in context

Stateful Human-Centered Visual Captioning System to Aid Video Surveillance

Understanding Subtitles by Character-Level Sequence-to-Sequence Learning.

Visual Subtitle Feature Enhanced Video Outline Generation

Aligning Subtitles in Sign Language Videos

Attention and Language Ensemble for Scene Text Recognition with Convolutional Sequence Modeling.

Rolling and Non-Rolling Subtitle Detection with Temporal and Spatial Analysis for News Video

An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos

Collaborative Detection and Caption Network

A Character-Level Sequence-to-sequence Method for Subtitle Learning

Automatic video superimposed text detection based on Nonsubsampled Contourlet Transform

Ensemble Chinese End-to-End Spoken Language Understanding for Abnormal Event Detection from audio stream

Video Captioning With Attention-Based LSTM and Semantic Consistency

End-to-End Video Captioning Based on Multiview Semantic Alignment for Human–Machine Fusion

Understanding Pictograph with Facial Features: End-to-end Sentence-Level Lip Reading of Chinese

CSLNSpeech: solving the extended speech separation problem with the help of Chinese Sign Language