Formal Foundations for MARTE-SystemC Interoperability

P. Peñil,F. Herrera,E. Villar

DOI: https://doi.org/10.1049/IC.2010.0152

Abstract:

What problem does this paper attempt to address?

Film Trailer Generation via Task Decomposition

Pinelopi Papalampidi,Frank Keller,Mirella Lapata

DOI: https://doi.org/10.48550/arXiv.2111.08774

2021-11-16

Computer Vision and Pattern Recognition

Abstract:Movie trailers perform multiple functions: they introduce viewers to the story, convey the mood and artistic style of the film, and encourage audiences to see the movie. These diverse functions make automatic trailer generation a challenging endeavor. We decompose it into two subtasks: narrative structure identification and sentiment prediction. We model movies as graphs, where nodes are shots and edges denote semantic relations between them. We learn these relations using joint contrastive training which leverages privileged textual information (e.g., characters, actions, situations) from screenplays. An unsupervised algorithm then traverses the graph and generates trailers that human judges prefer to ones generated by competitive supervised approaches.
Creating Memorable Video Summaries That Satisfy the User's Intention for Taking the Videos.

Mengjuan Fei,Wei Jiang,Weijie Mao

DOI: https://doi.org/10.1016/j.neucom.2017.10.030

IF: 6

2017-01-01

Neurocomputing

Abstract:Video summarization facilitates rapid browsing and efficient video indexing in many applications. A good user video summary should be interesting and satisfy the user's intention for taking the video. While many existing methods generate trailers from conventionally constructed videos based on low-level features, this paper focuses on creating interesting video summaries from user videos using three semantic features, namely memorability, snap points, and motion. In the proposed framework, we first apply a novel memorability-based video segmentation method. We use a large image dataset with annotated memorability to fine-tune a Hybrid-AlexNet and predict the memorability scores of all video images using the fine-tuned deep network. Next, we estimate the visual importance scores of the segments based on their memorability scores, snap point scores, and motion scores. We then select an optimal subset of the segments to create a memorable and interesting summary that contains image sequences intentionally taken by the video capturer. Finally, we evaluate our method using the SumMe dataset, which contains 15 - 18 human-labeled video summaries. The experimental results demonstrate that our method generates high-quality video summaries that are comparable to human-created summaries and those created by state-of-the-art methods. (C) 2017 Elsevier B.V. All rights reserved.
Towards Automated Movie Trailer Generation

Dawit Mureja Argaw,Mattia Soldan,Alejandro Pardo,Chen Zhao,Fabian Caba Heilbron,Joon Son Chung,Bernard Ghanem

2024-04-04

Abstract:Movie trailers are an essential tool for promoting films and attracting audiences. However, the process of creating trailers can be time-consuming and expensive. To streamline this process, we propose an automatic trailer generation framework that generates plausible trailers from a full movie by automating shot selection and composition. Our approach draws inspiration from machine translation techniques and models the movies and trailers as sequences of shots, thus formulating the trailer generation problem as a sequence-to-sequence task. We introduce Trailer Generation Transformer (TGT), a deep-learning framework utilizing an encoder-decoder architecture. TGT movie encoder is tasked with contextualizing each movie shot representation via self-attention, while the autoregressive trailer decoder predicts the feature representation of the next trailer shot, accounting for the relevance of shots' temporal order in trailers. Our TGT significantly outperforms previous methods on a comprehensive suite of metrics.

Computer Vision and Pattern Recognition
Automatically selecting shots for action movie trailers

Alan F. Smeaton,Bart Lehane,Noel E. O'Connor,Conor Brady,Gary Craig

DOI: https://doi.org/10.1145/1178677.1178709

2006-01-01

Abstract:Movie trailers, or previews, are an important method of advertising movies. They are extensively shown before movies in cinemas, as well as on television and increasingly, over the Internet. Making a trailer is a creative process, in which a number of shots from a movie are selected in order to entice a viewer in to paying to see the full movie. Thus, the creation of these trailers is an integral part in the promotion of a movie. Action movies in particular rely on trailers as a form of advertising as it is possible to show short, exciting portions of an action movie, which are likely to appeal to the target audience. This paper presents an approach which automatically selects shots from action movies in order to assist in the creation of trailers. A set of audiovisual features are extracted that aim to model the characteristics of shots typically present in trailers, and a support vector machine is utilised in order to select the relevant shots. The approach taken is not particularly novel but the results show that the process may be used in order to ease the trailer creation process or to facilitate the creation of variable length, or personalised trailers.
Learning Trailer Moments in Full-Length Movies

Lezi Wang,Dong Liu,Rohit Puri,Dimitris N. Metaxas

DOI: https://doi.org/10.48550/arXiv.2008.08502

2020-08-19

Computer Vision and Pattern Recognition

Abstract:A movie's key moments stand out of the screenplay to grab an audience's attention and make movie browsing efficient. But a lack of annotations makes the existing approaches not applicable to movie key moment detection. To get rid of human annotations, we leverage the officially-released trailers as the weak supervision to learn a model that can detect the key moments from full-length movies. We introduce a novel ranking network that utilizes the Co-Attention between movies and trailers as guidance to generate the training pairs, where the moments highly corrected with trailers are expected to be scored higher than the uncorrelated moments. Additionally, we propose a Contrastive Attention module to enhance the feature representations such that the comparative contrast between features of the key and non-key moments are maximized. We construct the first movie-trailer dataset, and the proposed Co-Attention assisted ranking network shows superior performance even over the supervised approach. The effectiveness of our Contrastive Attention module is also demonstrated by the performance improvement over the state-of-the-art on the public benchmarks.
Editing Techniques of Movie Trailers based on Semantic Score Method

,,Suzuka HASEGAWA,Tsukasa KIKUCHI

DOI: https://doi.org/10.3756/artsci.15.125

2016-11-05

The Journal of the Society for Art and Science

Abstract:The movie trailers is important factor when the viewer selects a movie. "Movie trailer" is a promotional video for the viewer to think, "I want to see this movie." Movie content and performers, and a 2-minute atmosphere, such as from 30 seconds, held together in brief time as short as about 3 minutes long. In this paper, we propose a method of using the semantic score method to the scene extraction method of the main video to be used as a movie trailer in making a movie trailer video production. In the semantic graph based on semantic score, by extracting the high complexity change rate scene editing as trailer video (Figure 1), indicated by a verification experiment that can increase the interest of viewers.
An Inverse Partial Optimal Transport Framework for Music-guided Movie Trailer Generation

Yutong Wang,Sidan Zhu,Hongteng Xu,Dixin Luo

2024-07-30

Abstract:Trailer generation is a challenging video clipping task that aims to select highlighting shots from long videos like movies and re-organize them in an attractive way. In this study, we propose an inverse partial optimal transport (IPOT) framework to achieve music-guided movie trailer generation. In particular, we formulate the trailer generation task as selecting and sorting key movie shots based on audio shots, which involves matching the latent representations across visual and acoustic modalities. We learn a multi-modal latent representation model in the proposed IPOT framework to achieve this aim. In this framework, a two-tower encoder derives the latent representations of movie and music shots, respectively, and an attention-assisted Sinkhorn matching network parameterizes the grounding distance between the shots' latent representations and the distribution of the movie shots. Taking the correspondence between the movie shots and its trailer music shots as the observed optimal transport plan defined on the grounding distances, we learn the model by solving an inverse partial optimal transport problem, leading to a bi-level optimization strategy. We collect real-world movies and their trailers to construct a dataset with abundant label information called CMTD and, accordingly, train and evaluate various automatic trailer generators. Compared with state-of-the-art methods, our IPOT method consistently shows superiority in subjective visual effects and objective quantitative measurements.

Multimedia
Learning Trailer Moments in Full-Length Movies with Co-Contrastive Attention

Lezi Wang,Dong Liu,Rohit Puri,Dimitris N. Metaxas

DOI: https://doi.org/10.1007/978-3-030-58523-5_18

2020-01-01

Abstract:A movie’s key moments stand out of the screenplay to grab an audience’s attention and make movie browsing efficient. But a lack of annotations makes the existing approaches not applicable to movie key moment detection. To get rid of human annotations, we leverage the officially-released trailers as the weak supervision to learn a model that can detect the key moments from full-length movies. We introduce a novel ranking network that utilizes the Co-Attention between movies and trailers as guidance to generate the training pairs, where the moments highly corrected with trailers are expected to be scored higher than the uncorrelated moments. Additionally, we propose a Contrastive Attention module to enhance the feature representations such that the comparative contrast between features of the key and non-key moments are maximized. We construct the first movie-trailer dataset, and the proposed Co-Attention assisted ranking network shows superior performance even over the supervised(The term “supervised” refers to the approach with access to the manual ground-truth annotations for training.) approach. The effectiveness of our Contrastive Attention module is also demonstrated by the performance improvement over the state-of-the-art on the public benchmarks.
Find the Cliffhanger: Multi-Modal Trailerness in Soap Operas

Carlo Bretti,Pascal Mettes,Hendrik Vincent Koops,Daan Odijk,Nanne van Noord

DOI: https://doi.org/10.1007/978-3-031-53308-2_15

2024-01-29

Abstract:Creating a trailer requires carefully picking out and piecing together brief enticing moments out of a longer video, making it a challenging and time-consuming task. This requires selecting moments based on both visual and dialogue information. We introduce a multi-modal method for predicting the trailerness to assist editors in selecting trailer-worthy moments from long-form videos. We present results on a newly introduced soap opera dataset, demonstrating that predicting trailerness is a challenging task that benefits from multi-modal information. Code is available at

Computer Vision and Pattern Recognition,Multimedia
Predicting Movie Trailer Viewer's “like/dislike” Via Learned Shot Editing Patterns

Yimin Hou,Ting Xiao,Shu Zhang,Xi Jiang,Xiang Li,Xintao Hu,Junwei Han,Lei Guo,L. Stephen Miller,Richard Neupert,Tianming Liu

DOI: https://doi.org/10.1109/taffc.2015.2444371

IF: 13.99

2016-01-01

IEEE Transactions on Affective Computing

Abstract:Nowadays, there are many movie trailers publicly available on social media website such as YouTube, and many thousands of users have independently indicated whether they like or dislike those trailers. Although it is understandable that there are multiple factors that could influence viewers' like or dislike of the trailer, we aim to address a preference question in this work: Can subjective multimedia features be developed to predict the viewer's preference presented by like (by thumbs-up) or dislike (by thumbs-down) during and after watching movie trailers? We designed and implemented a computational framework that is composed of low-level multimedia feature extraction, feature screening and selection, and classification, and applied it to a collection of 725 movie trailers. Experimental results demonstrated that, among dozens of multimedia features, the single low-level multimedia feature of shot length variance is highly predictive of a viewer's “like/dislike” for a large portion of movie trailers. We interpret these findings such that variable shot lengths in a trailer tend to produce a rhythm that is likely to stimulate a viewer's positive preference. This conclusion was also proved by the repeatability experiments results using another 600 trailer videos and it was further interpreted by viewers'eye-tracking data.
Semantic Video Trailers

Harrie Oosterhuis,Sujith Ravi,Michael Bendersky

DOI: https://doi.org/10.48550/arXiv.1609.01819

IF: 5.414

2016-09-07

Machine Learning

Abstract:Query-based video summarization is the task of creating a brief visual trailer, which captures the parts of the video (or a collection of videos) that are most relevant to the user-issued query. In this paper, we propose an unsupervised label propagation approach for this task. Our approach effectively captures the multimodal semantics of queries and videos using state-of-the-art deep neural networks and creates a summary that is both semantically coherent and visually attractive. We describe the theoretical framework of our graph-based approach and empirically evaluate its effectiveness in creating relevant and attractive trailers. Finally, we showcase example video trailers generated by our system.
From Trailers to Storylines: An Efficient Way to Learn from Movies

Qingqiu Huang,Yuanjun Xiong,Yu Xiong,Yuqi Zhang,Dahua Lin

DOI: https://doi.org/10.48550/arXiv.1806.05341

2018-06-14

Computer Vision and Pattern Recognition

Abstract:The millions of movies produced in the human history are valuable resources for computer vision research. However, learning a vision model from movie data would meet with serious difficulties. A major obstacle is the computational cost -- the length of a movie is often over one hour, which is substantially longer than the short video clips that previous study mostly focuses on. In this paper, we explore an alternative approach to learning vision models from movies. Specifically, we consider a framework comprised of a visual module and a temporal analysis module. Unlike conventional learning methods, the proposed approach learns these modules from different sets of data -- the former from trailers while the latter from movies. This allows distinctive visual features to be learned within a reasonable budget while still preserving long-term temporal structures across an entire movie. We construct a large-scale dataset for this study and define a series of tasks on top. Experiments on this dataset showed that the proposed method can substantially reduce the training time while obtaining highly effective features and coherent temporal structures.
Competitive Analysis System for Theatrical Movie Releases Based on Movie Trailer Deep Video Representation

Miguel Campo,Cheng-Kang Hsieh,Matt Nickens,JJ Espinoza,Abhinav Taliyan,Julie Rieger,Jean Ho,Bettina Sherick

DOI: https://doi.org/10.48550/arXiv.1807.04465

2018-07-12

Abstract:Audience discovery is an important activity at major movie studios. Deep models that use convolutional networks to extract frame-by-frame features of a movie trailer and represent it in a form that is suitable for prediction are now possible thanks to the availability of pre-built feature extractors trained on large image datasets. Using these pre-built feature extractors, we are able to process hundreds of publicly available movie trailers, extract frame-by-frame low level features (e.g., a face, an object, etc) and create video-level representations. We use the video-level representations to train a hybrid Collaborative Filtering model that combines video features with historical movie attendance records. The trained model not only makes accurate attendance and audience prediction for existing movies, but also successfully profiles new movies six to eight months prior to their release.

Information Retrieval,Computer Vision and Pattern Recognition,Machine Learning,Multimedia
Video Content Marketing: The Making of Clips

Xuan Liu,Savannah Wei Shi,Thales Teixeira,Michel Wedel

DOI: https://doi.org/10.1509/jm.16.0048

IF: 15.36

2018-07-01

Journal of Marketing

Abstract:Consumers have an increasingly wide variety of options available to entertain themselves. This poses a challenge for content aggregators who want to effectively promote their video content online through original trailers of movies, sitcoms, and video games. Marketers are now trying to produce much shorter video clips to promote their content on a variety of digital channels. This research is the first to propose an approach to produce such clips and to study their effectiveness, focusing on comedy movies as an application. Web-based facial-expression tracking is used to study viewers’ real-time emotional responses when watching comedy movie trailers online. These data are used to predict both viewers’ intentions to watch the movie and the movie's box office success. The authors then propose an optimization procedure for cutting scenes from trailers to produce clips and test it in an online experiment and in a field experiment. The results provide evidence that the production of short clips using the proposed methodology can be an effective tool to market movies and other online content.

business
Multilevel profiling of situation and dialogue-based deep networks for movie genre classification using movie trailers

Dinesh Kumar Vishwakarma,Mayank Jindal,Ayush Mittal,Aditya Sharma

DOI: https://doi.org/10.48550/arXiv.2109.06488

2021-09-14

Computation and Language

Abstract:Automated movie genre classification has emerged as an active and essential area of research and exploration. Short duration movie trailers provide useful insights about the movie as video content consists of the cognitive and the affective level features. Previous approaches were focused upon either cognitive or affective content analysis. In this paper, we propose a novel multi-modality: situation, dialogue, and metadata-based movie genre classification framework that takes both cognition and affect-based features into consideration. A pre-features fusion-based framework that takes into account: situation-based features from a regular snapshot of a trailer that includes nouns and verbs providing the useful affect-based mapping with the corresponding genres, dialogue (speech) based feature from audio, metadata which together provides the relevant information for cognitive and affect based video analysis. We also develop the English movie trailer dataset (EMTD), which contains 2000 Hollywood movie trailers belonging to five popular genres: Action, Romance, Comedy, Horror, and Science Fiction, and perform cross-validation on the standard LMTD-9 dataset for validating the proposed framework. The results demonstrate that the proposed methodology for movie genre classification has performed excellently as depicted by the F1 scores, precision, recall, and area under the precision-recall curves.
Automating Video Thumbnails Selection and Generation with Multimodal and Multistage Analysis

Elia Fantini

2024-10-19

Abstract:This thesis presents an innovative approach to automate video thumbnail selection for traditional broadcast content. Our methodology establishes stringent criteria for diverse, representative, and aesthetically pleasing thumbnails, considering factors like logo placement space, incorporation of vertical aspect ratios, and accurate recognition of facial identities and emotions. We introduce a sophisticated multistage pipeline that can select candidate frames or generate novel images by blending video elements or using diffusion models. The pipeline incorporates state-of-the-art models for various tasks, including downsampling, redundancy reduction, automated cropping, face recognition, closed-eye and emotion detection, shot scale and aesthetic prediction, segmentation, matting, and harmonization. It also leverages large language models and visual transformers for semantic consistency. A GUI tool facilitates rapid navigation of the pipeline's output. To evaluate our method, we conducted comprehensive experiments. In a study of 69 videos, 53.6% of our proposed sets included thumbnails chosen by professional designers, with 73.9% containing similar images. A survey of 82 participants showed a 45.77% preference for our method, compared to 37.99% for manually chosen thumbnails and 16.36% for an alternative method. Professional designers reported a 3.57-fold increase in valid candidates compared to the alternative method, confirming that our approach meets established criteria. In conclusion, our findings affirm that the proposed method accelerates thumbnail creation while maintaining high-quality standards and fostering greater user engagement.

Computer Vision and Pattern Recognition
Story-driven Video Editing

Zheng Wang,Jianguo Li,Yu-Gang Jiang

DOI: https://doi.org/10.1109/tmm.2020.3037461

IF: 7.3

2021-01-01

IEEE Transactions on Multimedia

Abstract:This paper proposes a novel multimedia task: story-driven video editing. Given a story paragraph, this task aims to retrieve related video segments from a gallery of collected video segments and compose them into a video sequence by the storyline order. Our proposed baseline solution consists of three modules: a retrieval module, which returns lists of candidate segments for all query sentences in the story paragraph using an object-aware sentence-segment matching method; a sequence candidate proposal module, which aggregates the retrieved segment sets into a sequence proposal by the submodular optimization method; a sorting module, which arranges the candidates according to the storyline of the paragraph using the Sinkhorn network. We build a benchmark for this task including a reorganized version of the ActivityNet Captions dataset, a well-defined quantitative metric called Evaluation of Segment-to-Sequence Matching (ESSM) for measuring the difference between the generated video segment sequence and the ground truth. Quantitative results of the proposed baseline solution are reported. We hope this new task and benchmark will bring broad research attention and push forward a lot of novel online short video editing applications.

computer science, information systems,telecommunications, software engineering
Agent-based Video Trimming

Lingfeng Yang,Zhenyuan Chen,Xiang Li,Peiyang Jia,Liangqu Long,Jian Yang

2024-12-13

Abstract:As information becomes more accessible, user-generated videos are increasing in length, placing a burden on viewers to sift through vast content for valuable insights. This trend underscores the need for an algorithm to extract key video information efficiently. Despite significant advancements in highlight detection, moment retrieval, and video summarization, current approaches primarily focus on selecting specific time intervals, often overlooking the relevance between segments and the potential for segment arranging. In this paper, we introduce a novel task called Video Trimming (VT), which focuses on detecting wasted footage, selecting valuable segments, and composing them into a final video with a coherent story. To address this task, we propose Agent-based Video Trimming (AVT), structured into three phases: Video Structuring, Clip Filtering, and Story Composition. Specifically, we employ a Video Captioning Agent to convert video slices into structured textual descriptions, a Filtering Module to dynamically discard low-quality footage based on the structured information of each clip, and a Video Arrangement Agent to select and compile valid clips into a coherent final narrative. For evaluation, we develop a Video Evaluation Agent to assess trimmed videos, conducting assessments in parallel with human evaluations. Additionally, we curate a new benchmark dataset for video trimming using raw user videos from the internet. As a result, AVT received more favorable evaluations in user studies and demonstrated superior mAP and precision on the YouTube Highlights, TVSum, and our own dataset for the highlight detection task. The code and models are available at <a class="link-external link-https" href="https://ylingfeng.github.io/AVT" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
TeaserGen: Generating Teasers for Long Documentaries

Weihan Xu,Paul Pu Liang,Haven Kim,Julian McAuley,Taylor Berg-Kirkpatrick,Hao-Wen Dong

2024-10-08

Abstract:Teasers are an effective tool for promoting content in entertainment, commercial and educational fields. However, creating an effective teaser for long videos is challenging for it requires long-range multimodal modeling on the input videos, while necessitating maintaining audiovisual alignments, managing scene changes and preserving factual accuracy for the output teasers. Due to the lack of a publicly-available dataset, progress along this research direction has been hindered. In this work, we present DocumentaryNet, a collection of 1,269 documentaries paired with their teasers, featuring multimodal data streams of video, speech, music, sound effects and narrations. With DocumentaryNet, we propose a new two-stage system for generating teasers from long documentaries. The proposed TeaserGen system first generates the teaser narration from the transcribed narration of the documentary using a pretrained large language model, and then selects the most relevant visual content to accompany the generated narration through language-vision models. For narration-video matching, we explore two approaches: a pretraining-based model using pretrained contrastive language-vision models and a deep sequential model that learns the mapping between the narrations and visuals. Our experimental results show that the pretraining-based approach is more effective at identifying relevant visual content than directly trained deep autoregressive models.

Computer Vision and Pattern Recognition,Artificial Intelligence
MovieGraphs: Towards Understanding Human-Centric Situations from Videos

Paul Vicol,Makarand Tapaswi,Lluis Castrejon,Sanja Fidler

DOI: https://doi.org/10.48550/arXiv.1712.06761

2018-04-16

Abstract:There is growing interest in artificial intelligence to build socially intelligent robots. This requires machines to have the ability to "read" people's emotions, motivations, and other factors that affect behavior. Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips. Each graph consists of several types of nodes, to capture who is present in the clip, their emotional and physical attributes, their relationships (i.e., parent/child), and the interactions between them. Most interactions are associated with topics that provide additional details, and reasons that give motivations for actions. In addition, most interactions and many attributes are grounded in the video with time stamps. We provide a thorough analysis of our dataset, showing interesting common-sense correlations between different social aspects of scenes, as well as across scenes over time. We propose a method for querying videos and text with graphs, and show that: 1) our graphs contain rich and sufficient information to summarize and localize each scene; and 2) subgraphs allow us to describe situations at an abstract level and retrieve multiple semantically relevant situations. We also propose methods for interaction understanding via ordering, and reason understanding. MovieGraphs is the first benchmark to focus on inferred properties of human-centric situations, and opens up an exciting avenue towards socially-intelligent AI agents.

Computer Vision and Pattern Recognition

Formal Foundations for MARTE-SystemC Interoperability

Film Trailer Generation via Task Decomposition

Creating Memorable Video Summaries That Satisfy the User's Intention for Taking the Videos.

Towards Automated Movie Trailer Generation

Automatically selecting shots for action movie trailers

Learning Trailer Moments in Full-Length Movies

Editing Techniques of Movie Trailers based on Semantic Score Method

An Inverse Partial Optimal Transport Framework for Music-guided Movie Trailer Generation

Learning Trailer Moments in Full-Length Movies with Co-Contrastive Attention

Find the Cliffhanger: Multi-Modal Trailerness in Soap Operas

Predicting Movie Trailer Viewer's “like/dislike” Via Learned Shot Editing Patterns

Semantic Video Trailers

From Trailers to Storylines: An Efficient Way to Learn from Movies

Competitive Analysis System for Theatrical Movie Releases Based on Movie Trailer Deep Video Representation

Video Content Marketing: The Making of Clips

Multilevel profiling of situation and dialogue-based deep networks for movie genre classification using movie trailers

Automating Video Thumbnails Selection and Generation with Multimodal and Multistage Analysis

Story-driven Video Editing

Agent-based Video Trimming

TeaserGen: Generating Teasers for Long Documentaries

MovieGraphs: Towards Understanding Human-Centric Situations from Videos