Multimodal Language Models for Domain-Specific Procedural Video Summarization

Nafisa Hussain

2024-07-07

Abstract:Videos serve as a powerful medium to convey ideas, tell stories, and provide detailed instructions, especially through long-format tutorials. Such tutorials are valuable for learning new skills at one's own pace, yet they can be overwhelming due to their length and dense content. Viewers often seek specific information, like precise measurements or step-by-step execution details, making it essential to extract and summarize key segments efficiently. An intelligent, time-sensitive video assistant capable of summarizing and detecting highlights in long videos is highly sought after. Recent advancements in Multimodal Large Language Models offer promising solutions to develop such an assistant. Our research explores the use of multimodal models to enhance video summarization and step-by-step instruction generation within specific domains. These models need to understand temporal events and relationships among actions across video frames. Our approach focuses on fine-tuning TimeChat to improve its performance in specific domains: cooking and medical procedures. By training the model on domain-specific datasets like Tasty for cooking and MedVidQA for medical procedures, we aim to enhance its ability to generate concise, accurate summaries of instructional videos. We curate and restructure these datasets to create high-quality video-centric instruction data. Our findings indicate that when finetuned on domain-specific procedural data, TimeChat can significantly improve the extraction and summarization of key instructional steps in long-format videos. This research demonstrates the potential of specialized multimodal models to assist with practical tasks by providing personalized, step-by-step guidance tailored to the unique aspects of each domain.

Computer Vision and Pattern Recognition,Information Retrieval

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to efficiently extract and summarize key steps in long - videos in specific fields, especially cooking and medical procedure videos. Specifically, the researchers hope to develop a multimodal large - language model that can understand the temporal events and action relationships in videos. By fine - tuning on datasets in specific fields, the ability of the model to generate concise and accurate video summaries and step - by - step instructions is improved. This research aims to provide an intelligent time - sensitive video assistant that can effectively summarize and detect highlights in long - videos, thereby meeting users' needs for specific information, such as precise measurement values or detailed execution steps.