A Unified Framework for Video Browsing and Retrieval

Yong Rui,Thomas S. Huang

2000-01-01

Abstract: this paper we focus our attention on the links between Index entities and shots. Shots arethe building blocks of the ToC. Other links are generalizable from the shot link.

What problem does this paper attempt to address?

Efficient Access to Video Content in a Unified Framework

Y Rui,SX Zhou,TS Huang

DOI: https://doi.org/10.1109/mmcs.1999.778576

1999-01-01

Abstract:Book contents' browsing and retrieval have been greatly facilitated by its Table-of-Contents (ToC) and Index, respectively. Unfortunately, today's video lacks such powerful mechanisms. In this paper, we explore and present novel techniques for constructing video ToC and Index. Furthermore, we explore the relationship between video browsing and retrieval and propose a unified framework: to incorporate both entities in a seamless way. Experimental results on real-world video clips justify our proposed framework for providing efficient access to video content.
Description and Browsing of Video Story Structure

WANG Dong-hui,QIAN Hui,ZHU Miao-liang

DOI: https://doi.org/10.3321/j.issn:1000-436x.2005.03.021

2005-01-01

Abstract:This paper presented a method for the description and browsing of video story structure. An innovatory algorithm of video segmentation was introduced to find the shot edge. And another refined match algorithm based on content comparability was proposed to implement the map of video structure from one dimension to two dimensions. By describing video story structure as a storyboard and sequence structure graph, video content can be browsed summarily.
Hierarchical Video Data Modeling and Indexing for Virtual Scene Construction

Huiyu Wang,Ruofeng Tong

DOI: https://doi.org/10.1109/icmv.2009.15

2009-01-01

Abstract:Rapidly growing quantities of digital video have made video data become a more important role in many applications than ever. Virtual scene construction also uses video materials as a rich resource. In order to manage the video materials like salient objects and scenes, an effective video retrieval system is required. In this paper, we present a video database management system using a hierarchical video model consisting of Video-Scene-Shot-SalientObject. We first propose a hierarchical video data model which presents a hierarchical structuring of video material and a hierarchical annotation of video material structure. Then two effective indexing structures are proposed based on the model, including an indexing tree structure which considers the relationship between salient objects and a semantic indexing structure inspired by the inverted file indexing structure. Finally, an implementation of this model is described and the efficiency of this method is evaluated.
A Unified Framework for Video Summarization, Browsing, and Retrieval

Ziyou Xiong,Regunathan Radhakrishnan,Yong Rui,Ajay Divakaran,Tsuhan Chen,Thomas S. Huang

DOI: https://doi.org/10.1016/b978-012119792-6/50122-4

2006-01-01

Abstract:Research on how to efficiently access video content has become increasingly active in the past few years. Considerable progress has been made in video analysis, representation, browsing, and retrieval, which are the four fundamental bases for accessing video content. Video analysis deals with the signal processing part of the video system, including shot boundary detection, key frame extraction, etc. Video representation is concerned with the structure of the video. An example of a video representation is the tree structured key frame hierarchy. Built on top of the video representation, video browsing deals with how to use the representation structure to help viewers browse the video content. Finally, video retrieval is concerned with retrieving interesting video objects.
A new video retrieval approach based on clustering

Zhen Lei,Lingda Wu,Songyang Lao,Gang Wang,Chen Wang

DOI: https://doi.org/10.1109/ICMLC.2004.1382055

2004-01-01

Abstract:The shot based clustering and retrieval is very important for video database organization and access. We can take advantage of semantic clustering to enhance the performance of traditional content-based video retrieval. In this paper, text feature and color feature are first extracted directly from shots. Subsequently, a two-level hierarchical clustering structure that provides indexing scheme for retrieval is constructed by incorporating text and color features. The top level is clustered by text features while the bottom level is clustered by color features. The constructed structure is then used for the cluster-based retrieval. The effectiveness and efficiency of our approach are demonstrated by the experimental results in retrieving and ranking similar video shots.
A hierarchical organization scheme for video data

Y.J. Zhang,H.B. Lu

DOI: https://doi.org/10.1016/S0031-3203(01)00189-3

IF: 8

2002-01-01

Pattern Recognition

Abstract:Video stream is based on bits of imagery and is thus difficult to be perceived (by machine) in the content level. To access video content, a suitable organization of video data is critical. This paper proposes a hierarchical structure and a process scheme for organizing video data to facilitate indexing, browsing and querying. Four layers can be distinguished, that is: video program, episode, shot and frame. This hierarchy provides an efficient and flexible structure as well as compact and meaningful abstraction of video program. To achieve such an organization, not only the boundary detection of shots and episodes, but also the extraction of key-frames for shots and the selection of representative shots and frames for episodes are important. Suitable criteria and methods for above tasks are proposed and these techniques have been integrated into a workable system. A number of organization experiments using real video data are performed and some results are presented, which show the effectiveness of the proposed organization scheme and techniques.
A Unified Framework for Semantic Shot Representation of Sports Video

Xiaofeng Tong,Qingshan Liu,Lingyu Duan,Hanqing Lu,Changsheng Xu,Qi Tian

DOI: https://doi.org/10.1145/1101826.1101848

2005-01-01

Abstract:The development of mid-level shot description helps to bridge the gap between low-level feature and high-level semantics in video indexing and analysis. In this paper, we present a unified framework for semantic shot representation in field-ball sports genres, in which a video shot is characterized via three essential properties, namely, camera shot size, subject in a scene and video production technology. The three properties clearly represent the primary factors of a shot, and provide a unified viewpoint of semantic shot definition. Based on this framework, we design an effective architecture for semantic shot management comprising three main components as: 1) flexible shot clustering and retrieval by adjusting the weights of three properties according to different requirements; 2) semantics based video temporal segmentation for further event recognition; and 3) comprehensive sports video semantics analysis. Extensive experiments on soccer, basketball and tennis demonstrate the effectiveness and validity of this framework.
A Tree-Based Paradigm for Content-Based Video Retrieval and Management

H Fang,Y Yin,J Jiang

DOI: https://doi.org/10.1117/12.642160

2006-01-01

Abstract:As video databases become increasingly important for full exploitation of multimedia resources, this paper aims at describing our recent efforts in feasibility studies towards building up a content-based and high-level video retrieval/management system. The study is focused on constructing a semantic tree structure via combination of low-level image processing techniques and high-level interpretation of visual content. Specifically, two separate algorithms were developed to organise input videos in terms of two layers: the shot layer and the key-frame layer. While the shot layer is derived by developing a multi-featured shot cut detection, the key frame layer is extracted automatically by a genetic algorithm. This paves the way for applying pattern recognition techniques to analyse those key frames and thus extract high level information to interpret the visual content or objects. Correspondingly, content-based video retrieval can be conducted in three stages. The first stage is to browse the digital video via the semantic tree at structural level, the second stage is match the key frame in terms of low-level features, such as colour, shape of objects, and texture etc. Finally, the third stage is to match the high-level information, such as conversation with indoor background, moving vehicles along a seaside road etc. Extensive experiments are reported in this paper for shot cut detection and key frame extraction, enabling the tree structure to be constructed.
ClassView: Hierarchical Video Shot Classification, Indexing, and Accessing

JP Fan,AK Elmagarmid,XQ Zhu,WG Aref,LD Wu

DOI: https://doi.org/10.1109/tmm.2003.819583

IF: 7.3

2004-01-01

IEEE Transactions on Multimedia

Abstract:Recent advances in digital video compression and networks have made video more accessible than ever. However, the existing content-based video retrieval systems still suffer from the following problems. 1) Semantics-sensitive video classification problem because of the semantic gap between low-level visual features and high-level semantic visual concepts; 2) Integrated video access problem because of the lack of efficient video database indexing, automatic video annotation, and concept-oriented summary organization techniques. In this paper, we have proposed a novel framework, called ClassView, to make some advances toward more efficient video database indexing and access. 1) A hierarchical semantics-sensitive video classifier is proposed to shorten the semantic gap. The hierarchical tree structure of the semantics-sensitive video classifier is derived from the domain-dependent concept hierarchy of video contents in a database. Relevance analysis is used for selecting the discriminating visual features with suitable importances. The Expectation-Maximization (EM) algorithm is also used to determine the classification rule for each visual concept node in the classifier. 2) A hierarchical video database indexing and summary presentation technique is proposed to support more effective video access over a large-scale database. The hierarchical tree structure of our video database indexing scheme is determined by the domain-dependent concept hierarchy which is also used for video classification. The presentation of visual summary is also integrated with the inherent hierarchical video database indexing tree structure. Integrating video access with efficient database indexing tree structure has provided great opportunity for supporting more powerful video search engines.
Browsing Linked Video Archives of WWW Video.

Zhenxing Zhang,Cathal Gurrin,Jinlin Guo

DOI: https://doi.org/10.1007/978-3-642-35728-2_49

2013-01-01

Abstract:In this paper, we describe an interactive video browsing system based on a graph of linked video objects. The system automatically organizes unstructured video archives by exploiting visual content similarity between objects in the videos. By generating a video link graph, the system can conceptually groups the videos that contains same objects together for searching and browsing. Both the chosen measures of video object similarity and the video data mining technologies are discussed here and included in the related software demonstrator. In addition, the software offers a query-by-image-example video search capability to jump into the video graph at a certain point to begin browsing the archive.
A Novel Feature Fusion Based Framework for Efficient Shot Indexing to Massive Web Videos

Yuan Dong,Lezi Wang,Shiguo Lian,Shusheng Cen,Wei Liu

DOI: https://doi.org/10.1007/s11235-014-9945-9

2014-01-01

Abstract:This study addresses an automatic approach to analyze the structure of large scale web videos based on visual and acoustic information. In our approach, video streams are macro-segmented via mining the duplicate sequences. Acoustic and visual information are both adopted for mining so as to avoid missing true-positive. Web videos contain severe visual and acoustic distortions, differing to TV data, where duplicate clips are quite similar. In this case, we present novel visual-acoustic feature schemes to handle the distortions. And shot based indexing algorithm and several temporary constrains are presented to mine the duplicate sequences, where the weak geometric verification is combined with direct hashing to achieve high efficiency and superior performance of image-based duplicate sequences detection, and dynamic programming is introduced to recall missing true-positives in audio-based section. Experiments conducted on the dataset composed of 500 h content-unknown videos show that F-Measure of duplicate sequences mining for web videos can achieve the rate of 95 % and, in terms of efficiency and detection performance, the proposed algorithm outperforms the state-of-art approaches.
Mental Visual Browsing

Jun He,Xindi Shang,Hanwang Zhang,Tat-Seng Chua

DOI: https://doi.org/10.1007/978-3-319-27674-8_44

2016-01-01

Abstract:We present a surprisingly easy-to-use video browser for helping users to pinpoint a specific video shot in mind, within a long video. At each interactive iteration, the only user effort required is to click 1 shot, which most visually relates to the user’s mental target, out of 8 displayed shots. Then, the system updates the browsing model and display another 8 shots for the next iteration. The proposed system is underpinned by a theoretically-sound Bayesian framework that maintains the probabilities of all the video shots segmented from the long video. This framework guarantees that we can find the target shot out of around 1-h video within 3–5 iterations. We believe that our system will perform well in the Video Broswer Showdown game of MMM 2016.
A Unified Framework for Web Video Topic Discovery and Visualization

Jian Shao,Shuai Ma,Weiming Lu,Yueting Zhuang

DOI: https://doi.org/10.1016/j.patrec.2011.07.026

IF: 4.757

2012-01-01

Pattern Recognition Letters

Abstract:Together with the explosive growth of web video in sharing sites like YouTube, automatic topic discovery and visualization have become increasingly important in helping to organize and navigate such large-scale videos. Previous work dealt with the topic discovery and visualization problem separately, and did not take fully into account of the distinctive characteristics of multi-modality and sparsity in web video features. This paper tries to solve web video topic discovery problem with visualization under a single framework, and proposes a Star-structured K-partite Graph based co-clustering and ranking framework, which consists of three stages: (1) firstly, represent the web videos and their multi-model features (e.g., keyword, near-duplicate keyframe, near-duplicate aural frame, etc.) as a Star-structured K-partite Graph; (2) secondly, group videos and their features simultaneously into clusters (topics) and organize the generated clusters as a linked cluster network; (3) finally, rank each type of nodes in the linked cluster network by ''popularity'' and visualize them as a novel interface to let user interactively browse topics in multi-level scales. Experiments on a YouTube benchmark dataset demonstrate the flexibility and effectiveness of our proposed framework.
InsightVideo: toward hierarchical video content organization for efficient browsing, summarization and retrieval

Xingquan Zhu,A. K. Elmagarmid,Xiangyang Xue,Lide Wu,A. C. Catlin

DOI: https://doi.org/10.1109/TMM.2005.850977

IF: 7.3

2005-01-01

IEEE Transactions on Multimedia

Abstract:Hierarchical video browsing and feature-based video retrieval are two standard methods for accessing video content. Very little research, however, has addressed the benefits of integrating these two methods for more effective and efficient video content access. In this paper, we introduce InsightVideo, a video analysis and retrieval system, which joins video content hierarchy, hierarchical browsing and retrieval for efficient video access. We propose several video processing techniques to organize the content hierarchy of the video. We first apply a camera motion classification and key-frame extraction strategy that operates in the compressed domain to extract video features. Then, shot grouping, scene detection and pairwise scene clustering strategies are applied to construct the video content hierarchy. We introduce a video similarity evaluation scheme at different levels (key-frame, shot, group, scene, and video.) By integrating the video content hierarchy and the video similarity evaluation scheme, hierarchical video browsing and retrieval are seamlessly integrated for efficient content access. We construct a progressive video retrieval scheme to refine user queries through the interactions of browsing and retrieval. Experimental results and comparisons of camera motion classification, key-frame extraction, scene detection, and video retrieval are presented to validate the effectiveness and efficiency of the proposed algorithms and the performance of the system.
Effective Approach for Video Shot Retrieval

Deng Li,Jin Lizuo,Fei Shumin

DOI: https://doi.org/10.13382/j.jemi.2008.01.018

2008-01-01

JOURNAL OF ELECTRONIC MEASUREMENT AND INSTRUMENT

Abstract:Shot retrieval plays a critical role in content-based retrieval.Motivated by the theory of ensemble similarity,the paper proposed a novel approach based on probabilistic distance algorithm for shot retrieval. In contrast to existing algorithms,the proposed approach emphasizes the integral of shot for effective similarity measure.An ensemble similarity is used to calibrate the similarity between two shots: a shot can be treated as an ensemble that consists of a sequence of multiple frames.By kernel method,in a high dimension space the feature vector represented frames can be assumed to follow a Gaussian distribution model.Then the probabilistic distance between two Gaussian distributions is computed as the similarity value between two shots.To improve the retrieval speed effectively,improved algorithms of Chernoff distance and KL divergence are also proposed.Experimental results indicate that the proposed approach achieves superior performance than some existing methods.
Structured and Content-Based Video Browsing

HJ Zhang,WY Ma

DOI: https://doi.org/10.1109/acssc.1998.751014

1998-01-01

Abstract:We argue the importance of content-based video browsing and present a set of power tools for video browsing. We propose a content representation scheme that captures both structure and visual content of video to support content-based video browsing. To extract video structure and content primitives and the visual abstraction needed by the representation scheme, a set of robust algorithms for video parsing, abstraction and clustering have been developed and is presented in this paper.
A Unified Framework for Shot Type Classification Based on Subject Centric Lens

Anyi Rao,Jiaze Wang,Linning Xu,Xuekun Jiang,Qingqiu Huang,Bolei Zhou,Dahua Lin

DOI: https://doi.org/10.48550/arXiv.2008.03548

2020-08-08

Abstract:Shots are key narrative elements of various videos, e.g. movies, TV series, and user-generated videos that are thriving over the Internet. The types of shots greatly influence how the underlying ideas, emotions, and messages are expressed. The technique to analyze shot types is important to the understanding of videos, which has seen increasing demand in real-world applications in this era. Classifying shot type is challenging due to the additional information required beyond the video content, such as the spatial composition of a frame and camera movement. To address these issues, we propose a learning framework Subject Guidance Network (SGNet) for shot type recognition. SGNet separates the subject and background of a shot into two streams, serving as separate guidance maps for scale and movement type classification respectively. To facilitate shot type analysis and model evaluations, we build a large-scale dataset MovieShots, which contains 46K shots from 7K movie trailers with annotations of their scale and movement types. Experiments show that our framework is able to recognize these two attributes of shot accurately, outperforming all the previous methods.

Computer Vision and Pattern Recognition,Machine Learning,Multimedia,Image and Video Processing
An Approach for Shot Retrieval by Optimal Matching in the Bipartite Graph

彭宇新,Ngo Chong-Wah,肖建国

DOI: https://doi.org/10.3321/j.issn:0372-2112.2004.07.020

2004-01-01

Tien Tzu Hsueh Pao/Acta Electronica Sinica

Abstract:Shot retrieval plays a critical role in content-based video retrieval. Motivated by the theory of optimal matching in bipartite graph, we propose a novel approach based on the Kuhn-Munkres algorithm for shot retrieval. In contrast to existing algorithms, the proposed approach emphasizes one-to-one mapping among frames between two shots for effective similarity measure. A weighted bipartite graph is constructed to model the similarity between two shots: every vertex in a bipartite graph represents one frame in a shot, and the weight of every edge represents the similarity value for a pair of frames between two shots. Then Kuhn-Munkres algorithm is employed to compute the maximum weight of a constructed bipartite graph as the similarity value between two shots by guaranteeing the one-to-one mapping among frames. To improve the speed efficiency, we also propose two improved algorithms. Experimental results indicate that the proposed approach achieves superior performance than some existing methods.
Video Summarization Based on User Log Enhanced Link Analysis

Bin Yu,Wei-Ying Ma,Klara Nahrstedt,HongJiang Zhang

DOI: https://doi.org/10.1145/957013.957095

2003-01-01

Abstract:Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which generally fails to capture the semantics of video content. Our vision is that users unintentionally embed their understanding of the video content in their interaction with computers. This valuable knowledge, which is difficult for computers to learn autonomously, can be utilized for video summarization process. In this paper, we present an intelligent video browsing and summarization system that utilizes previous viewers' browsing log to facilitate future viewers. Specifically, a novel ShotRank notion is proposed as a measure of the subjective interestingness and importance of each video shot. A ShotRank computation framework is constructed to seamlessly unify low-level video analysis and user browsing log mining. The resulting ShotRank is used to organize the presentation of video shots and generate video skims. Experimental results from user studies have strongly confirmed that ShotRank indeed represents the subjective notion of interestingness and importance of each video shot, and it significantly improves future viewers' browsing experience.
Hierarchical Indexing Structure for Efficient Similarity Search in Video Retrieval

Hong Lu,Beng Chin Ooi,Heng Tao Shen,Xiangyang Xue

DOI: https://doi.org/10.1109/TKDE.2006.174

IF: 9.235

2006-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:With the rapid increase in both centralized video archives and distributed WWW video resources, content-based video retrieval is gaining its importance. To support such applications efficiently, content-based video indexing must be addressed. Typically, each video is represented by a sequence of frames. Due to the high dimensionality of frame representation and the large number of frames, video indexing introduces an additional degree of complexity. In this paper, we address the problem of content-based video indexing and propose an efficient solution, called the Ordered VA-File (OVA-File) based on the VA-file. OVA-File is a hierarchical structure and has two novel features: 1) partitioning the whole file into slices such that only a small number of slices are accessed and checked during k Nearest Neighbor (kNN) search and 2) efficient handling of insertions of new vectors into the OVA-File, such that the average distance between the new vectors and those approximations near that position is minimized. To facilitate a search, we present an efficient approximate kNN algorithm named Ordered VA-LOW (OVA-LOW) based on the proposed OVA-File. OVA-LOW first chooses possible OVA-Slices by ranking the distances between their corresponding centers and the query vector, and then visits all approximations in the selected OVA-Slices to work out approximate kNN. The number of possible OVA-Slices is controlled by a user-defined parameter \delta. By adjusting \delta, OVA-LOW provides a trade-off between the query cost and the result quality. Query by video clip consisting of multiple frames is also discussed. Extensive experimental studies using real video data sets were conducted and the results showed that our methods can yield a significant speed-up over an existing VA-file-based method and iDistance with high query result quality. Furthermore, by incorporating temporal correlation of video content, our methods achieved much more efficient performance.

A Unified Framework for Video Browsing and Retrieval

Efficient Access to Video Content in a Unified Framework

Description and Browsing of Video Story Structure

Hierarchical Video Data Modeling and Indexing for Virtual Scene Construction

A Unified Framework for Video Summarization, Browsing, and Retrieval

A new video retrieval approach based on clustering

A hierarchical organization scheme for video data

A Unified Framework for Semantic Shot Representation of Sports Video

A Tree-Based Paradigm for Content-Based Video Retrieval and Management

ClassView: Hierarchical Video Shot Classification, Indexing, and Accessing

Browsing Linked Video Archives of WWW Video.

A Novel Feature Fusion Based Framework for Efficient Shot Indexing to Massive Web Videos

Mental Visual Browsing

A Unified Framework for Web Video Topic Discovery and Visualization

InsightVideo: toward hierarchical video content organization for efficient browsing, summarization and retrieval

Effective Approach for Video Shot Retrieval

Structured and Content-Based Video Browsing

A Unified Framework for Shot Type Classification Based on Subject Centric Lens

An Approach for Shot Retrieval by Optimal Matching in the Bipartite Graph

Video Summarization Based on User Log Enhanced Link Analysis

Hierarchical Indexing Structure for Efficient Similarity Search in Video Retrieval