Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-Media Retrieval

YangYi,ZhuangYue-Ting,WuFei,PanYun-He

IF: 7.3

2008-01-01

IEEE Transactions on Multimedia

Abstract:In this paper, we consider the problem of multimedia document (MMD) semantics understanding and content-based cross-media retrieval. An MMD is a set of media objects of different modalities but car...

What problem does this paper attempt to address?

Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-Media Retrieval

Yi Yang,Yue-Ting Zhuang,Fei Wu,Yun-He Pan

DOI: https://doi.org/10.1109/tmm.2008.917359

IF: 7.3

2008-01-01

IEEE Transactions on Multimedia

Abstract:In this paper, we consider the problem of multimedia document (MMD) semantics understanding and content-based cross-media retrieval. An MMD is a set of media objects of different modalities but carrying the same semantics and the content-based cross-media retrieval is a new kind of retrieval method by which the query examples and search results can be of different modalities. Two levels of manifolds are learned to explore the relationships among all the data in the level of MMD and in the level of media object respectively. We first construct a Laplacian media object space for media object representation of each modality and an MMD semantic graph to learn the MMD semantic correlations. The characteristics of media objects propagate along the MMD semantic graph and an MMD semantic space is constructed to perform cross-media retrieval. Different methods are proposed to utilize relevance feedback and experiment shows that the proposed approaches are effective.
Understanding Multimedia Document Semantics for Cross-Media Retrieval

Fei Wu,Yi Yang,Yueting Zhuang,Yunhe Pan

DOI: https://doi.org/10.1007/11581772_87

2005-01-01

Abstract:Multimedia Document (MMD) such as Web Page and Multimedia cyclopedias is composed of media objects of different modalities, and its integrated semantics is always expressed by the combination of all media objects in it. Since the contents in MMDs are enormous and the amount of them is increasing rapidly, effective management of MMDs is in great demand. Meanwhile, it is meaningful to provide users cross-media retrieval facilities so that users can query media objects by examples of different modalities, e.g. users may query an MMD (or an image) by submitting a audio clip and vice versa. However, there exist two challenges to achieve the above goals. First, how can we represent an MMD and fuse media objects together to achieve Cross-index and facilitate Cross-media retrieval? Second, how can we understand MMD semantics? Taking into account of the two problems, we give the definition of MMD and propose a manifold learning method to discover MMD semantics in this paper. We first construct an MMD semi-semantic graph (SSG) and then adopt Multidimensional scaling to create an MMD semantic space (MMDSS). We also propose two periods' feedbacks. The first one is used to refine SSG and the second one is adopted to introduce new MMD that is not in the MMDSS into MMDSS. Since all of the MMDs and their component media objects of different modalities lie in MMDSS, cross-media retrieval can be easily performed. Experiment results are encouraging and indicate that the performance of the proposed approach is effective.
Manifold Learning Based Cross-media Retrieval: A Solution to Media Object Complementary Nature

Yueting Zhuang,Yi Yang,Fei Wu,Yunhe Pan

DOI: https://doi.org/10.1007/s11265-006-0020-y

2007-01-01

Abstract:Media objects of different modalities always exist jointly and they are naturally complementary of each other, either in the view of semantics or in the view of modality. In this paper, we propose a manifold learning based cross-media retrieval approach that gives solutions to the two intrinsically basic but crucial questions of media objects semantics understanding and cross-media retrieval. First, considering the semantic complementary, how can we represent the concurrent media objects and fuse the complementary information they carry to understand the integrated semantics precisely. Second, considering the modality complementary, how can we accomplish the modality bridge to establish the cross-index and facilitate the cross-media retrieval? To solve the two problems, we first construct a Multimedia Document (MMD) Semi-Semantic Graph (MMDSSG) and then adopt Multidimensional Scaling to create an MMD Semantic Space (MMDSS). Both long-term and short-term feedbacks are proposed to boost the system performance. The first one is used to refine the MMDSSG and the second one is adopted to introduce new items that are not in the training set into the MMDSS. Since all of the MMDs and their component media objects of different modalities lie in the MMDSS and they are indexed uniformly by their coordinates in the MMDSS regardless of their modalities, the semantic subspace is actually a bridge of media objects which are of different modalities and the cross-media retrieval can be easily achieved. Experiment results are encouraging and indicate that the proposed approach is effective.
Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-Media Retrieval

Yue-Ting Zhuang,Yi Yang,Fei Wu

DOI: https://doi.org/10.1109/TMM.2007.911822

IF: 7.3

2008-01-01

IEEE Transactions on Multimedia

Abstract:Although multimedia objects such as images, audios and texts are of different modalities, there are a great amount of semantic correlations among them. In this paper, we propose a method of transductive learning to mine the semantic correlations among media objects of different modalities so that to achieve the cross-media retrieval. Cross-media retrieval is a new kind of searching technology by which the query examples and the returned results can be of different modalities, e.g., to query images by an example of audio. First, according to the media objects features and their co-existence information, we construct a uniform cross-media correlation graph, in which media objects of different modalities are represented uniformly. To perform the cross-media retrieval, a positive score is assigned to the query example; the score spreads along the graph and media objects of target modality or MMDs with the highest scores are returned. To boost the retrieval performance, we also propose different approaches of long-term and short-term relevance feedback to mine the information contained in the positive and negative examples.
Heterogeneous multimedia data semantics mining using content and location context.

Yi Yang,Yueting Zhuang,Wenhua Wang

DOI: https://doi.org/10.1145/1459359.1459452

2008-01-01

Abstract:Because it is very common that the heterogeneous multimedia data of the same semantics always exist jointly in many domain and application specific databases, it is very helpful to consider the location information when analyzing multimedia data. In this paper we propose a method of integrating the content and location context for multimedia data mining to enable the cross-media retrieval, by which the query examples and the returned results can be of different modalities, e.g. to query audios by an example of image. We construct a graph model by combing the multimedia content and location information. The graph model is then refined according to different strategies. The semantic correlations among multimedia data are calculated by learning the high-order neighborhood structure of the graph and the Multimedia Correlation Space is constructed in which the cross-media retrieval can be performed. We also propose different methods of Relevance Feedback to improve the search results. Experiments demonstrate the promise of the proposed method.
Cross-Media Retrieval: Concepts, Advances And Challenges

Yueting Zhuang,Fei Wu,Hong Zhang,Yi Yang

2006-01-01

Abstract:Cross-media - retrieval is an emerging comprehensive research topic, which seeks to provide, more effective retrieval approach so that internet users could query multimedia objects by examples in the form of different media. For example, users can query images by submitting an example audio clip in a cross-media retrieval system and vice versa. Clearly, a cross-media retrieval system better fits for human habits and is thus more powerful in retrieval performance. In order to achieve cross-media retrieval, we need to resolve the problem of semantic understanding and mappings among heterogeneous low-level multi-modal features spaces, such as judging the correlation between visual contents and auditory contents in accordance with human perception. In this paper we give the concept of cross-media retrieval, and two effective approaches for the two kinds of cross-media retrieval, namely Correlation Isomorphic Space Learning (CISL) for media object retrieval and Manifold Semantic Space Learning (MSSL) for multimedia document retrieval. CISL uses canonical correlation analysis to map pairs of heterogeneous multi-modal features into an integrity semantic subspace where canonical correlations are furthest preserved. MSSL implements manifolds learning to explore the relationship among multimedia documents and media objects within them respectively. Experiment results are encouraging and indicate that the performance of the proposed approaches is effective.
Improving Web-Based Learning: Automatic Annotation of Multimedia Semantics and Cross-Media Indexing

Yueting Zhuang,Congmiao Wu,Fei Wu,Xiang Liu

DOI: https://doi.org/10.1007/978-3-540-27859-7_33

2004-01-01

Abstract:Multimedia plays an important role in the web-based learning environment. In this paper, we validate management and retrieval of large multimedia collections through high-level semantics. A novel algorithm is proposed for automatic annotation of image based on support vector machine and statistical learning. In addition, we construct cross-media indexing for multi-modal data upon the annotation result to support cross-media search. Experiments show that our algorithm can interpret multimedia semantics accurately and cross-media indexing can support cross-media search effectively.
The Research of Multimedia Cross Reference Retrieval System

Yueting Zhuang,Congmiao Wu,Fei Wu,Xiang Liu

DOI: https://doi.org/10.3321/j.issn:1003-9775.2005.04.034

2005-01-01

Abstract:In this paper, we propose a novel multimedia cross-reference retrieval system for digital libraries. The most important feature of the system is to integrate the multi-modal data into a seamless retrieval system. The cross-reference search engine, based on the cross-reference graph, serves as a bridge in the whole system. It calculates both the semantic and content level similarity between media object and the query. It is also able to adjust the cross-reference graph model based on relevance feedback conducted by users, which progressively improves the retrieval accuracy. As demonstrated by the experiments, the system can achieve satisfactory retrieval efficiency on multi-modal datasets.
A MULTI-CHANNEL RETRIEVAL SYSTEM FOR MULTIMEDIA DOCUMENTS

Wang Yanyun,Liu Xiang,Wu Fei

DOI: https://doi.org/10.3969/j.issn.1000-386X.2009.07.004

2009-01-01

Abstract:A multi-channel,corresponding to multiple modals,e.g.text,image,video,etc.,retrieval system for multimedia documents is proposed and implemented.A novel framework model is defined in system to describe the contents of multimedia documents.In the framework,content-based bottom features in each channel extracted from multimedia documents and the links between multimedia objects of multimedia documents are both recorded.Moreover,a graph model-based cross-reference knowledge base is proposed to store the semantic relation between multimedia objects mined from link relations.A valid algorithm for semantic context analysis is proposed to calculate the semantic similarity between each object and query when retrieving.The algorithm not only endues content-based multimedia information retrieval method with semantic information of multimedia objects,but also provides a flexible query model,which enables a user to interact with the system through channel exchange and relevance feedback.It is demonstrated in the experiments that the retrieval performance(e.g.the content coverage rate) for multimedia objects in the proposed multi-channel retrieval system is much better than that of a system with conventional content-based retrieval methods.
Multi-Feature Fusion Via Hierarchical Regression for Multimedia Analysis

Yi Yang,Jingkuan Song,Zi Huang,Zhigang Ma,Nicu Sebe,Alexander G. Hauptmann

DOI: https://doi.org/10.1109/tmm.2012.2234731

IF: 7.3

2013-01-01

IEEE Transactions on Multimedia

Abstract:Multimedia data are usually represented by multiple features. In this paper, we propose a new algorithm, namely Multi-feature Learning via Hierarchical Regression for multimedia semantics understanding, where two issues are considered. First, labeling large amount of training data is labor-intensive. It is meaningful to effectively leverage unlabeled data to facilitate multimedia semantics understanding. Second, given that multimedia data can be represented by multiple features, it is advantageous to develop an algorithm which combines evidence obtained from different features to infer reliable multimedia semantic concept classifiers. We design a hierarchical regression model to exploit the information derived from each type of feature, which is then collaboratively fused to obtain a multimedia semantic concept classifier. Both label information and data distribution of different features representing multimedia data are considered. The algorithm can be applied to a wide range of multimedia applications and experiments are conducted on video data for video concept annotation and action recognition. Using Trecvid and CareMedia video datasets, the experimental results show that it is beneficial to combine multiple features. The performance of the proposed algorithm is remarkable when only a small amount of labeled training data are available.
Extracting Multimedia Semantics Based On Independent Modality Discovering And Fusion

Ruo-Gui Xiao,Yue-Ting Zhuang,Fei Wu

DOI: https://doi.org/10.1109/ICMLC.2006.258993

2006-01-01

Abstract:Learning semantics from low-level features of multimedia learning resources enables high-level access to multimedia content. Considerable amount of researches have been focused on multi-modal analysis to detect multimedia semantics. However, two fundamental issues have not been adequately addressed. First, given a set of raw features extracted from multimedia sources, what are the best independent modalities? Second, once a set of modalities has been identified, how are they optimally fused to map to the high-level semantics? In this paper, we apply statistical and machine learning techniques to answer the two questions. ISOMAP combining with support vector clustering are used to discover independent modalities from raw features. Then Maximum Entropy method is applied to optimally fuse the individual modalities. Experiments show that the proposed method can learn multimedia semantics more efficiently than traditional methods.
Cross-media retrieval using query dependent search methods

Yi Yang,Fei Wu,Dong Xu,Yueting Zhuang,Liang-Tien Chia

DOI: https://doi.org/10.1016/j.patcog.2010.02.015

IF: 8

2010-01-01

Pattern Recognition

Abstract:The content-based cross-media retrieval is a new type of multimedia retrieval in which the media types of query examples and the returned results can be different. In order to learn the semantic correlations among multimedia objects of different modalities, the heterogeneous multimedia objects are analyzed in the form of multimedia document (MMD), which is a set of multimedia objects that are of different media types but carry the same semantics. We first construct an MMD semi-semantic graph (MMDSSG) by jointly analyzing the heterogeneous multimedia data. After that, cross-media indexing space (CMIS) is constructed. For each query, the optimal dimension of CMIS is automatically determined and the cross-media retrieval is performed on a per-query basis. By doing this, the most appropriate retrieval approach for each query is selected, i.e. different search methods are used for different queries. The query dependent search methods make cross-media retrieval performance not only accurate but also stable. We also propose different learning methods of relevance feedback (RF) to improve the performance. Experiment is encouraging and validates the proposed methods.
Multimedia information retrieval at a crossroad

Qing Li,Jun Yang,Yueting Zhuang

DOI: https://doi.org/10.4018/978-1-60566-014-1.ch134

2008-01-01

Abstract:In the late 1990s, the availability of powerful computing capability, large storage devices, high-speed networking and especially the advent of the Internet, led to a phenomenal growth of digital multimedia content in terms of size, diversity and impact. As suggested by its name,“multimedia” is a name given to a collection of multiple types of data, which include not only “traditional multimedia” such as images and videos, but also emerging media such as 3D graphics (like VRML objects) and Web animations (like Flash animations). Furthermore, multimedia techniques have been penetrating into a growing number of applications, ranging from document-editing software to digital libraries and many Web applications. For example, most people who have used Microsoft Word have tried to insert pictures and diagrams into their documents, and they have the experience of watching online video clips, such as movie trailers …
Content-oriented Multimedia Document Understanding Through Cross-Media Correlation

Tong Lu,Yukang Jin,Feng Su,Palaiahnakote Shivakumara,Chew Lim Tan

DOI: https://doi.org/10.1007/s11042-014-2044-9

IF: 2.577

2014-01-01

Multimedia Tools and Applications

Abstract:This paper presents a novel method for multimedia document content analysis through modeling multimodal data correlations. We hypothesize that the correlation of different modalities from the same data source can help achieve better multimedia content understanding results than one which explores a single modality. We turn this task into two parts: multimedia data fusion and multimodal correlation propagation. During the first stage, we re-organize the training multimedia data into Modality semAntic Documents (MADs) after extracting quantized multimodal features, and then use multivariate Gaussian distributions to characterize the continuous quantity by latent topic modeling. Model parameters are asymmetrically learned to initialize multimodal correlations in the latent topic space. Accordingly, during the second stage, we construct a Multimodal Correlation Network (MCN) based on the initialized multimodal correlations, and a new mechanism of propagating inter-modality correlations and intra-modality similarities in MCN is further proposed to take the complementary from cross-modalities to facilitate multimedia content analysis. The experimental results of image-audio data retrieval on a 10-categories dataset and content-oriented web page recommendation on the USTODAY dataset show the effectiveness of our method.
MediaView: A Semantic View Mechanism for Multimedia Modeling

Qing Li,Jun Yang,Yueting Zhuang

DOI: https://doi.org/10.1007/3-540-36228-2_90

2002-01-01

Abstract:The semantics of multimedia data, which features context-dependency and media-independency, is of vital importance to multimedia applications but inadequately supported by the state-of-the-art database technology. In this paper, we address this problem by proposing MediaView as an extended object-oriented view mechanism to bridge the "semantic gap" between conventional databases and semantics-intensive multimedia applications. This mechanism captures the dynamic semantics of multimedia using a modeling construct named media view, which formulates a customized context where heterogeneous media objects with similar/related semantics are characterized by additional properties and user-defined semantic relationships. View operators are proposed for the manipulation and derivation of media views. The usefulness and elegancy of MediaView are demonstrated by its application in a multi-modal information retrieval system.
MHMS: Multimodal Hierarchical Multimedia Summarization

Jielin Qiu,Jiacheng Zhu,Mengdi Xu,Franck Dernoncourt,Trung Bui,Zhaowen Wang,Bo Li,Ding Zhao,Hailin Jin

DOI: https://doi.org/10.48550/arXiv.2204.03734

2022-04-07

Computer Vision and Pattern Recognition

Abstract:Multimedia summarization with multimodal output can play an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. In this work, we propose a multimodal hierarchical multimedia summarization (MHMS) framework by interacting visual and language domains to generate both video and textual summaries. Our MHMS method contains video and textual segmentation and summarization module, respectively. It formulates a cross-domain alignment objective with optimal transport distance which leverages cross-domain interaction to generate the representative keyframe and textual summary. We evaluated MHMS on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.
Parallel Field Alignment for Cross Media Retrieval

Xiangbo Mao,Binbin Lin,Deng Cai,Xiaofei He,Jian Pei

DOI: https://doi.org/10.1145/2502081.2502087

2013-01-01

Abstract:Cross media retrieval systems have received increasing interest in recent years. Due to the semantic gap between low-level features and high-level semantic concepts of multimedia data, many researchers have explored joint-model techniques in cross media retrieval systems. Previous joint-model approaches usually focus on two traditional ways to design cross media retrieval systems: (a) fusing features from different media data; (b) learning different models for different media data and fusing their outputs. However, the process of fusing features or outputs will lose both low- and high-level abstraction information of media data. Hence, both ways do not really reveal the semantic correlations among the heterogeneous multimedia data. In this paper, we introduce a novel method for the cross media retrieval task, named Parallel Field Alignment Retrieval (PFAR), which integrates a manifold alignment framework from the perspective of vector fields. Instead of fusing original features or outputs, we consider the cross media retrieval as a manifold alignment problem using parallel fields. The proposed manifold alignment algorithm can effectively preserve the metric of data manifolds, model heterogeneous media data and project their relationship into intermediate latent semantic spaces during the process of manifold alignment. After the alignment, the semantic correlations are also determined. In this way, the cross media retrieval task can be resolved by the determined semantic correlations. Comprehensive experimental results have demonstrated the effectiveness of our approach.
Supervised Semantic-Embedded Hashing for Multimedia Retrieval

Yunfei Chen,Jun Long,Lin Guo,Zhan Yang

DOI: https://doi.org/10.1016/j.knosys.2024.112023

IF: 8.139

2024-06-02

Knowledge-Based Systems

Abstract:With the rapid development of multimedia technologies, the efficient retrieval of large-scale multimedia information is regarded as a critical area to research. Hashing methods have achieved superiority as an effective solution for multimedia retrieval while offering the advantages of reduced storage demands and faster retrieval speeds. However, traditional hashing methods face challenges in addressing cross-modal heterogeneity and associating modal features with label information. In this paper, we introduce a novel supervised cross-modal hashing method named Supervised Semantic-Embedded Hashing (SSEH) for Multimedia Retrieval. Firstly, we designed a Semantic-Enhanced Representation module based on label mapping, which achieved deep integration of label information with multimedia features to ensure the completeness of semantic information in the hash codes. Secondly, a Class Structure Preservation module was constructed to comprehensively extract precise semantic information and association relationships from the labels, thereby ensuring the accuracy of semantic information and association relationships in the hash codes. We conducted extensive experiments on widely recognized datasets to demonstrate our SSEH method's efficacy and robustness. The code for our experiments is available at https://github.com/YunfeiChenMY/SSEH .

computer science, artificial intelligence
Multi-modal Deep Analysis for Multimedia

Wenwu Zhu,Xin Wang,Hongzhi Li

DOI: https://doi.org/10.1109/TCSVT.2019.2940647

2020-01-04

Abstract:With the rapid development of Internet and multimedia services in the past decade, a huge amount of user-generated and service provider-generated multimedia data become available. These data are heterogeneous and multi-modal in nature, imposing great challenges for processing and analyzing them. Multi-modal data consist of a mixture of various types of data from different modalities such as texts, images, videos, audios etc. In this article, we present a deep and comprehensive overview for multi-modal analysis in multimedia. We introduce two scientific research problems, data-driven correlational representation and knowledge-guided fusion for multimedia analysis. To address the two scientific problems, we investigate them from the following aspects: 1) multi-modal correlational representation: multi-modal fusion of data across different modalities, and 2) multi-modal data and knowledge fusion: multi-modal fusion of data with domain knowledge. More specifically, on data-driven correlational representation, we highlight three important categories of methods, such as multi-modal deep representation, multi-modal transfer learning, and multi-modal hashing. On knowledge-guided fusion, we discuss the approaches for fusing knowledge with data and four exemplar applications that require various kinds of domain knowledge, including multi-modal visual question answering, multi-modal video summarization, multi-modal visual pattern mining and multi-modal recommendation. Finally, we bring forward our insights and future research directions.

Multimedia,Computer Vision and Pattern Recognition,Information Retrieval,Machine Learning
Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval.

Xiaohua Zhai,Yuxin Peng,Jianguo Xiao

DOI: https://doi.org/10.1609/aaai.v27i1.8464

2013-01-01

Abstract:As the major component of big data, unstructured heterogeneous multimedia content such as text, image, audio, video and 3D increasing rapidly on the Internet. User demand a new type of cross-media retrieval where user can search results across various media by submitting query of any media. Since the query and the retrieved results can be of different media, how to learn a heterogeneous metric is the key challenge. Most existing metric learning algorithms only focus on a single media where all of the media objects share the same data representation. In this paper, we propose a joint graph regularized heterogeneous metric learning (JGRHML) algorithm, which integrates the structure of different media into a joint graph regularization. In JGRHML, different media are complementary to each other and optimizing them simultaneously can make the solution smoother for both media and further improve the accuracy of the final metric. Based on the heterogeneous metric, we further learn a high-level semantic metric through label propagation. JGRHML is effective to explore the semantic relationship hidden across different modalities. The experimental results on two datasets with up to five media types show the effectiveness of our proposed approach. © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-Media Retrieval

Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-Media Retrieval

Understanding Multimedia Document Semantics for Cross-Media Retrieval

Manifold Learning Based Cross-media Retrieval: A Solution to Media Object Complementary Nature

Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-Media Retrieval

Heterogeneous multimedia data semantics mining using content and location context.

Cross-Media Retrieval: Concepts, Advances And Challenges

Improving Web-Based Learning: Automatic Annotation of Multimedia Semantics and Cross-Media Indexing

The Research of Multimedia Cross Reference Retrieval System

A MULTI-CHANNEL RETRIEVAL SYSTEM FOR MULTIMEDIA DOCUMENTS

Multi-Feature Fusion Via Hierarchical Regression for Multimedia Analysis

Extracting Multimedia Semantics Based On Independent Modality Discovering And Fusion

Cross-media retrieval using query dependent search methods

Multimedia information retrieval at a crossroad

Content-oriented Multimedia Document Understanding Through Cross-Media Correlation

MediaView: A Semantic View Mechanism for Multimedia Modeling

MHMS: Multimodal Hierarchical Multimedia Summarization

Parallel Field Alignment for Cross Media Retrieval

Supervised Semantic-Embedded Hashing for Multimedia Retrieval

Multi-modal Deep Analysis for Multimedia

Heterogeneous Metric Learning with Joint Graph Regularization for Cross-Media Retrieval.