Multi-Modal And Multi-Granular Learning

Bo Zhang,Ling Zhang

DOI: https://doi.org/10.1007/978-3-540-71701-0_4

2007-01-01

Abstract:Under large scale data, machine learning becomes inefficient. In order to enhance its performances, new methodologies should be adopted. Taking video retrieval as an example, we discuss its two basic problems: representation and classification problems.

What problem does this paper attempt to address?

3D Model Retrieval with Multi-Granular Semantics Based on Gaussian Process Classifier

Bo-Yong GAO,San-Yuan ZHANG,Xiang PAN

DOI: https://doi.org/10.3969/j.issn.1003-6059.2011.05.001

2011-01-01

Pattern Recognition and Artificial Intelligence

Abstract:In order to solve the inconsistency between users' intentions in semantic 3D model retrieval system,a retrieval framework with multi-granular semantics is established,in which learning model can adapt to different user search intentions.Firstly,model classification is divided into different levels and the multi-granularity structure of semantic concept is formed.Then,a hybrid shape feature based on views is used to describe the shape characteristics of 3D model.And the Gaussian process classifier is used to associate low-level features with query concepts on a different level of semantic concept.Compared with existing research,the retrieval framework with multi-granular semantics allows the users to set their retrieval intentions according to selecting the granular level of semantics,and the results meet the user semantics as much as possible.The experimental results of retrieval performance evaluation using the benchmark show that the retrieval performance using proposed method is significantly higher than content-based retrieval and confident with human concept.
Effective Deep Learning-Based Multi-Modal Retrieval

Wei Wang,Xiaoyan Yang,Beng Chin Ooi,Dongxiang Zhang,Yueting Zhuang

DOI: https://doi.org/10.1007/s00778-015-0391-4

2015-01-01

The VLDB Journal

Abstract:Multi-modal retrieval is emerging as a new search paradigm that enables seamless information retrieval from various types of media. For example, users can simply snap a movie poster to search for relevant reviews and trailers. The mainstream solution to the problem is to learn a set of mapping functions that project data from different modalities into a common metric space in which conventional indexing schemes for high-dimensional space can be applied. Since the effectiveness of the mapping functions plays an essential role in improving search quality, in this paper, we exploit deep learning techniques to learn effective mapping functions. In particular, we first propose a general learning objective that effectively captures both intramodal and intermodal semantic relationships of data from heterogeneous sources. Given the general objective, we propose two learning algorithms to realize it: (1) an unsupervised approach that uses stacked auto-encoders and requires minimum prior knowledge on the training data and (2) a supervised approach using deep convolutional neural network and neural language model. Our training algorithms are memory efficient with respect to the data volume. Given a large training dataset, we split it into mini-batches and adjust the mapping functions continuously for each batch. Experimental results on three real datasets demonstrate that our proposed methods achieve significant improvement in search accuracy over the state-of-the-art solutions.
Cross-Modal Learning - The Learning Methodology Inspired by Human's Intelligence1

Bo Zhang,Dayong Ding,Ling Zhang

2007-01-01

Abstract:Human has an amazing cross-modal learning capability. In order to endow the computers with the same ability, we use a model based on the quotient space theory. In the quotient space model, representations at different modalities form a complete semi-order lattice and the translation from one modality to the others becomes easier. Therefore, it is suitable to be a mathematical model of cross-modal learning. Taking the video retrieval as an example, we show how to apply the cross-modal learning strategy to the field. The first problem of cross-modal learning in video retrieval is how to represent a video (content) so that the user expected videos can be found from a collection of videos precisely and entirely. A video can be represented by different modalities such as image, speech, text, etc. Each modality can be represented by several forms with different grain-sizes. Researches showed that, grain-size in the modality of image can bring compromise between precision and recall and multi-level feature may improve them both. But using only one modality to video retrieval is not enough. Speech and keyword are used as well. One of the strategies for cross-modal learning is to integrate information from different sense modalities. The second problem is how to integrate the results from different modalities. That is feature binding or information fusion problem. Multi-classifier technique will be discussed. We may consider each modality as a projection of the same object (video) and integrate information from the projections. Specifically, we propose the Probabilistic Model Supported Rank Aggregation (PMSRA) method to accomplish this integration. Theoretical analysis and experimental results show that cross-modal learning can significantly improve the performances of machine learning and that the quotient space model is powerful for it.
A New Class of Multi-Category Learning for Multiple Video Object Extraction

Yi Liu,Yuan F. Zheng,Xiaotong Sheny

2006-01-01

Abstract:Video object (VO) extraction is of great importance in multimedia processing. In recent years approaches have been proposed to deal with VO extraction as a classification problem. This type of methods calls for state-of-the-art classifiers because the performance is directly related to the accuracy of classification. Promising results have been reported for single object extraction using Support Vector Machines (SVM) and its extensions. Multiple object extraction, on the other hand, still imposes great difficulty as multi-category classification is an ongoing research topic in machine learning. This paper introduces a new class of multi-category learning scheme for multiple VO extraction, and demonstrates its effectiveness and advantages by experiments.
Multi-Granularity Aggregation Transformer for Joint Video-Audio-Text Representation Learning

Mengge He,Wenjing Du,Zhiquan Wen,Qing Du,Yutong Xie,Qi Wu

DOI: https://doi.org/10.1109/tcsvt.2022.3225549

IF: 5.859

2022-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Many real-world video-text tasks involve different levels of granularity to represent local and global information with distinct semantics, such as frames and words, clips and sentences, or videos and paragraphs. Most existing multimodal representation learning methods suffer from limitations: (i) Adopting expert systems or manual design to extract more fine-grained local information (such as objects and actions in a video frame) for supervision may lead to information asymmetry since there may no corresponding information among modalities; (ii) Neglecting the hierarchical nature of the data to aggregate different levels of information from different modalities will cause insufficient representations. To alleviate the above issues, in this paper, we propose a Multi-Granularity Aggregation Transformer (MGAT) for joint video-audio-text representation learning. Specifically, for intra-modality, we first design a multi-granularity transformer module to relieve information asymmetry by making full use of local and global information within a single modality from different perspectives. Then, for inter-modality, we develop an attention-guided aggregation module to fuse audio and video information hierarchically. Last, we align the aggregated information with text information at different hierarchical levels via intra- and inter-modality consistency loss and contrastive loss. With the help of more granularity of information, we are able to obtain a well-performed representation model for a variety of tasks, e.g., video-paragraph retrieval and video captioning. Extensive experiments on two challenging benchmarks, i.e., ActivityNet-captions and Youcook2, demonstrate the superiority of our proposed method.

engineering, electrical & electronic
Deep Multimodal Learning: An Effective Method for Video Classification

Tianqi Zhao

DOI: https://doi.org/10.48550/arXiv.1811.12563

2018-11-30

Abstract:Videos have become ubiquitous on the Internet. And video analysis can provide lots of information for detecting and recognizing objects as well as help people understand human actions and interactions with the real world. However, facing data as huge as TB level, effective methods should be applied. Recurrent neural network (RNN) architecture has wildly been used on many sequential learning problems such as Language Model, Time-Series Analysis, etc. In this paper, we propose some variations of RNN such as stacked bidirectional LSTM/GRU network with attention mechanism to categorize large-scale video data. We also explore different multimodal fusion methods. Our model combines both visual and audio information on both video and frame level and received great result. Ensemble methods are also applied. Because of its multimodal characteristics, we decide to call this method Deep Multimodal Learning(DML). Our DML-based model was trained on Google Cloud and our own server and was tested in a well-known video classification competition on Kaggle held by Google.

Computer Vision and Pattern Recognition,Multimedia
Multi-View Concept Learning for Data Representation

Ziyu Guan,Lijun Zhang,Jinye Peng,Jianping Fan

DOI: https://doi.org/10.1109/tkde.2015.2448542

IF: 9.235

2015-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Real-world datasets often involve multiple views of data items, e.g., a Web page can be described by both its content and anchor texts of hyperlinks leading to it; photos in Flickr could be characterized by visual features, as well as user contributed tags. Different views provide information complementary to each other. Synthesizing multi-view features can lead to a comprehensive description of the data items, which could benefit many data analytic applications. Unfortunately, the simple idea of concatenating different feature vectors ignores statistical properties of each view and usually incurs the "curse of dimensionality" problem. We propose Multi-view Concept Learning (MCL), a novel nonnegative latent representation learning algorithm for capturing conceptual factors from multi-view data. MCL exploits both multi-view information and label information. The key idea is to learn a common latent space across different views which (1) captures the semantic relationships between data items through graph embedding regularization on labeled items, and (2) allows each latent factor to be associated with a subset of views via sparseness constraints. In this way, MCL could capture flexible conceptual patterns hidden in multi-view features. Experiments on a toy problem and three real-world datasets show that MCL performs well and outperforms baseline methods.
One-Pass Multi-View Learning.

Yue Zhu,Wei Gao,Zhi-Hua Zhou

2015-01-01

Abstract:Multi-view learning has been an important learning paradigm where data come from multiple channels or appear in multiple modalities. Many approaches have been developed in this eld, and have achieved better performance than single-view ones. Those approaches, however, always work on small-size datasets with low dimensionality, owing to their high computational cost. In recent years, it has been witnessed that many applications involve large-scale multi-view data, e.g., hundreds of hours of video (including visual, audio and text views) is uploaded to YouTube every minute, bringing a big challenge to previous multi-view algorithms. This work concentrates on the large-scale multi-view learning for classication and proposes the One-Pass Multi-View (OPMV) framework which goes through the training data only once without storing the entire training examples. This approach jointly optimizes the composite objective functions with consistency linear constraints for dierent views. We verify, both theoretically and empirically, the eectiveness of the proposed algorithm.
Multi-Granular Computing In Web Age

Bo Zhang,Ling Zhang

DOI: https://doi.org/10.1007/978-3-642-41218-9_2

2013-01-01

Abstract:In web age, the traditional information processing faces a new challenge. Due to the change of man-machine interaction modes, computers have to know the intention or interest of users. So computer information processing has to use the human brain processing principle for reference. One of its key principles is the multi-granular computing. In the talk, we will discuss the problem both from artificial intelligence and traditional information processing viewpoints. And we show that the new trend of information processing is to combine these two methods.
Effective Multi-Modal Retrieval Based on Stacked Auto-Encoders

Wei Wang,Beng Chin Ooi,Xiaoyan Yang,Dongxiang Zhang,Yueting Zhuang

DOI: https://doi.org/10.14778/2732296.2732301

IF: 2.5

2014-01-01

Proceedings of the VLDB Endowment

Abstract:Multi-modal retrieval is emerging as a new search paradigm that enables seamless information retrieval from various types of media. For example, users can simply snap a movie poster to search relevant reviews and trailers. To solve the problem, a set of mapping functions are learned to project high-dimensional features extracted from data of different media types into a common low-dimensional space so that metric distance measures can be applied. In this paper, we propose an effective mapping mechanism based on deep learning (i.e., stacked auto-encoders) for multi-modal retrieval. Mapping functions are learned by optimizing a new objective function, which captures both intra-modal and inter-modal semantic relationships of data from heterogeneous sources effectively. Compared with previous works which require a substantial amount of prior knowledge such as similarity matrices of intra-modal data and ranking examples, our method requires little prior knowledge. Given a large training dataset, we split it into mini-batches and continually adjust the mapping functions for each batch of input. Hence, our method is memory efficient with respect to the data volume. Experiments on three real datasets illustrate that our proposed method achieves significant improvement in search accuracy over the state-of-the-art methods.
Multi-Granularity Context Network for Efficient Video Semantic Segmentation

Zhiyuan Liang,Xiangdong Dai,Yiqian Wu,Xiaogang Jin,Jianbing Shen

DOI: https://doi.org/10.1109/TIP.2023.3269982

Abstract:Current video semantic segmentation tasks involve two main challenges: how to take full advantage of multi-frame context information, and how to improve computational efficiency. To tackle the two challenges simultaneously, we present a novel Multi-Granularity Context Network (MGCNet) by aggregating context information at multiple granularities in a more effective and efficient way. Our method first converts image features into semantic prototypes, and then conducts a non-local operation to aggregate the per-frame and short-term contexts jointly. An additional long-term context module is introduced to capture the video-level semantic information during training. By aggregating both local and global semantic information, a strong feature representation is obtained. The proposed pixel-to-prototype non-local operation requires less computational cost than traditional non-local ones, and is video-friendly since it reuses the semantic prototypes of previous frames. Moreover, we propose an uncertainty-aware and structural knowledge distillation strategy to boost the performance of our method. Experiments on Cityscapes and CamVid datasets with multiple backbones demonstrate that the proposed MGCNet outperforms other state-of-the-art methods with high speed and low latency.
Multi-Modality Transfer Based on Multi-Graph Optimization for Domain Adaptive Video Concept Annotation

Shaoxi Xu,Sheng Tang,Yongdong Zhang,Jintao Li

DOI: https://doi.org/10.1109/PSIVT.2010.38

2010-01-01

Abstract:Multi-modality, the unique and important property of video data, is typically ignored in existing video adaptation processes. To solve this problem, we propose a novel approach, named multi-modality transfer based on multi- graph optimization (MMT-MGO) in this paper, which leverages multi-modality knowledge generalized by auxiliary classifiers in the source domain to assist multi-graph optimization (a graph-based semi-supervised learning method) in the target domain for video concept annotation. To our best knowledge, it is the first time to introduce multi-modality transfer into domain adaptive video concept detection and annotation. Moreover, we propose an efficient incremental extension scheme to sequentially estimate a small batch of new emerging data without modifying the structure of multi-graph scheme. The proposed scheme can achieve a comparable accuracy with that of the brand-new round optimization which combines these data with the data corpus for the nearest round optimization, while the time for estimation has been greatly reduced. Extensive experiments over TRECVID2005 and 2007 data sets demonstrate the effectiveness of both the multi-modality transfer scheme and the incremental extension scheme.
Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval

Yawen Zeng,Da Cao,Xiaochi Wei,Meng Liu,Zhou Zhao,Zheng Qin

DOI: https://doi.org/10.1109/CVPR46437.2021.00225

2021-01-01

Abstract:Given an untrimmed video and a query sentence, cross-modal video moment retrieval aims to rank a video moment from pre-segmented video moment candidates that best matches the query sentence. Pioneering work typically learns the representations of the textual and visual content separately and then obtains the interactions or alignments between different modalities. However, the task of cross-modal video moment retrieval is not yet thoroughly addressed as it needs to further identify the fine-grained differences of video moment candidates with high repeatability and similarity. Moveover, the relation among objects in both video and sentence is intuitive and efficient for understanding semantics but is rarely considered. Toward this end, we contribute a multi-modal relational graph to capture the interactions among objects from the visual and textual content to identify the differences among similar video moment candidates. Specifically, we first introduce a visual relational graph and a textual relational graph to form relation-aware representations via message propagation. Thereafter, a multi-task pre-training is designed to capture domain-specific knowledge about objects and relations, enhancing the structured visual representation after explicitly defined relation. Finally, the graph matching and boundary regression are employed to perform the cross-modal retrieval. We conduct extensive experiments on two datasets about daily activities and cooking activities, demonstrating significant improvements over state-of-the-art solutions.
A Survey on Multi-view Learning

Chang Xu,Dacheng Tao,Chao Xu

DOI: https://doi.org/10.48550/arxiv.1304.5634

2013-01-01

Abstract:In recent years, a great many methods of learning from multi-view data by considering the diversity of different views have been proposed. These views may be obtained from multiple sources or different feature subsets. In trying to organize and highlight similarities and differences between the variety of multi-view learning approaches, we review a number of representative multi-view learning algorithms in different areas and classify them into three groups: 1) co-training, 2) multiple kernel learning, and 3) subspace learning. Notably, co-training style algorithms train alternately to maximize the mutual agreement on two distinct views of the data; multiple kernel learning algorithms exploit kernels that naturally correspond to different views and combine kernels either linearly or non-linearly to improve learning performance; and subspace learning algorithms aim to obtain a latent subspace shared by multiple views by assuming that the input views are generated from this latent subspace. Though there is significant variance in the approaches to integrating multiple views to improve learning performance, they mainly exploit either the consensus principle or the complementary principle to ensure the success of multi-view learning. Since accessing multiple views is the fundament of multi-view learning, with the exception of study on learning a model from multiple views, it is also valuable to study how to construct multiple views and how to evaluate these views. Overall, by exploring the consistency and complementary properties of different views, multi-view learning is rendered more effective, more promising, and has better generalization ability than single-view learning.
Collaborative learning for image and video annotation.

Bo Geng,Linjun Yang,Chao Xu,Xian-Sheng Hua

DOI: https://doi.org/10.1145/1460096.1460168

2008-01-01

Abstract:Classical machine learning methods, such as Support Vector Machines, by taking each concept detection as an independent classification problem, can not achieve a sound performance for image and video annotation due to the overfitting problems. Thus, some prior knowledge is required to assist the learning of independent concept detectors, e.g. some concepts look much more alike to each other. In this paper, we assume that visually similar concepts should share resembled detectors. Based on the assumption, Collaborative Learning is proposed, to incorporate cross-concept collaborations into the joint learning of similar detectors over related concepts. Besides the collaborations, different concepts should also perform discriminations for classifying each other. To benefit from different trade-offs between collaboration and discrimination, we propose Multi-Granularity Boosting strategy, where each granularity corresponds to a specific balance between collaboration and discrimination for Collaborative Learning. The ultimate concept detector is an additive model that combines classifiers under different collaboration granularities together. Evaluations on both image and video annotation benchmark demonstrate that our method achieves a superior performance over independent annotation.
Multi-View Embedding Learning for Incompletely Labeled Data.

Wei Zhang,Ke Zhang,Pan Gu,Xiangyang Xue

2013-01-01

Abstract:In many applications, the data may be high dimensional, represented by multiple features, and associated with more than one labels. Embedding learning is an effective strategy for dimensionality reduction and for nearest neighbor search in massive datasets. We propose a novel method to seek compact embedding that allows efficient retrieval with incompletely-labeled multi-view data. Based on multi-graph Laplacian, we achieve the optimal combination of heterogeneous features to effectively describe data, which exploits the feature correlations between different views. We learn the embedding that preserves the neighborhood context in the original spaces, and obtain the complete labels simultaneously. Inter-label correlations are sufficiently leveraged in the proposed framework. Our goal is to find the maps from multiple input spaces to the compact embedding space and to the semantic concept space at the same time. There is semantic gap between the input multi-view feature spaces and the semantic concept space; and the compact embedding space can be looked on as the bridge between the above spaces. Experimental evaluation on three real-world datasets demonstrates the effectiveness of the proposed method.
Dual graph-structured semantics multi-subspace learning for cross-modal retrieval

Yirong Li,Xianghong Tang,Jianguang Lu,Yong Huang

DOI: https://doi.org/10.1007/s00530-024-01471-0

IF: 3.9

2024-09-28

Multimedia Systems

Abstract:As the era of big data develops rapidly, cross-modal retrieval is a research field that has received widespread attention. Most current methods of cross-modal retrieval just pursue the macro alignment of modal data in a shared space to gain a common representation. However, they cannot achieve the satisfactory performance of cross-modal retrieval since they neglect the deep semantic alignment and the inherent differences between modalities. Being aware of these, this paper presents a dual graph-structured semantics multi-subspace learning (DGMS) method for cross-modal retrieval. Specifically in DGMS, the double semantics graph is established to represent the deep semantics of modal data, and the multiple subspace learning network constructs public and independent subspaces to capture the relevance and dissimilarity of modal data. Finally, a dual learning method based on the generative adversarial network is employed further to catch the joint probability distribution of the different modalities. The superiority of DGMS is demonstrated by experiments on Wikipedia, XMedia, and Pascal Sentence, for it can not only learn deep structural semantics but also explore the consistency and diversity of modalities.

computer science, information systems, theory & methods
Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Chenzhuang Du,Yue Zhao,Chonghua Liao,Jiacheng You,Jie Fu,Hang Zhao

DOI: https://doi.org/10.48550/arXiv.2310.05193

2023-10-08

Computer Vision and Pattern Recognition

Abstract:This paper investigates how to better leverage large-scale pre-trained uni-modal models to further enhance discriminative multi-modal learning. Even when fine-tuned with only uni-modal data, these models can outperform previous multi-modal models in certain tasks. It's clear that their incorporation into multi-modal learning would significantly improve performance. However, multi-modal learning with these models still suffers from insufficient learning of uni-modal features, which weakens the resulting multi-modal model's generalization ability. While fine-tuning uni-modal models separately and then aggregating their predictions is straightforward, it doesn't allow for adequate adaptation between modalities, also leading to sub-optimal results. To this end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA). By freezing the weights of uni-modal fine-tuned models, adding extra trainable rank decomposition matrices to them, and subsequently performing multi-modal joint training, our method enhances adaptation between modalities and boosts overall performance. We demonstrate the effectiveness of MMLoRA on three dataset categories: audio-visual (e.g., AVE, Kinetics-Sound, CREMA-D), vision-language (e.g., MM-IMDB, UPMC Food101), and RGB-Optical Flow (UCF101).
Multi-Feature Fusion Via Hierarchical Regression for Multimedia Analysis

Yi Yang,Jingkuan Song,Zi Huang,Zhigang Ma,Nicu Sebe,Alexander G. Hauptmann

DOI: https://doi.org/10.1109/tmm.2012.2234731

IF: 7.3

2013-01-01

IEEE Transactions on Multimedia

Abstract:Multimedia data are usually represented by multiple features. In this paper, we propose a new algorithm, namely Multi-feature Learning via Hierarchical Regression for multimedia semantics understanding, where two issues are considered. First, labeling large amount of training data is labor-intensive. It is meaningful to effectively leverage unlabeled data to facilitate multimedia semantics understanding. Second, given that multimedia data can be represented by multiple features, it is advantageous to develop an algorithm which combines evidence obtained from different features to infer reliable multimedia semantic concept classifiers. We design a hierarchical regression model to exploit the information derived from each type of feature, which is then collaboratively fused to obtain a multimedia semantic concept classifier. Both label information and data distribution of different features representing multimedia data are considered. The algorithm can be applied to a wide range of multimedia applications and experiments are conducted on video data for video concept annotation and action recognition. Using Trecvid and CareMedia video datasets, the experimental results show that it is beneficial to combine multiple features. The performance of the proposed algorithm is remarkable when only a small amount of labeled training data are available.

Multi-Modal And Multi-Granular Learning

3D Model Retrieval with Multi-Granular Semantics Based on Gaussian Process Classifier

Effective Deep Learning-Based Multi-Modal Retrieval

Cross-Modal Learning - The Learning Methodology Inspired by Human's Intelligence1

A New Class of Multi-Category Learning for Multiple Video Object Extraction

Multi-Granularity Aggregation Transformer for Joint Video-Audio-Text Representation Learning

Deep Multimodal Learning: An Effective Method for Video Classification

Multi-View Concept Learning for Data Representation

One-Pass Multi-View Learning.

Multi-Granular Computing In Web Age

Effective Multi-Modal Retrieval Based on Stacked Auto-Encoders

Multi-Granularity Context Network for Efficient Video Semantic Segmentation

Multi-Modality Transfer Based on Multi-Graph Optimization for Domain Adaptive Video Concept Annotation

Multi-Modal Relational Graph for Cross-Modal Video Moment Retrieval

A Survey on Multi-view Learning

Collaborative learning for image and video annotation.

Multi-View Embedding Learning for Incompletely Labeled Data.

Dual graph-structured semantics multi-subspace learning for cross-modal retrieval

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Multi-Feature Fusion Via Hierarchical Regression for Multimedia Analysis