Multi-modal Hash Learning

Lei Zhu,Jingjing Li,Weili Guan

2024-01-01

Synthesis Lectures on Information Concepts, Retrieval, and Services

Abstract:This book presents key concepts of multi-modal hashing technology and recent advances in large-scale efficient multimedia search and recommendation.

What problem does this paper attempt to address?

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Dekui Ma,Jian Liang,Xiangwei Kong,Ran He,Ying Li

DOI: https://doi.org/10.1109/ism.2016.0017

2016-01-01

Abstract:Hashing techniques have been widely adopted for cross-modal retrieval due to its low storage cost and fast query speed. Most existing cross-modal hashing methods aim to map heterogeneous data into the common low-dimensional hamming space and then threshold to obtain binary codes by relaxing the discrete constraint. However, this independent relaxation step also brings quantization errors, resulting in poor retrieval performances. Other cross-modal hashing methods try to directly optimize the challenging objective function with discrete binary constraints. Inspired by [1], we propose a novel supervised cross-modal hashing method called Discrete Cross-Modal Hashing (DCMH) to learn the discrete binary codes without relaxing them. DCMH is formulated through reconstructing the semantic similarity matrix and learning binary codes as ideal features for classification. Furthermore, DCMH alternately updates binary codes of each modality, and iteratively learns the discrete hashing codes bit by bit efficiently, which is quite promising for large-scale datasets. Extensive empirical results on three real-world datasets show that DCMH outperforms the baseline approaches significantly.
Nonlinear Discrete Cross-Modal Hashing for Visual-Textual Data

Dekui Ma,Jian Liang,Ran He,Xiangwei Kong

DOI: https://doi.org/10.1109/mmul.2017.28

IF: 3.4911

2017-01-01

IEEE Multimedia

Abstract:Hashing techniques have been widely adopted for cross-modal retrieval due to their low storage cost and fast query speed. Recently, some unimodal hashing methods have tried to directly optimize the objective function with discrete binary constraints. Inspired by these methods, the authors propose a novel supervised cross-modal hashing method called Discrete Cross-Modal Hashing (DCMH) to learn the binary codes without relaxing them. DCMH is formulated through semantic similarity reconstruction, and it learns binary codes for use as ideal features for classification. Furthermore, DCMH alternately updates binary codes for each modality, and its discrete hashing codes are learned efficiently, bit by bit, which is quite promising for large-scale datasets. To evaluate the effectiveness of the proposed discrete optimization, the authors optimize their objective function in a relax-and-threshold manner. Extensive empirical results on both image-text and image-tag datasets demonstrate that DCMH is a significant improvement over previous approaches in terms of training time and retrieval performance.
Online latent semantic hashing for cross-media retrieval.

Tao Yao,Gang Wang,Lianshan Yan,Xiangwei Kong,Qingtang Su,Caiming Zhang,Qi Tian

DOI: https://doi.org/10.1016/j.patcog.2018.12.012

IF: 8

2019-01-01

Pattern Recognition

Abstract:Hashing based cross-media method has been become an increasingly popular technique in facilitating large-scale multimedia retrieval task, owing to its effectiveness and efficiency. Most existing cross-media hashing methods learn hash functions in a batch based mode. However, in practical applications, data points often emerge in a streaming manner, which makes batch based hashing methods loss their efficiency. In this paper, we propose an Online Latent Semantic Hashing (OLSH) method to address this issue. Only newly arriving multimedia data points are utilized to retrain hash functions efficiently and meanwhile preserve the semantic correlations in old data points. Specifically, for learning discriminative hash codes, discrete labels are mapped to a continuous latent semantic space where the relative semantic distances in data points can be measured more accurately. And then, we propose an online optimization scheme towards the challenging task of learning hash functions efficiently on streaming data points, and the computational complexity and memory cost are much less than the size of training dataset at each round. Extensive experiments across many real-world datasets, e.g. Wiki, Mir-Flickr25K and NUS-WIDE, show the effectiveness and efficiency of the proposed method.
Multi-modal Hashing for Efficient Multimedia Retrieval: A Survey

Lei Zhu,Chaoqun Zheng,Weili Guan,Jingjing Li,Yang Yang,Heng Tao Shen

DOI: https://doi.org/10.1109/tkde.2023.3282921

IF: 9.235

2023-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:With the explosive growth of multimedia contents, multimedia retrieval is facing unprecedented challenges on both storage cost and retrieval speed. Hashing technique can project the high-dimensional data into compact binary hash codes. With it, the most time-consuming semantic similarity computation during the multimedia retrieval process can be significantly accelerated with fast Hamming distance computation, and meanwhile the storage cost can be reduced greatly by the binary embedding. In the light of this, multi-modal hashing has recently received considerable attention to support large-scale multimedia retrieval. Different from uni-modal hashing, the multi-modal hashing focuses on modeling the multi-modal semantics and further preserving them into binary hash codes with hash learning. In this paper, we first systematically review the existing learning to hash methods for efficient multimedia retrieval, categorizing them according to the multimedia retrieval tasks, the specific multi-modal semantic modeling techniques, and hash learning strategies. Thereafter, we present the performance comparison results. We ultimately discuss the challenges and potential research directions that may require further investigation in multi-modal hash learning. To facilitate the research on multi-modal hashing, we develop an open-source performance comparison tool at https://github.com/BMC-SDNU/Hashing-Retrieval.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
Composite Multi-modal Hashing

Lei Zhu,Jingjing Li,Weili Guan

2023-01-01

Synthesis Lectures on Information Concepts, Retrieval, and Services

Abstract:Multi-modal data describe the multi-perception observed information of the object in the real world from various aspects. Different from uni-modal [1–3] and cross-modal hashing [4–8] mentioned in the above chapters, composite multi-modal hashing technology [9–15] focuses on how to collaboratively fuse heterogeneous multi-modal features and perform the retrieval among different instances based on fused features.
Flexible Multi-modal Hashing for Scalable Multimedia Retrieval

Lei Zhu,Xu Lu,Zhiyong Cheng,Jingjing Li,Huaxiang Zhang

DOI: https://doi.org/10.1145/3365841

IF: 5

2020-01-01

ACM Transactions on Intelligent Systems and Technology

Abstract:Multi-modal hashing methods could support efficient multimedia retrieval by combining multi-modal features for binary hash learning at the both offline training and online query stages. However, existing multi-modal methods cannot binarize the queries, when only one or part of modalities are provided. In this article, we propose a novel Flexible Multi-modal Hashing (FMH) method to address this problem. FMH learns multiple modality-specific hash codes and multi-modal collaborative hash codes simultaneously within a single model. The hash codes are flexibly generated according to the newly coming queries, which provide any one or combination of modality features. Besides, the hashing learning procedure is efficiently supervised by the pair-wise semantic matrix to enhance the discriminative capability. It could successfully avoid the challenging symmetric semantic matrix factorization and O(n2) storage cost of semantic matrix. Finally, we design a fast discrete optimization to learn hash codes directly with simple operations. Experiments validate the superiority of the proposed approach.
Fast Discrete Collaborative Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Chaoqun Zheng,Lei Zhu,Xu Lu,Jingjing Li,Zhiyong Cheng,Hanwang Zhang

DOI: https://doi.org/10.1109/tkde.2019.2913388

IF: 9.235

2020-11-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Many achievements have been made on learning to hash for uni-modal and cross-modal retrieval. However, it is still an unsolved problem that how to directly and efficiently learn discriminative discrete hash codes for the multimedia retrieval, where both query and database samples are represented with heterogeneous multi-modal features. With this motivation, we propose a Fast Discrete Collaborative Multi-modal Hashing (FDCMH) method in this paper. We first propose an efficient collaborative multi-modal mapping that first transforms heterogeneous multi-modal features into the unified factors to exploit the complementarity of multi-modal features and preserve the semantic correlations in multiple modalities with linear computation and space complexity. Such shared factors also bridge the heterogeneous modality gap and remove the inter-modality redundancy. Further, we develop an asymmetric hashing learning module to simultaneously correlate the learned hash codes with low-level data distribution and high-level semantics. In particular, this design could avoid the challenging symmetric semantic matrix factorization and $O(n^2)$<math>O(n2)</math> memory cost ($n$<math>n</math> is the number of training samples). It can support both computation and memory efficient discrete hash optimization. Experiments on several public multimedia retrieval datasets demonstrate the superiority of the proposed approach compared with state-of-the-art h-shing techniques, in terms of both model learning efficiency and retrieval accuracy.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
Partial Multi-Modal Hashing via Neighbor-aware Completion Learning

Wentao Tan,Lei Zhu,Jingjing Li,Zheng Zhang,Huaxiang Zhang

DOI: https://doi.org/10.1109/tmm.2023.3238308

IF: 7.3

2023-01-01

IEEE Transactions on Multimedia

Abstract:Multi-modal hashing technology can support large-scale multimedia retrieval well, because of its fast query speed and low storage consumption. Although many multi-modal hashing methods have been developed in the literature, they still suffer from three important problems: 1) Most multi-modal hashing methods assume that multi-modal data are complete. This ideal assumption limits their application to practical retrieval scenarios, where the modality-missing is common. 2) Existing partial multi-modal hashing methods directly model incomplete multi-modal data for hash learning, which may result in partial multi-modal semantics in the learned hash codes. 3) Most of the methods are based on the shallow learning framework, which inevitably suffers from limited representation capability. To solve the above problems, we propose a flexible deep partial multi-modal hash learning framework, named Neighbor-aware Completion Hashing (NCH). Our framework jointly performs the cross-modal completion learning for incomplete multi-modal data and the multi-modal hash learning. It can not only support model training with incomplete multi-modal data but also handle incomplete multi-modal queries. Besides, we design a neighbor-aware completion learning module to capture neighbor semantics and generate distribution-consistent completed features. Finally, we conduct extensive experiments to evaluate our method on both fully-paired and partial multi-modal retrieval scenarios. The experimental results verify the superiority of our proposed method over the state-of-the-art baselines. The experimental codes of our method are released at: https://github.com/FutureTwT/NCH.

computer science, information systems,telecommunications, software engineering
Efficient Multi-modal Hashing with Online Query Adaption for Multimedia Retrieval

Lei Zhu,Chaoqun Zheng,Xu Lu,Zhiyong Cheng,Liqiang Nie,Huaxiang Zhang

DOI: https://doi.org/10.1145/3477180

IF: 4.657

2022-04-30

ACM Transactions on Information Systems

Abstract:Multi-modal hashing supports efficient multimedia retrieval well. However, existing methods still suffer from two problems: (1) Fixed multi-modal fusion. They collaborate the multi-modal features with fixed weights for hash learning, which cannot adaptively capture the variations of online streaming multimedia contents. (2) Binary optimization challenge. To generate binary hash codes, existing methods adopt either two-step relaxed optimization that causes significant quantization errors or direct discrete optimization that consumes considerable computation and storage cost. To address these problems, we first propose a Supervised Multi-modal Hashing with Online Query-adaption method. A self-weighted fusion strategy is designed to adaptively preserve the multi-modal features into hash codes by exploiting their complementarity. Besides, the hash codes are efficiently learned with the supervision of pair-wise semantic labels to enhance their discriminative capability while avoiding the challenging symmetric similarity matrix factorization. Further, we propose an efficient Unsupervised Multi-modal Hashing with Online Query-adaption method with an adaptive multi-modal quantization strategy. The hash codes are directly learned without the reliance on the specific objective formulations. Finally, in both methods, we design a parameter-free online hashing module to adaptively capture query variations at the online retrieval stage. Experiments validate the superiority of our proposed methods.

computer science, information systems
One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Chaoqun Zheng,Fengling Li,Lei Zhu,Zheng Zhang,Wenpeng Lu

DOI: https://doi.org/10.1016/j.eswa.2023.120913

IF: 8.5

2023-06-01

Expert Systems with Applications

Abstract:Hashing has been widely studied in support of efficient multimedia retrieval due to its high computation and storage efficiency. However, existing multimedia hashing models are designed only for specific multimedia retrieval tasks, e.g., cross-modal retrieval and composite multi-modal retrieval. To simultaneously support multiple multimedia retrieval tasks, multiple different hashing models should be equipped in a multimedia retrieval system. This paper proposes a Structured Multi-Modal Hashing (SMMH) method, which can learn only a hash model for multiple multimedia retrieval tasks. Specifically, we generate structured hash codes by independently projecting the modality data into the Hamming space, where each part of hash codes preserves the specific characteristic of the specific modality data, and they are collaborated to represent the whole multi-modal data. To bridge the heterogeneous gap between multiple modalities, a fusion multi-modal space is constructed based on structured hash codes with label and pair-wise semantic supervision. Besides, we learn hash functions for composite multi-modal retrieval by projecting heterogeneous modalities into the collaborative multi-modal features. Finally, we develop a discrete hash optimization strategy to efficiently solve binary hash codes. Extensive experiments show that our model outperforms the strong baseline methods on multiple multimedia retrieval tasks. Source codes are available at https://github.com/ChaoqunZheng/SMMH.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
Scalable Multimedia Retrieval By Deep Learning Hashing With Relative Similarity Learning

Lianli Gao,Jingkuan Song,Fuhao Zou,Dongxiang Zhang,Jie Shao

DOI: https://doi.org/10.1145/2733373.2806360

2015-01-01

Abstract:Learning-based hashing methods are becoming the mainstream for approximate scalable multimedia retrieval. They consist of two main components: hash codes learning for training data and hash functions learning for new data points. Tremendous efforts have been devoted to designing novel methods for these two components, i.e., supervised and unsupervised methods for learning hash codes, and different models for inferring hashing functions. However, there is little work integrating supervised and unsupervised hash codes learning into a single framework. Moreover, the hash function learning component is usually based on hand-crafted visual features extracted from the training images. The performance of a content-based image retrieval system crucially depends on the feature representation and such hand-crafted visual features may degrade the accuracy of the hash functions. In this paper, we propose a semi-supervised deep learning hashing (DLH) method for fast multimedia retrieval. More specifically, in the first component, we utilize both visual and label information to learn an relative similarity graph that can more precisely reflect the relationship among training data, and then generate the hash codes based on the graph. In the second stage, we apply a deep convolutional neural network (CNN) to simultaneously learn a good multimedia representation and hash functions. Extensive experiments on three popular datasets demonstrate the superiority of our DLH over both supervised and unsupervised hashing methods.
Enhanced Discrete Multi-Modal Hashing: More Constraints Yet Less Time to Learn

Yong Chen,Hui Zhang,Zhibao Tian,Jun Wang,Dell Zhang,Xuelong Li

DOI: https://doi.org/10.1109/tkde.2020.2995195

IF: 9.235

2022-03-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Due to the exponential growth of multimedia data, multi-modal hashing as a promising technique to make cross-view retrieval scalable is attracting more and more attention. However, most of the existing multi-modal hashing methods either divide the learning process unnaturally into two separate stages or treat the discrete optimization problem simplistically as a continuous one, which leads to suboptimal results. Recently, a few discrete multi-modal hashing methods that try to address such issues have emerged, but they still ignore several important discrete constraints (such as the balance and decorrelation of hash bits). In this paper, we overcome those limitations by proposing a novel method named "Enhanced Discrete Multi-modal Hashing (EDMH)" which learns binary codes and hashing functions simultaneously from the pairwise similarity matrix of data, under the aforementioned discrete constraints. Although the model of EDMH looks a lot more complex than the other models for multi-modal hashing, we are actually able to develop a fast iterative learning algorithm for it, since the subproblems of its optimization all have closed-form solutions after introducing a couple of auxiliary variables. Our experimental results on three real-world datasets have revealed the usefulness of those previously ignored discrete constraints and demonstrated that EDMH not only performs much better than state-of-the-art competitors according to several retrieval metrics but also runs much faster than most of them.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
Unsupervised Multi-modal Hashing for Cross-Modal Retrieval

Jun Yu,Xiao-Jun Wu,Donglin Zhang

DOI: https://doi.org/10.1007/s12559-021-09847-4

IF: 4.89

2021-03-04

Cognitive Computation

Abstract:The explosive growth of multimedia data on the Internet has magnified the challenge of information retrieval. Multimedia data usually emerges in different modalities, such as image, text, video, and audio. Unsupervised cross-modal hashing techniques that support searching among multi-modal data have gained importance in large-scale retrieval tasks because of the advantage of low storage cost and high efficiency. Current methods learn the hash functions by transforming high-dimensional data into discrete hash codes. However, the original manifold structure and semantic correlation are not preserved well in compact hash codes. We propose a novel unsupervised cross-modal hashing method to cope with this problem from two perspectives. On the one hand, the semantic correlation in textual space and the locally geometric structure in the visual space are reconstructed by unified hashing features seamlessly and simultaneously. On the other hand, the <span class="mathjax-tex">$\ell _{2,1}$</span>-norm penalties are imposed on the projection matrices separately to learn the relevant and discriminative hash codes. The experimental results indicate that our proposed method achieves an improvement of 1%, 6%, 9%, and 2% over the best comparison method on the four publicly available datasets (WiKi, PASCAL-VOC, UCI Handwritten Digit, and NUS-WIDE), respectively. In conclusion, the proposed framework which combines hash functions learning and multimodal graph embedding is effective in learning hash codes and achieves superior retrieval performance compared to state-of-the-art methods.

computer science, artificial intelligence,neurosciences
Multi-View Latent Hashing for Efficient Multimedia Search

Xiaobo Shen,Fumin Shen,Quan-Sen Sun,Yun-Hao Yuan

DOI: https://doi.org/10.1145/2733373.2806342

2015-01-01

Abstract:Hashing techniques have attracted broad research interests in recent multimedia studies. However, most of existing hashing methods focus on learning binary codes from data with only one single view, and thus cannot fully utilize the rich information from multiple views of data. In this paper, we propose a novel unsupervised hashing approach, dubbed multi-view latent hashing (MVLH), to effectively incorporate multi-view data into hash code learning. Specifically, the binary codes are learned by the latent factors shared by multiple views from an unified kernel feature space, where the weights of different views are adaptively learned according to the reconstruction error with each view. We then propose to solve the associate optimization problem with an efficient alternating algorithm. To obtain high-quality binary codes, we provide a novel scheme to directly learn the codes without resorting to continuous relaxations, where each bit is efficiently computed in a closed form. We evaluate the proposed method on several large-scale datasets and the results demonstrate the superiority of our method over several other state-of-the-art methods.
CLIP Multi-modal Hashing for Multimedia Retrieval

Jian Zhu,Mingkai Sheng,Zhangmin Huang,Jingfei Chang,Jinling Jiang,Jian Long,Cheng Luo,Lei Liu

2024-10-10

Abstract:Multi-modal hashing methods are widely used in multimedia retrieval, which can fuse multi-source data to generate binary hash code. However, the individual backbone networks have limited feature expression capabilities and are not jointly pre-trained on large-scale unsupervised multi-modal data, resulting in low retrieval accuracy. To address this issue, we propose a novel CLIP Multi-modal Hashing (CLIPMH) method. Our method employs the CLIP framework to extract both text and vision features and then fuses them to generate hash code. Due to enhancement on each modal feature, our method has great improvement in the retrieval performance of multi-modal hashing methods. Compared with state-of-the-art unsupervised and supervised multi-modal hashing methods, experiments reveal that the proposed CLIPMH can significantly improve performance (a maximum increase of 8.38% in mAP).

Computer Vision and Pattern Recognition
Transitive Hashing Network for Heterogeneous Multimedia Retrieval

Zhangjie Cao,Mingsheng Long,Qiang Yang

DOI: https://doi.org/10.1609/aaai.v31i1.10487

2016-01-01

Abstract: Hashing has been widely applied to large-scale multimedia retrieval due to the storage and retrieval efficiency. Cross-modal hashing enables efficient retrieval from database of one modality in response to a query of another modality. Existing work on cross-modal hashing assumes heterogeneous relationship across modalities for hash function learning. In this paper, we relax the strong assumption by only requiring such heterogeneous relationship in an auxiliary dataset different from the query/database domain. We craft a hybrid deep architecture to simultaneously learn the cross-modal correlation from the auxiliary dataset, and align the dataset distributions between the auxiliary dataset and the query/database domain, which generates transitive hash codes for heterogeneous multimedia retrieval. Extensive experiments exhibit that the proposed approach yields state of the art multimedia retrieval performance on public datasets, i.e. NUS-WIDE, ImageNet-YahooQA.
Efficient Supervised Discrete Multi-View Hashing for Large-Scale Multimedia Search

Xu Lu,Lei Zhu,Jingjing Li,Huaxiang Zhang,Heng Tao Shen

DOI: https://doi.org/10.1109/tmm.2019.2947358

IF: 7.3

2020-08-01

IEEE Transactions on Multimedia

Abstract:Hashing has recently received substantial attention in large-scale multimedia search for its extremely low-cost storage cost and high retrieval efficiency. However, most existing hashing techniques focus on learning hash codes for single-view or cross-view retrieval. It is still an unsolved problem that how to efficiently learn discriminative binary codes for multi-view data that is common in real world multimedia search. In this paper, we propose an efficient Supervised Discrete Multi-view Hashing (SDMH) to solve the problem. SDMH first properly detects the shared binary hash codes, with an integrated multi-view feature mapping and latent hash coding, by exploiting the complementarity of different view-specific features and removing the involved inter-view redundancy. To further enhance the discriminative capability of hash codes, SDMH directly represses the explicit semantic labels of data samples with their corresponding binary codes. Different from most existing multi-view hashing methods that adopt "relaxing+rounding" hash optimization strategy or the discrete optimization method based on discrete cyclic coordinate descent, an efficient augmented Lagrangian multiplier (ALM) based discrete hash optimization method is developed in this paper to optimize the hash codes within a single step. Experimental results on four benchmark datasets demonstrate the superior performance of the proposed approach over state-of-the-art hashing techniques, in terms of both learning efficiency and retrieval accuracy.

computer science, information systems,telecommunications, software engineering
Multi-modal discrete tensor decomposition hashing for efficient multimedia retrieval

Xize Wu,Lei Zhu,Liang Xie,Zheng Zhang,Huaxiang Zhang

DOI: https://doi.org/10.1016/j.neucom.2021.08.125

IF: 6

2021-11-01

Neurocomputing

Abstract:In the research field of multimedia retrieval, unsupervised multi-modal hashing has received widespread attention because of its high retrieval efficiency, low storage cost and semantic label independence. However, there are still several problems that need to be resolved: 1) All existing methods are based on matrix factorizations, which has limited capability on fusing heterogeneous multi-modal features under the unsupervised learning paradigm. 2) Most methods adopt two-step learning strategy to separately learn the hash codes and functions, which may generate sub-optimal hash functions. 3) The optimization strategies used by most methods produce significant quantization loss and computation cost. In this paper, an efficient unsupervised Multi-modal Discrete Tensor Decomposition Hashing (MDTDH) approach is proposed to solve the above problems. Specifically, all multi-modal features are first stacked together into a three-dimensional tensor after nonlinear mapping, and then it is decomposed further into a core tensor and two factor matrices by Tucker decomposition. A series of hash functions are learned simultaneously by mapping the nonlinear features of the training instances to their corresponding hash codes. To reduce the quantization loss and computation cost, a fast discrete optimization strategy is proposed to generate hash codes directly without relaxing quantization loss. Extensive experimental results on three benchmarks demonstrate that our proposed MDTDH obtains superior performance than state-of-the-art unsupervised baselines, and even defeats several supervised baselines. Our source codes and testing datasets are available at https://github.com/XizeWu/MDTDH.

computer science, artificial intelligence
Cross-Media Hashing with Neural Networks

Yueting Zhuang,Zhou Yu,Wei Wang,Fei Wu,Siliang Tang,Jian Shao

DOI: https://doi.org/10.1145/2647868.2655059

2014-01-01

Abstract:Cross-media hashing, which conducts cross-media retrieval by embedding data from different modalities into a common low-dimensional hamming space, has attracted intensive attention in recent years. This is motivated by the facts a) the multi-modal data is widespread, e.g., the web images on Flickr are associated with tags, and b) hashing is an effective technique towards large-scale high-dimensional data processing, which is exactly the situation of cross-media retrieval. Inspired by recent advances in deep learning, we propose a cross-media hashing approach based on multi-modal neural networks. By restricting in the learning objective a) the hash codes for relevant cross-media data being similar, and b) the hash codes being discriminative for predicting the class labels, the learned Hamming space is expected to well capture the cross-media semantic relationships and to be semantically discriminative. The experiments on two real-world data sets show that our approach achieves superior cross-media retrieval performance compared with the stateof-the-art methods.
Sparse Multi-Modal Hashing

Fei Wu,Zhou Yu,Yi Yang,Siliang Tang,Yin Zhang,Yueting Zhuang

DOI: https://doi.org/10.1109/tmm.2013.2291214

IF: 7.3

2013-01-01

IEEE Transactions on Multimedia

Abstract:Learning hash functions across heterogenous high-dimensional features is very desirable for many applications involving multi-modal data objects. In this paper, we propose an approach to obtain the sparse codesets for the data objects across different modalities via joint multi-modal dictionary learning, which we call sparse multi-modal hashing (abbreviated as SM2H). In SM2H, both intra-modality similarity and inter-modality similarity are first modeled by a hypergraph, then multi-modal dictionaries are jointly learned by Hypergraph Laplacian sparse coding. Based on the learned dictionaries, the sparse codeset of each data object is acquired and conducted for multi-modal approximate nearest neighbor retrieval using a sensitive Jaccard metric. The experimental results show that SM2H outperforms other methods in terms of mAP and Percentage on two real-world data sets.

Multi-modal Hash Learning

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Nonlinear Discrete Cross-Modal Hashing for Visual-Textual Data

Online latent semantic hashing for cross-media retrieval.

Multi-modal Hashing for Efficient Multimedia Retrieval: A Survey

Composite Multi-modal Hashing

Flexible Multi-modal Hashing for Scalable Multimedia Retrieval

Fast Discrete Collaborative Multi-Modal Hashing for Large-Scale Multimedia Retrieval

Partial Multi-Modal Hashing via Neighbor-aware Completion Learning

Efficient Multi-modal Hashing with Online Query Adaption for Multimedia Retrieval

One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Scalable Multimedia Retrieval By Deep Learning Hashing With Relative Similarity Learning

Enhanced Discrete Multi-Modal Hashing: More Constraints Yet Less Time to Learn

Unsupervised Multi-modal Hashing for Cross-Modal Retrieval

Multi-View Latent Hashing for Efficient Multimedia Search

CLIP Multi-modal Hashing for Multimedia Retrieval

Transitive Hashing Network for Heterogeneous Multimedia Retrieval

Efficient Supervised Discrete Multi-View Hashing for Large-Scale Multimedia Search

Multi-modal discrete tensor decomposition hashing for efficient multimedia retrieval

Cross-Media Hashing with Neural Networks

Sparse Multi-Modal Hashing