Multicenter clinical trial of implanted norethindrone pellets for long-acting contraception in women. Program for Applied Research on Fertility Regulation.

G. N. Gupta,R. Landesman,Saxena Bb,Mukul Singh,R. Asch,H. Croxatto,S. Dı́az,M. Pavez,E. Coutinho,R. Rivera,A. Goldsmith

DOI: https://doi.org/10.1016/0010-7824(84)90087-8

IF: 3.051

1984-09-01

Contraception

Abstract:

What problem does this paper attempt to address?

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

Weikuo Guo,Jian Liang,Xiangwei Kong,Lingxiao Song,Ran He

DOI: https://doi.org/10.1007/978-3-030-20873-8_33

2018-01-01

Abstract:How to bridge heterogeneous gap between different modalities is one of the main challenges in cross-modal retrieval task. Most existing methods try to tackle this problem by projecting data from different modalities into a common space. In this paper, we introduce a novel X-Shaped Generative Adversarial Cross-Modal Network (X-GACMN) to learn a better common space between different modalities. Specifically, the proposed architecture combines the process of synthetic data generation and distribution adapting into a unified framework to make sure the heterogeneous modality distributions similar to each other in the learned common subspace. To promote the discriminative ability, a new loss function that combines intra-modality angular softmax loss and cross-modality pair-wise consistent loss is further imposed on the common space, hence the learned features can well preserve both inter-modality structure and intra-modality structure on a hypersphere manifold. Extensive experiments on three benchmark datasets show the effectiveness of the proposed approach.
Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval

Shengsheng Qian,Dizhan Xue,Huaiwen Zhang,Quan Fang,Changsheng Xu

DOI: https://doi.org/10.1609/aaai.v35i3.16345

2021-05-18

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Cross-modal retrieval has become an active study field with the expanding scale of multimodal data. To date, most existing methods transform multimodal data into a common representation space where semantic similarities between items can be directly measured across different modalities. However, these methods typically suffer from following limitations: 1) They usually attempt to bridge the modality gap by designing losses in the common representation space which may not be sufficient to eliminate potential heterogeneity of different modalities in the common space. 2) They typically treat labels as independent individuals and ignore label relationships which are important for constructing semantic links between multimodal data. In this work, we propose a novel Dual Adversarial Graph Neural Networks (DAGNN) composed of the dual generative adversarial networks and the multi-hop graph neural networks, which learn modality-invariant and discriminative common representations for cross-modal retrieval. Firstly, we construct the dual generative adversarial networks to project multimodal data into a common representation space. Secondly, we leverage the multi-hop graph neural networks, in which a layer aggregation mechanism is proposed to exploit multi-hop propagation information, to capture the label correlation dependency and learn inter-dependent classifiers. Comprehensive experiments conducted on two cross-modal retrieval benchmark datasets, NUS-WIDE and MIRFlickr, indicate the superiority of DAGNN.
Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

Lei Liao,Meng Yang,Bob Zhang

DOI: https://doi.org/10.1109/tcsvt.2022.3203247

IF: 5.859

2023-02-07

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Cross-modal retrieval tasks, which are more natural and challenging than traditional retrieval tasks, have attracted increasing interest from researchers in recent years. Although different modalities with the same semantics have some potential relevance, the feature space heterogeneity still seriously weakens the performance of cross-modal retrieval models. To solve this problem, common space-based methods in which multimodal data is projected into a learned common space for similarity measurement have become the mainstream approach for cross-modal retrieval tasks. However, current methods entangle the modality style and semantic content in the common space and neglect to fully explore the semantic and discriminative representation/reconstruction of the semantic content. This often results in an unsatisfactory retrieval performance. To solve these issues, this paper proposes a new Deep Supervised Dual Cycle Adversarial Network (DSDCAN) model based on common space learning. It is composed of two cross-modal cycle GANs, one for the image and one for the text. The proposed cycle GAN model disentangles the semantic content and modality style features by making the data of one modality well reconstructed from the extracted modal style feature and the content feature of the other modality. Then, a discriminative semantic and label loss is proposed by fully considering the category, sample contrast, and label supervision to enhance the semantic discrimination of the common space representation. Besides this, to make the data distribution between two modalities similar, a second-order similarity is presented as a distance measurement of the cross-modal representation in the common space. Extensive experiments have been conducted on the Wikipedia, Pascal Sentence, NUS-WIDE-10k, PKU XMedia, MSCOCO, NUS-WIDE, Flickr30k and MIRFlickr datasets. The results demonstrate that the proposed method can achieve a higher performance than the state-of-the-art methods.
Multi-label adversarial fine-grained cross-modal retrieval

Chunpu Sun,Huaxiang Zhang,Li Liu,Dongmei Liu,Lin Wang

DOI: https://doi.org/10.1016/j.image.2023.117018

2023-07-08

Abstract:Most supervised cross-modal approaches transform features into a common representation space in which semantic similarity can be measured directly. How- ever, there exist modal specific features in the common semantic space and most methods can not fully eliminate them. In order to bridge the semantic gap and eliminate modal specific features, we propose a novel Multi-label Adversarial Fine-grained Cross-modal Retrieval Based on Transformer (MLAT). MLAT constructs a semantic consistency enhanced module (SCE) which includes the semantic mask attention module and a fine-grained feature generator based on transformer. It learns fine-grained semantic information to preserve the high-level semantic relevance and eliminate modal specific features. In order to narrow the distance between common representations and further eliminate modal specific features, we construct a multi-stage adversarial learning module to optimize feature representations. Furthermore, we design a label graph network based on graph attention network (GAT) to better explore the semantic correlations between labels and learn a classifier. Three benchmark datasets are synthesized to demonstrate the superiority of MLAT method.

engineering, electrical & electronic
Discriminative Dictionary Learning with Common Label Alignment for Cross-Modal Retrieval.

Cheng Deng,Xu Tang,Junchi Yan,Wei Liu,Xinbo Gao

DOI: https://doi.org/10.1109/tmm.2015.2508146

IF: 7.3

2015-01-01

IEEE Transactions on Multimedia

Abstract:Cross-modal retrieval has attracted much attention in recent years due to its widespread applications. In this area, how to capture and correlate heterogeneous features originating from different modalities remains a challenge. However, most existing methods dealing with cross-modal learning only focus on learning relevant features shared by two distinct feature spaces, therefore overlooking discriminative feature information of them. To remedy this issue and explicitly capture discriminative feature information, we propose a novel cross-modal retrieval approach based on discriminative dictionary learning that is augmented with common label alignment. Concretely, a discriminative dictionary is first learned to account for each modality, which boosts not only the discriminating capability of intra-modality data from different classes but also the relevance of inter-modality data in the same class. Subsequently, all the resulting sparse codes are simultaneously mapped to a common label space, where the cross-modal data samples are characterized and associated. Also in the label space, the discriminativeness and relevance of the considered cross-modal data can be further strengthened by enforcing a common label alignment. Finally, cross-modal retrieval is performed over the common label space. Experiments conducted on two public cross-modal datasets show that the proposed approach outperforms several state-of-the-art methods in term of retrieval accuracy.
Dual graph-structured semantics multi-subspace learning for cross-modal retrieval

Yirong Li,Xianghong Tang,Jianguang Lu,Yong Huang

DOI: https://doi.org/10.1007/s00530-024-01471-0

IF: 3.9

2024-09-28

Multimedia Systems

Abstract:As the era of big data develops rapidly, cross-modal retrieval is a research field that has received widespread attention. Most current methods of cross-modal retrieval just pursue the macro alignment of modal data in a shared space to gain a common representation. However, they cannot achieve the satisfactory performance of cross-modal retrieval since they neglect the deep semantic alignment and the inherent differences between modalities. Being aware of these, this paper presents a dual graph-structured semantics multi-subspace learning (DGMS) method for cross-modal retrieval. Specifically in DGMS, the double semantics graph is established to represent the deep semantics of modal data, and the multiple subspace learning network constructs public and independent subspaces to capture the relevance and dissimilarity of modal data. Finally, a dual learning method based on the generative adversarial network is employed further to catch the joint probability distribution of the different modalities. The superiority of DGMS is demonstrated by experiments on Wikipedia, XMedia, and Pascal Sentence, for it can not only learn deep structural semantics but also explore the consistency and diversity of modalities.

computer science, information systems, theory & methods
Dual discriminant adversarial cross-modal retrieval

Pei He,Meng Wang,Ding Tu,Zhuo Wang,He, Pei,Wang, Meng,Tu, Ding,Wang, Zhuo

DOI: https://doi.org/10.1007/s10489-022-03653-7

IF: 5.3

2022-06-07

Applied Intelligence

Abstract:In order to improve the accuracy of cross-modal retrieval tasks and achieve flexible retrieval between different modalities, we propose a Dual Discriminant Adversarial cross-modal Retrieval (DDAC) method in this paper. First, DDAC integrates adversarial learning and minimization of feature projection distances and introduces label information in it. It can eliminate the same semantic heterogeneity between modalities while maintaining the distinguishability of different semantic features between modalities. Then, cosine distance is used to minimize and maximize the inter-modal distance of features with the same and different labels respectively to solve the inter-modal discrimination problem. Different from the general method, DDAC performs dual discrimination in the label space and solves the intra-modal discrimination problem from two perspectives of probability distribution and distance. Extensive experiments carried out on three public datasets validate that the proposed DDAC outperforms the state-of-the-art methods.

computer science, artificial intelligence
Rethinking Label-Wise Cross-Modal Retrieval from A Semantic Sharing Perspective

Yang,Chubing Zhang,Yi-Chu Xu,Dianhai Yu,De-Chuan Zhan,Jian Yang

DOI: https://doi.org/10.24963/ijcai.2021/454

2021-01-01

Abstract:The main challenge of cross-modal retrieval is to learn the consistent embedding for heterogeneous modalities. To solve this problem, traditional label-wise cross-modal approaches usually constrain the inter-modal and intra-modal embedding consistency relying on the label ground-truths. However, the experiments reveal that different modal networks actually have various generalization capacities, thereby end-to-end joint training with consistency loss usually leads to sub-optimal uni-modal model, which in turn affects the learning of consistent embedding. Therefore, in this paper, we argue that what really needed for supervised cross-modal retrieval is a good shared classification model. In other words, we learn the consistent embedding by ensuring the classification performance of each modality on the shared model, without the consistency loss. Specifically, we consider a technique called Semantic Sharing, which directly trains the two modalities interactively by adopting a shared self-attention based classification model. We evaluate the proposed approach on three representative datasets. The results validate that the proposed semantic sharing can consistently boost the performance under NDCG metric.
Integrating information theory and adversarial learning for cross-modal retrieval

Wei Chen,Yu Liu,Erwin M. Bakker,Michael S. Lew

DOI: https://doi.org/10.1016/j.patcog.2021.107983

IF: 8

2021-09-01

Pattern Recognition

Abstract:Accurately matching visual and textual data in cross-modal retrieval has been widely studied in the multimedia community. To address these challenges posited by the heterogeneity gap and the semantic gap, we propose integrating Shannon information theory and adversarial learning. In terms of the heterogeneity gap, we integrate modality classification and information entropy maximization adversarially. For this purpose, a modality classifier (as a discriminator) is built to distinguish the text and image modalities according to their different statistical properties. This discriminator uses its output probabilities to compute Shannon information entropy, which measures the uncertainty of the modality classification it performs. Moreover, feature encoders (as a generator) project uni-modal features into a commonly shared space and attempt to fool the discriminator by maximizing its output information entropy. Thus, maximizing information entropy gradually reduces the distribution discrepancy of cross-modal features, thereby achieving a domain confusion state where the discriminator cannot classify two modalities confidently. To reduce the semantic gap, Kullback-Leibler (KL) divergence and bi-directional triplet loss are used to associate the intra- and inter-modality similarity between features in the shared space. Furthermore, a regularization term based on KL-divergence with temperature scaling is used to calibrate the biased label classifier caused by the data imbalance issue. Extensive experiments with four deep models on four benchmarks are conducted to demonstrate the effectiveness of the proposed approach.

computer science, artificial intelligence,engineering, electrical & electronic
Effective Deep Learning-Based Multi-Modal Retrieval

Wei Wang,Xiaoyan Yang,Beng Chin Ooi,Dongxiang Zhang,Yueting Zhuang

DOI: https://doi.org/10.1007/s00778-015-0391-4

2015-01-01

The VLDB Journal

Abstract:Multi-modal retrieval is emerging as a new search paradigm that enables seamless information retrieval from various types of media. For example, users can simply snap a movie poster to search for relevant reviews and trailers. The mainstream solution to the problem is to learn a set of mapping functions that project data from different modalities into a common metric space in which conventional indexing schemes for high-dimensional space can be applied. Since the effectiveness of the mapping functions plays an essential role in improving search quality, in this paper, we exploit deep learning techniques to learn effective mapping functions. In particular, we first propose a general learning objective that effectively captures both intramodal and intermodal semantic relationships of data from heterogeneous sources. Given the general objective, we propose two learning algorithms to realize it: (1) an unsupervised approach that uses stacked auto-encoders and requires minimum prior knowledge on the training data and (2) a supervised approach using deep convolutional neural network and neural language model. Our training algorithms are memory efficient with respect to the data volume. Given a large training dataset, we split it into mini-batches and adjust the mapping functions continuously for each batch. Experimental results on three real datasets demonstrate that our proposed methods achieve significant improvement in search accuracy over the state-of-the-art solutions.
Principles of invasive hemodynamic monitoring.

G. Hankins

DOI: https://doi.org/10.1016/S0095-5108(18)30798-X

1986-12-01

Abstract:
Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Xin Wen,Zhizhong Han,Xinyu Yin,Yu-Shen Liu

DOI: https://doi.org/10.1109/ICME.2019.00089

2019-01-01

Abstract:Cross-modal retrieval aims to retrieve relevant data across different modalities (e.g., texts vs. images). The common strategy is to apply element-wise constraints between manually labeled pair-wise items to guide the generators to learn the semantic relationships between the modalities, so that the similar items can be projected close to each other in the common representation subspace. However, such constraints often fail to preserve the semantic structure between unpaired but semantically similar items (e.g. the unpaired items with the same class label are more similar than items with different labels). To address the above problem, we propose a novel cross-modal similarity transferring (CMST) method to learn and preserve the semantic relationships between unpaired items in an unsupervised way. The key idea is to learn the quantitative similarities in single-modal representation subspace, and then transfer them to the common representation subspace to establish the semantic relationships between unpaired items across modalities. Experiments show that our method outperforms the state-of-the-art approaches both in the class-based and pair-based retrieval tasks.
Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Yi He,Xin Liu,Yiu-Ming Cheung,Shu-Juan Peng,Jinhan Yi,Wentao Fan

DOI: https://doi.org/10.1145/3404835.3463031

2021-07-11

Abstract:Fine-grained Image-text retrieval is challenging but vital technology in the field of multimedia analysis. Existing methods mainly focus on learning the common embedding space of images (or patches) and sentences (or words), whereby their mapping features in such embedding space can be directly measured. Nevertheless, most existing image-text retrieval works rarely consider the shared semantic concepts that potentially correlated the heterogeneous modalities, which can enhance the discriminative power of learning such embedding space. Toward this end, we propose a Cross-Graph Attention model (CGAM) to explicitly learn the shared semantic concepts, which can be well utilized to guide the feature learning process of each modality and promote the common embedding learning. More specifically, we build semantic-embedded graph for each modality, and smooth the discrepancy between two modalities via cross-graph attention model to obtain shared semantic-enhanced features. Meanwhile, we reconstruct image and text features via the shared semantic concepts and original embedding representations, and leverage multi-head mechanism for similarity calculation. Accordingly, the semantic-enhanced cross-modal embedding between image and text is discriminatively obtained to benefit the fine-grained retrieval with high retrieval performance. Extensive experiments evaluated on benchmark datasets show the performance improvements in comparison with state-of-the-arts.
Semi-supervised constrained graph convolutional network for cross-modal retrieval

Lei Zhang,Leiting Chen,Weihua Ou,Chuan Zhou

DOI: https://doi.org/10.1016/j.compeleceng.2022.107994

2022-07-01

Abstract:Exploiting relationship among samples in cross-modal data plays a key role in the task of cross-modal retrieval, but most of existing methods only extract the correlation from pairwise samples and ignore the relations of unpaired samples. Some graph regularization methods proposed a reasonable paradigm to exploit the correlation from multiple samples. However, limited by the traditional framework, the performance has much room to improve. Moreover, although some existing DNN-based methods achieve excellent performance, the requirement of massive labeled data is also a shortcoming. In this paper, we propose a novel semi-supervised method, named Semi-supervised Constrained Graph Convolutional Network (SCGCN), which adopts graph convolutional network to exploit correlation from batch samples of data with different modalities. For reducing the requirement of labeled data, we design a two stage training procedure: deep supervised learning stage and unsupervised learning stage. In deep supervised learning stage, we integrate two DNN-based semantic encoding networks and a shared classifier into Deep Cross-modal Semantic Encoding (DCSE) module which is trained by supervised learning with labeled data. From DCSE module, we learn a temporary modality-invariant space where the semantic embeddings of samples with different modalities are modality-invariant, and we also learn a classifier which can generate predicted label from the unlabeled data. In unsupervised learning stage, for fully exploiting the correlation from cross-modal data, we design a Constrained Graph Convolutional Network (CGCN) module which utilizes GCN to exploit the correlation and adopts both intra-modal discriminative loss and inter-modal pairwise similar loss to ensure the generated common representation modality-invariant and semantical discriminative. We perform extensive experiments on four conventional datasets and a large scale dataset to demonstrate the effectiveness of proposed approach.

engineering, electrical & electronic,computer science, interdisciplinary applications, hardware & architecture
Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Lei Zhu,Chengyuan Zhang,Jiayu Song,Shichao Zhang,Chunwei Tian,Xinghui Zhu

DOI: https://doi.org/10.1109/mmul.2022.3144138

IF: 3.4911

2022-01-01

IEEE Multimedia

Abstract:The main challenge of cross-modal retrieval is how to efficiently realize cross-modal semantic alignment and reduce the heterogeneity gap. However, existing approaches either ignore the multigrained semantic knowledge learning from different modalities, or fail to learn consistent relation distributions of semantic details in multimodal instances. To this end, this article proposes a novel end-to-end cross-modal representation method, termed as deep multigraph-based hierarchical enhanced semantic representation (MG-HESR). This method is an integration of MG-HESR with cross-modal adversarial learning, which captures multigrained semantic knowledge from cross-modal samples and realizes fine-grained semantic relation distribution alignment, and then generates modalities-invariant representations in a common subspace. To evaluate the performance, extensive experiments are conducted on four benchmarks. The experimental results show that our method is superior than the state-of-the-art methods.

computer science, information systems, theory & methods, software engineering, hardware & architecture
Adversarial Graph Convolutional Network for Cross-Modal Retrieval

Xinfeng Dong,Li Liu,Lei Zhu,Liqiang Nie,Huaxiang Zhang

DOI: https://doi.org/10.1109/tcsvt.2021.3075242

IF: 5.859

2022-03-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:The completeness of semantic expression plays an important role in cross-modal retrieval tasks, which contributes to align the cross-modal data and thus narrow the modality gap. But due to the abstractness of semantics, the same topic may have different aspects to be well described so it may be incomplete to express semantics with only one sample. In order to obtain semantic complementary information and strengthen similar information for samples with the same semantics, we utilize a graph convolutional network (GCN) to reconstruct the sample representation based on the adjacency relationship between the sample itself and its neighborhoods. We construct a local graph for each instance, and propose a novel Graph Feature Generator based on GCN and a fully-connected network to reconstruct node features based on local graph and map the features of two modalities into a common space. The Graph Feature Generator and Graph Feature Discriminator adopt a minimax game strategy to generate modality-invariant graph feature representations. Experiments on three benchmark datasets demonstrate the superiority of our proposed model compared with several state-of-the-art methods.

engineering, electrical & electronic
Cross‐modal retrieval with dual multi‐angle self‐attention

Wenjie Li,Yi Zheng,Yuejie Zhang,Rui Feng,Tao Zhang,Weiguo Fan

DOI: https://doi.org/10.1002/asi.24373

2020-07-16

Journal of the Association for Information Science and Technology

Abstract:In recent years, cross‐modal retrieval has been a popular research topic in both fields of computer vision and natural language processing. There is a huge semantic gap between different modalities on account of heterogeneous properties. How to establish the correlation among different modality data faces enormous challenges. In this work, we propose a novel end‐to‐end framework named Dual Multi‐Angle Self‐Attention (DMASA) for cross‐modal retrieval. Multiple self‐attention mechanisms are applied to extract fine‐grained features for both images and texts from different angles. We then integrate coarse‐grained and fine‐grained features into a multimodal embedding space, in which the similarity degrees between images and texts can be directly compared. Moreover, we propose a special multistage training strategy, in which the preceding stage can provide a good initial value for the succeeding stage and make our framework work better. Very promising experimental results over the state‐of‐the‐art methods can be achieved on three benchmark datasets of Flickr8k , Flickr30k , and MSCOCO .

information science & library science,computer science, information systems
Adversarial Cross-Modal Retrieval

Bokun Wang,Yang Yang,Xing Xu,Alan Hanjalic,Heng Tao Shen

DOI: https://doi.org/10.1145/3123266.3123326

2017-10-19

Abstract:Cross-modal retrieval aims to enable flexible retrieval experience across different modalities (e.g., texts vs. images). The core of cross-modal retrieval research is to learn a common subspace where the items of different modalities can be directly compared to each other. In this paper, we present a novel Adversarial Cross-Modal Retrieval (ACMR) method, which seeks an effective common subspace based on adversarial learning. Adversarial learning is implemented as an interplay between two processes. The first process, a feature projector, tries to generate a modality-invariant representation in the common subspace and to confuse the other process, modality classifier, which tries to discriminate between different modalities based on the generated representation. We further impose triplet constraints on the feature projector in order to minimize the gap among the representations of all items from different modalities with same semantic labels, while maximizing the distances among semantically different images and texts. Through the joint exploitation of the above, the underlying cross-modal semantic structure of multimedia data is better preserved when this data is projected into the common subspace. Comprehensive experimental results on four widely used benchmark datasets show that the proposed ACMR method is superior in learning effective subspace representation and that it significantly outperforms the state-of-the-art cross-modal retrieval methods.
Weighted Graph-structured Semantics Constraint Network for Cross-Modal Retrieval

Lei Zhang,Leiting Chen,Chuan Zhou,Xin Li,Fan Yang,Zhang Yi

DOI: https://doi.org/10.1109/tmm.2023.3282894

IF: 7.3

2023-01-01

IEEE Transactions on Multimedia

Abstract:Cross-modal retrieval aims to retrieve relevant content of different modalities by giving a query of another modality. The biggest difficulty is how to bridge the heterogeneous gap between different modalities. The commonly-used methods tend to focus on exploiting individual image-text pair and mining the relations of cross-modality data thereof, but ignore the role of multi-sample correlation. Moreover, more global, structural inter-pair knowledge contained by the training dataset will be under-used. To fully exploit graph-structured semantics and mine the semantic information in the dataset for learning discriminative representations, we propose Weighted Graph-structured Semantics Constraint Network (WGSCN), a unified, graph-based, semantic-constrained learning framework,in which GCN is used to mine comprehensive relation information from cross modality data. Our main inspiration is to design a novel two-branch GCN-based Cross-modal Semantic Encoding (GCSE) module to produce semantic embeddings with the both modality-specific and modality-shared correlation. Moreover, a GAN-based dual learning approach is used to further improve the discriminability and model the joint distribution across different modalities. Our proposed GDL uses semantic embeddings as supervisory signal to make the common representation semantically discriminative while adversarial learning and dual learning are used to make the common representation modality-invariant. Through comparative experiments on five commonly used cross-modal datasets, we have shown the superior retrieval accuracy of our WGSCN.

computer science, information systems,telecommunications, software engineering
Cross-modal Contrastive Learning for Generalizable and Efficient Image-text Retrieval

Haoyu Lu,Yuqi Huo,Mingyu Ding,Nanyi Fei,Zhiwu Lu

DOI: https://doi.org/10.1007/s11633-022-1386-4

2023-05-04

Machine Intelligence Research

Abstract:Cross-modal image-text retrieval is a fundamental task in bridging vision and language. It faces two main challenges that are typically not well addressed in previous works. 1) Generalizability: Existing methods often assume a strong semantic correlation between each text-image pair, which are thus difficult to generalize to real-world scenarios where the weak correlation dominates. 2) Efficiency: Many latest works adopt the single-tower architecture with heavy detectors, which are inefficient during the inference stage because the costly computation needs to be repeated for each text-image pair. In this work, to overcome these two challenges, we propose a two-tower cross-modal contrastive learning (CMCL) framework. Specifically, we first devise a two-tower architecture, which enables a unified feature space for the text and image modalities to be directly compared with each other, alleviating the heavy computation during inference. We further introduce a simple yet effective module named multi-grid split (MGS) to learn fine-grained image features without using detectors. Last but not the least, we deploy a cross-modal contrastive loss on the global image/text features to learn their weak correlation and thus achieve high generalizability. To validate that our CMCL can be readily generalized to real-world scenarios, we construct a large multi-source image-text dataset called weak semantic correlation dataset (WSCD). Extensive experiments show that our CMCL outperforms the state-of-the-arts while being much more efficient.

Multicenter clinical trial of implanted norethindrone pellets for long-acting contraception in women. Program for Applied Research on Fertility Regulation.

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

Multi-label adversarial fine-grained cross-modal retrieval

Discriminative Dictionary Learning with Common Label Alignment for Cross-Modal Retrieval.

Dual graph-structured semantics multi-subspace learning for cross-modal retrieval

Dual discriminant adversarial cross-modal retrieval

Rethinking Label-Wise Cross-Modal Retrieval from A Semantic Sharing Perspective

Integrating information theory and adversarial learning for cross-modal retrieval

Effective Deep Learning-Based Multi-Modal Retrieval

Principles of invasive hemodynamic monitoring.

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Semi-supervised constrained graph convolutional network for cross-modal retrieval

Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Adversarial Graph Convolutional Network for Cross-Modal Retrieval

Cross‐modal retrieval with dual multi‐angle self‐attention

Adversarial Cross-Modal Retrieval

Weighted Graph-structured Semantics Constraint Network for Cross-Modal Retrieval

Cross-modal Contrastive Learning for Generalizable and Efficient Image-text Retrieval