Exploring statistical correlations for image retrieval

WangXin-Jing,MaWei-Ying,LiXing

IF: 3.9

2006-01-01

Multimedia Systems

Abstract:Bridging the cognitive gap in image retrieval has been an active research direction in recent years, of which a key challenge is to get enough training data to learn the mapping functions from low-...

What problem does this paper attempt to address?

Exploring Statistical Correlations for Image Retrieval

Xin-Jing Wang,Wei-Ying Ma,Xing Li

DOI: https://doi.org/10.1007/s00530-006-0013-5

IF: 3.9

2006-01-01

Multimedia Systems

Abstract:Bridging the cognitive gap in image retrieval has been an active research direction in recent years, of which a key challenge is to get enough training data to learn the mapping functions from low-level feature spaces to high-level semantics. In this paper, image regions are classified into two types: key regions representing the main semantic contents and environmental regions representing the contexts. We attempt to leverage the correlations between types of regions to improve the performance of image retrieval. A Context Expansion approach is explored to take advantages of such correlations by expanding the key regions of the queries using highly correlated environmental regions according to an image thesaurus. The thesaurus serves as both a mapping function between image low-level features and concepts and a store of the statistical correlations between different concepts. It is constructed through a data-driven approach which uses Web data (images, their surrounding textual annotations) as training data source to learn the region concepts and to explore the statistical correlations. Experimental results on a database of 10,000 general-purpose images show the effectiveness of our proposed approach in both improving search precision (i.e. filter irrelevant images) and recall (i.e. retrieval relevant images whose context may be varied). Several major factors which have impact on the performance of our approach are also studied.
Exploring Geometric Information in CNN for Image Retrieval.

Ying Li,Xiangwei Kong,Haiyan Fu

DOI: https://doi.org/10.1007/s11042-018-6414-6

IF: 2.577

2018-01-01

Multimedia Tools and Applications

Abstract:Convolutional Neural Network (CNN) has brought significant improvements for various multimedia tasks. In contrast, image retrieval has not yet benefited as much since no training database is available. In this paper, we propose an unsupervised weighting scheme for pre-trained CNN models to adaptively emphasize image center. Different from the general preference for fully connected layers which represent abstract semantics, we aggregate the activations of convolutional layers on image patches to depict local patterns in details. It is an empirical observation that the target of searching is naturally the focus of an image. Thus we pooling the features with respect to their positions, since they innately maintain the geometric layout of an image. Experimental results on two benchmarks prove the effectiveness of our methods.
Learning and Exploiting Interclass Visual Correlations for Medical Image Classification

Dong Wei,Shilei Cao,Kai Ma,Yefeng Zheng

DOI: https://doi.org/10.1007/978-3-030-59710-8_11

2020-01-01

Abstract:Deep neural network-based medical image classifications often use “hard” labels for training, where the probability of the correct category is 1 and those of others are 0. However, these hard targets can drive the networks over-confident about their predictions and prone to overfit the training data, affecting model generalization and adaption. Studies have shown that label smoothing and softening can improve classification performance. Nevertheless, existing approaches are either non-data-driven or limited in applicability. In this paper, we present the Class-Correlation Learning Network (CCL-Net) to learn interclass visual correlations from given training data, and produce soft labels to help with classification tasks. Instead of letting the network directly learn the desired correlations, we propose to learn them implicitly via distance metric learning of class-specific embeddings with a lightweight plugin CCL block. An intuitive loss based on a geometrical explanation of correlation is designed for bolstering learning of the interclass correlations. We further present end-to-end training of the proposed CCL block as a plugin head together with the classification backbone while generating soft labels on the fly. Our experimental results on the International Skin Imaging Collaboration 2018 dataset demonstrate effective learning of the interclass correlations from training data, as well as consistent improvements in performance upon several widely used modern network structures with the CCL block.
Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Dan Song,Xue-Jing Jiang,Yue Zhang,Fang-Lue Zhang,Yao Jin,Yun Zhang

DOI: https://doi.org/10.1016/j.cag.2023.06.033

2023-01-01

Abstract:Image-based 3D model retrieval aims to search for 3D models according to 2D image queries, which provides a convenient way for the management of large 3D model datasets. Most of the related works put the emphasis on bridging the modality gap between 2D images and 3D models, which faces a lot of challenges due to the huge domain discrepancy. In this paper, we explicitly model and eliminate the domain-specific features of 2D images and 3D models. To alleviate the negative effect of complex background of natural images, we adopt semantic focus loss to constrain networks to learn the most semantically relevant feature representations for both 2D images and 3D models. We conduct extensive experiments on two cross-domain 3D model retrieval datasets, MI3DOR and MI3DOR-2, to show the effectiveness of the proposed method.
Data-Driven Approach for Bridging the Cognitive Gap in Image Retrieval

XJ Wang,WY Ma,X Li

DOI: https://doi.org/10.1109/icme.2004.1394714

2004-01-01

Abstract:Bridging the cognitive gap in image retrieval has been an active research direction in recent years. Existing solutions typically require a large volume of training data that could be difficult to obtain in practice. In this paper, we propose a data-driven approach that uses Web images and their surrounding textual annotations as the source of training data to bridge the cognitive gap. We construct an image thesaurus that contains a set of codewords, each representing a semantically related subspace in the feature space. We also explore the use of query expansion based on the constructed image thesaurus for improving image retrieval performance.
Exploring Region Correlation and Context Expansion for Image Retrieval

Xinjing Wang,Wei‐Ying Ma,Xing Li

2005-01-01

Abstract:ABSTRACT We explore the statistical correlations between homogeneous regions after image segmentation for improving image retrieval performance. Our proposed approach, called Context Expansion , is analogous to query expansion in text retrieval where the statistical correlations between words are used to expand query for improving search precision and recall. In our approach, regions in images are classified into two types: key region representing the main semantic content and environmental region representing the context. We perform context expansion for the key region in the query example using highly correlated environmental regions. In our experiments, we found that this method is able to filter irrelevant images (i.e. improving precision) as well as retrieve relevant images whose context may be varied (i.e. improving recall). We also study several major factors which have impact on the performance of context expansion. Categories and Subject Descriptors H.3.3 [ Information Search and Retrieval
Correlation Space Embedding Algorithm and Its Application to Image Retrieval

Ling ZHUANG,Chao WANG,Feng ZHOU,Wei-Ming LU,Jiang-Qin WU

DOI: https://doi.org/10.3969/j.issn.1003-6059.2014.04.011

2014-01-01

Abstract:An effective approach to semantic-based image retrieval is to find the correlation between low-level visual features and high-level semantics expressed by free text. Inspired by kernel method and graph Laplacian, the correlation space embedding algorithm ( CSEA ) is proposed in this paper. The latent semantic indexing and the visual word are used to construct the correlation between low-level image feature and semantic text feature which are heterogeneous with each other. The underlying cross-modal relationship between the free text and the image is established, and then the semantic-based image retrieval can be realized naturally. The consistency of manifold structure is regarded as a prior constraint in CSEA. By using CSEA, both the low-level image feature and the semantic text feature are embedded into a same intermediate space. Compared with the canonical correlation analysis, the proposed method models the correlation between two different feature spaces and preserves the manifold structure of each data distribution. Thus, the reliability of the proposed algorithm is improved. The experimental results show the effectiveness and the feasibility of the proposed algorithm in image retrieval.
Image Retrieval Approach Based on Sparse Canonical Correlation Analysis

ZHUANG Ling,ZHUANG Yue-Ting,WU Jiang-Qin,YE Zhen-Chao,WU Fei

DOI: https://doi.org/10.3724/sp.j.1001.2012.04032

2012-01-01

Abstract:A key issue of semantic-based image retrieval is how to bridge the semantic gap between the low-level feature of image and high-level semantics, which can be expressed by means of free text effectively. The cross-modal relationship between the text and image is studied by a modeling semantic correlation between text and image. Based on the model, an approach to image retrieval is proposed so that images are retrieved according to meaning of the query text rather than query keywords. First, an algorithm for solving sparse canonical correlation analysis (CCA) is designed in this paper. Then a semantic space is learned by way of latent semantic analysis from text corpus, and images are represented by bag of visual words. After that, a semantic correlation space, by which the map between visual words of image and the high-level semantics is made explicit, can be constructed. The proposed method solves CCA in a sparse framework in order to make the result more interpretable and stable. The experimental result demonstrates that Sparse CCA outperform CCA in the context, and also substantiates the feasibility of the proposed approach to image retrieval.
A Three-Layered Graph-Based Learning Approach for Remote Sensing Image Retrieval.

Yuebin Wang,Liqiang Zhang,Xiaohua Tong,Liang Zhang,Zhenxin Zhang,Hao Liu,Xiaoyue Xing,P. Takis Mathiopoulos

DOI: https://doi.org/10.1109/TGRS.2016.2579648

IF: 8.2

2016-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:With the emergence of huge volumes of high-resolution remote sensing images produced by all sorts of satellites and airborne sensors, processing and analysis of these images require effective retrieval techniques. To alleviate the dramatic variation of the retrieval accuracy among queries caused by the single image feature algorithms, we developed a novel graph-based learning method for effectivel...
Codedretrieval: Joint Image Compression and Retrieval with Neural Networks.

Linfeng Liu,Haojie Liu,Tong Chen,Qiu Shen,Zhan Ma

DOI: https://doi.org/10.1109/vcip47243.2019.8965918

2019-01-01

Abstract:With the explosive increase of image data, the efficiency of both image compression and retrieval becomes unprecedentedly significant. However, these two tasks are usually isolated executed, which waste great computational resources in large-scale image applications. In this work, we propose a joint framework called CodedRetrieval, which can find a general feature expression for both compression and retrieval based on neural network. Additionally, a two stage training strategy is designed to achieve better balance between the two distinct tasks. Experimental results show that our method can achieve competitive perfomance on both compression and retrieval comparing to classic methods, while saving great amount of computation time.
Learning similarity for image retrieval with locally spatial information feedback

Xiaohang Ma,Dianhui Wang

DOI: https://doi.org/10.1109/ICISIP.2004.1287639

2004-01-01

Abstract:For content-based image retrieval techniques, systems adopt certain metric to measure the similarity between images using some visual features. In most of existing approaches, the measurements are fixed up and calculated based on the whole image. The usefulness of those systems is limited because 1. The end-users are not involved in the retrieval processes. 2. The spatial information is not taken into account and end users can not put different emphasizes on different regions in the image when retrieve. Therefore, the systems demonstrate a disability to use the locally spatial relevance feedback. This paper describes an approach towards those problems to improve the discrimination power using locally spatial information. Images are divided into 5 overlapped regions and 5 neural networks are employed to measure the similarity between corresponding regions in a query image and images in the database. The relevance feedbacks for different regions from end users are used to refine the neural networks for adjusting the measurements. Results demonstrate a good potential of the methodology.
Exploring Spatial Correlation for Visual Object Retrieval

Miaojing Shi,Xinghai Sun,Dacheng Tao,Chao Xu,George Baciu,Hong Liu

DOI: https://doi.org/10.1145/2641576

IF: 5

2015-01-01

ACM Transactions on Intelligent Systems and Technology

Abstract:Bag-of-visual-words (BOVW)-based image representation has received intense attention in recent years and has improved content-based image retrieval (CBIR) significantly. BOVW does not consider the spatial correlation between visual words in natural images and thus biases the generated visual words toward noise when the corresponding visual features are not stable. This article outlines the construction of a visual word co-occurrence matrix by exploring visual word co-occurrence extracted from small affine-invariant regions in a large collection of natural images. Based on this co-occurrence matrix, we first present a novel high-order predictor to accelerate the generation of spatially correlated visual words and a penalty tree (PTree) to continue generating the words after the prediction. Subsequently, we propose two methods of co-occurrence weighting similarity measure for image ranking: Co-Cosine and Co-TFIDF. These two new schemes down-weight the contributions of the words that are less discriminative because of frequent co-occurrences with other words. We conduct experiments on Oxford and Paris Building datasets, in which the ImageNet dataset is used to implement a large-scale evaluation. Cross-dataset evaluations between the Oxford and Paris datasets and Oxford and Holidays datasets are also provided. Thorough experimental results suggest that our method outperforms the state of the art without adding much additional cost to the BOVW model.
Image retrieval using compact deep semantic correlation descriptors

Bo-Jian Zhang,Guang-Hai Liu,Zuoyong Li,Shu-Xiang Song

DOI: https://doi.org/10.1016/j.ipm.2023.103608

IF: 7.466

2024-02-03

Information Processing & Management

Abstract:Significant progress has been made in instance image retrieval based on deep feature aggregation. However, existing approaches are limited by two issues: 1) The inability of deep features to localize target objects generates inaccurate feature descriptions and 2) using short vector feature representations provides unsatisfactory retrieval performance. To address these issues, we propose the compact deep semantic correlation descriptor (DSCD) approach, which has three main highlights: (1) Unlike manual labeling approaches, we propose a channel semantic correlation learning method to localize target objects. This method can learn semantic correlation information between channels from an auxiliary dataset and use this information as a priori knowledge to locate target objects well. (2) We propose a hierarchical attention mechanism to integrate multi-level features within target regions. It utilizes object and focus attention modules to capture global and local features, and constructs channel attention coefficients to enhance channels containing important target features. (3) We propose a general, yet efficient dimensionality reduction method named adaptive PCA-whitening to improve short vector feature retrieval performance. This method can dynamically fuse the feature vectors of different dimensions via adaptive coefficients, yielding a more compact and robust representation. Extensive experiments on six benchmark datasets show that our method achieves better retrieval performance compared to existing state-of-the-art unsupervised methods. Compared with the next-best unsupervised methods, the mAP scores of our method (dim = 128) are 11.2 %, 6.1 %, 9.3 %, 8.3 %, 14.3 %, and 22.1 % higher on the Oxford5K, Paris6K, Oxford105K, Paris106K, ROxford, and RParis datasets, respectively.

computer science, information systems,information science & library science
Towards Improving Canonical Correlation Analysis for Cross-modal Retrieval.

Jie Shao,Zhicheng Zhao,Fei Su,Ting Yue

DOI: https://doi.org/10.1145/3126686.3126726

2017-01-01

Abstract:Building correlations for cross-modal retrieval, i.e., image-to-text retrieval and text-to-image retrieval, is a feasible solution to bridge the semantic gap between different modalities. Canonical correlation analysis (CCA) based methods have ever achieved great successes. However, conventional 2-view CCA suffers from three inherent problems: 1) it fails to capture the intra-modal semantic consistency, which is a necessary element to improve the retrieval performance, 2) it is hard to learn the non-linear correlation between different modalities, and 3) there exists problem in similarity measure due to the fact that the latent space learned by CCA is not directly optimized with certain distance measure. To address above problem, in this paper, we propose an improved CCA algorithm (ICCA) from three aspects. First, we propose two effective semantic features based on text features to improve intra-modal semantic consistency. Second, we expand traditional CCA from 2-view to 4-view, and embed 4-view CCA into a progressive framework to alleviate the over-fitting. Our progressive framework combines the training of linear projection and nonlinear hidden layers to ensure that good representations of the input raw data are learned at the output of the network. Third, inspired by large scale similarity learning (LSSL), a similarity metric is proposed to improve the distance measure. Experiments on three publicly data sets demonstrate the effectiveness of the proposed ICCA method.
Relation-Aggregated Cross-Graph Correlation Learning for Fine-Grained Image–Text Retrieval

Shu-Juan Peng,Yi He,Xin Liu,Yiu-ming Cheung,Xing Xu,Zhen Cui

DOI: https://doi.org/10.1109/tnnls.2022.3188569

IF: 14.255

2022-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Fine-grained image-text retrieval has been a hot research topic to bridge the vision and languages, and its main challenge is how to learn the semantic correspondence across different modalities. The existing methods mainly focus on learning the global semantic correspondence or intramodal relation correspondence in separate data representations, but which rarely consider the intermodal relation that interactively provide complementary hints for fine-grained semantic correlation learning. To address this issue, we propose a relation-aggregated cross-graph (RACG) model to explicitly learn the fine-grained semantic correspondence by aggregating both intramodal and intermodal relations, which can be well utilized to guide the feature correspondence learning process. More specifically, we first build semantic-embedded graph to explore both fine-grained objects and their relations of different media types, which aim not only to characterize the object appearance in each modality, but also to capture the intrinsic relation information to differentiate intramodal discrepancies. Then, a cross-graph relation encoder is newly designed to explore the intermodal relation across different modalities, which can mutually boost the cross-modal correlations to learn more precise intermodal dependencies. Besides, the feature reconstruction module and multihead similarity alignment are efficiently leveraged to optimize the node-level semantic correspondence, whereby the relation-aggregated cross-modal embeddings between image and text are discriminatively obtained to benefit various image-text retrieval tasks with high retrieval performance. Extensive experiments evaluated on benchmark datasets quantitatively and qualitatively verify the advantages of the proposed framework for fine-grained image-text retrieval and show its competitive performance with the state of the arts.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Towards Sketch-Based Image Retrieval with Deep Cross-Modal Correlation Learning.

Fei Huang,Cheng Jin,Yuejie Zhang,Tao Zhang

DOI: https://doi.org/10.1109/icme.2017.8019432

2017-01-01

Abstract:A novel scheme with deep cross-modal correlation learning is developed in this paper to facilitate more effective Sketch-based Image Retrieval (SBIR) for large-scale annotated images. It integrates the deep multimodal feature generation, deep cross-modal correlation learning and similarity search optimization through mining all the beneficial multimodal information sources in sketches and images, which can be treated as an inter-related correlation distribution over deep representations of sketches and images. Very positive results were obtained in our experiments using a large quantity of public data.
Understanding Visual-Auditory Correlation from Heterogeneous Features for Cross-Media Retrieval

Hong Zhang,Yan-yun Wang,Hong Pan,Fei Wu

DOI: https://doi.org/10.1631/jzus.a071191

2008-01-01

Abstract:Cross-media retrieval is an interesting research topic, which seeks to remove the barriers among different modalities. To enable cross-media retrieval, it is needed to find the correlation measures between heterogeneous low-level features and to judge the semantic similarity. This paper presents a novel approach to learn cross-media correlation between visual features and auditory features for image-audio retrieval. A semi-supervised correlation preserving mapping (SSCPM) method is described to construct the isomorphic SSCPM subspace where canonical correlations between the original visual and auditory features are further preserved. Subspace optimization algorithm is proposed to improve the local image cluster and audio cluster quality in an interactive way. A unique relevance feedback strategy is developed to update the knowledge of cross-media correlation by learning from user behaviors, so retrieval performance is enhanced in a progressive manner. Experimental results show that the performance of our approach is effective.
Bi-Directional Image-Text Retrieval with Position Attention and Similarity Filtering

Wei Zhang,Xiaowei Xu,Ye Tao,Xiaodong Wang,Chenglin Wang,Zhimin Wei

DOI: https://doi.org/10.1109/icivc55077.2022.9886989

2022-01-01

Abstract:In recent years, cross-modal retrieval has gradually become the frontier and hotspot of academic research at home and abroad, and it is an important direction for the future development of information retrieval. Some current methods improve the performance of image-text retrieval by exploring more comprehensive global image-text alignment information or capturing region-word local fine-grained alignment. However, previous methods did not mine more useful information to obtain more accurate matching scores. In this paper, we propose a location attention and similarity filtering network for image text retrieval. Specifically, we enhance visual-text joint embedding learning with global and local alignments. We then enhance more reliable relationships between images and text sentences by exploring the location information of objects in images through location attention. In addition, we use a similarity filtering mechanism to selectively focus on important and representative alignment information while leaving the distraction of meaningless alignment information aside to effectively integrate these alignments. Experiments on our proposed method on the public datasets Flickr30K and MS-COCO validate the effectiveness and superiority of our method.
Correlation learning method based on image internal semantic model for CBIR

Lijuan Duan,Guojun Mao,Wen Gao

DOI: https://doi.org/10.1007/978-3-540-30542-2_22

2004-01-01

Abstract:Semantic-based image retrieval is the desired target of Content-based image retrieval (CBIR). In this paper, we proposed a new method to extract semantic information for CBIR using the relevance feedback results. Firstly it is assumed that positive and negative examples in relevant feedback are containing semantic content added by users. Then image internal semantic model (IISM) is proposed to represent comprehensive pair-wise correlation information for images through analyzing the feedback results. Finally, correlation learning method is proposed to represent the images' pair-wise relationship based on statistical value of access path, access frequency, similarity factor and correlation factor. Experimental results on Corel datasets show the effectiveness of the proposed model and method.
Constraint Based Region Matching for Image Retrieval

Tao Wang,Yong Rui,Jia-Guang Sun

DOI: https://doi.org/10.1023/B:VISI.0000004831.53436.88

IF: 13.369

2004-01-01

International Journal of Computer Vision

Abstract:Objects and their spatial relationships are important features for human visual perception. In most existing content-based image retrieval systems, however, only global features extracted from the whole image are used. While they are easy to implement, they have limited power to model semantic-level objects and spatial relationship. To overcome this difficulty, this paper proposes a constraint-based region matching approach to image retrieval. Unlike existing region-based approaches where either individual regions are used or only first-order constraints are modeled, the proposed approach formulates the problem in a probabilistic framework and simultaneously models both first-order region properties and second-order spatial relationships for all the regions in the image. Specifically, in this paper we present a complete system that includes image segmentation, local feature extraction, first- and second-order constraints, and probabilistic regionweight estimation. Extensive experiments have been carried out on a large heterogeneous image collection with 17,000 images. The proposed approach achieves significantly better performance than the state-of-the-art approaches.

Exploring statistical correlations for image retrieval

Exploring Statistical Correlations for Image Retrieval

Exploring Geometric Information in CNN for Image Retrieval.

Learning and Exploiting Interclass Visual Correlations for Medical Image Classification

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Data-Driven Approach for Bridging the Cognitive Gap in Image Retrieval

Exploring Region Correlation and Context Expansion for Image Retrieval

Correlation Space Embedding Algorithm and Its Application to Image Retrieval

Image Retrieval Approach Based on Sparse Canonical Correlation Analysis

A Three-Layered Graph-Based Learning Approach for Remote Sensing Image Retrieval.

Codedretrieval: Joint Image Compression and Retrieval with Neural Networks.

Learning similarity for image retrieval with locally spatial information feedback

Exploring Spatial Correlation for Visual Object Retrieval

Image retrieval using compact deep semantic correlation descriptors

Towards Improving Canonical Correlation Analysis for Cross-modal Retrieval.

Relation-Aggregated Cross-Graph Correlation Learning for Fine-Grained Image–Text Retrieval

Towards Sketch-Based Image Retrieval with Deep Cross-Modal Correlation Learning.

Understanding Visual-Auditory Correlation from Heterogeneous Features for Cross-Media Retrieval

Bi-Directional Image-Text Retrieval with Position Attention and Similarity Filtering

Correlation learning method based on image internal semantic model for CBIR

Constraint Based Region Matching for Image Retrieval