Retrieving Relevant and Diverse Image from Social Media Images.

Xi Chen,Haokun Liu,Zhi-Hong Deng,Yunlun Yang

2015-01-01

Abstract:We describe our approach and its result for MediaEval 2015 Retrieving Diverse Social Images Task. The basic idea is removing the irrelevant images and then obtaining the diverse image using a greedy strategy. Experiment results show our method can retrieve diverse images with a moderate relevance to the topic.

What problem does this paper attempt to address?

Image Retrieval by Cross-Media Relevance Fusion.

Jianfeng Dong,Xirong Li,Shuai Liao,Jieping Xu,Duanqing Xu,Xiaoyong Du

DOI: https://doi.org/10.1145/2733373.2809929

2015-01-01

Abstract:How to estimate cross-media relevance between a given query and an unlabeled image is a key question in the MSR-Bing Image Retrieval Challenge. We answer the question by proposing cross-media relevance fusion, a conceptually simple framework that exploits the power of individual methods for cross-media relevance estimation. Four base cross-media relevance functions are investigated, and later combined by weights optimized on the development set. With DCG25 of 0.5200 on the test dataset, the proposed image retrieval system secures the first place in the evaluation.
Exploiting Visual-based Intent Classification for Diverse Social Image Retrieval.

Bo Wang,Martha A. Larson

2017-01-01

Abstract:In the 2017 MediaEval Retrieving Diverse Social Images task, we (TUD-MMC team) propose a novel method, namely an intent-based approach, for social image search result diversification. The underlying assumption is that the visual appearance of social images is impacted by the underlying photographic act, i.e., why the images were taken. Better understanding the rationale behind the photographic act could potentially benefit social image search result diversification. To investigate this idea, we employ a manual content analysis approach to create a taxonomy of intent classes. Our experiments show that a CNN-based neural network classifier is able to capture the visual difference between the classes in the intent taxonomy. We cluster images of the Flickr baseline based on predicted intent class and generate a re-ranked list by alternating images from different clusters. Our results reveal that, compared to conventional diversification strategies, intent-based search result diversification is able to bring a considerable improvement in terms of cluster recall with several extra benefits.
Cross-media relevance mining for evaluating text-based image search engine

Zhongwen Xu,Yi Yang,Ashraf A. Kassim,Shuicheng Yan

DOI: https://doi.org/10.1109/ICMEW.2014.6890606

2014-01-01

Abstract:Targeted at MSR-Bing Image Retrieval grand challenge, we accumulate the experience from the one in 2013, and the make further investigation into different models to solve the relevance assessment problem. Generally speaking, the assessment of relevance between text query and image list is very hard due to the semantic gap. It's not easy to find the “mapping” from text query into the visual world. Solutions from 2013 MSR-Bing grand challenge are discussed in this paper. Combining with our own observation, we give some insights into why some solutions work well, while others not. Our main solution is to combine the deep learning features with the wining solution of last year (average similarity measurement and page rank), since the deep learning features have more compact representation than the traditional BoWs features, and deep learning features are efficient (on a descent GPU) with very good performance. Our solution achieved the 1st place in MSR-Bing grand challenge 2014. Finally, we give the running time of our solution in the testing phase for the 2014 ICME testing set and development set, respectively.
Social-Sensed Image Search.

Peng Cui,Shao-Wei Liu,Wen-Wu Zhu,Huan-Bo Luan,Tat-Seng Chua,Shi-Qiang Yang

DOI: https://doi.org/10.1145/2590974

2014-01-01

Abstract:Although Web search techniques have greatly facilitate users’ information seeking, there are still quite a lot of search sessions that cannot provide satisfactory results, which are more serious in Web image search scenarios. How to understand user intent from observed data is a fundamental issue and of paramount significance in improving image search performance. Previous research efforts mostly focus on discovering user intent either from clickthrough behavior in user search logs (e.g., Google), or from social data to facilitate vertical image search in a few limited social media platforms (e.g., Flickr). This article aims to combine the virtues of these two information sources to complement each other, that is, sensing and understanding users’ interests from social media platforms and transferring this knowledge to rerank the image search results in general image search engines. Toward this goal, we first propose a novel social-sensed image search framework, where both social media and search engine are jointly considered. To effectively and efficiently leverage these two kinds of platforms, we propose an example-based user interest representation and modeling method, where we construct a hybrid graph from social media and propose a hybrid random-walk algorithm to derive the user-image interest graph. Moreover, we propose a social-sensed image reranking method to integrate the user-image interest graph from social media and search results from general image search engines to rerank the images by fusing their social relevance and visual relevance. We conducted extensive experiments on real-world data from Flickr and Google image search, and the results demonstrated that the proposed methods can significantly improve the social relevance of image search results while maintaining visual relevance well.
Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

Xing Gao,Xiangyu Shi,Guangyu Zhang,Juncong Lin,Minghong Liao,Kuan-Ching Li,Chaoyong Li

DOI: https://doi.org/10.1109/access.2018.2842796

IF: 3.9

2018-01-01

IEEE Access

Abstract:Because of the diversity and popularity of image acquisition techniques, data-driven methods for image analysis and editing have become popular. However, the explosive growth of images also presents challenges. Helping users to retrieve their expected images quickly and effectively is one of the most difficult tasks. Although various methods have been proposed, most methods cannot guarantee the quality of the image, which is typically required for analysis and authoring tasks. In this paper, we present a progressive image retrieval method with a quality guarantee. Images are gradually filtered by various criteria: starting from the quickest textual comparison, proceeding through a series of quality criteria, and ending with the most time-consuming contour match. The entire framework is parallelized under MapReduce to improve the performance. Various experiments are conducted to validate the performance and accuracy of the algorithm and the quality of the retrieved results. We also demonstrate the potential of the algorithm with an image synthesis prototype system.
Towards Automatic Construction of Diverse, High-Quality Image Datasets

Yazhou Yao,Jian Zhang,Fumin Shen,Li Liu,Fan Zhu,Dongxiang Zhang,Heng Tao Shen

DOI: https://doi.org/10.1109/tkde.2019.2903036

IF: 9.235

2020-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:The availability of labeled image datasets has been shown critical for high-level image understanding, which continuously drives the progress of feature designing and models developing. However, constructing labeled image datasets is laborious and monotonous. To eliminate manual annotation, in this work, we propose a novel image dataset construction framework by employing multiple textual queries. We aim at collecting diverse and accurate images for given queries from the Web. Specifically, we formulate noisy textual queries removing and noisy images filtering as a multi-view and multi-instance learning problem separately. Our proposed approach not only improves the accuracy but also enhances the diversity of the selected images. To verify the effectiveness of our proposed approach, we construct an image dataset with 100 categories. The experiments show significant performance gains by using the generated data of our approach on several tasks, such as image classification, cross-dataset generalization, and object detection. The proposed method also consistently outperforms existing weakly supervised and web-supervised approaches.
CUHK at ImageCLEF 2005: cross-language and cross-media image retrieval

Steven C. H. Hoi,Jianke Zhu,Michael R. Lyu

DOI: https://doi.org/10.1007/11878773_67

2006-01-01

Abstract:In this paper, we describe our studies of cross-language and cross-media image retrieval at the ImageCLEF 2005. This is the first participation of our CUHK (The Chinese University of Hong Kong) group at ImageCLEF. The task in which we participated is the “bilingual ad hoc retrieval” task. There are three major focuses and contributions in our participation. The first is the empirical evaluation of language models and smoothing strategies for cross-language image retrieval. The second is the evaluation of cross-media image retrieval, i.e., combining text and visual contents for image retrieval. The last is the evaluation of bilingual image retrieval between English and Chinese. We provide an empirical analysis of our experimental results, in which our approach achieves the best mean average precision result in the monolingual query task in the campaign. Finally we summarize our empirical experience and address the future improvement of our work.
Diversifying the image retrieval results.

Kai Song,Yonghong Tian,Wen Gao,Tiejun Huang

DOI: https://doi.org/10.1145/1180639.1180789

2006-01-01

Abstract:In the area of image retrieval, post-retrieval processing is often used to refine the retrieval results to better satisfy users' requirements. Previous methods mainly focus on presenting users with relevant results. However, in most cases, users cannot clearly present their requirements by several query words. Therefore, relevant results with rich topic coverage are more likely to meet users' ambiguous needs. In this paper, a re-ranking method based on topic richness analysis is proposed to enrich topic coverage in retrieval results. Furthermore, a quantitative criterion called diversity scores (DS) is proposed to evaluate the improvement. Given a set of images, topics that are rarely included in the set are scarce topics, as oppose to rich topics that are widely distributed among the set. Scarce topics contribute more than rich topics do to the DS of images. Five researchers are invited to evaluate the re-ranked results both in topic coverage and relevance. Experimental results on over 20,000 images demonstrate that our proposed approach is effective in improving the topic coverage of retrieval results without loss of relevance.
Cross-Modal Image-Tag Relevance Learning for Social Images

Yong Cheng,Zhengxiang Cai,Rui Feng,Cheng Jin,Yuejie Zhang,Tao Zhang

DOI: https://doi.org/10.1145/2733373.2806294

2015-01-01

Abstract:A new algorithm is developed in this paper to support more effective cross-modal image-tag relevance learning for large-scale social images, which integrates the multimodal feature representation, multimodal relevance measurement, and cross- modal relevance fusion. The main contribution of our work is that we provide a more reasonable base to learn cross-modal relevance among social images, which can be acquired from integrating multimodal image and tag relevance with multiple features in different modalities. Very positive results were obtained in our experiments using a large quantity of public social image data.
Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and Transformer.

Minyi Zhao,Jinpeng Wang,Dongliang Liao,Yiru Wang,Huanzhong Duan,Shuigeng Zhou

DOI: https://doi.org/10.1145/3539618.3591705

2023-01-01

Abstract:In addition to relevance, diversity is an important yet less studied performance metric of cross-modal image retrieval systems, which is critical to user experience. Existing solutions for diversity-aware image retrieval either explicitly post-process the raw retrieval results from standard retrieval systems or try to learn multi-vector representations of images to represent their diverse semantics. However, neither of them is good enough to balance relevance and diversity. On the one hand, standard retrieval systems are usually biased to common semantics and seldom exploit diversity-aware regularization in training, which makes it difficult to promote diversity by post-processing. On the other hand, multi-vector representation methods are not guaranteed to learn robust multiple projections. As a result, irrelevant images and images of rare or unique semantics may be projected inappropriately, which degrades the relevance and diversity of the results generated by some typical algorithms like top-k. To cope with these problems, this paper presents a new method called CoLT that tries to generate much more representative and robust representations for accurately classifying images. Specifically, CoLT first extracts semantics-aware image features by enhancing the preliminary representations of an existing one-to-one cross-modal system with semantics-aware contrastive learning. Then, a transformer-based token classifier is developed to subsume all the features into their corresponding categories. Finally, a post-processing algorithm is designed to retrieve images from each category to form the final retrieval result. Extensive experiments on two real-world datasets Div400 and Div150Cred show that CoLT can effectively boost diversity, and outperforms the existing methods as a whole (with a higher F1 score).
Hypergraph Spectral Hashing for image retrieval with heterogeneous social contexts

Yang Liu,Jian Shao,Jun Xiao,Fei Wu,Yueting Zhuang

DOI: https://doi.org/10.1016/j.neucom.2012.02.051

IF: 6

2013-01-01

Neurocomputing

Abstract:The development of social media brings great challenges to image retrieval on both efficiency and accuracy. In addition to achieving fast similarity search over large scale data, it is very crucial to represent the complex and high-order relationships among the social contents to improve the semantic understanding of social images. In this paper, unified hypergraph is implemented to model the various relationships among images and other contexts in social media. Then we extend traditional spectral hashing to hypergraph to accelerate similarity search of social images by mapping semantically related vertices into similar binary codes within a short Hamming distance. In addition, out-of-sample extension is implemented in a supervised manner and different strategies of fusing different social contexts are compared and discussed in this work. We evaluated our approach on the dataset crawled from Flickr and the experiment results indicate that our proposed HSH approach is both efficient and effective.
Social Media Analytics and Learning

Zheng-Jun Zha,Tao Mei,Abdulmotaleb El Saddik

DOI: https://doi.org/10.1016/j.neucom.2015.04.082

IF: 6

2015-01-01

Neurocomputing

Abstract:The goal of this paper is to simultaneously segment the object regions in a set of images with the same object class, known as object co-segmentation. Different from typical methods, simply assuming that the common regions among images are the object regions, we additionally consider the disturbance from consistent backgrounds, and indicate not only common regions but salient ones among images to be the object regions. To this end, we propose an adaptive discriminative low rank matrix recovery (ADLRR) algorithm to divide the over-completely segmented regions (i.e., super-pixels) of a given image set into object and non-object ones. The proposed ADLRR is formulated from two views: a low-rank matrix recovery term for salient regions detection and a discriminative learning term adopted to distinguish object regions from all super-pixels. An additional regularized term is incorporated to jointly measure the disagreement between the predicted saliency and the objectiveness probability. For the unified learning problem by connecting the above three terms, we design an efficient alternate optimization procedure based on block-coordinate descent and augmented Lagrange multipliers method. Extensive experiments are conducted on three public datasets, i.e., MSRC, iCoseg and Caltech101, and the comparisons with some state-of-the-arts demonstrate the effectiveness of our work.
Geometric Discriminative Features for Aerial Image Retrieval in Social Media

Yingjie Xia,Jinlong Chen,Jun Li,Ying Zhang

DOI: https://doi.org/10.1007/s00530-014-0412-y

IF: 3.9

2014-01-01

Multimedia Systems

Abstract:The aerial image recognition is an important problem in multimedia information retrieval in social media. In this paper, we propose a new approach by integrating aerial image's local features into a discriminative one which reflects both the geometric property and the color distribution of aerial image. Firstly, each aerial image is segmented into several regions in terms of their color intensities. And region connected graph (RCG), the links between the spatial neighboring regions, is presented to encode the spatial context of aerial images. Secondly, we mine frequent structures in the RCGs corresponding to training aerial images collected from social media. And a set of refined structures are selected among the frequent ones towards being more discriminative and less redundant. Finally, given a new aerial image, its sub-RCGs corresponding to all the refined structures are extracted and quantized into a discriminative feature for aerial image recognition. The experimental results validate the proposed method by providing a more accurate recognition result of the aerial images on different datasets from different social medias.
RetriBooru: Leakage-Free Retrieval of Conditions from Reference Images for Subject-Driven Generation

Haoran Tang,Jieren Deng,Zhihong Pan,Hao Tian,Pratik Chaudhari,Xin Zhou

2024-10-23

Abstract:Diffusion-based methods have demonstrated remarkable capabilities in generating a diverse array of high-quality images, sparking interests for styled avatars, virtual try-on, and more. Previous methods use the same reference image as the target. An overlooked aspect is the leakage of the target's spatial information, style, etc. from the reference, harming the generated diversity and causing shortcuts. However, this approach continues as widely available datasets usually consist of single images not grouped by identities, and it is expensive to recollect large-scale same-identity data. Moreover, existing metrics adopt decoupled evaluation on text alignment and identity preservation, which fail at distinguishing between balanced outputs and those that over-fit to one aspect. In this paper, we propose a multi-level, same-identity dataset RetriBooru, which groups anime characters by both face and cloth identities. RetriBooru enables adopting reference images of the same character and outfits as the target, while keeping flexible gestures and actions. We benchmark previous methods on our dataset, and demonstrate the effectiveness of training with a reference image different from target (but same identity). We introduce a new concept composition task, where the conditioning encoder learns to retrieve different concepts from several reference images, and modify a baseline network RetriNet for the new task. Finally, we introduce a novel class of metrics named Similarity Weighted Diversity (SWD), to measure the overlooked diversity and better evaluate the alignment between similarity and diversity.

Computer Vision and Pattern Recognition,Artificial Intelligence
Dataset and Case Studies for Visual Near-Duplicates Detection in the Context of Social Media

Hana Matatov,Mor Naaman,Ofra Amir

DOI: https://doi.org/10.48550/arXiv.2203.07167

2022-03-14

Abstract:The massive spread of visual content through the web and social media poses both challenges and opportunities. Tracking visually-similar content is an important task for studying and analyzing social phenomena related to the spread of such content. In this paper, we address this need by building a dataset of social media images and evaluating visual near-duplicates retrieval methods based on image retrieval and several advanced visual feature extraction methods. We evaluate the methods using a large-scale dataset of images we crawl from social media and their manipulated versions we generated, presenting promising results in terms of recall. We demonstrate the potential of this method in two case studies: one that shows the value of creating systems supporting manual content review, and another that demonstrates the usefulness of automatic large-scale data analysis.

Information Retrieval,Computer Vision and Pattern Recognition,Machine Learning
A unified approach of detecting misleading images via tracing its instances on web and analyzing its past context for the verification of multimedia content

Deepika Varshney,Dinesh Kumar Vishwakarma

DOI: https://doi.org/10.1007/s13735-022-00235-8

Abstract:The verification of multimedia content over social media is one of the challenging and crucial issues in the current scenario and gaining prominence in an age where user-generated content and online social web-platforms are the leading sources in shaping and propagating news stories. As these sources allow users to share their opinions without restriction, opportunistic users often post misleading/unreliable content on social media such as Twitter, Facebook, etc. At present, to lure users toward the news story, the text is often attached with some multimedia content (images/videos/audios). Verifying these contents to maintain the credibility and reliability of social media information is of paramount importance. Motivated by this, we proposed a generalized system that supports the automatic classification of images into credible or misleading. In this paper, we investigated machine learning-based as well as deep learning-based approaches utilized to verify misleading multimedia content, where the available image traces are used to identify the credibility of the content. The experiment is performed on the real-world dataset (Media-eval-2015 dataset) collected from Twitter. It also demonstrates the efficiency of our proposed approach and features using both Machine and Deep Learning Model (Bi-directional LSTM). The experiment result reveals that the Microsoft BING image search engine is quite effective in retrieving titles and performs better than our study's Google image search engine. It also shows that gathering clues from attached multimedia content (image) is more effective than detecting only posted content-based features.
Social-oriented Visual Image Search

Shaowei Liu,Peng Cui,Huanbo Luan,Wenwu Zhu,Shiqiang Yang,Qi Tian

DOI: https://doi.org/10.1016/j.cviu.2013.06.011

IF: 4.886

2014-01-01

Computer Vision and Image Understanding

Abstract:Many research have been focusing on how to match the textual query with visual images and their surrounding texts or tags for Web image search. The returned results are often unsatisfactory due to their deviation from user intentions, particularly for queries with heterogeneous concepts (such as “apple”, “jaguar”) or general (non-specific) concepts (such as “landscape”, “hotel”). In this paper, we exploit social data from social media platforms to assist image search engines, aiming to improve the relevance between returned images and user intentions (i.e., social relevance). Facing the challenges of social data sparseness, the tradeoff between social relevance and visual relevance, and the complex social and visual factors, we propose a community-specific Social-Visual Ranking (SVR) algorithm to rerank the Web images returned by current image search engines. The SVR algorithm is implemented by PageRank over a hybrid image link graph, which is the combination of an image social-link graph and an image visual-link graph. By conducting extensive experiments, we demonstrated the importance of both visual factors and social factors, and the advantages of social-visual ranking algorithm for Web image search.
Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Jie Xu,Zhoujun Li,Feiran Huang,Chaozhuo Li,Philip S. Yu

DOI: https://doi.org/10.1109/tii.2020.3005405

IF: 12.3

2021-04-01

IEEE Transactions on Industrial Informatics

Abstract:In the circumstance of social big data, sentiment analysis is attracting increasing attention for its capacity in understanding individuals' attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifestations. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social platforms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we propose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is proposed here to highlight semantically rich image channels and a region attention schema is further designed to highlight the emotional regions based on the attended channels. After that, we construct a heterogeneous relation network and extend graph convolutional network to aggregate the content information from social contexts as complements to learn high-quality representations of social images. Our proposal is thoroughly evaluated on two benchmark datasets, and experimental results demonstrate the superiority of the proposed model.

automation & control systems,computer science, interdisciplinary applications,engineering, industrial
On the Diversity of Realistic Image Synthesis.

Zichen Yang,Haifeng Liu,Deng Cai

2017-01-01

Abstract:Many image processing tasks can be formulated as translating images between two image domains, such as colorization, super resolution and conditional image synthesis. In most of these tasks, an input image may correspond to multiple outputs. However, current existing approaches only show very minor diversity of the outputs. In this paper, we present a novel approach to synthesize diverse realistic images corresponding to a semantic layout. We introduce a diversity loss objective, which maximizes the distance between synthesized image pairs and links the input noise to the semantic segments in the synthesized images. Thus, our approach can not only produce diverse images, but also allow users to manipulate the output images by adjusting the noise manually. Experimental results show that images synthesized by our approach are significantly more diverse than that of the current existing works and equipping our diversity loss does not degrade the reality of the base networks.
Social Visual Image Ranking for Web Image Search.

Shaowei Liu,Peng Cui,Huanbo Luan,Wenwu Zhu,Shiqiang Yang,Qi Tian

DOI: https://doi.org/10.1007/978-3-642-35725-1_22

2013-01-01

Abstract:Many research have been focusing on how to match the textual query with visual images and their surrounding texts or tags for Web image search. The returned results are often unsatisfactory due to their deviation from user intentions. In this paper, we propose a novel image ranking approach to web image search, in which we use social data from social media platform jointly with visual data to improve the relevance between returned images and user intentions (i.e., social relevance). Specifically, we propose a community-specific Social-Visual Ranking(SVR) algorithm to rerank the Web images by taking social relevance into account. Through extensive experiments, we demonstrated the importance of both visual factors and social factors, and the effectiveness and superiority of the social-visual ranking algorithm for Web image search.

Retrieving Relevant and Diverse Image from Social Media Images.

Image Retrieval by Cross-Media Relevance Fusion.

Exploiting Visual-based Intent Classification for Diverse Social Image Retrieval.

Cross-media relevance mining for evaluating text-based image search engine

Social-Sensed Image Search.

Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

Towards Automatic Construction of Diverse, High-Quality Image Datasets

CUHK at ImageCLEF 2005: cross-language and cross-media image retrieval

Diversifying the image retrieval results.

Cross-Modal Image-Tag Relevance Learning for Social Images

Keyword-Based Diverse Image Retrieval by Semantics-aware Contrastive Learning and Transformer.

Hypergraph Spectral Hashing for image retrieval with heterogeneous social contexts

Social Media Analytics and Learning

Geometric Discriminative Features for Aerial Image Retrieval in Social Media

RetriBooru: Leakage-Free Retrieval of Conditions from Reference Images for Subject-Driven Generation

Dataset and Case Studies for Visual Near-Duplicates Detection in the Context of Social Media

A unified approach of detecting misleading images via tracing its instances on web and analyzing its past context for the verification of multimedia content

Social-oriented Visual Image Search

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

On the Diversity of Realistic Image Synthesis.

Social Visual Image Ranking for Web Image Search.