Abstract:Cross-media retrieval is to return the results of various media types corresponding to the query of any media type. Existing researches generally focus on coarse-grained cross-media retrieval. When users submit an image of "Slaty-backed Gull" as a query, coarse-grained cross-media retrieval treats it as "Bird", so that users can only get the results of "Bird", which may include other bird species with similar appearance (image and video), descriptions (text) or sounds (audio), such as "Herring Gull". Such coarse-grained cross-media retrieval is not consistent with human lifestyle, where we generally have the fine-grained requirement of returning the exactly relevant results of "Slaty-backed Gull" instead of "Herring Gull". However, few researches focus on fine-grained cross-media retrieval, which is a highly challenging and practical task. Therefore, in this paper, we first construct a new benchmark for fine-grained cross-media retrieval, which consists of 200 fine-grained subcategories of the "Bird", and contains 4 media types, including image, text, video and audio. To the best of our knowledge, it is the first benchmark with 4 media types for fine-grained cross-media retrieval. Then, we propose a uniform deep model, namely FGCrossNet, which simultaneously learns 4 types of media without discriminative treatments. We jointly consider three constraints for better common representation learning: classification constraint ensures the learning of discriminative features, center constraint ensures the compactness characteristic of the features of the same subcategory, and ranking constraint ensures the sparsity characteristic of the features of different subcategories. Extensive experiments verify the usefulness of the new benchmark and the effectiveness of our FGCrossNet. They will be made available at <a class="link-external link-https" href="https://github.com/PKU-ICST-MIPL/FGCrossNet_ACMMM2019" rel="external noopener nofollow">this https URL</a>.

Facebook5k: A Novel Evaluation Resource Dataset for Cross-Media Search

A Benchmark Dataset and Learning High-Level Semantic Embeddings of Multimedia for Cross-Media Retrieval.

Learning a Semantic Space for Modeling Images, Tags and Feelings in Cross-Media Search.

Image Retrieval Based on Fuzzy Semantic Relevance Matrix

Community-Aware Photo Quality Evaluation by Deeply Encoding Human Perception

What Are The High-Level Concepts With Small Semantic Gaps?

Measuring the Semantic Relatedness Between Images Using Social Tags.

AI-Based Semantic Multimedia Indexing and Retrieval for Social Media on Smartphones

Social-Sensed Image Search.

On the Sampling of Web Images for Learning Visual Concept Classifiers

Measuring semantic relatedness between Flickr images: from a social tag based view.

Cross‐media search method based on complementary attention and generative adversarial network for social networks

Flickr30K-CFQ: A Compact and Fragmented Query Dataset for Text-image Retrieval

Sampled Image Tagging and Retrieval Methods on User Generated Content

An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges

Cross-Media Similarity Evaluation for Web Image Retrieval in the Wild

Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrieval.

Sampling and Ontologically Pooling Web Images for Visual Concept Learning

Brand Data Gathering From Live Social Media Streams

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

Cross-Modal Image-Tag Relevance Learning for Social Images