Abstract:Image search reranking which aims to improve the text-based image search results with the help of other cues has grown into a hot research topic. Most existing reranking methods only focus on image visual cues. However, the visual cues cannot always guarantee to provide enough information for the reranking process. Although, some approaches try to fuse multiple image cues for reranking, they do not or weakly exploit the relationships among multiple image cues. In this paper,we present a novel image reranking framework--Joint-Rerank which considers multiple modalities of images (or multiple cues) jointly as interdependent attributes of an image entity. Joint-Rerank models the images as a multigraph where each image is a node with multimodal attributes (textual and visual cues) and the parallel edges between nodes measure both image intra-modal and inter-modal similarities. Besides, each node has a "self-consistency" that measures how much the multiple modalities of an image may be consistent. To solve the reranking problem, we first degenerate the multigraph into a new complete graph, and then employ a random walk on the degenerated graph to propagate the relevance scores of each node. Finally, the relevance scores of multiple modalities are fused to rank the images. Moreover, in Joint-Rerank, "cross-modal" walk is possible, i.e., a surfer can jump from one image to another following both intra-modal and inter-modal links. In this framework, we propose two methods: Sym-Joint-Rerank and Asym-Joint-Rerank which use different approaches to measure the inter-modal similarities between two nodes. Experimental results on a large web queries dataset which contains 353 image search queries show that both of them are superior or highly competitive to several state-of-the-art reranking algorithms.

Unified Hypergraph For Image Ranking In A Multimodal Context

Multimodal Information Joint Learning for Geotagged Image Search.

Image ranking via attribute boosted hypergraph

Combinative Hypergraph Learning in Subspace for Cross-Modal Ranking.

Multi-Modal Knowledge Hypergraph for Diverse Image Retrieval.

Adaptive Multimodal Hypergraph Learning for Image Classification

Multimodal graph-based reranking for web image search.

Hypergraph-Enhanced Hashing for Unsupervised Cross-Modal Retrieval via Robust Similarity Guidance

Content-Based Visual Landmark Search Via Multimodal Hypergraph Learning.

Multimodal graph-based reranking for web image search

Joint Hypergraph Learning for Tag-Based Image Retrieval

Scalable graph based non-negative multi-view embedding for image ranking

Joint-Rerank: a Novel Method for Image Search Reranking

Multiple Hypergraph Ranking for Video Concept Detection

Unsupervised Concatenation Hashing Via Combining Subspace Learning and Graph Embedding for Cross-Modal Image Retrieval

Hypergraph Spectral Hashing for image retrieval with heterogeneous social contexts

Joint-rerank

Multimodal collaborative graph for image recommendation

Social Neighborhood Graph and Multigraph Fusion Ranking for Multifeature Image Retrieval

A Unified Framework of Image Latent Feature Learning on Sina Microblog

Feature Correlation Hypergraph: Exploiting High-order Potentials for Multimodal Recognition