Abstract:The ubiquitous mobile devices have led to the unprecedented growing of personal photo collections on the phone. One significant pain point of today's mobile users is instantly finding specific photos of what they want. Existing applications (e.g., Google Photo and OneDrive) have predominantly focused on cloud-based solutions, while leaving the client-side challenges (e.g., query formulation, photo tagging and search, etc.) unsolved. This considerably hinders user experience on the phone. In this paper, we present an innovative personal photo search system on the phone, which enables instant and accurate photo search by visual query suggestion and joint text-image hashing. Specifically, the system is characterized by several distinctive properties: 1) visual query suggestion (VQS) to facilitate the formulation of queries in a joint text-image form, 2) light-weight convolutional and sequential deep neural networks to extract representations for both photos and queries, and 3) joint text-image hashing (with compact binary codes) to facilitate binary image search and VQS. It is worth noting that all the components run on the phone with client optimization by deep learning techniques. We have collected 270 photo albums taken by 30 mobile users (corresponding to 37,000 personal photos) and conducted a series of field studies. We show that our system significantly outperforms the existing client-based solutions by 10 x in terms of search efficiency, and 92.3% precision in terms of search accuracy, leading to a remarkably better user experience of photo discovery on the phone.

Opportunistic Content Search of Smartphone Photos

Searching Personal Photos on the Phone with Instant Visual Query Suggestion and Joint Text-Image Hashing.

Analysis and Applications of Smartphone User Mobility

Photo-to-Search: Using Camera Phones to Inquire of the Surrounding World

Personalized Multi-Modality Image Management and Search for Mobile Devices

Scalable Mobile Image Retrieval by Exploring Contextual Saliency

Photo-to-search: using multimodal queries to search the web from mobile devices.

Storage Management Strategy in Mobile Phones for Photo Crowdsensing.

Mobile Image Search with Multimodal Context-Aware Queries

Crowdsourcing Based Event Reporting System Using Smartphones with Accurate Localization and Photo Tamper Detection

POP: Privacy-Preserving Outsourced Photo Sharing and Searching for Mobile Devices

A Novel Image Retrieval System Based on Mobile Platform

Privacy-preserving content-based image retrieval for mobile computing

Photo search by face positions and facial attributes on touch devices.

From Uncertain Photos to Certain Coverage: a Novel Photo Selection Approach to Mobile Crowdsensing.

Semantic-Based Graph Index for Mobile Photo Search

Imagilar: A Real-Time Image Similarity Search System On Mobile Platform

Photo 4W: Mobile Photo Management on What, Where, Who and When

Energy-aware keyword search on mobile phones.

Spatial Verification for Scalable Mobile Image Retrieval

Smartphone picture organization: A hierarchical approach