Abstract:Episodic memory retrieval aims to enable wearable devices with the ability to recollect from past video observations objects or events that have been observed (e.g., "where did I last see my smartphone?"). Despite the clear relevance of the task for a wide range of assistive systems, current task formulations are based on the "offline" assumption that the full video history can be accessed when the user makes a query, which is unrealistic in real settings, where wearable devices are limited in power and storage capacity. We introduce the novel task of Online Episodic Memory Visual Queries Localization (OEM-VQL), in which models are required to work in an online fashion, observing video frames only once and relying on past computations to answer user queries. To tackle this challenging task, we propose ESOM - Egocentric Streaming Object Memory, a novel framework based on an object discovery module to detect potentially interesting objects, a visual object tracker to track their position through the video in an online fashion, and a memory module to store spatio-temporal object coordinates and image representations, which can be queried efficiently at any moment. Comparisons with different baselines and offline methods show that OEM-VQL is challenging and ESOM is a viable approach to tackle the task, with results outperforming offline methods (81.92 vs 55.89 success rate %) when oracular object discovery and tracking are considered. Our analysis also sheds light on the limited performance of object detection and tracking in egocentric vision, providing a principled benchmark based on the OEM-VQL downstream task to assess progress in these areas.

A Two-Level Lifelog Search Engine at the LSC 2019

A Multi-level Interactive Lifelog Search Engine with User Feedback.

Improving Multi-Faceted Book Search by Incorporating Sparse Latent Semantic Analysis of Click-Through Logs

Lifelogging Retrieval based on Semantic Concepts Fusion

Sign Language Video Retrieval with Free-Form Textual Queries

An Interactive Video Search Platform for Multi-modal Retrieval with Advanced Concepts

Lifelog Moment Retrieval With Interactive Watershed-Based Clustering and Hierarchical Similarity Search

Aiding autobiographic memory by using wearable devices

Computer Vision for Lifelogging: Characterizing Everyday Activities Based on Visual Semantics

On the Place of Text Data in Lifelogs, and Text Analysis via Semantic Facets

Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach

Ten Questions in Lifelog Mining and Information Recall

DeepDiary: Automatic Caption Generation for Lifelogging Image Streams

Robust Semantic Sketch Based Specific Image Retrieval

Visual Information Retrieval in Endoscopic Video Archives

Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory

LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

Overview of LiLAS 2020 -- Living Labs for Academic Search

Web Search Via an Efficient and Effective Brain-Machine Interface

Light-Weight Global Feature for Mobile Clothing Search

LifeRec: A Mobile App for Lifelog Recording and Ubiquitous Recommendation