Abstract:<p>Weakly-supervised video object localization is a challenging yet important task. The system should spatially localize the object of interest in videos, where only the descriptive sentences and their corresponding video segments are given in the training stage. Recent efforts propose to apply image-based Multiple Instance Learning (MIL) theory in this video task, and propagate the supervision from the video into frames by applying different frame-weighting strategies. Despite their promising progress, the spatio-temporal correlation between different object regions in videos has been largely ignored. To fill the research gap, in this work we introduce a simple but effective feature expression and aggregation framework, which utilizes the self-attention mechanism to capture the latent spatio-temporal correlation between multimodal object features and design a multimodal interaction module to model the similarity between the semantic query in sentences and the object regions in videos. We conduct extensive experimental evaluation on the YouCookII and ActivityNet-Entities datasets, which demonstrates significant improvements over multiple competitive baselines.</p>

Weakly-supervised video object localization with attentive spatio-temporal correlation