Abstract:Weakly supervised phrase grounding aims to map the phrases in an image caption to the objects appearing in the image under the supervision of image-caption correspondence. We observe that the current studies are insufficient to model the complicated interactions between the visual components (i.e., the visual regions) and between the visual and textual components (i.e., the phrases). Therefore, this paper presents a novel weakly supervised learning approach to phrase grounding in which we systematically model the visual contextualized representation with three modules: (1) object proposals pooling (OPP), (2) visual self-attention (VSA) and (3) visual-textual cross-modal attention (VTCA). OPP alleviates the suppression of the object proposals and benefits the visual representation in terms of trading off the richness of the visual components and the computational efficiency. VSA aims to capture the correlation among the object proposals and generate a representation of each proposal by incorporating the visual information of the others. To measure the cross-modal compatibility in terms of topics, we introduce the VTCA module to represent the visual topic corresponding to each textual component in a cross-modal common vector space. In the training process, we build a mixed contrastive loss function by considering both the cross-modal compatibility and the differences in the visual representations in the VSA module. Compared with the state-of-the-art methods, the proposed approach improves the performance by 3.88% points and 1.24% points on R@1 , and by 2.23% points and 0.26% points on Pt_Acc , when trained on the MS COCO and Flickr30K Entities training sets, respectively. We have made our code available for follow-up research.

Grounding learning of modifier dynamics: An application to color naming

Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention

Pragmatically Informative Color Generation by Grounding Contextual Modifiers

Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

Learning to Color from Language

Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language

L-C4: Language-Based Video Colorization for Creative and Consistent Color

Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling

Modeling the emergence of universality in color naming patterns

Exemplar-Based Image Colorization with A Learning Framework

Generating Bilingual Pragmatic Color References

MultiColor: Image Colorization by Learning from Multiple Color Spaces

Colors $-$Messengers of Concepts: Visual Design Mining for Learning Color Semantics

Grounded Language Acquisition From Object and Action Imagery

Deep auxiliary learning for visual localization using colorization task

Improving Weakly Supervised Phrase Grounding Via Visual Representation Contextualization with Contrastive Learning.

Color2Vec: Web-based Modeling of Word-color Association with Sociocultural Contexts

Learning Representations for Automatic Colorization

L-CAD: Language-based Colorization with Any-level Descriptions Using Diffusion Priors

Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision

Illumination Robust Color Naming Via Label Propagation