Abstract:Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at <a class="link-external link-https" href="https://github.com/linhuixiao/OneRef" rel="external noopener nofollow">this https URL</a>.

Modeling Collaborative Referring for Situated Referential Grounding

Towards Situated Dialogue: Revisiting Referring Expression Generation.

Towards Mediating Shared Perceptual Basis in Situated Dialogue.

Reference-Centric Models for Grounded Collaborative Dialogue

Probabilistic Labeling for Efficient Referential Grounding Based on Collaborative Discourse.

Grounding Language in Multi-Perspective Referential Communication

Collaborative Language Grounding Toward Situated Human‐Robot Dialogue

Collaborative Effort Towards Common Ground in Situated Human-Robot Dialogue.

Integrating Word Acquisition and Referential Grounding Towards Physical World Interaction

Collaboration on reference to objects that are not mutually known

Any Other Thoughts, Hedgehog? Linking Deliberation Chains in Collaborative Dialogues

Reference Resolution and Context Change in Multimodal Situated Dialogue for Exploring Data Visualizations

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

Joint Visual Grounding with Language Scene Graphs

Resolving References in Visually-Grounded Dialogue via Text Generation

Transcrib3D: 3D Referring Expression Resolution through Large Language Models

Towards Collaborative Plan Acquisition through Theory of Mind Modeling in Situated Dialogue