Abstract:Language is highly structured, with syntactic and semantic structures, to some extent, agreed upon by speakers of the same language. With implicit or explicit awareness of such structures, humans can learn and use language efficiently and generalize to sentences that contain unseen words. Motivated by human language learning, in this dissertation, we consider a family of machine learning tasks that aim to learn language structures through grounding. We seek distant supervision from other data sources (i.e., grounds), including but not limited to other modalities (e.g., vision), execution results of programs, and other languages. We demonstrate the potential of this task formulation and advocate for its adoption through three schemes. In Part I, we consider learning syntactic parses through visual grounding. We propose the task of visually grounded grammar induction, present the first models to induce syntactic structures from visually grounded text and speech, and find that the visual grounding signals can help improve the parsing quality over language-only models. As a side contribution, we propose a novel evaluation metric that enables the evaluation of speech parsing without text or automatic speech recognition systems involved. In Part II, we propose two execution-aware methods to map sentences into corresponding semantic structures (i.e., programs), significantly improving compositional generalization and few-shot program synthesis. In Part III, we propose methods that learn language structures from annotations in other languages. Specifically, we propose a method that sets a new state of the art on cross-lingual word alignment. We then leverage the learned word alignments to improve the performance of zero-shot cross-lingual dependency parsing, by proposing a novel substructure-based projection method that preserves structural knowledge learned from the source language.

Grounded Lexicon Acquisition - Case Studies in Spatial Language

Grounding Language for Robotic Manipulation via Skill Library

Grounded Language Acquisition From Object and Action Imagery

Interactive Grounded Language Acquisition and Generalization in a 2D World

Crossmodal Language Grounding in an Embodied Neurocognitive Model

Grounded language acquisition through the eyes and ears of a single child

Collaborative Language Grounding Toward Situated Human‐Robot Dialogue

Grounding Dynamic Spatial Relations for Embodied (Robot) Interaction

Grounded 3D-LLM with Referent Tokens

Experience Grounds Language

Grounding Language with Visual Affordances over Unstructured Data

Context-based Word Acquisition for Situated Dialogue in a Virtual World

Grounding Language Plans in Demonstrations Through Counterfactual Perturbations

Grounding Language to Autonomously-Acquired Skills via Goal Generation

Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction

Learning Language Structures through Grounding

Grounding Object Relations in Language-Conditioned Robotic Manipulation with Semantic-Spatial Reasoning

Exploring the acquisition and production of grammatical constructions through human-robot interaction with echo state networks

Pretraining on Interactions for Learning Grounded Affordance Representations

LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent

Language bootstrapping: learning word meanings from perception-action association