Abstract:Author(s): Wang, Xin | Advisor(s): Wang, William Y.; Wang, Yuan-Fang | Abstract: Grounding natural language onto real-world perception is a fundamental challenge to empower various practical applications that require human-machine communication.This Ph.D. dissertation presents our research thrusts on developing intelligent embodied agents that connect language, vision, and actions. It consists of two major directions towards human-machine communication. First, from visual recognition to cognition, we study using natural language to describe the visual information and express the understanding of the visual world.We show that scalable learning approaches based on semantics and external knowledge can generate coherent, fine-grained, and generalizable visual descriptions.We also investigate the automatic evaluation metrics for language generation, and propose an adversarial reward learning method to overcome their limitations in optimizing the policy for human-like visual storytelling. Besides, we introduce a multilingual dataset for video-and-language research, which goes beyond monolingual language grounding in vision.Second, in order to connect language and vision to actions, we have situated natural language inside interactive environments where communication often occurs.We systematically study the task of vision-language navigation, aiming to associate human commands with visual perception and navigate the 3D world.Then we demonstrate novel methods to tackle generalization and data scarcity issues from various perspectives like counterfactual thinking, transfer learning, multitask learning and agnostic learning. Our efforts shed light on generalizing embodied navigation agents to more challenging and practical scenarios.Finally, we summarize the strengths, weaknesses, and implications of our work, and discuss the future research plan of pushing embodied AI research further.

Egocentric perception of walking environments using an interactive vision-language system

SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction

Egocentric Vision Language Planning

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

Multimodal Large Language Model for Visual Navigation

TEXT2TASTE: A Versatile Egocentric Vision System for Intelligent Reading Assistance Using Large Language Model

I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots

Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination

Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Images

Enhancing Robot Explanation Capabilities through Vision-Language Models: a Preliminary Study by Interpreting Visual Inputs for Improved Human-Robot Interaction

Closing the Loop Between Language and Vision for Embodied Agents

Proactive Human-Robot Interaction using Visuo-Lingual Transformers

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning

Vision and language: from visual perception to content creation

Octopus: Embodied Vision-Language Programmer from Environmental Feedback

Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation

Tackling Vision Language Tasks Through Learning Inner Monologues