Abstract:Vision-and-Language Navigation (VLN) is unique in that it requires turning relatively general natural-language instructions into robot agent actions, on the basis of visible environments. This requires to extract value from two very different types of natural-language information. The first is object description (e.g., ‘table’, ‘door’), each presenting as a tip for the agent to determine the next action by finding the item visible in the environment, and the second is action specification (e.g., ‘go straight’, ‘turn left’) which allows the robot to directly predict the next movements without relying on visual perceptions. However, most existing methods pay few attention to distinguish these information from each other during instruction encoding and mix together the matching between textual object/action encoding and visual perception/orientation features of candidate viewpoints. In this paper, we propose an Object-and-Action Aware Model (OAAM) that processes these two different forms of natural language based instruction separately. This enables each process to match object-centered/action-centered instruction to their own counterpart visual perception/action orientation flexibly. However, one side-issue caused by above solution is that an object mentioned in instructions may be observed in the direction of two or more candidate viewpoints, thus the OAAM may not predict the viewpoint on the shortest path as the next action. To handle this problem, we design a simple but effective path loss to penalize trajectories deviating from the ground truth path. Experimental results demonstrate the effectiveness of the proposed model and path loss, and the superiority of their combination with a 50%documentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$$50\%$$end{document} SPL score on the R2R dataset and a 40%documentclass[12pt]{minimal}usepackage{amsmath}usepackage{wasysym}usepackage{amsfonts}usepackage{amssymb}usepackage{amsbsy}usepackage{mathrsfs}usepackage{upgreek}setlength{oddsidemargin}{-69pt}egin{document}$$40\%$$end{document} CLS score on the R4R dataset in unseen environments, outperforming the previous state-of-the-art.

Design and Implementation of the Navigation Tool of AML Model

Improvement and Implementation of ODMG Object Manipulating Language

Design and implementation of modeling language AML

Modeling Principles of the Sequence Diagram and Its Application in MDA Software Paradigm

Object-oriented Methodology Meets MDA Software Paradigm

A Task-Oriented Navigation Approach to Enhance Architectural Description Comprehension

Modeling Business for MDA Software Paradigm

Motion navigation for arc welding robots based on feature mapping in a simulation environment

Modeling Mechanism and Application of Sequence Diagram

Data Model For Moving Objects To Support Intelligent Vehicle Navigation

A Study on Object-Oriented Modeling in AML

Separating Data And View: Support View-Wandering Between Different Trades During Engineering Design

Study On The Real Time Navigation Data Model For Dynamic Navigation

Object-and-Action Aware Model for Visual Language Navigation

Integration System for Overall Design of Unmanned Underwater Vehicle (UUV) Warhead Based on Adapted Model Language(AML)

OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graph

An adaptive interactive multi-model navigation method based on UUV

A Mapping Approach for MDA Model Transformation Based on J2EE Platform

An LOD model for graph visualization and its application in web navigation

Development of Software Modeling Tools Based on UML

Design and implementation of a navigation system for autonomous mobile robots