Abstract:When determining navigation actions, it is important to design effective visual and semantic representations of the observation scenes and robust navigation strategies. The paper proposes a goal-oriented visual semantic navigation method using semantic knowledge graph and transformer. Two kinds of knowledge graphs representing the location relationship between objects are constructed, namely current knowledge graph and prior knowledge graph. The pre-constructed prior knowledge graph is periodically updated by the current knowledge graph obtained in real time, and embedded into the semantic feature vector through graph convolutional network (GCN). The semantic features and extracted scene features are jointly embedded and stored, they are jointly fed into the transformer module to explore the spatio-temporal dependencies between objects in the environment. The navigation strategy is obtained from the Asynchronous Advantage Actor-Critic (A3C) model composed of Long-Short Term Memory (LSTM) and Multi-Layer Perception (MLP). Experiments show that the knowledge graph can significantly improve the navigation performance. More importantly, our experimental results show that our method can improve the generalization ability of navigation across novel scenes and novel objects. Video can be available at https://youtu.be/ZMjNvoK2rbY. Note to Practitioners — The motivation of this work is to develop an efficient visual semantic navigation method. Conventional navigation algorithms lack semantic information and learning ability, and can not adapt to the complex unknown environments. When semantic information is included in navigation, the location relationship between objects can be obtained as a prior knowledge, which can be combined with reinforcement learning to achieve autonomous navigation of agents. In this article, a knowledge graph representing the location relationships between objects has been constructed and regularly updated in real-time. The proposed visual semantic navigation method further improves the generalization ability of navigation. This navigation method can be applied to mobile robots and deployed in many scenarios such as home, restaurant, hospitals, and even factories.

Knowledge Distillation on Driving Intention Generator: Learn Human-like Semantic Reasoning.

Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectory

DeepGoal: Learning to drive with driving intention from human control demonstration

World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

Coaching a Teachable Student

DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models

Spirit Distillation: Precise Real-time Semantic Segmentation of Road Scenes with Insufficient Data

Semantically Adversarial Scenario Generation with Explicit Knowledge Guidance

Humanlike Driving: Empirical Decision-Making System for Autonomous Vehicles

Goal-Oriented Visual Semantic Navigation Using Semantic Knowledge Graph and Transformer

SurrealDriver: Designing LLM-powered Generative Driver Agent Framework based on Human Drivers' Driving-thinking Data

Transferable and Adaptable Driving Behavior Prediction

Learning a Decision Module by Imitating Driver's Control Behaviors

Intuitive Decision-Making Modeling for Self-Driving Vehicles

Driver Behavioral Cloning for Route Following in Autonomous Vehicles Using Task Knowledge Distillation

Improving Synthetic to Realistic Semantic Segmentation with Parallel Generative Ensembles for Autonomous Urban Driving

Towards Knowledge-driven Autonomous Driving

Towards Scalable & Efficient Interaction-Aware Planning in Autonomous Vehicles using Knowledge Distillation

A Language Agent for Autonomous Driving

Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

Human experience knowledge induction based intelligent train driving