Embodied question answering

Das Abhishek, Datta Samyak, Gkioxari Georgia, Lee Stefan, Devi Parikh

2018-06-01

Abstract:

What problem does this paper attempt to address?

Embodied Question Answering

Samyak Datta,Georgia Gkioxari,Stefan Lee,Devi Parikh,Abhishek Das,Dhruv Batra

DOI: https://doi.org/10.1109/CVPR.2018.00008

2017-11-30

Abstract:We present a new AI task - Embodied Question Answering (EmbodiedQA) - where an agent is spawned at a random location in a 3D environment and asked a question ('What color is the car?'). In order to answer, the agent must first intelligently navigate to explore the environment, gather necessary visual information through first-person (egocentric) vision, and then answer the question ('orange'). EmbodiedQA requires a range of AI skills - language understanding, visual recognition, active perception, goal-driven navigation, commonsense reasoning, long-term memory, and grounding language into actions. In this work, we develop a dataset of questions and answers in House3D environments [1], evaluation metrics, and a hierarchical model trained with imitation and reinforcement learning.

Computer Science
Knowledge-based Embodied Question Answering

Sinan Tan,Mengmeng Ge,Di Guo,Huaping Liu,Fuchun Sun

DOI: https://doi.org/10.48550/arXiv.2109.07872

2021-09-16

Abstract:In this paper, we propose a novel Knowledge-based Embodied Question Answering (K-EQA) task, in which the agent intelligently explores the environment to answer various questions with the knowledge. Different from explicitly specifying the target object in the question as existing EQA work, the agent can resort to external knowledge to understand more complicated question such as "Please tell me what are objects used to cut food in the room?", in which the agent must know the knowledge such as "knife is used for cutting food". To address this K-EQA problem, a novel framework based on neural program synthesis reasoning is proposed, where the joint reasoning of the external knowledge and 3D scene graph is performed to realize navigation and question answering. Especially, the 3D scene graph can provide the memory to store the visual information of visited scenes, which significantly improves the efficiency for the multi-turn question answering. Experimental results have demonstrated that the proposed framework is capable of answering more complicated and realistic questions in the embodied environment. The proposed method is also applicable to multi-agent scenarios.

Robotics
S-EQA: Tackling Situational Queries in Embodied Question Answering

Vishnu Sashank Dorbala,Prasoon Goyal,Robinson Piramuthu,Michael Johnston,Reza Ghanadhan,Dinesh Manocha

2024-10-26

Abstract:We present and tackle the problem of Embodied Question Answering (EQA) with Situational Queries (S-EQA) in a household environment. Unlike prior EQA work tackling simple queries that directly reference target objects and properties ("What is the color of the car?"), situational queries (such as "Is the house ready for sleeptime?") are more challenging requiring the agent to identify multiple objects (Doors: Closed, Lights: Off, etc.) and reach a consensus on their states for an answer. Towards this objective, we first introduce a novel Prompt-Generate-Evaluate (PGE) scheme that wraps around an LLM's output to create a dataset of unique situational queries and corresponding consensus object information. PGE maintains uniqueness among the generated queries, using semantic similarity via a feedback loop. We annotate the generated data for ground truth answers via a large scale user-study conducted on M-Turk, and with a high answerability rate of 97.26%, establish that LLMs are good at generating situational data. However, using the same LLM to answer the queries gives a low success rate of 46.2%; indicating that while LLMs are good at generating query data, they are poor at answering them. We use images from the VirtualHome simulator with the S-EQA queries establish an evaluation benchmark via Visual Question Answering (VQA). We report an improved accuracy of 15.31% while using queries framed from the generated object consensus for VQA over directly answering situational ones, indicating that such simplification is necessary for improved performance. To the best of our knowledge, this is the first work to introduce EQA in the context of situational queries that also uses a generative approach for query creation. We aim to foster research on improving the real-world usability of embodied agents in household environments through this work.

Robotics,Artificial Intelligence
Embodied Question Answering in Photorealistic Environments With Point Cloud Perception

Erik Wijmans,Samyak Datta,Oleksandr Maksymets,Abhishek Das,Georgia Gkioxari,Stefan Lee,Irfan Essa,Devi Parikh,Dhruv Batra

DOI: https://doi.org/10.1109/cvpr.2019.00682

2019-06-01

Abstract:To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task – Embodied Question Answering [1] in photo-realistic environments (Matterport 3D). We thoroughly study navigation policies that utilize 3D point clouds, RGB images, or their combination. Our analysis of these models reveals several key findings. We find that two seemingly naive navigation baselines, forward-only and random, are strong navigators and challenging to outperform, due to the specific choice of the evaluation setting presented by [1]. We find a novel loss-weighting scheme we call Inflection Weighting to be important when training recurrent models for navigation with behavior cloning and are able to out perform the baselines with this technique. We find that point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating the use (and continued study) of 3D deep learning models for embodied navigation.
GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering

Saumya Saxena,Blake Buchanan,Chris Paxton,Bingqing Chen,Narunas Vaskevicius,Luigi Palmieri,Jonathan Francis,Oliver Kroemer

2024-12-19

Abstract:In Embodied Question Answering (EQA), agents must explore and develop a semantic understanding of an unseen environment in order to answer a situated question with confidence. This remains a challenging problem in robotics, due to the difficulties in obtaining useful semantic representations, updating these representations online, and leveraging prior world knowledge for efficient exploration and planning. Aiming to address these limitations, we propose GraphEQA, a novel approach that utilizes real-time 3D metric-semantic scene graphs (3DSGs) and task relevant images as multi-modal memory for grounding Vision-Language Models (VLMs) to perform EQA tasks in unseen environments. We employ a hierarchical planning approach that exploits the hierarchical nature of 3DSGs for structured planning and semantic-guided exploration. Through experiments in simulation on the HM-EQA dataset and in the real world in home and office environments, we demonstrate that our method outperforms key baselines by completing EQA tasks with higher success rates and fewer planning steps.

Robotics,Computation and Language,Computer Vision and Pattern Recognition,Machine Learning
Embodied Referring Expression for Manipulation Question Answering in Interactive Environment

Qie Sima,Sinan Tan,Huaping Liu

DOI: https://doi.org/10.48550/arXiv.2210.02709

IF: 3.7

2022-10-06

Robotics

Abstract:Embodied agents are expected to perform more complicated tasks in an interactive environment, with the progress of Embodied AI in recent years. Existing embodied tasks including Embodied Referring Expression (ERE) and other QA-form tasks mainly focuses on interaction in term of linguistic instruction. Therefore, enabling the agent to manipulate objects in the environment for exploration actively has become a challenging problem for the community. To solve this problem, We introduce a new embodied task: Remote Embodied Manipulation Question Answering (REMQA) to combine ERE with manipulation tasks. In the REMQA task, the agent needs to navigate to a remote position and perform manipulation with the target object to answer the question. We build a benchmark dataset for the REMQA task in the AI2-THOR simulator. To this end, a framework with 3D semantic reconstruction and modular network paradigms is proposed. The evaluation of the proposed framework on the REMQA dataset is presented to validate its effectiveness.
Revisiting EmbodiedQA: A Simple Baseline and Beyond.

Yu Wu,Lu Jiang,Yi Yang

DOI: https://doi.org/10.1109/tip.2020.2967584

IF: 10.6

2020-01-01

IEEE Transactions on Image Processing

Abstract:In Embodied Question Answering (EmbodiedQA), an agent interacts with an environment to gather necessary information for answering user questions. Existing works have laid a solid foundation towards solving this interesting problem. But the current performance, especially in navigation, suggests that EmbodiedQA might be too challenging for the contemporary approaches. In this paper, we empirically study this problem and introduce 1) a simple yet effective baseline that achieves promising performance; 2) an easier and practical setting for EmbodiedQA where an agent has a chance to adapt the trained model to a new environment before it actually answers users questions. In this new setting, we randomly place a few objects in new environments, and upgrade the agent policy by a distillation network to retain the generalization ability from the trained model. On the EmbodiedQA v1 benchmark, under the standard setting, our simple baseline achieves very competitive results to the-state-of-the-art; in the new setting, we found the introduced small change in settings yields a notable gain in navigation.
Map-based Modular Approach for Zero-shot Embodied Question Answering

Koya Sakamoto,Daichi Azuma,Taiki Miyanishi,Shuhei Kurita,Motoaki Kawanabe

2024-10-13

Abstract:Embodied Question Answering (EQA) serves as a benchmark task to evaluate the capability of robots to navigate within novel environments and identify objects in response to human queries. However, existing EQA methods often rely on simulated environments and operate with limited vocabularies. This paper presents a map-based modular approach to EQA, enabling real-world robots to explore and map unknown environments. By leveraging foundation models, our method facilitates answering a diverse range of questions using natural language. We conducted extensive experiments in both virtual and real-world settings, demonstrating the robustness of our approach in navigating and comprehending queries within unknown environments.

Robotics,Computer Vision and Pattern Recognition
Learning by Asking for Embodied Visual Navigation and Task Completion

Ying Shen,Ismini Lourentzou

DOI: https://doi.org/10.48550/arXiv.2302.04865

2023-02-09

Computer Vision and Pattern Recognition

Abstract:The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Despite recent progress on related vision-language benchmarks, most prior work has focused on building agents that follow instructions rather than endowing agents the ability to ask questions to actively resolve ambiguities arising naturally in embodied environments. To empower embodied agents with the ability to interact with humans, in this work, we propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. We evaluate our model on the TEACH vision-dialog navigation and task completion dataset. Experimental results show that ELBA achieves improved task performance compared to baseline models without question-answering capabilities.
Explore until Confident: Efficient Exploration for Embodied Question Answering

Allen Z. Ren,Jaden Clark,Anushri Dixit,Masha Itkina,Anirudha Majumdar,Dorsa Sadigh

2024-07-08

Abstract:We consider the problem of Embodied Question Answering (EQA), which refers to settings where an embodied agent such as a robot needs to actively explore an environment to gather information until it is confident about the answer to a question. In this work, we leverage the strong semantic reasoning capabilities of large vision-language models (VLMs) to efficiently explore and answer such questions. However, there are two main challenges when using VLMs in EQA: they do not have an internal memory for mapping the scene to be able to plan how to explore over time, and their confidence can be miscalibrated and can cause the robot to prematurely stop exploration or over-explore. We propose a method that first builds a semantic map of the scene based on depth information and via visual prompting of a VLM - leveraging its vast knowledge of relevant regions of the scene for exploration. Next, we use conformal prediction to calibrate the VLM's question answering confidence, allowing the robot to know when to stop exploration - leading to a more calibrated and efficient exploration strategy. To test our framework in simulation, we also contribute a new EQA dataset with diverse, realistic human-robot scenarios and scenes built upon the Habitat-Matterport 3D Research Dataset (HM3D). Both simulated and real robot experiments show our proposed approach improves the performance and efficiency over baselines that do no leverage VLM for exploration or do not calibrate its confidence. Webpage with experiment videos and code: <a class="link-external link-https" href="https://explore-eqa.github.io/" rel="external noopener nofollow">this https URL</a>

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning
EfficientEQA: An Efficient Approach for Open Vocabulary Embodied Question Answering

Kai Cheng,Zhengyuan Li,Xingpeng Sun,Byung-Cheol Min,Amrit Singh Bedi,Aniket Bera

2024-10-27

Abstract:Embodied Question Answering (EQA) is an essential yet challenging task for robotic home assistants. Recent studies have shown that large vision-language models (VLMs) can be effectively utilized for EQA, but existing works either focus on video-based question answering without embodied exploration or rely on closed-form choice sets. In real-world scenarios, a robotic agent must efficiently explore and accurately answer questions in open-vocabulary settings. To address these challenges, we propose a novel framework called EfficientEQA for open-vocabulary EQA, which enables efficient exploration and accurate answering. In EfficientEQA, the robot actively explores unknown environments using Semantic-Value-Weighted Frontier Exploration, a strategy that prioritizes exploration based on semantic importance provided by calibrated confidence from black-box VLMs to quickly gather relevant information. To generate accurate answers, we employ Retrieval-Augmented Generation (RAG), which utilizes BLIP to retrieve useful images from accumulated observations and VLM reasoning to produce responses without relying on predefined answer choices. Additionally, we detect observations that are highly relevant to the question as outliers, allowing the robot to determine when it has sufficient information to stop exploring and provide an answer. Experimental results demonstrate the effectiveness of our approach, showing an improvement in answering accuracy by over 15% and efficiency, measured in running steps, by over 20% compared to state-of-the-art methods.

Robotics,Artificial Intelligence,Computer Vision and Pattern Recognition
Central Answer Modeling for an Embodied Multi-LLM System

Bhrij Patel,Vishnu Sashank Dorbala,Amrit Singh Bedi,Dinesh Manocha

2024-09-16

Abstract:Embodied Question Answering (EQA) is an important problem, which involves an agent exploring the environment to answer user queries. In the existing literature, EQA has exclusively been studied in single-agent scenarios, where exploration can be time-consuming and costly. In this work, we consider EQA in a multi-agent framework involving multiple large language models (LLM) based agents independently answering queries about a household environment. To generate one answer for each query, we use the individual responses to train a Central Answer Model (CAM) that aggregates responses for a robust answer. While prior Question Answering (QA) work has used a central module based on answers from multiple LLM-based experts, we specifically look at applying this framework to embodied LLM-based agents that must physically explore the environment first to become experts on their given environment to answer questions. Our work is the first to utilize a central answer model framework with embodied agents that must rely on exploring an unknown environment. We set up a variation of EQA where instead of the agents exploring the environment after the question is asked, the agents first explore the environment for a set amount of time and then answer a set of queries. Using CAM, we observe a $46\%$ higher EQA accuracy when compared against aggregation methods for ensemble LLM, such as voting schemes and debates. CAM does not require any form of agent communication, alleviating it from the associated costs. We ablate CAM with various nonlinear (neural network, random forest, decision tree, XGBoost) and linear (logistic regression classifier, SVM) algorithms. We experiment in various topological graph environments and examine the case where one of the agents is malicious and purposes contribute responses it believes to be wrong.

Machine Learning,Artificial Intelligence,Computation and Language
ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion

Ying Shen,Ismini Lourentzou

2024-12-06

Abstract:The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Although there have been significant advancements in related vision-language benchmarks, most prior work has focused on building agents that follow instructions rather than endowing agents the ability to ask questions to actively resolve ambiguities arising naturally in embodied environments. To address this gap, we propose an Embodied Learning-By-Asking (ELBA) model that learns when and what questions to ask to dynamically acquire additional information for completing the task. We evaluate ELBA on the TEACh vision-dialog navigation and task completion dataset. Experimental results show that the proposed method achieves improved task performance compared to baseline models without question-answering capabilities.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Machine Learning
Depth and Video Segmentation Based Visual Attention for Embodied Question Answering

Haonan Luo,Guosheng Lin,Yazhou Yao,Fayao Liu,Zichuan Liu,Zhenmin Tang

DOI: https://doi.org/10.1109/TPAMI.2021.3139957

Abstract:Embodied Question Answering (EQA) is a newly defined research area where an agent is required to answer the user's questions by exploring the real-world environment. It has attracted increasing research interests due to its broad applications in personal assistants and in-home robots. Most of the existing methods perform poorly in terms of answering and navigation accuracy due to the absence of fine-level semantic information, stability to the ambiguity, and 3D spatial information of the virtual environment. To tackle these problems, we propose a depth and segmentation based visual attention mechanism for Embodied Question Answering. First, we extract local semantic features by introducing a novel high-speed video segmentation framework. Then guided by the extracted semantic features, a depth and segmentation based visual attention mechanism is proposed for the Visual Question Answering (VQA) sub-task. Further, a feature fusion strategy is designed to guide the navigator's training process without much additional computational cost. The ablation experiments show that our method effectively boosts the performance of the VQA module and navigation module, leading to 4.9 % and 5.6 % overall improvement in EQA accuracy on House3D and Matterport3D datasets respectively.
Entailment-based Question Answering for Structured Data.

Bogdan Sacaleanu,Constantin Orasan,Christian Spurk,Shiyan Ou,Óscar Ferrández,Milen Kouylekov,Matteo Negri

2008-01-01

Abstract:This paper describes a Question Answering system which retrieves answers from structured data regarding cinemas and movies. The system represents the first prototype of a multilingual and multimodal QA system for the domain of tourism. Based on specially designed domain ontology and using Textual Entailment as a means for semantic inference, the system can be used in both monolingual and cross-language settings with slight adjustments for new input languages.
Multi-agent Embodied Question Answering in Interactive Environments

Sinan Tan,Weilai Xiang,Huaping Liu,Di Guo,Fuchun Sun

DOI: https://doi.org/10.1007/978-3-030-58601-0_39

2020-01-01

Abstract:We investigate a new AI task—Multi-Agent Interactive Question Answering—where several agents explore the scene jointly in interactive environments to answer a question. To cooperate efficiently and answer accurately, agents must be well-organized to have balanced work division and share knowledge about the objects involved. We address this new problem in two stages: Multi-Agent 3D Reconstruction in Interactive Environments and Question Answering. Our proposed framework features multi-layer structural and semantic memories shared by all agents, as well as a question answering model built upon a 3D-CNN network to encode the scene memories. During the reconstruction, agents simultaneously explore and scan the scene with a clear division of work, organized by next viewpoints planning. We evaluate our framework on the IQuADv1 dataset and outperform the IQA baseline in a single-agent scenario. In multi-agent scenarios, our framework shows favorable speedups while remaining high accuracy.
A Visual Question and Answering System with Support for Compound Emotions using Facial Landmark Identification with MediaPipe and CNN classifier

Lavika Goel,Nilarnab Debnath,Sanskar Mundaniya

DOI: https://doi.org/10.1016/j.neucom.2024.127623

IF: 6

2024-04-05

Neurocomputing

Abstract:Visual Question and Answering is a fast-evolving field of research where we attempt to answer the question based on an image as a context. In this paper, we try to add support for answering questions that are based on emotions with face detection and basic emotion extraction. With the basic emotions, we further try to find out compound emotions to expand the area of answerable questions. We complete the said operations in two stages: i) Caption generation. ii) Question and answering with that caption as context. The context generation process can be further divided into i) general caption generator ii) emotional caption generator. In general caption generator, we try to find a single sentence providing a basic information about the input context image, using an Attention [9] based model. In the later phase, the emotional caption generator finds the information of the emotions of the faces of the image using RetinaFace [20] and DeepFace (Taigman et al., 2014). In emotional caption generator, apart from basic emotion generation, we also try to find compound emotions depicted by each of the faces by learning the patterns in facial landmarks generated by MediaPipe (Vardhan et al., 2020) with CNN based architecture. In this module, we also generate a set of know question and answers to use in the next phase. As for the question and answering part, we first use BERT (Devlin et al., 2020) for finding an answer, given a question and the captions generated as context of the question. If the confidence of answer exceeds a threshold value, we report that answer as final. If not, we try to find the known question similar to the asked question. The answer generated then is the final answer.

computer science, artificial intelligence
Grounded Question-Answering in Long Egocentric Videos

Shangzhe Di,Weidi Xie

2024-04-01

Abstract:Existing approaches to video understanding, mainly designed for short videos from a third-person perspective, are limited in their applicability in certain fields, such as robotics. In this paper, we delve into open-ended question-answering (QA) in long, egocentric videos, which allows individuals or robots to inquire about their own past visual experiences. This task presents unique challenges, including the complexity of temporally grounding queries within extensive video content, the high resource demands for precise data annotation, and the inherent difficulty of evaluating open-ended answers due to their ambiguous nature. Our proposed approach tackles these challenges by (i) integrating query grounding and answering within a unified model to reduce error propagation; (ii) employing large language models for efficient and scalable data synthesis; and (iii) introducing a close-ended QA task for evaluation, to manage answer ambiguity. Extensive experiments demonstrate the effectiveness of our method, which also achieves state-of-the-art performance on the QaEgo4D and Ego4D-NLQ benchmarks. Code, data, and models are available at

Computer Vision and Pattern Recognition

Embodied question answering

Embodied Question Answering

Knowledge-based Embodied Question Answering

S-EQA: Tackling Situational Queries in Embodied Question Answering

Embodied Question Answering in Photorealistic Environments With Point Cloud Perception

GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering

Embodied Referring Expression for Manipulation Question Answering in Interactive Environment

Revisiting EmbodiedQA: A Simple Baseline and Beyond.

Map-based Modular Approach for Zero-shot Embodied Question Answering

Learning by Asking for Embodied Visual Navigation and Task Completion

Explore until Confident: Efficient Exploration for Embodied Question Answering

EfficientEQA: An Efficient Approach for Open Vocabulary Embodied Question Answering

Central Answer Modeling for an Embodied Multi-LLM System

ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion

Depth and Video Segmentation Based Visual Attention for Embodied Question Answering

Entailment-based Question Answering for Structured Data.

Multi-agent Embodied Question Answering in Interactive Environments

A Visual Question and Answering System with Support for Compound Emotions using Facial Landmark Identification with MediaPipe and CNN classifier

Grounded Question-Answering in Long Egocentric Videos