Abstract:We present and tackle the problem of Embodied Question Answering (EQA) with Situational Queries (S-EQA) in a household environment. Unlike prior EQA work tackling simple queries that directly reference target objects and properties ("What is the color of the car?"), situational queries (such as "Is the house ready for sleeptime?") are more challenging requiring the agent to identify multiple objects (Doors: Closed, Lights: Off, etc.) and reach a consensus on their states for an answer. Towards this objective, we first introduce a novel Prompt-Generate-Evaluate (PGE) scheme that wraps around an LLM's output to create a dataset of unique situational queries and corresponding consensus object information. PGE maintains uniqueness among the generated queries, using semantic similarity via a feedback loop. We annotate the generated data for ground truth answers via a large scale user-study conducted on M-Turk, and with a high answerability rate of 97.26%, establish that LLMs are good at generating situational data. However, using the same LLM to answer the queries gives a low success rate of 46.2%; indicating that while LLMs are good at generating query data, they are poor at answering them. We use images from the VirtualHome simulator with the S-EQA queries establish an evaluation benchmark via Visual Question Answering (VQA). We report an improved accuracy of 15.31% while using queries framed from the generated object consensus for VQA over directly answering situational ones, indicating that such simplification is necessary for improved performance. To the best of our knowledge, this is the first work to introduce EQA in the context of situational queries that also uses a generative approach for query creation. We aim to foster research on improving the real-world usability of embodied agents in household environments through this work.

Central Answer Modeling for an Embodied Multi-LLM System

Multi-target embodied question answering

Explore until Confident: Efficient Exploration for Embodied Question Answering

EfficientEQA: An Efficient Approach for Open Vocabulary Embodied Question Answering

S-EQA: Tackling Situational Queries in Embodied Question Answering

Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering

GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering

AQA: Adaptive Question Answering in a Society of LLMs via Contextual Multi-Armed Bandit

Leveraging Large Language Models for Multiple Choice Question Answering

CoQ:AN Empirical Framework for Multi-hop Question Answering Empowered by Large Language Models

Knowledge-Based Embodied Question Answering

An Interactive Multi-modal Query Answering System with Retrieval-Augmented Large Language Models

Openeqa: Embodied question answering in the era of foundation models

Asking Before Action: Gather Information in Embodied Decision Making with Language Models

LONGAGENT: Achieving Question Answering for 128K-Token-long Documents Through Multi-Agent Collaboration

Multi-agent Embodied Question Answering in Interactive Environments

LLM Harmony: Multi-Agent Communication for Problem Solving

IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering

Depth and Video Segmentation Based Visual Attention for Embodied Question Answering