"Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations

James F. Mullen Jr,Prasoon Goyal,Robinson Piramuthu,Michael Johnston,Dinesh Manocha,Reza Ghanadan

2024-04-13

Abstract:Home robots intend to make their users lives easier. Our work assists in this goal by enabling robots to inform their users of dangerous or unsanitary anomalies in their home. Some examples of these anomalies include the user leaving their milk out, forgetting to turn off the stove, or leaving poison accessible to children. To move towards enabling home robots with these abilities, we have created a new dataset, which we call SafetyDetect. The SafetyDetect dataset consists of 1000 anomalous home scenes, each of which contains unsafe or unsanitary situations for an agent to detect. Our approach utilizes large language models (LLMs) alongside both a graph representation of the scene and the relationships between the objects in the scene. Our key insight is that this connected scene graph and the object relationships it encodes enables the LLM to better reason about the scene -- especially as it relates to detecting dangerous or unsanitary situations. Our most promising approach utilizes GPT-4 and pursues a categorization technique where object relations from the scene graph are classified as normal, dangerous, unsanitary, or dangerous for children. This method is able to correctly identify over 90% of anomalous scenarios in the SafetyDetect Dataset. Additionally, we conduct real world experiments on a ClearPath TurtleBot where we generate a scene graph from visuals of the real world scene, and run our approach with no modification. This setup resulted in little performance loss. The SafetyDetect Dataset and code will be released to the public upon this papers publication.

Robotics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of how household robots can detect unsafe or unsanitary anomalies in the home. Specifically, the researchers aim to enable robots to recognize potential dangerous situations and alert users. For example, scenarios such as milk not being put back in the refrigerator, the stove not being turned off, or toxic items being accessed by children. To achieve this goal, the research team created a new dataset called SafetyDetect, which contains 1,000 household scenes with unsafe or unsanitary conditions. Additionally, the research proposes a method that utilizes large language models (LLMs) combined with scene graphs to better understand the relationships between objects in a scene, thereby effectively identifying these anomalies. Experimental results show that this method can correctly identify over 90% of anomalous scenes in the SafetyDetect dataset. Furthermore, the research was tested in real-world environments by generating scene graphs through a ClearPath TurtleBot and validating the method's effectiveness. The main contributions of the research include: the creation of the SafetyDetect dataset; the proposal of a scene graph-based LLM method that achieves high-precision anomaly detection; the demonstration of the importance of scene graphs in creating LLM prompts; and the showcasing of the method's transferability from simulated environments to the real world.