Abstract:In this paper we present a neurosymbolic architecture for coupling language-guided visual reasoning with robot manipulation. A non-expert human user can prompt the robot using unconstrained natural language, providing a referring expression (REF), a question (VQA), or a grasp action instruction. The system tackles all cases in a task-agnostic fashion through the utilization of a shared library of primitive skills. Each primitive handles an independent sub-task, such as reasoning about visual attributes, spatial relation comprehension, logic and enumeration, as well as arm control. A language parser maps the input query to an executable program composed of such primitives, depending on the context. While some primitives are purely symbolic operations (e.g. counting), others are trainable neural functions (e.g. visual grounding), therefore marrying the interpretability and systematic generalization benefits of discrete symbolic approaches with the scalability and representational power of deep networks. We generate a 3D vision-and-language synthetic dataset of tabletop scenes in a simulation environment to train our approach and perform extensive evaluations in both synthetic and real-world scenes. Results showcase the benefits of our approach in terms of accuracy, sample-efficiency, and robustness to the user's vocabulary, while being transferable to real-world scenes with few-shot visual fine-tuning. Finally, we integrate our method with a robot framework and demonstrate how it can serve as an interpretable solution for an interactive object-picking task, both in simulation and with a real robot. We make our datasets available in <a class="link-external link-https" href="https://gtziafas.github.io/neurosymbolic-manipulation" rel="external noopener nofollow">this https URL</a>.

Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Language

Prompt2Model: Generating Deployable Models from Natural Language Instructions

KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts

ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answering

Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic

Towards a fully declarative neuro-symbolic language

SNeL: A Structured Neuro-Symbolic Language for Entity-Based Multimodal Scene Understanding

The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning

Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approach

Conversational Neuro-Symbolic Commonsense Reasoning

DANA: Domain-Aware Neurosymbolic Agents for Consistency and Accuracy

Collaborative Development of NLP models

Neurosymbolic AI for Enhancing Instructability in Generative AI

CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding

Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design

Parallel Neurosymbolic Integration with Concordia

DomiKnowS: A Library for Integration of Symbolic Domain Knowledge in Deep Learning

A Neural Conversational Model

Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models

A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues