Abstract:The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language-based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement a novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use language-based definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules as needed. We benchmark AmadeusGPT and show we can produce state-of-the-art performance on the MABE 2022 behavior challenge tasks. Note, an end-user would not need to write any code to achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: <a class="link-external link-https" href="https://github.com/AdaptiveMotorControlLab/AmadeusGPT" rel="external noopener nofollow">this https URL</a>.

HUB-DT: A tool for unsupervised behavioural discovery and analysis

DeepBhvTracking: A Novel Behavior Tracking Method for Laboratory Animals Based on Deep Learning

Clustering for Automated Exploratory Pattern Discovery in Animal Behavioral Data

BehaviorDEPOT is a simple, flexible tool for automated behavioral detection based on markerless pose tracking

DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour

SUBTLE: An Unsupervised Platform with Temporal Link Embedding that Maps Animal Behavior

DeepRod: A human-in-the-loop system for automatic rodent behavior analysis

Statistical signature of subtle behavioural changes in large-scale behavioural assays

Automated Behavioral Analysis Using Instance Segmentation

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

From raw footage to behaviour: an automated pipeline for activity monitoring and behaviour identification in the wild

YOLO-Behaviour: A simple, flexible framework to automatically quantify animal behaviours from videos

AmadeusGPT: a natural language interface for interactive animal behavioral analysis

Digitally-Enhanced Dog Behavioral Testing: Getting Help from the Machine

DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels

Open-Source Tools for Behavioral Video Analysis: Setup, Methods, and Development

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

Mapping the landscape of social behavior

LarvaTagger: manual and automatic tagging of Drosophila larval behaviour

The choice-wide behavioral association study: data-driven identification of interpretable behavioral components

Fine decomposition of rodent behavior via unsupervised segmentation and clustering of inertial signals