Large Language Models as Zero-shot Dialogue State Tracker through Function Calling

Zekun Li,Zhiyu Zoey Chen,Mike Ross,Patrick Huber,Seungwhan Moon,Zhaojiang Lin,Xin Luna Dong,Adithya Sagar,Xifeng Yan,Paul A. Crook

2024-05-30

Abstract:Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, their effectiveness in task-oriented dialogues (TOD), which requires not only response generation but also effective dialogue state tracking (DST) within specific tasks and domains, remains less satisfying. In this work, we propose a novel approach FnCTOD for solving DST with LLMs through function calling. This method improves zero-shot DST, allowing adaptation to diverse domains without extensive data collection or model tuning. Our experimental results demonstrate that our approach achieves exceptional performance with both modestly sized open-source and also proprietary LLMs: with in-context prompting it enables various 7B or 13B parameter models to surpass the previous state-of-the-art (SOTA) achieved by ChatGPT, and improves ChatGPT's performance beating the SOTA by 5.6% average joint goal accuracy (JGA). Individual model results for GPT-3.5 and GPT-4 are boosted by 4.8% and 14%, respectively. We also show that by fine-tuning on a small collection of diverse task-oriented dialogues, we can equip modestly sized models, specifically a 13B parameter LLaMA2-Chat model, with function-calling capabilities and DST performance comparable to ChatGPT while maintaining their chat capabilities. We have made the code publicly available at <a class="link-external link-https" href="https://github.com/facebookresearch/FnCTOD" rel="external noopener nofollow">this https URL</a>

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - language models (LLMs) to achieve effective dialogue state tracking (DST) in task - oriented dialogue (TOD) without specific task - fine - tuning. Specifically, the author proposes a new method, FNCTOD, which improves zero - sample DST through function calls, enabling the model to adapt to different domains without extensive data collection or model adjustment. This method aims to overcome the problem of poor performance of existing methods when dealing with unseen domains and can add DST functionality to existing chat - tuned LLMs without sacrificing chat capabilities. Experimental results show that this method can not only make open - source medium - scale models (such as models with 7B or 13B parameters) surpass the previous best levels achieved by advanced proprietary models such as ChatGPT, but also significantly improve the DST performance of ChatGPT itself.

Large Language Models as Zero-shot Dialogue State Tracker through Function Calling

Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation

Zero-shot language extension for dialogue state tracking via pre-trained models and multi-auxiliary-tasks fine-tuning

GLMDST: Chinese Dialogue State Tracking Framework Driven by LLM

InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems

Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries

ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?

A Two-dimensional Zero-shot Dialogue State Tracking Evaluation Method using GPT-4

DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models

A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding

Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT

Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

Controllable Conversations: Planning-Based Dialogue Agent with Large Language Models

Improving Zero-Shot Cross-Lingual Dialogue State Tracking Via Contrastive Learning

S3-dst: Structured open-domain dialogue segmentation and state tracking in the era of llms

Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking

Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue

Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation

Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models