Abstract:Recent advanced methods in Natural Language Understanding for Task-oriented Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. In reality, token-level annotations (slot labels) are time-consuming and difficult to acquire. In this work, we study the Slot Induction (SI) task whose objective is to induce slot boundaries without explicit knowledge of token-level slot annotations. We propose leveraging Unsupervised Pre-trained Language Model (PLM) Probing and Contrastive Learning mechanism to exploit (1) unsupervised semantic knowledge extracted from PLM, and (2) additional sentence-level intent label signals available from TOD. Our approach is shown to be effective in SI task and capable of bridging the gaps with token-level supervised models on two NLU benchmark datasets. When generalized to emerging intents, our SI objectives also provide enhanced slot label representations, leading to improved performance on the Slot Filling tasks.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the Slot Induction (SI) problem in Task - oriented Dialogue (TOD) systems. Specifically, the author focuses on how to automatically identify slot boundaries without explicit word - level slot labels. #### Background and Challenges 1. **Scarcity of Annotated Data**: The current state - of - the - art Natural Language Understanding (NLU) methods (such as intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. However, obtaining word - level slot labels is both time - consuming and expensive. 2. **Generalization Ability in the Open - World Setting**: Traditional NLU models perform poorly in the open - world setting (i.e., when facing new intents and slots). 3. **Limitations of Existing Methods**: Most existing slot - filling methods rely on predefined sets of categories, and some methods also require additional word - level task knowledge, which may not be easily obtainable. #### Solutions To solve the above problems, the author proposes to use unsupervised pre - trained language model (PLM) probing and a multi - level contrastive learning mechanism to complete the slot induction task. Specifically, it includes the following aspects: 1. **Unsupervised PLM Probing (UPL)**: - Use pre - trained language models (such as BERT) to extract semantic information and generate semantic segments. - These segments can cover word - level slot labels without explicit word - level annotation. 2. **Multi - level Contrastive Learning**: - **Segment - level Contrastive Learning (SegCL)**: Optimize the quality of segments by comparing the similarity between the [CLS] representation and the segment representations generated by UPL. - **Sentence - level Contrastive Learning (SentCL)**: Use sentence - level intent labels as auxiliary signals to further improve the quality of segment representations. #### Main Contributions - Propose a method of using unsupervised PLM probing to generate semantic segments to induce phrases containing word - level slot labels. - Introduce a multi - level contrastive learning mechanism to improve the performance of slot induction through segment - level and sentence - level supervision signals. - Demonstrate the effectiveness of the proposed SI framework, especially when dealing with new intents, it can produce more refined slot representations, thereby improving the performance of the slot - filling task. ### Summary This paper proposes a novel slot induction method. By combining unsupervised pre - trained language model probing and multi - level contrastive learning, it can effectively identify slot boundaries in the absence of explicit word - level annotation and show good generalization ability in the open - world setting.

Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning

A Unified Approach to Nested and Non-Nested Slots for Spoken Language Understanding

SLIM: Explicit Slot-Intent Mapping with BERT for Joint Multi-Intent Detection and Slot Filling

A Novel Bi-Directional Interrelated Model For Joint Intent Detection And Slot Filling

Automatic Intent-Slot Induction for Dialogue Systems

A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding

Joint Training Model of Intent Detection and Slot Filling for Multi Granularity Implicit Guidance.

Dual Level Intent-Slot Interaction for Improved Multi-Intent Spoken Language Understanding

Effective Slot Filling via Weakly-Supervised Dual-Model Learning

Position-aware Interactive Attention Network for Multi-Intent Spoken Language Understanding

POS Scaling Attention Model for Joint Slot Filling and Intent Classification

Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning

Multi-domain Language Understanding of Task Oriented Dialogue Based on Intent Enhancement.

BERT for Joint Intent Classification and Slot Filling

Zero-Shot Slot and Intent Detection in Low-Resource Languages

Transforming Slot Schema Induction with Generative Dialogue State Inference

Intent-Slot Correlation Modeling for Joint Intent Prediction and Slot Filling

Semantically Guided Enhanced Fusion for Intent Detection and Slot Filling

SPM: A Split-Parsing Method for Joint Multi-Intent Detection and Slot Filling.

Novel Slot Detection with an Incremental Setting

Novel Slot Detection: A Benchmark for Discovering Unknown Slot Types in the Task-Oriented Dialogue System