Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correction

Yuanchao Li,Yuan Gong,Chao-Han Huck Yang,Peter Bell,Catherine Lai

2024-09-24

Abstract:Annotating and recognizing speech emotion using prompt engineering has recently emerged with the advancement of Large Language Models (LLMs), yet its efficacy and reliability remain questionable. In this paper, we conduct a systematic study on this topic, beginning with the proposal of novel prompts that incorporate emotion-specific knowledge from acoustics, linguistics, and psychology. Subsequently, we examine the effectiveness of LLM-based prompting on Automatic Speech Recognition (ASR) transcription, contrasting it with ground-truth transcription. Furthermore, we propose a Revise-Reason-Recognize prompting pipeline for robust LLM-based emotion recognition from spoken language with ASR errors. Additionally, experiments on context-aware learning, in-context learning, and instruction tuning are performed to examine the usefulness of LLM training schemes in this direction. Finally, we investigate the sensitivity of LLMs to minor prompt variations. Experimental results demonstrate the efficacy of the emotion-specific prompts, ASR error correction, and LLM training schemes for LLM-based emotion recognition. Our study aims to refine the use of LLMs in emotion recognition and related domains.

Audio and Speech Processing,Artificial Intelligence,Computation and Language,Multimedia,Sound

What problem does this paper attempt to address?

The paper attempts to address the effectiveness and reliability of using large language models (LLMs) for speech emotion recognition. Specifically, the paper focuses on the following aspects: 1. **Proposing new emotion-specific prompts**: Combining knowledge from acoustics, linguistics, and psychology to design new emotion-specific prompts to improve the performance of LLMs in emotion recognition tasks. 2. **Error correction in automatic speech recognition (ASR) transcriptions**: Investigating how to improve the accuracy of emotion recognition through prompt engineering in the presence of errors in ASR transcriptions. 3. **R3 prompt pipeline**: Proposing a prompt pipeline named REVISE-REASON-RECOGNIZE (R3) for robust emotion recognition from spoken text with ASR errors. 4. **Exploration of LLM training schemes**: Evaluating the effects of context-aware learning, in-context learning, and instruction tuning on emotion recognition tasks. 5. **Sensitivity of LLMs to minor prompt changes**: Studying the sensitivity of LLMs to minor changes in prompts to understand how these changes affect the performance of emotion recognition tasks. Through these studies, the paper aims to optimize the application of LLMs in the field of emotion recognition and explore their potential in related areas.

Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correction

AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models

Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversations

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMs

Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models

Speak From Heart: An Emotion-Guided LLM-Based Multimodal Method for Emotional Dialogue Generation

Large Language Models Understand and Can be Enhanced by Emotional Stimuli

Recent Advancement of Emotion Cognition in Large Language Models

InstructERC: Reforming Emotion Recognition in Conversation with Multi-task Retrieval-Augmented Large Language Models

Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models

EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models

Visual Prompting in LLMs for Enhancing Emotion Recognition

EmotionPrompt: Leveraging Psychology for Large Language Models Enhancement via Emotional Stimulus

Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench

From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs

Conditioning LLMs with Emotion in Neural Machine Translation

Human-LLM Collaborative Construction of a Cantonese Emotion Lexicon

ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting