Abstract:Prompting has become a practical method for utilizing pre-trained language models (LMs). This approach offers several advantages. It allows an LM to adapt to new tasks with minimal training and parameter updates, thus achieving efficiency in both storage and computation. Additionally, prompting modifies only the LM's inputs and harnesses the generative capabilities of language models to address various downstream tasks in a unified manner. This significantly reduces the need for human labor in designing task-specific models. These advantages become even more evident as the number of tasks served by the LM scales up. Motivated by the strengths of prompting, we are the first to explore the potential of prompting speech LMs in the domain of speech processing. Recently, there has been a growing interest in converting speech into discrete units for language modeling. Our pioneer research demonstrates that these quantized speech units are highly versatile within our unified prompting framework. Not only can they serve as class labels, but they also contain rich phonetic information that can be re-synthesized back into speech signals for speech generation tasks. Specifically, we reformulate speech processing tasks into speech-to-unit generation tasks. As a result, we can seamlessly integrate tasks such as speech classification, sequence generation, and speech generation within a single, unified prompting framework. The experiment results show that the prompting method can achieve competitive performance compared to the strong fine-tuning method based on self-supervised learning models with a similar number of trainable parameters. The prompting method also shows promising results in the few-shot setting. Moreover, with the advanced speech LMs coming into the stage, the proposed prompting framework attains great potential.

Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Model

An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks

SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks

SpeechPrompt V2: Prompt Tuning for Speech Classification Tasks

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models

Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition

Exploring efficient-tuning methods in self-supervised speech models

Efficient Decoding Self-Attention for End-to-end Speech Synthesis

Are Soft Prompts Good Zero-shot Learners for Speech Recognition?

Dynamic Prompting: A Unified Framework for Prompt Tuning

Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion

Audio Prompt Adapter: Unleashing Music Editing Abilities for Text-to-Music with Lightweight Finetuning

PromptST: Abstract Prompt Learning for End-to-End Speech Translation

Wav2Prompt: End-to-End Speech Prompt Generation and Tuning For LLM in Zero and Few-shot Learning

Meta-Prompt: Boosting Whisper's Performance in Low-Resource Speech Recognition

ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks

Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter

On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model

Revisiting the Power of Prompt for Visual Tuning

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding