Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment

Huma Ameer,Seemab Latif,Iram Tariq Bhatti,Rabia Latif

2024-07-21

Abstract:The automated classification of stuttered speech has significant implications for timely assessments providing assistance to speech language pathologists. Despite notable advancements in the field, the cases in which multiple disfluencies occur in speech require attention. We have taken a progressive approach to fill this gap by classifying multi-stuttered speech more efficiently. The problem has been addressed by firstly curating a dataset of multi-stuttered disfluencies from open source dataset SEP-28k audio clips. Secondly, employing Whisper, a state-of-the-art speech recognition model has been leveraged by using its encoder and taking the problem as multi label classification. Thirdly, using a 6 encoder layer Whisper and experimenting with various layer freezing strategies, a computationally efficient configuration of the model was identified. The proposed configuration achieved micro, macro, and weighted F1-scores of 0.88, 0.85, and 0.87, correspondingly on an external test dataset i.e. Fluency-Bank. In addition, through layer freezing strategies, we were able to achieve the aforementioned results by fine-tuning a single encoder layer, consequently, reducing the model's trainable parameters from 20.27 million to 3.29 million. This research study unveils the contribution of the last encoder layer in the identification of disfluencies in stuttered speech. Consequently, it has led to a computationally efficient approach, 83.7% less parameters to train, making the proposed approach more adaptable for various dialects and languages.

Sound,Machine Learning,Audio and Speech Processing

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem of automated classification of multi - stuttered speech. Specifically, the authors attempt to improve existing methods in the following ways:

1. **Construct a high - quality multi - stuttered dataset**:
- Researchers integrated and enhanced existing open - source datasets (such as SEP - 28k and FluencyBank). By manually annotating speakers and strategically splicing audio segments, they simulated speech samples containing multiple stuttering types.

2. **Use the encoder of the Whisper model for efficient parameter reduction**:
- The authors utilized the encoder part of the Whisper model and redefined the problem as a multi - label classification task. Through this method, they successfully achieved effective classification of multi - stuttered speech.
- On an external test dataset, the model's micro - average, macro - average, and weighted F1 - scores reached 0.88, 0.85, and 0.87 respectively, significantly improving the ability to handle various types of disfluencies.

3. **Optimize the model to adapt to different dialects and languages**:
- By freezing specific layers (such as the feature extractor and encoder layers), researchers significantly reduced the number of trainable parameters from 20.27 million to 3.29 million, a reduction of 83.7%. This makes the model more computationally efficient while maintaining high performance and suitable for various dialects and languages.

### Formula representation

- **BCE Loss (Binary Cross - Entropy Loss)**:
\[
\text{BCE Loss}=-\frac{1}{N}\sum_{i = 1}^{N}[y_i\cdot\log(p_i)+(1 - y_i)\cdot\log(1 - p_i)]
\]
where:
- \( N \) is the number of instances in the dataset.
- \( y_i \) is the actual binary label (0 or 1) of the \( i \)-th sample.
- \( p_i \) is the predicted probability that the \( i \)-th sample belongs to the positive category.

Through these improvements, researchers not only improved the accuracy of multi - stuttered speech classification but also greatly reduced the complexity and training cost of the model, making it more suitable for practical application scenarios.