Abstract:Objective: Despite speech being the primary communication medium, it carries valuable information about a speaker's health, emotions, and identity. Various conditions can affect the vocal organs, leading to speech difficulties. Extensive research has been conducted by voice clinicians and academia in speech analysis. Previous approaches primarily focused on one particular task, such as differentiating between normal and dysphonic speech, classifying different voice disorders, or estimating the severity of voice disorders. Methods and procedures: This study proposes an approach that combines transfer learning and multitask learning (MTL) to simultaneously perform dysphonia classification and severity estimation. Both tasks use a shared representation; network is learned from these shared features. We employed five computer vision models and changed their architecture to support multitask learning. Additionally, we conducted binary 'healthy vs. dysphonia' and multiclass 'healthy vs. organic and functional dysphonia' classification using multitask learning, with the speaker's sex as an auxiliary task. Results: The proposed method achieved improved performance across all classification metrics compared to single-task learning (STL), which only performs classification or severity estimation. Specifically, the model achieved F1 scores of 93% and 90% in MTL and STL, respectively. Moreover, we observed considerable improvements in both classification tasks by evaluating beta values associated with the weight assigned to the sex-predicting auxiliary task. MTL achieved an accuracy of 77% compared to the STL score of 73.2%. However, the performance of severity estimation in MTL was comparable to STL. Conclusion: Our goal is to improve how voice pathologists and clinicians understand patients' conditions, make it easier to track their progress, and enhance the monitoring of vocal quality and treatment procedures. Clinical and Translational Impact Statement: By integrating both classification and severity estimation of dysphonia using multitask learning, we aim to enable clinicians to gain a better understanding of the patient's situation, effectively monitor their progress and voice quality.

Multimodal Transfer Learning for Oral Presentation Assessment

Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides

Estimating Presentation Competence using Multimodal Nonverbal Behavioral Cues

AI-Assisted Enhancement of Student Presentation Skills: Challenges and Opportunities

A Survey of Transformer-Based Multimodal Pre-Trained Modals.

Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching

A Visual Analytical Approach for Transfer Learning in Classification.

Multimodal Pre-Train then Transfer Learning Approach for Speaker Recognition

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

Multilingual Multimodal Learning with Machine Translated Text

Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback

Multimodal Representation Learning by Alternating Unimodal Adaptation

Multi-Modal Knowledge Transfer for Target Speaker Lipreading with Improved Audio-Visual Pretraining and Cross-Lingual Fine-Tuning

Multitask and Transfer Learning Approach for Joint Classification and Severity Estimation of Dysphonia

Improving Multimodal Speech Recognition by Data Augmentation and Speech Representations

Multimodal Inductive Transfer Learning for Detection of Alzheimer's Dementia and its Severity

Design of oral English teaching model based on multi-modal perception of the Internet of Things and improved conventional neural networks

Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach