Abstract:In today's fast-paced world, the rates of stress and depression present a surge. Social media provide assistance for the early detection of mental health conditions. Existing methods mainly introduce feature extraction approaches and train shallow machine learning classifiers. Other researches use deep neural networks or transformers. Despite the fact that transformer-based models achieve noticeable improvements, they cannot often capture rich factual knowledge. Although there have been proposed a number of studies aiming to enhance the pretrained transformer-based models with extra information or additional modalities, no prior work has exploited these modifications for detecting stress and depression through social media. In addition, although the reliability of a machine learning model's confidence in its predictions is critical for high-risk applications, there is no prior work taken into consideration the model calibration. To resolve the above issues, we present the first study in the task of depression and stress detection in social media, which injects extra linguistic information in transformer-based models, namely BERT and MentalBERT. Specifically, the proposed approach employs a Multimodal Adaptation Gate for creating the combined embeddings, which are given as input to a BERT (or MentalBERT) model. For taking into account the model calibration, we apply label smoothing. We test our proposed approaches in three publicly available datasets and demonstrate that the integration of linguistic features into transformer-based models presents a surge in the performance. Also, the usage of label smoothing contributes to both the improvement of the model's performance and the calibration of the model. We finally perform a linguistic analysis of the posts and show differences in language between stressful and non-stressful texts, as well as depressive and non-depressive posts.

A Million Tweets Are Worth a Few Points: Tuning Transformers for Customer Service Tasks

Improving Sentiment Analysis over non-English Tweets using Multilingual Transformers and Automatic Translation for Data-Augmentation

The Dark Side of the Language: Pre-trained Transformers in the DarkNet

Understanding Transformers for Bot Detection in Twitter

Investigation on task effect analysis and optimization strategy of multimodal large model based on Transformers architecture for various languages

Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality

On the Limitations of Sociodemographic Adaptation with Transformers

Transformer-based Multi-task Learning for Disaster Tweet Categorisation

Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers

Muppet: Massive multi-task representations with pre-finetuning

CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts

Data Movement is All You Need: A Case Study on Optimizing Transformers.

Performance Evaluations of Large Language Models for Customer Service

Practical Text Classification With Large Pre-Trained Language Models

Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?

Database Tuning using Natural Language Processing

Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media

Analyzing COVID-19 Tweets with Transformer-based Language Models

Benchmarking Transformers-based models on French Spoken Language Understanding tasks

Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social Commonsense

Uncovering suggestions in MOOC discussion forums: a transformer-based approach