MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

Shaoxiong Ji,Tianlin Zhang,Luna Ansari,Jie Fu,Prayag Tiwari,Erik Cambria
DOI: https://doi.org/10.48550/arXiv.2110.15621
2021-10-29
Computation and Language
Abstract:Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.
What problem does this paper attempt to address?