Abstract:Background: Comprehensive session summaries enable effective continuity in mental health counseling, facilitating informed therapy planning. However, manual summarization presents a significant challenge, diverting experts' attention from the core counseling process. Leveraging advances in automatic summarization to streamline the summarization process addresses this issue because this enables mental health professionals to access concise summaries of lengthy therapy sessions, thereby increasing their efficiency. However, existing approaches often overlook the nuanced intricacies inherent in counseling interactions. Objective: This study evaluates the effectiveness of state-of-the-art large language models (LLMs) in selectively summarizing various components of therapy sessions through aspect-based summarization, aiming to benchmark their performance. Methods: We first created Mental Health Counseling-Component-Guided Dialogue Summaries, a benchmarking data set that consists of 191 counseling sessions with summaries focused on 3 distinct counseling components (also known as counseling aspects). Next, we assessed the capabilities of 11 state-of-the-art LLMs in addressing the task of counseling-component-guided summarization. The generated summaries were evaluated quantitatively using standard summarization metrics and verified qualitatively by mental health professionals. Results: Our findings demonstrated the superior performance of task-specific LLMs such as MentalLlama, Mistral, and MentalBART evaluated using standard quantitative metrics such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-1, ROUGE-2, ROUGE-L, and Bidirectional Encoder Representations from Transformers Score across all aspects of the counseling components. Furthermore, expert evaluation revealed that Mistral superseded both MentalLlama and MentalBART across 6 parameters: affective attitude, burden, ethicality, coherence, opportunity costs, and perceived effectiveness. However, these models exhibit a common weakness in terms of room for improvement in the opportunity costs and perceived effectiveness metrics. Conclusions: While LLMs fine-tuned specifically on mental health domain data display better performance based on automatic evaluation scores, expert assessments indicate that these models are not yet reliable for clinical application. Further refinement and validation are necessary before their implementation in practice.

Mental Health Counseling From Conversational Content With Transformer-Based Machine Learning

Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health

Towards Automated Psychotherapy via Language Modeling

Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions

Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models

Towards Automated Real-time Evaluation in Text-based Counseling

VCounselor: A Psychological Intervention Chat Agent Based on a Knowledge-Enhanced Large Language Model

AI-Enhanced Mental Health Diagnosis: Leveraging Transformers for Early Detection of Depression Tendency in Textual Data

ChatCounselor: A Large Language Models for Mental Health Support

Advancing Conversational Psychotherapy: Integrating Privacy, Dual-Memory, and Domain Expertise with Large Language Models

Evaluating Large Language Models for Anxiety and Depression Classification using Counseling and Psychotherapy Transcripts

CALLM: Enhancing Clinical Interview Analysis through Data Augmentation with Large Language Models

LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devices

A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Health

Conversational Topic Recommendation in Counseling and Psychotherapy with Decision Transformer and Large Language Models

Optimizing Psychological Counseling with Instruction-Tuned Large Language Models

Modeling Motivational Interviewing Strategies On An Online Peer-to-Peer Counseling Platform

Transfer Learning for Risk Classification of Social Media Posts: Model Evaluation Study

The Most Effective Interventions for Classification Model Development to Predict Chat Outcomes Based on the Conversation Content in Online Suicide Prevention Chats: Machine Learning Approach

Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: Benchmark Study

Finding Your Voice: The Linguistic Development of Mental Health Counselors