Abstract:Background: The working environment of a suicide prevention helpline requires high emotional and cognitive awareness from chat counselors. A shared opinion among counselors is that as a chat conversation becomes more difficult, it takes more effort and a longer amount of time to compose a response, which, in turn, can lead to writer's block. Objective: This study evaluates and then designs supportive technology to determine if a support system that provides inspiration can help counselors resolve writer's block when they encounter difficult situations in chats with help-seekers. Methods: A content-based recommender system with sentence embedding was used to search a chat corpus for similar chat situations. The system showed a counselor the most similar parts of former chat conversations so that the counselor would be able to use approaches previously taken by their colleagues as inspiration. In a within-subject experiment, counselors' chat replies when confronted with a difficult situation were analyzed to determine if experts could see a noticeable difference in chat replies that were obtained in 3 conditions: (1) with the help of the support system, (2) with written advice from a senior counselor, or (3) when receiving no help. In addition, the system's utility and usability were measured, and the validity of the algorithm was examined. Results: A total of 24 counselors used a prototype of the support system; the results showed that, by reading chat replies, experts were able to significantly predict if counselors had received help from the support system or from a senior counselor (P=.004). Counselors scored the information they received from a senior counselor (M=1.46, SD 1.91) as significantly more helpful than the information received from the support system or when no help was given at all (M=-0.21, SD 2.26). Finally, compared with randomly selected former chat conversations, counselors rated the ones identified by the content-based recommendation system as significantly more similar to their current chats (β=.30, P<.001). Conclusions: Support given to counselors influenced how they responded in difficult conversations. However, the higher utility scores given for the advice from senior counselors seem to indicate that specific actionable instructions are preferred. We expect that these findings will be beneficial for developing a system that can use similar chat situations to generate advice in a descriptive style, hence helping counselors through writer's block.

SolutionChat: Real-time Moderator Support for Chat-based Structured Discussion

Group Chat Ecology in Enterprise Instant Messaging: How Employees Collaborate Through Multi-User Chat Channels on Slack

Moderator Chatbot for Deliberative Discussion

StoryChat: Designing a Narrative-Based Viewer Participation Tool for Live Streaming Chatrooms

PsyChat: A Client-Centric Dialogue System for Mental Health Support

Content-Based Recommender Support System for Counselors in a Suicide Prevention Chat Helpline: Design and Evaluation Study

MeetScript: Designing Transcript-based Interactions to Support Active Participation in Group Video Meetings

Proactive Moderation of Online Discussions: Existing Practices and the Potential for Algorithmic Support

The Unsung Heroes of Facebook Groups Moderation: A Case Study of Moderation Practices and Tools

Augmenting Online Teamwork Dynamics: Investigating the Effects of a Generative AI Chatbot on Perceived Assistance and the Assessment of Team Members

A First Look at Developers' Live Chat on Gitter

Supervised ensemble sentiment-based framework to measure chatbot quality of services

The Effect of Moderation on Online Mental Health Conversations

OntoChat: a Framework for Conversational Ontology Engineering using Language Models

Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation

Challenges in Restructuring Community-based Moderation

Coordination and Collaboration: How do Volunteer Moderators Work as a Team in Live Streaming Communities?

Designing and Evaluating Multi-Chatbot Interface for Human-AI Communication: Preliminary Findings from a Persuasion Task

Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations

Can Language Model Moderators Improve the Health of Online Discourse?

A trial of a chat service for patients and their family members in an emergency department