Abstract:Background: Health care and well-being are 2 main interconnected application areas of conversational agents (CAs). There is a significant increase in research, development, and commercial implementations in this area. In parallel to the increasing interest, new challenges in designing and evaluating CAs have emerged. Objective: This study aims to identify key design, development, and evaluation challenges of CAs in health care and well-being research. The focus is on the very recent projects with their emerging challenges. Methods: A review study was conducted with 17 invited studies, most of which were presented at the ACM (Association for Computing Machinery) CHI 2020 conference workshop on CAs for health and well-being. Eligibility criteria required the studies to involve a CA applied to a health or well-being project (ongoing or recently finished). The participating studies were asked to report on their projects' design and evaluation challenges. We used thematic analysis to review the studies. Results: The findings include a range of topics from primary care to caring for older adults to health coaching. We identified 4 major themes: (1) Domain Information and Integration, (2) User-System Interaction and Partnership, (3) Evaluation, and (4) Conversational Competence. Conclusions: CAs proved their worth during the pandemic as health screening tools, and are expected to stay to further support various health care domains, especially personal health care. Growth in investment in CAs also shows the value as a personal assistant. Our study shows that while some challenges are shared with other CA application areas, safety and privacy remain the major challenges in the health care and well-being domains. An increased level of collaboration across different institutions and entities may be a promising direction to address some of the major challenges that otherwise would be too complex to be addressed by the projects with their limited scope and budget.

Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024

AAAI-07 Workshop Reports.

Conversational Search -- A Report from Dagstuhl Seminar 19461

Report on the Search Futures Workshop at ECIR 2024

CUI@CHI 2024: Building Trust in CUIs-From Design to Deployment

SCAI-QReCC Shared Task on Conversational Question Answering

CUI@IUI: Theoretical and Methodological Challenges in Intelligent Conversational User Interface Interactions

Artificial intelligence empowered conversational agents: A systematic literature review and research agenda

Report on the Workshop on Simulations for Information Access (Sim4IA 2024) at SIGIR 2024

Findings of the First Workshop on Simulating Conversational Intelligence in Chat

Report on the 1st Workshop on Task Focused IR in the Era of Generative AI

A Conceptual Framework for Implicit Evaluation of Conversational Search Interfaces

Perspectives for Evaluating Conversational AI

Design and Evaluation Challenges of Conversational Agents in Health Care and Well-being: Selective Review Study

A Survey of Conversational Search

Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024

Advancing the Search Frontier with AI Agents

The 1st InterAI Workshop: Interactive AI for Human-centered Robotics

Broadening the perspective for sustainable artificial intelligence: sustainability criteria and indicators for Artificial Intelligence systems

Towards a model for spoken conversational search

Report on the 10th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS 2023) at ACM RecSys 2023