Abstract:The evolution of language models, particularly the development of Large Language Models like ChatGPT, has opened new avenues for psychological assessment, potentially revolutionizing the rating scale methods that have been used for over a century. This study introduces a new Automated Assessment Paradigm (AAP), which aims to integrate natural language processing (NLP) techniques with traditional measurement methods. This integration enhances the accuracy and depth of mental health evaluations, while also addressing the acceptance and subjective experience of participants - areas that have not been extensively measured before. A pilot study was conducted with 32 participants, seven of whom were diagnosed with depression by licensed psychiatrists using the Clinical Interview Schedule-Revised (CIS-R). The participants completed the BDI-Fast Screen (BDI-FS) using a custom ChatGPT(GPTs) interface and the Chinese version of the PHQ-9 in a private setting. Following these assessments, participants also completed the Subjective Evaluation Scale. Spearman's correlation analysis showed a high correlation between the total scores of the PHQ-9 and the BSI-FS-GPTs. The agreement of diagnoses between the two measures, as measured by Cohen's kappa, was also significant. BSI-FS-GPTs diagnosis showed significantly higher agreement with the current diagnosis of depression. However, given the limited sample size of the pilot study, the AUC value of 1.00 and a sensitivity of 0.80 at a cutoff of 0.5, with zero false positive rate, likely overstate the classifier's performance. Bayesian factors suggest that participants may feel more comfortable expressing their true feelings and opinions through this method. For ongoing follow-up research, a total sample size of approximately 104 participants, including about 26 diagnosed individuals, may be required to ensure the analysis maintains a necessary power of 0.80 and an alpha level of 0.05. Nonetheless, these findings provide a promising foundation for the ongoing validation of the new AAP in larger-scale studies, aiming to confirm its validity and reliability.

Open Brain AI. Automatic Language Assessment

Open Brain AI and language assessment

Assessing Language Disorders using Artificial Intelligence: a Paradigm Shift

AI-powered Language Assessment Tools for Dementia

AI-assisted assessment and treatment of aphasia: a review

Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models

Testing the Limits of Language Models: A Conversational Framework for Medical AI Assessment

Beyond rating scales: With targeted evaluation, large language models are poised for psychological assessment

Language Analytics for Assessment of Mental Health Status and Functional Competency

Automatic Screening for Children with Speech Disorder using Automatic Speech Recognition: Opportunities and Challenges

A Systematic Review and Meta‐analysis of AI‐enabled Assessment in Language Learning: Design, Implementation, and Effectiveness

Can AI Relate: Testing Large Language Model Response for Mental Health Support

Robot-assisted language assessment: development and evaluation of feasibility and usability

Evaluating the Efficacy of AI-Based Interactive Assessments Using Large Language Models for Depression Screening

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

A framework for language technologies in behavioral research and clinical applications: Ethical challenges, implications, and solutions.

OpenClinicalAI: enabling AI to diagnose diseases in real-world clinical settings

An overview of diagnostics and therapeutics using large language models

A Literature Survey on SpeakSmart: AI – Enhanced language learning guide

An AI Generated Test of Pragmatic Competence and Connected Speech

Impact of an Artificial Intelligence in Language Learning - A survey