Abstract:Background: Sentiment analysis is a significant yet difficult task in natural language processing. The linguistic peculiarities of Cantonese, including its high similarity with Standard Chinese, its grammatical and lexical uniqueness, and its colloquialism and multilingualism, make it different from other languages and pose additional challenges to sentiment analysis. Recent advances in models such as ChatGPT offer potential viable solutions. Objective: This study investigated the efficacy of GPT-3.5 and GPT-4 in Cantonese sentiment analysis in the context of web-based counseling and compared their performance with other mainstream methods, including lexicon-based methods and machine learning approaches. Methods: We analyzed transcripts from a web-based, text-based counseling service in Hong Kong, including a total of 131 individual counseling sessions and 6169 messages between counselors and help-seekers. First, a codebook was developed for human annotation. A simple prompt ("Is the sentiment of this Cantonese text positive, neutral, or negative? Respond with the sentiment label only.") was then given to GPT-3.5 and GPT-4 to label each message's sentiment. GPT-3.5 and GPT-4's performance was compared with a lexicon-based method and 3 state-of-the-art models, including linear regression, support vector machines, and long short-term memory neural networks. Results: Our findings revealed ChatGPT's remarkable accuracy in sentiment classification, with GPT-3.5 and GPT-4, respectively, achieving 92.1% (5682/6169) and 95.3% (5880/6169) accuracy in identifying positive, neutral, and negative sentiment, thereby outperforming the traditional lexicon-based method, which had an accuracy of 37.2% (2295/6169), and the 3 machine learning models, which had accuracies ranging from 66% (4072/6169) to 70.9% (4374/6169). Conclusions: Among many text analysis techniques, ChatGPT demonstrates superior accuracy and emerges as a promising tool for Cantonese sentiment analysis. This study also highlights ChatGPT's applicability in real-world scenarios, such as monitoring the quality of text-based counseling services and detecting message-level sentiments in vivo. The insights derived from this study pave the way for further exploration into the capabilities of ChatGPT in the context of underresourced languages and specialized domains like psychotherapy and natural language processing.

Public sentiment analysis and topic modeling regarding ChatGPT in mental health on Reddit: Negative sentiments increase over time

The public attitude towards ChatGPT on reddit: A study based on unsupervised learning from sentiment analysis and topic modeling

Public Opinions on ChatGPT: An Analysis of Reddit Discussions by Using Sentiment Analysis, Topic Modeling, and SWOT Analysis

Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and Occupations

Exploring Public Response to ChatGPT With Sentiment Analysis and Knowledge Mapping

"The ChatGPT bot is causing panic now – but it'll soon be as mundane a tool as Excel": analysing topics, sentiment and emotions relating to ChatGPT on Twitter

From Optimism to Concern: Unveiling Sentiments and Perceptions Surrounding ChatGPT on Twitter

Powerful tool or too powerful? Early public discourse about ChatGPT across 4 million tweets

Higher Education Faculty Perceptions of ChatGPT and the Influencing Factors: A Sentiment Analysis of X

Can ChatGPT provide a better support: a comparative analysis of ChatGPT and dataset responses in mental health dialogues

Last Week with ChatGPT: A Weibo Study on Social Perspective Regarding ChatGPT for Education and Beyond

"I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data

"ChatGPT, a Friend or Foe for Education?" Analyzing the User's Perspectives on the Latest AI Chatbot Via Reddit

Societal Impacts of Chatbot and Mitigation Strategies for Negative Impacts: A Large-Scale Qualitative Survey of ChatGPT Users

ChatGPT and mental health: Friends or foes?

ChatGPT as a Complementary Mental Health Resource: A Boon or a Bane

Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study

ChatGPT in education: A discourse analysis of worries and concerns on social media

Factors Predicting Intentions of Adoption and Continued Use of Artificial Intelligence Chatbots for Mental Health: Examining the Role of UTAUT Model, Stigma, Privacy Concerns, and Artificial Intelligence Hesitancy.

Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study

The high dimensional psychological profile and cultural bias of ChatGPT