Google Translate Error Analysis for Mental Healthcare Information: Evaluating Accuracy, Comprehensibility, and Implications for Multilingual Healthcare Communication

Jaleh Delfani,Constantin Orasan,Hadeel Saadany,Ozlem Temizoz,Eleanor Taylor-Stilgoe,Diptesh Kanojia,Sabine Braun,Barbara Schouten
2024-02-06
Abstract:This study explores the use of Google Translate (GT) for translating mental healthcare (MHealth) information and evaluates its accuracy, comprehensibility, and implications for multilingual healthcare communication through analysing GT output in the MHealth domain from English to Persian, Arabic, Turkish, Romanian, and Spanish. Two datasets comprising MHealth information from the UK National Health Service website and information leaflets from The Royal College of Psychiatrists were used. Native speakers of the target languages manually assessed the GT translations, focusing on medical terminology accuracy, comprehensibility, and critical syntactic/semantic errors. GT output analysis revealed challenges in accurately translating medical terminology, particularly in Arabic, Romanian, and Persian. Fluency issues were prevalent across various languages, affecting comprehension, mainly in Arabic and Spanish. Critical errors arose in specific contexts, such as bullet-point formatting, specifically in Persian, Turkish, and Romanian. Although improvements are seen in longer-text translations, there remains a need to enhance accuracy in medical and mental health terminology and fluency, whilst also addressing formatting issues for a more seamless user experience. The findings highlight the need to use customised translation engines for Mhealth translation and the challenges when relying solely on machine-translated medical content, emphasising the crucial role of human reviewers in multilingual healthcare communication.
Computation and Language
What problem does this paper attempt to address?
The paper primarily explores the accuracy, comprehensibility, and impact of using Google Translate (hereinafter referred to as GT) in translating mental health care information, and its effect on multilingual medical communication. The study evaluates GT's performance by analyzing its translation results in the mental health field from English to Persian, Arabic, Turkish, Romanian, and Spanish. ### Research Background and Objectives - **Background**: Global mental health issues are becoming increasingly severe, especially among refugee and immigrant populations, who face language barriers that hinder their access to effective medical services. - **Challenges**: Using human translation in medical settings faces challenges such as long waiting times, high costs, and limited resources. - **Solution**: Machine Translation (MT) is a potential tool that can overcome language barriers and provide critical information to people with limited language proficiency. However, the translation quality of general MT tools like GT is inconsistent, particularly in situations where safety is paramount. ### Key Findings - **Translation Accuracy**: The study found that GT faces challenges in translating medical terms, especially in Arabic, Romanian, and Persian. - **Fluency Issues**: There are common fluency issues in translations across various languages, affecting the comprehensibility of the content, particularly more pronounced in Arabic and Spanish. - **Formatting Issues**: Formatting-related issues, such as bullet point formatting, were observed in specific contexts, especially in Persian, Turkish, and Romanian. - **Improvement Trend**: Despite these issues, the study observed an improvement trend in longer text translations. However, there is still a need to improve the accuracy and fluency of medical and mental health terms and address formatting issues to enhance user experience. ### Methodology Overview - **Datasets**: The study used two datasets, one from the National Health Service (NHS) website in the UK and another from the information booklets of the Royal College of Psychiatrists. - **Translation Process**: GT was used to translate these English texts into the target languages, and the translations were manually evaluated by native speakers of the target languages. - **Evaluation Methods**: The evaluation included aspects such as accuracy, comprehensibility, and critical grammatical/semantic errors, with a particular focus on the accuracy of medical terms. ### Conclusions and Recommendations - **Conclusions**: The study emphasizes the importance of using customized translation engines in mental health communication and highlights the challenges of relying solely on machine translation for medical content. - **Future Directions**: Future research should explore how to further improve the application of machine translation in the mental health field, including developing translation tools specifically for the medical domain and enhancing the role of human review to ensure the quality and safety of multilingual medical communication.