Abstract:Background: Identifying health problems in audio-recorded patient-nurse communication is important to improve outcomes in home healthcare patients who have complex conditions with increased risks of hospital utilization. Training machine learning classifiers for identifying problems requires resource-intensive human annotation. Objective: To generate synthetic patient-nurse communication and to automatically annotate for common health problems encountered in home healthcare settings using GPT-4. We also examined whether augmenting real-world patient-nurse communication with synthetic data can improve the performance of machine learning to identify health problems. Design: Secondary data analysis of patient-nurse verbal communication data in home healthcare settings. Methods: The data were collected from one of the largest home healthcare organizations in the United States. We used 23 audio recordings of patient-nurse communications from 15 patients. The audio recordings were transcribed verbatim and manually annotated for health problems (e.g., circulation, skin, pain) indicated in the Omaha System Classification scheme. Synthetic data of patient-nurse communication were generated using the in-context learning prompting method, enhanced by chain-of-thought prompting to improve the automatic annotation performance. Machine learning classifiers were applied to three training datasets: real-world communication, synthetic communication, and real-world communication augmented by synthetic communication. Results: Average F1 scores improved from 0.62 to 0.63 after training data were augmented with synthetic communication. The largest increase was observed using the XGBoost classifier where F1 scores improved from 0.61 to 0.64 (about 5% improvement). When trained solely on either real-world communication or synthetic communication, the classifiers showed comparable F1 scores of 0.62-0.61, respectively. Conclusion: Integrating synthetic data improves machine learning classifiers' ability to identify health problems in home healthcare, with performance comparable to training on real-world data alone, highlighting the potential of synthetic data in healthcare analytics. Clinical relevance: This study demonstrates the clinical relevance of leveraging synthetic patient-nurse communication data to enhance machine learning classifier performances to identify health problems in home healthcare settings, which will contribute to more accurate and efficient problem identification and detection of home healthcare patients with complex health conditions.

Constructing synthetic datasets with generative artificial intelligence to train large language models to classify acute renal failure from clinical notes

Uncovering Language Disparity of ChatGPT in Healthcare: Non-English Clinical Environment for Retinal Vascular Disease Classification (Preprint)

Does Synthetic Data Generation of LLMs Help Clinical Text Mining?

Assessing the Ability of a Large Language Model to Score Free-Text Medical Student Clinical Notes: Quantitative Study

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Data

Robust Privacy Amidst Innovation with Large Language Models Through a Critical Assessment of the Risks

Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes

Does synthetic data augmentation improve the performances of machine learning classifiers for identifying health problems in patient-nurse verbal communications in home healthcare settings?

A Study of Generative Large Language Model for Medical Research and Healthcare

Utilizing Large Language Models to Generate Synthetic Data to Increase the Performance of BERT-Based Neural Networks

On the limitations of large language models in clinical diagnosis

Performance of a large language model for identifying central line-associated bloodstream infections (CLABSI) using real clinical notes

The Case Records of ChatGPT: Language Models and Complex Clinical Questions

Investigating the clinical reasoning abilities of large language model GPT-4: an analysis of postoperative complications from renal surgeries

Natural Language Processing Methods to Identify Oncology Patients at High Risk for Acute Care with Clinical Notes

Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: Advancing Practical Applications

A large language model for electronic health records

Leveraging large language models for generating responses to patient messages-a subjective analysis

Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification