Abstract:Data science plays a critical role in clinical research, but it requires professionals with expertise in coding and medical data analysis. Large language models (LLMs) have shown great potential in supporting medical tasks and performing well in general coding tests. However, these tests do not assess LLMs' ability to handle data science tasks in medicine, nor do they explore their practical utility in clinical research. To address this, we developed a dataset consisting of 293 real-world data science coding tasks, based on 39 published clinical studies, covering 128 tasks in Python and 165 tasks in R. This dataset simulates realistic clinical research scenarios using patient data. Our findings reveal that cutting-edge LLMs struggle to generate perfect solutions, frequently failing to follow input instructions, understand target data, and adhere to standard analysis practices. Consequently, LLMs are not yet ready to fully automate data science tasks. We benchmarked advanced adaptation methods and found two to be particularly effective: chain-of-thought prompting, which provides a step-by-step plan for data analysis, which led to a 60% improvement in code accuracy; and self-reflection, enabling LLMs to iteratively refine their code, yielding a 38% accuracy improvement. Building on these insights, we developed a platform that integrates LLMs into the data science workflow for medical professionals. In a user study with five medical doctors, we found that while LLMs cannot fully automate coding tasks, they significantly streamline the programming process. We found that 80% of their submitted code solutions were incorporated from LLM-generated code, with up to 96% reuse in some cases. Our analysis highlights the potential of LLMs, when integrated into expert workflows, to enhance data science efficiency in clinical research.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is whether large language models (LLMs) can replace data scientists to perform complex data science tasks in clinical research. Specifically, the paper focuses on the following aspects: 1. **Evaluating the capabilities of LLMs**: The paper developed a dataset containing 293 real - world data science coding tasks. These tasks are based on 39 published clinical studies, covering 128 tasks in Python and 165 tasks in R. Through this dataset, the paper evaluated the performance of the current state - of - the - art LLMs in handling data science tasks in clinical research. 2. **Exploring adaptation methods**: In order to improve the performance of LLMs in data science tasks in clinical research, the paper tested several adaptation methods, including chain - of - thought prompting, few - shot prompting, automatic prompting, self - reflection and retrieval - augmented generation. These methods aim to enhance the code generation and debugging capabilities of LLMs through different strategies. 3. **Constructing a collaborative platform**: Based on the research results of the above - mentioned evaluation and adaptation methods, the paper developed a platform that integrates LLMs into the data science workflow of medical professionals. This platform aims to simplify the programming process and improve the productivity of clinical researchers through human - machine collaboration. ### Main findings - **Performance of LLMs**: Although current LLMs perform well on some simple tasks, they still have significant difficulties in handling complex clinical data science tasks. For example, for more difficult tasks, the success rate of LLMs' first attempts is low. Especially in Hard tasks, the Pass@1 score is only 0.05 to 0.15. - **Effects of adaptation methods**: - **Chain - of - thought prompting (CoT)**: Providing step - by - step guidance significantly improves the accuracy of code generation, especially in Medium and Hard tasks, with the Pass@1 score increasing by 60%. - **Self - reflection**: By correcting errors through multiple iterations, it significantly improves the performance of LLMs. After five rounds of self - reflection, the Pass@1 score of Easy tasks increases by about 60 - 80%, that of Medium tasks increases by 40 - 50%, and that of Hard tasks increases by 20 - 25%. - **Practical application of the platform**: In a user study, 5 medical doctors used this platform for programming tasks. The results show that although LLMs cannot fully automate programming tasks, they significantly simplify the programming process. 80% of the submitted code solutions come from the code generated by LLMs, and in some cases, the reuse rate even reaches 96%. ### Conclusion The paper shows that although current LLMs are not sufficient to completely replace the role of data scientists in clinical research, through appropriate adaptation methods and human - machine collaboration, LLMs can significantly improve the efficiency of data science tasks and provide strong support for clinical research.

Can Large Language Models Replace Data Scientists in Clinical Research?

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Demystifying Large Language Models for Medicine: A Primer

Evaluating large language models in medical applications: a survey

Large language models encode clinical knowledge

Large language models in medical and healthcare fields: applications, advances, and challenges

Transforming clinical trials: the emerging roles of large language models

Is larger always better? Evaluating and prompting large language models for non-generative medical tasks

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Large Language Models in the Medical Field: Principles and Applications

Large Language Models as Agents in the Clinic

Large language models in medicine: the potentials and pitfalls

Large Language Model Prompting Techniques for Advancement in Clinical Medicine

Large Language Models Illuminate a Progressive Pathway to Artificial Intelligent Healthcare Assistant

Impact of Large Language Models on Medical Education and Teaching Adaptations

ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

Adapted large language models can outperform medical experts in clinical text summarization

A Survey of Clinicians’ Views of the Utility of Large Language Models