Perceptions and Detection of AI Use in Manuscript Preparation for Academic Journals

Nir Chemaya,Daniel Martin

2024-01-30

Abstract:The emergent abilities of Large Language Models (LLMs), which power tools like ChatGPT and Bard, have produced both excitement and worry about how AI will impact academic writing. In response to rising concerns about AI use, authors of academic publications may decide to voluntarily disclose any AI tools they use to revise their manuscripts, and journals and conferences could begin mandating disclosure and/or turn to using detection services, as many teachers have done with student writing in class settings. Given these looming possibilities, we investigate whether academics view it as necessary to report AI use in manuscript preparation and how detectors react to the use of AI in academic writing.

Computers and Society,Artificial Intelligence,General Economics

What problem does this paper attempt to address?

The paper primarily explores the use of artificial intelligence (especially large language models like ChatGPT) in the preparation of academic journal manuscripts and whether such usage should be disclosed.

The paper first points out that with the development of large language models, these tools have been widely used in academic writing, such as for text analysis and experimental design. However, there is little research on the application and potential impact of these tools in the preparation of academic manuscripts. In particular, there is currently no clear consensus on whether the use of such tools to revise or rewrite manuscripts should be disclosed.

To investigate these issues, the authors conducted two main tasks:
1. **Survey Design**: An online survey was conducted to collect opinions from the academic community on whether the use of ChatGPT for grammar correction and text rewriting needs to be disclosed. The results showed that most respondents believed that using ChatGPT to rewrite text should be disclosed, while using it to correct grammar does not need to be disclosed.
2. **Detection Design**: GPT-3.5 was used to revise the abstracts of articles from the journal "Management Science" over the past 10 years, and AI detection services were used to evaluate whether the revised content could be identified as AI-generated. The results showed that even when used only for grammar correction, the revised text was easily detected as AI-generated.

The main findings of the paper include:
- There is a difference in attitudes within the academic community towards using AI tools for grammar correction and text rewriting, but AI detection software may not be able to distinguish between these two uses.
- There is no significant difference in reporting preferences between using ChatGPT and using the help of a research assistant, but there is a significant difference in reporting preferences compared to other auxiliary tools (such as Grammarly, Word).
- There is a division within the academic community on whether the use of ChatGPT to rewrite text should be reported, and this division is related to ethical views, academic roles, and English background.

This paper provides preliminary insights into how to handle the application of AI tools in academic writing in the future and raises issues worth further discussion regarding disclosure requirements and detection methods.