Large language models and academic writing: Five tiers of engagement

Martin Bekker

DOI: https://doi.org/10.17159/sajs.2024/17147

2024-02-17

South African Journal of Science

Abstract:Against a backdrop of the rapidly expanding use of large language models (LLMs) across diverse domains, this discussion breaks LLM usage into tiers of use, offering practical guidance to cautiously embrace the benefits of this significant new tool.

multidisciplinary sciences

What problem does this paper attempt to address?

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Jeff Z. Pan,Simon Razniewski,Jan-Christoph Kalo,Sneha Singhania,Jiaoyan Chen,Stefan Dietze,Hajira Jabeen,Janna Omeliyanenko,Wen Zhang,Matteo Lissandrini,Russa Biswas,Gerard de Melo,Angela Bonifati,Edlira Vakaj,Mauro Dragoni,Damien Graux

DOI: https://doi.org/10.48550/arxiv.2308.06374

2023-01-01

Abstract:Large Language Models (LLMs) have taken Knowledge Representation -- and the world -- by storm. This inflection point marks a shift from explicit knowledge representation to a renewed focus on the hybrid representation of both explicit knowledge and parametric knowledge. In this position paper, we will discuss some of the common debate points within the community on LLMs (parametric knowledge) and Knowledge Graphs (explicit knowledge) and speculate on opportunities and visions that the renewed focus brings, as well as related research topics and challenges.
The rise of large language models: challenges for Critical Discourse Studies

Mathew Gillings,Tobias Kohn,Gerlinde Mautner

DOI: https://doi.org/10.1080/17405904.2024.2373733

IF: 1.642

2024-07-13

Critical Discourse Studies

Abstract:Large language models (LLMs) such as ChatGPT are opening up new areas of research and teaching potential across a variety of domains. The purpose of the present conceptual paper is to map this new terrain from the point of view of Critical Discourse Studies (CDS). We demonstrate that the usage of LLMs raises concerns that definitely fall within the remit of CDS; among them, power and inequality. After an initial explanation of LLMs, we focus on three key areas of reflection. The first is a general stock-taking, where we look at CDS' theoretical underpinnings and what they imply for working with AI-generated language data. The second issue is authorship, where we assess the traceability of linguistic metadata and the ethically sensitive situation with regard to ownership of texts. The third area is linguistic homogenisation, where we examine how LLM usage privileges the mainstream. Afterwards, we explore ways in which LLMs could be used in research, and we discuss the implications of exploring their use in the classroom through a CDS lens. We close the paper with some observations on likely future developments in AI and how CDS can contribute with its distinctive theoretical, methodological and critical apparatus.

communication
Eight Things to Know about Large Language Models

Samuel R. Bowman

2023-04-03

Abstract:The widespread public deployment of large language models (LLMs) in recent months has prompted a wave of new attention and engagement from advocates, policymakers, and scholars from many fields. This attention is a timely response to the many urgent questions that this technology raises, but it can sometimes miss important considerations. This paper surveys the evidence for eight potentially surprising such points: 1. LLMs predictably get more capable with increasing investment, even without targeted innovation. 2. Many important LLM behaviors emerge unpredictably as a byproduct of increasing investment. 3. LLMs often appear to learn and use representations of the outside world. 4. There are no reliable techniques for steering the behavior of LLMs. 5. Experts are not yet able to interpret the inner workings of LLMs. 6. Human performance on a task isn't an upper bound on LLM performance. 7. LLMs need not express the values of their creators nor the values encoded in web text. 8. Brief interactions with LLMs are often misleading.

Computation and Language,Artificial Intelligence
An Interdisciplinary Outlook on Large Language Models for Scientific Research

James Boyko,Joseph Cohen,Nathan Fox,Maria Han Veiga,Jennifer I-Hsiu Li,Jing Liu,Bernardo Modenesi,Andreas H. Rauch,Kenneth N. Reid,Soumi Tribedi,Anastasia Visheratina,Xin Xie

DOI: https://doi.org/10.48550/arXiv.2311.04929

2023-11-03

Computation and Language

Abstract:In this paper, we describe the capabilities and constraints of Large Language Models (LLMs) within disparate academic disciplines, aiming to delineate their strengths and limitations with precision. We examine how LLMs augment scientific inquiry, offering concrete examples such as accelerating literature review by summarizing vast numbers of publications, enhancing code development through automated syntax correction, and refining the scientific writing process. Simultaneously, we articulate the challenges LLMs face, including their reliance on extensive and sometimes biased datasets, and the potential ethical dilemmas stemming from their use. Our critical discussion extends to the varying impacts of LLMs across fields, from the natural sciences, where they help model complex biological sequences, to the social sciences, where they can parse large-scale qualitative data. We conclude by offering a nuanced perspective on how LLMs can be both a boon and a boundary to scientific progress.
Apprentices to Research Assistants: Advancing Research with Large Language Models

M. Namvarpour,A. Razi

2024-04-09

Abstract:Large Language Models (LLMs) have emerged as powerful tools in various research domains. This article examines their potential through a literature review and firsthand experimentation. While LLMs offer benefits like cost-effectiveness and efficiency, challenges such as prompt tuning, biases, and subjectivity must be addressed. The study presents insights from experiments utilizing LLMs for qualitative analysis, highlighting successes and limitations. Additionally, it discusses strategies for mitigating challenges, such as prompt optimization techniques and leveraging human expertise. This study aligns with the 'LLMs as Research Tools' workshop's focus on integrating LLMs into HCI data work critically and ethically. By addressing both opportunities and challenges, our work contributes to the ongoing dialogue on their responsible application in research.

Human-Computer Interaction,Artificial Intelligence,Machine Learning
Large Language Models: A Survey

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, Jianfeng Gao

2024-02-09

Abstract:Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Artificial Intelligence,Computation and Language
Large language models are changing landscape of academic publications. A positive transformation?

Martin Májovský,Martin Černý,David Netuka

Abstract:The advent of large language models (LLMs) based on neural networks marks a significant shift in academic writing, particularly in medical sciences. These models, including OpenAI's GPT-4, Google's Bard, and Anthropic's Claude, enable more efficient text processing through transformer architecture and attention mechanisms. LLMs can generate coherent texts that are indistinguishable from human-written content. In medicine, they can contribute to the automation of literature reviews, data extraction, and hypothesis formulation. However, ethical concerns arise regarding the quality and integrity of scientific publications and the risk of generating misleading content. This article provides an overview of how LLMs are changing medical writing, the ethical dilemmas they bring, and the possibilities for detecting AI-generated text. It concludes with a focus on the potential future of LLMs in academic publishing and their impact on the medical community.
Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding

Bram M.A. van Dijk,Tom Kouwenhoven,Marco R. Spruit,Max J. van Duijn

2023-10-31

Abstract:Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. LLMs are appearing rapidly, and debates on LLM capacities have taken off, but reflection is lagging behind. Thus, in this position paper, we first zoom in on the debate and critically assess three points recurring in critiques of LLM capacities: i) that LLMs only parrot statistical patterns in the training data; ii) that LLMs master formal but not functional language competence; and iii) that language learning in LLMs cannot inform human language learning. Drawing on empirical and theoretical arguments, we show that these points need more nuance. Second, we outline a pragmatic perspective on the issue of `real' understanding and intentionality in LLMs. Understanding and intentionality pertain to unobservable mental states we attribute to other humans because they have pragmatic value: they allow us to abstract away from complex underlying mechanics and predict behaviour effectively. We reflect on the circumstances under which it would make sense for humans to similarly attribute mental states to LLMs, thereby outlining a pragmatic philosophical context for LLMs as an increasingly prominent technology in society.

Computation and Language,Artificial Intelligence
Exploring the landscape of large language models: Foundations, techniques, and challenges

Milad Moradi,Ke Yan,David Colwell,Matthias Samwald,Rhona Asgari

2024-04-18

Abstract:In this review paper, we delve into the realm of Large Language Models (LLMs), covering their foundational principles, diverse applications, and nuanced training processes. The article sheds light on the mechanics of in-context learning and a spectrum of fine-tuning approaches, with a special focus on methods that optimize efficiency in parameter usage. Additionally, it explores how LLMs can be more closely aligned with human preferences through innovative reinforcement learning frameworks and other novel methods that incorporate human feedback. The article also examines the emerging technique of retrieval augmented generation, integrating external knowledge into LLMs. The ethical dimensions of LLM deployment are discussed, underscoring the need for mindful and responsible application. Concluding with a perspective on future research trajectories, this review offers a succinct yet comprehensive overview of the current state and emerging trends in the evolving landscape of LLMs, serving as an insightful guide for both researchers and practitioners in artificial intelligence.

Artificial Intelligence
Practical Applications of Large Language Models for Health Care Professionals and Scientists

Florian Reis,Christian Lenz,Manfred Gossen,Hans-Dieter Volk,Norman Michael Drzeniek

DOI: https://doi.org/10.2196/58478

2024-09-05

Abstract:With the popularization of large language models (LLMs), strategies for their effective and safe usage in health care and research have become increasingly pertinent. Despite the growing interest and eagerness among health care professionals and scientists to exploit the potential of LLMs, initial attempts may yield suboptimal results due to a lack of user experience, thus complicating the integration of artificial intelligence (AI) tools into workplace routine. Focusing on scientists and health care professionals with limited LLM experience, this viewpoint article highlights and discusses 6 easy-to-implement use cases of practical relevance. These encompass customizing translations, refining text and extracting information, generating comprehensive overviews and specialized insights, compiling ideas into cohesive narratives, crafting personalized educational materials, and facilitating intellectual sparring. Additionally, we discuss general prompting strategies and precautions for the implementation of AI tools in biomedicine. Despite various hurdles and challenges, the integration of LLMs into daily routines of physicians and researchers promises heightened workplace productivity and efficiency.
Challenges and Applications of Large Language Models

Jean Kaddour,Joshua Harris,Maximilian Mozes,Herbie Bradley,Roberta Raileanu,Robert McHardy

2023-07-20

Abstract:Large Language Models (LLMs) went from non-existent to ubiquitous in the machine learning discourse within a few years. Due to the fast pace of the field, it is difficult to identify the remaining challenges and already fruitful application areas. In this paper, we aim to establish a systematic set of open problems and application successes so that ML researchers can comprehend the field's current state more quickly and become productive.

Computation and Language,Artificial Intelligence,Machine Learning
A Comprehensive Overview of Large Language Models

Asad Ullah Khan,Saeed Anwar,Muhammad Saqib,Shi Qiu,Humza Naveed,Nick Barnes,Muhammad Usman,A. Mian

DOI: https://doi.org/10.48550/arXiv.2307.06435

2023-07-12

Abstract:Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the existing literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to not only provide a systematic survey but also a quick comprehensive reference for the researchers and practitioners to draw insights from extensive informative summaries of the existing works to advance the LLM research.

Computer Science,Linguistics
Large Language Models for Mathematicians

Simon Frieder,Julius Berner,Philipp Petersen,Thomas Lukasiewicz

2024-04-02

Abstract:Large language models (LLMs) such as ChatGPT have received immense interest for their general-purpose language understanding and, in particular, their ability to generate high-quality text or computer code. For many professions, LLMs represent an invaluable tool that can speed up and improve the quality of work. In this note, we discuss to what extent they can aid professional mathematicians. We first provide a mathematical description of the transformer model used in all modern language models. Based on recent studies, we then outline best practices and potential issues and report on the mathematical abilities of language models. Finally, we shed light on the potential of LLMs to change how mathematicians work.

Computation and Language,Artificial Intelligence,Machine Learning,History and Overview
Large language model programs

Imanol Schlag, Sainbayar Sukhbaatar, Asli Celikyilmaz, Wen-tau Yih, Jason Weston, Jürgen Schmidhuber, Xian Li

2023-05-09

Abstract:In recent years, large pre-trained language models (LLMs) have demonstrated the ability to follow instructions and perform novel tasks from a few examples. The possibility to parameterise an LLM through such in-context examples widens their capability at a much lower cost than finetuning. We extend this line of reasoning and present a method which further expands the capabilities of an LLM by embedding it within an algorithm or program. To demonstrate the benefits of this approach, we present an illustrative example of evidence-supported question-answering. We obtain a 6.4\% improvement over the chain of thought baseline through a more algorithmic approach without any finetuning. Furthermore, we highlight recent work from this perspective and discuss the advantages and disadvantages in comparison to the standard approaches.
Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

Gašper Beguš,Maksymilian Dąbkowski,Ryan Rhodes

2023-08-22

Abstract:The performance of large language models (LLMs) has recently improved to the point where the models can perform well on many language tasks. We show here that for the first time, the models can also generate coherent and valid formal analyses of linguistic data and illustrate the vast potential of large language models for analyses of their metalinguistic abilities. LLMs are primarily trained on language data in the form of text; analyzing and evaluating their metalinguistic abilities improves our understanding of their general capabilities and sheds new light on theoretical models in linguistics. In this paper, we probe into GPT-4's metalinguistic capabilities by focusing on three subfields of formal linguistics: syntax, phonology, and semantics. We outline a research program for metalinguistic analyses of large language models, propose experimental designs, provide general guidelines, discuss limitations, and offer future directions for this line of research. This line of inquiry also exemplifies behavioral interpretability of deep learning, where models' representations are accessed by explicit prompting rather than internal representations.

Computation and Language,Artificial Intelligence
Ten simple rules to leverage large language models for getting grants

Elizabeth Seckel,Brandi Y. Stephens,Fatima Rodriguez

DOI: https://doi.org/10.1371/journal.pcbi.1011863

2024-03-02

PLoS Computational Biology

Abstract:The recent leap in performance of large language models (LLMs), a subclass of artificial intelligence (AI) algorithms that includes OpenAI's ChatGPT, Google Bard, and Microsoft's Copilot (formerly Bing Chat), ushered a revolution in artificial text generation. These systems, trained on billions of documents, are sophisticated enough to fool human users into thinking they are conversing with other humans [1,2]. In academia, LLM-driven chatbots have become popular tools to help draft and revise scientific text [3,4], with some going as far as including them as coauthors [5]. Enthusiasts highlight the ability of these systems to summarize entire articles, simplify jargon-laden paragraphs, and improve the clarity and conciseness of drafts, particularly for non-native English writers [6–8]. On the other hand, others have advocated for strict boundaries and restrictions [5,9,10], citing ethical and privacy concerns as well as the tendency of these tools to "hallucinate"—or confabulate and fabricate—facts and references [11]. LLMs are fed enormous amounts of information and use statistics to predict the next word in a sentence [12]. By doing so, they generate grammatically and semantically correct text in response to prompts but are unable to estimate the uncertainty or truth of their predictions—resulting in hallucinations. This also means that the generated text can be borrowed verbatim from existing sources, which has led to a growing number of copyright lawsuits [13,14]. As writers of scientific proposals, we believe that writing proposals is a very personal exercise where the final product is best when imbued with the ideas, style, and personality of the writer. The iterative process of drafting and refining also helps develop scientific writing skills [15], which are essential for a successful long-term career in academia. We also believe, however, that scientists can benefit immensely from including AI in this process, as assistants or makeshift reviewers, in particular as the algorithms that power these systems become better and more widely available. This article aims to strike a delicate balance—an enthusiastic yet cautionary tale outlining 10 best practice tips (summarized in Fig 1) for using LLMs during your grant writing journey.

biochemical research methods,mathematical & computational biology
On the use of Large Language Models in Model-Driven Engineering

Juri Di Rocco,Davide Di Ruscio,Claudio Di Sipio,Phuong T. Nguyen,Riccardo Rubei

2024-10-23

Abstract:Model-Driven Engineering (MDE) has seen significant advancements with the integration of Machine Learning (ML) and Deep Learning (DL) techniques. Building upon the groundwork of previous investigations, our study provides a concise overview of current Language Large Models (LLMs) applications in MDE, emphasizing their role in automating tasks like model repository classification and developing advanced recommender systems. The paper also outlines the technical considerations for seamlessly integrating LLMs in MDE, offering a practical guide for researchers and practitioners. Looking forward, the paper proposes a focused research agenda for the future interplay of LLMs and MDE, identifying key challenges and opportunities. This concise roadmap envisions the deployment of LLM techniques to enhance the management, exploration, and evolution of modeling ecosystems. By offering a compact exploration of LLMs in MDE, this paper contributes to the ongoing evolution of MDE practices, providing a forward-looking perspective on the transformative role of Language Large Models in software engineering and model-driven practices.

Software Engineering
LARGE LANGUAGE MODEL-BASED ARTIFICIAL INTELLIGENCE IN THE LANGUAGE CLASSROOM: PRACTICAL IDEAS FOR TEACHING

Euan Bonner,Ryan Lege,Erin Frazier

DOI: https://doi.org/10.56297/bkam1691/wieo1749

Abstract:Large Language Models (LLMs) are a powerful type of Artificial Intelligence (AI) that simulates how humans organize language and are able to interpret, predict, and generate text. This allows for contextual understanding of natural human language which enables the LLM to understand conversational human input and respond in a natural manner. Recent examples of this, such as the Generative Pre-Trained Transformer (GPT) model, popularized by OpenAI’s web application, ChatGPT, are able to complete an astounding variety of tasks when provided with simple language input. For education, LLMs can alleviate teacher curriculum and grading workloads and even perform specific tasks such as generating creative ideas for activities. Specifically for language learning, LLMs can draw on their immense corpus of language content to generate learner-centric materials to aid teachers in delivering targeted, personalized language instruction. The aim of this paper is to provide the reader with examples of how LLMs can be utilized for materials development, classroom activities, and providing feedback. After giving specific examples and explanations, the paper will conclude with a discussion of how this technology can provide teachers with new innovative ways to streamline the teaching process to focus on learner needs.

Linguistics,Education,Computer Science
Large Language Models in Computer Science Education: A Systematic Literature Review

Nishat Raihan,Mohammed Latif Siddiq,Joanna C.S. Santos,Marcos Zampieri

2024-10-22

Abstract:Large language models (LLMs) are becoming increasingly better at a wide range of Natural Language Processing tasks (NLP), such as text generation and understanding. Recently, these models have extended their capabilities to coding tasks, bridging the gap between natural languages (NL) and programming languages (PL). Foundational models such as the Generative Pre-trained Transformer (GPT) and LLaMA series have set strong baseline performances in various NL and PL tasks. Additionally, several models have been fine-tuned specifically for code generation, showing significant improvements in code-related applications. Both foundational and fine-tuned models are increasingly used in education, helping students write, debug, and understand code. We present a comprehensive systematic literature review to examine the impact of LLMs in computer science and computer engineering education. We analyze their effectiveness in enhancing the learning experience, supporting personalized education, and aiding educators in curriculum development. We address five research questions to uncover insights into how LLMs contribute to educational outcomes, identify challenges, and suggest directions for future research.

Machine Learning,Human-Computer Interaction
Transforming Scholarly Landscapes: Influence of Large Language Models on Academic Fields beyond Computer Science

Aniket Pramanick,Yufang Hou,Saif M. Mohammad,Iryna Gurevych

2024-09-29

Abstract:Large Language Models (LLMs) have ushered in a transformative era in Natural Language Processing (NLP), reshaping research and extending NLP's influence to other fields of study. However, there is little to no work examining the degree to which LLMs influence other research fields. This work empirically and systematically examines the influence and use of LLMs in fields beyond NLP. We curate $106$ LLMs and analyze $\sim$$148k$ papers citing LLMs to quantify their influence and reveal trends in their usage patterns. Our analysis reveals not only the increasing prevalence of LLMs in non-CS fields but also the disparities in their usage, with some fields utilizing them more frequently than others since 2018, notably Linguistics and Engineering together accounting for $\sim$$45\%$ of LLM citations. Our findings further indicate that most of these fields predominantly employ task-agnostic LLMs, proficient in zero or few-shot learning without requiring further fine-tuning, to address their domain-specific problems. This study sheds light on the cross-disciplinary impact of NLP through LLMs, providing a better understanding of the opportunities and challenges.

Computation and Language

Large language models and academic writing: Five tiers of engagement

Large Language Models and Knowledge Graphs: Opportunities and Challenges

The rise of large language models: challenges for Critical Discourse Studies

Eight Things to Know about Large Language Models

An Interdisciplinary Outlook on Large Language Models for Scientific Research

Apprentices to Research Assistants: Advancing Research with Large Language Models

Large Language Models: A Survey

Large language models are changing landscape of academic publications. A positive transformation?

Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding

Exploring the landscape of large language models: Foundations, techniques, and challenges

Practical Applications of Large Language Models for Health Care Professionals and Scientists

Challenges and Applications of Large Language Models

A Comprehensive Overview of Large Language Models

Large Language Models for Mathematicians

Large language model programs

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

Ten simple rules to leverage large language models for getting grants

On the use of Large Language Models in Model-Driven Engineering

LARGE LANGUAGE MODEL-BASED ARTIFICIAL INTELLIGENCE IN THE LANGUAGE CLASSROOM: PRACTICAL IDEAS FOR TEACHING

Large Language Models in Computer Science Education: A Systematic Literature Review

Transforming Scholarly Landscapes: Influence of Large Language Models on Academic Fields beyond Computer Science