A Reality check of the benefits of LLM in business

Ming Cheung

2024-06-09

Abstract:Large language models (LLMs) have achieved remarkable performance in language understanding and generation tasks by leveraging vast amounts of online texts. Unlike conventional models, LLMs can adapt to new domains through prompt engineering without the need for retraining, making them suitable for various business functions, such as strategic planning, project implementation, and data-driven decision-making. However, their limitations in terms of bias, contextual understanding, and sensitivity to prompts raise concerns about their readiness for real-world applications. This paper thoroughly examines the usefulness and readiness of LLMs for business processes. The limitations and capacities of LLMs are evaluated through experiments conducted on four accessible LLMs using real-world data. The findings have significant implications for organizations seeking to leverage generative AI and provide valuable insights into future research directions. To the best of our knowledge, this represents the first quantified study of LLMs applied to core business operations and challenges.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

This paper primarily discusses the practicality and limitations of large language models (LLMs) in commercial applications. Despite the potential shown by LLMs in business functions such as strategic planning, project implementation, and data-driven decision-making due to their adaptability to new domains without retraining, they face issues related to bias, context understanding, and sensitivity to prompts, raising doubts about their readiness for real-world applications. The paper evaluates the utility and capabilities of LLMs in business processes through empirical experiments, using four accessible LLMs to assess real-world data. These experiments hold significant importance for organizations seeking to leverage generative AI and provide valuable insights for future research directions. The authors point out that this is the first quantitative study on the application of LLMs to core business operations and challenges. The paper also discusses common types of LLMs such as ChatGPT, Claude, Llama, and PaLM, and provides examples of their applications in tasks such as text analysis, content generation, translation, and code generation. However, the paper also highlights the limitations of LLMs concerning bias, context understanding, and prompt sensitivity, demonstrating through experiments how these limitations impact the performance of LLMs in tasks like reference generation, code generation, and context-based question answering. In conclusion, this paper aims to provide a practical evaluation of LLMs in commercial environments, emphasizing their advantages and limitations, and offering guidance for future research and practice.

A Reality check of the benefits of LLM in business

LLMs' Understanding of Natural Language Revealed

Large Language Models are legal but they are not: Making the case for a powerful LegalLLM

Easy Problems That LLMs Get Wrong

The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?

Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Review

Investigating LLM Applications in E-Commerce

"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

Several categories of Large Language Models (LLMs): A Short Survey

LLM4DS: Evaluating Large Language Models for Data Science Code Generation

A Survey on Human-Centric LLMs

A Survey of Useful LLM Evaluation

Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong

Eight Things to Know about Large Language Models

Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis

Spoken Language Intelligence of Large Language Models for Language Learning

When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey

LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

Unveiling the Competitive Dynamics: A Comparative Evaluation of American and Chinese LLMs