An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support

Yongkun Liu,Jiachi Chen,Tingting Bi,John Grundy,Yanlin Wang,Jianxing Yu,Ting Chen,Yutian Tang,Zibin Zheng

2024-06-06

Abstract:Low-code programming (LCP) refers to programming using models at higher levels of abstraction, resulting in less manual and more efficient programming, and reduced learning effort for amateur developers. Many LCP tools have rapidly evolved and have benefited from the concepts of visual programming languages (VPLs) and programming by demonstration (PBD). With huge increase in interest in using large language models (LLMs) in software engineering, LLM-based LCP has began to become increasingly important. However, the technical principles and application scenarios of traditional approaches to LCP and LLM-based LCP are significantly different. Understanding these key differences and characteristics in the application of the two approaches to LCP by users is crucial for LCP providers in improving existing and developing new LCP tools, and in better assisting users in choosing the appropriate LCP technology. We conducted an empirical study of both traditional LCP and LLM-based LCP. We analyzed developers' discussions on Stack Overflow (SO) over the past three years and then explored the similarities and differences between traditional LCP and LLM-based LCP features and developer feedback. Our findings reveal that while traditional LCP and LLM-based LCP share common primary usage scenarios, they significantly differ in scope, limitations and usage throughout the software development lifecycle, particularly during the implementation phase. We also examine how LLMs impact and integrate with LCP, discussing the latest technological developments in LLM-based LCP, such as its integration with VPLs and the application of LLM Agents in software engineering.

Software Engineering

What problem does this paper attempt to address?

This paper mainly explores two types of Low-Code Programming (LCP) methods: traditional LCP and LCP based on Large Language Models (LLM). LCP aims to reduce manual coding, improve development efficiency, and lower the learning curve for amateur developers through higher-level abstract models. The paper compares the characteristics, application scenarios, and limitations of these two LCP methods by analyzing developer discussions on Stack Overflow over the past three years. The research found that although they have similar main application scenarios, traditional LCP is usually limited to specific domains, such as web development, while LCP based on LLM can be applied to more domains due to its extensive programming knowledge base. Both of them focus on the implementation stage in the software development lifecycle, but LCP based on LLM also plays an important role in the deployment stage. Traditional LCP has intuitiveness and readability, while LCP based on LLM may result in decreased program accuracy due to inaccurate user prompts, LLM hallucination problems, and code generation errors. The paper identifies the limitations of traditional LCP and LCP based on LLM through research, providing guidance for improving LCP tools and selecting appropriate technologies for users. Furthermore, the paper also discusses how LLM influences LCP, such as integration with Visual Programming Languages (VPLs) and the application of LLM agents in software engineering.

An Empirical Study on Low Code Programming using Traditional vs Large Language Model Support

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

Low-code LLM: Graphical User Interface over Large Language Models

Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks

If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

Planning-Driven Programming: A Large Language Model Programming Workflow

Characterizing Developers' Behaviors in LLM -Supported Software Development

A Survey on Large Language Models for Code Generation

Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation

An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project

An Empirical Study on Challenges for LLM Application Developers

A Survey on Large Language Models for Software Engineering

Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

Requirements are All You Need: From Requirements to Code with LLMs

Experimenting a New Programming Practice with LLMs

Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code

LMs: Understanding Code Syntax and Semantics for Code Analysis

Empirical Evaluation of Large Language Models for Novice Program Fault Localization