Abstract:Code generation aims to generate source code implementing human requirements illustrated with natural language specifications. With the rapid development of intelligent software engineering, automated code generation has become a hot research topic in both artificial intelligence and software engineering, and researchers have made significant achievements on code generation. More recently, large language models (LLMs) have demonstrated outstanding performance on code generation tasks, such as ChatGPT released by OpenAI presents the fantastic potential on automated code generation. However, the existing studies are limited to exploring LLMs' ability for generating code snippets to solve simple programming problems, the task of competition-level code generation has never been investigated. The specifications of the programming competition are always complicated and require the specific input/output format as well as the high-level algorithmic reasoning ability. In this study, we conduct the first large empirical study to investigate the zero-shot learning ability of ChatGPT for solving competition programming problems. Specifically, we warm up the design of prompts by using the Human-Eval dataset. Then, we apply the well-designed prompt to the competition-level code generation dataset, namely APPS, to further explore the effectiveness of using ChatGPT for solving competition problems. We collect ChatGPT's outputs on 5,000 code competition problems, the evaluation results show that it can successfully pass 25.4% test cases. By further feeding extra information (e.g, test failed information) to ChatGPT, we observe that ChatGPT has the potential to fix partial pass into a fully pass program. Moreover, we investigate the solutions generated by LLMs and the existing solutions, we find that it prefers to directly copy the code instead of re-write when facing more difficult problems. Finally, we evaluate the code quality generated by ChatGPT in terms of “code cleanness”, we observe that the generated codes are with small functions and file sizes, which are in line with the standard of clean code.

C3: Zero-shot Text-to-SQL with ChatGPT.

A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

Zero-shot Text-to-SQL Learning with Auxiliary Task

A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding

ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT

Is ChatGPT a General-Purpose Natural Language Processing Task Solver?

ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?

ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language

Eval-GCSC: A New Metric for Evaluating ChatGPT's Performance in Chinese Spelling Correction

The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge

Enhancing Text-based Knowledge Graph Completion with Zero-Shot Large Language Models: A Focus on Semantic Enhancement

SelECT-SQL: Self-correcting ensemble Chain-of-Thought for Text-to-SQL

Empirical Evaluation of ChatGPT on Requirements Information Retrieval Under Zero-Shot Setting

$R^3$: "This is My SQL, Are You With Me?" A Consensus-Based Multi-Agent System for Text-to-SQL Tasks

Zero-shot information extraction from radiological reports using ChatGPT

Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval

Empirical Study of Zero-Shot NER with ChatGPT

Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems

Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chain

A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT.

Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection Via Querying ChatGPT.