Abstract:Creating programs to correctly manipulate data is a difficult task, as the underlying programming languages and APIs can be challenging to learn for many users who are not skilled programmers. Large language models (LLMs) demonstrate remarkable potential for generating code from natural language, but in the data manipulation domain, apart from the natural language (NL) description of the intended task, we also have the dataset on which the task is to be performed, or the "data context". Existing approaches have utilized data context in a limited way by simply adding relevant information from the input data into the prompts sent to the LLM. In this work, we utilize the available input data to execute the candidate programs generated by the LLMs and gather their outputs. We introduce semantic reranking, a technique to rerank the programs generated by LLMs based on three signals coming the program outputs: (a) semantic filtering and well-formedness based score tuning: do programs even generate well-formed outputs, (b) semantic interleaving: how do the outputs from different candidates compare to each other, and (c) output-based score tuning: how do the outputs compare to outputs predicted for the same task. We provide theoretical justification for semantic interleaving. We also introduce temperature mixing, where we combine samples generated by LLMs using both high and low temperatures. We extensively evaluate our approach in three domains, namely databases (SQL), data science (Pandas) and business intelligence (Excel's Power Query M) on a variety of new and existing benchmarks. We observe substantial gains across domains, with improvements of up to 45% in top-1 accuracy and 34% in top-3 accuracy.

Safurai-Csharp: Harnessing Synthetic Data to improve language-specific Code LLM

Safurai 001: New Qualitative Approach for Code LLM Evaluation

AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation

AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data

CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation

Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

Coarse-Tuning Models of Code with Reinforcement Learning Feedback

From Words to Code: Harnessing Data for Program Synthesis from Natural Language

Multi-Programming Language Ensemble for Code Generation in Large Language Model

Assured Automatic Programming via Large Language Models

LLM Security Guard for Code

CodeMind: A Framework to Challenge Large Language Models for Code Reasoning

dafny-annotator: AI-Assisted Verification of Dafny Programs

LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward

Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting

S3LLM: Large-Scale Scientific Software Understanding with LLMs using Source, Metadata, and Document

UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback

LLM-Assisted Code Cleaning For Training Accurate Code Generators

Verified Code Transpilation with LLMs

Instruction Tuning for Secure Code Generation