Abstract:Some companies(e.g., Microsoft Research and Google DeepMind) have discovered some of the limitations of GPTs autoregressive paradigm next-word prediction, manifested in the model lack of planning, working memory, backtracking, and reasoning skills. GPTs rely on a local and greedy process of generating the next word, without a global understanding of the task or the output.We have confirmed the above limitations through specialized empirical studies of code comprehension. Although GPT4 is good at producing fluent and coherent text, it cannot handle complex logic and generate new code that haven not been seen, and it relies too much on the formatting of the prompt to generate the correct code.We propose a new paradigm for code understanding that goes beyond the next-word prediction paradigm, inspired by the successful application of diffusion techniques to image generation(Dalle2, Sora) and protein structure generation(AlphaFold3), which have no autoregressive constraints.Instead of encoding the code in a form that mimics natural language, we encode the code as a heterogeneous image paradigm with a memory of global information that mimics both images and protein structures.We then refer to Sora's CLIP upstream text-to-image encoder model to design a text-to-code encoder model that can be applied to various downstream code understanding tasks.The model learns the global understanding of code under the new paradigm heterogeneous image, connects the encoding space of text and code, and encodes the input of text into the vector of code most similar to it.Using self-supervised comparative learning on 456,360 text-code pairs, the model achieved a zero-shot prediction of new data. This work is the basis for future work on code generation using diffusion techniques under a new paradigm to avoid autoregressive limitations.

A Grammar-Based Structural CNN Decoder for Code Generation

Learning to Decode for Future Success

TreeGen: A Tree-Based Transformer Architecture for Code Generation

StructCoder: Structure-Aware Transformer for Code Generation

GrammarT5: Grammar-Integrated Pretrained Encoder-Decoder Neural Model for Code

Code Generation with Hybrid of Structural and Semantic Features Retrieval

Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation

CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation

SeCNN: A semantic CNN parser for code comment generation

Toward Exploring the Code Understanding Capabilities of Pre-trained Code Generation Models

Planning with Large Language Models for Code Generation

A new approach for encoding code and assisting code understanding

A Compiler Design for a Programmable CNN Accelerator

Compilable Neural Code Generation with Compiler Feedback

GANCoder: An Automatic Natural Language-to-Programming Language Translation Approach based on GAN

JumpCoder: Go Beyond Autoregressive Coder via Online Modification

Context-aware Code Generation with Synchronous Bidirectional Decoder

XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

Grammar-based Neural Text-to-SQL Generation

Automatic Code Annotation Generation Based on Multi-dimensional Heterogeneous Graph Structure

Genetic CNN