Abstract:Some companies(e.g., Microsoft Research and Google DeepMind) have discovered some of the limitations of GPTs autoregressive paradigm next-word prediction, manifested in the model lack of planning, working memory, backtracking, and reasoning skills. GPTs rely on a local and greedy process of generating the next word, without a global understanding of the task or the output.We have confirmed the above limitations through specialized empirical studies of code comprehension. Although GPT4 is good at producing fluent and coherent text, it cannot handle complex logic and generate new code that haven not been seen, and it relies too much on the formatting of the prompt to generate the correct code.We propose a new paradigm for code understanding that goes beyond the next-word prediction paradigm, inspired by the successful application of diffusion techniques to image generation(Dalle2, Sora) and protein structure generation(AlphaFold3), which have no autoregressive constraints.Instead of encoding the code in a form that mimics natural language, we encode the code as a heterogeneous image paradigm with a memory of global information that mimics both images and protein structures.We then refer to Sora's CLIP upstream text-to-image encoder model to design a text-to-code encoder model that can be applied to various downstream code understanding tasks.The model learns the global understanding of code under the new paradigm heterogeneous image, connects the encoding space of text and code, and encodes the input of text into the vector of code most similar to it.Using self-supervised comparative learning on 456,360 text-code pairs, the model achieved a zero-shot prediction of new data. This work is the basis for future work on code generation using diffusion techniques under a new paradigm to avoid autoregressive limitations.

Pseudocode to Code Based on Adaptive Global and Local Information

PinNet: Pinpoint Instructive Information for Retrieval Augmented Code-to-Text Generation

Fine-grained Pseudo-code Generation Method via Code Feature Extraction and Transformer

CodeAttention: Translating Source Code to Comments by Exploiting the Code Constructs

Automatic Generation of Pseudocode with Attention Seq2seq Model

Compilable Neural Code Generation with Compiler Feedback

Automatic Program Synthesis of Long Programs with a Learned Garbage Collector

Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation

A new approach for encoding code and assisting code understanding

SkCoder: A Sketch-based Approach for Automatic Code Generation

Code Attention: Translating Code to Comments by Exploiting Domain Features

AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

Learn to Align - A Code Alignment Network for Code Clone Detection.

Code Generation with Hybrid of Structural and Semantic Features Retrieval

Hierarchical Attention Graph Embedding Networks for Binary Code Similarity against Compilation Diversity

CodeDPO: Aligning Code Models with Self Generated and Verified Source Code

deGraphCS : Embedding Variable-based Flow Graph for Neural Code Search

Adaptive Code Completion with Meta-learning.

Code2Text: Dual Attention Syntax Annotation Networks for Structure-Aware Code Translation

Revisiting Code Search in a Two-Stage Paradigm