Prototype2Code: End-to-end Front-end Code Generation from UI Design Prototypes

Shuhong Xiao,Yunnong Chen,Jiazhi Li,Liuqing Chen,Lingyun Sun,Tingting Zhou

2024-05-08

Abstract:UI-to-code technology has streamlined the front-end development process, reducing repetitive tasks for engineers. prior research mainly use design prototypes as inputs, with the effectiveness of the generated code heavily dependent on these prototypes' quality, leading to compromised robustness. Moreover, these approaches also exhibit shortcomings in code quality, including issues such as disorganized UI structures and the inability to support responsive layouts. To address these challenges, we introduce Prototype2Code, which achieves end-to-end front-end code generation with business demands. For Prototype2Code, we incorporate design linting into the workflow, addressing the detection of fragmented elements and perceptual groups, enhancing the robustness of the generated outcomes. By optimizing the hierarchical structure and intelligently recognizing UI element types, Prototype2Code generates code that is more readable and structurally clearer. To meet responsive design requirements, Prototype2Code primarily supports flexbox layout model, ensuring code compatibility across various device sizes. To validate the efficacy, we compare Prototype2Code with the commercial code generation platform CodeFun and Screenshot-to-code based on GPT-4 with vision. Employing structural similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), and mean squared error (MSE) for visual similarity assessment, Prototype2Code's rendered UI effects align most closely with the design prototypes, exhibiting the minimal errors. We also conduct a user study with five experienced front-end engineers, inviting them to review and revise code generated by the three methods. As a result, Prototype2Code surpasses other methods in readability, usability, and maintainability, better meeting the business needs of industrial development.

Software Engineering

What problem does this paper attempt to address?

This paper mainly explores the problem of generating front-end code from UI design prototypes. Existing techniques rely on high-quality design prototypes, but this approach lacks robustness and the generated code quality is problematic, such as messy UI structure and lack of support for responsive layout. To address these issues, the paper proposes Prototype2Code, an end-to-end front-end code generation system that combines design linting (the process of detecting and fixing code errors) to enhance the robustness of the generated results. Prototype2Code optimizes the hierarchical structure of UI elements, intelligently recognizes the types of UI elements, and generates code that is more readable and structured. Additionally, it supports flexbox layout mode to ensure compatibility across different device sizes.

To validate the effectiveness of Prototype2Code, the paper compares it with the commercial code generation platform CodeFun and the GPT-4-based Screenshot-to-code method. The visual similarity is evaluated using the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Mean Squared Error (MSE). Prototype2Code performs the best in reproducing design intentions and has the fewest errors. User studies also indicate that Prototype2Code surpasses other methods in readability, usability, and maintainability, better meeting the needs of industrial development.