Frontend Diffusion: Exploring Intent-Based User Interfaces through Abstract-to-Detailed Task Transitions

Qinshi Zhang,Latisha Besariani Hendra,Mohan Chi,Zijian Ding

2024-07-17

Abstract:The emergence of Generative AI is catalyzing a paradigm shift in user interfaces from command-based to intent-based outcome specification. In this paper, we explore abstract-to-detailed task transitions in the context of frontend code generation as a step towards intent-based user interfaces, aiming to bridge the gap between abstract user intentions and concrete implementations. We introduce Frontend Diffusion, an end-to-end LLM-powered tool that generates high-quality websites from user sketches. The system employs a three-stage task transition process: sketching, writing, and coding. We demonstrate the potential of task transitions to reduce human intervention and communication costs in complex tasks. Our work also opens avenues for exploring similar approaches in other domains, potentially extending to more complex, interdependent tasks such as video production.

Human-Computer Interaction,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of how to achieve intent-based user interfaces through abstract-to-detailed task transitions, thereby reducing user intervention and communication costs in complex tasks. Specifically, the paper introduces an end-to-end tool called Frontend Diffusion, which leverages large language models (LLMs) to generate high-quality website code from user sketches.

### Main Issues:
1. **Reducing User Intervention**: Current user interfaces still require a significant amount of user intervention when handling command-intensive tasks. Users typically need to specify tasks step by step, while AI generates the corresponding code. This mode is similar to agile programming but is less efficient and prone to errors.
2. **Improving Output Quality**: Existing generative AI performs well in handling tasks within a fixed scope but still requires considerable user intervention when dealing with information-rich tasks. This limits the application of generative AI in complex tasks.
3. **Achieving Seamless Transition from Intent to Command**: How to transform users' abstract intentions (such as sketches, theme descriptions) into specific commands (such as code) to generate high-quality final products.

### Solution:
- **Task Transition Paradigm**: The paper proposes a three-stage task transition process: sketching, writing, and coding. This process aims to gradually transform users' abstract intentions into concrete implementations.
- **Frontend Generation Tool**: The Frontend Diffusion tool utilizes large language models (such as Claude 3.5 Sonnet) to complete the generation of high-quality website code from sketches. The tool includes the following main components:
- **Sketching**: Users can draw website layouts on a canvas and input theme descriptions.
- **Writing**: The system converts users' visual and textual inputs into a Product Requirements Document (PRD), serving as a blueprint for website development.
- **Coding**: The system generates initial code based on the generated PRD and the original user prompts, and automatically improves code quality and functionality through an iterative optimization process.

### Potential Impact:
- **Simplifying Complex Tasks**: The task transition paradigm can simplify complex, interdependent tasks, such as video production.
- **Promoting Intent-Driven Interfaces**: This approach provides new ideas for exploring intent-driven user interfaces in other fields, potentially reducing communication costs between users and generative AI systems.

In summary, the paper aims to achieve seamless transitions from users' abstract intentions to concrete implementations by proposing a new task transition paradigm, thereby enhancing the application of generative AI in complex tasks.