On AI-Inspired UI-Design

Jialiang Wei,Anne-Lise Courbis,Thomas Lambolais,Gérard Dray,Walid Maalej

2024-06-19

Abstract:Graphical User Interface (or simply UI) is a primary mean of interaction between users and their device. In this paper, we discuss three major complementary approaches on how to use Artificial Intelligence (AI) to support app designers create better, more diverse, and creative UI of mobile apps. First, designers can prompt a Large Language Model (LLM) like GPT to directly generate and adjust one or multiple UIs. Second, a Vision-Language Model (VLM) enables designers to effectively search a large screenshot dataset, e.g. from apps published in app stores. The third approach is to train a Diffusion Model (DM) specifically designed to generate app UIs as inspirational images. We discuss how AI should be used, in general, to inspire and assist creative app design rather than automating it.

Human-Computer Interaction,Artificial Intelligence,Software Engineering

What problem does this paper attempt to address?

The paper primarily explores how to leverage Artificial Intelligence (AI) technology to support mobile application User Interface (UI) design, aiming to enhance designers' creativity and efficiency. Specifically, the paper proposes three complementary approaches:

1. **Generating and adjusting UI through Large Language Models (LLM)**: Designers can directly generate and adjust one or more UI design schemes with the help of large language models like GPT.
2. **Using Visual-Language Models (VLM) to search large screenshot datasets**: This method enables designers to effectively find relevant UI designs from published applications, such as application screenshots in app stores.
3. **Training Diffusion Models (DM) to generate inspirational UI images**: This approach is specifically used to generate application UI images that can spark inspiration.

The paper emphasizes that although these AI technologies can significantly enhance creativity in the design process, they cannot completely replace the experience and creativity of human designers. Therefore, designers should view AI as an auxiliary tool rather than a fully automated design method. Additionally, the paper proposes an AI-based design process and provides detailed explanations of its various steps, aiming to guide designers on how to better utilize AI tools for innovative design.