Flowy: Supporting UX Design Decisions Through AI-Driven Pattern Annotation in Multi-Screen User Flows

Yuwen Lu,Ziang Tong,Qinyi Zhao,Yewon Oh,Bryan Wang,Toby Jia-Jun Li

2024-06-24

Abstract:Many recent AI-powered UX design tools focus on generating individual static UI screens from natural language. However, they overlook the crucial aspect of interactions and user experiences across multiple screens. Through formative studies with UX professionals, we identified limitations of these tools in supporting realistic UX design workflows. In response, we designed and developed Flowy, an app that augments designers' information foraging process in ideation by supplementing specific user flow examples with distilled design pattern knowledge. Flowy utilizes large multimodal AI models and a high-quality user flow dataset to help designers identify and understand relevant abstract design patterns in the design space for multi-screen user flows. Our user study with professional UX designers demonstrates how Flowy supports realistic UX tasks. Our design considerations in Flowy, such as representations with appropriate levels of abstraction and assisted navigation through the solution space, are generalizable to other creative tasks and embody a human-centered, intelligence augmentation approach to using AI in UX design.

Human-Computer Interaction

What problem does this paper attempt to address?

This paper aims to address the issue of artificial intelligence support in user experience (UX) design, specifically for multi-screen user flow design. Existing AI-driven UX design tools mainly focus on generating single static user interface (UI) screens from natural language, but overlook the critical aspect of interaction and user experience across multiple screens. Through research with professional UX designers, the authors identified limitations of these tools in supporting real-world UX design workflows, including the inability to generate designs based on enterprise design systems, a lack of design rationale or justification, and an excessive focus on individual UI screens while neglecting key UX details and context.

To tackle these issues, the paper introduces the Flowy system, an application that leverages multimodal artificial intelligence to automatically annotate design patterns in multi-screen user flows. Flowy supports the decision-making process of UX designers by analyzing latent design patterns in high-quality user flow examples and showcasing other high-quality designs with relevant patterns, thereby accelerating their information gathering process during the design concept stage and helping them to fully understand the design space. Additionally, Flowy considers the appropriate level of abstraction for representation and auxiliary navigation of the solution space, which can be extended to other creative tasks, reflecting a human-centered approach to intelligence augmentation in the use of AI in UX design.

The main contributions of the paper include:

1. The design and implementation of Flowy, a novel system that supports the conceptualization of UX design as part of the information gathering process;
2. A pipeline utilizing state-of-the-art multimodal AI technology to automatically annotate design pattern information on user flow design examples;
3. Research findings that demonstrate the effectiveness of Flowy, along with three design implications based on the findings, providing directions for future research in tools that support UX design conceptualization.

In summary, Flowy enhances the decision-making process of UX designers by automating design pattern analysis and providing insights into multi-screen user flow design, especially when dealing with complex user journeys and design patterns.