Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow

Tao Long,Katy Ilonka Gero,Lydia B. Chilton

2024-06-01

Abstract:Generative AI brings novel and impressive abilities to help people in everyday tasks. There are many AI workflows that solve real and complex problems by chaining AI outputs together with human interaction. Although there is an undeniable lure of AI, it is uncertain how useful generative AI workflows are after the novelty wears off. Additionally, workflows built with generative AI have the potential to be easily customized to fit users' individual needs, but do users take advantage of this? We conducted a three-week longitudinal study with 12 users to understand the familiarization and customization of generative AI tools for science communication. Our study revealed that there exists a familiarization phase, during which users were exploring the novel capabilities of the workflow and discovering which aspects they found useful. After this phase, users understood the workflow and were able to anticipate the outputs. Surprisingly, after familiarization the perceived utility of the system was rated higher than before, indicating that the perceived utility of AI is not just a novelty effect. The increase in benefits mainly comes from end-users' ability to customize prompts, and thus potentially appropriate the system to their own needs. This points to a future where generative AI systems can allow us to design for appropriation.

Human-Computer Interaction,Artificial Intelligence,Computation and Language,Computers and Society

What problem does this paper attempt to address?

The problem this paper attempts to address is the long-term utility and customization potential of Generative AI in solving real-world problems. Specifically, the researchers focus on the following points:

1. **Novelty Effect**: Is the initial popularity of Generative AI workflows among users merely due to their novelty? Does their actual utility decline once users become familiar with these tools?
2. **Customization Capability**: Will users leverage the customization features of Generative AI workflows to suit their own needs? Can this customization enhance the long-term utility of the system?
3. **Familiarization Process**: Is there a familiarization phase for users when using Generative AI tools? How long is this phase? How do users' behaviors and perceptions change during this phase?

To answer these questions, the researchers conducted a three-week longitudinal study, recruiting 12 PhD students in the field of computer science to use a Generative AI-based workflow to write scientific communication Tweetorial hooks. Through this study, they hope to understand users' experiences, customization behaviors, and perceived changes in system utility after prolonged use of these tools.