Rethinking Software Engineering in the Era of Foundation Models: A Curated Catalogue of Challenges in the Development of Trustworthy FMware
Ahmed E. Hassan,Dayi Lin,Gopi Krishnan Rajbahadur,Keheliya Gallaba,Filipe R. Cogo,Boyuan Chen,Haoxiang Zhang,Kishanthan Thangarajah,Gustavo Ansaldi Oliva,Jiahuei Lin,Wali Mohammad Abdullah,Zhen Ming Jiang
2024-02-25
Abstract:Foundation models (FMs), such as Large Language Models (LLMs), have
revolutionized software development by enabling new use cases and business
models. We refer to software built using FMs as FMware. The unique properties
of FMware (e.g., prompts, agents, and the need for orchestration), coupled with
the intrinsic limitations of FMs (e.g., hallucination) lead to a completely new
set of software engineering challenges. Based on our industrial experience, we
identified 10 key SE4FMware challenges that have caused enterprise FMware
development to be unproductive, costly, and risky. In this paper, we discuss
these challenges in detail and state the path for innovation that we envision.
Next, we present FMArts, which is our long-term effort towards creating a
cradle-to-grave platform for the engineering of trustworthy FMware. Finally, we
(i) show how the unique properties of FMArts enabled us to design and develop a
complex FMware for a large customer in a timely manner and (ii) discuss the
lessons that we learned in doing so. We hope that the disclosure of the
aforementioned challenges and our associated efforts to tackle them will not
only raise awareness but also promote deeper and further discussions, knowledge
sharing, and innovative solutions across the software engineering discipline.
Software Engineering,Artificial Intelligence