Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design

Michelle S. Lam,Zixian Ma,Anne Li,Izequiel Freitas,Dakuo Wang,James A. Landay,Michael S. Bernstein

DOI: https://doi.org/10.1145/3544548.3581290

2023-03-06

Abstract:Machine learning practitioners often end up tunneling on low-level technical details like model architectures and performance metrics. Could early model development instead focus on high-level questions of which factors a model ought to pay attention to? Inspired by the practice of sketching in design, which distills ideas to their minimal representation, we introduce model sketching: a technical framework for iteratively and rapidly authoring functional approximations of a machine learning model's decision-making logic. Model sketching refocuses practitioner attention on composing high-level, human-understandable concepts that the model is expected to reason over (e.g., profanity, racism, or sarcasm in a content moderation task) using zero-shot concept instantiation. In an evaluation with 17 ML practitioners, model sketching reframed thinking from implementation to higher-level exploration, prompted iteration on a broader range of model designs, and helped identify gaps in the problem formulation$\unicode{x2014}$all in a fraction of the time ordinarily required to build a model.

Human-Computer Interaction,Artificial Intelligence,Machine Learning