Abstract:Deep learning models for natural language processing (NLP) are increasingly adopted and deployed by analysts without formal training in NLP or machine learning (ML). However, the documentation intended to convey the model's details and appropriate use is tailored primarily to individuals with ML or NLP expertise. To address this gap, we conduct a design inquiry into interactive model cards, which augment traditionally static model cards with affordances for exploring model documentation and interacting with the models themselves. Our investigation consists of an initial conceptual study with experts in ML, NLP, and AI Ethics, followed by a separate evaluative study with non-expert analysts who use ML models in their work. Using a semi-structured interview format coupled with a think-aloud protocol, we collected feedback from a total of 30 participants who engaged with different versions of standard and interactive model cards. Through a thematic analysis of the collected data, we identified several conceptual dimensions that summarize the strengths and limitations of standard and interactive model cards, including: stakeholders; design; guidance; understandability & interpretability; sensemaking & skepticism; and trust & safety. Our findings demonstrate the importance of carefully considered design and interactivity for orienting and supporting non-expert analysts using deep learning models, along with a need for consideration of broader sociotechnical contexts and organizational dynamics. We have also identified design elements, such as language, visual cues, and warnings, among others, that support interactivity and make non-interactive content accessible. We summarize our findings as design guidelines and discuss their implications for a human-centered approach towards AI/ML documentation.

The Model Card Authoring Toolkit: Toward Community-centered, Deliberation-driven AI Design.

Fostering Children's Dispositional Autonomy and AI Understanding Through Co-Designing AI Systems: A Learning Science Perspective

Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation

Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation

Interactive Model Cards: A Human-Centered Approach to Model Documentation

inML Kit: empowering the prototyping of ML-enhanced products by involving designers in the ML lifecycle

Towards "Anytime, Anywhere" Community Learning and Engagement around the Design of Public Sector AI

Codesigning AI with End-Users: An AI Literacy Toolkit for Nontechnical Audiences

What's documented in AI? Systematic Analysis of 32K AI Model Cards

Automatic Generation of Model and Data Cards: A Step Towards Responsible AI

Human-centered mechanism design with Democratic AI

The AI-DEC: A Card-based Design Method for User-centered AI Explanations

Bridging the gap: Towards an Expanded Toolkit for AI-driven Decision-Making in the Public Sector

Systematic analysis of 32,111 AI model cards characterizes documentation practice in AI

Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

AI Usage Cards: Responsibly Reporting AI-generated Content

The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources

Evolving AI Collectives to Enhance Human Diversity and Enable Self-Regulation

Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience

From Experts to the Public: Governing Multimodal Language Models in Politically Sensitive Video Analysis