Abstract:Stakeholders constantly make assumptions in the development of deep learning (DL) frameworks. These assumptions are related to various types of software artifacts (e.g., requirements, design decisions, and technical debt) and can turn out to be invalid, leading to system failures. Existing approaches and tools for assumption management usually depend on manual identification of assumptions. However, assumptions are scattered in various sources (e.g., code comments, commits, pull requests, and issues) of DL framework development, and manually identifying assumptions has high costs. This study intends to evaluate different classification models for the purpose of identification with respect to assumptions from the point of view of developers and users in the context of DL framework projects (i.e., issues, pull requests, and commits) on GitHub. First, we constructed a new and largest dataset (i.e., the AssuEval dataset) of assumptions collected from the TensorFlow and Keras repositories on GitHub. Then we explored the performance of seven non-transformers based models (e.g., Support Vector Machine, Classification and Regression Trees), the ALBERT model, and three decoder-only models (i.e., ChatGPT, Claude, and Gemini) for identifying assumptions on the AssuEval dataset. The study results show that ALBERT achieves the best performance (f1-score: 0.9584) for identifying assumptions on the AssuEval dataset, which is much better than the other models (the 2nd best f1-score is 0.8858, achieved by the Claude 3.5 Sonnet model). Though ChatGPT, Claude, and Gemini are popular models, we do not recommend using them to identify assumptions in DL framework development because of their low performance. Fine-tuning ChatGPT, Claude, Gemini, or other language models (e.g., Llama3, Falcon, and BLOOM) specifically for assumptions might improve their performance for assumption identification.

Decide: Knowledge-Based Version Incompatibility Detection in Deep Learning Stacks

Knowledge-Based Version Incompatibility Detection for Deep Learning

Compatibility Issues in Deep Learning Systems: Problems and Opportunities

InterpretStack: Interpretable Exploration and Interactive Visualization Construction of Stacking Algorithm

Why Do Deep Learning Projects Differ in Compatible Framework Versionsƒ an Exploratory Study

Demystifying Dependency Bugs in Deep Learning Stack

What Do Programmers Discuss about Deep Learning Frameworks

Task-Oriented ML/DL Library Recommendation Based on a Knowledge Graph

Topological Interpretability for Deep-Learning

Interpreting Deep Learning Models for Knowledge Tracing

StackGenVis: Alignment of Data, Algorithms, and Models for Stacking Ensemble Learning Using Performance Metrics

Explainable AI: A Neurally-Inspired Decision Stack Framework

Understanding the Implementation Issues when Using Deep Learning Frameworks

LwDeepStack: A lightweight DeepStack based on knowledge distillation

Towards Interpretable Deep Learning Models for Knowledge Tracing

An Empirical Study of the Dependency Networks of Deep Learning Libraries

Interpretable Coronary Artery Disease Prediction Based on K-Stacking Model

StackOverflow vs Kaggle: A Study of Developer Discussions About Data Science

A comprehensive study on challenges in deploying deep learning based software

An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks