Model Lakes

Koyena Pal,David Bau,Renée J. Miller

2024-03-05

Abstract:Given a set of deep learning models, it can be hard to find models appropriate to a task, understand the models, and characterize how models are different one from another. Currently, practitioners rely on manually-written documentation to understand and choose models. However, not all models have complete and reliable documentation. As the number of machine learning models increases, this issue of finding, differentiating, and understanding models is becoming more crucial. Inspired from research on data lakes, we introduce and define the concept of model lakes. We discuss fundamental research challenges in the management of large models. And we discuss what principled data management techniques can be brought to bear on the study of large model management.

Databases,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper primarily explores how to manage and utilize a large number of machine learning models, particularly addressing a series of challenges related to the management and understanding of deep learning models. It introduces the concept of "Model Lakes" to address these issues.

#### Main Issues:
1. **Model Selection**: How to find the most suitable model for a specific task among the many available models? For example, a user wants to find a model that can non-technically summarize legal documents. There are numerous related models on the HuggingFace platform, but choosing the optimal model is a challenge.
2. **Model Understanding and Comparison**: How to understand the differences between various models and their respective strengths and weaknesses?
3. **Model Documentation**: Current model documentation is often incomplete or unreliable. How to ensure the accuracy and completeness of model documentation?
4. **Model Provenance**: How to track the origin and modification history of a model? For instance, is a model fine-tuned from another model?
5. **Version Management**: How to effectively manage and track different versions of a model and the changes during its training process?

### Specific Challenges:
- **Content Search**: Current model searches mainly rely on keyword matching, which is not semantic enough and prone to errors. How to achieve content-based model search?
- **Related Model Search**: How to identify and display other models related to a given model?
- **Documentation Verification**: How to verify the accuracy and completeness of model documentation? How to automate this process?
- **Data Citation**: How to standardize the citation of training data and its sources for traceability and verification?
- **Model Provenance Research**: How to effectively track the evolution of a model, including dataset updates, algorithm changes, etc.?

By introducing the concept of "Model Lakes," the paper aims to leverage research outcomes from the data lake domain to develop a systematic model management and analysis framework to address the above issues.