Simulation-Based Optimization of User Interfaces for Quality-Assuring Machine Learning Model Predictions

Yu Zhang,Martijn Tennekes,Tim de Jong,Lyana Curier,Bob Coecke,Min Chen

DOI: https://doi.org/10.1145/3594552

2023-08-31

Abstract:Quality-sensitive applications of machine learning (ML) require quality assurance (QA) by humans before the predictions of an ML model can be deployed. QA for ML (QA4ML) interfaces require users to view a large amount of data and perform many interactions to correct errors made by the ML model. An optimized user interface (UI) can significantly reduce interaction costs. While UI optimization can be informed by user studies evaluating design options, this approach is not scalable because there are typically numerous small variations that can affect the efficiency of a QA4ML interface. Hence, we propose using simulation to evaluate and aid the optimization of QA4ML interfaces. In particular, we focus on simulating the combined effects of human intelligence in initiating appropriate interaction commands and machine intelligence in providing algorithmic assistance for accelerating QA4ML processes. As QA4ML is usually labor-intensive, we use the simulated task completion time as the metric for UI optimization under different interface and algorithm setups. We demonstrate the usage of this UI design method in several QA4ML applications.

Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of how to optimize the user interface (UI) in quality-sensitive applications to improve the efficiency of quality assurance (QA) for machine learning (ML) model predictions. Specifically, the paper focuses on the QA process where users need to review large amounts of data and correct the erroneous predictions of ML models. This process is often very time-consuming and labor-intensive, so optimizing the UI can significantly reduce the interaction cost for users.

### Background and Problem Description
1. **Accuracy of Machine Learning Model Predictions**:
- In many real-world applications, the prediction accuracy of ML models often cannot reach 100%.
- For some quality-sensitive applications (such as medical diagnosis, fraud detection, and content moderation), prediction errors are unacceptable, thus requiring strict QA.

2. **User Interface in the QA Process**:
- During the QA process, users need to review the predictions of ML models through the UI and correct errors.
- These UIs usually require users to view large amounts of data and perform multiple interactions to correct the model's erroneous predictions.
- Optimizing the UI can significantly reduce the interaction cost for users and improve work efficiency.

3. **Limitations of Existing Methods**:
- Traditional user research methods (such as user testing and usability evaluation) can help improve the UI, but these methods are usually costly and time-consuming.
- Due to the numerous design options for UIs, traditional methods find it difficult to evaluate the effects of different design options on a large scale.

### Proposed Solution
The paper proposes a simulation-based method to evaluate and optimize QA UIs (QA4ML interfaces). The specific methods include:

1. **Simulating User and Machine Operations**:
- By modeling and simulating the routine operation sequences and time costs of users in the QA process, the effects of different UI design options are evaluated.
- The simulation method can consider various factors, such as user skills, operation strategies, and the accuracy of ML models.

2. **Task Completion Time as an Evaluation Metric**:
- Task completion time is used as the main evaluation metric because the QA process is usually labor-intensive, and reducing task completion time can significantly improve user work efficiency.

3. **Application Examples**:
- The paper demonstrates how this method can be applied to optimize QA UIs in data extraction and aerial image classification systems.

### Main Contributions
1. **Establishing a Simulation Model of User Routine Operation Sequences and Time Costs**:
- This model can be used to evaluate and optimize QA UIs.
2. **Applying the Model to Real Scenarios**:
- The simulation method was used to optimize QA UIs in data extraction and aerial image classification systems.

### Conclusion
By using a simulation-based method, researchers can efficiently evaluate and optimize QA UIs, thereby reducing user interaction costs and improving work efficiency. This method is particularly suitable for quality-sensitive applications that require extensive user interaction.