Abstract:We introduce BRIDGET, a novel human-in-the-loop system for hybrid decision-making, aiding the user to label records from an un-labeled dataset, attempting to ``bridge the gap'' between the two most popular Hybrid Decision-Making paradigms: those featuring the human in a leading position, and the other with a machine making most of the decisions. BRIDGET understands when either a machine or a human user should be in charge, dynamically switching between two statuses. In the different statuses, BRIDGET still fosters the human-AI interaction, either having a machine learning model assuming skeptical stances towards the user and offering them suggestions, or towards itself and calling the user back. We believe our proposal lays the groundwork for future synergistic systems involving a human and a machine decision-makers.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the gap between the two main paradigms in existing Hybrid Decision - Making Systems (HDM): one is the human - dominated decision - making mode, and the other is the machine - dominated decision - making mode. Specifically:
1. **Human - dominated decision - making mode**:
- For example, Skeptical Learning (SL). This mode assumes that the human user is always in control, but needs the help of the machine to maintain consistency.
2. **Machine - dominated decision - making mode**:
- For example, Learning - to - Defer (LtD). This mode assumes that some decisions are more suitable for the machine or the user to make. The model not only needs to classify instances but also needs to decide when to hand over the decision - making power to an external human supervisor.
The differences between these two paradigms are as follows:
- **Role assignment**: In SL, the human is always the final decision - maker; while in LtD, the machine undertakes most of the decision - making tasks and only hands over the decision - making power to the human when the uncertainty is high.
- **Training process**: The training of SL is continuous, and the model keeps learning from the user's final decisions; while LtD is trained with a static data set, resulting in different ground truths for classification and deferral strategies.
To solve these problems, the author proposes a new system named Bridget. Bridget aims to "fill" the gap between these two paradigms by dynamically switching between two states (Human - in - Command and Machine - in - Command), while promoting human - machine interaction and adaptively adjusting according to different scenarios.
### How Bridget Works
The Bridget system has two potential states:
- **Human - in - Command (HiC) phase**: The human user H is responsible for all decisions, and the machine M provides suggestions and explanations, especially when being skeptical about the user's decisions.
- **Machine - in - Command (MiC) phase**: The machine M is responsible for most of the decisions, but in uncertain situations, it will recall the user H for decision - making and be able to explain its behavior.
Bridget can cycle between these two phases to adapt to the user's needs, potential drops in model accuracy, or new situations in the data. The specific process is as follows:
#### Human - in - Command phase
1. The user H and the machine M jointly process each new record xi.
2. Calculate the model's skepticism. If the skepticism exceeds the threshold α, prompt the user whether to accept the machine's suggestion.
3. The user can choose to ask for an explanation and decide whether to accept the machine's label according to the explanation.
4. Update the model and parameters and check whether it is necessary to switch to the MiC phase.
#### Machine - in - Command phase
1. The machine M calculates its belief (prediction probability b) for each new record xi.
2. If the belief is lower than the threshold β, immediately recall the user H for decision - making and provide an explanation.
3. If the belief is higher than the threshold β, in order to avoid excessive dependence on the machine, randomly select some records for the user to check.
4. Update the model and parameters and check whether it is necessary to return to the HiC phase.
### Conclusions and Future Work
Bridget trains the ML model through a co - evolution process to imitate the user's behavior, realizing the interactive dynamics between humans and machines, thus allowing continuous updates of system parameters. Future research plans include:
- Comparing the effect of Bridget by testing it against an independent LtD system.
- Considering different data types (such as time series) to provide alternative explainability.
- Studying more deeply the influence of concept drift in phase transitions.
- Exploring other methods, such as using the internal uncertainty of the model instead of FEA values, or comparing the number of leaves of incremental decision trees at different times to evaluate the change of the model.
Through these improvements, Bridget is expected to further improve the performance and flexibility of hybrid decision - making systems.