Establishing and Evaluating Trustworthy AI: Overview and Research Challenges

Dominik Kowald,Sebastian Scher,Viktoria Pammer-Schindler,Peter Müllner,Kerstin Waxnegger,Lea Demelius,Angela Fessl,Maximilian Toller,Inti Gabriel Mendoza Estrada,Ilija Simic,Vedran Sabol,Andreas Truegler,Eduardo Veas,Roman Kern,Tomislav Nad,Simone Kopeinik

2024-11-15

Abstract:Artificial intelligence (AI) technologies (re-)shape modern life, driving innovation in a wide range of sectors. However, some AI systems have yielded unexpected or undesirable outcomes or have been used in questionable manners. As a result, there has been a surge in public and academic discussions about aspects that AI systems must fulfill to be considered trustworthy. In this paper, we synthesize existing conceptualizations of trustworthy AI along six requirements: 1) human agency and oversight, 2) fairness and non-discrimination, 3) transparency and explainability, 4) robustness and accuracy, 5) privacy and security, and 6) accountability. For each one, we provide a definition, describe how it can be established and evaluated, and discuss requirement-specific research challenges. Finally, we conclude this analysis by identifying overarching research challenges across the requirements with respect to 1) interdisciplinary research, 2) conceptual clarity, 3) context-dependency, 4) dynamics in evolving systems, and 5) investigations in real-world contexts. Thus, this paper synthesizes and consolidates a wide-ranging and active discussion currently taking place in various academic sub-communities and public forums. It aims to serve as a reference for a broad audience and as a basis for future research directions.

Machine Learning,Information Retrieval

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to explore and evaluate the requirements and challenges in building Trustworthy AI. With the wide application of artificial intelligence technology in various fields, some AI systems have produced unexpected or unwanted results or have been used in controversial ways. This has triggered extensive discussions among the public and academia about what requirements an AI system must meet to be considered trustworthy. The paper synthesizes the existing conceptualizations of Trustworthy AI and focuses on six main requirements:

1. **Human Agency and Oversight**
2. **Fairness and Non - Discrimination**
3. **Transparency and Explainability**
4. **Robustness and Accuracy**
5. **Privacy and Security**
6. **Accountability**

For each requirement, the paper provides a definition, describes how to establish and evaluate these requirements, and discusses the research challenges of specific requirements. In addition, the paper also identifies the overall research challenges across these requirements, including interdisciplinary research, conceptual clarity, context - dependence, dynamically evolving systems, and research in real - world situations.

### Main contributions

1. **A comprehensive overview of the requirements for Trustworthy AI**: The paper covers different perspectives, including technical, human - centered, and legal considerations.
2. **Discussion of open issues and challenges in defining, establishing, and evaluating these requirements**: It emphasizes the need for further research, especially in high - risk areas (such as healthcare), to ensure that human values and rights are not compromised.

### Research methods

The paper adopts a semi - structured literature review method and conducts extensive literature searches and screenings through Scopus and Google Scholar, and finally collects 183 relevant literatures. These literatures cover the above six requirements and aim to generate a comprehensive understanding of relevant research directions and their existing challenges.

### Conclusions

The paper points out that while the evaluation of technical requirements (such as robustness) can rely on established indicators and test procedures, human - centered considerations (such as fairness and accountability) often require more detailed methods, taking into account ethical, legal, and cultural factors. Therefore, the paper emphasizes the need to develop robust evaluation schemes applicable to various AI systems, especially in high - risk areas.

### Future research directions

1. **Interdisciplinary research**: Combine knowledge from different disciplines to more comprehensively understand the requirements for Trustworthy AI.
2. **Conceptual clarity**: Clarify the definitions and scopes of each requirement for better implementation and evaluation.
3. **Context - dependence**: Consider the specific needs and challenges in different application scenarios.
4. **Dynamically evolving systems**: Research the trust issues of AI systems in a constantly changing environment.
5. **Research in real - world situations**: Verify the requirements and evaluation methods of Trustworthy AI in practical applications.

In summary, this paper provides a comprehensive framework for understanding and evaluating Trustworthy AI and points out important directions for future research.