A Comprehensive Survey and Classification of Evaluation Criteria for Trustworthy Artificial Intelligence

Louise McCormack,Malika Bendechache

2024-10-10

Abstract:This paper presents a systematic review of the literature on evaluation criteria for Trustworthy Artificial Intelligence (TAI), with a focus on the seven EU principles of TAI. This systematic literature review identifies and analyses current evaluation criteria, maps them to the EU TAI principles and proposes a new classification system for each principle. The findings reveal both a need for and significant barriers to standardising criteria for TAI evaluation. The proposed classification contributes to the development, selection and standardization of evaluation criteria for TAI governance.

Computers and Society,Artificial Intelligence,Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to systematically review and analyze the existing evaluation criteria for measuring the seven EU principles of Trustworthy Artificial Intelligence (TAI). Specifically, it attempts to solve the following key problems:

1. **The need for standardized evaluation criteria**:
- The paper points out that although there are some frameworks and guidelines to guide the governance of trustworthy AI systems, there is still a lack of unified standards to evaluate the trustworthiness of these systems. Therefore, one of the goals of the paper is to identify the current evaluation criteria and propose a classification system to support standardized evaluation.

2. **Classification and mapping of evaluation criteria**:
- The author conducts a systematic literature review of the existing evaluation criteria and maps these criteria to the seven TAI principles proposed by the EU. These seven principles include: human autonomy and supervision, technical robustness and safety, privacy and data governance, transparency, diversity, non - discrimination and fairness, social and environmental well - being, and accountability.

3. **Deficiencies in existing research**:
- The paper finds that although there are some studies on evaluation criteria for individual TAI principles, few studies comprehensively cover all seven principles. Therefore, the paper attempts to fill this gap by comprehensively analyzing the existing literature, providing a classification of evaluation criteria for each principle, and discussing how to evaluate the trade - offs between multiple principles.

4. **Promoting the governance and development of trustworthy AI**:
- Ultimately, the paper hopes to help promote the governance and development of trustworthy AI by proposing a new classification system. This will not only help in formulating more effective evaluation criteria but also provide valuable references for policymakers, researchers, and practitioners to ensure the trustworthiness and ethical compliance of AI systems.

### Specific objectives

- **RQ1**: Determine the main initiatives for establishing the trustworthiness standards and best practices of AI systems.
- **RQ2**: Identify the current indicators or criteria used to evaluate the trustworthiness of AI systems.
- **RQ3**: Analyze whether there are differences in the current evaluation and scoring methods for each trustworthy AI principle.
- **RQ4**: Explore whether there are studies focusing on scoring all TAI principles.
- **RQ5**: Identify the challenges and problems that hinder the development of scoring and evaluation systems for trustworthy AI systems.

By answering these questions, the paper hopes to provide a comprehensive and systematic framework for the evaluation and governance of trustworthy AI, thereby promoting the safe, fair, and transparent application of AI technology.