Why scientists trust AI too much — and what to do about it

DOI: https://doi.org/10.1038/d41586-024-00639-y

IF: 64.8

2024-03-07

Nature

Abstract:Some researchers see superhuman qualities in artificial intelligence. All scientists need to be alert to the risks this creates. Some researchers see superhuman qualities in artificial intelligence. All scientists need to be alert to the risks this creates.

multidisciplinary sciences

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the multiple risks that scientists may face when using artificial intelligence (AI) tools. Specifically, through analyzing approximately 100 peer - reviewed papers, pre - prints, conference proceedings, and books, the authors Messeri and Crockett identified four "Visions" that scientists hold regarding AI tools and explored the risks that these visions may lead to:

1. **AI as Oracle**: This vision holds that AI tools can tirelessly read and digest scientific literature and survey scientific literature more comprehensively than humans. However, this may lead researchers to selectively cite literature to support their hypotheses or show partiality in peer review.

2. **AI as Arbiter**: In this vision, AI systems are considered to be able to evaluate scientific findings more objectively than humans because they are not influenced by personal emotions or biases. But this view may lead to a false sense of objectivity, that is, researchers think that AI systems represent all possible viewpoints or no viewpoints. In fact, these tools can only reflect the viewpoints in their training data and may inherit the biases in the data.

3. **AI as Quant**: Here it refers to the fact that AI tools seem to transcend the limitations of human thinking in analyzing large and complex data sets. However, this ability may lead research to be biased towards questions that can be tested by AI systems, while ignoring other important research areas.

4. **AI as Surrogate**: In this case, AI tools simulate data that is difficult or complex to obtain. For example, in the social sciences, this vision may encourage conducting human behavior experiments that can be simulated by AI, while ignoring those behavior studies that require physical manifestation.

The paper also pointed out several specific pitfalls, including:
- **The illusion of depth of explanation**: People tend to mistakenly think that the knowledge obtained through algorithms is their own knowledge, thus overestimating their depth of understanding.
- **The illusion of breadth of exploration**: Research may be biased towards areas that can be tested by AI systems, ignoring other important but unprocessable - by - AI research directions.
- **The illusion of objectivity**: Researchers think that AI systems are completely objective, forgetting that AI tools can only reflect the viewpoints in their training data and may carry biases in the data.

To reduce these risks, the paper suggests that scientists adopt some strategies when planning to use AI, such as mapping the proposed AI uses to one of the above visions and considering the most likely pitfalls; and making conscious decisions when using AI, avoiding using it to provide expertise that the team itself does not possess. At the same time, journal editors, funders, and research institutions should also consider these risks and ensure the diversity and balance of research.

In conclusion, this paper aims to remind the scientific community that although AI tools have great potential, they must be used with caution and their limitations and potential risks must be fully recognized.