Abstract:The study provides a comprehensive review of OpenAI's Generative Pre-trained Transformer 4 (GPT-4) technical report, with an emphasis on applications in high-risk settings like healthcare. A diverse team, including experts in artificial intelligence (AI), natural language processing, public health, law, policy, social science, healthcare research, and bioethics, analyzed the report against established peer review guidelines. The GPT-4 report shows a significant commitment to transparent AI research, particularly in creating a systems card for risk assessment and mitigation. However, it reveals limitations such as restricted access to training data, inadequate confidence and uncertainty estimations, and concerns over privacy and intellectual property rights. Key strengths identified include the considerable time and economic investment in transparent AI research and the creation of a comprehensive systems card. On the other hand, the lack of clarity in training processes and data raises concerns about encoded biases and interests in GPT-4. The report also lacks confidence and uncertainty estimations, crucial in high-risk areas like healthcare, and fails to address potential privacy and intellectual property issues. Furthermore, this study emphasizes the need for diverse, global involvement in developing and evaluating large language models (LLMs) to ensure broad societal benefits and mitigate risks. The paper presents recommendations such as improving data transparency, developing accountability frameworks, establishing confidence standards for LLM outputs in high-risk settings, and enhancing industry research review processes. It concludes that while GPT-4's report is a step towards open discussions on LLMs, more extensive interdisciplinary reviews are essential for addressing bias, harm, and risk concerns, especially in high-risk domains. The review aims to expand the understanding of LLMs in general and highlights the need for new reflection forms on how LLMs are reviewed, the data required for effective evaluation, and addressing critical issues like bias and risk.

An Evaluation of GPT-4 on the ETHICS Dataset

Peer review of GPT-4 technical report and systems card

GPTEval: A Survey on Assessments of ChatGPT and GPT-4

Mind meets machine: Unravelling GPT-4's cognitive psychology

Exploring the potential utility of AI large language models for medical ethics: an expert panel evaluation of GPT-4

DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Is ETHICS about ethics? Evaluating the ETHICS benchmark

Evaluating ChatGPT's moral competence in health care-related ethical problems

The model student: GPT-4 performance on graduate biomedical science exams

Using cognitive psychology to understand GPT-3

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

GPT-4 Can't Reason

Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency

Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?

Is GPT-4 a Good Data Analyst?

An Early Evaluation of GPT-4V(ision)

GPT-4: A Review on Advancements and Opportunities in Natural Language Processing

GPT in Data Science: A Practical Exploration of Model Selection

GPT versus Humans: Uncovering Ethical Concerns in Conversational Generative AI-empowered Multi-Robot Systems

Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks