ConceptPsy:A Benchmark Suite with Conceptual Comprehensiveness in Psychology

Junlei Zhang,Hongliang He,Nirui Song,Zhanchao Zhou,Shuyuan He,Shuai Zhang,Huachuan Qiu,Anqi Li,Yong Dai,Lizhi Ma,Zhenzhong Lan

2024-06-16

Abstract:The critical field of psychology necessitates a comprehensive benchmark to enhance the evaluation and development of domain-specific Large Language Models (LLMs). Existing MMLU-type benchmarks, such as C-EVAL and CMMLU, include psychology-related subjects, but their limited number of questions and lack of systematic concept sampling strategies mean they cannot cover the concepts required in psychology. Consequently, despite their broad subject coverage, these benchmarks lack the necessary depth in the psychology domain, making them inadequate as psychology-specific evaluation suite. To address this issue, this paper presents ConceptPsy, designed to evaluate Chinese complex reasoning and knowledge abilities in psychology. ConceptPsy includes 12 core subjects and 1383 manually collected concepts. Specifically, we prompt GPT-4 to generate questions for each concept using carefully designed diverse prompts and hire professional psychologists to review these questions. To help to understand the fine-grained performances and enhance the weaknesses, we annotate each question with a chapter label and provide chapter-wise accuracy. Based on ConceptPsy, we evaluate a broad range of LLMs. We observe that, although some LLMs achieve similar accuracies on overall performances, they exhibit significant performance variations across different psychology concepts, even when they are models from the same series. We hope our work can facilitate the development of LLMs in the field of psychology.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is the issue of insufficient conceptual coverage in the evaluation benchmarks of current large language models (LLMs) in the field of psychology. Existing Chinese evaluation benchmarks such as C-EVAL and CMMLU, although covering disciplines like psychology, have low conceptual coverage and a limited number of questions, leading to "conceptual bias" during the sampling process. This bias can result in skewed evaluation results of model performance, failing to fully reflect the actual capabilities of the models in the field of psychology.

To tackle this challenge, the authors propose ConceptPsy, a benchmark specifically designed to evaluate the complex reasoning and knowledge capabilities of Chinese LLMs in the field of psychology. ConceptPsy includes 12 core psychology subjects and 1383 concepts, which are derived from official exams. To generate high-quality questions, the authors used GPT-4 to generate questions based on each concept, which were then verified by psychology professionals. Additionally, each question is tagged with chapter labels to provide more detailed result analysis.

By evaluating a series of LLMs on ConceptPsy, the authors found that even within the same series of models, there are significant differences in performance across different chapters. ConceptPsy aims to promote the development of LLMs in the Chinese psychology field through comprehensive conceptual coverage and detailed analysis of strengths and weaknesses.