AlignBench: Benchmarking Chinese Alignment of Large Language Models
Xiao Liu,Xuanyu Lei,Shengyuan Wang,Yue Huang,Zhuoer Feng,Bosi Wen,Jiale Cheng,Pei Ke,Yifan Xu,Weng Lam Tam,Xiaohan Zhang,Lichao Sun,Xiaotao Gu,Hongning Wang,Jing Zhang,Minlie Huang,Yuxiao Dong,Jie Tang
2024-08-25
Abstract:Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{<a class="link-external link-https" href="https://github.com/THUDM/AlignBench" rel="external noopener nofollow">this https URL</a>}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab.
Computation and Language,Artificial Intelligence,Machine Learning