Abstract:Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{<a class="link-external link-https" href="https://github.com/THUDM/AlignBench" rel="external noopener nofollow">this https URL</a>}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab.

Experimental Investigation into Alignment-based Acoustic Confidence Measures in Keyword Verification for Mandarin Speech

Fintegrating Confidence Scores for Utterance Verification

Keyword Spotting Based on Hypothesis Boundary Realignment and State-Level Confidence Weighting

Effective Acoustic Modeling for Pronunciation Quality Scoring of Strongly Accented Mandarin Speech

Word Spotting Based on a Posterior Measure of Keyword Confidence

Improved Keyword Spotting System by Optimizing Posterior Confidence Measure Vector Using Feed-Forward Neural Network.

Keyword Spotting Based on Syllable Confusion Network.

Bayesian Estimation of Keyword Confidence in Chinese Continuous Speech Recognition

Keyword Spotting Based on Phoneme Confusion Matrix

An utterance verification algorithm in keyword spotting system

A New Keyword Spotting Approach for Spontaneous Mandarin Speech

A New Framework for Large Vocabulary Keyword Spotting Using Two-Pass Confidence Measure

LEXICAL ACCESS-BASED CONFIDENCE MEASURE FOR A SPANISH KEYWORD SPOTTING SYSTEM

Investigation of Frame Alignments for GMM-based Digit-prompted Speaker Verification.

Approach of N-Best Lists-Based Confidence Measure Using Mismatching Frame Proportion Weighting

AlignBench: Benchmarking Chinese Alignment of Large Language Models

Calibration of Word Posterior Estimation in Confusion Networks for Keyword Search

Confidence Measures and Integrating Algorithm in Utterance Verification

ACOUSTIC MODEL COMPARISONS FOR AN EMBEDDED PHONEME-BASED MANDARIN NAME DIALING SYSTEM

An Evolutionary Confidence Measure for Spotting Words in Speech Recognition

A Novel Discriminative Score Calibration Method for Keyword Search