Abstract:Abstract Objectives The aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans. Methods Abdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC). Results Twenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396–0.814) to 0.785 (95%CI 0.676–0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034–0.500) to 0.562 (95%CI 0.337–0.729). Intraobserver agreement increased for four out of six observers. Conclusion The PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement. Clinical relevance statement This study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms. Key Points • Subjective assessment of diagnostic image quality via Likert scale has limited reliability. • A pairwise comparison method improves the inter- and intraobserver agreement. • The pairwise comparison method is more reliable for CT optimization studies. Graphical Abstract

Ranking Consistent Rate: New Evaluation Criterion on Pairwise Subjective Experiments

A New Objective Assessment Method for Image Coding Quality

Pairwise comparison and rank learning for image quality assessment

Active Sampling Exploiting Reliable Informativeness for Subjective Image Quality Assessment Based on Pairwise Comparison

Quality Assessment for Comparing Image Enhancement Algorithms

Beyond MOS: Subjective Image Quality Score Preprocessing Method Based on Perceptual Similarity

A Perceptually Weighted Rank Correlation Indicator for Objective Image Quality Assessment

Predictive Sampling for Efficient Pairwise Subjective Image Quality Assessment

Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment

Multiscale contrast similarity deviation: An effective and efficient index for perceptual image quality assessment

Subjective Opinions Matter: Controllable Image Quality Assessment Using Pseudo Reference Images

Strategy for Boosting Pair Comparison and Improving Quality Assessment Accuracy

A Comprehensive Performance Evaluation of Image Quality Assessment Algorithms

Pairwise Comparisons Are All You Need

How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale

Comparison of Image Quality Models for Optimization of Image Processing Systems

Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare

Contrast and visual saliency similarity-induced index for assessing image quality

GMC-IQA: Exploiting Global-correlation and Mean-opinion Consistency for No-reference Image Quality Assessment

Comparison of Full-Reference Image Quality Models for Optimization of Image Processing Systems

Inconsistency indices in pairwise comparisons: an improvement of the Consistency Index