How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine

Haya R Rubin, Donald A Redelmeier, Albert W Wu, Earl P Steinberg
1993-05-01
Abstract:Objective: To evaluate the interrater reproducibility of scientific abstract review. Design: Retrospective analysis. Setting: Review for the 1991 Society of General Internal Medicine (SGIM) annual meeting. Subjects: 426 abstracts in seven topic categories evaluated by 55 reviewers. Measurements: Reviewers rated abstracts from 1 (poor) to 5 (excellent), globally and on three specific dimensions: interest to the SGIM audience, quality of methods, and quality of presentation. Each abstract was reviewed by five to seven reviewers. Each reviewer’s ratings of the three dimensions were added to compute that reviewer’ssummary score for a given abstract. The mean of all reviewers’ summary scores for an abstract, thefinal score, was used by SGIM to select abstracts for the meeting. Results: Final scores ranged from 4.6 to 136 (mean=9.9). Although 222 abstracts (5296) were …
What problem does this paper attempt to address?