Abstract:Abstract Independent evaluations of grant applications by subject experts are an important part of the peer-review system. However, little is known about the real-time experiences of peer reviewers or experts who perform reviews of a grant application independently. This study sought to gain insight into this stage of the grant evaluation process by observing how experts conduct an independent review in near real time. Using the think aloud approach and Critical Decision Method of interviewing, in-depth interviews were conducted with 16 peer reviewers from a range of roles and disciplines within the medical humanities and social sciences. Participants were asked to think aloud while reviewing applications to different grant schemes from a single prestigious funder. The analysis shows reviewers encountered five dilemmas during the evaluation process. These dilemmas were related to whether or not one should (1) accept an invitation to review, (2) rely exclusively on the information presented in the application, (3) pay attention to institutional prestige, (4) offer comments about aspects that are not directly related to academics’ area of expertise, and (5) to take risks and overlook shortcomings rather than err on the side of caution. In order to decide on the appropriate course of action, reviewers often engaged in a series of deliberations and trade-offs—varying in length and complexity. However, their interpretation of what was ‘right’ was influenced by their values, preferences and experiences, but also by relevant norms and their understanding of the funder’s guidelines and priorities. As a result, the way reviewers approached the identified dilemmas was idiosyncratic and sometimes diametrically opposed to other reviewers’ views, which could lead to variation in peer-review outcomes. The dilemmas we have uncovered suggest that peer reviewers engage in thoughtful considerations during the peer-review process. We should, therefore, be wary of reducing the absence of consensus as resulting from biased, instinctive thinking. Rather, these findings highlight the diversity of values, priorities and habits and ways of working each reviewer brings to the fore when reviewing the applicants and their project proposals and call for further reflection on, and study of, this “invisible work” to better understand and continue to improve the peer-reviewing process.

Ethics review, reflective equilibrium and reflexivity

Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

Reflections on reviews

The role of research ethics committees: Friend or foe in educational research? An exploratory study

Ethical Approval and Being a Virtuous Social Work Researcher. The Experience of Multi-site Research in UK Health and Social Care: An Approved Mental Health Professional Case Study

Evaluating interventions to improve ethical decision making in clinical practice: a review of the literature and reflections on the challenges posed

Applying a framework for critical reflection in educational psychology practice: views of trainee educational psychologists

Research ethics committees and post-approval activities: a qualitative study on the perspectives of European research ethics committee representatives

Regulation (EU) 536/2014 and the role of ethics committees: a proposal for a review system model

Ethics review of big data research: What should stay and what should be reformed?

Behind the scenes of research ethics committee oversight: a qualitative research study with committee chairs in the Middle East and North Africa region

Justice in human research ethics. A conceptual and practical guide

Ranking Research Methodology by Risk — a cross-sectional study to determine the opinion of research ethics committee members

Ethics approval: responsibilities of journal editors, authors and research ethics committees

Psychology, ethics, and research ethics boards

The UK research ethics committee: Making the case for better serving the underserved – can we do better?

Reclaiming a theoretical orientation to reflection in medical education research: a critical narrative review

Post-approval process: A challenge for ethics committees

Does team reflexivity impact teamwork and communication in interprofessional hospital-based healthcare teams? A systematic review and narrative synthesis

Impact and Assessment of Research Integrity Teaching: A Systematic Literature Review

Process of risk assessment by research ethics committees: foundations, shortcomings and open questions