Quantitative Measurements of the Influence of Participant Roles during Peer Review Meetings

Patrick D'Astous,Pierre Robillard,Françoise Détienne,Willemien Visser

DOI: https://doi.org/10.48550/arXiv.cs/0612005

2007-03-02

Abstract:Peer review meetings (PRMs) are formal meetings during which peers systematically analyze artifacts to improve their quality and report on non-conformities. This paper presents an approach based on protocol analysis for quantifying the influence of participant roles during PRMs. Three views are used to characterize the seven defined participant roles. The project view defines three roles supervisor, procedure expert and developer. The meeting view defines two roles: author and reviewer, and the task view defines the roles reflecting direct and indirect interest in the artifact under review. The analysis, based on log-linear modeling, shows that review activities have different patterns, depending on their focus: form or content. The influence of each role is analyzed with respect to this focus. Interpretation of the quantitative data leads to the suggestion that PRMs could be improved by creating three different types of reviews, each of which collects together specific roles: form review, cognitive synchronization review and content review.

Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in Peer Review Meetings (PRMs), how to quantitatively measure the impact of different participant roles on the meeting process and effectiveness. Specifically, the author hopes to reveal how these roles affect the content and form of the meeting by studying the behavior patterns of different roles (such as project supervisors, procedure experts, developers, authors, and reviewers, etc.) in the meeting, and put forward improvement suggestions.

### Research Background
PRMs are a formal meeting form in the software development process, aiming to improve the quality of artifacts (such as design documents, code, etc.) through systematic analysis among peers and report non - compliant areas. Although existing research has shown that the roles of team members may affect the effectiveness of the meeting, there is a lack of quantitative data support. Therefore, this study attempts to fill this gap through quantitative analysis methods.

### Main Problems
1. **Quantification of Role Influence**: How to quantify the influence of different roles in PRMs?
2. **Relationship between Roles and Activities**: How do different roles behave differently in terms of discussion form and content?
3. **Improvement Suggestions**: Based on the results of quantitative analysis, how to optimize PRMs to improve efficiency?

### Methods
To answer these questions, the author adopted the following methods:
- **Protocol Analysis**: By recording and transcribing real PRMs, record each participant's speech (referred to as "move") and code it.
- **Multivariate Statistical Analysis**: Use Log - Linear Modeling (LLM) to analyze the relationships between different roles and their impact on meeting activities.

### Key Findings
- **Impact of Roles on Discussion Content**: The author found that different roles showed obvious differences in terms of discussion form and content. For example, authors are more active in discussing content but less involved in discussing form.
- **Three Types of Reviews**: Based on the analysis results, the author suggests dividing PRMs into three types:
- **Formal Review**: Mainly focuses on the form of the artifact, usually led by reviewers, and the discussion is short and direct.
- **Cognitive Synchronization Review**: Helps participants adjust their understanding of each other, with more information and assumption activities.
- **Content Review**: Discusses the content of the artifact in - depth to ensure that all participants reach a consensus.

### Conclusion
Through the quantitative analysis of PRMs, the author revealed the behavior patterns of different roles in the meeting and put forward specific suggestions for improving meeting efficiency. This provides a valuable reference for future research and practice.

### Example of Formula
In the Log - Linear Model, assume that we have three variables \(X\), \(Y\), and \(Z\), and the relationship between them can be expressed by the following formula:

\[ \log(\mu_{ijk})=\lambda+\lambda_{i}^{X}+\lambda_{j}^{Y}+\lambda_{k}^{Z}+\lambda_{ij}^{XY}+\lambda_{ik}^{XZ}+\lambda_{jk}^{YZ}+\lambda_{ijk}^{XYZ} \]

where \(\mu_{ijk}\) represents the observed frequency, \(\lambda\) is a constant term, and other terms represent the effects of each variable and its interactions.

Through this model, we can better understand the influence of different roles in PRMs and provide data support for optimizing the meeting process.