Abstract:Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve.Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed.Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the inspections were based on a set of questions derived by brainstorming. The inspection using the Author perspective reviewed each section of the guidelines sequentially. Results. The question-based perspective inspections detected 42 issues where the guidelines would benefit from amendment or clarification and 8 defects.Conclusions. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. Software engineering researchers need to be cautious about adopting reporting guidelines that differ from those used by other disciplines. The current guidelines need to be revised and the revised guidelines need to be subjected to further theoretical and empirical validation. Perspective-based inspection is a useful validation method but the practitioner/consultant perspective presents difficulties.

Analyzing a practitioner perspective on relevance of published empirical research in Requirements Engineering

How do Practitioners Perceive the Relevance of Requirements Engineering Research? An Ongoing Study

The Role of Requirements Engineering Practices in Agile Development: An Empirical Study

Views on Quality Requirements in Academia and Practice: Commonalities, Differences, and Context-Dependent Grey Areas

Practical relevance of software engineering research: Synthesizing the community's voice

Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys

Relevance Versus Rigor in IS Research: Need to Introduce A New Institution

A Systematic Literature Review of Empirical Research on Quality Requirements

Rigour versus relevance for purchasing trends and processes : An analysis of how research trends and business needs are in line

A systematic literature review of requirements engineering education

How Mature is Requirements Engineering for AI-based Systems? A Systematic Mapping Study on Practices, Challenges, and Future Research Directions

Naming the Pain in Requirements Engineering: Design of a Global Family of Surveys and First Results from Germany

Requirements Engineering Practice and Problems in Agile Projects: Results from an International Survey

What Industry Wants from Requirements Engineers in China?

Between Sweet Harmony and a Clash of Cultures: Does a Joint Academic–Practitioner Review Reconcile Rigor and Relevance?

Ethnographic field work in requirements engineering

Evaluating Guidelines for Reporting Empirical Software Engineering Studies

A Blueprint of IR Evaluation Integrating Task and User Characteristics: Test Collection and Evaluation Metrics

Status Quo and Problems of Requirements Engineering for Machine Learning: Results from an International Survey

Identifying relevant Factors of Requirements Quality: an industrial Case Study