Abstract:Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve.Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed.Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the inspections were based on a set of questions derived by brainstorming. The inspection using the Author perspective reviewed each section of the guidelines sequentially. Results. The question-based perspective inspections detected 42 issues where the guidelines would benefit from amendment or clarification and 8 defects.Conclusions. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. Software engineering researchers need to be cautious about adopting reporting guidelines that differ from those used by other disciplines. The current guidelines need to be revised and the revised guidelines need to be subjected to further theoretical and empirical validation. Perspective-based inspection is a useful validation method but the practitioner/consultant perspective presents difficulties.

Valuing Evaluation: Methodologies to Bridge Research and Practice

Qualitative software engineering research -- reflections and guidelines

How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological Guidance

A Methodology for Deriving Evaluation Criteria for Software Solutions

The state of user experience evaluation practice

Evaluating Guidelines for Reporting Empirical Software Engineering Studies

Practical relevance of software engineering research: Synthesizing the community's voice

Assessing the Effectiveness of Quantitative and Qualitative Methods for R&D Project Proposal Evaluations

Impact of Software Engineering Research in Practice: A Patent and Author Survey Analysis

How do we Evaluate Self-adaptive Software Systems?

Practitioner-generated blog posts as evidence for software engineering research: attitudinal survey and preliminary checklist

Empirical Research in Software Engineering - A Literature Survey.

Catching up with Method and Process Practice: An Industry-Informed Baseline for Researchers

Bridging the Research-Practice Gap in Requirements Engineering through Effective Teaching and Peer Learning

Software Engineering Practices for Scientific Software Development: A Systematic Mapping Study

Replicated Study of Effectiveness Evaluation of Cutting-Edge Software Engineering

The Role of Evaluation in Research—Practice Integration Working Toward the ‘‘Golden Spike’’

An Experience Report On Applying Software Testing Academic Results In Industry: We Need Usable Automated Test Generation

Towards a Framework for Normative Software Engineering Research.

Towards a Methodology for Participant Selection in Software Engineering Experiments. A Vision of the Future

Seeking Enlightenment: Incorporating Evidence-Based Practice Techniques in a Research Software Engineering Team