An Analysis of Research in Software Engineering: Assessment and Trends

Zhi Wang,Bing Li,Yutao Ma

DOI: https://doi.org/10.48550/arXiv.1407.4903

2014-07-18

Abstract:Glass published the first report on the assessment of systems and software engineering scholars and institutions two decades ago. The ongoing, annual survey of publications in this field provides fund managers, young scholars, graduate students, etc. with useful information for different purposes. However, the studies have been questioned by some critics because of a few shortcomings of the evaluation method. It is actually very hard to reach a widely recognized consensus on such an assessment of scholars and institutions. This paper presents a module and automated method for assessment and trends analysis in software engineering compared with the prior studies. To achieve a more reasonable evaluation result, we take into consideration more high-quality publications, the rank of each publication analyzed, and the different roles of authors named on each paper in question. According to the 7638 papers published in 36 publications from 2008 to 2013, the statistics of research subjects roughly follow power laws, implying the interesting Matthew Effect. We then identify the Top 20 scholars, institutions and countries or regions in terms of a new evaluation rule based on the frequently-used one. The top-ranked scholar is Mark Harman of the University College London, UK, the top-ranked institution is the University of California, USA, and the top-ranked country is the USA. Besides, we also show two levels of trend changes based on the EI classification system and user-defined uncontrolled keywords, as well as noteworthy scholars and institutions in a specific research area. We believe that our results would provide a valuable insight for young scholars and graduate students to seek possible potential collaborators and grasp the popular research topics in software engineering.

Software Engineering

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are several deficiencies in the existing evaluation methods for software engineering research, specifically including:

1. **Small sample size**: Previous studies usually selected at most seven journals as survey objects, resulting in a small sample size (that is, the number of papers published in these journals), which may cause the results to be biased towards certain specific journals or fields and fail to comprehensively reflect the research status of the entire software engineering field.

2. **Subjectivity of keyword analysis**: The keywords used in previous studies were collected from the top 15 scholars to describe their research focuses. This practice may lead to the selection of keywords being subjective and biased, and cannot reasonably reflect the trends and hot topics in the software engineering field.

3. **Imperfect scoring scheme**: When evaluating leading scholars and institutions, previous studies used a certain evaluation rule, but this method ignores the leading roles of a few scholars in papers with multiple authors.

To overcome the above problems, this paper proposes a new evaluation method and trend analysis method, aiming to provide more reasonable evaluation results by considering more high - quality publications, the rankings of each publication, and the different roles of authors in each paper. Specifically, the goals of this paper include:

- Conduct a new evaluation of scholars, institutions, and countries (or regions) in the software engineering field from 2008 to 2013.
- Investigate the research trends in the software engineering field during this period.
- Propose a more reasonable and general evaluation and trend analysis method that can handle more publication data than before.
- Improve the efficiency and accuracy of data processing through automated software programs.

In addition, this paper also finds that the distributions of scholars, institutions, countries (or regions), and keywords roughly follow the power - law distribution, which indicates that the Matthew effect also exists in software engineering research, further highlighting the importance of evaluating top scholars and institutions and trend analysis.