Mining Software Repository:A Survey

BAI Jie,LI Chun-ping

DOI: https://doi.org/10.3969/j.issn.1001-3695.2008.01.005

2008-01-01

Abstract:This paper summarized the research on software repository with data mining techniques in recent years,including the main research areas and methods,and furthermore discussed the future work.

What problem does this paper attempt to address?

Software intelligence: the future of mining software engineering data.

Ahmed E. Hassan,Tao Xie

DOI: https://doi.org/10.1145/1882362.1882397

2010-01-01

Abstract:ABSTRACTSoftware engineering data (such as code bases, execution traces, historical code changes, mailing lists, and bug databases) contains a wealth of information about a project's status, progress, and evolution. Using well-established data mining techniques, practitioners and researchers have started exploring the potential of this valuable data in order to better manage their projects and to produce higher quality software systems that are delivered on time and within budget. This tutorial presents the latest research in mining software engineering data, discusses challenges associated with mining software engineering data, highlights success stories of mining software engineering data, and outlines future research directions. Attendees will acquire the knowledge and skills needed to integrate the mining of software engineering data in their own research or practice. This tutorial builds on several successful offerings at ICSE since 2007.
Mining Software Engineering Data

Tao Xie,Jian Pei,Ahmed E. Hassan

DOI: https://doi.org/10.1145/1810295.1810451

2007-01-01

Abstract:Software engineering data (such as code bases, exe- cution traces, historical code changes, mailing lists, and bug databases) contains a wealth of information about a projectÂ¿s status, progress, and evolution. Using well- established data mining techniques, practitioners and re- searchers can explore the potential of this valuable data in order to better manage their projects and to produce higher-quality software systems that are delivered on time and within budget. This tutorial presents the latest research in mining Soft- ware Engineering (SE) data, discusses challenges associ- ated with mining SE data, highlights SE data mining suc- cess stories, and outlines future research directions. Partic- ipants will acquire knowledge and skills needed to perform research or conduct practice in the field and to integrate data mining techniques in their own research or practice.
Research on Web Mining Technique Facing Electronic Business and Application

DONG De-min,HE Qin-ming

DOI: https://doi.org/10.3969/j.issn.1000-7024.2006.01.029

2006-01-01

Abstract:Enterprise and organize face a series of problems to capture information of web.Technique of solving these problems were summed up including web content mining,web structure mining and web usage mining.And some application of web mining was intro-duced,such as intelligent search engine,enterprise crisis management,client relation management and personalization service system.
eLORM: learning object relationship mining-based repository

Ouyang Yang,Miaoliang Zhu

DOI: https://doi.org/10.1108/14684520810879863

2008-01-01

Online Information Review

Abstract:Purpose - This paper aims to explore the feasibility of using web-mining technology on learning object (LO) usage information to discover the LO relation pattern and provide valuable recommendations on related learning resources. Design/methodology/approach - This paper proposes three kinds of learning object relation patterns and gives a specific definition of each pattern based on analysing the learners' usage data stored in the learning object repository. These relation patterns can be used to make effective recommendations to learners. Findings - LO usage data indicate the potential relation patterns between LOs. By using web-mining technology on the usage data, it is possible to discover valuable relation patterns. Originality/value - The authors propose a set of LO relation patterns and indicate how they are closely related to users' learning behaviour.
Review of Defect Analysis-oriented Software Repositories Mining Methods

LIU Ying-Bo,WANG Jian-Min

DOI: https://doi.org/10.3969/j.issn.1002-137X.2007.09.001

2007-01-01

Computer Science

Abstract:Defect analysis is an important topic in the field of software engineering.The history information of software development(like defect records,different version of source code etc.)have provided valuable empirical data for this topic.How to utilize this data to better analysis the defects of software is a challenge faced by the research of Mining Software Repositories.This review introduces two approaches of using development history information to analysis software defects.The first approach is statistical defect analysis and the other approach is program analysis based de- fect analysis.
Data Mining in Software Maintenance

SU Shao-Yong,PAN Jin-Gui

2005-01-01

Computer Science

Abstract:Software maintenance is one of the most important and difficulty stages in software process. Fortunately, the data mining technology has greatly improved the situation of software maintenance. This paper introduces the history of the application of data mining in software maintenance with three aspects: theapplicability of the technology, the typical steps and the major application fields; and based on it we study the key issues. This paper concludes with the directions for future work.
Software Artifact Mining in Software Engineering Conferences: A Meta-Analysis

Zeinab Abou Khalil,Stefano Zacchiroli

DOI: https://doi.org/10.1145/3544902.3546239

2022-07-18

Abstract:Background: Software development results in the production of various types of artifacts: source code, version control system metadata, bug reports, mailing list conversations, test data, etc. Empirical software engineering (ESE) has thrived mining those artifacts to uncover the inner workings of software development and improve its practices. But which artifacts are studied in the field is a moving target, which we study empirically in this <a class="link-external link-http" href="http://paper.Aims" rel="external noopener nofollow">this http URL</a>: We quantitatively characterize the most frequently mined and co-mined software artifacts in ESE research and the research purposes they <a class="link-external link-http" href="http://support.Method" rel="external noopener nofollow">this http URL</a>: We conduct a meta-analysis of artifact mining studies published in 11 top conferences in ESE, for a total of 9621 papers. We use natural language processing (NLP) techniques to characterize the types of software artifacts that are most often mined and their evolution over a 16-year period (2004-2020). We analyze the combinations of artifact types that are most often mined together, as well as the relationship between study purposes and mined <a class="link-external link-http" href="http://artifacts.Results" rel="external noopener nofollow">this http URL</a>: We find that: (1) mining happens in the vast majority of analyzed papers, (2) source code and test data are the most mined artifacts, (3) there is an increasing interest in mining novel artifacts, together with source code, (4) researchers are most interested in the evaluation of software systems and use all possible empirical signals to support that goal.

Software Engineering
Mining Extremely Small Data Sets with Application to Software Reuse

Yuan Jiang,Ming Li,Zhi-Hua Zhou

DOI: https://doi.org/10.1002/spe.905

2008-01-01

Software Practice and Experience

Abstract:A serious problem encountered by machine learning and data mining techniques in software engineering is the lack of sufficient data. For example, there are only 24 examples in the current largest data set on software reuse. In this paper, a recently proposed machine learning algorithm is modified for mining extremely small data sets. This algorithm works in a twice-learning style. In detail, a random forest is trained from the original data set at first. Then, virtual examples are generated from the random forest and used to train a single decision tree. In contrast to the numerous discrepancies between the empirical data and expert opinions reported by previous research, our mining practice shows that the empirical data are actually consistent with expert opinions. Copyright © 2008 John Wiley & Sons, Ltd.
Empirical Standards for Repository Mining

Preetha Chatterjee,Tushar Sharma,Paul Ralph

DOI: https://doi.org/10.1145/3524842.3528032

2022-03-30

Abstract:The purpose of scholarly peer review is to evaluate the quality of scientific manuscripts. However, study after study demonstrates that peer review neither effectively nor reliably assesses research quality. Empirical standards attempt to address this problem by modelling a scientific community's expectations for each kind of empirical study conducted in that community. This should enhance not only the quality of research but also the reliability and predictability of peer review, as scientists adopt the standards in both their researcher and reviewer roles. However, these improvements depend on the quality and adoption of the standards. This tutorial will therefore present the empirical standard for mining software repositories, both to communicate its contents and to get feedback from the attendees. The tutorial will be organized into three parts: (1) brief overview of the empirical standards project; (2) detailed presentation of the repository mining standard; (3) discussion and suggestions for improvement.

Digital Libraries,Software Engineering
Data Mining for Software Engineering

Tao Xie,Suresh Thummalapenta,David Lo,Chao Liu

DOI: https://doi.org/10.1109/MC.2009.256

2009-01-01

Computer

Abstract:To improve software productivity and quality, software engineers are increasingly applying data mining algorithms to various software engineering tasks. However, mining SE data poses several challenges. The authors present various algorithms to effectively mine sequences, graphs, and text from such data.
Thirteen Years of Mining Software Repositories (MSR) Conference - What is the Bibliography Data Telling Us?

Lov Kumar,Ashish Sureka

DOI: https://doi.org/10.48550/arXiv.1609.06266

2016-09-20

Software Engineering

Abstract:The Mining Software Repositories (MSR) conference is a reputed, long-running and flagship conference in the area of Software Analytics which has successfully completed more than one decade as of year 2016. We conduct a bibliometric and scientific publication mining based study to study how the conference has evolved over the recent past 13 years (from 2004 to 2007 as a workshop and then from 2008 to 2016 as a conference). Our objective is to perform an examination of the state of MSR so that the MSR community can identify strengths, areas of improvements and future directions for the conference.
Finding Trends in Software Research

George Mathew,Amritanshu Agrawal,Tim Menzies

DOI: https://doi.org/10.1109/tse.2018.2870388

IF: 7.4

2019-01-01

IEEE Transactions on Software Engineering

Abstract:Text mining methods can find large scale trends within research communities. For example, using stable Latent Dirichlet Allocation (a topic modeling algorithm) this study found 10 major topics in 35,391 SE research papers from 34 leading SE venues over the last 25 years (divided, evenly, between conferences and journals). Out study also shows how those topics have changed over recent years. Also, we note that (in the historical record) mono-focusing on a single topic can lead to fewer citations than otherwise. Further, while we find no overall gender bias in SE authorship, we note that women are under-represented in the top-most cited papers in our field. Lastly, we show a previously unreported dichotomy between software conferences and journals (so research topics that succeed at conferences might not succeed at journals, and vice versa). An important aspect of this work is that it is automatic and quickly repeatable (unlike prior SE bibliometric studies that used tediously slow and labor intensive methods). Automation is important since, like any data mining study, its conclusions are skewed by the data used in the analysis. The automatic methods of this paper make it far easier for other researchers to re-apply the analysis to new data, or if they want to use different modeling assumptions.

engineering, electrical & electronic,computer science, software engineering
Mining architectural information: A systematic mapping study

Musengamana Jean de Dieu,Peng Liang,Mojtaba Shahin,Chen Yang,Zengyang Li

DOI: https://doi.org/10.1007/s10664-024-10480-6

IF: 3.762

2024-06-06

Empirical Software Engineering

Abstract:Mining Software Repositories (MSR) has become an essential activity in software development. Mining architectural information (e.g., architectural models) to support architecting activities, such as architecture understanding, has received significant attention in recent years. However, there is a lack of clarity on what literature on mining architectural information is available. Consequently, this may create difficulty for practitioners to understand and adopt the state-of-the-art research results, such as what approaches should be adopted to mine what architectural information in order to support architecting activities. It also hinders researchers from being aware of the challenges and remedies for the identified research gaps. We aim to identify, analyze, and synthesize the literature on mining architectural information in software repositories in terms of architectural information and sources mined, architecting activities supported, approaches and tools used, and challenges faced. A Systematic Mapping Study (SMS) has been conducted on the literature published between January 2006 and December 2022. Of the 104 primary studies finally selected, 7 categories of architectural information have been mined, among which architectural description is the most mined architectural information; 11 categories of sources have been leveraged for mining architectural information, among which version control system (e.g., GitHub) is the most popular source; 11 architecting activities can be supported by the mined architectural information, among which architecture understanding is the most supported activity; 95 approaches and 56 tools were proposed and employed in mining architectural information; and 4 types of challenges in mining architectural information were identified. This SMS provides researchers with promising future directions and help practitioners be aware of what approaches and tools can be used to mine what architectural information from what sources to support various architecting activities.

computer science, software engineering
A Study on Mining Bibliographic Records by Designed Software SATI:Case Study on Library and Information Science

Liu Qiyuan,Ye Ying

DOI: https://doi.org/10.13365/j.jirm.2012.01.012

2012-01-01

Abstract:A bibliographic information analysis software named SATI(Statistical Analysis Toolkit for Informetrics)is developed using C#based on Microsoft.NET platform.The national data fitting EndNote,NoteExpress and NoteFirst can be imported into SATI as well as international data as HTML(output from WoS).For the purpose of getting the clustering graph,multidimensional scaling map,network knowledge map,and strategic diagram,four basic functions including transforming raw data into XML,extracting selected elements,counting terms frequency and building knowledge units matrices have been implemented.Taking 17440articles published in ten core Library and Information Science journals both at home and abroad during 2006to 2010as the sample,this paper revealed three potential research fields of LIS research area based on the consistency between clustering analysis and multidimensional scaling analysis results,and figured out relations and features of subject areas by interpreting the network knowledge map and strategic diagram.
A Mining Approach to Obtain the Software Vulnerability Characteristics

Xiang Li,Jinfu Chen,Zhechao Lin,Lin Zhang,Zibin Wang,Minmin Zhou,Wanggen Xie

DOI: https://doi.org/10.1109/cbd.2017.58

2017-08-01

Abstract:Software vulnerability remains a serious challenge to the software engineering domain over the past decade as a result of the recent technological advancement in information systems. The rapid development in software applications and failure on the part of system developers to properly analyze program codes before been released to the market increases the chance for data breaches. It is a known fact that most system failures are as a result of bugs and errors detected in software applications. Although code errors significantly affect software quality, there is no effective method that can be used in eliminating software errors to improve its reliability. Data Mining and its related algorithms are an active area which can successfully be applied in analyzing software vulnerability. However the concept of applying data mining techniques has not been empirically proven as an effective method for obtaining the essential characteristics of software vulnerability. To investigate this effect, we propose a vulnerability mining algorithm to analyze and obtain the essential characteristics of Software vulnerability based data mining techniques. We first extracted and preprocessed the software vulnerabilities using data mining techniques and common vulnerability database. We evaluate the proposed technique using the Common Vulnerability and Exposure (CVE) Database, Common Weakness Enumeration (CWE) Database, National Vulnerability Database (NVD) datasets. Empirical results show that the proposed vulnerability mining algorithm has a remarkable improvement in the vulnerability mining process. The most interesting finding is that, we observed that across all the three projects, recall was around 70% and precision was approximately 60%.
A Mining Software Repository Extended Cookbook: Lessons learned from a literature review

Daniel Barros,Flavio Horita,Igor Wiese,Kanan Silva

DOI: https://doi.org/10.48550/arXiv.2110.04095

2021-10-08

Abstract:The main purpose of Mining Software Repositories (MSR) is to discover the latest enhancements and provide an insight into how to make improvements in a software project. In light of it, this paper updates the MSR findings of the original MSR Cookbook, by first conducting a systematic mapping study to elicit and analyze the state-of-the-art, and then proposing an extended version of the Cookbook. This extended Cookbook was built on four high-level themes, which were derived from the analysis of a list of 112 selected studies. Hence, it was used to consolidate the extended Cookbook as a contribution to practice and research in the following areas by: 1) including studies published in all available and relevant publication venues; 2) including and updating recommendations in all four high-level themes, with an increase of 84% in comments in this study when compared with the original MSR Cookbook; 3) summarizing the tools employed for each high-level theme; and 4) providing lessons learned for future studies. Thus, the extended Cookbook examined in this work can support new research projects, as upgraded recommendations and the lessons learned are available with the aid of samples and tools.

Software Engineering,Artificial Intelligence
Application of Data Mining Technology in Software Engineering

Xiaobin Hong

DOI: https://doi.org/10.1088/1742-6596/2066/1/012013

2021-11-01

Journal of Physics: Conference Series

Abstract:Abstract With the rapid development of informatization, computer database software systems have entered various fields of society, which has brought about the explosive growth of industry data. Faced with massive amounts of data, computers with limited storage capacity have to abandon some outdated data, and the application of various data mining technologies related to it has gradually matured. The purpose of this article is to discuss the application research of data mining technology in software engineering. This article analyzes the correlation analysis of a large number of bug repair source code update data and bug defect reports in the version control system SVN and the defect tracking system Bugzilla in the software engineering project development process, and tries to classify the bug report by data mining technology: defect changes and potential defects change. Starting from large-scale software engineering projects, apply data mining technology to the huge software engineeri ng knowledge base. Especially the software development and maintenance are explained, as well as the more challenging problems in the future. This paper uses data mining technology to study the dependency of the source code files of each module of the software system, and helps software developers quickly understand the software architecture by understanding the interrelationships between the modules, and provides suggestions for modification paths. Experimental research shows that this paper compares with F-measure and concludes that FL-M-GSpan algorithm is better than TS-M-GSpan algorithm. At the same time, it is found that the FL-M-GSpan algorithm always has a better accuracy rate close to 95%, while the TS-M-GSpan algorithm always has a better recall rate.
Empirical Study on the Software Engineering Practices in Open Source ML Package Repositories

Minke Xiu,Ellis E. Eghan,Zhen Ming,Jiang,Bram Adams

DOI: https://doi.org/10.48550/arXiv.2012.01403

2020-12-02

Software Engineering

Abstract:Recent advances in Artificial Intelligence (AI), especially in Machine Learning (ML), have introduced various practical applications (e.g., virtual personal assistants and autonomous cars) that enhance the experience of everyday users. However, modern ML technologies like Deep Learning require considerable technical expertise and resources to develop, train and deploy such models, making effective reuse of the ML models a necessity. Such discovery and reuse by practitioners and researchers are being addressed by public ML package repositories, which bundle up pre-trained models into packages for publication. Since such repositories are a recent phenomenon, there is no empirical data on their current state and challenges. Hence, this paper conducts an exploratory study that analyzes the structure and contents of two popular ML package repositories, TFHub and PyTorch Hub, comparing their information elements (features and policies), package organization, package manager functionalities and usage contexts against popular software package repositories (npm, PyPI, and CRAN). Through these studies, we have identified unique SE practices and challenges for sharing ML packages. These findings and implications would be useful for data scientists, researchers and software developers who intend to use these shared ML packages.

Mining Software Repository:A Survey

Software intelligence: the future of mining software engineering data.

Mining Software Engineering Data

Research on Web Mining Technique Facing Electronic Business and Application

eLORM: learning object relationship mining-based repository

Review of Defect Analysis-oriented Software Repositories Mining Methods

Data Mining in Software Maintenance

Software Artifact Mining in Software Engineering Conferences: A Meta-Analysis

Mining Extremely Small Data Sets with Application to Software Reuse

Empirical Standards for Repository Mining

Data Mining for Software Engineering

Thirteen Years of Mining Software Repositories (MSR) Conference - What is the Bibliography Data Telling Us?

Finding Trends in Software Research

Mining architectural information: A systematic mapping study

A Study on Mining Bibliographic Records by Designed Software SATI:Case Study on Library and Information Science

A Mining Approach to Obtain the Software Vulnerability Characteristics

A Mining Software Repository Extended Cookbook: Lessons learned from a literature review

Application of Data Mining Technology in Software Engineering

Empirical Study on the Software Engineering Practices in Open Source ML Package Repositories