Abstract:Software Engineering activities are information intensive. Research proposes Information Retrieval (IR) techniques to support engineers in their daily tasks, such as establishing and maintaining traceability links, fault identification, and software maintenance. We describe an engineering task, test case selection, and illustrate our problem analysis and solution discovery process. The objective of the study is to gain an understanding of to what extent IR techniques (one potential solution) can be applied to test case selection and provide decision support in a large-scale, industrial setting. We analyze, in the context of the studied company, how test case selection is performed and design a series of experiments evaluating the performance of different IR techniques. Each experiment provides lessons learned from implementation, execution, and results, feeding to its successor. The three experiments led to the following observations: 1) there is a lack of research on scalable parameter optimization of IR techniques for software engineering problems; 2) scaling IR techniques to industry data is challenging, in particular for latent semantic analysis; 3) the IR context poses constraints on the empirical evaluation of IR techniques, requiring more research on developing valid statistical approaches. We believe that our experiences in conducting a series of IR experiments with industry grade data are valuable for peer researchers so that they can avoid the pitfalls that we have encountered. Furthermore, we identified challenges that need to be addressed in order to bridge the gap between laboratory IR experiments and real applications of IR in the industry.

Evaluation of information retrieval for E-discovery

Case law retrieval: problems, methods, challenges and evaluations in the last 20 years

High Recall, Small Data: The Challenges of Within-System Evaluation in a Live Legal Search System

Evaluation of Diversification Techniques for Legal Information Retrieval

Large-scale information retrieval in software engineering -- an experience report from industrial application

Investigating User Behavior in Legal Case Retrieval

Cheap IR Evaluation: Fewer Topics, No Relevance Judgements, and Crowdsourced Assessments

Using Context to Improve the Evaluation of Information Retrieval Systems

Information Retrieval beyond the Text Document

Learning from Litigation: Graphs and LLMs for Retrieval and Reasoning in eDiscovery

An Empirical Study on Query Expansion and Documentexpans Ion in Information Retrieval.

An overview of semantic search evaluation initiatives

Conversational Vs Traditional: Comparing Search Behavior and Outcome in Legal Case Retrieval

Legal information retrieval for understanding statutory terms

Performance Evaluation in Multimedia Retrieval

Evaluate Retrieval Systems Based on Ontology Vocabulary.

Evaluation of Temporal Change in IR Test Collections

Sequential/Spatial, a Survey of Interactive Information Retrieval Methods for Controlled Experimentation and Evaluation

Understanding Relevance Judgments in Legal Case Retrieval

Investigating the Influence of Legal Case Retrieval Systems on Users' Decision Process

Active Sampling for Large-scale Information Retrieval Evaluation