Abstract:Continuous integration at scale is costly but essential to software development. Various test optimization techniques including test selection and prioritization aim to reduce the cost. Test batching is an effective alternative, but overlooked technique. This study evaluates parallelization's effect by adjusting machine count for test batching and introduces two novel approaches. We establish TestAll as a baseline to study the impact of parallelism and machine count on feedback time. We re-evaluate ConstantBatching and introduce DynamicBatching, which adapts batch size based on the remaining changes in the queue. We also propose TestCaseBatching, enabling new builds to join a batch before full test execution, thus speeding up continuous integration. Our evaluations utilize Ericsson's results and 276 million test outcomes from open-source Chrome, assessing feedback time, execution reduction, and providing access to Chrome project scripts and data. The results reveal a non-linear impact of test parallelization on feedback time, as each test delay compounds across the entire test queue. ConstantBatching, with a batch size of 4, utilizes up to 72% fewer machines to maintain the actual average feedback time and provides a constant execution reduction of up to 75%. Similarly, DynamicBatching maintains the actual average feedback time with up to 91% fewer machines and exhibits variable execution reduction of up to 99%. TestCaseBatching holds the line of the actual average feedback time with up to 81% fewer machines and demonstrates variable execution reduction of up to 67%. We recommend practitioners use DynamicBatching and TestCaseBatching to reduce the required testing machines efficiently. Analyzing historical data to find the threshold where adding more machines has minimal impact on feedback time is also crucial for resource-effective testing.

Creating a Virtuous Cycle in Performance Testing at MongoDB

Automated System Performance Testing at MongoDB

Schema-Driven Performance Evaluation for Highly Concurrent Scenarios.

Distance Based Root Cause Analysis and Change Impact Analysis of Performance Regressions

What Makes a Real Change in Software Performance? an Empirical Study on Analyzing the Factors That Affect the Triagement of Performance Change Points

Change Point Detection in Software Performance Testing

High-Volume Performance Test Framework using Big Data

Improving the Performance of MongoDB with RDMA.

Analyzing Performance-Aware Code Changes in Software Development Process

Embedding Performance Testing in Agile Software Model

First Past the Post: Evaluating Query Optimization in MongoDB

Enhancement of database access performance by improving data consistency in a non-relational database system (NoSQL)

DevOps Improvements for Reduced Cycle Times with Integrated Test Optimizations for Continuous Integration

Improving Development Practices through Experimentation: An Industrial TDD Case

Lessons learned from the early performance evaluation of Intel Optane DC Persistent Memory in DBMS

Accelerating Continuous Integration with Parallel Batch Testing

Uncovering the Benefits and Challenges of Continuous Integration Practices

From Relational Databases to NoSQL Databases: Performance Evaluation

Does the performance of TDD hold across software companies and premises? A group of industrial experiments on TDD

Deep variability modeling to enhance reproducibility of database performance testing

Contrasting test selection, prioritization, and batch testing at scale