Abstract:Software testing activities scrutinize the artifacts and the behavior of a software product to find possible defects and ensure that the product meets its expected requirements. Recently, Deep Reinforcement Learning (DRL) has been successfully employed in complex testing tasks such as game testing, regression testing, and test case prioritization to automate the process and provide continuous adaptation. Practitioners can employ DRL by implementing from scratch a DRL algorithm or using a DRL framework. DRL frameworks offer well-maintained implemented state-of-the-art DRL algorithms to facilitate and speed up the development of DRL applications. Developers have widely used these frameworks to solve problems in various domains including software testing. However, to the best of our knowledge, there is no study that empirically evaluates the effectiveness and performance of implemented algorithms in DRL frameworks. Moreover, some guidelines are lacking from the literature that would help practitioners choose one DRL framework over another. In this paper, we empirically investigate the applications of carefully selected DRL algorithms on two important software testing tasks: test case prioritization in the context of Continuous Integration (CI) and game testing. For the game testing task, we conduct experiments on a simple game and use DRL algorithms to explore the game to detect bugs. Results show that some of the selected DRL frameworks such as Tensorforce outperform recent approaches in the literature. To prioritize test cases, we run experiments on a CI environment where DRL algorithms from different frameworks are used to rank the test cases. Our results show that the performance difference between implemented algorithms in some cases is considerable, motivating further investigation.

BDD-Based Framework with RL Integration: An approach for videogames automated testing

Augmenting Automated Game Testing with Deep Reinforcement Learning

A Comparison of Reinforcement Learning Frameworks for Software Testing Tasks

Using Reinforcement Learning for Load Testing of Video Games

Robust Imitation Learning for Automated Game Testing

Automated Play-Testing Through RL Based Human-Like Play-Styles Generation

Technical Challenges of Deploying Reinforcement Learning Agents for Game Testing in AAA Games

DRIFT: Deep Reinforcement Learning for Functional Software Testing

Deep Reinforcement Learning for Black-Box Testing of Android Apps

Intelligent Agent-Based Stimulation for Testing Robotic Software in Human-Robot Interactions

On Automating Video Game Regression Testing by Planning and Learning

Ontology and Reinforcement Learning Based Intelligent Agent Automatic Penetration Test

System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games

Deep Reinforcement Learning with Godot Game Engine

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Coverage-guided fuzzing for deep reinforcement learning systems

A Behavior-Based Ontology for Supporting Automated Assessment of Interactive Systems

Reinforcement Learning for Test Case Prioritization

Evolving Testing Scenario Generation Method and Intelligence Evaluation Framework for Automated Vehicles

FDQN: A Flexible Deep Q-Network Framework for Game Automation

Challenges of real-world reinforcement learning: definitions, benchmarks and analysis