Abstract:The term crowdsourcing – a compound contraction of crowd and outsourcing – is a new paradigm for utilizing the power of crowds of people to facilitate large-scale tasks that are costly or time consuming with traditional methods. This paradigm offers mobile application companies the possibility to outsource their testing activities to crowdsourced testers (crowdtesters) who have various testing facilities and environments, as well as different levels of skills and expertise. With this so-called Crowdsourced Mobile Application Testing (CMAT), some of the well-recognized issues in testing mobile applications, such as multitude of mobile devices, fragmentation of device models, variety of OS versions, and omnifariousness of testing scenarios, could be mitigated. However, how effective is CMAT in practice? What are the challenges and issues presented by the process of applying CMAT? How can these issues and challenges be overcome and CMAT be improved? Although CMAT has attracted attention from both academia and industry, these questions have not been addressed or researched in depth based on a large-scale and real-life industrial study. Since June 2015, we have worked with Mooctest, Inc., a CMAT intermediary, on testing five real-life Android applications using their CMAT platform – Kikbug. Throughout the process, we have collected 1013 bug reports from 258 crowdtesters and found 247 bugs in total. This paper will present our industrial study thoroughly and give an insightful analysis to investigate the successes and challenges of applying CMAT.

Mobile Testing-as-a-Service (MTaaS) -- Infrastructures, Issues, Solutions and Needs

Testing as a Service (TaaS) on Clouds

SaaS Testing on Clouds - Issues, Challenges and Needs

A cloud-based TaaS infrastructure with tools for SaaS validation, performance and scalability evaluation

Cloud testing-issues, challenges, needs and practice

[email protected]: the virtual test lab on the cloud

Testing As a Service over Cloud

Automated Model-Based Performance Testing for PaaS Cloud Services

Mobile Application Testing: A Tutorial

A Framework of Testing As a Service

Towards Scalable Automated Mobile App Testing

Vee@Cloud: the virtual test lab on the cloud

Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing

Performing unit testing based on testing as a service (TaaS) approach

Survey of End-to-End Mobile Network Measurement Testbeds, Tools, and Services

MTISS: A novel mobile transportation information service system based on SOA

Adaptive Fault Detection for Testing Tenant Applications in Multi-tenancy SaaS Systems

Towards a Scalable and Robust Multi-Tenancy SaaS

Regression Testing As a Service

Successes, Challenges, and Rethinking – an Industrial Investigation on Crowdsourced Mobile Application Testing

Research and Design of Mobile Service Testing System Based on Wireless Sensor Network