Collaborative Software Development Platforms for Crowdsourcing

Xin Peng,Muhammad Ali Babar,Christof Ebert

DOI: https://doi.org/10.1109/ms.2014.31

IF: 3

2014-01-01

IEEE Software

Abstract:Outsourcing to the crowd, or crowdsourcing, has launched extremely successful businesses, such as Linux. But platforms for efficient collaboration and crowdsourcing support are still emerging. This article provides an overview of current technologies for crowdsourcing.

What problem does this paper attempt to address?

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Xiaoyan Yin,Yanjiao Chen,Baochun Li

DOI: https://doi.org/10.1109/iwqos.2017.7969165

2017-01-01

Abstract:Crowdsourcing leverages the collective intelligence of the massive crowd workers to accomplish tasks in a cost-effective way. On a crowdsourcing platform, it is challenging to assign tasks to workers in an appropriate way due to heterogeneity in both tasks and workers. In this paper, we explore the problem of assigning workers with various skill levels to tasks with different quality requirements and budget constraints. We first formulate the task assignment as a many-to-one matching problem, in which multiple workers are assigned to a task, and the task can be successfully completed only if a minimum quality requirement can be satisfied within its limited budget. Different from traditional task assignment mechanisms which focus on utility maximization for the crowdsourcing platform, our proposed matching framework takes into consideration the preferences of individual crowdsourcers and workers towards each other. We design a novel algorithm that can generate a stable outcome for the many-to-one matching problem with lower and upper bounds (i.e., quality requirement and budget constraint), as well as heterogeneous worker skill levels. Through extensive simulations, we show that the proposed algorithm can greatly improve the success ratio of task accomplishment and worker happiness, when compared with existing algorithms.
Guest Editorial: The Age of Crowdsourcing and Crowdfunding for Technological Innovation: Where We Are, and Where to Go?

Alexander M Brem,Christopher L Tucci,Terrence Brown,Jin Chen

DOI: https://doi.org/10.1109/TEM.2023.3270022

IF: 8.702

2023-01-01

IEEE Transactions on Engineering Management

Abstract:Taking advantage of the combined labor of individual users or even groups of users, organizations can increase the resources to which they have access. Crowdsourcing enables companies to create and capture more value than either the company or the user could do on their own. One of the chief enablers of crowdsourcing's rapid increase is the development and pervasive growth of information and communications technology (ICT), which has led to an explosion of possible crowdsourcing opportunities [1], [20]. Although crowdsourcing has had a significant impact on the practice of management, especially in entrepreneurship [16], there continues to be a multitude of research gaps and, therefore, many openings for future research. Thus, the purpose of this article is to not only introduce our special issue on the topic, but to highlight and summarize some of the research gaps and directions that have the potential to influence the future development of the field of crowdsourcing.
Artificial Intelligence-Powered Worker Engagement in Software Crowdsourcing

Junjie Wang,Ye Yang,Qing Wang

DOI: https://doi.org/10.1109/ms.2020.3017035

IF: 3

2020-11-01

IEEE Software

Abstract:Crowdsourced Software Engineering (CSE) evolved from outsourcing and open source development. It has created a fundamental shift; there are now many open-call software minitasks that are advertised and performed through popular CSE platforms. For instance, TopCoder currently has more than 1.5 million registered designers, developers, and quality engineers; uTest has 400,000-plus testing specialists with diverse expertise to validate various aspects of digital quality.

computer science, software engineering
Crowd work with or without crowdsourcing platforms

Xin Yan,Xianghua Ding,Ning Gu

DOI: https://doi.org/10.1109/CSCWD.2016.7565963

2016-01-01

Abstract:Crowdsourcing platforms such as Amazon Mechanical Turk (AMT), oDesk, InnoCentive, Science Exchange and others are on rapid rise in recent years. By outsourcing work to a network of people, they provide a more economical and efficient means of production. But essentially, crowdsourcing platforms are not the only means to outsource work to a crowd through the Internet. Studies in China, for example, have shown that quite some crowd works are accomplished through QQ (an Chinese instant messaging software) or other communication tools. So what are the advantages and disadvantages of using current crowdsourcing platforms, and how can they better designed? With these questions, and to better inform the design of future crowdsourcing platforms, we conducted a comparative study of crowd work with and without crowdsourcing platforms. In this paper, we present the study, and discuss advantages and disadvantages of using current crowdsourcing platforms in comparison to using QQ and other means for crowd work. We then end by suggesting implications for design for future crowdsourcing platforms.
Crowdsourcing Database Systems: Overview and Challenges

Chengliang Chai,Ju Fan,Guoliang Li,Jiannan Wang,Yudian Zheng

DOI: https://doi.org/10.1109/icde.2019.00237

2019-01-01

Abstract:Many data management and analytics tasks, such as entity resolution, cannot be solely addressed by automated processes. Crowdsourcing is an effective way to harness the human cognitive ability to process these computer-hard tasks. Thanks to public crowdsourcing platforms, e.g., Amazon Mechanical Turk and CrowdFlower, we can easily involve hundreds of thousands of ordinary workers (i.e., the crowd) to address these computer-hard tasks. However it is rather inconvenient to interact with the crowdsourcing platforms, because the platforms require one to set parameters and even write codes. Inspired by traditional DBMS, crowdsourcing database systems have been proposed and widely studied to encapsulate the complexities of interacting with the crowd. In this tutorial, we will survey and synthesize a wide spectrum of existing studies on crowdsourcing database systems. We first give an overview of crowdsourcing, and then summarize the fundamental techniques in designing crowdsourcing databases, including task design, truth inference, task assignment, answer reasoning and latency reduction. Next we review the techniques on designing crowdsourced operators, including selection, join, sort, top-k, max/min, count, collect, and fill. Finally, we discuss the emerging challenges.
Research on the Optimal Combination and Scheduling Method of Crowdsourcing Members in a Cloud Design Platform

Jian Chen,Rong Mo,Jianjie Chu,Suihuai Yu,Jiashuang Fan,Fangmin Cheng

DOI: https://doi.org/10.1177/0954405419855874

2019-01-01

Abstract:Cloud design has become a potential emerging design service mode. More and more design service providers are clustered in a cloud design platform to become crowdsourcing members. Collaborative tasks published by the platform are usually performed by several crowdsourcing members, and the collaborative mode of crowdsourcing members is an asynchronous collaborative working mode. However, due to the variety of crowdsourcing members in terms of ability, time, and so on, how to combine crowdsourcing members into a rational crowdsourcing team has become a challenge. At the same time, due to the looseness and uncertainty of their collaborative mode, there is no effective guidance mechanism in the cooperation process. Therefore, we propose a method of crowdsourcing the optimal combination and scheduling of members. By evaluating the comprehensive abilities of crowdsourcing members, the method optimizes the selection of suitable crowdsourcing members for each subtask and combines them into a crowdsourcing team. At the same time, besides establishing a collaborative workflow simulation model to plan the timing of subtasks and the workflow of crowdsourcing members, the method also optimizes and schedules crowdsourcing members to process different subtasks in turn. Finally, in the case of the collaborative design task of a medical analgesia pump, this method is used to establish a rational crowdsourcing team for collaborative tasks and effectively plan the collaborative work process of crowdsourcing members. The results demonstrate that the proposed method is feasible and effective.
An Evolutionary And Automated Virtual Team Making Approach For Crowdsourcing Platforms

Tao Yue,Shaukat Ali,Shuai Wang

DOI: https://doi.org/10.1007/978-3-662-47011-4_7

2015-01-01

Abstract:Crowdsourcing has demonstrated its capability of supporting various software development activities including development and testing as it can be seen by several successful crowdsourcing platforms such as TopCoder and uTest. However, to crowd source large-scale and complex software development and testing tasks, there are several optimization challenges to be addressed such as division of tasks, searching a set of registrants, and assignment of tasks to registrants. Since in crowdsourcing a task can be assigned to registrants geographically distributed with various backgrounds, the quality of final task deliverables is a key issue. As the first step to improve the quality, we propose a systematic and automated approach to optimize the assignment of registrants in a crowdsourcing platform to a crowdsourcing task. The objective is to find the best fit of a group of registrants to the defined task. A few examples of factors forming the optimization problem include budget defined by the task submitter and pay expectation from a registrant, skills required by a task, skills of a registrant, task delivering deadline, and availability of a registrant. We first collected a set of commonly seen factors that have impact on the perfect matching between tasks submitted and a virtual team that consists of a selected set of registrants. We then formulated the optimization objective as a fitness functionlthe heuristics used by search algorithms (e.g., Genetic Algorithms) to find an optimal solution. We empirically evaluated a set of well-known search algorithms in software engineering, along with the proposed fitness function, to identify the best solution for our optimization problem. Results of our experiments are very positive in terms of solving optimization problems in a crowdsourcing context.
Crowdsourcing as a Model for Problem Solving

Daren C. Brabham

DOI: https://doi.org/10.1177/1354856507084420

2008-02-01

Abstract:Crowdsourcing is an online, distributed problem-solving and production model that has emerged in recent years. Notable examples of the model include Threadless, iStockphoto, InnoCentive, the Goldcorp Challenge, and user-generated advertising contests. This article provides an introduction to crowdsourcing, both its theoretical grounding and exemplar cases, taking care to distinguish crowdsourcing from open source production. This article also explores the possibilities for the model, its potential to exploit a crowd of innovators, and its potential for use beyond forprofit sectors. Finally, this article proposes an agenda for research into crowdsourcing.
Towards a characterization of crowdsourcing practices

Eric Schenk,Claude Guittard

DOI: https://doi.org/10.3917/jie.007.0093

2011-04-06

Abstract:The word Crowdsourcing –a compound contraction of Crowd and Outsourcing, was used by Howe in order to define outsourcing to the crowd. Beyond cost, benefits for the company can be substantial. It can externalize the risk of failure and it only pays for products or services that meet its expectations. The Crowdsourcing phenomenon covers heterogeneous situations and it has inspired a number of authors. However, we are still lacking a general and synthetic view of this concept. The aim of our work is to characterize Crowdsourcing in its various aspects. First we define of Crowdsourcing, and provide examples that illustrate the diversity of Crowdsourcing practices and we present similarities and differences between Crowdsourcing and established theories (Open Innovation, User Innovation and Open Source Software). Then, we propose and illustrate a typology of Crowdsourcing practices based on two criteria: the integrative or selective nature of the process and the type of tasks that are crowd sourced (simple, complex and creative tasks). In either case, the client firm seeks to mobilize external competencies. Relying upon the crowd can be an adequate method, because of its unique characteristics that are fostered by the Internet. Finally, we present some potential benefits and pitfalls of Crowdsourcing.JEL Codes: O32, L17, L24, L29, M19

English Else
Human–Computer Interaction and Participation in Software Crowdsourcing

Habib Ullah Khan,Farhad Ali,Yazeed Yasin Ghadi,Shah Nazir,Inam Ullah,Heba G. Mohamed

DOI: https://doi.org/10.3390/electronics12040934

IF: 2.9

2023-02-13

Electronics

Abstract:Improvements in communication and networking technologies have transformed people's lives and organizations' activities. Web 2.0 innovation has provided a variety of hybridized applications and tools that have changed enterprises' functional and communication processes. People use numerous platforms to broaden their social contacts, select items, execute duties, and learn new things. Context: Crowdsourcing is an internet-enabled problem-solving strategy that utilizes human–computer interaction to leverage the expertise of people to achieve business goals. In crowdsourcing approaches, three main entities work in collaboration to solve various problems. These entities are requestors (job providers), platforms, and online users. Tasks are announced by requestors on crowdsourcing platforms, and online users, after passing initial screening, are allowed to work on these tasks. Crowds participate to achieve various rewards. Motivation: Crowdsourcing is gaining importance as an alternate outsourcing approach in the software engineering industry. Crowdsourcing application development involves complicated tasks that vary considerably from the micro-tasks available on platforms such as Amazon Mechanical Turk. To obtain the tangible opportunities of crowdsourcing in the realm of software development, corporations should first grasp how this technique works, what problems occur, and what factors might influence community involvement and co-creation. Online communities have become more popular recently with the rise in crowdsourcing platforms. These communities concentrate on specific problems and help people with solving and managing these problems. Objectives: We set three main goals to research crowd interaction: (1) find the appropriate characteristics of social crowd utilized for effective software crowdsourcing, (2) highlight the motivation of a crowd for virtual tasks, and (3) evaluate primary participation reasons by assessing various crowds using Fuzzy AHP and TOPSIS method. Conclusion: We developed a decision support system to examine the appropriate reasons of crowd participation in crowdsourcing. Rewards and employments were evaluated as the primary motives of crowds for accomplishing tasks on crowdsourcing platforms, knowledge sharing was evaluated as the third reason, ranking was the fourth, competency was the fifth, socialization was sixth, and source of inspiration was the seventh.

engineering, electrical & electronic,computer science, information systems,physics, applied
Crowdsourcing Intelligent Design

Wei Xiang,Ling-yun Sun,Wei-tao You,Chang-yuan Yang

DOI: https://doi.org/10.1631/fitee.1700810

IF: 2.526

2018-01-01

Frontiers of Information Technology & Electronic Engineering

Abstract:Design intelligence, namely, artificial intelligence to solve creative problems and produce creative ideas, has improved rapidly with the new generation artificial intelligence. However, existing methods are more skillful in learning from data and have limitations in creating original ideas different from the training data. Crowdsourcing offers a promising method to produce creative designs by combining human inspiration and machines’ computational ability. We propose a crowdsourcing intelligent design method called ‘flexible crowdsourcing design’. Design ideas produced through crowdsourcing design can be unreliable and inconsistent because they rely solely on selection among participants’ submissions of ideas. In contrast, the flexible crowdsourcing design method employs a cultivation procedure that integrates the ideas from crowd participants and cultivates these ideas to improve design quality at the same time. We introduce a series of studies to show how flexible crowdsourcing design can produce original design ideas consistently. Specifically, we will describe the typical procedure of flexible crowdsourcing design, the refined crowdsourcing tasks, the factors that affect the idea development process, the method for calculating idea development potential, and two applications of the flexible crowdsourcing design method. Finally, it summarizes the design capabilities enabled by crowdsourcing intelligent design. This method enhances the performance of crowdsourcing design and supports the development of design intelligence.
"The Whole Is Greater Than the Sum of Its Parts": Optimization in Collaborative Crowdsourcing

Habibur Rahman,Senjuti Basu Roy,Saravanan Thirumuruganathan,Sihem Amer-Yahia,Gautam Das

DOI: https://doi.org/10.48550/arXiv.1502.05106

2015-04-13

Abstract:In this work, we initiate the investigation of optimization opportunities in collaborative crowdsourcing. Many popular applications, such as collaborative document editing, sentence translation, or citizen science resort to this special form of human-based computing, where, crowd workers with appropriate skills and expertise are required to form groups to solve complex tasks. Central to any collaborative crowdsourcing process is the aspect of successful collaboration among the workers, which, for the first time, is formalized and then optimized in this work. Our formalism considers two main collaboration-related human factors, affinity and upper critical mass, appropriately adapted from organizational science and social theories. Our contributions are (a) proposing a comprehensive model for collaborative crowdsourcing optimization, (b) rigorous theoretical analyses to understand the hardness of the proposed problems, (c) an array of efficient exact and approximation algorithms with provable theoretical guarantees. Finally, we present a detailed set of experimental results stemming from two real-world collaborative crowdsourcing application us- ing Amazon Mechanical Turk, as well as conduct synthetic data analyses on scalability and qualitative aspects of our proposed algorithms. Our experimental results successfully demonstrate the efficacy of our proposed solutions.

Databases
Crowdsourcing Team Formation With Worker-Centered Modeling

Federica Lucia Vinella,Jiayuan Hu,Ioanna Lykourentzou,Judith Masthoff

DOI: https://doi.org/10.3389/frai.2022.818562

2022-05-27

Frontiers in Artificial Intelligence

Abstract:Modern crowdsourcing offers the potential to produce solutions for increasingly complex tasks requiring teamwork and collective labor. However, the vast scale of the crowd makes forming project teams an intractable problem to coordinate manually. To date, most crowdsourcing collaborative platforms rely on algorithms to automate team formation based on worker profiling data and task objectives. As a top-down strategy, algorithmic crowd team formation tends to alienate workers causing poor collaboration, interpersonal clashes, and dissatisfaction. In this paper, we investigate different ways that crowd teams can be formed through three team formation models namely bottom-up, top-down, and hybrid. By simulating an open collaboration scenario such as a hackathon, we observe that the bottom-up model forms the most competitive teams with the highest teamwork quality. Furthermore, we note that bottom-up approaches are particularly suitable for populations with high-risk appetites (most workers being lenient toward exploring new team configurations) and high degrees of homophily (most workers preferring to work with similar teammates). Our study highlights the importance of integrating worker agency in algorithm-mediated team formation systems, especially in collaborative/competitive settings, and bears practical implications for large-scale crowdsourcing platforms.
How to Work a Crowd: Developing Crowd Capital Through Crowdsourcing

J. Prpic,P. P. Shukla,J. H. Kietzmann,I. P. McCarthy

DOI: https://doi.org/10.48550/arXiv.1702.04214

2017-02-13

Abstract:Traditionally, the term crowd was used almost exclusively in the context of people who self-organized around a common purpose, emotion or experience. Today, however, firms often refer to crowds in discussions of how collections of individuals can be engaged for organizational purposes. Crowdsourcing, the use of information technologies to outsource business responsibilities to crowds, can now significantly influence a firms ability to leverage previously unattainable resources to build competitive advantage. Nonetheless, many managers are hesitant to consider crowdsourcing because they do not understand how its various types can add value to the firm. In response, we explain what crowdsourcing is, the advantages it offers and how firms can pursue crowdsourcing. We begin by formulating a crowdsourcing typology and show how its four categories (crowd-voting, micro-task, idea and solution crowdsourcing) can help firms develop crowd capital, an organizational-level resource harnessed from the crowd. We then present a three-step process model for generating crowd capital. Step one includes important considerations that shape how a crowd is to be constructed. Step two outlines the capabilities firms need to develop to acquire and assimilate resources (knowledge, labor, funds) from the crowd. Step three addresses key decision-areas that executives need to address to effectively engage crowds.

Computers and Society,Human-Computer Interaction
Tasks Decomposition Approaches in Crowdsourcing Software Development

Abdullah Khanfor

DOI: https://doi.org/10.48550/arXiv.2302.05099

2023-02-10

Abstract:A main characteristic of crowdsourcing software development (CSD) is the complexity of tasks and skills required by workers to achieve successful software crowdsourcing. The tasks proposed to the crowd in CSD are checked to ensure they are manageable and achievable. In general, individual tasks come from general goal-oriented projects. There are practices for breaking down software projects into manageable tasks, known as task decomposition. This study identified task decomposition techniques in software engineering, particularly in the context of CSD. Then, we defined the experienced developers who lead the requester in decomposing the project, preparing tasks, and reviewing submissions. This study explored and addressed decomposition approaches in CSD. Next, we selected projects in TopCoder to identify the task decomposition process in the CSD context. Finally, we concluded with future research directions for investigating decomposition approaches and their effects in the CSD context to ensure successful crowdsourced software projects.

Software Engineering
Exploiting mobile crowdsourcing for pervasive cloud services: challenges and solutions

Ju Ren,Yaoxue Zhang,Kuan Zhang,Xuemin Shen

DOI: https://doi.org/10.1109/MCOM.2015.7060488

IF: 9.03

2015-01-01

IEEE Communications Magazine

Abstract:With the proliferation of increasingly powerful mobile devices, mobile users can collaboratively form a mobile cloud to provide pervasive services, such as data collecting, processing, and computing. With this mobile cloud, mobile crowdsourcing, as an emerging service paradigm, can enable mobile users to take over the outsourced tasks. By leveraging the sensing capabilities of mobile devices and integrating human intelligence and machine-computation, mobile crowdsourcing has the potential to revolutionize the approach of data collecting and processing. In this article we investigate the mobile crowdsourcing architecture and applications, then discuss some research challenges and countermeasures for developing mobile crowdsourcing. Some research orientations are finally envisioned for further studies.
Towards AI-Empowered Crowdsourcing

Shipeng Wang,Qingzhong Li,Lizhen Cui,Zhongmin Yan,Yonghui Xu,Zhuan Shi,Xinping Min,Zhiqi Shen,Han Yu

2023-09-19

Abstract:Crowdsourcing, in which human intelligence and productivity is dynamically mobilized to tackle tasks too complex for automation alone to handle, has grown to be an important research topic and inspired new businesses (e.g., Uber, Airbnb). Over the years, crowdsourcing has morphed from providing a platform where workers and tasks can be matched up manually into one which leverages data-driven algorithmic management approaches powered by artificial intelligence (AI) to achieve increasingly sophisticated optimization objectives. In this paper, we provide a survey presenting a unique systematic overview on how AI can empower crowdsourcing to improve its efficiency - which we refer to as AI-Empowered Crowdsourcing(AIEC). We propose a taxonomy which divides AIEC into three major areas: 1) task delegation, 2) motivating workers, and 3) quality control, focusing on the major objectives which need to be accomplished. We discuss the limitations and insights, and curate the challenges of doing research in each of these areas to highlight promising future research directions.

Computers and Society,Artificial Intelligence
Crowdsourcing as a strategic IS sourcing phenomenon: Critical review and insights for future research

Dorit Nevo,Julia Kotlarsky

DOI: https://doi.org/10.1016/j.jsis.2020.101593

IF: 14.682

2020-12-01

The Journal of Strategic Information Systems

Abstract:Fueled by the development of Internet-based platforms that provided its technological foundation, and the need for an agile and uniquely skilled workforce, crowdsourcing has grown from the grassroots, with a burgeoning body of research investigating its many aspects. To gain insight into organizational crowdsourcing as a strategic IS sourcing phenomenon, this paper thoroughly reviews the literature to identify both areas of saturation and gaps, with a focus on the strategic organizational context. Pulling together knowledge on specific aspects of crowdsourcing, we first offer a high-level analysis of definitions to reveal rather broad coverage of various activities involving the crowd, many of which do not involve sourcing. We further build on the literature to establish boundary conditions and clarify the focus on crowdsourcing. This is followed by an in-depth critical analysis of selected studies published in top IS and general management journals to date. Through this review, we identify key themes that emerge out of the crowdsourcing literature and synthesize the literature to chart a more focused research path moving forward. Guided by our analysis, we offer a road map for future research that brings together fine-grained insights from existing crowdsourcing studies towards developing a high-level, macro-perspective of the crowdsourcing phenomenon and its strategic impact.
Optimal Team Recruitment Strategies for Collaborative Mobile Crowdsourcing Systems

Aymen Hamrouni,Hakim Ghazzai,Turki Alelyani,Yehia Massoud

DOI: https://doi.org/10.48550/arXiv.2004.11512

2020-04-28

Abstract:The wide spread of mobile devices has enabled a new paradigm of innovation called Mobile Crowdsourcing (MCS) where the concept is to allow entities, e.g., individuals or local authorities, to hire workers to help from the crowd of connected people, to execute a task or service. Some complex tasks require the collaboration of multiple workers to ensure its successful completion. In this context, the task requester needs to hire a group of socially connected and collaborative workers that, at the same time, have sufficient skills to accomplish the task. In this paper, we develop two recruitment strategies for collaborative MCS frameworks in which, virtual teams are formed according to four different criteria: level of expertise, social relationship strength, recruitment cost, and recruiter's confidence level. The first proposed strategy is a platform-based approach which exploits the platform knowledge to form the team. The second one is a leader-based approach that uses team members' knowledge about their social network (SN) neighbors to designate a group leader that recruits its suitable team. Both approaches are modeled as integer linear programs resulting in optimal team formation. Experimental results show a performance trade-off between the two virtual team grouping strategies when varying the members SN edge degree. Compared to the leader-based strategy, the platform-based strategy recruits a more skilled team but with lower SN relationships and higher cost.

Social and Information Networks,Computers and Society,Systems and Control
Deploying Crowdsourcing for Workflow Driven Business Process

Rafał Masłyk,Kinga Skorupska,Piotr Gago,Marcin Niewiński,Barbara Karpowicz,Anna Jaskulska,Katarzyna Abramczuk,Wiesław Kopeć

DOI: https://doi.org/10.48550/arXiv.2101.01168

IF: 6.4588

2021-01-04

Human-Computer Interaction

Abstract:The main goal of this paper is to discuss how to integrate the possibilities of crowdsourcing platforms with systems supporting workflow to enable the engagement and interaction with business tasks of a wider group of people. Thus, this work is an attempt to expand the functional capabilities of typical business systems by allowing selected process tasks to be performed by unlimited human resources. Opening business tasks to crowdsourcing, within established Business Process Management Systems (BPMS) will improve the flexibility of company processes and allow for lower work-load and greater specialization among the staff employed on-site. The presented conceptual work is based on the current international standards in this field, promoted by Workflows Management Coalition. To this end, the functioning of business platforms was analysed and their functionality was presented visually, followed by a proposal and a discussion of how to implement crowdsourcing into workflow systems.

Collaborative Software Development Platforms for Crowdsourcing

Task Assignment with Guaranteed Quality for Crowdsourcing Platforms.

Guest Editorial: The Age of Crowdsourcing and Crowdfunding for Technological Innovation: Where We Are, and Where to Go?

Artificial Intelligence-Powered Worker Engagement in Software Crowdsourcing

Crowd work with or without crowdsourcing platforms

Crowdsourcing Database Systems: Overview and Challenges

Research on the Optimal Combination and Scheduling Method of Crowdsourcing Members in a Cloud Design Platform

An Evolutionary And Automated Virtual Team Making Approach For Crowdsourcing Platforms

Crowdsourcing as a Model for Problem Solving

Towards a characterization of crowdsourcing practices

Human–Computer Interaction and Participation in Software Crowdsourcing

Crowdsourcing Intelligent Design

"The Whole Is Greater Than the Sum of Its Parts": Optimization in Collaborative Crowdsourcing

Crowdsourcing Team Formation With Worker-Centered Modeling

How to Work a Crowd: Developing Crowd Capital Through Crowdsourcing

Tasks Decomposition Approaches in Crowdsourcing Software Development

Exploiting mobile crowdsourcing for pervasive cloud services: challenges and solutions

Towards AI-Empowered Crowdsourcing

Crowdsourcing as a strategic IS sourcing phenomenon: Critical review and insights for future research

Optimal Team Recruitment Strategies for Collaborative Mobile Crowdsourcing Systems

Deploying Crowdsourcing for Workflow Driven Business Process