Harnessing the Wisdom of Crowds

Zhi Da,Xing Huang

DOI: https://doi.org/10.2139/ssrn.2731884

2015-01-01

SSRN Electronic Journal

Abstract:Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI

What problem does this paper attempt to address?

Crowd Intelligence in AI 2.0 Era.

Wei Li,Wen-jun Wu,Huai-min Wang,Xue-qi Cheng,Hua-jun Chen,Zhi-hua Zhou,Rong Ding

DOI: https://doi.org/10.1631/fitee.1601859

2017-01-01

Abstract:The Internet based cyber-physical world has profoundly changed the information environment for the development of artificial intelligence (AI), bringing a new wave of AI research and promoting it into the new era of AI 2.0. As one of the most prominent characteristics of research in AI 2.0 era, crowd intelligence has attracted much attention from both industry and research communities. Specifically, crowd intelligence provides a novel problem-solving paradigm through gathering the intelligence of crowds to address challenges. In particular, due to the rapid development of the sharing economy, crowd intelligence not only becomes a new approach to solving scientific challenges, but has also been integrated into all kinds of application scenarios in daily life, e.g., online-to-offline (O2O) application, real-time traffic monitoring, and logistics management. In this paper, we survey existing studies of crowd intelligence. First, we describe the concept of crowd intelligence, and explain its relationship to the existing related concepts, e.g., crowdsourcing and human computation. Then, we introduce four categories of representative crowd intelligence platforms. We summarize three core research problems and the state-of-the-art techniques of crowd intelligence. Finally, we discuss promising future research directions of crowd intelligence.
Harnessing the Power of the General Public for Crowdsourced Business Intelligence: A Survey.

Bin Guo,Yan Liu,Yi Ouyang,Vincent W. Zheng,Daqing Zhang,Zhiwen Yu

DOI: https://doi.org/10.1109/access.2019.2901027

IF: 3.9

2019-01-01

IEEE Access

Abstract:Crowdsourced business intelligence (CrowdBI), which leverages the crowdsourced user-generated data to extract useful knowledge about business and create marketing intelligence to excel in the business environment, has become a surging research topic in recent years. Compared with the traditional business intelligence that is based on the firm-owned data and survey data, CrowdBI faces numerous unique issues, such as customer behavior analysis, brand tracking, and product improvement, demand forecasting and trend analysis, competitive intelligence, business popularity analysis and site recommendation, and urban commercial analysis. This paper first characterizes the concept model and unique features and presents a generic framework for CrowdBI. It also investigates novel application areas as well as the key challenges and techniques of CrowdBI. Furthermore, we make discussions about the future research directions of CrowdBI.
Crowd characteristics and crowd wisdom: Evidence from an online investment community

Hong Hong,Qiang Ye,Qianzhou Du,G. Alan Wang,Weiguo Fan

DOI: https://doi.org/10.1002/asi.24255

2019-05-15

Journal of the Association for Information Science and Technology

Abstract:<p>Fueled by the explosive growth of Web 2.0 and social media, online investment communities have become a popular venue for individual investors to interact with each other. Investor opinions extracted from online investment communities capture "crowd wisdom" and have begun to play an important role in financial markets. Existing research confirms the importance of crowd wisdom in stock predictions, but fails to investigate factors influencing crowd performance (that is, crowd prediction accuracy). In order to help improve crowd performance, our research strives to investigate the impact of crowd characteristics on crowd performance. We conduct an empirical study using a large data set collected from a popular online investment community, StockTwits. Our findings show that experience diversity, participant independence, and network decentralization are all positively related to crowd performance. Furthermore, crowd size moderates the influence of crowd characteristics on crowd performance. From a theoretical perspective, our work enriches extant literature by empirically testing the relationship between crowd characteristics and crowd performance. From a practical perspective, our findings help investors better evaluate social sensors embedded in user‐generated stock predictions, based upon which they can make better investment decisions.</p>

information science & library science,computer science, information systems
Crowdsourcing with Multiple-Source Knowledge Transfer

Guangyang Han,Jinzheng Tu,Guoxian Yu,Jun Wang,Carlotta Domeniconi

DOI: https://doi.org/10.24963/ijcai.2020/402

2020-01-01

Abstract:Crowdsourcing is a new computing paradigm that harnesses human effort to solve computer-hard problems. Budget and quality are two fundamental factors in crowdsourcing, but they are antagonistic and their balance is crucially important. Induction and inference are principled ways for humans to acquire knowledge. Transfer learning can also enable induction and inference processes. When a new task comes, we may not know how to go about approaching it. On the other hand, we may have easy access to relevant knowledge that can help us with the new task. As such, via appropriate knowledge transfer, for example, an improved annotation can be achieved for the task at a small cost. To make this idea concrete, we introduce the Crowdsourcing with Multiple-source Knowledge Transfer (CrowdMKT)approach to transfer knowledge from multiple, similar, but different domains for a new task, and to reduce the negative impact of irrelevant sources. CrwodMKT first learns a set of concentrated high-level feature vectors of tasks using knowledge transfer from multiple sources, and then introduces a probabilistic graphical model to jointly model the tasks with high-level features, workers, and their annotations. Finally, it adopts an EM algorithm to estimatethe workers strengths and consensus. Experimental results on real-world image and text datasets prove the effectiveness of CrowdMKT in improving quality and reducing the budget.
Wisdom of the crowd from unsupervised dimension reduction

Lingfei Wang,Tom Michoel

DOI: https://doi.org/10.1098/rsos.181806

2017-11-29

Abstract:Wisdom of the crowd, the collective intelligence derived from responses of multiple human or machine individuals to the same questions, can be more accurate than each individual, and improve social decision-making and prediction accuracy. This can also integrate multiple programs or datasets, each as an individual, for the same predictive questions. Crowd wisdom estimates each individual's independent error level arising from their limited knowledge, and finds the crowd consensus that minimizes the overall error. However, previous studies have merely built isolated, problem-specific models with limited generalizability, and mainly for binary (yes/no) responses. Here we show with simulation and real-world data that the crowd wisdom problem is analogous to one-dimensional unsupervised dimension reduction in machine learning. This provides a natural class of crowd wisdom solutions, such as principal component analysis and Isomap, which can handle binary and also continuous responses, like confidence levels, and consequently can be more accurate than existing solutions. They can even outperform supervised-learning-based collective intelligence that is calibrated on historical performance of individuals, e.g. penalized linear regression and random forest. This study unifies crowd wisdom and unsupervised dimension reduction, and thereupon introduces a broad range of highly-performing and widely-applicable crowd wisdom methods. As the costs for data acquisition and processing rapidly decrease, this study will promote and guide crowd wisdom applications in the social and natural sciences, including data fusion, meta-analysis, crowd-sourcing, and committee decision making.

Machine Learning,Quantitative Methods,Methodology
Crowd-Powered Data Mining

Chengliang Chai,Ju Fan,Guoliang Li,Jiannan Wang,Yudian Zheng

DOI: https://doi.org/10.48550/arXiv.1806.04968

2018-10-19

Abstract:Many data mining tasks cannot be completely addressed by auto- mated processes, such as sentiment analysis and image classification. Crowdsourcing is an effective way to harness the human cognitive ability to process these machine-hard tasks. Thanks to public crowdsourcing platforms, e.g., Amazon Mechanical Turk and Crowd- Flower, we can easily involve hundreds of thousands of ordinary workers (i.e., the crowd) to address these machine-hard tasks. In this tutorial, we will survey and synthesize a wide spectrum of existing studies on crowd-powered data mining. We first give an overview of crowdsourcing, and then summarize the fundamental techniques, including quality control, cost control, and latency control, which must be considered in crowdsourced data mining. Next we review crowd-powered data mining operations, including classification, clustering, pattern mining, machine learning using the crowd (including deep learning, transfer learning and semi-supervised learning) and knowledge discovery. Finally, we provide the emerging challenges in crowdsourced data mining.

Databases,Artificial Intelligence
When social influence promotes the wisdom of crowds

Abdullah Almaatouq,M. Amin Rahimian,Jason W. Burton,Abdulla Alhajri,Jason Burton

DOI: https://doi.org/10.31234/osf.io/f5ky3

2020-07-14

Abstract:Whether, and under what conditions, groups exhibit "crowd wisdom" has been a major focus of research across the social and computational sciences. Much of this work has focused on the role of social influence in promoting the wisdom of the crowd versus leading the crowd astray, resulting in conflicting conclusions about how the social network structure determines the impact of social influence. Here, we demonstrate that it is not enough to consider the network structure in isolation. Using theoretical analysis, numerical simulation, and reanalysis of four experimental datasets (totaling 2,885 human subjects), we find that the wisdom of crowds critically depends on the interaction between (i) the centralization of the social influence network and (ii) the distribution of the initial, individual estimates. By adopting a framework that integrates both the structure of the social influence and the distribution of the initial estimates, we bring previously conflicting results under one theoretical framework and clarify the effects of social influence on the wisdom of crowds.
Crowdsourcing Intelligent Design

Wei Xiang,Ling-yun Sun,Wei-tao You,Chang-yuan Yang

DOI: https://doi.org/10.1631/fitee.1700810

IF: 2.526

2018-01-01

Frontiers of Information Technology & Electronic Engineering

Abstract:Design intelligence, namely, artificial intelligence to solve creative problems and produce creative ideas, has improved rapidly with the new generation artificial intelligence. However, existing methods are more skillful in learning from data and have limitations in creating original ideas different from the training data. Crowdsourcing offers a promising method to produce creative designs by combining human inspiration and machines’ computational ability. We propose a crowdsourcing intelligent design method called ‘flexible crowdsourcing design’. Design ideas produced through crowdsourcing design can be unreliable and inconsistent because they rely solely on selection among participants’ submissions of ideas. In contrast, the flexible crowdsourcing design method employs a cultivation procedure that integrates the ideas from crowd participants and cultivates these ideas to improve design quality at the same time. We introduce a series of studies to show how flexible crowdsourcing design can produce original design ideas consistently. Specifically, we will describe the typical procedure of flexible crowdsourcing design, the refined crowdsourcing tasks, the factors that affect the idea development process, the method for calculating idea development potential, and two applications of the flexible crowdsourcing design method. Finally, it summarizes the design capabilities enabled by crowdsourcing intelligent design. This method enhances the performance of crowdsourcing design and supports the development of design intelligence.
Enabling Human-Centric Smart Cities: Crowdsourcing-Based Practice in China

Jiangtao Wang,Junfeng Zhao,Yong Zhang,Xin Peng,Ying Li,Yun Xie

DOI: https://doi.org/10.1109/mc.2018.2880016

2018-01-01

Computer

Abstract:Existing smart city systems are designed mainly for urban management from a governmental perspective, but they fail to cover the wide spectrum of citizens' daily lives. We propose a crowdsourcing-based platform through which multiple players can collaborate to produce more abundant, personalized, and proactive services.
Crowd-Selection Query Processing in Crowdsourcing Databases: A Task-Driven Approach.

Zhou Zhao,Furu Wei,Ming Zhou,Weikeng Chen,Wilfred Ng

DOI: https://doi.org/10.5441/002/edbt.2015.35

2015-01-01

Abstract:Crowd-selection is essential to crowdsourcing applications, since choosing the right workers with particular expertise to carry out specific crowdsourced tasks is extremely important. The central problem is simple but tricky: given a crowdsourced task, is the right worker to ask? Currently, most existing work has mainly studied the problem of crowd-selection for simple crowdsourced tasks such as decision making and sentiment analysis. Their crowd- selection procedures are based on the trustworthiness of workers. However, for some complex tasks such as document review and question answering, selecting workers based on the latent category of tasks is a better solution. In this paper, we formulate a new problem of task-driven crowd- selection for complex tasks. We first develop a bayesian gener- ative model to exploit who knows what for the workers in the crowdsourcing environment. The model provides a principle and natural framework for capturing the latent skills of workers as well as the latent categories of crowdsourced tasks. The inference of the latent skills of workers is based on past resolved crowdsourced tasks with feedback scores. We assume that the feedback scores can illustrate the performance of the workers for the tasks. We then devise a variational algorithm that transforms the latent skill infer- ence with the proposed model into a standard optimization prob- lem, which can be solved efficiently. We verify the performance of our method through extensive experiments on the data collected from three well-known crowdsourcing platforms for question an- swering tasks such as Quora, Yahoo ! Answer and Stack Overflow.
Brief Survey of Crowdsourcing for Data Mining

Guo Xintong,Wang Hongzhi,Yangqiu Song,Gao Hong

DOI: https://doi.org/10.1016/j.eswa.2014.06.044

IF: 8.5

2014-01-01

Expert Systems with Applications

Abstract:Crowdsourcing allows large-scale and flexible invocation of human input for data gathering and analysis, which introduces a new paradigm of data mining process. Traditional data mining methods often require the experts in analytic domains to annotate the data. However, it is expensive and usually takes a long time. Crowdsourcing enables the use of heterogeneous background knowledge from volunteers and distributes the annotation process to small portions of efforts from different contributions. This paper reviews the state-of-the-arts on the crowdsourcing for data mining in recent years. We first review the challenges and opportunities of data mining tasks using crowdsourcing, and summarize the framework of them. Then we highlight several exemplar works in each component of the framework, including question designing, data mining and quality control. Finally, we conclude the limitation of crowdsourcing for data mining and suggest related areas for future research.
Using the Wisdom of Crowds to Prevent Internet Frauds

Tao Zheng,Yong Jiang,Lei Zhang

2013-01-01

Abstract:With the rapid growth of the netizen population in China, more and more internet frauds are committed. Many people suffer from internet frauds by losing wealth or other valuable things. To prevent internet frauds, we first need to discover the methods in which internet frauds are conducted. In this paper, we investigate and categorize the internet frauds in China. So far, there are typically six kinds of internet frauds, including email fraud, website fraud, e-commerce fraud, virus fraud, password fraud, and message fraud. To prevent these internet frauds, many approaches, such as in- trusion detection and access control, have been proposed to help users. However, most of these methods are limited to detecting a small volume of internet frauds. To address this issue, we propose a methodology to use the wisdom of crowds, with the help of Semantic Web and Web 2.0 tech- nologies, to detect a large volume of internet frauds. The proposed framework is composed of eight modules: internet fraud report module, key element extraction module, linked data generation module, linked data repository, query inter- face, query interpretation module, SPARQL module, and answer generation module. Based on the framework, users are able to input the internet fraud reports in a controlled natural language. The internet fraud reports are converted into linked data automatically. Then, the users can query the linked data in a semantic fashion. A case study and a survey preliminarily indicate that the proposed method is able to help users share and identify the internet frauds effectively.
Towards AI-Empowered Crowdsourcing

Shipeng Wang,Qingzhong Li,Lizhen Cui,Zhongmin Yan,Yonghui Xu,Zhuan Shi,Xinping Min,Zhiqi Shen,Han Yu

2023-09-19

Abstract:Crowdsourcing, in which human intelligence and productivity is dynamically mobilized to tackle tasks too complex for automation alone to handle, has grown to be an important research topic and inspired new businesses (e.g., Uber, Airbnb). Over the years, crowdsourcing has morphed from providing a platform where workers and tasks can be matched up manually into one which leverages data-driven algorithmic management approaches powered by artificial intelligence (AI) to achieve increasingly sophisticated optimization objectives. In this paper, we provide a survey presenting a unique systematic overview on how AI can empower crowdsourcing to improve its efficiency - which we refer to as AI-Empowered Crowdsourcing(AIEC). We propose a taxonomy which divides AIEC into three major areas: 1) task delegation, 2) motivating workers, and 3) quality control, focusing on the major objectives which need to be accomplished. We discuss the limitations and insights, and curate the challenges of doing research in each of these areas to highlight promising future research directions.

Computers and Society,Artificial Intelligence
The Paradox of Interaction: Communication Network Centralization, Shared Task Experience, and the Wisdom of Crowds in Online Crowdsourcing Communities

Yan Bei,Jian Lian,Ren Ruqin,Janet Fulk,Emily Sidnam-Mauch,Peter Monge

DOI: https://doi.org/10.1177/0093650220915033

IF: 6.3

2020-01-01

Communication Research

Abstract:Research on the wisdom of crowds (WOC) identifies two paradoxical effects of communication. The social influence effect hampers the WOC, whereas the collective learning effect improves crowd wisdom. Yet it remains unclear under what conditions such communication impedes or enhances collective wisdom. The current study examined two features characterizing communication in online communities, communication network centralization and shared task experience, and their effect on the WOC. Both these features can serve as indicators of the likelihood that underlying communication may facilitate either social influence or collective learning. With an 8-year longitudinal behavioral-trace data set of 269,871 participants and 1,971 crowds, we showed that communication network centralization negatively affected the WOC. By contrast, shared task experience positively predicted the WOC. Shared task experience also moderated the effect of communication network centralization such that centralized communication networks became more beneficial for crowd performance as shared task experience increased.
T-Crowd: Effective Crowdsourcing for Tabular Data

Caihua Shan,Nikos Mamoulis,Guoliang Li,Reynold Cheng,Zhipeng Huang,Yudian Zheng

DOI: https://doi.org/10.48550/arXiv.1708.02125

2017-08-07

Abstract:Crowdsourcing employs human workers to solve computer-hard problems, such as data cleaning, entity resolution, and sentiment analysis. When crowdsourcing tabular data, e.g., the attribute values of an entity set, a worker's answers on the different attributes (e.g., the nationality and age of a celebrity star) are often treated independently. This assumption is not always true and can lead to suboptimal crowdsourcing performance. In this paper, we present the T-Crowd system, which takes into consideration the intricate relationships among tasks, in order to converge faster to their true values. Particularly, T-Crowd integrates each worker's answers on different attributes to effectively learn his/her trustworthiness and the true data values. The attribute relationship information is also used to guide task allocation to workers. Finally, T-Crowd seamlessly supports categorical and continuous attributes, which are the two main datatypes found in typical databases. Our extensive experiments on real and synthetic datasets show that T-Crowd outperforms state-of-the-art methods in terms of truth inference and reducing the cost of crowdsourcing.

Databases
Similarity-driven and task-driven models for diversity of opinion in crowdsourcing markets

Chen Jason Zhang,Yunrui Liu,Pengcheng Zeng,Ting Wu,Lei Chen,Pan Hui,Fei Hao

DOI: https://doi.org/10.1007/s00778-024-00853-0

2024-05-18

The VLDB Journal

Abstract:The recent boom in crowdsourcing has opened up a new avenue for utilizing human intelligence in the realm of data analysis. This innovative approach provides a powerful means for connecting online workers to tasks that cannot effectively be done solely by machines or conducted by professional experts due to cost constraints. Within the field of social science, four elements are required to construct a sound crowd—Diversity of Opinion, Independence, Decentralization and Aggregation. However, while the other three components have already been investigated and implemented in existing crowdsourcing platforms, 'Diversity of Opinion' has not been functionally enabled yet. From a computational point of view, constructing a wise crowd necessitates quantitatively modeling and taking diversity into account. There are usually two paradigms in a crowdsourcing marketplace for worker selection: building a crowd to wait for tasks to come and selecting workers for a given task. We propose similarity-driven and task-driven models for both paradigms. Also, we develop efficient and effective algorithms for recruiting a limited number of workers with optimal diversity in both models. To validate our solutions, we conduct extensive experiments using both synthetic datasets and real data sets.

computer science, information systems, hardware & architecture
Fair Task Assignment in Spatial Crowdsourcing.

Zhao Chen,Peng Cheng,Lei Chen,Xuemin Lin,Cyrus Shahabi

DOI: https://doi.org/10.14778/3407790.3407839

2020-01-01

Abstract:With the pervasiveness of mobile devices, wireless broadband and sharing economy, spatial crowdsourcing is becoming part of our daily life. Existing studies on spatial crowdsourcing usually focus o...
Crowd-Machine Hybrid Urban Sensing and Computing

Jiangtao Wang,Leye Wang,Yanhui Song

DOI: https://doi.org/10.1109/mc.2020.3023931

2021-01-01

Computer

Abstract:With progress in the Internet of Things, artificial intelligence, and cloud/edge computing, urban sensing and computing (USC) has become a promising solution to address the challenges in modern cities. This article investigates how to combine the power of human/crowd and machine intelligence to enable more innovative USC applications.
An Introduction to the Special Issue on Participatory Sensing and Crowd Intelligence

Bin Guo,Alvin Chin,Zhiwen Yu,Runhe Huang,Daqing Zhang

DOI: https://doi.org/10.1145/2745712

IF: 5

2015-01-01

ACM Transactions on Intelligent Systems and Technology

Abstract:editorial Free AccessAn Introduction to the Special Issue on Participatory Sensing and Crowd Intelligence Authors: Bin Guo Northwestern Polytechnical University, China Northwestern Polytechnical University, ChinaView Profile , Alvin Chin Nokia, Beijing, China Nokia, Beijing, ChinaView Profile , Zhiwen Yu Northwestern Polytechnical University, China Northwestern Polytechnical University, ChinaView Profile , Runhe Huang Hosei University, Tokyo, Japan Hosei University, Tokyo, JapanView Profile , Daqing Zhang Institut TELECOM SudParis, France Institut TELECOM SudParis, FranceView Profile Authors Info & Claims ACM Transactions on Intelligent Systems and TechnologyVolume 6Issue 3May 2015 Article No.: 36pp 1–4https://doi.org/10.1145/2745712Published:21 April 2015Publication History 5citation311DownloadsMetricsTotal Citations5Total Downloads311Last 12 Months18Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF
How the crowd impacts commercial applications: a user-oriented approach.

Huihong He,Zhiyi Ma,Hongjie Chen,Weizhong Shao

DOI: https://doi.org/10.1145/2666539.2666566

2014-01-01

Abstract:ABSTRACT As crowdsourcing has been applied to a variety of disciplines, e.g. marketing and operationalization, more and more scientists turn their sights to how the crowd innovate software engineering to produce high quality software. However, they mainly focus on the impacts brought by domain experts or experienced developers on developing and managing open source softwares, whereas how softwares are influenced by the ordinary people e.g. end users is seldom discussed and easily omitted. To fill up the research gaps, we investigate into commercial application improvement paradigm with assistance of user crowd. The approach focuses on end users by proposing a workflow loop to form a healthy cycle between them and applications. Especially, the approach propose a suggestion model to encourage users to participate into application runtime adaptation. So far, a prototype is developed to enable the crowd to raise and modify their advices, and our prior work has proven the effectiveness in which applications consider the users’ advices to adapt themselves.

Harnessing the Wisdom of Crowds

Crowd Intelligence in AI 2.0 Era.

Harnessing the Power of the General Public for Crowdsourced Business Intelligence: A Survey.

Crowd characteristics and crowd wisdom: Evidence from an online investment community

Crowdsourcing with Multiple-Source Knowledge Transfer

Wisdom of the crowd from unsupervised dimension reduction

Crowd-Powered Data Mining

When social influence promotes the wisdom of crowds

Crowdsourcing Intelligent Design

Enabling Human-Centric Smart Cities: Crowdsourcing-Based Practice in China

Crowd-Selection Query Processing in Crowdsourcing Databases: A Task-Driven Approach.

Brief Survey of Crowdsourcing for Data Mining

Using the Wisdom of Crowds to Prevent Internet Frauds

Towards AI-Empowered Crowdsourcing

The Paradox of Interaction: Communication Network Centralization, Shared Task Experience, and the Wisdom of Crowds in Online Crowdsourcing Communities

T-Crowd: Effective Crowdsourcing for Tabular Data

Similarity-driven and task-driven models for diversity of opinion in crowdsourcing markets

Fair Task Assignment in Spatial Crowdsourcing.

Crowd-Machine Hybrid Urban Sensing and Computing

An Introduction to the Special Issue on Participatory Sensing and Crowd Intelligence

How the crowd impacts commercial applications: a user-oriented approach.