Conference PaperPDF Available

Self-selection In Crowdsourced Democracy: A Bug Or A Feature?


Abstract and Figures

This paper examines the tension between the norm of equal representation in democracy and the ideals of crowdsourcing, which strive for a large undefined, self-selected crowd. We argue that crowdsourcing, when used in democratic processes, can never meet the standard for statistical representativeness, which is the often-strived standard in democratic processes. We also argue that crowdsourcing shouldn't strive for statistical representativeness of the public, otherwise the virtues of crowdsourcing would be compromised and its benefits in crowdwork would not be achieved.
Content may be subject to copyright.
Self-selection In Crowdsourced Democracy:
A Bug Or A Feature?
Tanja Aitamurto
School of Engineering,
Stanford University
475 Via Ortega, Stanford,
California, 94305
Jorge Saldivar Galli
Department of Information
Engineering and Computer
Science, University of Trento
Via Sommarive 9, Povo, 38123
Juho Salminen
Lahti School of Innovation
Lappeenranta University of
Saimaankatu 11, LaHti, 15140
This paper examines the tension between the norm of equal
representation in democracy and the ideals of
crowdsourcing, which strive for a large undefined, self-
selected crowd. We argue that crowdsourcing, when used in
democratic processes, can never meet the standard for
statistical representativeness, which is the often-strived
standard in democratic processes. We also argue that
crowdsourcing shouldn’t strive for statistical
representativeness of the public, otherwise the virtues of
crowdsourcing would be compromised and its benefits in
crowdwork would not be achieved.
Author Keywords
Crowdsourcing; democratic innovations; representativeness
Crowdsourcing in democratic processes, such as in public
policy-making, has become more common in the recent
years. Crowdsourcing has been used in knowledge search
for law-reforms [1, 2], in policy-making in local
governance [3] and in eliciting ideas for state policy agenda
[4]. Crowdsourcing for democracy is a democratic
innovation [5] in that it engages citizens in policy-making
and brings them closer to the political decision-making
power. An often-heard objection against crowdsourcing in
democratic processes is the lack of representativeness of the
participant crowd, which leads to input from a biased
sample of population. This can be problematic in the
context of democratic processes, because the
crowdsourcing initiative can attract only certain
demographic groups, or groups that share a certain political
view. Therefore, it is justified to pose the following
question: How does crowdsourcing as a participatory
mechanism fit to the democratic ideal of equal
representation? In this paper we address that question.
For the purposes of this paper we consider equal
representation consisting of two complementary aspects:
each potential participant should have equal probability of
participating, and each participant should contribute an
equal amount. In polls and surveys, which are aimed to
support democratic decision-making, representativeness is
ensured by using representative sampling. A representative
sample is a selected subset from a population that reflects
the relevant features of the population accurately [6]. For
example, if the population consists of 50% females, also the
representative sample from that population should have
about 50% females [7]. A form of random selection
procedure is usually employed in sampling.
To ensure equal and comparative contributions, surveys for
instance, aim to collect exactly the same amount of
information from each participant [8]. Combination of
random sampling and fixed contributions allows
generalization of findings from the sample to the population
as a whole. The deliberative poll [16] is a democratic
innovation, which employs the idea of equal representation.
It relies on the notion of “mini-publics”, i.e. random
samples of citizens, who are invited to deliberate about
societal issues. Their opinion is measured after the
deliberation and is considered to be a representative opinion
about the issue, because it is assumed to represent the views
of the larger public.
This traditional survey approach contrasts with
crowdsourcing, in which participants can contribute as
much or as little as they wish. Therefore, the participation
in crowdsourcing is ruled by highly unequal levels of
contributions, when most of the participants contribute very
little and a few active ones contribute a lot [8]. This
phenomenon of unequal participation has been first featured
by Horowitz, who found that the participation of the crowd
in content-generation online sites, such as YouTube,
Wikipedia and Yahoo Groups, is governed by the rule of
1%. The rule of 1% means that out of every 100 visitors
only 1% of them will create new content, of the remainder,
10% will refine and improve existing content while 89%
will just consume it [9].
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
CSCW ’14, February 1519, 2014, Baltimore, Maryland, USA.
Copyright 2014 ACM xxx-x-xxxx-xxxx-x/xx/xx...$15.00.
Crowdsourcing, one form of online content-generation, is
an open call for anyone to participate in a task online [10,
11, 12]. It has been used to engage people from urban
planning [13] to new product design and solving complex
scientific problems [14]. A crowdsourcing system invites a
crowd of people to help solve a problem defined by the
system owners [11]. The “crowd” here refers to an
undefined collection of people who participate in the open
call and the system owner or task initiator, i.e., the
crowdsourcer, can be any given entity, whether it is a
company, institution, nonprofit organization or an
individual. The term crowdsourcing is also used in contexts
in which the task is open only to a restricted group like
employees in a certain organization. Technology
companies, such as IBM and Microsoft, have been using
crowdsourcing to harvest ideas from employees to fuel
grassroot innovation processes within their companies [15].
When used in democratic processes, like policy-making,
crowdsourcing turns into a method for democratic
innovations [5]. Democratic innovations engage citizens in
democratic processes between the elections. Democratic
innovations involve a variety of methods ranging from
deliberative polls [16] and citizen assemblies to
crowdsourced policy-making. In crowdsourced policy-
making the citizens are invited to share their knowledge and
ideas for improving the policy. Iceland, for instance, used
crowdsourcing in its constitution reform in 2011 [2], and
Finland crowdsourced its off-road traffic law reform in
[17]. Crowdsourcing has also been used to extend the
governments’ capacities in data processing and analytics.
For instance, the State of Minnesota has run an initiative
called Minnesota’s Citizen Lake in which citizens were
invited to analyze and monitor the quality of the state’s
water resources [18].
Crowdsourcing has also been used for searching solutions
for complex societal challenges, such as predicting solar
flares. In this type of innovation challenges the participants
propose innovative solutions for solving the problem. The
SAVE Award launched by the White House is an example
of employing the knowledge and skills of an online
community to address a public concern, which in this case
is reducing the public budget [19].
In contrast to random sampling used in polls and surveys, in
crowdsourcing the participants self-select to work on the
solution to the problem defined by the crowdsourcing
system owner [20]. This means that the participants are not
invited randomly to participate, but they initiate the
participation themselves, leading to a biased sample the
crowd is not a representative sample of the larger public.
Crowdsourcing thus collides with the notion of equal
This contrast is illustrated in Figure 1, which presents the
rank-ordered distributions of the amounts of information
contributed in a survey using random sampling and in a
crowdsourcing system. The diagram on left illustrates the
distribution of contributions from a survey using random
sampling and fixed contributions [21]. The figure on right
illustrates the distribution of participant contributions to a
crowdsourcing system [17].
In order for crowdsourcing to work the crowd needs to be
large and at least some participants need to be
knowledgeable and motivated to self-select to participate in
problem solving [20]. At least in crowdsourced scientific
problems the best solutions tend to come from people in
technical and social marginality, who, thus, have different
perspectives and problem solving approaches compared to
the majority of the participants [22].
Figure 1. Comparing contributions between equal
representation (fixed contributions in survey) and in
crowdsourced knowledge search.
Altogether, crowdsourcing relies on the self-selection of
participants, leading to highly unequal levels of
contributions. Many of the best contributions come from
the most unrepresentative individuals in the crowd. The
contrasts between equal representation and crowdsourcing
are outlined in Table 1.
Typical rank-
distribution of
from sample
Finding non-
typical individuals
that provide non-
typical solutions
“The public
opinion” (poll
result) /Majority
ted knowledge
Table 1: Comparison of equal representation and
Because of the tension between the nature of crowdsourcing
as a method based on self-selection and the strive to equal
representation, an often heard objection against
crowdsourcing in democratic processes is the lack of
statistical representativeness of the participant crowd,
which leads to biased samples. In democracy this can be
problematic because the crowdsourced process may attract
only certain demographic groups, or groups that share a
certain political view. It is deceiving to consider the
crowdsourced input as the public opinion.
In this paper we have exposed the tension between the
ideals of equal representation and self-selection in
crowdsourcing. Crowdsourcing aims for large, diverse
participation, which is based on self-selection, and those are
the virtues of crowdsourcing. Equal representation, instead,
presumes equal representation (in the form of statistical
representativeness of the public) and equal contributions. If
crowdsourcing aims for equal representation, its virtues are
compromised and that can undermine the method’s
advantages. Therefore, crowdsourcing, even when applied
in democratic processes, shouldn’t attempt to follow the
norm of equal representation. Instead, crowdsourcing as a
method for participatory democracy should be cherished by
enhancing its virtues and developing the method using
those virtues, whether crowdsourcing is used in distributed
group-work or for large online crowds.
1. Aitamurto, T. Crowdsourcing for Democracy:
New Era In Policy Making. Publications of the
Committee for the Future, Parliament of Finland.
2. Landmore, H. Inclusive ConstitutionMaking: The
Icelandic Experiment. Journal of Political
Philosophy. 2014
3. Brabham, D. C. Crowdsourcing the public
participation process for planning projects.
Planning Theory, 8, 242262. 2009
4. Nelimarkka, M., Nonnecke B., Krishnan S.,
Aitamurto T., Catterson D., Crittenden C., Garland
C., Gregory C., Huang C.C., Newsom G., Patel J.,
Scott J., Goldberg K. Comparing Online Civic
Engagement Interfaces using the “Spectrum of
Public Participation” Framework. Conference on
Crowdsourcing for Politics and Policy. Oxford.
5. Smith, G. Democratic Innovations: Designing
Institutions for Citizen Participation. Cambridge
University Press. 2009
6. Bryman, A. Social Research Methods. Oxford
University Press. 2008.
7. Cherry, K. What is a representative sample? Retrieved from:
8. Salganik, M., and Levy, K. Wiki surveys: Open
and quantifiable social data collection. arXiv
preprint arXiv:1202.0500. 2012
9. Horowitz, B. Creators, Synthesizers, and
Consumers. 2006. Retrieved from:
10. Brabham, D. C. Crowdsourcing as a model for
problem solving an introduction and cases.
Convergence: the international journal of
research into new media technologies, 14(1), 75-
90. 2008
11. Doan, A., Ramakrishnan, R., Halevy, & A. Y.
Crowdsourcing systems on the world-wide web.
Communications of the ACM, 54(4), 86-96. 2011
12. Howe, J. Crowdsourcing: How the power of the
crowd is driving the future of business. Random
House, 223-246. 2008.
13. Brabham, D. C., Sanchez, T. W., & Bartholomew,
K. Crowdsourcing public participation in transit
planning: preliminary results from the next stop
design case. In TRB 89th Annual Meeting
Compendium. 2010
14. Aitamurto, T., Leiponen, A., & Tee, R. The
Promise of idea crowdsourcingBenefits,
contexts, limitations. White paper for Nokia Ideas
Project. 2011.
15. Bailey, B., and Horvitz, E. What's your idea? a
case study of a grassroots innovation pipeline
within a large software company. Proceedings of
the SIGCHI Conference on Human Factors in
Computing Systems. ACM, 2010.
16. Fishkin, J. When the people speak: Deliberative
democracy and public consultation. Oxford
University Press. 2009
17. Aitamurto, T., and Landemore, & H. Democratic
participation and deliberation in crowdsourced
legislative processes: The case of the law on off-
road traffic in Finland. 6th Conference on
Communities and Technologies. 2013
18. Nambisan S., and Nambisan P. Engaging Citizens
in Co-Creation in Public Services: Lessons
Learned and Best Practices. IBM Center for The
Business of Government. 2013.
19. Long, E. Administration announces finalists in
cost-cutting contest.
2009. Retrieved from:
20. Afuah, A. N., and Tucci, C. L. Crowdsourcing as a
solution to distant search. Academy of
Management Review 37(3). 355375. 2012
21. Saunila, M. Innovation capability for SME
success: Perspectives of financial and operational
performance, Journal of Advances in Management
Research, Vol. 11, No. 2, pp. 163-175. 2014
22. Jeppesen, L. B., and Lakhani, K. R. Marginality
and problem-solving effectiveness in broadcast
search. Organization Science: Articles in Advance,
21(5), 10161033. 2010
... Yet, Urbinati denies this incompatibility to be the only legacy of 18th century's political philosophy when it comes to the idea of representation. 1 In supporting her claim for a 'democratic under-standing of representation' she draws on Condorcet's Plan de Constitution submitted to the French National Assembly in 1793. Condorcet's proposal, eventually rejected by both his fellow Girondins and the Jacobins, contains what Urbinati describes as 'an institutional order that is one of the most democratically advanced and imaginative Europe has produced in the last two centuries' (2004: 56): ...
... This is what most of the current online crowd-civic platforms provide, but what we gather in this case is a 'self-selected microcosm of deliberators', highly engaged and yet, far from being representative of the broader population (so we would be violating the principle of political equality). Tanja Aitamurto et al. (2014) have also highlighted the tension between the norm of equal representation in democracy and the self-selection bias of crowdsourcing, suggesting that 'crowdsourcing shouldn't strive for statistical representativeness of the public, otherwise the virtues of crowdsourcing would be compromised and its benefits in crowd work would not be achieved.' (2014: 1). ...
Full-text available
This paper aims at connecting democratic theory with civic technologies in order to highlight the links between some theoretical tensions and trilemmas and design trade-offs. First, it reviews some tensions and trilemmas raised by political philosophers and democratic theorists. Second, it considers both the role and the limitations of civic technologies in mitigating these tensions and trilemmas. Third, it proposes to adopt a meso-level approach, in between the macro-level of democratic theories and the micro-level of tools, to situate the interplay between people, digital technologies, and data.
... In contrast, designed sampling reflects the appeal of purposeful, structured inclusion, but at a cost to voluntary participation. These tensions between design and volunteerism and between representatives selected to be exceptional or broadly inclusive will not be new to social and political scientists-but they manifest anew, for example, in debates about crowdsourcing today [9,137,146]. Finally, the case highlights that the inclusiveness of a representative sample is no guarantee of power over how data are used. ...
Full-text available
Representativeness is a foundational yet slippery concept. Though familiar at first blush, it lacks a single precise meaning. Instead, meanings range from typical or characteristic, to a proportionate match between sample and population, to a more general sense of accuracy, generalizability, coverage, or inclusiveness. Moreover, the concept has long been contested. In statistics, debates about the merits and methods of selecting a representative sample date back to the late 19th century; in politics, debates about the value of likeness as a logic of political representation are older still. Today, as the concept crops up in the study of fairness and accountability in machine learning, we need to carefully consider the term's meanings in order to communicate clearly and account for their normative implications. In this paper, we ask what representativeness means, how it is mobilized socially, and what values and ideals it communicates or confronts. We trace the concept's history in statistics and discuss normative tensions concerning its relationship to likeness, exclusion, authority, and aspiration. We draw on these analyses to think through how representativeness is used in FAccT debates, with emphasis on data, shift, participation, and power.
... Given its potential to advantage participants over non-participating citizens, researchers have long been concerned with the possible distributional aspects of joint citizen-government production of services (for example, Aitamurto et al. 2014;Auer 2011;Bovaird and Loeffler 2015;Verschuere et al. 2012;Voorberg et al. 2015). In this article we seek to extend this inquiry by examining data from individual citizens regarding their use of 311, to assess whether they facilitate a representative form of crowdsourcing. ...
Full-text available
This article examines the citizen representativeness of crowdsourcing achieved through 311 systems—the non-emergency and quality of life service request reporting systems used by local governments. Based on surveys of San Francisco residents conducted in 2011, 2013, and 2015, our findings suggest that no systematic biases exist in participation rates across a range of socio-economic indicators. In addition, the findings provide evidence that participation may be responding positively to the city’s responsiveness, thus creating a self-reinforcing process that benefits an increasingly diverse and representative body of users. This inquiry builds on earlier studies of Boston and San Francisco that show that 311 systems did not bias response to traditionally disadvantaged groups (lower socioeconomic status or racial/ethnic minorities) at the demand level nor from high-volume users.
Smart Cities evolve into complex and pervasive urban environments with a citizens’ mandate to meet sustainable development goals. Repositioning democratic values of citizens’ choices in these complex ecosystems has turned out to be imperative in an era of social media filter bubbles, fake news and opportunities for manipulating electoral results with such means. This paper introduces a new paradigm of augmented democracy that promises actively engaging citizens in a more informed decision-making augmented into public urban space. The proposed concept is inspired by a digital revive of the Ancient Agora of Athens, an arena of public discourse, a Polis where citizens assemble to actively deliberate and collectively decide about public matters. The core contribution of the proposed paradigm is the concept of proving witness presence: making decision-making subject of providing secure evidence and testifying for choices made in the physical space. This paper shows how the challenge of proving witness presence can be tackled with blockchain consensus to empower citizens’ trust and overcome security vulnerabilities of GPS localization. Moreover, a novel platform for collective decision-making and crowd-sensing in urban space is introduced: Smart Agora. It is shown how real-time collective measurements over citizens’ choices can be made in a fully decentralized and privacy-preserving way. Witness presence is tested by deploying a decentralized system for crowd-sensing the sustainable use of transport means. Furthermore, witness presence of cycling risk is validated using official accident data from public authorities compared against wisdom of the crowd. The paramount role of dynamic consensus, self-governance and ethically aligned artificial intelligence in the augmented democracy paradigm is outlined.
Conference Paper
Full-text available
The Workshop on ‘Linked Democracy: Artificial Intelligence for Democratic Innovation’ is one of the official workshops of the International Joint Conference on Artificial Intelligence (IJCAI 2017) held in Melbourne (19-26 August 2017). The goal of this workshop is to provide a multidisciplinary forum to address questions such as: How to model the interactions between people, data, and digital tools that create new spaces and forms of civic action in the digital era? How to analyse emerging properties and types of knowledge in these contexts? How to design socio-technical systems that effectively leverage data and knowledge for deliberation (or other types of participation) and collective decision making? Can we design the meta-rules of the emergent ecosystems? The Workshop brings together participants from universities and research centers in Australia, New Zealand, Spain, Brazil, Israel, UK, and the USA. The workshop received 10 submissions, covering a number of different areas in AI (e.g. multi-agent systems and machine learning), economics, political sciences, and law. All submitted versions were reviewed by at least two members of the Program Committee. These proceedings finally include nine of these papers and an invited keynote speech by Patrick Keyzer. We sincerely thank the Program Committee members for reviewing all submitted papers and providing feedback to improve their revised versions. We are also grateful to the IJCAI 2017 chairs (Program Chair Carles Sierra, Workshops Chair Daniele Magazzeni and Local Arrangements co-Chair Andy Song) for their support in preparing this workshop. Last but not least, we would like to thank the participants who submitted their papers and afterwards produced the revised versions that are now composing these proceedings.
Conference Paper
Full-text available
This paper reports on a pioneering case study of a legislative process open to the direct online participation of the public. The empirical context of the study is a crowdsourced off-road traffic law in Finland. On the basis of our analysis of the user content generated to date and a series of interviews with key participants, we argue that the process qualifies as a promising case of deliberation on a mass-scale. This case study will make an important contribution to the understanding of online methods for participatory and deliberative democracy. The preliminary findings indicate that there is deliberation in the crowdsourcing process, which occurs organically (to a certain degree) among the participants, despite the lack of incentives for it. Second, the findings strongly indicate that there is a strong educative element in crowdsourced lawmaking process, as the participants share information and learn from each other. The peer-learning aspect could be made even stronger through the addition of design elements in the process and on the crowdsourcing software.
Full-text available
Purpose – The relationship between overall innovation and innovation capability, and performance has been a topic of several earlier studies. However, the effects of the aspects of innovation capability on performance of a firm have stayed unfamiliar. The purpose of this paper is to study the relationship between organizational innovation capability and firm performance. The study contributes to the current understanding by presenting the important aspects of organizational innovation capability that affect firm performance. The effects are studied to both financial and operational performance. Design/methodology/approach – The approach of this study is quantitative. The data used to test the hypotheses were gathered from Finnish small- and medium-sized enterprises (SMEs) with a web-based questionnaire. The sample covered 2,400 SMEs employing 11-249 persons and having a revenue of two to 50 Meuro. The sample was randomly selected. Findings – The findings showed that three aspects of innovation capability, namely ideation and organizing structures, participatory leadership culture, and know-how development, has some effect on different aspects of firm performance. Surprisingly, the aspects of innovation capability were found to be more influential to the financial performance than operational performance. Practical implications – The paper contains suggestions for improving performance through developing innovation capability. The paper aims to support practice in two ways. First, organizations can identify aspects of innovation capability that affect operational and financial performance. In that way, organizations can benefit the results by applying these aspects in their everyday operations. Second, the results of the paper may help professionals to begin to understand that leveraging innovation capability may improve an organization’s performance. Originality/value – Previous research has often either concentrated on innovation capability as a one dimension without studying the relationship aspect by aspect or studying only the effects of one aspect of innovation capability. The results of the study take one step further by investigating the relationship of multiple aspects of innovation capability and firm performance.
Full-text available
Idea crowdsourcing has become a hype term associated with unrealistic expectations for innovation and unclear understanding of its requirements and challenges. This paper intends to bring together and evaluate the key insights that shed light on the phenomenon conceptually or empirically and help us qualify and attenuate the expectations. What do we know about crowdsourcing as of early 2011? We review the concept of idea crowdsourcing, focusing particularly on findings reported in the academic literature. Going beyond studies on the sources of inputs for innovation, we review research published in journals, conferences, and working papers in a variety of fields and disciplines, including strategic management, innovation, information systems,communication, and marketing. Overall, we find that existing work clearly indicates that crowdsourcing potentially contributes significantly to innovation. When applied in the right circumstances, crowdsourcing can deliver considerable benefits to firms in terms of inputs into innovation. However, success requires careful analysis of goals, problem-solving environment, required expertise, firm‟s strategies, complementary activities and capabilities, and the competitive environment. We also attempt to evaluate the direct costs and benefits of crowdsourcing with the limiteddata available from a few well-documented crowdsourcing platforms. Meanwhile, there islittle empirical research on the indirect strategic value of crowdsourcing such as brandleverage or innovation real options. We see this as a potential avenue for further research.Future research should also seek to quantify the impact of crowdsourcing compared to and complementing more traditional methods of innovation. Furthermore, research on companies absorptive capacity to utilize crowdsourced ideas and to co-create and refine ideas furtherwith customers would benefit both academia and industry.Our report is inspired by Nokia‟s recent and ongoing experimen ts in this area. The paper isstructured as follows. In section 1, we define the concept of crowdsourcing, position it inrelation to other literatures, and discuss it in the wider context of open innovation. Section 2analyzes how crowdsourcing can be utilized and implemented. In section 3 we investigatewhat existing evidence can tell us about the benefits of crowdsourcing. Section 4 looks at thecontext in which crowdsourcing can best be utilized. In section 5 we discuss possibledownsides and limitations of crowdsourcing and in section 6 how crowdsourcing might becombined with more traditional methods. Section 7 assesses the crowdsourcing business case.In section 8 we look at the methods used in crowdsourcing research. Finally, in section 9 weassess open research questions and conclude in section 10.
Can we design institutions that increase and deepen citizen participation in the political decision making process? At a time when there is growing disillusionment with the institutions of advanced industrial democracies, there is also increasing interest in new ways of involving citizens in the political decisions that affect their lives. This book draws together evidence from a variety of democratic innovations from around the world, including participatory budgeting in Brazil, Citizens’ Assemblies on Electoral Reform in Canada, direct legislation in California and Switzerland and emerging experiments in e-democracy. The book offers a rare systematic analysis of this diverse range of democratic innovations, drawing lessons for the future development of both democratic theory and practice.
We argue that under certain circumstances crowdsourcing transforms distant search into local search, improving the efficiency and effectiveness of problem solving. Under such circumstances a firm may choose to crowdsource problem solving rather than solve the problem internally or contract it to a designated supplier. These circumstances depend on the characteristics of the problem, the knowledge required for the solution, the crowd, and the solutions to be evaluated.
Public involvement is a central concern for urban planners, but the challenge for planners is how best to implement such programs, given many difficulties inherent in the typical public involvement process. The medium of the Web enables us to harness collective intellect among a population in ways face-to-face planning meetings cannot. This article argues that the crowdsourcing model, a successful, Web-based, distributed problem solving and production model for business, is an appropriate model for enabling the citizen participation process in public planning projects. This article begins with an exploration of the challenges of public participation in urban planning projects, particularly in the harnessing of creative solutions. An explanation of the theories of collective intelligence and crowd wisdom follows, arguing for the medium of the Web as an appropriate technology for harnessing far-flung genius. An exploration of crowdsourcing in a hypothetical neighborhood planning example, along with a consideration of the challenges of implementing crowdsourcing, concludes the article.