Conference Paper

Learning while Earning? A Literature Review and Case Study on Learning Opportunities in Crowdwork

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Crowdsourcing has emerged as a pivotal force driving innovation and problem-solving in today’s digital landscape. Yet, to sustain this momentum and meet the dynamic demands of the labor market, prioritizing learning opportunities within crowdsourcing is imperative. Current platforms face criticism for neglecting skill development, leaving it primarily the workers’ responsibility. This article delves into how crowdsourcing platforms can enhance support for workers’ learning. Through a structured literature review, we synthesize learning approaches within crowdwork literature in a conceptual mapping. We then practically apply theoretical findings to the successful platform “Kaggle”, examining how the platform supports data literacy learning. Through our investigations, we offer both theoretical insights and practical observations, aiming to catalyze further exploration and enhancement of learning opportunities within crowdwork.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
This study examines how ChatGPT affects the labor market. I first thoroughly analyzed the prior research that has been done on the subject in order to start understanding how ChatGPT and other AI-related services are influencing the labor market. Using the supply and demand model, I then assess ChatGPT's impact. This paper examines this innovation's short- and long-term effects on the labor market, concentrating on its challenges and opportunities. Furthermore, I employ a text-mining approach to extract various tasks from the International Standard Occupation Classification to present a comprehensive list of occupations most sensitive to ChatGPT.
Article
Full-text available
Crowdsourcing platform owners and operators constantly search for ways to improve contestant performance. One novel proposal for improving performance is the introduction of an online community to the crowdsourcing contest platform. However, research regarding the potential benefits of an online community on such platforms is unclear. Furthermore, prior research often assumes the single dimensionality of prior experience, whose impacts on crowdsourcing performance are also inconclusive. Building on knowledge collaboration and cognitive diversity research, we model the direct effects of introducing an online community on contestant performance and the moderating effects of the amount of experience and experience diversity. Leveraging a natural quasi‐experiment in a large crowdsourcing contest platform, we collected 24 months of contestant data to test our hypotheses. Our propensity score matching and difference in differences analysis demonstrated that contestants' performance (winning contests and crowdsourcing income) increases significantly with the presence of an online community. Additionally, the positive effect of an online community on performance is more pronounced for contestants with less experience and those with more diverse experience. Our findings provide insights into the causality of incorporating an online community and inform community investment decisions for crowdsourcing contest platforms.
Conference Paper
Full-text available
Solving real-world problems with innovative and novel artifacts is at the core of design science research (DSR). Given DSR’s emphasis on a strong connection to the real-world, artifacts for solving a particular problem may not only be described in extant literature, but also exist in practice. This is particularly the case for software artifacts. Therefore, DSR scholars need to explore the state of the art and demonstrate the novelty of their software artifact relative to existing artifacts in research and practice. However, while methodological guidance for conducting literature reviews is abundant, there is little guidance on how to review software artifacts from practice. This paper takes a first step toward addressing this gap by proposing and illustrating a seven-step method for reviewing software artifacts from practice. Our research provides actionable guidance for DSR scholars on how to support the claim that their software artifact constitutes a substantial contribution to knowledge.
Article
Full-text available
This work intends to lay the foundations for a theoretical framework of citizen science combining social and organizational implications with the support of information technologies. The proposed theoretical framework moves towards a shared and common research process between experts and citizens to deal with environmental and social challenges. The role and capacity of online communities is explored and their engagement capacity by means of web-based digital platforms supporting crowdsourcing activities. In this contribution, authors highlight the most common practices, methods and issues of citizen science approaches adopted from multidisciplinary application fields to obtain insights for designing a new participative approach for organizational studies. To reach this goal, authors illustrate the results of a systematic meta-review analysis, consisting of an accurate selection and revision of journal review articles in order to highlight concepts, methods, research design approaches and tools adopted in citizen science approaches.
Article
Full-text available
Self-correction for crowdsourced tasks is a two-stage setting that allows a crowd worker to review the task results of other workers; the worker is then given a chance to update their results according to the review.Self-correction was proposed as a complementary approach to statistical algorithms, in which workers independently perform the same task.It can provide higher-quality results with low additional costs. However, thus far, the effects have only been demonstrated in simulations, and empirical evaluations are required.In addition, as self-correction provides feedback to workers, an interesting question arises: whether perceptual learning is observed in self-correction tasks.This paper reports our experimental results on self-corrections with a real-world crowdsourcing service.We found that:(1) Self-correction is effective for making workers reconsider their judgments.(2) Self-correction is effective more if workers are shown the task results of higher-quality workers during the second stage.(3) A perceptual learning effect is observed in some cases. Self-correction can provide feedback that shows workers how to provide high-quality answers in future tasks.(4) A Perceptual learning effect is observed, particularly with workers who moderately change answers in the second stage. This suggests that we can measure the learning potential of workers.These findings imply that requesters/crowdsourcing services can construct a positive loop for improved task results by the self-correction approach.However, (5) no long-term effects of the self-correction task were transferred to other similar tasks in two different settings.
Article
Full-text available
Kaggle is a data modeling competition service, where participants compete to build a model with lower predictive error than other participants. Several years ago they released a simplified service that is ideal for instructors to run competitions in a classroom setting. This paper describes the results of an experiment to determine if participating in a predictive modeling competition enhances learning. The evidence suggests it does. In addition, students were surveyed to examine if the competition improved engagement and interest in the class.
Article
Full-text available
Purpose The digitalization of society results in challenges and opportunities for learning and education. This paper describes exemplary transformations from current to future practices. It illustrates multi-dimensional aspects of learning which complement and transcend current frameworks of learning focused on schools. While digital technologies are necessary for these transformations, they are not sufficient. The paper briefly illustrates the applicability of the conceptual framework to the COVID-19 pandemic. It concludes that design opportunities and design trade-offs in relation to digital technologies and learning should be explored by envisioning the cultural transformation that are desirable for making learning a part of life. Design/methodology/approach This paper is based on the work conducted at the symposium “Rethinking and Reinventing Learning, Education, and Collaboration in the Digital Age—From Creating Technologies to Transforming Cultures” that took place in Engeltofta outside of Gävle, Sweden in September 2019. The symposium invited scholars in collaborative analysis of design opportunities and design trade-offs in relation to digital technologies and learning and explored design strategies for systematically and proactively increasing digital technology's contributions to learning and collaborating. The paper first provides a condensed introduction of a conceptual framework summarizing current practices, their problems and promising alternatives. Multi-dimensional aspects of learning and lifelong learning will be briefly described as promising future alternatives to school learning. Examples of transformative practices are supporting the major argument of the paper that creating new technologies is an important prerequisite to address the fundamental challenge of transforming cultures. The unanticipated but fundamental event of the occurrence of COVID-19 will be briefly described to provide further evidence for the need and the applicability of our conceptual framework for rethinking and reinventing learning, education and collaboration in the digital age. Findings The paper provides a condensed introduction of a conceptual framework summarizing current practices, their problems and promising alternatives. The framework includes multi-dimensional aspects of learning and lifelong learning as a promising future alternative to a focus on school learning. Originality/value This paper describes exemplary transformations from current to future practices. It illustrates multi-dimensional aspects of learning which complement and transcend current frameworks of learning focused on schools.
Article
Full-text available
This article analyses crowdwork platforms where various forms of digital labour are outsourced to digital workers across the globe. The labour of these workers is, among other things, a crucial component in the production, development and support of artificial intelligence. Crowdwork platforms are an extreme example of new forms of automated measurement, management and control of labour allowing, in turn, for the creation of hyperflexible and highly scalable workforces. Particularly on so-called microtask platforms, work is characterised by decomposition, standardisation, automated management and surveillance, as well as algorithmically organised cooperation between a great number of workers. Analysing these platforms as a paradigmatic example of an emerging digital Taylorism, the article goes on to argue that this allows the platforms to assemble a deeply heterogeneous set of workers while bypassing the need to spatially and subjectively homogenise them. These platforms create a global on-demand workforce, working in their private homes or Internet cafes. As a result, crowdwork taps into labour pools hitherto almost inaccessible to wage labour. The second part of the article investigates this tendency by looking at two sets of workers: women shouldering care responsibilities, who now can work on crowdwork platforms while performing domestic labour, as well as digital workers in the Global South. While there are clear specifics of digital crowdwork, it is also an expression of broader transformations within the world of work, concerning, for example, new forms of algorithmic management just as the return of very old forms of exploitation such as the piece wage.
Book
Full-text available
The quickly evolving platform-based reorganisation of work comes with a whole set of opportunities and risks. First, this demands a precise terminology and analysis of its functionalities. Secondly, a broad public debate is needed about what values society deems worth protecting and what constitutes decent work. Finally, it is the role of government to enforce the agreed-upon values through regulations and ensure that labour laws also apply in the digital realm. However, due to the structure of the platforms, this will be a serious challenge. The present publication offers first and foremost a categorisation of the different types of commercial digital labour platforms. It also discusses the particular characteristics and challenges of the different categories and maps how to tackle them politically.
Conference Paper
Full-text available
Learning from captioned foreign language videos is highly effective, but the availability of such videos is limited. By using speech-to-text technology to generate partially correct transcripts as a starting point, we see an opportunity for learners to build accurate foreign language captions while learning at the same time. We present a system where learners correct captions using automatic transcription and machine-generated suggested alternative words for scaffolding. In a lab study of 49 participants, we found that compared to watching the video with accurate caption, learning and quality of experience were not significantly impaired by the secondary caption correction task using interface designs either with or without scaffolding from speech-to-text generated alternative words. Nevertheless, aggregating corrections reduced word error rate from 19% to 5.5% without scaffolding from suggested-alternatives, and 1.8% with scaffolding. Feedback from participants suggest that emphasizing the learning community contribution aspect is important for motivating learners and reducing frustration.
Conference Paper
Full-text available
This paper reports findings of a survey exploring how crowdworkers develop their knowledge and skills in the course of their work on digital platforms. The focus is on informal learning initiated and self-regulated by crowdworkers: engaging in challenging tasks; studying professional literature/online resources; sharing knowledge and collaborating with others. The survey was run within two platforms representing two types of crowdwork – microwork (CrowdFlower) and online freelancing (Upwork). The survey uncovered evidence for considerable individual and social learning activity within both types of crowdwork. Findings suggest that both microwork and online freelancing are learning-intensive and both groups of workers are learning-oriented and self-regulated. Crowdwork is a growing form of employment in developed and developing countries. Improved understanding of learning practices within crowdwork would inform the design of crowdwork platforms; empower crowdworkers to direct their own learning and work; and help platforms, employers, and policymakers enhance the learning potential of crowdwork.
Article
Full-text available
Online citizen science projects have demonstrated their usefulness for research, however little is known about the potential benefits for volunteers. We conducted 39 interviews (28 volunteers, 11 researchers) to gain a greater understanding of volunteers' motivations, learning and creativity (MLC). In our MLC model we explain that participating and progressing in a project community provides volunteers with many indirect opportunities for learning and creativity. The more aspects that volunteers are involved in, the more likely they are to sustain their participation in the project. These results have implications for the design and management of online citizen science projects. It is important to provide users with tools to communicate in order to supporting social learning, community building and sharing.
Article
Full-text available
Digital competence is one of the eight key competences for lifelong learning and is essential for participation in our increasingly digitalised society. It is therefore necessary to understand and define what digital competence is and consists of. The paper discusses various aspects of digital competence firstly differentiating it from other similar or overlapping concepts, then discussing the implication of the historic evolution of the term, finally detailing the digital competence framework in its constituting parts.
Article
Full-text available
Traditionally, the term ‘crowd’ was used almost exclusively in the context of people who self-organized around a common purpose, emotion or experience. Today, however, firms often refer to ‘crowds’ in discussions of how collections of individuals can be engaged for organizational purposes. Crowdsourcing, the use of information technologies to outsource business responsibilities to crowds, can now significantly influence a firm’s ability to leverage previously unattainable resources to build competitive advantage. Nonetheless, many managers are hesitant to consider crowdsourcing because they don’t understand how its various types can add value to the firm. In response, we explain what crowdsourcing is, the advantages it offers and how firms can pursue crowdsourcing. We begin by formulating a crowdsourcing typology and show how its four categories (crowd-voting, micro-task, idea and solution crowdsourcing) can help firms develop ‘crowd capital’, an organizational-level resource harnessed from the crowd. We then present a three-step process model for generating crowd capital. Step one includes important considerations that shape how a crowd is to be constructed. Step two outlines the capabilities firms need to develop to acquire and assimilate resources (e.g., knowledge, labor, funds) from the crowd. Step three addresses key decision-areas that executives need to address to effectively engage crowds. http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=1919614#show2489648
Conference Paper
Full-text available
Crowdsourcing has become a popular approach for capitalizing on the potential of large and open crowds of people external to the organization. While crowdsourcing as a phenomenon is studied in a variety of fields, research mostly focuses on isolated aspects and little is known about the integrated design of crowdsourcing efforts. We introduce a socio-technical systems perspective on crowdsourcing, which provides a deeper understanding of the components and relationships in crowdsourcing systems. By considering the function of crowdsourcing systems within their organizational context, we develop a typology of four distinct system archetypes. We analyze the characteristics of each type and derive a number of design requirements for the respective system components. The paper lays a foundation for IS-based crowdsourcing research, channels related academic work, and helps guiding the study and design of crowdsourcing information systems.
Article
Full-text available
Crowdsourcing is one of the emerging Web 2.0 based phenomenon and has attracted great attention from both practitioners and scholars over the years. It can facilitate the connectivity and collaboration of people, organizations, and societies. We believe that Information Systems scholars are in a unique position to make significant contributions to this emerging research area and consider it as a new research frontier. However, so far, few studies have elaborated what have been achieved and what should be done. This paper seeks to present a critical examination of the substrate of crowdsourcing research by surveying the landscape of existing studies, including theoretical foundations, research methods, and research foci, and identifies several important research directions for IS scholars from three perspectives—the participant, organization, and system—and which warrant further study. This research contributes to the IS literature and provides insights for researchers, designers, policy-makers, and managers to better understand various issues in crowdsourcing systems and projects.
Conference Paper
Full-text available
Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd work will fail to achieve its potential, focusing on assembly-line piecework. Can we foresee a future crowd workplace in which we would want our children to participate? This paper frames the major challenges that stand in the way of this goal. Drawing on theory from organizational behavior and distributed computing, as well as direct feedback from workers, we outline a framework that will enable crowd work that is complex, collaborative, and sustainable. The framework lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.
Conference Paper
Governments and public institutions increasingly embrace digital opportunities to involve citizens in public issues and decision making. While public participation is generally seen as an important and promising venture, the design of the participation processes and the utilized digital infrastructure poses challenges, especially to the public sector. Instead of limiting conceptual guidance and exchange to one domain, we therefore develop a taxonomy for digital involvement projects that unites the domains of eparticipation, citizen science and crowd-X. Embedded in a design science research approach, we follow an iterative design process to elaborate the key characteristics of a digital involvement project based on the participation process, its individuals and digital infrastructure. Through evaluating the artifact in a focus group with domain practitioners, we find support for the usefulness of our taxonomy and its ability to provide guidance and a basis for discussion of digital involvement projects across domains.
Article
Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles (n = 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT's intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003-about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.
Article
The paper analyzes social interactions among crowdworkers on Discord. The explored crowdtesting platform uses this social media platform to offer their crowdworkers various opportunities for work-related and private communication and to host events that encourage learning practices. The paper investigates to what extent the interactions on Discord can be analyzed as social learning practices and can be understood as a specific form of solidarity among crowdworkers. In an explorative online ethnographic study, two learning related channels of the platform company’s Discord server were observed: the question channel in which testers can ask for help, and the quiz channel in which a testing related quiz event takes place. Additionally, interviews with moderators and crowdtesters were conducted. The observation of learning practices on Discord makes clear that the social media tool is mostly used by testers for situational and functional information exchange like helping each other with bug classifications or solving technical problems. Testers mostly provide each other with brief information that can directly be applied in the work context. Further information are mainly shared by moderators that offer supplementary explanations as a possibility to self-help. The study highlights that a form of weak cooperative solidarity emerges among testers as they support each other via Discord to fulfill individual work tasks. This differs from resistant solidarity in other contexts of platform work, because in the observed case, platform workers’ solidarity is not directed against the platform company.
Article
Workers in crowdsourcing are evolving from one-off, independent micro-workers to on-demand collaborators with a long-term orientation. They were expected to collaborate as transient teams to solve more complex, non-trivial tasks. However, collaboration as a team may not be as prevalent as possible, given the lack of support for synchronous collaboration and the "competition, collaboration but transient" nature of crowdsourcing. Aiming at unfolding how individuals collaborate as a transient team and how such teamwork can affect an individual's long-term success, this study investigates the individuals' collaborations on Kaggle, a crowdsourcing contest platform for data analysis. The analysis reveals a growing trend of collaborating as a transient team, which is influenced by contest designs like complexity and reward. However, compared with working independently, the surplus of teamwork in a contest varies over time. Furthermore, the teamwork experience is beneficial for individuals in the short term and long term. Our study distinguishes the team-related intellectual capital and solo-related intellectual capital, and finds a path dependency effect for the individual to work solely or collectively. These findings allow us to contribute insights into the collaborative strategies for crowd workers, contest designers, and platform operators like Kaggle.
Article
One of the most significant differences of M5 over previous forecasting competitions is that it was held on Kaggle, an online platform for data scientists and machine learning practitioners. Kaggle provides a gathering place, or virtual community, for web users who are interested in the M5 competition. Users can share code, models, features, and loss functions through online notebooks and discussion forums. Here, we study the social influence of this virtual community on user behavior in the M5 competition. We first research the content of the M5 virtual community by topic modeling and trend analysis. Further, we perform social media analysis to identify the potential relationship network of the virtual community. We study the roles and characteristics of some key participants who promoted the diffusion of information within the M5 virtual community. Overall, this study provides in-depth insights into the mechanism of the virtual community’s influence on the participants and has potential implications for future online competitions.
Article
Workers are the most important resource in crowdsourcing. However, only investing in worker-centric needs, such as skill improvement, often conflicts with short-term platform-centric needs, such as task throughput. This paper studies learning strategies in task assignment in crowdsourcing and their impact on platform-centric needs. We formalize learning potential of individual tasks and collaborative tasks, and devise an iterative task assignment and completion approach that implements strategies grounded in learning theories. We conduct experiments to compare several learning strategies in terms of skill improvement, and in terms of task throughput and contribution quality. We discuss how our findings open new research directions in learning and collaboration.
Article
The growing number of tools for data novices are not designed with the goal of learning in mind. This paper proposes a set of pedagogical design principles for tool development to support data literacy learners. We document their use in the creation of three digital tools and activities that help learners build data literacy, showing design decisions driven by our pedagogy. Sketches students created during the activities reflect their adeptness with key data literacy skills. Based on early results, we suggest that tool designers and educators should orient their work from the outset around strong pedagogical principles.
Book
Do you want to create data analysis reports without writing a line of code? This book introduces SAS Studio, a free data science web browser-based product for educational and non-commercial purposes. The power of SAS Studio comes from its visual point-and-click user interface that generates SAS code. It is easier to learn SAS Studio than to learn R and Python to accomplish data cleaning, statistics, and visualization tasks. The book includes a case study about analyzing the data required for predicting the results of presidential elections in the state of Maine for 2016 and 2020. In addition to the presidential elections, the book provides real-life examples including analyzing stocks, oil and gold prices, crime, marketing, and healthcare. You will see data science in action and how easy it is to perform complicated tasks and visualizations in SAS Studio. You will learn, step-by-step, how to do visualizations, including maps. In most cases, you will not need a line of code as you work with the SAS Studio graphical user interface. The book includes explanations of the code that SAS Studio generates automatically. You will learn how to edit this code to perform more complicated advanced tasks. The book introduces you to multiple SAS products such as SAS Viya, SAS Analytics, and SAS Visual Statistics. You will: • Become familiar with SAS Studio IDE • Understand essential visualizations • Know the fundamental statistical analysis required in most data science and analytics reports • Clean the most common data set problems • Use linear progression for data prediction • Write programs in SAS • Get introduced to SAS-Viya, which is more potent than SAS studio
Article
Peer production projects involve people in many tasks, from editing articles to analyzing datasets. To facilitate mastery of these practices, projects offer a number of learning resources, ranging from project-defined FAQsto individually-oriented search tools and communal discussion boards. However, it is not clear which project resources best support participant learning, overall and at different stages of engagement. We draw onSørensen's framework of forms of presence to distinguish three types of engagement with learning resources:authoritative, agent-centered and communal. We assigned resources from the Gravity Spy citizen-science into these three categories and analyzed trace data recording interactions with resources using a mixed-effects logistic regression with volunteer performance as an outcome variable. The findings suggest that engagement with authoritative resources (e.g., those constructed by project organizers) facilitates performance initially. However, as tasks become more difficult, volunteers seek and benefit from engagement with their own agent-centered resources and community-generated resources. These findings suggest a broader scope for the design of learning resources for peer production
Article
Purpose This paper aims to explore workplace learning practices within two types of crowdwork – microwork and online freelancing. Specifically, the paper scopes and compares the use of workplace learning activities (WLAs) and self-regulatory learning (SRL) strategies undertaken by microworkers (MWs) and online freelancers (OFs). We hypothesised that there may be quantitative differences in the use of WLAs and SRL strategies within these two types of crowdwork, because of the underpinning differences in the complexity of tasks and skill requirements. Design/methodology/approach To test this hypothesis, a questionnaire survey was carried out among crowdworkers from two crowdwork platforms – Figure Eight (microwork) and Upwork (online freelancing). Chi-square test was used to compare WLAs and SRL strategies among OFs and MWs. Findings Both groups use many WLAs and SRL strategies. Several significant differences were identified between the groups. In particular, moderate and moderately strong associations were uncovered, whereby OFs were more likely to report (i) undertaking free online courses/tutorials and (ii) learning by receiving feedback. In addition, significant but weak or very weak associations were identified, namely, OFs were more likely to learn by (i) collaborating with others, (ii) self-study of literature and (iii) making notes when learning. In contrast, MWs were more likely to write reflective notes on learning after the completion of work tasks, although this association was very weak. Originality/value The paper contributes empirical evidence in an under-researched area – workplace learning practices in crowdwork. Crowdwork is increasingly taken up across developed and developing countries. Therefore, it is important to understand the learning potential of this form of work and where the gaps and issues might be. Better understanding of crowdworkers’ learning practices could help platform providers and policymakers to shape the design of crowdwork in ways that could be beneficial to all stakeholders.
Article
This paper develops a set of innovative tools for labor market intelligence by applying machine learning techniques to web vacancies on the Italian labor market. Our approach allows to calculate, for each occupation, the different types of skills required by the market alongside a set of relevant variables such as region, sector, education and level of experience. We construct a taxonomy for skills and map it into the recently developed ESCO classification system. We subsequently develop measures of the relevance of soft and hard skills and we analyze their detailed composition. We apply the dataset constructed to the debate on computerization of work. We show that soft and digital skills are related to the probability of automation of a given occupation and we shed some light on the complementarity/substitutability of hard and soft skills.
Article
Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.
Conference Paper
Audio transcription is an important task for making content accessible to people who are deaf or hard of hearing. Much of the transcription work is increasingly done by crowd workers, people online who pick up the work as it becomes available often in small bits at a time. Whereas work typically provides a ladder for skill development -- a series of opportunities to acquire new skills that lead to advancement -- crowd transcription work generally does not. To demonstrate how crowd work might create a skill ladder, we created Scopist, a JavaScript application for learning an efficient text-entry method known as stenotype while doing audio transcription tasks. Scopist facilitates on-the-job learning to prepare crowd workers for remote, real-time captioning by supporting both touch-typing and chording. Real-time captioning is a difficult skill to master but is important for making live events accessible. We conducted 3 crowd studies of Scopist focusing on Scopist's performance and support for learning. We show that Scopist can distinguish touch-typing from stenotyping with 94% accuracy. Our research demonstrates a new way for workers on crowd platforms to align their work and skill development with the accessibility domain while they work.
Conference Paper
Crowd work provides solutions to complex problems effectively, efficiently, and at low cost. Previous research showed that feedback, particularly correctness feedback can help crowd workers improve their performance; yet such feedback, particularly when generated by experts, is costly and difficult to scale. In our research we investigate approaches to facilitating continuous observational learning in crowdsourcing communities. In a study conducted with workers on Amazon Mechanical Turk, we asked workers to complete a set of tasks identifying nutritional composition of different meals. We examined workers' accuracy gains after being exposed to expert-generated feedback and to two types of peer-generated feedback: direct accuracy assessment with explanations of errors, and a comparison with solutions generated by other workers. The study further confirmed that expert-generated feedback is a powerful mechanism for facilitating learning and leads to significant gains in accuracy. However, the study also showed that comparing one's own solutions with a variety of solutions suggested by others and their comparative frequencies leads to significant gains in accuracy. This solution is particularly attractive because of its low cost, minimal impact on time and cost of job completion, and high potential for adoption by a variety of crowdsourcing platforms.
Conference Paper
We explore how crowdworkers can be trained to tackle complex crowdsourcing tasks. We are particularly interested in training novice workers to perform well on solving tasks in situations where the space of strategies is large and workers need to discover and try different strategies to be successful. In a first experiment, we perform a comparison of five different training strategies. For complex web search challenges, we show that providing expert examples is an effective form of training, surpassing other forms of training in nearly all measures of interest. However, such training relies on access to domain expertise, which may be expensive or lacking. Therefore, in a second experiment we study the feasibility of training workers in the absence of domain expertise. We show that having workers validate the work of their peer workers can be even more effective than having them review expert examples if we only present solutions filtered by a threshold length. The results suggest that crowdsourced solutions of peer workers may be harnessed in an automated training pipeline.
Conference Paper
Expert crowdsourcing marketplaces have untapped potential to empower workers' career and skill development. Currently, many workers cannot afford to invest the time and sacrifice the earnings required to learn a new skill, and a lack of experience makes it difficult to get job offers even if they do. In this paper, we seek to lower the threshold to skill development by repurposing existing tasks on the marketplace as mentored, paid, real-world work experiences, which we refer to as micro-internships. We instantiate this idea in Atelier, a micro-internship platform that connects crowd interns with crowd mentors. Atelier guides mentor-intern pairs to break down expert crowdsourcing tasks into milestones, review intermediate output, and problem-solve together. We conducted a field experiment comparing Atelier's mentorship model to a non-mentored alternative on a real-world programming crowdsourcing task, finding that Atelier helped interns maintain forward progress and absorb best practices.
Conference Paper
This research investigates how to introduce synchronous interactive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in massive online courses (learning course material). We present an interaction framework in which groups of learners are formed on demand and then proceed through a sequence of activities that include synchronous group discussion about learner-generated responses. Via controlled experiments with crowdworkers, we show that discussing challenging problems leads to better outcomes than working individually, and incentivizing people to help one another yields still better results. We then show that providing a mini-lesson in which workers consider the principles underlying the tested concept and justify their answers leads to further improvements. Combining the mini-lesson with the discussion of the multiple-choice question leads to significant improvements on that question. We also find positive subjective responses to the peer interactions, suggesting that discussions can improve morale in remote work or learning settings.
Conference Paper
Integrating crowd-based systems to organizations is highly complex. A major of source of this complexity stems from the nature of organizational work design. Work design and configurations of work performance, in the traditional organizational model, are tightly woven into the structure and functions of organizations, whereas crowdwork leverages an undefined network of people without an organized managerial or hierarchical model. This paper examines work design theories in organizational studies with a view to exploring their potential for addressing the fundamental challenges of work design in organizational crowdwork. Drawing on review of extant literature on crowdwork and analysis of perspectives in work design theories, the paper outlines ways in which crowdsourcing research, on one hand, can interpret and utilize work design theories, and on the other, contribute to redesigning work design theories to keep pace with the important and rapid transformation of work from the traditional staffing paradigm to crowd and open models.
Chapter
Within volunteered geographic information (VGI), citizen science stands out as a class of activities that require special attention and analysis. Citizen science is likely to be the longest running of VGI activities, with some projects showing continuous effort over a century. In addition, many projects are characterised by a genuine element of volunteering and contribution of information for the benefit of human Knowledge and science. They are also tasks where data quality and uncertainty come to the fore when evaluating the validity of the results. This chapter provides an overview of citizen science in the context of VGI - hence the focus on geographic citizen science. This chapter highlights the historical context of citizen science and its more recent incarnation. It also covers some of the cultural and conceptual challenges that citizen science faces and the resulting limitation on the level of engagement. By drawing parallels with the Participatory Geographic Information Systems (PGIS) literature, the chapter offers a framework for participation in citizen science and concludes with the suggestion that a more participatory mode of citizen science is possible. © 2013 Springer Science+Business Media Dordrecht. All rights reserved.
Article
The development of a fast and reliable Internet, new technologies online payment systems, and changes in work structure that enable and demand flexible working patterns have driven a move to a new form of Internet-enabled labour exchange called crowdsourcing. Evidence from an in-depth qualitative study is presented, focusing on selected users' interactions and experiences of working on two UK-based crowdsourcing platforms. The paper shows that workers engaged in this form of labour exchange need to deploy existing employability skills and networks to effectively meet the challenges, and take advantage of the opportunities, that crowdsourcing presents. Individual factors and circumstances emerge as paramount for workers' continued engagement in this form of employment. Using selected components from an employability framework, the findings suggest that crowdsourcing can offer new pathways to practising skills and enhancing employability for some workers.
Conference Paper
In modern crowdsourcing markets, requesters face the challenge of training and managing large transient workforces. Requesters can hire peer workers to review others' work, but the value may be marginal, especially if the reviewers lack requisite knowledge. Our research explores if and how workers learn and improve their performance in a task domain by serving as peer reviewers. Further, we investigate whether peer reviewing may be more effective in teams where the reviewers can reach consensus through discussion. An online between-subjects experiment compares the trade-offs of reviewing versus producing work using three different organization strategies: working individually, working as an interactive team, and aggregating individuals into nominal groups. The results show that workers who review others' work perform better on subsequent tasks than workers who just produce. We also find that interactive reviewer teams outperform individual reviewers on all quality measures. However, aggregating individual reviewers into nominal groups produces better quality assessments than interactive teams, except in task domains where discussion helps overcome individual misconceptions.
Conference Paper
In collaborative crowdsourcing, the problem of maintaining solvers' motivation and participation plays a crucial role. It has been noticed that one of the major issues is related to the scarce capacity of novices to face expert users in finding good solutions, which leads to an high rate of community abandonment. In order to overcome such a problem the best solvers should be asked to illustrate the rationale of their solutions and to share the adopted heuristics before receiving the award. This procedure allows the less expert to learn by doing and see how best solvers innovate products. The paper presents the “You solve, I learn” approach and how it has been implemented in an original platform able to promote innovation, creativity and effective knowledge sharing. The software use and achieved results are demonstrated through a case study.
Article
Several organizations have developed ongoing crowdsourcing communities that repeatedly collect ideas for new products and services from a large, dispersed “crowd” of nonexperts (consumers) over time. Despite its promises, little is known about the nature of an individual's ideation efforts in such an online community. Studying Dell's IdeaStorm community, serial ideators are found to be more likely than consumers with only one idea to generate an idea the organization finds valuable enough to implement, but they are unlikely to repeat their early success once their ideas are implemented. As ideators with past success attempt to again come up with ideas that will excite the organization, they instead end up proposing ideas similar to their ideas that were already implemented (i.e., they generate less diverse ideas). The negative effects of past success are somewhat mitigated for ideators with diverse commenting activity on others' ideas. These findings highlight some of the challenges in maintaining an ongoing supply of quality ideas from the crowd over time. This paper was accepted by Kamalini Ramdas, entrepreneurship and innovation.