No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Abstract
A growing number of people are working as part of on-line crowd work, which has been characterized by its low wages; yet, we know little about wage distribution and causes of low/high earnings. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. The average requester pays more than $11/h, although lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is included in our wage calculations, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs future platform design and worker tools to create a more positive future for crowd work.
... On the one hand, this low reimbursement might be a product of researchers trying to optimize the total number of annotations given a particular budget. On the other hand, it could be a lack of awareness of what appropriate compensation should be [Hara et al., 2017]. ...
... There are important ethical issues which are largely not mentioned in the papers we surveyed. First of all, details about compensation are often missing, whereas this can have an important effect on the crowd [Hara et al., 2017]. Furthermore, what is reasonable compensation in one country, may be too low for another country due to different cost of living. ...
... This is particularly important because high throughput workers are more likely to discuss HITs [Chandler et al., 2014]. This subgroup (less than 10 % of the workers do more than 75% of the work [Hara et al., 2017]) is likely to have experience with similar tasks [Chandler et al., 2014], and interaction with these workers may result in various improvements such as improvements of the user interface as in [Bruggemann et al., 2018]. ...
Rapid advances in image processing capabilities have been seen across many domains, fostered by the application of machine learning algorithms to "big-data". However, within the realm of medical image analysis, advances have been curtailed, in part, due to the limited availability of large-scale, well-annotated datasets. One of the main reasons for this is the high cost often associated with producing large amounts of high-quality meta-data. Recently, there has been growing interest in the application of crowdsourcing for this purpose; a technique that has proven effective for creating large-scale datasets across a range of disciplines, from computer vision to astrophysics. Despite the growing popularity of this approach, there has not yet been a comprehensive literature review to provide guidance to researchers considering using crowdsourcing methodologies in their own medical imaging analysis. In this survey, we review studies applying crowdsourcing to the analysis of medical images, published prior to July 2018. We identify common approaches, challenges and considerations, providing guidance of utility to researchers adopting this approach. Finally, we discuss future opportunities for development within this emerging domain.
... Gig economy platforms have undermined local governments and use technology for "regulatory arbitrage" [10,20]. Crowdsourcing has been associated with (and sometimes predicated upon) subminimum wage pay [7,18]. And it doesn't stop there. ...
... Crowdwork: A researcher who invents a new crowdwork framework likely motivates her work by highlighting the problem the framework solves and often the financial benefits of the solution. Crowdwork, however, also comes with serious negative externalities such as incentivizing very low pay [18]. Under our recommendations, this researcher should ideally find ways to engineer her crowdwork framework such that these externalities are structurally mitigated. ...
The computing research community needs to work much harder to address the downsides of our innovations. Between the erosion of privacy, threats to democracy, and automation's effect on employment (among many other issues), we can no longer simply assume that our research will have a net positive impact on the world. While bending the arc of computing innovation towards societal benefit may at first seem intractable, we believe we can achieve substantial progress with a straightforward step: making a small change to the peer review process. As we explain below, we hypothesize that our recommended change will force computing researchers to more deeply consider the negative impacts of their work. We also expect that this change will incentivize research and policy that alleviates computing's negative impacts.
... This initiative stands in sharp contrast to the utilization of outsourced, contracted laborers to improve metadata with datasets, an approach common among machine learning datasets in the machine learning community. Crowd workers such as Mechanical Turk workers are paid extremely low hourly wages (Hara et al., 2017). This checklist section is thus provided to encourage project stakeholders to consider the project's relationship to labor and to improve transparency to the project's audience. ...
Within the cultural heritage sector, there has been a growing and concerted effort to consider a critical sociotechnical lens when applying machine learning techniques to digital collections. Though the cultural heritage community has collectively developed an emerging body of work detailing responsible operations for machine learning in libraries and other cultural heritage institutions at the organizational level, there remains a paucity of guidelines created specifically for practitioners embarking on machine learning projects. The manifold stakes and sensitivities involved in applying machine learning to cultural heritage underscore the importance of developing such guidelines. This paper contributes to this need by formulating a detailed checklist with guiding questions and practices that can be employed while developing a machine learning project that utilizes cultural heritage data. I call the resulting checklist the "Collections as ML Data" checklist, which, when completed, can be published with the deliverables of the project. By surveying existing projects, including my own project, Newspaper Navigator, I justify the "Collections as ML Data" checklist and demonstrate how the formulated guiding questions can be employed and operationalized.
... This drives people to alternative work arrangements where they often find themselves in the role of entrepreneurs, drawing on their own personal assets, with all the attendant risks and rewards for this kind of economic activity (Berg, 2016;Berg et al., 2018). Gig workers, similarly, find themselves in the grip of the so-called platform economy, controlled by machines and managed by algorithms, the working of which they struggle to understand, and against which they have no recourse to legal labour protections (Hara et al., 2018). In brief, not only has the 15-hour week not materialized, for the majority of people almost the opposite has happened. ...
Resumen
El miedo a la automatización y al «futuro del trabajo», a la superfluidad de los trabajadores y del ser humano en general, se basa en ideas recurrentes sobre la tecnología, el trabajo y el valor económico. El debate se remonta a destacados pensadores como Karl Marx y John Maynard Keynes. Para entender el momento actual, los autores revisan este debate en relación con la historia del capitalismo. Desde una perspectiva centrada en el trabajo y la tecnología, examinan las formas ocultas de creación de valor en la economía actual y las lagunas del debate histórico, y esbozan varias situaciones hipotéticas de cara al futuro.
... Silberman (2010) reported in detail on the issues that crowdworkers face. A lot has been written about the low rates of pay that crowdworks often suffer, with the average worker earning a median of USD$2 per hour (Hara et al., 2017). This is not the only issue. ...
Crowdsourcing psychometric data is common in areas of Human-Computer Interaction (HCI) such as information visualization, text entry, and interface design. In some of the social sciences, crowdsourcing data is now considered routine, and even standard. In this chapter, we explore the collection of data in this manner, beginning by describing the variety of approaches can be used to crowdsource data. Then, we evaluate past literature that has compared the results of these approaches to more traditional data-collection techniques. From this literature, we synthesize a set of design and implementation guidelines for crowdsourcing studies. Finally, we describe how particular analytic techniques can be recruited to aid the analysis of large-scale crowdsourced data. The goal of this chapter it to clearly enumerate the difficulties of crowdsourcing psychometric data and to explore how, with careful planning and execution, these limitations can be overcome.
... The micro- tasks that are sometimes described as repetitive and tedious are poorly paid, workers lack income stability but also bargaining power and legal resource to defend themselves against unfair work practices (Bergvall-Kåreborn and Howcroft, 2014, Irani, 2015, Irani and Silberman, 2013). For instance, Hara et al. (2017) found that the majority of AMT workers earn on average 2 USD an hour. This is far below the national minimum wage in the United ...
https://www.riigikogu.ee/wpcms/wp-content/uploads/2017/09/Virtual-work-size-and-trends_final1.pdf
How do borrowers repay their debts? In two simple debt repayment experiments, we elicit different types of severe deviations from optimal, i.e. debt minimizing, repayment decisions. We show how these deviations can be triggered using supposedly irrelevant information and find evidence for a novel heuristic, the “Cuckoo Fallacy”, which is based on the amount of new debt rather than the interest rate. We also demonstrate that simple framing can decrease repayment misallocation, nudging borrowers to more optimal behavior. Our results inform scholars and policy makers on how to improve household’s financial decisions.
Invisible workers—who is behind the artificial intelligence? The article is dedicated to microworkers—highly precarious, often “invisible”workers whose labour is pivotal for the development of artificial intelligence (AI). It examines how crowdwork platforms such as Amazon Mechanical Turk, Appen and Clickworker organise the work of globally dispersed workers annotating data. It argues that, more often than not, this kind of work is performed in a highly structured, yet intransparent way, and is subject to substantial algorithmic control. This gives rise to the question of the employment classification of formally independent microworkers. The paper discusses this issue in light of the proposed Directive on the improved working conditions of platform workers.
This paper studies the response of saving decisions to two alternative types of capital taxation: the wealth tax and the capital income tax. First, we conducted a lifecycle experiment using Amazon MTurk, wherein subjects made dynamic saving decisions. Subjects exhibited an overreaction to wealth taxes but not to capital income taxes: subjects’ savings decreased more with the introduction of a wealth tax than with a financially equivalent drop in capital returns. Second, we built a parsimonious behavioral model of individual optimization to characterize this overreaction and illustrate its welfare implications. Finally, additional treatments began to explore the root cause of this overreaction bias. We ruled out the explanation that the intrinsic aversion to paying wealth taxes drives overreaction. Instead, overreaction seems to be driven by subjects’ incorrect understanding of the effects of wealth taxes on their savings.
This paper undertakes a comparative analysis of the famous eighteenth-century chess-playing automaton known as Mechanical Turk and the Amazon microwork platform of the same name. The original Mechanical Turk was a life-sized automaton made in 1770 and publicly exhibited until the mid-1800s, and which played games of chess with the audience. Its movements were fully mechanical, but even more remarkably, it was promoted as the world’s first ‘thinking machine,’ deciding each move of the chess pieces for itself. From the outset, it was widely assumed that the Mechanical Turk was a hoax, and that a human must be hidden inside the machine, directing the game. But it was a clever hoax whose trick was never discovered, and widely admired as such. Amazon’s Mechanical Turk job platform functions in a similarly open way, this paper argues, as a sort of open technological hoax. Mechanical Turk provides a source of human cognitive labour that can be used to invisibly operate digital systems and programs that are widely assumed to be fully automated. Artificial intelligence is a twenty-first century thinking machine, it requires a human brain to make it work. Amazon’s Mechanical Turk is a marketplace in which companies can hire piecemeal cognitive labour to patch gaps and train programs to keep those systems functioning. Providing what Jeff Bezos has called ‘artificial artificial intelligence,’ Amazon’s Mechanical Turk, like Kempelen’s automaton, both draws attention to and obfuscates the limits of automation and artificial intelligence. Taking the two iterations of the Mechanical Turk as rich cultural figures of automation for their respective periods, this paper will argue that the open secret of their artificial artificial intelligence is itself a form of misdirection that hides other, more successfully guarded secrets: the true extent of that labour, and the conditions in which it is performed.
usinesses today are discovering the significance of an online presence. However, only a handful of them, know the importance of identifying what users are doing on their blog or business website. Here Web Analytics plays a critical role. Web Analytics, also known as Web data analytics, help boost website traffic & provide the website visitors’ data. This study shows how web analytics help in collecting the important information of users behaviours, age, demographics, gender, conversions & source of traffic details. Startups & small businesses can optimise their content based on users’ interest & take productive decisions. India which has become a hub of the startups in the world. Studies have showed that 90% of the startups get failed, in this paper it has been tried to find out how can Web Analytics play the role of catalyst for the success of the startup business. The sooner businesses embrace web analytics, the …
While contemporaneous exposure to polluted air has been shown to reduce labor supply and worker productivity, little is known about the underlying channels. We present first causal evidence that psychological exposure to pollution - the “thought of pollution” - can influence employment performance. Over 2000 recruits on a leading micro-task platform are exposed to otherwise identical images of polluted (treated) or unpolluted (control) scenes. Randomization across the geographically-dispersed workforce ensures that treatment is orthogonal to physical pollution exposure. Treated workers are less likely to accept a subsequent offer of work (labor supply) despite being offered a piece-rate much higher than is typical for the setting. Conditional on accepting the offer, treated workers complete between 5.1% to 10.1% less work depending on the nature of their assigned task. We find no effect on work quality. Suggestive evidence points to the role of induced negative sentiment. Decrements to productivity through psychological mechanisms are plausibly additional to any from physical exposure to polluted air.
In the past decade, crowdworking on online labor market platforms has become the main source of income for a growing number of people worldwide. This development has led to increasing political and scientific interest in the wages that people can earn on such platforms. In this article, we extend the literature based on a single platform, region, or category of crowdworking by conducting a meta-analysis of the prevalent hourly wages. After a systematic and rigorous literature search, we consider 20 primary empirical studies, including 104 wages and 76,282 data points from 22 platforms, eight different countries, and a time span of 12 years. We find that, on average, microwork results in an hourly wage of less than $6. This wage is significantly lower than the mean wage of online freelancers, which is roughly three times higher. We find that hourly wages accounting for unpaid work, such as searching for tasks and communicating with requesters, tend to be significantly lower than wages not considering unpaid work. Legislators and researchers evaluating wages in crowdworking should be aware of this bias when assessing hourly wages, given that the majority of the literature does not account for the effect of unpaid work time on crowdworking wages. To foster the comparability of different research results, we suggest that scholars consider a wage malus to account for unpaid work. Finally, we find that hourly wages collected through surveys tend to be lower than wages collected via browser plugins or other technical data collection methods.
We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark as well as the accompanying API that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement.
This article illustrates the trade-off between online work and leisure. We use the introduction of the widely played mobile game Pokémon Go to observe its impact on the online labour platform, Amazon Mechanical Turk. To measure whether a mobile game, for which players must physically go outside, affects the proportion of American workers on the platform, we create an index to measure the intensity of Pokémon Go use. This article highlights that American online workers, for whom Amazon Mechanical Turk is a secondary source of revenue, are susceptible to exploit the work-leisure trade-off on a daily basis. We have two main findings. First, we show that a one s.d. increase in the relative use of Pokémon Go leads to a daily decrease in the proportion of American workers on Amazon Mechanical Turk of 4.38 percentage points, corresponding to a decrease of up to 1,095 workers. Second, we find that an increase in the search for part-time jobs raises the proportion of American workers on Amazon Mechanical Turk.
Theoretical and empirical findings suggest that individuals are sensitive to the observability of their actions and the downstream consequences of this observability. We connect three streams of literature (social preferences, behavior change, and social norms) to investigate the conditions for which these claims are valid. Across multiple high-powered studies, we examine the mechanisms through which observability of one’s actions affects pro-sociality, when and why it sometimes fails, and how to utilize social and economic incentives to enact behavior change. Our three main results are: (i) observability alone has very little positive effect and can even backfire; (ii) inequality aversion drives the observed backfiring of observability; (iii) increasing the salience of norms can mitigate unintended consequences and successfully increase pro-sociality. From a policy perspective, our results highlight the potential pitfalls of simple behavioral interventions.
A new wave of collaborative robots designed to work alongside humans is bringing the automation historically seen in large-scale industrial settings in new, diverse contexts[95]. However, the research has shown that the innovation gap between robot hardware and software doesn’t seem to close- the hardware development reached extremely high levels.
In contrast, the software side remained quite rudimentary at the level of programming languages from the early 1990s, known almost exclusively among engineers. Consequently, the newest robotic technologies don’t experience high acceptance levels or even recognition from the biggest part of the audience without engineering profiles.
The main goal of this thesis is to improve robot programming environments usability evaluation by developing a methodology such that creating and executing crowdsourced evaluation can be done quickly and with the lowest cost possible, resulting in higher user acceptance overall user groups. With our approach, we then evaluate Assembly[63], a web-based robot programming environment developed at TU Wien as a part of a publicly-funded research project called Cobot Meets Makerspace (COMEMAK)[8].
This cross-sectional study examined the feasibility of using four different web-based strategies to recruit rural and urban adults who use opioids non-medically for a survey on opioid use disorder (OUD) treatment preferences, and compared the treatment preferences of rural versus urban participants. Preferences for medication for opioid use disorder (MOUD) formulation and OUD treatment models were assessed through an online survey. Recruitment advertisements were shown on Facebook, Google AdWords, Reddit, and Amazon Mechanical Turk (MTurk). Participants were categorized by zip code into urban versus rural residence using the Centers for Medicaid and Medicaid Health Resources and Services Administration definitions. OUD treatment preferences were compared using chi-square and t-tests. Among the 851 participants recruited, 815 provided zip codes and were classified as residing in rural (n = 200, 24.5%) or urban (n = 615, 75.4%) regions. A crowdsourcing service (MTurk) recruited the most rural participants, while posts on a social news website (Reddit) recruited the most urban participants (χ²3 = 17.0, p < 0.01). While preferred MOUD formulation and OUD treatment model did not differ by rurality, rural participants were more likely to report a willingness to receive OUD treatment integrated with general medical care (χ²1 = 18.9, p < 0.0001). This study demonstrated that web-based strategies are feasible for recruiting rural adults who misuse opioids. Results suggest OUD treatment preferences largely did not differ by rural residence, and highlight the importance of enhancing the availability and increasing education about MOUD formulations in rural regions.
Unemployment is a serious issue in many areas of Global South. Over the last years, microwork platforms have started to emerge offering opportunities for earnings and promising improved livelihoods. Although the gig economy facilitated by these platforms has been criticized in the literature, we argue that in the context of informal economy, participation on these platforms does not always worsen the livelihood and thus should be examined. This paper investigates the requirements for using such platforms and assesses their applicability in an informal settlement in Namibia. The Sustainable Livelihoods Approach provides the conceptual framework for the research. We devised a simulation by using design science research methods, where the community members were able to complete digital micro‐tasks for monetary compensation. The experiment evoked excitement among participants. This paper argues that the digital micro‐tasks have promise for the individuals, and elaborates on human, financial, physical, and social capitals required to use these platforms. Finally, we propose a new way, trained peers called “tech mediators,” that can act as facilitators to make these platforms more available for the communities in Global South in the future.
This paper studies the response of saving decisions to two alternative types of capital taxation: the wealth tax and the capital income tax. First, we conducted a lifecycle experiment using Amazon Mechanical Turk, wherein subjects made dynamic saving decisions. Subjects exhibited an overreaction to wealth taxes but not to capital income taxes: subjects' savings decreased more with the introduction of a wealth tax than with a financially equivalent drop in capital returns. Second, we built a parsimonious behavioral model of individual optimization to characterize this overreaction and illustrate its welfare implications. Finally, additional treatments began to explore the root cause of this overreaction bias. We ruled out the explanation that the intrinsic aversion to paying wealth taxes drives overreaction. Instead, overreaction seems to be driven by subjects' incorrect understanding of the effects of wealth taxes on their savings.
We advance the hypothesis that women are as competitive as men once the incentive for winning includes factors that matter to women. Allowing winners an opportunity to share some of their winnings with the low performers has gendered consequences for competitive behavior. We ground our work in an evolutionary framework in which winning competitions brings asymmetric benefits and costs to men and women. In the new environment, the potential to share some of the rewards from competition with others may afford women the benefit of reaping competitive gains without incurring some of its potential costs. An experiment (N = 438 in an online convenience sample of U.S. adults) supports our hypothesis: a 26% gender gap in performance vanishes once a sharing option is included to an otherwise identical winner-take-all incentive scheme. Besides providing a novel experiment that challenges the paradigm that women are not as motivated to compete as men, our work proposes some suggestions for policy: including socially-oriented rewards to contracts may offer a novel tool to close the persistent labor market gender gap.
Privacy-by-Design (PbD) suggests designing the fundamental architecture and features of computing systems with privacy in mind. Although widely adopted by regulatory frameworks, a growing number of critics have questioned whether PbD's focus on compliance with privacy regulation may prevent it from addressing users' specific privacy attitudes and expectations. Motivated to enhance user-centered privacy-by-design processes, we examine what are the consequences of the way privacy questions are framed to crowd users, and how personal characteristics of the crowd users impact their responses. We recruited a total of 665 participants, of which 456 were recruited via Amazon Mechanical Turk (AMT), and 209 were university students. We show that the framing of computing systems' features using data flows results in features' evaluations that are less critical, compared to using descriptions of personal experiences. We also found, based on the student sample, that students with professional engineering experience are less critical than those with no work experience when assessing the features' appropriateness. We discuss how our results can be used to enhance privacy-by-design processes and encourage user-centered privacy engineering.
Well labelled Data is paramount in Artificial Intelligence tasks and its collection is often the first step taken by Developers. Current Data Annotation methods can be controversial and ethically questionable, which is the result of large corporations innovating while also trying to reduce their costs. Gamification is the use of games and their concepts in a non-gaming context, and it has seen several creative applications in recent years. We propose a method to use Gamification to streamline and simplify the data annotation process while eradicating its ethical concerns. Our method defines three game modes that provide players with labelling tasks as part of a competitive challenge. Gamifying the data annotation process provides flexibility, scalability and reliability to the creation and improvement of labelled datasets while also improving ethicality and accessibility. Our method is catalysed by widespread adoption to deliver these benefits. With sufficient adoption, it will further the possibilities of creating relevant solutions in the domain of Artificial Intelligence.
John Stuart Mill claimed that “men do not desire merely to be rich, but richer than other men.” Are people made happy by being richer than others? Or are people made happy by favorable comparisons to others more generally, and being richer is merely a proxy for this ineffable relativity? We conduct an online experiment absent choice in which we measure subjective well-being (SWB) before and after an exogenous shock that reveals to subjects how many experimental points they and another subject receive, and whether or not points are worth money. We find that subjects are made significantly happier when they receive monetized rather than non-monetized points, suggesting money is valued more than the points it represents. In contrast, subjects are made equally unhappy when they receive fewer monetized points as when they receive fewer non-monetized points than others, suggesting relative money is not valued more than the relative points it represents. We find no evidence that subjects are made happier by being “richer” than others (i.e., by receiving more points—either monetized or non-monetized—than others). Subgroup analyses reveal women are made unhappier (than men) by being “richer” and “poorer” than others, and conservatives are made unhappier (than progressives) by being “poorer” than others. Our experimental-SWB approach is easy to administer and may complement choice-based tasks in future experiments to better estimate preference parameters.
Digital technologies have led to the emergence of new forms of flexible work, such as crowdwork or information and communication technology-enabled mobile work, over the past decade. This is considered as a positive development for its high flexibility in hours and place of work, and also for providing employment opportunities to workers in developing countries, individuals with disabilities or those with care responsibilities. This article examines the on-going trend in outsourcing low-skilled work to the crowd through online platforms, and how platform design features and algorithmic management are used by digital labour platforms to assign, monitor and evaluate work. The paper argues that these distinctive features of digital platforms have an impact on working conditions in terms of access to tasks, remuneration and working time flexibility. The paper draws on a global survey of 675 workers on five globally operating platforms in 27 developing countries. Its findings highlight the need for transparency in platform architecture, design and algorithms to ensure that workers are protected from the vulnerabilities they face with digital work.
Resumo Discutimos as relações de comunicação, circulação e produção de sentidos no cenário de manipulação, controle e poder da informação. Procuramos responder à questão: como os estudos de recepção tratam os desafios colocados pelo Big Data na extração, mineração e análise de dados com vistas à manipulação de comportamentos? A metodologia adotada é a pesquisa bibliográfica para a análise crítica e comparativa das perspectivas autorais. Trabalha-se o conceito de sistema, para contestar sua produtividade nas pesquisas de recepção. Como resultado da discussão, os mapas barberianos são retomados em perspectiva, para recolocarmos a história e os conflitos sociais no centro das disputas por hegemonia de sentidos, os quais devem ser problematizados pelos estudos de recepção no contexto social atual.
The purpose of this paper is to investigate the effect of internal motivators (psychological ownership, perceived control, and self-efficacy) on employees' intention to protect organizational data. The threat of cybersecurity breaches is increasing, as are the realization of cybersecurity breaches and the subsequent loss of data. While information security policies increase employees' intentions to protect organizational data, they do not guarantee employee compliance. Results indicate that self-efficacy is associated with perceived control, which, in turn, is associated with perceived psychological ownership of organizational data. Furthermore, self-efficacy and psychological ownership are both associated with an increase in intention to protect organizational data.
Beauty of the World Wide Web (WWW) is not only a huge collection of data and information but it also brings people closer to each other from all over the world. A new and growing area on the web is freelancing, which enables the people around the globe to use skills, expertise, and mind of talented freelancers from anywhere in the world. Freelancing platforms work for two types of peoples i.e. freelancer and a contractor/ client or buyer. A freelancer is a person who is searching for jobs, while the contractor is actually searching for a freelancer for solving his work and business goals. In this, approach the freelancing platforms recommends jobs to the freelancers and freelancers to contractors. However, there are lots of issues in these stereotype recommendation systems, which are discussed in this paper. Furthermore, this paper presents a novel approach for recommending expert freelancers to contractors, not only to save the time of the contractors but also to reduce cognitive overload for them. Experimental results show that the proposed approach is highly effective and recommend relevant expert freelancers to contractors in a better way than the existing recommendation techniques.
The global reach of online experiments and their wide adoption in fields ranging from political science to computer science poses an underexplored opportunity for learning at scale: the possibility of participants learning about the research to which they contribute data. We conducted three experiments on Amazon's Mechanical Turk to evaluate whether participants of paid online experiments are interested in learning about research, what information they find most interesting, and whether providing them with such information actually leads to learning gains. Our findings show that 40% of our participants on Mechanical Turk actively sought out post-experiment learning opportunities despite having already received their financial compensation. Participants expressed high interest in a range of research topics, including previous research and experimental design. Finally, we find that participants comprehend and accurately recall facts from post-experiment learning opportunities. Our findings suggest that Mechanical Turk can be a valuable platform for learning at scale and scientific outreach.
The relations between technology, work organization, worker power, workers’ rights, and workers’ experience of work have long been central concerns of CSCW. European CSCW research, especially, has a tradition of close collaboration with workers and trade unionists in which researchers aim to develop technologies and work processes that increase workplace democracy. This paper contributes a practitioner perspective on this theme in a new context: the (sometimes global) labor markets enabled by digital labor platforms. Specifically, the paper describes a method for rating working conditions on digital labor platforms (e.g., Amazon Mechanical Turk, Uber) developed within a trade union setting. Preliminary results have been made public on a website that is referred to by workers, platform operators, journalists, researchers, and policy makers. This paper describes this technical project in the context of broader cross-sectoral efforts to safeguard worker rights and build worker power in digital labor platforms. Not a traditional research paper, this article instead takes the form of a case study documenting the process of incorporating a human-centered computing perspective into contemporary trade union activities and communicating a practitioner’s perspective on how CSCW research and computational artifacts can come to matter outside of the academy. The paper shows how practical applications can benefit from the work of CSCW researchers, while illustrating some practical constraints of the trade union context. The paper also offers some practical contributions for researchers studying digital platform workers’ experiences and rights: the artifacts and processes developed in the course of the work.
Micro-task crowdsourcing is rapidly gaining popularity among research communities and businesses as a means to leverage Human Computation in their daily operations. Unlike any other service, a crowdsourcing platform is in fact a marketplace subject to human factors that affect its performance, both in terms of speed and quality. Indeed, such factors shape the dynamics of the crowdsourcing market. For example, a known behavior of such markets is that increasing the reward of a set of tasks would lead to faster results. However, it is still unclear how different dimensions interact with each other: reward, task type, market competition, requester reputation, etc. In this paper, we adopt a data-driven approach to (A) perform a long-term analysis of a popular micro-task crowdsourcing platform and understand the evolution of its main actors (workers, requesters, and platform). (B) We leverage the main findings of our five year log analysis to propose features used in a predictive model aiming at determining the expected performance of any batch at a specific point in time. We show that the number of tasks left in a batch and how recent the batch is are two key features of the prediction. (C) Finally, we conduct an analysis of the demand (new tasks posted by the requesters) and supply (number of tasks completed by the workforce) and show how they affect task prices on the marketplace.
Micro-task crowdsourcing is rapidly gaining popularity among research communities and businesses as a means to leverage Human Computation in their daily operations. Unlike any other service, a crowdsourcing platform is in fact a marketplace subject to human factors that affect its performance, both in terms of speed and quality. Indeed, such factors shape the \emph{dynamics} of the crowdsourcing market. For example, a known behavior of such markets is that increasing the reward of a set of tasks would lead to faster results. However, it is still unclear how different dimensions interact with each other: reward, task type, market competition, requester reputation, etc.
In this paper, we adopt a data-driven approach to (A) perform a long-term analysis of a popular micro-task crowdsourcing platform and understand the evolution of its main actors (workers, requesters, tasks, and platform). (B) We leverage the main findings of our five year log analysis to propose features used in a predictive model aiming at determining the expected performance of any batch at a specific point in time. We show that the number of tasks left in a batch and how recent the batch is are two key features of the prediction. (C) Finally, we conduct an analysis of the demand (new tasks posted by the requesters) and supply (number of tasks completed by the workforce) and show how they affect task prices on the marketplace.
This article assesses the validity of many of the assumptions made about work in the on-demand economy and analyses whether proposals advanced for improving workers' income security are sufficient for remedying current shortcomings. It draws on findings from a survey of crowdworkers conducted in late 2015 on the Amazon Mechanical Turk and Crowdflower platforms on workers' employment patterns, work histories, and financial security. Based on this information, it provides an analysis of crowdworkers' economic dependence on the platform, including the share of workers who depend on crowdwork as their main source of income, as well as their working conditions, the problems they encounter while crowdworking and their overall income security. Based on these findings, the article recommends an alternative way of organizing work that can improve the income security of crowdworkers as well as overall efficiency and productivity of crowdwork.
Previous studies on Amazon Mechanical Turk (AMT), the most well-known marketplace for microtasks, show that the largest population of workers on AMT is U.S. based, while the second largest is based in India. In this paper, we present insights from an ethnographic study conducted in India to introduce some of these workers or "Turkers" -- who they are, how they work and what turking means to them. We examine the work they do to maintain their reputations and their work-life balance. In doing this, we illustrate how AMT's design practically impacts on turk-work. Understanding the "lived work" of crowdwork is a valuable first step for technology design.
Nowadays, a substantial number of people are turning to crowdsourcing, in order to solve tasks that require human intervention. Despite a considerable amount of research done in the field of crowdsourcing, existing works fall short when it comes to classifying typically crowdsourced tasks. Understanding the dynamics of the tasks that are crowdsourced and the behaviour of workers, plays a vital role in efficient task-design. In this paper, we propose a two-level categorization scheme for tasks, based on an extensive study of 1000 workers on CrowdFlower. In addition, we present insights into certain aspects of crowd behaviour; the task affinity of workers, effort exerted by workers to complete tasks of various types, and their satisfaction with the monetary incentives.
Crowdsourcing is a key current topic in CSCW. We build upon findings of a few qualitative studies of crowdworkers. We conducted an ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (AMT) users. Using forum data we provide novel depth and detail on how the Turker Nation members operate as economic actors, working out which Requesters and jobs are worthwhile to them. We show some of the key ways Turker Nation functions as a community and also look further into Turker-Requester relationships from the Turker perspective -- considering practical, emotional and moral aspects. Finally, following Star and Strauss [25] we analyse Turking as a form of invisible work. We do this to illustrate practical and ethical issues relating to working with Turkers and AMT, and to promote design directions to support Turkers and their relationships with Requesters.
Paid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd work will fail to achieve its potential, focusing on assembly-line piecework. Can we foresee a future crowd workplace in which we would want our children to participate? This paper frames the major challenges that stand in the way of this goal. Drawing on theory from organizational behavior and distributed computing, as well as direct feedback from workers, we outline a framework that will enable crowd work that is complex, collaborative, and sustainable. The framework lays out research challenges in twelve major areas: workflow, task assignment, hierarchy, real-time response, synchronous collaboration, quality control, crowds guiding AIs, AIs guiding crowds, platforms, job design, reputation, and motivation.
Large corpora are ubiquitous in today’s world and memory quickly becomes the limiting factor in practical applications of the Vector Space Model (VSM). In this paper, we identify a gap in existing implementations of many of the popular algorithms, which is their scalability and ease of use. We describe a Natural Language Processing software framework which is based on the idea of document streaming, i.e. processing corpora document after document, in a memory independent fashion. Within this framework, we implement several popular algorithms for topical inference, including Latent Semantic Analysis and Latent Dirichlet Allocation, in a way that makes them completely independent of the training corpus size. Particular emphasis is placed on straightforward and intuitive framework design, so that modifications and extensions of the methods and/or their application by interested practitioners are effortless. We demonstrate the
usefulness of our approach on a real-world scenario of computing document similarities within an existing digital library DML-CZ.
Statistical topic models can help analysts discover patterns in large text corpora by identifying recurring sets of words and enabling exploration by topical concepts. However, understanding and validating the output of these models can itself be a challenging analysis task. In this paper, we offer two design considerations - interpretation and trust - for designing visualizations based on data-driven models. Interpretation refers to the facility with which an analyst makes inferences about the data through the lens of a model abstraction. Trust refers to the actual and perceived accuracy of an analyst's inferences. These considerations derive from our experiences developing the Stanford Dissertation Browser, a tool for exploring over 9,000 Ph.D. theses by topical similarity, and a subsequent review of existing literature. We contribute a novel similarity measure for text collections based on a notion of "word-borrowing" that arose from an iterative design process. Based on our experiences and a literature review, we distill a set of design recommendations and describe how they promote interpretable and trustworthy visual analysis tools.
The recent influx in generation, storage and availability of textual data presents researchers with the challenge of developing suitable methods for their analysis. Latent Semantic Analysis (LSA), a member of a family of methodological approaches that offers an opportunity to address this gap by describing the semantic content in textual data as a set of vectors, was pioneered by researchers in psychology, information retrieval, and bibliometrics. LSA involves a matrix operation called singular value decomposition, an extension of principal component analysis. LSA generates latent semantic dimensions that are either interpreted, if the researcher’s primary interest lies with the understanding of the thematic structure in the textual data, or used for purposes of clustering, categorisation and predictive modelling, if the interest lies with the conversion of raw text into numerical data, as a precursor to subsequent analysis. This paper reviews five methodological issues that need to be addressed by the researcher who will embark on LSA. We examine the dilemmas, present the choices, and discuss the considerations under which good methodological decisions are made. We illustrate these issues with the help of four small studies, involving the analysis of abstracts for papers published in the European Journal of Information Systems.
Self-report and mono-method bias often threaten the validity of research conducted in business settings and thus hinder the development of theories of organizational behavior. This paper outlines a conceptual framework for understanding factors that influence the motivation of an employee to bias his or her responses to questions posed by organizational researchers. Using a longitudinal, multitrait-multimethod dataset, we illustrate various aspects of the problem and argue that traditional approaches for controlling self-report bias do not adequately prevent the problem. The results suggest the need for developing a theory of method effects and companion analytic techniques to improve the accuracy of psychological research in business settings.
How can the development of ideas in a sci- entific field be studied over time? We ap- ply unsupervised topic modeling to the ACL Anthology to analyze historical trends in the field of Computational Linguistics from 1978 to 2006. We induce topic clusters using Latent Dirichlet Allocation, and examine the strength of each topic over time. Our methods find trends in the field including the rise of prob- abilistic methods starting in 1988, a steady in- crease in applications, and a sharp decline of research in semantics and understanding be- tween 1978 and 2001, possibly rising again after 2001. We also introduce a model of the diversity of ideas, topic entropy, using it to show that COLING is a more diverse confer- ence than ACL, but that both conferences as well as EMNLP are becoming broader over time. Finally, we apply Jensen-Shannon di- vergence of topic distributions to show that all three conferences are converging in the topics they cover.
Visual information pervades our environment. Vision is used to decide everything from what we want to eat at a restaurant and which bus route to take to whether our clothes match and how long until the milk expires. Individually, the inability to interpret such visual information is a nuisance for blind people who often have effective, if inefficient, work-arounds to overcome them. Collectively, however, they can make blind people less independent. Specialized technology addresses some problems in this space, but automatic approaches cannot yet answer the vast majority of visual questions that blind people may have. VizWiz addresses this shortcoming by using the Internet connections and cameras on existing smartphones to connect blind people and their questions to remote paid workers' answers. VizWiz is designed to have low latency and low cost, making it both competitive with expensive automatic solutions and much more versatile.
This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.
The relationship between financial incentives and performance, long of interest to social scientists, has gained new relevance with the advent of web-based "crowd-sourcing" models of production. Here we investigate the effect of compensation on performance in the context of two experiments, conducted on Amazon's Mechanical Turk (AMT). We find that increased financial incentives increase the quantity, but not the quality, of work performed by participants, where the difference appears to be due to an "anchoring" effect: workers who were paid more also perceived the value of their work to be greater, and thus were no more motivated than workers paid less. In contrast with compensation levels, we find the details of the compensation scheme do matter---specifically, a "quota" system results in better work for less pay than an equivalent "piece rate" system. Although counterintuitive, these findings are consistent with previous laboratory studies, and may have real-world analogs as well.
Crowdsourcing is a form of "peer production" in which work traditionally performed by an employee is outsourced to an "undefined, generally large group of people in the form of an open call." We present a model of workers supplying labor to paid crowdsourcing projects. We also introduce a novel method for estimating a worker's reservation wage--the smallest wage a worker is willing to accept for a task and the key parameter in our labor supply model. It shows that the reservation wages of a sample of workers from Amazon's Mechanical Turk (AMT) are approximately log normally distributed, with a median wage of $1.38/hour. At the median wage, the point elasticity of extensive labor supply is 0.43. We discuss how to use our calibrated model to make predictions in applied work. Two experimental tests of the model show that many workers respond rationally to offered incentives. However, a non-trivial fraction of subjects appear to set earnings targets. These "target earners" consider not just the offered wage--which is what the rational model predicts--but also their proximity to earnings goals. Interestingly, a number of workers clearly prefer earning total amounts evenly divisible by 5, presumably because these amounts make good targets.
Since the concept of crowd sourcing is relatively new, many potential participants have questions about the AMT marketplace. For example, a common set of questions that pop up in an 'introduction to crowd sourcing and AMT' session are the following: What type of tasks can be completed in the marketplace? How much does it cost? How fast can I get results back? How big is the AMT marketplace? The answers for these questions remain largely anecdotal and based on personal observations and experiences. To understand better what types of tasks are being completed today using crowd sourcing techniques, we started collecting data about the AMT marketplace. We present a preliminary analysis of the dataset and provide directions for interesting future research.
Many practitioners currently use rules of thumb to price tasks on online labor markets. Incorrect pricing leads to task starvation or inefficient use of capital. Formal pricing policies can address these challenges. In this paper we argue that a pricing policy can be based on the trade-off between price and desired completion time. We show how this duality can lead to a better pricing policy for tasks in online labor markets. This paper makes three contributions. First, we devise an algorithm for job pricing using a survival analysis model. We then show that worker arrivals can be modeled as a non-homogeneous Poisson Process (NHPP). Finally using NHPP for worker arrivals and discrete choice models we present an abstract mathematical model that captures the dynamics of the market when full market information is presented to the task requester. This model can be used to predict completion times and pricing policies for both public and private crowds.
Scikit-learn is a Python module integrating a wide range of state-of-the-art
machine learning algorithms for medium-scale supervised and unsupervised
problems. This package focuses on bringing machine learning to non-specialists
using a general-purpose high-level language. Emphasis is put on ease of use,
performance, documentation, and API consistency. It has minimal dependencies
and is distributed under the simplified BSD license, encouraging its use in
both academic and commercial settings. Source code, binaries, and documentation
can be downloaded from http://scikit-learn.sourceforge.net.
Adults with Autism Spectrum Disorders (ASD) are unemployed at a high rate, in part because the constraints and expectations of traditional employment can be difficult for them. In this paper, we report on our work in introducing people with ASD to remote work on a crowdsourcing platform and a prototype tool we developed by working with participants. We conducted a six-week long user-centered design study with three participants with ASD. The early stage of the study focused on assessing the abilities of our participants to search and work on micro-tasks available on the crowdsourcing market. Based on our preliminary findings, we designed, developed, and evaluated a prototype tool to facilitate image transcription tasks that are increasingly popular on crowd labor markets. Our findings suggest that people with ASD have varying levels of ability to work on micro-tasks, but are likely to be able to work on tasks like image transcription. The tool we introduce, Assistive Task Queue (ATQ), facilitated our participants' completion of image transcription tasks by removing ambiguity in finding the next task to work on and in simplifying tasks into discrete steps. ATQ may serve as a general platform for finding and delivering appropriate tasks to workers with autism.
The internet is empowering the rise of crowd work, gig work, and other forms of on-demand labor. A large and growing body of scholarship has attempted to predict the socio-technical outcomes of this shift, especially addressing three questions: 1) What are the complexity limits of on-demand work?, 2) How far can work be decomposed into smaller microtasks?, and 3) What will work and the place of work look like for workers? In this paper, we look to the historical scholarship on piecework — a similar trend of work decomposition, distribution, and payment that was popular at the turn of the 20th century — to understand how these questions might play out with modern on-demand work. We identify the mechanisms that enabled and limited piecework historically, and identify whether on-demand work faces the same pitfalls or might differentiate itself. This approach introduces theoretical grounding that can help address some of the most persistent questions in crowd work, and suggests design interventions that learn from history rather than repeat it.
We conduct an experimental analysis of a dataset comprising over 27 million microtasks performed by over 70,000 workers issued to a large crowdsourcing marketplace between 2012-2016. Using this data---never before analyzed in an academic context---we shed light on three crucial aspects of crowdsourcing: (1) Task design --- helping requesters understand what constitutes an effective task, and how to go about designing one; (2) Marketplace dynamics --- helping marketplace administrators and designers understand the interaction between tasks and workers, and the corresponding marketplace load; and (3) Worker behavior --- understanding worker attention spans, lifetimes, and general behavior, for the improvement of the crowdsourcing ecosystem as a whole.
Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-Telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.
Online crowd labor markets often address issues of risk and mistrust between employers and employees from the employers' perspective, but less often from that of employees. Based on 437 comments posted by crowd workers (Turkers) on the Amazon Mechanical Turk (AMT) participation agreement, we identified work rejection as a major risk that Turkers experience. Unfair rejections can result from poorly-designed tasks, unclear instructions, technical errors, and malicious Requesters. Because the AMT policy and platform provide little recourse to Turkers, they adopt strategies to minimize risk: avoiding new and known bad Requesters, sharing information with other Turkers, and choosing low-risk tasks. Through a series of ideas inspired by these findings-including notifying Turkers and Requesters of a broken task, returning rejected work to Turkers for repair, and providing collective dispute resolution mechanisms-we argue that making reducing risk and building trust a first-class design goal can lead to solutions that improve outcomes around rejected work for all parties in online labor markets.
Diversifying participation in crowd work can benefit the worker and requester. Increasing numbers of older adults are online, but little is known about their awareness of or how they engage in mainstream crowd work. Through an online survey with 505 seniors, we found that most have never heard of crowd work but would be motivated to complete tasks by earning money or working on interesting or stimulating tasks. We follow up results from the survey with interviews and observations of 14 older adults completing crowd work tasks. While our survey data suggests that financial incentives are encouraging, in-depth interviews reveal that a combination of personal and social incentives may be stronger drivers of participation, but only if older adults can overcome accessibility issues and understand the purpose of crowd work. This paper contributes insights into how crowdsourcing sites could better engage seniors and other users.
Paid crowdsourcing marketplaces have gained popularity by using piecework, or payment for each microtask, to incentivize workers. This norm has remained relatively unchallenged. In this paper, we ask: is the pay-per-task method the right one? We draw on behavioral economic research to examine whether payment in bulk after every ten tasks, saving money via coupons instead of earning money, or material goods rather than money will increase the number of completed tasks. We perform a twenty-day, between-subjects field experiment (N=300) on a mobile crowdsourcing application and measure how often workers responded to a task notification to fill out a short survey under each incentive condition. Task completion rates increased when paying in bulk after ten tasks: doing so increased the odds of a response by 1.4x, translating into 8% more tasks through that single intervention. Payment with coupons instead of money produced a small negative effect on task completion rates. Material goods were the most robust to decreasing participation over time.
This paper argues that designers committed to advancing justice and other non-market values must attend not only to the design of objects, processes, and situations, but also to the wider economic and cultural imaginaries of design as a social role. The paper illustrates the argument through the case of Turkopticon, originally an activist tool for workers in Amazon Mechanical Turk (AMT), built by the authors and maintained since 2009. The paper analyzes public depictions of Turkopticon which cast designers as creative innovators and AMT workers as without agency or capacity to change their situation. We argue that designers' elevated status as workers in knowledge economies can have practical consequences for the politics of their design work. We explain the consequences of this status for Turkopticon and how we adapted our approach in response over the long term. We argue for analyses of power in design work that account for and develop counters to hegemonic beliefs and practices about design as high-status labor.
Crowdsourcing and human computation enable organizations to accomplish tasks that are currently not possible for fully automated techniques to complete, or require more flexibility and scalability than traditional employment relationships can facilitate. In the area of data processing, companies have benefited from crowd workers on platforms such as Amazon's Mechanical Turk or Upwork to complete tasks as varied as content moderation, web content extraction, entity resolution, and video/audio/image processing. Several academic researchers from diverse areas ranging from the social sciences to computer science have embraced crowdsourcing as a research area, resulting in algorithms and systems that improve crowd work quality, latency, or cost. Given the relative nascence of the field, the academic and the practitioner communities have largely operated independently of each other for the past decade, rarely exchanging techniques and experiences. In this book, we aim to narrow the gap between academics and practitioners. On the academic side, we summarize the state of the art in crowd-powered algorithms and system design tailored to large-scale data processing. On the industry side, we survey 13 industry users (e.g., Google, Facebook, Microsoft) and 4 marketplace providers of crowd work (e.g., CrowdFlower, Upwork) to identify how hundreds of engineers and tens of million dollars are invested in various crowdsourcing solutions. Through the book, we hope to simultaneously introduce academics to real problems that practitioners encounter every day, and provide a survey of the state of the art for practitioners to incorporate into their designs. Through our surveys, we also highlight the fact that crowdpowered data processing is a large and growing field. Over the next decade, we believe that most technical organizations will in some way benefit from crowd work, and hope that this book can help guide the effective adoption of crowdsourcing across these organizations.
Amazon's Mechanical Turk (MTurk) is an online marketplace for work, where Requesters post Human Intelligence Tasks (HITs) for Workers to complete for varying compensation. Past research has focused on the quality and generalizability of social and behavioral science research conducted using MTurk as a source of research participants. However, MTurk and other crowdsourcing platforms also exemplify trends toward extremely short-term contract work. We apply principles of industrial–organizational (I–O) psychology to investigate MTurk Worker job satisfaction, information sharing, and turnover. We also report the top best and worst Requester behaviors (e.g., building a relationship, unfair pay) that affect Worker satisfaction. Worker satisfaction was consistently negatively related to turnover as expected, indicating that this traditional variable operates similarly in the MTurk work context. However, few of the traditional predictors of job satisfaction were significant, signifying that new operational definitions or entirely new variables may be needed in order to adequately understand the experiences of crowdsourced workers. Coworker friendships consistently predicted information sharing among Workers. The findings of this study are useful for understanding the experiences of crowdsourced workers from the perspective of I–O psychology, as well as for researchers using MTurk as a recruitment tool.
An Introduction to Statistical Learning provides an accessible overview of the field of statistical learning, an essential toolset for making sense of the vast and complex data sets that have emerged in fields ranging from biology to finance to marketing to astrophysics in the past twenty years. This book presents some of the most important modeling and prediction techniques, along with relevant applications. Topics include linear regression, classification, resampling methods, shrinkage approaches, tree-based methods, support vector machines, clustering, and more. Color graphics and real-world examples are used to illustrate the methods presented. Since the goal of this textbook is to facilitate the use of these statistical learning techniques by practitioners in science, industry, and other fields, each chapter contains a tutorial on implementing the analyses and methods presented in R, an extremely popular open source statistical software platform. Two of the authors co-wrote The Elements of Statistical Learning (Hastie, Tibshirani and Friedman, 2nd edition 2009), a popular reference book for statistics and machine learning researchers. An Introduction to Statistical Learning covers many of the same topics, but at a level accessible to a much broader audience. This book is targeted at statisticians and non-statisticians alike who wish to use cutting-edge statistical learning techniques to analyze their data. The text assumes only a previous course in linear regression and no knowledge of matrix algebra.
By lowering the costs of communication, the web promises to enable distributed collectives to act around shared issues. However, many collective action efforts never succeed: while the web's affordances make it easy to gather, these same decentralizing characteristics impede any focus towards action. In this paper, we study challenges to collective action efforts through the lens of online labor by engaging with Amazon Mechanical Turk workers. Through a year of ethnographic fieldwork, we sought to understand online workers' unique barriers to collective action. We then created Dynamo, a platform to support the Mechanical Turk community in forming publics around issues and then mobilizing. We found that collective action publics tread a precariously narrow path between the twin perils of stalling and friction, balancing with each step between losing momentum and flaring into acrimony. However, specially structured labor to maintain efforts' forward motion can help such publics take action.
Social scientists are turning to online retail giant Amazon.com to cheaply recruit people around the world for research studies
Electronic Health Records (EHRs) have emerged as a cost-effective data source for conducting medical research. The difficulty in using EHRs for research purposes, however, is that both patient selection and record analysis must be conducted across very large, and typically very noisy datasets. Our previous work introduced EventFlow, a visualization tool that transforms an entire dataset of temporal event records into an aggregated display, allowing researchers to analyze population-level patterns and trends. As datasets become larger and more varied, however, it becomes increasingly difficult to provide a succinct, summarizing display. This paper presents a series of user-driven data simplifications that allow researchers to pare event records down to their core elements. Furthermore, we present a novel metric for measuring visual complexity, and a language for codifying disjoint strategies into an overarching simplification framework. These simplifications were used by real-world researchers to gain new and valuable insights from initially overwhelming datasets.
In this paper, we examine the results of applying Term Frequency Inverse Document Frequency (TF-IDF) to determine what words in a corpus of documents might be more favorable to use in a query. As the term implies, TF-IDF calculates values for each word in a document through an inverse proportion of the frequency of the word in a particular document to the percentage of documents the word appears in. Words with high TF-IDF numbers imply a strong relationship with the document they appear in, suggesting that if that word were to appear in a query, the document could be of interest to the user. We provide evidence that this simple algorithm efficiently categorizes relevant words that can enhance query retrieval.
Individuals with autism spectrum disorder (ASD) have the ability and desire to work, but there are still several obstructions. Research overwhelmingly demonstrates disappointing employment outcomes for this group. The vast majority is unemployed and for those who do have gainful employment, underemployment is common. The increased prevalence of ASD coupled with unique social, communication, and behavioral characteristics translate into the need for services to help them achieve employment success. Consideration of individual characteristics including strengths, needs, as well as specific interests, coupled with implementation of proper supports can result in successful and ongoing employment. This paper provides a review of evi-dence based research related to employment for individuals with ASD. Specific areas addressed include benefits of employment, state of employment, obstacles to employment, current service options, and an in depth review of supports needed for success. These supports focus not only on job tasks, but also the interpersonal skills needed to foster a positive work experience.
A model is proposed that specifies the conditions under which individuals will become internally motivated to perform effectively on their jobs. The model focuses on the interaction among three classes of variables: (a) the psychological states of employees that must be present for internally motivated work behavior to develop; (b) the characteristics of jobs that can create these psychological states; and (c) the attributes of individuals that determine how positively a person will respond to a complex and challenging job. The model was tested for 658 employees who work on 62 different jobs in seven organizations, and results support its validity. A number of special features of the model are discussed (including its use as a basis for the diagnosis of jobs and the evaluation of job redesign projects), and the model is compared to other theories of job design.
The technique of latent semantic indexing is used in a wide variety of commercial applications. In these applications, the processing time and RAM required for SVD computation, and the processing time and RAM required during LSI retrieval operations are all roughly linear in the number of dimensions, k, chosen for the LSI representation space. In large-scale commercial LSI applications, reducing k values could be of significant value in reducing server costs. This paper explores the effects of varying dimensionality. The approach taken here focuses on term comparisons. Pairs of terms are considered which have strong real-world associations. The proximities of members of these pairs in the LSI space are compared at multiple values of k. The testing is carried out for collections of from one to five million documents. For the five million document collection, a value of k ≈ 400 provides the best performance. The results suggest that there is something of an 'island of stability' in the k = 300 to 500 range. The results also indicate that there is relatively little room to employ k values outside of this range without incurring significant distortions in at least some term-term correlations.
In this paper we present a mechanism for determining near-optimal prices for tasks in online labor markets, often used for crowdsourcing. In particular, the mechanisms are designed to handle the intricacies of markets like Mechanical Turk where workers arrive online and requesters have budget constraints. The mechanism is incentive compatible, budget feasible, and has competitive ratio performance and also performs well in practice. To demonstrate the mechanism's practical effectiveness we conducted experiments on the Mechanical Turk platform. Copyright © 2011, Association for the Advancement of Artificial Intelligence. All rights reserved.
The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.
The ongoing rise of human computation as a means of solving computational problems has created an environment where human workers are often regarded as nameless, faceless computational resources. Some people have begun to think of online tasks as a "remote person call". In this paper, we summarize ethical and practical labor issues surrounding online labor, and offer a set of guidelines for designing and using online labor in ways that support more positive relationships between workers and requesters, so that both can gain the most benefit from the interaction.
Paid crowd workers are not just an API call---but all too often, they are treated like one.
We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.
No work is inherently either visible or invisible. We always “see” work through a selection of indicators: straining muscles, finished artifacts, a changed state of affairs. The indicators change with context, and that context becomes a negotiation about the relationship between visible and invisible work. With shifts in industrial practice these negotiations require longer chains of inference and representation, and may become solely abstract.
This article provides a framework for analyzing invisible work in CSCW systems. We sample across a variety of kinds of work to enrich the understanding of how invisibility and visibility operate. Processes examined include creating a “non-person” in domestic work; disembedding background work; and going backstage. Understanding these processes may inform the design of CSCW systems and the development of related social theory.