A Guide to Conducting Behavioral Research on Amazon's Mechanical Turk

Yahoo! Research, New York, USA.
Behavior Research Methods (Impact Factor: 2.12). 06/2011; 44(1):1-23. DOI: 10.3758/s13428-011-0124-6
Source: PubMed

ABSTRACT Amazon's Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, stable, and diverse subject pool, the low cost of doing experiments, and faster iteration between developing theory and executing experiments. While other methods of conducting behavioral research may be comparable to or even better than Mechanical Turk on one or more of the axes outlined above, we will show that when taken as a whole Mechanical Turk can be a useful tool for many researchers. We will discuss how the behavior of workers compares with that of experts and laboratory subjects. Then we will illustrate the mechanics of putting a task on Mechanical Turk, including recruiting subjects, executing the task, and reviewing the work that was submitted. We also provide solutions to common problems that a researcher might face when executing their research on this platform, including techniques for conducting synchronous experiments, methods for ensuring high-quality work, how to keep data private, and how to maintain code security.

783 Reads
  • Source
    • "online microtask market that can be used to crowdsource user studies (Kittur, Chi, & Suh, 2008; Heer & Bostock, 2010; Mason & Suri, 2011). The HIT consisted of 20 test-cases, two verification cases, and five demographic questions. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Online syndicated text-based advertising is ubiquitous on news sites, blogs, personal websites, and on search result pages. Until recently, a common distinguishing feature of these text-based advertisements has been their background color. Following intervention by the Federal Trade Commission (FTC), the format of these advertisements has undergone a subtle change in their design and presentation. Using three empirical experiments, we investigate the effect of industry-standard advertising practices on click rates, and demonstrate changes in user behavior when this familiar differentiator is modified. Using three large-scale experiments (N1 = 101, N2 = 84, N3 = 176) we find that displaying advertisement and content results with a differentiated background results in significantly lower click rates. Our results demonstrate the strong link between background color differentiation and advertising, and reveal how alternative differentiation techniques influence user behavior.
    Computers in Human Behavior 09/2015; 50. DOI:10.1016/j.chb.2015.04.031 · 2.69 Impact Factor
  • Source
    • "We surveyed Amazon Mechanical Turk (MTurk) workers from diverse professions, organizations, and industries. MTurk is crowd-sourcing website that connects researchers with participants, and has been found to produce reasonably high quality and low cost data (Berinsky et al. 2012; Buhrmester et al. 2011; Mason and Suri 2012). At Time 1, N = 681 MTurk workers completed the survey and were reimbursed $1 for their time. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Purpose: We examine the interaction between trait resilience and control in predicting coping and performance. Drawing on a person–environment fit perspective, we hypothesized resilient individuals would cope and perform better in demanding work situations when control was high. In contrast, those low in resilience would cope and perform better when control was low. Recognizing the relationship between trait resilience and performance also could be indirect, adaptive coping was examined as a mediating mechanism through which high control enables resilient individuals to demonstrate better performance. Methodology: In Study 1 (N = 78) and Study 2 (N = 94), participants completed a demanding inbox task in which trait resilience was measured and high and low control was manipulated. Study 3 involved surveying 368 employees on their trait resilience, control, and demand at work (at Time 1), and coping and performance 1 month later at Time 2. Findings: For more resilient individuals, high control facilitated problem-focused coping (Study 1, 2, and 3), which was indirectly associated with higher subjective performance (Study 1), mastery (Study 2), adaptive, and proficient performance (Study 3). For more resilient individuals, high control also facilitated positive reappraisal (Study 2 and 3), which was indirectly associated with higher adaptive and proficient performance (Study 3). Implications: Individuals higher in resilience benefit from high control because it enables adaptive coping. Originality/value: This research makes two contributions: (1) an experimental investigation into the interaction of trait resilience and control, and (2) investigation of coping as the mechanism explaining better performance.
    Journal of Business and Psychology 08/2015; 30(3):583-604. DOI:10.1007/s10869-014-9383-4 · 1.25 Impact Factor
  • Source
    • "Two independent samples were recruited through Amazon's Mechanical Turk, which is a large crowd sourcing internet marketplace (database currently consists of over 500,000 individuals from 190 countries ) shown to produce demographically diverse samples and reliable data (see Buhrmester, Kwang, & Gosling, 2011; Mason & Suri, 2012; Paolacci, Chandler, & Ipeirotis, 2010). We recruited participants living in the U.S. and 19 years of age or older. "
    Personality and Individual Differences 08/2015; 87:130-135. DOI:10.1016/j.paid.2015.07.041 · 1.86 Impact Factor
Show more