Conducting behavioral research on Amazon’s Mechanical Turk

Yahoo! Research, New York, USA.
Behavior Research Methods (Impact Factor: 2.12). 06/2011; 44(1):1-23. DOI: 10.3758/s13428-011-0124-6
Source: PubMed

ABSTRACT Amazon's Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, stable, and diverse subject pool, the low cost of doing experiments, and faster iteration between developing theory and executing experiments. While other methods of conducting behavioral research may be comparable to or even better than Mechanical Turk on one or more of the axes outlined above, we will show that when taken as a whole Mechanical Turk can be a useful tool for many researchers. We will discuss how the behavior of workers compares with that of experts and laboratory subjects. Then we will illustrate the mechanics of putting a task on Mechanical Turk, including recruiting subjects, executing the task, and reviewing the work that was submitted. We also provide solutions to common problems that a researcher might face when executing their research on this platform, including techniques for conducting synchronous experiments, methods for ensuring high-quality work, how to keep data private, and how to maintain code security.

  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Online syndicated text-based advertising is ubiquitous on news sites, blogs, personal websites, and on search result pages. Until recently, a common distinguishing feature of these text-based advertisements has been their background color. Following intervention by the Federal Trade Commission (FTC), the format of these advertisements has undergone a subtle change in their design and presentation. Using three empirical experiments, we investigate the effect of industry-standard advertising practices on click rates, and demonstrate changes in user behavior when this familiar differentiator is modified. Using three large-scale experiments (N1 = 101, N2 = 84, N3 = 176) we find that displaying advertisement and content results with a differentiated background results in significantly lower click rates. Our results demonstrate the strong link between background color differentiation and advertising, and reveal how alternative differentiation techniques influence user behavior.
    Computers in Human Behavior 09/2015; 50. DOI:10.1016/j.chb.2015.04.031 · 2.69 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Mathematical fluency is important for academic and mathematical success. Fluency training programs have typically focused on fostering retrieval, which leads to math performance that does not reliably transfer to non-trained problems. More recent studies have focused on training number understanding and representational precision, but few have directly investigated whether training improvements also transfer to more advanced mathematics. In one previous study, university undergraduates who extensively trained on mental computation demonstrated improvements on a complex mathematics test. These improvements were also associated with changes in number representation precision. Because such far transfer is both rare and educationally important, we investigated whether these transfer and precision effects would occur when using a more diverse population and after removing several features of the mental computation training that are difficult to implement in classrooms. Trained participants showed significant, robust improvements, suggesting that mental computation training can reliably lead to mathematical transfer and improvements in number representation precision.
    Instructional Science 07/2015; 43(4). DOI:10.1007/s11251-015-9350-0 · 1.83 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: The present study investigates how the construct intellect, according to the Theoretical Intellect Framework (TIF), predicts creativity. The TIF is a theoretical model describing the structure of the construct intellect, a sub-dimension of the Big Five domain openness to experience. People (N = 2709) from two sub-samples (undergraduate students and Amazon MTurkers) completed one of three creativity tasks (self-reported, remote associates, or rated photo caption) and the Intellect Scale. The results support hypotheses derived from the TIF, as the operation Create, rather than the operations Think or Learn, significantly and in some cases uniquely predicted the self-reported creativity indicators. Creativity indicators with a strong cognitive load (remote associates test and rated photo caption), however, were predicted by the operation Think. Results are discussed with regards to the nomological net of the operation Create and the construct validity of the creativity assessments. We provide implications for applied purposes and call for further examination of the TIF with additional creativity measures. Copyright © 2015 European Association of Personality Psychology
    European Journal of Personality 04/2015; DOI:10.1002/per.2000 · 2.44 Impact Factor