A Guide to Conducting Behavioral Research on Amazon's Mechanical Turk

Yahoo! Research, New York, USA.
Behavior Research Methods (Impact Factor: 2.93). 06/2011; 44(1):1-23. DOI: 10.3758/s13428-011-0124-6
Source: PubMed


Amazon's Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, stable, and diverse subject pool, the low cost of doing experiments, and faster iteration between developing theory and executing experiments. While other methods of conducting behavioral research may be comparable to or even better than Mechanical Turk on one or more of the axes outlined above, we will show that when taken as a whole Mechanical Turk can be a useful tool for many researchers. We will discuss how the behavior of workers compares with that of experts and laboratory subjects. Then we will illustrate the mechanics of putting a task on Mechanical Turk, including recruiting subjects, executing the task, and reviewing the work that was submitted. We also provide solutions to common problems that a researcher might face when executing their research on this platform, including techniques for conducting synchronous experiments, methods for ensuring high-quality work, how to keep data private, and how to maintain code security.

820 Reads
  • Source
    • "2. Method 2.1. Participants and procedure A total of 1248 U.S. residents were recruited from Amazon's Mechanical Turk (Mason & Suri, 2012 "
    [Show abstract] [Hide abstract]
    ABSTRACT: "Selfies" are amateur photographs people take of themselves, usually with a smartphone. Sharing selfies on social media has become a popular activity, prompting questions about its psychological meaning and dispositionally-relevant motives. This study was performed to examine the association between narcissism, a personality trait characterized by inflated self-views and attempts to seek attention and admiration from others, and frequency of posting selfies on social networking sites. In addition, the association between posting selfies and three facets of narcissism (i.e., Leadership/Authority, Grandiose Exhibitionism, Entitlement/Exploitativeness) was explored. These questions were addressed in a nationally representative sample of 1204 men and women who completed an online survey. Results showed that narcissism, as well as the Leadership/Authority and Grandiose Exhibitionism facets, but not Entitlement/Exploitativeness, exhibited positive and significant associations with selfie-posting frequency. Age did not moderate the predictive effects of narcissism or any of its three dimensions, indicating that the relationship between narcissism, its facets, and posting selfies is not age dependent. However, the more adaptive Leadership/Authority facet emerged as a stronger predictor of selfie posting among women than men, whereas the maladaptive Entitlement/Exploitativeness facet predicted selfie posting among men, but not women. Interpretations and implications of these findings are discussed.
    Personality and Individual Differences 11/2015; 86:477-481. DOI:10.1016/j.paid.2015.07.007 · 1.86 Impact Factor
  • Source
    • "Such directions are currently already being successfully pursued using social media data (e.g., Eichstaedt et al., 2015; Park et al., 2015), and we expect that they will soon also start to shape the field of psychology as a whole. Online testing comes with many opportunities (see also Crump et al., 2013; Behav Res Griffiths, 2015; Mason & Suri, 2012). For example, it will allow us to test hypotheses within a much larger domain of potentially interesting psychological dimensions, thus allowing us to draw landscapes of generalizability (Brunswik, 1955), instead of the traditional approach of testing a small convenience sample and merely hoping that the effects will generalize to different populations (cf. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In a recent letter, Plant (2015) reminded us that proper calibration of our laboratory experiments is important for the progress of psychological science. Therefore, carefully controlled laboratory studies are argued to be preferred over Web-based experimentation, in which timing is usually more imprecise. Here we argue that there are many situations in which the timing of Web-based experimentation is acceptable and that online experimentation provides a very useful and promising complementary toolbox to available lab-based approaches. We discuss examples in which stimulus calibration or calibration against response criteria is necessary and situations in which this is not critical. We also discuss how online labor markets, such as Amazon's Mechanical Turk, allow researchers to acquire data in more diverse populations and to test theories along more psychological dimensions. Recent methodological advances that have produced more accurate browser-based stimulus presentation are also discussed. In our view, online experimentation is one of the most promising avenues to advance replicable psychological science in the near future.
    Behavior Research Methods 11/2015; DOI:10.3758/s13428-015-0677-x · 2.93 Impact Factor
  • Source
    • "A requester can reward the work or can refuse to pay for poor quality of work. Majority of the workers are at age range of 20-40 years and are females (Mason and Suri, 2012; Paolacci et al., 2010). Qualitative constraints and quality management are essential in successful microtasking (Kern et al., 2009). "
Show more