A Guide to Conducting Behavioral Research on Amazon's Mechanical Turk

Yahoo! Research, New York, USA.
Behavior Research Methods (Impact Factor: 2.93). 06/2011; 44(1):1-23. DOI: 10.3758/s13428-011-0124-6
Source: PubMed


Amazon's Mechanical Turk is an online labor market where requesters post jobs and workers choose which jobs to do for pay. The central purpose of this article is to demonstrate how to use this Web site for conducting behavioral research and to lower the barrier to entry for researchers who could benefit from this platform. We describe general techniques that apply to a variety of types of research and experiments across disciplines. We begin by discussing some of the advantages of doing experiments on Mechanical Turk, such as easy access to a large, stable, and diverse subject pool, the low cost of doing experiments, and faster iteration between developing theory and executing experiments. While other methods of conducting behavioral research may be comparable to or even better than Mechanical Turk on one or more of the axes outlined above, we will show that when taken as a whole Mechanical Turk can be a useful tool for many researchers. We will discuss how the behavior of workers compares with that of experts and laboratory subjects. Then we will illustrate the mechanics of putting a task on Mechanical Turk, including recruiting subjects, executing the task, and reviewing the work that was submitted. We also provide solutions to common problems that a researcher might face when executing their research on this platform, including techniques for conducting synchronous experiments, methods for ensuring high-quality work, how to keep data private, and how to maintain code security.

Download full-text


Available from: Winter Mason
    • "All registered users of MTurk were eligible for our pilot study if they were between 18 and 25 years old, were racial/ethnic minorities (i.e., Latino, African Americans, Asian Americans), and attended high school in the United States. To ensure the quality of the sample and data, we also limited participants to those who had U.S. IP addresses with a 95% approval rate (an indicator of response quality) on MTurk (Mason & Suri, 2012). Users who met all criteria were invited to participate in the study. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Racial/ethnic minority youth live at the intersection of diverse cultures, yet little is known about cultural socialization outside families or how cultural socialization in multiple settings conjointly influences adolescent well-being. In a sample of 236 8th graders (51 % female; 89 % Latinos, 11 % African Americans), we examined adolescents’ perceptions of family and peer cultural socialization toward the heritage culture and the mainstream American culture. A variable-centered approach demonstrated that the socioemotional and academic benefits of family cultural socialization were most evident when peer cultural socialization was congruently high. Although family and peer cultural contexts are often assumed to be drastically different, we identified similar proportions of adolescents experiencing congruently high, congruently low, and incongruent cultural socialization from families and peers using a person-centered approach. Although the incongruent group received relatively high levels of cultural socialization in one setting, their well-being was similar to the congruently low group. The findings highlight the importance of considering cultural socialization across multiple developmental settings in understanding racial/ethnic minority youth’s well-being.
    No preview · Article · Jan 2016 · Journal of Youth and Adolescence
  • Source
    • "Mechanical Turk and received $1 for completing the task. The effectiveness of this method of subject recruitment has been demonstrated in several studies that find MTurk samples to be generally reliable and comparable to more expensive methods [14] [15] [16] [17] [18]. This new platform opens up the possibility of recruiting this larger number of subjects with relative ease and at low cost. "
    [Show abstract] [Hide abstract]
    ABSTRACT: As systems grow in complexity, there are more tasks that people want to complete simultaneously. Prior research has shown that receiving interruptions hurts people's performance. Our research intends to better understand the performance implications of users multitasking during simple and complex tasks by examining task difficulty both objectively and subjectively. We created a web-based word search puzzle as the primary task with an objectively easy and hard version. 726 were randomized into one of four conditions. In the first two conditions, participants received the objectively easy or hard version of the primary task and did not receive any interruptions. In the latter two conditions participants received the easy and hard versions with interruptions. Subjective difficulty was measured based on the participants’ opinions of the primary task. While we did not find any significant differences in conditions with the objective or subjective divisions, however, when participants perceived the interrupting task as difficult receiving interruptions during both the easy and hard conditions helped participants perform better. This was only true for the subjective breakdown. These findings suggest that the difficulty level of the interrupting task may impact users’ performance outcome when receiving interruptions. We also found that while there was no significant correlation between participants’ propensity to multitask and performance in the interrupting conditions, when examining those who did not receive interruptions, participants’ performance significantly positively correlated with their propensity to multitask. This implies that multitasking users perform better than non-multitasking users when mono-tasking.
    Full-text · Article · Dec 2015
  • Source
    • "Several studies have assessed this tool and concluded that this approach results in high-quality and reliable data (e.g., Buhrmester et al., 2011; Saunders et al., 2013; Rouse, 2015) that is more representative than many other samples (Mason and Suri, 2012; Rouse, 2015). Participants were given the following information before taking the survey: " Take a short survey asking your opinion of dairy farms. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Practices in agriculture can have negative effects on the environment, rural communities, food safety, and animal welfare. Although disagreements are possible about specific issues and potential solutions, it is widely recognized that public input is needed in the development of socially sustainable agriculture systems. The aim of this study was to assess the views of people not affiliated with the dairy industry on what they perceived to be the ideal dairy farm and their associated reasons. Through an online survey, participants were invited to respond to the following open-ended question: "What do you consider to be an ideal dairy farm and why are these characteristics important to you?" Although participants referenced social, economic, and ecological aspects of dairy farming, animal welfare was the primary issue raised. Concern was expressed directly about the quality of life for the animals, and the indirect effect of animal welfare on milk quality. Thus participants appeared to hold an ethic for dairy farming that included concern for the animal, as well as economic, social, and environmental aspects of the dairy system.
    Full-text · Article · Dec 2015 · Journal of Dairy Science
Show more