ArticlePDF Available

Abstract and Figures

The present article reviews web-based research in psychology. It captures principles, learnings, and trends in several types of web-based research that show similar developments related to web technology and its major shifts (e.g., appearance of search engines, browser wars, deep web, commercialization, web services, HTML5…) as well as distinct challenges. The types of web-based research discussed are web surveys and questionnaire research, web-based tests, web experiments, Mobile Experience Sampling, and non-reactive web research, including big data. A number of web-based methods are presented and discussed that turned out to become important in research methodology. These are one-item-one-screen design, seriousness check, instruction manipulation and other attention checks, multiple site entry technique, subsampling technique, warm-up technique, and web-based measurement. Pitfalls and best practices are described then, especially regarding dropout and other non-response, recruitment of participants, and interaction between technology and psychological factors. The review concludes with a discussion of important concepts that have developed over 25 years and an outlook on future developments in web-based research.
Content may be subject to copyright.
Review Article
Web-Based Research
in Psychology
A Review
Ulf-Dietrich Reips
Psychological Methods and Assessment / Experimental Psychology and Internet Science, Department of Psychology,
University of Konstanz, Germany
Abstract: The present article reviews web-based research in psychology. It captures principles, learnings, and trends in several types of web-
based research that show similar developments related to web technology and its major shifts (e.g., appearance of search engines, browser
wars, deep web, commercialization, web services, HTML5...) as well as distinct challenges. The types of web-based research discussed are
web surveys and questionnaire research, web-based tests, web experiments, Mobile Experience Sampling, and non-reactive web research,
including big data. A number of web-based methods are presented and discussed that turned out to become important in research
methodology. These are one-item-one-screen design, seriousness check, instruction manipulation and other attention checks, multiple site
entry technique, subsampling technique, warm-up technique, and web-based measurement. Pitfalls and best practices are described then,
especially regarding dropout and other non-response, recruitment of participants, and interaction between technology and psychological
factors. The review concludes with a discussion of important concepts that have developed over 25 years and an outlook on future
developments in web-based research.
Keywords: Internet-based research, online experiments, online research, online assessment, web surveys
While Internet-based research began shortly after the
invention of the Internet in the 1960s, web-based research
in psychology began only in the mid-1990s once the world
wide web (short: web) had been invented by Tim Berners-
Lee in Geneva in 1992. Web browsers had become avail-
able, and subsequently, the http protocol had been
amended by the functionality of forms.Formsalloweda
web browser user to send back responses to what someone
had set up on a web page as response options, for example,
radio buttons, drop-down menus, check boxes, or text fields
(Birnbaum, 2004;Reips,2000).
Web-based research naturally relies on technology and
the principles of remote research. As we have seen, a num-
ber of technologies have competed and continue to com-
pete for a place among essential requirements. Surviving
technologies and concepts are from both server- and cli-
ent-side (with advancements in quick switching and inte-
gration first via AJAX, then in HTML5technology).
Recently, however, Garaizar and Reips (2019)havecon-
cluded that the complexity of browser technologies has
increased so much that web-based research is facing diffi-
culties that will continue to cumulate. Anwyl-Irvine and col-
leagues (2021) compared the precision and accuracy of
some online experiment platforms, web browsers, and
devices. Their data confirm what became clear as an intrin-
sic difference at the onset of web-based research: Variance
in any measurement is higher than under laboratory-based
conditions, especially compared to calibrated equipment of
a limited type. Variance on the web comes unsystematically
and systematically, as in earlier research, Anwyl-Irvine and
colleagues found differences between online experiment
platforms, web browsers (and operating systems), and
devices. While during the first decades of web-based
research, browsers (and main technologies) were updated
only every few years or months, browser vendors now
update weekly or even daily (Garaizar & Reips, 2019;see
e.g., for Firefox at
fox_version_history). From a researchers point of view,
both the stimulus display technology and the measurement
device change frequently during data collection. Further-
more, browsers are not being optimized to meet research-
ersneeds. The main reason behind some of the
difficulties is the browser vendorsmotivation to optimize
browsers in terms of the user experience to serve their com-
mercial outlook. Browser vendors primarily want to sell, not
help science find the truth.
With the advent of new technologies, the story of web-
based research in psychology continues in a similar vein,
yet to new frontiers. Similar to Musch and Reips (2000,
replicated by Krantz & Reips, 2017), who once surveyed
the first web experimenters, Ratcliffe and colleagues
(2021) recently web-surveyed those who currently pioneer
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under the license CC BY 4.0 (
Open Access, available from
remote research with Extended Reality (XR) technology
such as virtual and augmented reality. They found a similar
hope by XR researchers to the early web researchers for
benefits of remote research for increasing the amount,
diversity and segmentation of participants compared with
in-lab studies(Wolfe, 2017,p.9) and also in parallel to
web-based research. In reality that this hope rarely materi-
alizes, probably as a result of many technical and procedu-
ral limitations. While XR researchers see opportunities to
further develop web-based XR, the history of web-based
research shows that research requiring any addition to the
core interface the browser may be doomed to fail or
end up in a small niche of limited showcases that never
make it to routine research methods.
Types of Web-Based Research
Essentially, all traditional methods in behavioral research
have been transferred to digital research in one way or
the other. However, naturally, some methods transfer more
easily than others, which may change over time with the
change of technology. Major shifts (now sometimes trendily
called disruptions) in web technology always caused new
methods to flourish. Examples are (1) the implementation
of forms in HTML in the mid-90s suddenly allowed people
to easily send data back to the servers from websites, inspir-
ing web pioneers in psychology to create the first web sur-
veys, experiments, and tests (like André Hahn, who owned
the domain and is believed to have devel-
oped the first web-based psychological test,
on perceived
self-efficacy); (2) big data methods are evolving from the
appearance of search engines and large hubs on the web,
for example, picture sharing sites or public databases, with
their data being made available (e.g., a method is to use
SECEs search engine count estimates in research,
Janetzko, 2008); (3)webmethodsthatrelyonquickinter-
action between client and server (e.g., AJAX, social interac-
tion online, video transmission) have become possible and
increasingly better accessible with higher bandwidth and
ubiquity of mobile connections and flat rates; (4)digital
experience sampling methodology evolving from the inven-
tion and subsequent proliferation of smartphones, smart-
watches, and other wearables. Some other technologies
have been on the verge of a breakthrough for a long time
and may make it in the future or not, for example, VR
and augmented reality in, for example, Google glasses.
From a taxonomy valid for the first decade of web-based
research Reips and Lengler (2005) empirically determined
from data for reactive studies on the web that within psy-
chology web-based research mostly was conducted in cog-
nitive and social psychology, followed by perception and a
few studies each in personality, clinical and Internet
science. A later review and survey by Krantz and Reips
(2017) revealed a largely unchanged picture. Reips and Len-
gler also explicitly introduced the concept of web services,
which is software that runs on a server on the Internet:
Users access it via a web browser and can use it only while
they are connected to the Internet. Because the functional-
ity of web browsers is less dependent on the operating sys-
tem ... all who access a web service are likely to see and
experience almost the same interface (but see, e.g., Dillman
&Bowker,2001, for browser-related problems in Internet-
based research). Web services spare the user from upgrad-
ing and updating since this is done by the web service
administrators at the server. Nothing is installed on the
users computer, saving space and time(p. 287). Curiously,
in a way with web services (now often called apps)and
the cloudwe have returned to the server-terminal model
common in the 70sand80s that was interrupted by a brief
phase of only temporarily connected personal computers.
Importantly, when discussing characteristics of web-
based research, it is important to not mistakenly attribute
the advantages of computerized assessment to the web
method only. Many useful functionalities such as item
branching, precise timing, filtering, automatic checks of
plausibility during data entry, and so on, were already intro-
duced to experimenting during the computer revolution in
the late 1960s and early 1970s (Reips & Krantz, 2010).
Characteristics of computerized research are valid in
web-based research also, but its true advancement comes
from its structure and reach as a world-wide network.
In the following, I will present characteristics and develop-
ments of the methods of web surveys and questionnaire
research (including web-based tests), web experiments,
Mobile Experience Sampling, and non-reactive web research.
Web Surveys and Questionnaire Research
This type of web-based research is the most frequently
used, yet it also may be the most error-prone. A general
advantage of surveying as a methodology is its ease, and
thus we see much ad hoc use with everything associated
with quick-and-dirty approaches. Related, many errors fre-
quently seen in web questionnaires came and come from a
lack of understanding of the technology and that a direct
transfer from the paper-based or computer-based formats
to an Internet-based format is impossible.
Still available at the web archive also contains it at*/userpage. The self-scoring script, a CGI, is not working anymore, at the time it would calculate and show the test takers their own
score, their percentile and the distribution of all results.
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 199 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
Some of the typical pitfalls are technological. For exam-
ple, Birnbaum (2004) observed a case of erroneous coding:
in that web questionnaire, if a person were from India or if
he or she did not respond, the coded value was the same,
99.Reips(2002a) highlights that the way HTML forms
work may lead to overwriting of data due to the same
names associated with form elements, for example, assign-
ing the variable name sexto both an item reporting ones
own sex and an item asking about frequency of sexual
In a recent meta-analysis, Daikeler and colleagues
(2020) confirmed earlier findings that response rates in
web surveys are lower than in other survey modes, roughly
by 12%. The expected length of web questionnaires is, of
course, a major factor in a respondents decision-making
for participation and response. Galesic and Bosnjak
(2009) experimentally varied both expected (10,20,and
30 min) and actual questionnaire length. They found, as
expected, the longer the stated length, the fewer respon-
dents started and completed the questionnaire. In addition,
answers to questions positioned later in the questionnaire
were faster, shorter, and more uniform than answers to
questions positioned near the beginning(p. 349).
Reips (2010) lists several typical errors that frequently
happen when constructing online questionnaires:
preselected answer options in drop-down menus,
resulting either in submission of the default answer
option as a chosen answer when the item is really
skipped or provoking an anchoring effect,
overlapping answer categories,
no limitations set or announced to size of text to be
entered in text fields,
lack of options that indicate reluctance to answer (e.g.,
dont want to answeror no answer), especially for
sensitive items,
all items on one run-on web page (see OIOS Technique
section below),
incorrect writing (e.g., errors in instructions or
An excellent resource for many topics around web survey
methodology is Callegaro and colleagues (2015). It summa-
rizes many important learnings from the scientific literature
on moving surveying to the web and supersedes many ear-
lier attempts in scientific depth and applicability.
Despite the many benefits of web-based research,
researchers and others have expressed concerns about
online data collection in surveying. For the important case,
in which generalization to a country population is needed,
Dillman and colleagues (2010), in their editorial Advice
in surveying the general public over the Internetfor the
International Journal of Internet Science, make the prediction
The first decade of this century in web surveying is likely
to be recalled as a time of much uncertainty on whether
random samples of the general public could be surveyed
effectively over the Internet. A significant proportion of
households in most countries are not connected to the
Internet, and within households, some members lack the
Internet skills or frequency of use needed for responding
to survey requests. In addition, households without Internet
access differ sharply from those with it. Non-Internet
households are older, have less education, and have lower
incomes. Their inclusion in surveys is critical for producing
acceptable population estimates, especially for public policy
purposes. Web survey response rates in general public sur-
veys are often dismal.(p. 1). Despite the skeptical state of
web surveying the general public, the authors made five
recommendations that are likely to improve the situation,
namely (1) use of a mixed-mode approach, (2) delivering
a token cash incentive with the initial mail request, (3)using
a mail follow-up to improve response rates and obtain a bet-
ter representation of the general public, (4) refraining from
offering people a choice of whether to respond by web or
mail in initial contacts, and (5) using an experimental
approach, in order to generate estimates for the meaning
and sizes of various effects on response rates and non-
response. Loomis and Paterson (2018) empirically investi-
gated the challenges identified by Dillman and colleagues
and found limited differences between survey modes when
aggregating all results (in their case from 11 studies). Only
the non-deliverable rate seemed consistently higher in
online surveying than in mail surveying. Non-response
rates, item non-response, and content of the results showed
no or only small differences for the aggregated data. The
authors interpreted differences in response rates according
to mode as random or idiosyncratic in nature, and perhaps
more a matter of study population or topic than of mode
(p. 145).
A specific form of web surveying, if it considers verbal
items, is web-based psychological testing (e.g., of personality
or other individual differences constructs). A researcher
who has pioneered this branch of web-based research is
Tom Buchanan (e.g., Buchanan, 2000,2001; Buchanan
& Smith, 1999; Buchanan et al., 2005). His and others
studies generally find equivalency to offline testing, with
notable exceptions that indicate that any web-based test
should be scrutinized with the standard reliability and valid-
ity checks during test development.
Web Experiments
Web-based experimenting has evolved since 1994 and was
first presented at the SCiP Conference in Chicago 1996,
with Krantz presenting the first within-subjects web experi-
ment (Krantz et al., 1997)andReips(1996)presenting
the first between-subjects web experiment and the first
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
200 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
experimental laboratory on the world wide web, the Web
Experimental Psychology Lab (Reips, 2001). On track of
the question whether lab and web and type of participant
recruitment are comparable, Germine and colleagues
(2012) addressed the question of data quality across a range
of cognitive and perceptual tasks. Their findings on key per-
formance metrics demonstrate that collecting data from
uncompensated, anonymous, unsupervised, self-selected
participants on the web need not reduce data quality, even
for demanding cognitive and perceptual experiments.
Reips points out the ultimate reason for using the web to
conduct experiments:
The fundamental asymmetry of accessibility (Reips,
2002b, 2006): What is programmed to be accessible
from any Internet-connected computer in the world
will surely also be accessible in a university labora-
tory, but what is programmed to work in a local com-
puter lab may not necessarily be accessible anywhere
else. A laboratory experiment cannot simply be
turned into a web experiment, because it may be pro-
grammed in a stand-alone programming language
and lack Internet-based research methodology, but
any web experiment can also be used by connecting
the laboratory computer to the Internet. Conse-
quently, it is a good strategy to design a study web-
based, if possible. (2007b, pp. 375376).
As a consequence, the many advantages of web-based
experimenting, for example,
(1) web experiments are more cost-effective in adminis-
tration, time, space, and work in comparison with
laboratory research,
(2) ease of access for participants, also from different
cultures and for people with rare characteristics (for
accessibility see Vereenooghe, 2021),
(3) web experiments are generally truly voluntary,
(4) detectability of confounding with motivational aspects
of experiment participation,
(5) replicability and re-usability, as the materials are
publicly accessible,
quickly led to the new methods proliferation (Birnbaum,
2004;Krantz&Reips,2017;Musch&Reips,2000; Reips,
2000,2007b; Wolfe, 2017). As a consequence of these
characteristics of web experiments, they frequently have
been shown to collect higher quality data than laboratory
experiments (Birnbaum, 2001; Buchanan & Smith, 1999;
Reips, 2000). For example, in a field where all past studies
from more than two decades had two digit sample sizes,
spatialnumerical association, web experiments finally pro-
vided detailed results with small confidence intervals
(Cipora et al., 2019). Tan and colleagues (2021) were highly
successful in objectively measuring singing pitch accuracy
on the web. With moderate-to-high test-retest reliabilities
(.65.80), even across an average 4.5-year period between
test and retest (!) they see high potential for large-scale
web-based investigations of singing and music ability. In
some areas of psychology, more than half of all studies pub-
lished are now conducted online, for example, in social psy-
chology however, due to a certain participant recruitment
method discussed further below, the widespread use is not
without problems (Anderson et al., 2019).
Beyond what to researchers continues to be a new way
and sometimes challenging and questionable advancement
beyond laboratory experimental research, web experiments
have truly revolutionized digital business. In his bestseller
Seth Stephens-Davidowitz (2017)writes:
Experiments in the digital world have a huge advan-
tage relative to experiments in the offline world. As
convincing as offline randomized experiments can
be, they are also resource-intensive. ... Offline exper-
iments can cost thousands or hundreds of thousands
of dollars and take months or years to conduct. In the
digital world, randomized experiments can be cheap
and fast. You do not need to recruit and pay partici-
pants. Instead, you can write a line of code to ran-
domly assign them to a group. You do not need
users to fill out surveys. Instead, you can measure
mouse movements and clicks. You do not need to
hand-code and analyze the responses. You can build
a program to automatically do that for you. You do
not have to contact anybody. You do not even have
to tell users that they are part of an experiment. This
is the fourth power of big data: it makes randomized
experiments, which can find truly causal effects,
much, much easier to conduct anytime, more or less
anywhere, as long as youre online. In the era of big
data, all the worlds a lab. This insight quickly spread
through Google and then the rest of Silicon Valley,
where randomized controlled experiments have been
renamed A/B testing.In 2011, Google engineers ran
seven thousand A/B tests. And this number is only
rising. Facebook now runs a thousand A/B tests per
day, which means that a small number of engineers
at Facebook start more randomized, controlled
experiments in a given day than the entire pharma-
ceutical industry starts in a year (pp. 210211).
While early and quickly, a number of web survey generators
appeared on the market, with Internet-Rogatorby Heid-
ingsfelder (1997) being one of the first, only a few web experi-
ment generators are available today. For a recent listing of
software for experiments (laboratory or web) and links to
further resources, see the helpful Google page by Weichsel-
gartner (2021). Figure 1shows the newest version of
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 201 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
WEXTOR (, also available on the iScience
server at, one of the longest-standing
web experiment generators.
Mobile Experience Sampling
This method is sometimes also described under the terms
ecological momentary assessmentor ambulatory assess-
ment(Shiffman et al., 2008). It is a modern form of the
diary method and has its strength in the current ubiquitous
presence of smartphones, smartwatches, and other wear-
ables in large proportions of populations in many technolog-
ically advanced societies. For instance, Stieger and Reips
(2019) were able to replicate and refine past research about
the dynamics of well-being fluctuations during the day (low
in the morning, high in the evening) and over the course of
a week (low just before the beginning of the week, highest
near the end of the week) (Akay & Martinsson, 2009). The
method is more accurate in capturing the frequency and
intensity of experiences (Shiffman et al., 2008), but it is rel-
atively burdensome for both participants and researchers,
may lead to non-compliance, produces correlational data
only, and its reliability is hard to determine. With context-
sensitive experience sampling (see e.g., Klein & Reips,
2017), researchers can trigger questions based on times,
events, app usage or location or combinations thereof, for
example, ask for subjective well-being when someone
leaves university only on afternoons when smartphone sen-
sors registered a practical sports class was attended. More-
over, by using the experience sampling method, different
research questions can be analyzed regarding the use of
mobile devices in research, for example, whether well-
being can be derived from the tilt of the smartphone (Kuhl-
mann et al., 2016).
Akin to upbeat statements regarding the potential of
computers in psychology in the 1960sand1970sandexten-
sion of hopes for the impact of the web in psychological
research in the 1990s, the proponents of mobile research
declared, Smartphones could revolutionize all fields of
psychology and other behavioral sciences if we grasp their
potential and develop the right research skills, psych apps,
data analysis tools, and human subjects protections.
(Miller, 2012,p.221). Critically, however, the reliance on
consumer-grade devices in research comes with their lim-
ited reliability and fast-changing variance. For example,
smartwatches do not agree much in their recordings of
workout parameters, and their measures depend on walk-
ing speed (Fokkema et al., 2017). Furthermore, our study
on smartphone sensors showed that measurements and
their reliability vary by type and brand of smartphone and
operating system (Kuhlmann et al., 2021). Smartphone
sensing in the field will thus systematically suffer from
worse measurements than possible in controlled studies
in the laboratory.
Figure 1. WEXTOR 2021, a web experiment generator available from The figure shows the good methods by designphilosophy
implemented by WEXTORs authors, that is, methods and best practices for web-based research were implemented in a way that the
experimenter is nudged toward using them.
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
202 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
While mobile experience sampling research is often not
exactly web-based, the free open-source platform Samply
(Shevchenko et al., 2021; for experi-
ence sampling enables researchers to access the complete
interface via a web browser and manage their present stud-
ies. It allows researchers to easily schedule, customize and
send notifications linking to online surveys or experiments
created in any web-based service or software (e.g., Google
Forms, lab.js, SurveyMonkey, Qualtrics, A
flexible schedule builder enables a customized notification
schedule, which can be randomized for each participant.
The Samply research mobile application preserves partici-
pantsanonymity and is freely available at the Google
and Apple App Stores. Shevchenko and colleagues (2021)
demonstrated via two empirical studies the appsfunction-
ality and usability.
Non-reactive web-based methodsrefer to the use and
analysis of existing databases and text or media collections
on the Internet (e.g., forums, picture collections, server log
files, scanned document collections, newsgroup contribu-
tions). Such data can also include geolocation, that is, infor-
mation about the place that may allow analysts to identify
routes and timelines. The Internet provides an ocean of
opportunities for non-reactive data collection. The sheer
size of Internet corpora multiplies the specific strengths of
this class of methods: Non-manipulable events can be stud-
ied in natura, facilitating the examination of rare behavioral
patterns.(Reips, 2006,p.74). While lab-based research
creates social expectations that might motivate participants
to answer and perform in unusual ways, data from archival
or non-reactive research will not contain biases that come
from reacting to the research situation or personnel (hence
non-reactive). Such non-reactive research is easy to do on
the web.
For some archival data, there are even specific interfaces.
For example, upon its initiative to scan as many of the
worlds books as possible and make them available to the
public, Google also created a specific search engine to
search this corpus, Google Books Ngram Viewer (short: Goo-
gle Ngram, With it,
relative frequencies of words within the corpus can be ana-
lyzed per year, and thus, it is possible to create timelines
that show word use over time since the year 1800.Michel
and colleagues (2011) describe how this tool allows for
research options unprecedented in the history of science,
in data mining cultural trends as reflected in books. Younes
and Reips (2019) provide guidelines for such research, for
example, the use of synonyms, word inflections, and control
words to assess a words base rate in a given year and lan-
guage corpus.
Research That Could Not Be Done
Without the Web
Of course, there is a lot of web-based research that cannot
be done without the web because it is research that con-
cerns the web. I will not review this research here. Instead,
I will focus in this section on research that was impossible
or only very difficult to do before the advancement of the
Sensitive or Illegal Topics
Research with people who have rare and unusual character-
istics of interest used to be impossible to do or very costly
and burdensome. Similarly, for research asking sensitive
questions about illegal or taboo behaviors (e.g., drug deal-
ing, Coomber, 1997; or ecstasy and cannabis consumption,
Rodgers et al., 2001,2003)orforinformationthatthe
responders may be reluctant to disclose the web with its veil
of anonymity has become a promising route.
With just two web-based surveys conducted via the hub
of people concerned with having the rare condition sexsom-
nia and their family and friends,,
Mangan and
Reips (2007) reached more than five times as many partic-
ipants from the target population than in all previously pub-
lished studies combined.
Large Crowdsourcing Studies
In most cases, studies that rely on thousands of participants
or even more can be conducted much more efficiently and
less burdensome on the web. Deshpande and colleagues
(2016), for example, took a collection of thousands of so
far unanalyzed hand-marked forms, with which hundreds
of researchers and research assistants before the advent
of modern media and the Internet had collected color
names in almost 200 middle American cultures that used
different yet related languages, and crowdsourced helpers
on the web to categorize these entries. The results of this
ongoing project promise unparalleled insights into the
essentials of perception and language. Similar projects can
be found on Citizen science websites, for example, zooni-, where people from the general public can help
researchers via the web by categorizing images from outer
space or remote zones on Earth or read and classify diaries
and letters by soldiers who fought in the American Civil
War. Honing (2021) discusses citizen science studies in
music cognition.
Large collections of entries or traces from human behav-
ior on the Internet have become an accessible source for
research. Examples include the definition of points of inter-
est via data mining in uploaded pictures (Barras, 2009),
Site now dysfunctional, see web archive at, for example, for a
snapshot of the site.
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 203 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
prediction of influenza outbreaks from searches (Ginsberg
et al., 2009), and our own work on attributions of person-
ality characteristics to first names accessed via Twitter min-
ing (Reips & Garaizar, 2011). Upon the big success of its
search engine that became available on the web in 1997,
Google has created freely available interfaces to their
search data. These include Google Trends, Google Insights,
Google Correlations (now disconnected, like the more
specific services Google Flu Trends and Google Eurovi-
sion). These services can be used in psychological research,
sometimes with profound results achieved within hours, but
of course, there are also limitations, ethical issues, and sci-
entific principles that are sometimes at odds with character-
istics of big data services provided by companies.
Other big data research is, of course, possible without the
web, and all of it comes with a number of limitations. For
example, Back et al. (2011) showed for a big data study
on emotions in digital messaging how a term from an auto-
matically generated message that was frequently sent by
the system unknowingly completely messed up the results
and the conclusions that were first drawn from it because
it contained the word critical,which was interpreted as
a marker for anger. I expect us to see many more cases with
such artifacts in the future, along with the proliferation of
large-scale studies.
In principle, for many studies conducted in psychology,
no large participant crowds are needed. Even though power
was notoriously low in pre-web psychology research and
promises to become more adequate as web-based research
is more easily scalable, overpowering studies is just as bad
(e.g., Faul et al., 2009;Reips1997). There has been a bit of
bragging in some articles relying on web-based data collec-
tion in terms of how many participants had been reached
millions and millions”–, some of which then report pitiful
effect sizes, but really for methodological and ethical rea-
sons to request such large numbers of people to devote
their time usually is not needed and thus may fall back
on the authors of such articles as questionable practices.
Methods in Web-Based Research
While most methods in principle have been transferred to
the web, many needed to be modified and adapted to the
online format, so often, new challenges arose. Psychological
tests, for example, should not simply be transferred from
paper-and-pencil to the computerized format and then be
handled similarly as usual on the web. In principle, they
need to be evaluated and validated as web-based instru-
ments (Buchanan, 2001; Buchanan & Smith, 1999). For
reasons of space, the selection of methods presented here
will be limited to those that have large effects and decent
Design: One-Item-One-Screen (OIOS)
When designing a web-based instrument that consists of a
number of items, a researcher has to decide how many
items are going to be presented on one screen or whether
this may vary by device or participant. The OIOS or one-
item-one-screen design has several advantages, namely
Context effects (interference between items) are reduced
(Reips, 2002a), meaningful response times and drop out
can be measured and used as dependent variables, and
the frequency of interferences in hidden formatting is vastly
reduced(Reips, 2010,pp.3334). The design thus is rou-
tinely used in research studies, for example, in urban plan-
ning (Roth, 2006), where even a variant was developed, the
Two-item-one-screen design(Lindquist et al., 2016).
OIOS has been proposed as a recommended strategy in a
general framework for technology-based assessments by
Kroehne and Goldhammer (2018). They write, Item-level
response times from questionnaire items (e.g., Likert type)
are an interesting source of information above and beyond
item responses.(p. 543) and go on to criticize current
implementations of the Programme for International Stu-
dent Assessment (PISA) and other large-scale assessments
as missing out on such opportunities in web-based
Including other data (para-data, meta-data) apart from
the response itself, especially behavioral data that indicate
timing, navigation, and switching of answers (Stieger &
Reips, 2010) and thus can be important in identifying issues
during test and questionnaire construction as well as diag-
nostic indicators beyond content responses, likely will
become much more important in future surveying. Issues
with adaptivity (e.g., in responsive design) and ethical appli-
cation will continue to be discussed (see e.g., Hilbig &
Thielmann, 2021).
Seriousness Check
The seriousness check is a technique that can be used in all
reactive types of web-based research to significantly
improve data quality (Aust et al., 2013;Bayram,2018;
Musch & Klauer, 2002;Reips,2000,2009; Verbree et al.
2020). Revilla (2016) found only limited evidence support-
ing the technique, but hers was an underpowered study
with online panelists and a wording geared towards com-
mitment that resulted in very few non-serious
Nowadays, online studies are accessible to a large diver-
sity of participants. However, many people just click
through a questionnaire out of curiosity, rather than provid-
ing well-thought-out answers (Reips, 2009). This poses a
serious threat to the validity of online research (Oppen-
heimer et al., 2009;Reips,2002b, 2009). The seriousness
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
204 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
check addresses this problem (Figure 2): In this approach,
the respondents are asked about the seriousness of their
participation or for a probability estimate that they will
complete the entire study or experiment (Musch & Klauer,
2002). Thus, by using the seriousness check, irrelevant data
entries can be easily identified and be excluded from the
data analysis. To provide a rough idea of how large the seri-
ousness checks effect can be: It routinely was observed
that of those answering I would like to look at the pages
onlyaround 75% will drop, while of those answering I
would like to seriously participate nowonly ca. 1015%
will drop during the study. Overall, about 3050%ofvisi-
tors will fail the seriousness check, that is, answer Iwould
like to look at the pages only(Reips, 2009).
Figure 2depicts a possible seriousness check proposed by
Reips (2009). The participants are asked whether or not
they want to participate seriously in the experiment (I
would like to seriously participate now./I would like to
look at the pages only.). Seriousness checks can be imple-
mented before (e.g., Bernecker & Job, 2011)andafter(Aust
et al., 2013) participation in the study. Reips (2002b, 2008,
2009) has argued to conduct seriousness checks on the first
page of the experiment because this is the best predictor of
dropout rates and thus a measure of motivation. Addition-
ally, conducting a seriousness check before the completion
of the study can reduce the dropout rates (Reips, 2002a).
The participants answer to the check question serves as
self-commitment, as predicted by dissonance theory (Frick
et al., 2001). Bayram (2018) experimentally showed that
emphasizing the seriousness increased the degree of infor-
mation participants accessed and their time they spent on
the study. The technique has been shown to predict dropout
and control for motivational confounding (Aust et al., 2013;
Bayram, 2018; Musch & Klauer, 2002;Reips,2002b, 2008,
2009). Some tools for web-based research, for example,
WEXTOR (Reips & Neuhaus, 2002;,
implement the seriousness check by default.
In a study with 5,077 participants representative of the
Dutch population in education, gender, and age over 15
years, Verbree and colleagues (2020) recently confirmed
that self-reported seriousness and motivation significantly
predict multiple data quality indicators. In preparing a study
that includes a seriousness check, it may be important to
know that their results showed that self-reported serious-
ness varies with demography.
Instructional Manipulation Check (IMC)
and Other Attention Checks
The IMC (Oppenheimer et al., 2009) was created with the
same intention in mind as the seriousness check, to identify
and avoid data from participants who are not as attentive in
online studies as is required to gather quality data. In the
case of the IMC, the focus is on attention during instruc-
tions. Of course, if participants do not properly attend to
instructions, it is unlikely they will provide valid data during
any subsequent task. Hence, screening them out at the
beginning makes sense. However, there are several other
attention checks that were designed to verify attention dur-
ing tasks we will also briefly look at those further below.
In the IMC, a request for unusual navigation is hidden
within the instructions, for example, to not click on the sub-
the text or a small blue dot. Only those who read the
instruction carefully and comply will follow the navigation,
and only their data will be analyzed later.
Another attention check, the Cognitive Reflection Test
(CRT; Frederick, 2005) is a frequently used measure of
cognitive vs. intuitive reflection, sometimes also discussed
as a measure of numerical ability. A typical task it includes
is the bat-and-the-ball problem: A bat and a ball cost $1.10
in total. The bat costs $1.00 more than the ball. How much
does the ball cost? ...cents. The intuitive answer most
inattentive participants go for is 10 cents, but the correct
answer is 5cents. With the widespread use of the task as
a nice riddle and as an attention check in web-based
research, many people have become familiar with this task.
Indeed, Stieger and Reips (2016)foundinalargestudythat
44%ofthemorethan2,200 participants were familiar with
the task or a similar task and scored substantially higher on
the test (Cohensd=0.41). They also found that familiarity
varies with sociodemographics. Web researchers should,
therefore, better use lesser-known attention check items
or the methods discussed above.
Multiple Site Entry Technique
The multiple site entry technique I first proposed in the
based research to target different samples via different
recruitment sites and compare their data. The method
can be used in behavioral and social research to assess
Figure 2. Seriousness check technique (adapted from Reips, 2009).
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 205 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
the presence and impact of self-selection effects. Self-selec-
tion effects can be considered a major challenge in social
science research. With the invention of online research in
the 1990s, the multiple site entry technique became possi-
ble because the recruitment of participants via different
links (URLs) is very easy to implement. It can be assumed
that there is no or very limited self-selection bias if the data
sets coming from different recruitment sites do not differ
systematically (Reips, 2000). This implies that the results
find no sample differences indicate high generalizability.
Implementing the multiple site entry technique works as
follows: Several links to the study are placed on different
websites, in Internet forums, social media platforms, or off-
line media that are likely to attract different types of partic-
ipants or are mailed out to different mailing lists. In order to
identify the recruitment sources, the published URLs con-
tain source-identifying information, and the HTTP protocol
is analyzed by different referrer information (Schmidt,
2000). This means a unique string of characters is
appended to the URL for each recruitment source, for
example, ...index.html?source=studentlistfor a study
announcement mailed to a list of students. The data file will
haveacolumn(sourcein the example) containing an
entry of the referring source for each participant (stu-
dentlistin the example). The collected datasets can then
be compared for differences in results and differences in
relative degree of appeal (measured via dropout), demo-
graphic data, central results, and data quality (Reips,
2002b). Figure 3illustrates the multiple-site entry
Several studies have shown that the multiple site entry
technique is useful for determining the presence and
impact of self-selection in web-based research (Reips,
2000,2002a; Roth, 2006). Now, in the spring of 2021,
Google Scholar shows more than 75 publications that men-
tion the technique. It has been used in memory (Kristo
et al., 2009), personality (Bluemke & Zumbach, 2012;
Buchanan et al., 2005;Trapnell&Campbell,1999), trauma
surveys (Hiskey & Troop, 2002), cross-cultural music-lis-
tening (Boer & Fischer, 2011), landscape research (Roth,
2006), criminological psychology (Buchanan & Whitty,
2014), and political psychology (Kus et al., 2014)andit
has entered the methodological discussion in the fields of
experimental survey research (Holbrook & Lavrakas,
2019), sex research (Mustanski, 2001), and health science
(Whitehead, 2007). Rodgers and colleagues (2003)used
the multiple site entry technique to detect biased responses
to their web questionnaire on drug use (subsequently vali-
dated by finding discussions of their research in forums
on that particular recruitment website). The multiple site
entry technique thus helps to detect potential sampling
problems, which in turn ensures the quality of the data
collection over the Internet (Dillman et al., 2010;Reips,
2002b). Therefore, the generalizability of the research
results can be estimated when using the multiple-site entry
technique (Reips, 2002a).
Subsampling Technique
Data quality may vary with a number of factors (e.g.,
whether personal information is requested at the beginning
or end of a study, Frick et al., 2001, or whether participants
are not allowed to leave any items unanswered and, there-
fore, show psychological reactance, Reips, 2002b). Subsam-
pling analysis is a verification procedure. For a random
sample drawn from all data submitted (e.g., from 50 out
of 1,500 participants), every possible measure is taken to
verify the responses, resulting in an estimate for the whole
dataset. This technique can help estimate the prevalence of
wrong answers by checking verifiable responses (e.g., age,
gender, occupation) both by specific item and via the aggre-
gate for any item. Ray and colleagues (2010)considerthis
technique as one possible option to better verify age and
thus protect children on the Internet.
Buchanan and colleagues (2005,p.120) noted that such
ways of estimating the degree of potential data contamina-
tionalong with other control procedures would need to be
developed and researched more in the future. While a lot
has happened in data analysis, for example, in the applica-
tion of Benfords law (Benford, 1938) to survey data (Judge
& Schechter, 2009), the advances in design and procedure
of web-based research (or all types of research, for that
matter) need further research and development.
Warm-Up Technique
The warm-up technique in Internet-based experimenting,
first proposed by Reips (2001), is a method that can be used
to avoid dropout during the experimental phase of a study
to maximize the quality of the data. This technique is based
Figure 3. Illustration of the multiple site entry technique (adapted
from Reips, 2009).
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
206 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
on the finding that most dropout will occur at the beginning
of an online study (Reips, 2002c). Therefore, participants
are presented with tasks and materials similar to the exper-
imental materials before the actual experimental manipula-
tion is introduced.
By using this technique, one can counter three main
problems in web-based research: Firstly, the dropout during
the actual experiment will be lower (Reips, 2001). Secondly,
the dropout cannot be attributed to the experimental
manipulation (Reips, 2002a, 2002b). Thirdly, only highly
committed participants will stay in the experiment, and
thus, the quality of the collected data is improved. Reips
and colleagues (2001) showed that by using the warm-up
technique the dropout during the actual experiment was
extremely low (< 2%). In comparison, the average dropout
rate in web-based research is much higher, in a review sum-
marizing previous web-based experimental research,
Musch and Reips (2000) found it to average at 34%.
The technique was used in personality and health behav-
ior research (Hagger-Johnson & Whiteman, 2007), motiva-
tion (Bernecker & Job, 2011), landscape perception
(Lindquist et al., 2016;Roth,2006), gender research (Fleis-
chmann et al., 2016), polar research (Summerson & Bishop,
2012), and has been implemented in software to conduct
web experiments (Naumann et al., 2007). It entered the
methodological discourse in Poland (Siuda, 2009)and
China (Wang et al., 2015), where it is called .
Pitfalls, Best Practices
In 2010,Iwrote,At the core of many of the more impor-
tant methodological problems with design and formatting
in Internet-based research are interactions between psycho-
logical processes in Internet use and the widely varying
technical context.(p. 32). The interaction between psy-
chology and technology can lead to advances, but foremost
in research, many pitfalls were discovered that subse-
quently led to recommendations for best practices, which
I both will introduce in this section.
Psychology has a long history of finding strategies to
measure behaviors, mental processes, attitudes, emotions,
self-reported inner states, or other constructs. Pronk and
colleagues (2020), following-up on various others (Garaizar
et al., 2014; Plant, 2016; Reimers & Stewart, 2015;Reips,
2007a; Schmidt, 2001; van Steenbergen & Bocanegra,
2016), investigated timing accuracy of web applications,
here with a comparative focus on touchscreen and key-
board devices. Their results confirm what theoretically
was expected from the technical structure and limitations
of the web (Reips, 1997,2000):
... very accurate stimulus timing and moderately
accurate RT measurements could be achieved on
both touchscreen and keyboard devices, though RTs
were consistently overestimated. In uncontrolled cir-
cumstances, such as researchers may encounter
online, stimulus presentation may be less accurate.
...Differences in RT overestimation between devices
might not substantially affect the reliability with
which group differences can be found, but they may
affect reliability for individual differences (p. 1371).
An example of how the combination of computerized mea-
surement and large sample sizes achievable on the web pro-
vided a shift in measurement accuracy that was long
needed but overlooked in light of traditions and measure-
ment burden is the switch from Likert-type rating scales
to visual analogue scales. The latter has become much
easier to administer via the web than on paper (where dis-
tances have to be measured with a ruler) and on computers
(where software often lacks that type of scale as an option).
Figure 4shows an example of both types of scales. Note the
visual analogue scale turns 100 this year (Hayes & Patter-
son, 1921).
As Reips and Funke (2008) point out, as a result of their
experiment, data collected with web-based visual analogue
scales provide better measurement than Likert-type scales
and offer more options for statistical analysis. Importantly,
visual analogue scales are not to be confused with slider
scales, for which we know its handle causes potential prob-
lems because its default position may cause anchoring
effects or erroneous ratings (Funke, 2016). Slider scales
may also cause more problems for the less educated (Funke
et al., 2011).
Dropout and Other Nonresponse
Dropout is more prevalent in web-based research and may
have detrimental effects (Reips, 2002b; Zhou & Fishbach,
2016). Zhou and Fishbach describe how unattended selec-
tive dropout can lead to surprising yet false research con-
clusions. However, avoiding dropout (e.g., via high hurdle
and warm-up techniques, and low tech principle) or control-
ling it (e.g., via seriousness check) are not the only strate-
gies to deal with the higher prevalence of dropout on the
web. Rather, dropout can be used as a dependent variable
(Reips, 2002a). Bosnjak (2001) describes seven types of
non-response behavior that can be observed in surveys,
these all are good measures in web-based research.
In an early and widely cited study, Frick and col-
leagues (2001) experimentally manipulated the influence
of announcing an incentive (or not), anonymity, and
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 207 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
placement of demographic questions (beginning vs. end) on
dropout. Incentive announcement and position of demo-
graphic questions showed large main effects on dropout,
it ranged from 5.7% in the incentive known and demo-
graphic questions at the beginning condition to 21.9%in
the incentive unknown and demographic questions at the
end condition. They also found a strong effect (> 100
min difference in reported TV consumption per week) of
the order of questions asking both for time devoted to
charity work and TV consumption had created a context
that evoked socially desirable responses. Birnbaum (2021)
also discusses dropout and reflects on early discussions
and findings by those who adopted the web for research
and developed the associated methodology.
DropR ( is ShinyApp and R software that
we created to meet the increased need to calculate dropout
rates because dropout is much more frequent in web-based
research than in laboratory research. In the analysis and
reporting of web experiments, the commonly high dropout
makes it necessary to provide an analysis and often also
visualize dropout by condition. DropR supports web
researchers in both providing manuscript-ready Figures
specifically designed for accessibility (see Vereenooghe,
2021) and all major survival and common dropout analyses
on the fly, including Kaplan-Meier, chi-square, odds ratio,
and rho family tests. Visual inspection allows for quick
detection of critical differences in dropout. DropR is Open
Source software available from Github (Reips & Bannert,
Dropout is particularly useful in detecting motivational
confounding in experiments (Reips, 2000,2002b). When-
ever conditions differ in motivational aspects, there is the
danger that this confound may explain any between-condi-
tion findings. On the web this would likely show in differen-
tial dropout rates. On the contrary, because there is
usually very minor dropout in offline experiments, the lab-
oratory setting is not as sensitive as the Internet setting in
regard to detecting motivational confounding.(Reips,
2009). In a secondary analysis of two studies, Shapiro-Luft
and Cappella (2013) confirm that motivational confounding
can be detected in web-based studies with video content. It
has been of consideration in conducting real-time multi-
player experiments on the web (Hawkins, 2015)andis
now routinely used as an argument to conduct studies on
the web rather than in the lab (e.g., Lithopoulos et al.,
2020; Sheinbaum et al., 2013).
Interaction of Technology and Psychology
A chief issue in web-based research is interactions between
technology and human factors, such as the type of device
and personality of user. Which device someone uses is by
no way randomly determined, it depends on ones prefer-
ences either directly, for example, because one may be
an early adopterand likes to buy and use the newest
technology or indirectly because ones personality and
demographics drive one to follow a certain education and
ogy is more common than in other professions. Buchanan
and Reips (2001) analyzed responses of 2,148 participants
to a web-based Five-Factor personality inventory and com-
pared demographic items for users of different computing
platforms. The responses of participants whose web brow-
sers were Javascript-enabled were also compared with
those whose web browsers were not. Macintosh users were
significantly more Open to Experiencethan were PC
users, and users using Javascript-enabled browsers had sig-
nificantly lower education levels.
For research, an immediate consequence is to expect lar-
ger direct and indirect self-selection and coverage biases. If a
web-based study relies on certain specific technologies, it
will not reach every person with the same probability. For
Figure 4. Web-based Likert type scale (A) and Visual Analogue scale (B).
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
208 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
theory-guided and experimental basic research, this is less of
an issue, but it may be for any research that tries to general-
ize from samples to populations. Technical and situational
variance in itself in the presence of an effect strengthens
the case for its generalizability (Reips, 1997,2000), as it
diminishes the probability that an effect is an artifact result-
ing from a specific technological setup in the laboratory.
Krantz (2021) further notes that technical variance also sug-
gests a potential for modification of the theoretical under-
standing of the phenomenon(p. 233) in web-based
research and shows how this can be done with the famous
illusion named after the founder of this journal, Ebbinghaus.
Once one understands the willingness of people to partici-
pate in web-based research (Bosnjak & Batinic, 2002),
the question arises where to find them. Participants for
web-based research in psychology can be found via various
types of sources beyond any that also work for laboratory-
based research: Mailing lists, forums/newsgroups, online
panels, social media (Facebook, Twitter, Instagram, Snap-
chat, Tuenti, ...), frequented websites (e.g., for news),
special target websites (e.g., by genealogists), Google
ads. Further, there are dedicated websites like Psycholog-
ical research on the net(
research/exponnet.html) by John Krantz or the web exper-
iment listat
Within just a few years, it has become common to recruit
workersasparticipantsformini jobson crowdsourcing
platforms like Clickworker, Prolific Academic, Cloudflare,
or Amazon Mechanical Turk (AMT). Anderson and col-
leagues (2019) show social and personality psychology as
an example of how dominant recruitment via AMT became
a recruitment platform just within a few years. However,
this proliferation of its use among researchers stands in
stark contrast with much criticism about data quality from
AMT workers (MTurkers) and the sites limitations. Reips
(in press) writes,
Workers respond to be paid, whereas other research
participants respond to help with research. A second
reason why MTurkers provide lower quality data may
be tied to the forums they have established where
jobs are discussed, including online studies. It may
well be that rumors and experiences shared in these
forums lead to decreased data quality. A third reason
is artificial MTurkers that have appeared on the site,
these are computer scripts or bots, not humans
(Dreyfuss, 2018), ironically replacing the hidden
humanin the machine with machines. Stewart and
colleagues (2015) calculated that with more and more
laboratories in the Behavioral sciences moving to
MTurk the total size of the actual participant pool
for all studies approaches just 7,300 people rather
than the hundreds of thousands in the past.
Ironically, a service that was developed to employ people
who appear to work as a machine because the machine
(computer) cannot do the task as well is now going down
the drain because human beings have programmed scripts
that pose as human workers.
Web-based research has enabled psychologists to explore
new topics and do their research with previously impossible
options to reach large heterogeneous samples and people
with rare characteristics, run studies easily in several sam-
ples and cultures simultaneously (see Subsampling Tech-
nique section), and go deeper into multiple fine-grained
measurements that may bring a revival of behavior instead
of self-report and other measures in psychology.
At the same time, technological factors became more
dominant in the ways psychological research is conducted,
various dangers include (1) the dependency on non-scienti-
fic agents like big player companies who provide only
selected data access to embedded scientists,(2)lackof
reproducibility because of the many hardware and dynamic
software factors involved and the quickly changing technol-
ogy, and (3) the distance between the researcher and partic-
ipants, who may not even be human but vague agents
provided by commercial recruitment services.
Web methodologists are, of course, trying to keep up with
the fast development and provide multiple solutions to the
challenges posed by the web as a route for research. For
those of us who actively have experienced research before
the web revolution, it will be an important task to describe
our insights from comparing pre-web with web-based
research and teach a new generation of researchers in psy-
chology. We will be the only generation to have witnessed
and experienced the change.
Akay, A., & Martinsson, P. (2009). Sundays are blue: Arent they?
The day-of-the-week effect on subjective well-being and socio-
economic status. (No. 4563; IZA Discussion Paper). http://hdl.
Anderson, C. A., Allen, J. J., Plante, C., Quigley-McBride, A., Lovett, A.,
& Rokkum, J. N. (2019). The MTurkification of social and person-
ality psychology. Personality and Social Psychology Bulletin, 45(6),
Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., & Evershed, J. K.
(2021). Realistic precision and accuracy of online experiment
platforms, web browsers, and devices. Behavior Research
Methods, 53, 14071425.
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 209 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013). Serious-
ness checks are useful to improve data validity in online
research. Behavior Research Methods, 45(2), 527535. https://
Back, M. D., Küfner, A. C. P., & Egloff, B. (2011). Automatic or the
people?Anger on September 11, 2001, and lessons learned for
the analysis of large digital data sets. Psychological Science,
22(6), 837838.
Barras, G. (2009). Gallery: Flickr users make accidental maps.
New Scientist.
Bayram, A. B. (2018). Serious subjects: A test of the seriousness
technique to increase participant motivation in political science
experiments. Research and Politics, 5(2).
Benford, F. (1938). The law of anomalous numbers. Proceedings of
the American Philosophical Society, 78(4), 551572.
Bernecker, K., & Job, V. (2011). Assessing implicit motives with an
online version of the picture story exercise. Motivation and
Emotion, 35(3), 251266.
Birnbaum, M. H. (2001). A web-based program of research on
decision making. In U.-D. Reips & M. Bosnjak (Eds.), Dimen-
sions of Internet science (pp. 2355). Pabst Science.
Birnbaum, M. H. (2004). Human research and data collection via
the Internet. Annual Review of Psychology, 55, 803832.
Birnbaum, M. H. (2021). Advanced training in web-based psychol-
ogy research: Trends and future directions. Zeitschrift für
Psychologie, 229(4), 260265.
Bluemke, M., & Zumbach, J. (2012). Assessing aggressiveness via
reaction times online. Cyberpsychology: Journal of Psychosocial
Research on Cyberspace, 6(1), Article 5.
Boer, D., & Fischer, R. (2011). Towards a holistic model of
functions of music listening across cultures: A culturally
decentred qualitative approach. Psychology of Music, 40(2),
Bosnjak, M. (2001). Participation in non-restricted web surveys: A
typology and explanatory model for item-nonresponse. In U.-D.
Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp.
193207). Pabst Science.
Bosnjak, M., & Batinic, B. (2002). Understanding the willingness to
participate in online-surveys. In B. Batinic, U.-D. Reips, & M.
Bosnjak (Eds.), Online social sciences (pp. 8192). Hogrefe &
Buchanan, T. (2000). Potential of the Internet for personality
research. In M. H. Birnbaum (Ed.), Psychological experiments on
the Internet (pp. 121140). Academic Press.
Buchanan, T. (2001). Online personality assessment. In U.-D.
Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp.
5774). Pabst Science.
Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Imple-
menting a five-factor personality inventory for use on the
Internet. European Journal of Psychological Assessment, 21(2),
Buchanan, T., & Reips, U.-D. (2001). Platform-dependent biases in
online research: Do Mac users really think different? In K. J.
Jonas, P. Breuer, B. Schauenburg, & M. Boos (Eds.), Perspectives
on Internet research: Concepts and methods (Proceedings of the
4th German Online Research Conference (GOR), May 1718,
Göttingen, Germany) University of Göttingen. https://www.uni-
Buchanan, T., & Smith, J. L. (1999). Using the Internet for
psychological research: Personality testing on the world wide
web. British Journal of Psychology, 90(1), 125144. https://doi.
Buchanan, T., & Whitty, M. T. (2014). The online dating romance
scam: Causes and consequences of victimhood. Psychology,
Crime & Law, 20(3), 261283.
Callegaro, M., Manfreda, K. L., & Vehovar, V. (2015). Web survey
methodology. Sage.
Cipora, K., Soltanlou, M., Reips, U.-D., & Nuerck, H.-C. (2019). The
SNARC and MARC effects measured online: Large-scale
assessment methods in flexible cognitive effects. Behavior
Research Methods, 51, 16761692.
Coomber, R. (1997). Using the Internet for survey research.
Sociological Research Online, 2 .
Daikeler, J., Bosnjak, M., & Lozar Manfreda, K. (2020). Web versus
other survey modes: An updated and extended meta-analysis
comparingresponserates.Journal of Survey Statistics and
Methodology, 8(3), 513539.
Deshpande, P. S., Tauber, S., Chang, S. M., Gago, S., & Jameson,
K. A. (2016). Digitizing a large corpus of handwritten documents
using crowdsourcing and Cultural Consensus Theory. Interna-
tional Journal of Internet Science, 11(1), 832.
Dillman, D. A., Reips, U.-D., & Matzat, U. (2010). Advice in
surveying the general public over the Internet. International
Journal of Internet Science, 5(1), 14.
Faul, F., Erdfelder, E., Buchner, A., & Lang, A. G. (2009). Statistical
regression analyses. Behavior Research Methods, 41, 11491160.
Fleischmann, A., Sieverding, M., Hespenheide, U., Weiß, M., &
Koch, S. C. (2016). See feminine think incompetent? The
effects of a feminine outfit on the evaluation of womens
computer competence. Computers & Education, 95,6374.
Fokkema, T., Kooiman, T. J., Krijnen, W. P., van der Schans, C. P.,
& de Groot, M. (2017). Reliability and validity of ten consumer
activity trackers depend on walking speed. Medicine and
Science in Sports and Exercise, 49(4), 793800. https://doi.
Frederick, S. (2005). Cognitive reflection and decision making.
Journal of Economic Perspectives, 19(4), 2542.
Frick, A., Bächtiger, M. T., & Reips, U.-D. (2001). Financial
incentives, personal information and drop-out in online studies.
In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet
science (pp. 209219). Pabst Science.
Funke, F. (2016). A web experiment showing negative effects of
slider scales compared to visual analogue scales and radio
button scales. Social Science Computer Review, 34(2), 244254.
Funke, F., Reips, U.-D., & Thomas, R. K. (2011). Sliders for the
smart: Type of rating scale on the web interacts with educa-
tional level. Social Science Computer Review, 29, 221231.
Garaizar, P., & Reips, U.-D. (2019). Best practices: Two web
browser-based methods for stimulus presentation in behav-
ioral experiments with high resolution timing requirements.
Behavior Research Methods, 51, 14411453.
Garaizar, P., Vadillo, M. A., & López-de-Ipiña, D. (2014). Presen-
tation accuracy of the web revisited: Animation methods in the
HTML5 era. PLoS One, 9, Article e109812.
10.1371/journal.pone. 0109812
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
210 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F.,
Chatterjee, G., & Wilmer, J. B. (2012). Is the web as good as
the lab? Comparable performance from web and lab in cogni-
tive/perceptual experiments. Psychonomic Bulletin & Review,
19(5), 847857.
Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski,
M. S., & Brilliant, L. (2009). Detecting influenza epidemics using
search engine query data. Nature, 457, 10121015.
Hagger-Johnson, G. E., & Whiteman, M. C. (2007). Conscientious-
ness facets and health behaviors: A latent variable modeling
approach. Personality and Individual Differences, 43(5), 1235
Hawkins, R. X. D. (2015). Conducting real-time multiplayer exper-
iments on the web. Behavior Research Methods, 47(4), 966
Hayes, M. H. S., & Patterson, D. G. (1921). Experimental develop-
ment of the graphic rating method. Psychological Bulletin, 18,
Heidingsfelder, M. (1997). Der Internet-Rogator [The Internet-
Rogator]. Software demonstration at first German Online
Research (GOR) Conference, Cologne.
Hilbig, B. E., & Thielmann, I. (2021). On the (mis)use of deception
in web-based research: Challenges and recommendations.
Zeitschrift für Psychologie, 229(4), 225229.
Hiskey, S., & Troop, N. A. (2002). Online longitudinal survey
research: Viability and participation. Social Science Computer
Review, 20(3), 250259.
Holbrook, A., & Lavrakas, P. J. (2019). Vignette experiments in
surveys. In P. Lavrakas, M. Traugott, C. Kennedy, A. Holbrook, E.
de Leeuw, & B. West (Eds.), Experimental methods in survey
research (pp. 369370). Wiley.
Honing, H. (2021). Lured into listening: Engaging games as an
alternative to reward-based crowdsourcing in music research.
Zeitschrift für Psychologie, 229(4), 266268.
Janetzko, D. (2008). Objectivity, reliability, and validity of search
engine count estimates. International Journal of Internet
Science, 3,733.
Judge, G., & Schechter, L. (2009). Detecting problems in survey
data using Benfords Law. Journal of Human Resources, 44(1),
Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length
on participation and indicators of quality of answers in a web
survey. Public Opinion Quarterly, 73(2), 349360. https://doi.
Klein, B., & Reips, U.-D. (2017). Innovative social location-aware
services for mobile phones. In A. Quan-Haase & L. Sloan (Eds.),
Handbook of social media research methods (pp. 421438).
Krantz, J. H. (2021). Ebbinghaus illusion: Relative size as a
possible invariant under technically varied conditions? Zeits-
chrift für Psychologie, 229(4), 230235.
Krantz, J., & Reips, U.-D. (2017). The state of web-based research:
A survey and call for inclusion in curricula. Behavior Research
Methods, 49(5), 16211629.
Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results
of laboratory and world-wide web samples on the determinants
of female attractiveness. Behavior Research Methods, Instru-
ments, & Computers, 29, 264269.
Kristo, G., Janssen, S. M. J., & Murre, J. M. J. (2009). Retention of
autobiographical memories: An Internet-based diary study.
Memory, 17(8), 816829.
Kroehne, U., & Goldhammer, F. (2018). How to conceptualize,
represent, and analyze log data from technology-based
assessments? A generic framework and an application to
questionnaire items. Behaviormetrika, 45(2), 527563. https://
Kuhlmann, T., Garaizar, P., & Reips, U.-D. (2021). Smartphone
sensor accuracy varies from device to device: The case of
spatial orientation. Behavior Research Methods, 53,2233.
Kuhlmann, T., Reips, U.-D., & Stieger, S. (2016, November 11).
Smartphone tilt as a measure of well-being? Results from a
longitudinal smartphone app study. In U.-D. Reips (Ed.),
20 years of Internet-based research at SCiP: Surviving concepts,
new methodologies [Symposium]. 46th Society for Computers in
Psychology (SCiP) Conference, Boston, MA, USA.
Kus, L., Ward, C., & Liu, J. (2014). Interethnic factors as predictors
of the subjective well-being of minority individuals in a context
of recent societal changes. Political Psychology, 35(5), 703719.
Lindquist, M., Lange, E., & Kang, J. (2016). From 3D landscape
visualization to environmental simulation: The contribution of
sound to the perception of virtual environments. Landscape
and Urban Planning, 148, 216231.
Lithopoulos, A., Grant, S. J., Williams, D. M., & Rhodes, R. E.
(2020). Experimental comparison of physical activity self-
efficacy measurement: Do vignettes reduce motivational con-
founding? Psychology of Sport and Exercise, 47, Article 101642.
Loomis, D. K., & Paterson, S. (2018). A comparison of data
collection methods: Mail versus online surveys. Journal of
Leisure Research, 49(2), 133149.
Mangan, M. A., & Reips, U.-D. (2007). Sleep, sex, and the web:
Surveying the difficult-to-reach clinical population suffering
from sexsomnia. Behavior Research Methods, 39, 233236.
Michel, J.-B., Shen, Y. K., Aiden, A. P., Veres, A., Gray, M. K., The
Google Books Team, Pickett, J. P., Hoiberg, D., Clancy, D.,
Norvig, P., Orwant, J., Pinker, S., Nowak, M. A., & Lieberman
Aiden, E. (2011). Quantitative analysis of culture using millions
of digitized books. Science, 331(6014), 176182.
Miller, G. (2012). The smartphone psychology manifesto. Perspec-
tives on Psychological Science, 7(3), 221237.
Musch, J., & Klauer, K. C. (2002). Psychological experimenting on
the world wide web: Investigating content effects in syllogistic
reasoning. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online
social sciences (pp. 181212). Hogrefe & Huber.
Musch, J., & Reips, U.-D. (2000). A brief history of web experi-
menting. In M. H. Birnbaum (Ed.), Psychological experiments on
the Internet (pp. 6188). Academic Press.
Mustanski, B. S. (2001). Getting wired: Exploiting the Internet for
the collection of valid sexuality data. Journal of Sex Research,
38(4), 292301.
Naumann, A., Brunstein, A., & Krems, J. F. (2007). DEWEX: A
system for designing and conducting web-based experiments.
Behavior Research Methods, 39(2), 248258.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instruc-
tional manipulation checks: Detecting satisficing to increase
statistical power. Journal of Experimental Social Psychology,
45(4), 867872.
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 211 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
Plant, R. R. (2016). A reminder on millisecond timing accuracy and
potential replication failure in computer-based psychology
experiments: An open letter. Behavior Research Methods, 48,
Pronk, T., Wiers, R. W., Molenkamp, B., & Murre, J. (2020). Mental
chronometry in the pocket? Timing accuracy of web applica-
tions on touchscreen and keyboard devices. Behavior Research
Methods, 52(3), 13711382.
Ratcliffe, J., Soave, F., Bryan-Kinns, N., Tokarchuk, L., &
Farkhatdinov, I. (2021, May 813). Extended reality (XR) remote
research: A survey of drawbacks and opportunities. Paper
presented at the CHI Conference on Human Factors in
Computing Systems (CHI 21), Yokohama, Japan. https://doi.
Ray, J. V., Kimonis, E. R., & Donoghue, C. (2010). Legal, ethical,
and methodological considerations in the Internet-based study
of child pornography offenders. Behavioral Sciences and the
Law, 28(1), 84105.
Reimers, S., & Stewart, N. (2015). Presentation and response
timing accuracy in Adobe Flash and HTML5/JavaScript web
experiments. Behavior Research Methods, 47, 309327.
Reips, U.-D. (in press). Internet-based studies. In M. D. Gellman & J.
Rick Turner (Eds.), Encyclopedia of behavioral medicine (2nd ed.).
Reips, U.-D. (1996, October). Experimenting in the world wide web.
Paper presented at the Society for Computers in Psychology
conference, Chicago, IL.
Reips, U.-D. (1997). Das psychologische Experimentieren im
Internet [Psychological experimenting on the Internet]. In B.
Batinic (Ed.), Internet für Psychologen (pp. 245265). Hogrefe.
Reips, U.-D. (2000). The web experiment method: Advantages,
disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psycho-
logical experiments on the Internet (pp. 89117). Academic
Reips, U.-D. (2001). The web experimental psychology lab: Five
years of data collection on the Internet. Behavior Research
Methods, Instruments, & Computers, 33, 201211. https://doi.
Reips, U.-D. (2002a). Internet-based psychological experimenting:
Five dos and five donts. Social Science Computer Review, 20(3),
Reips, U.-D. (2002b). Standards for Internet-based experimenting.
Experimental Psychology, 49(4), 243256.
Reips, U.-D. (2002c). Theory and techniques of conducting web
experiments. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.),
Online social sciences (pp. 229250). Hogrefe & Huber.
Reips, U.-D. (2006). Web-based methods. In M. Eid & E. Diener
(Eds.), Handbook of multimethod measurement in psychology
(pp. 7385). American Psychological Association. https://doi.
Reips, U.-D. (2007a, November). Reaction times in Internet-based
research. Invited symposium talk at the 37th Meeting of the
Society for Computers in Psychology (SCiP) Conference, St.
Louis, MO.
Reips, U.-D. (2007b). The methodology of Internet-based experi-
ments. In A. Joinson, K. McKenna, T. Postmes, & U.-D. Reips
(Eds.), The Oxford handbook of Internet psychology (pp. 373390).
Oxford University Press.
Reips, U.-D. (2008). How Internet-mediated research changes
science. In A. Barak (Ed.), Psychological aspects of cyberspace:
Theory, research, applications (pp. 268294). Cambridge Univer-
sity Press.
Reips, U.-D. (2009). Internet experiments: Methods, guidelines,
metadata. Human Vision and Electronic Imaging XIV, Proceed-
ings of SPIE, 7240, Article 724008.
Reips, U.-D. (2010). Design and formatting in Internet-based
research. In S. D. Gosling & J. A. Johnson (Eds.), Advanced
methods for conducting online behavioral research (pp. 2943).
American Psychological Association.
Reips, U.-D., & Bannert, M. (2015). dropR: Analyze dropout of an
experiment or survey [Computer software] (R package version
0.9). Research Methods, Assessment, and iScience, Depart-
ment of Psychology, University of Konstanz. https://cran.
Reips, U.-D., & Funke, F. (2008). Interval-level measurement with
visual analogue scales in Internet-based research: VAS Gener-
ator. Behavior Research Methods, 40(3), 699704.
Reips, U.-D., & Garaizar, P. (2011). Mining Twitter: Microblogging
as a source for psychological wisdom of the crowds. Behavior
Research Methods, 43, 635642.
Reips, U.-D., & Krantz, J. H. (2010). Conducting true experiments on
the web. In S. D. Gosling & J. A. Johnson (Eds.), Advanced
methods for conducting online behavioral research (pp. 193216).
American Psychological Association.
Reips, U.-D., & Lengler, R. (2005). The web experiment list: A web
service for the recruitment of participants and archiving of
Internet-based experiments. Behavior Research Methods, 37,
Reips, U.-D., Morger, V., & Meier, B. (2001). Fünfe gerade sein
lassen: Listenkontexteffekte beim Kategorisieren [Letting five
be equal: List context effects in categorization]. https://www.
Reips, U.-D., & Neuhaus, C. (2002). WEXTOR: A web-based tool for
generating and visualizing experimental designs and proce-
dures. Behavior Research Methods, Instruments, and Comput-
ers, 34(2), 234240.
Revilla, M. (2016). Impact of raising awareness of respondents on
the measurement quality in a web survey. Quality and Quantity,
50(4), 14691486.
Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T. M., Ling, J.,
& Parrott, A. (2001). Differential effects of ecstasy and cannabis
on self-reports of memory ability: A web-based study. Human
Psychopharmacology: Clinical and Experimental, 16(8), 619
Rodgers, J., Buchanan, T., Scholey, A. B., Heffernan, T. M., Ling, J.,
& Parrott, A. C. (2003). Patterns of drug use and the influence
of gender on self-reports of memory ability in ecstasy users: A
web-based study. Journal of Psychopharmacology, 17(4), 389
Roth, M. (2006). Validating the use of Internet survey techniques in
visual landscape assessment: An empirical study from Ger-
many. Landscape and Urban Planning, 78(3), 179192. https://
Schmidt, W. C. (2000). The server-side of psychology web exper-
iments. In M. H. Birnbaum (Ed.), Psychological experiments on
the Internet (pp. 285310). Academic Press.
Schmidt, W. C. (2001). Presentation accuracy of web animation
methods. Behavior Research Methods, Instruments, & Comput-
ers, 33, 187200.
Shapiro-Luft, D., & Cappella, J. N. (2013). Video content in web
surveys: Effects on selection bias and validity. Public Opinion
Quarterly, 77(4), 936961.
Zeitschrift für Psychologie (2021), 229(4), 198213 Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under
the license CC BY 4.0 (
212 U.-D. Reips, A Review of Web-Based Research - Sunday, January 30, 2022 5:00:46 AM - IP Address:
Sheinbaum, T., Berry, K., & Barrantes-Vidal, N. (2013). Proceso de
adaptación al español y propiedades psicométricas de la
Psychosis Attachment Measure [Spanish version of the Psy-
chosis Attachment Measure: Adaptation process and psycho-
metric properties]. Salud Mental, 36(5), 403409. https://doi.
Shevchenko, Y., Kuhlmann, T., & Reips, U.-D. (2021). Samply: A
user-friendly smartphone app and web-based means of
scheduling and sending mobile notifications for experience-
sampling research. Behavior Research Methods, 53, 1710
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological
Momentary Assessment. Annual Review of Clinical Psychology, 4,
Siuda, P. (2009). Eksperyment w Internecie nowa metoda badan´
w naukach społecznych [Experiment on the Internet: New
social sciences research method analysis]. Studia Medioz-
nawcze, 3(38).
Summerson, R., & Bishop, I. D. (2012). The impact of human
activities on wilderness and aesthetic values in Antarctica.
Polar Research, 31, Article 10858.
Stephens-Davidowitz, S. (2017). Everybody lies: Big data, new
data, and what the Internet can tell us about who we really are.
Dey Street Books.
Stieger, S., & Reips, U.-D. (2010). What are participants doing
while filling in an online questionnaire: A paradata collection
tool and an empirical study. Computers in Human Behavior,
26(6), 14881495.
Stieger, S., & Reips, U.-D. (2016). A limitation of the Cognitive
Reflection Test: Familiarity. PeerJ, 4, Article e2395. https://doi.
Stieger, S., & Reips, U.-D. (2019). Well-being, smartphone sensors,
and data from open-access databases: A mobile experience
sampling study. Field Methods, 31(3), 277291.
Tan, Y. T., Peretz, I., McPherson, G. E., & Wilson, S. J. (2021).
Establishing the reliability and validity of web-based singing
research. Music Perception, 38(4), 386405.
Trapnell, P. D., & Campbell, J. D. (1999). Private self-conscious-
ness and the five-factor model of personality: Distinguishing
rumination from reflection. Journal of Personality and Social
Psychology, 76(2), 284304.
Van Steenbergen, H., & Bocanegra, B. R. (2016). Promises and
pitfalls of web-based experimentation in the advance of
replicable psychological science: A reply to Plant (2015).
Behavior Research Methods, 48, 17131717.
Verbree, A.-R., Toepoel, V., & Perada, D. (2020). The effect of
seriousness and device use on data quality. Social Science
Computer Review, 38(6), 720738.
Vereenooghe, L. (2021). Participation of people with disabilities in
web-based research. Zeitschrift für Psychologie, 229(4), 257
Wang, Y., Yu, Z., Luo, Y., Chen, J., & Cai, H. (2015). Conducting
psychological research via the Internet: In the west and China.
Advances in Psychological Science, 23(3), 510519. https://doi.
Weichselgartner, E. (2021, March 23). Software for psychology
experiments: Which software can be used to run experiments in
the behavioral sciences (online and offline)? https://docs.
Whitehead, L. C. (2007). Methodological and ethical issues in
Internet-mediated research in the field of health: An integrated
review of the literature. Social Science and Medicine, 65(4),
Wolfe, C. R. (2017). Twenty years of Internet-based research at
SCiP: A discussion of surviving concepts and new methodolo-
gies. Behavior Research Methods, 49, 16151620. https://doi.
Younes, N., & Reips, U.-D. (2019). Guidelines for improving the
reliability of Google Ngram studies: Evidence from religious
terms. PLoS One, 14(3), Article e0213554.
Zhou, H., & Fishbach, A. (2016). The pitfall of experimenting on the
web: How unattended selective attrition leads to surprising (yet
false) research conclusions. Journal of Personality and Social
Psychology, 111(4), 493504.
Received August 5, 2021
Accepted September 2, 2021
Published online December 17, 2021
I thank Tom Buchanan for his valuable feedback on this review
article. The action editor for this article was Michael Bošnjak.
For further information about the authors iScience research
group with their focus on experimental psychology and Internet
science, please visit
Conflict of Interest
The author declares that there is no conflict of interest.
Open access publication enabled by Hogrefe Publishing.
Ulf-Dietrich Reips
Department of Psychology
University of Konstanz
Universitätsstraße 10
Fach 31
78464 Konstanz
Ó2021 The Author(s). Distributed as a Hogrefe OpenMind article under Zeitschrift für Psychologie (2021), 229(4), 198213
the license CC BY 4.0 (
U.-D. Reips, A Review of Web-Based Research 213 - Sunday, January 30, 2022 5:00:46 AM - IP Address:
... However, given the sharp correspondence of the increased samples with online samples (Figure 3) and the many reasons to conduct research via the Internet (e.g. Reips, 2000Reips, , 2021, it seems likely that online experiments drive increased sample sizes. ...
... From only a very small number of Internet-based studies until the early 2010s, during the past decade online research has soared to the extent that in the recent years more than half of the subjects in experimental psychology has participated online. The concomitantly increased total sample sizes (despite only a slight increase in offline samples) likely bode well for replicability and robustness (Nosek et al., 2022;Reips, 2000Reips, , 2021, although the precise impact would require more detailed examination (see Sassenberg & Ditrich, 2019). As widely discussed in reviews, online research has its pitfalls as well (e.g., Chandler et al., 2014;Gagné & Franzen, 2021;Reips, 2021;Zhou & Fishbach, Zhou, H., & Fishbach, A. (2016). ...
... The concomitantly increased total sample sizes (despite only a slight increase in offline samples) likely bode well for replicability and robustness (Nosek et al., 2022;Reips, 2000Reips, , 2021, although the precise impact would require more detailed examination (see Sassenberg & Ditrich, 2019). As widely discussed in reviews, online research has its pitfalls as well (e.g., Chandler et al., 2014;Gagné & Franzen, 2021;Reips, 2021;Zhou & Fishbach, Zhou, H., & Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. ...
... A experimentação via internet não é uma novidade para a Psicologia de maneira geral, sendo executada desde a década de 1990 (Reips, 2021), e é uma possibilidade bastante utilizada em pesquisas analítico-comportamentais sobre desvalorização pelo atraso e ressurgência, por exemplo (Johnson et al., 2015;Ritchey et al., 2021 ...
... Aggression and negative reinforcement in humans: An initial experiment with remote participation Experimentos em psicologia têm sido executados remotamente como uma solução para situações nas quais o contato físico entre pesquisadores e participantes não é possível (e.g., Krantz, 2021;Ratcliffe et al., 2021). Nesses casos, ajustes metodológicos são necessários para garantir uma coleta de dados segura para ambas as partes (Reips, 2021) (Lane et al., 2020). É parcialmente baseada em investigações experimentais com humanos (Buss, 1961;Taylor, 1967) e não-humanos (Hutchinson, 1973;Ulrich, 1966) (Renfrew, 1997;Renfrew & Hutchinson, 1983; (Shull & Lawrence, 1998). ...
Full-text available
O Volume 14 da coleção Comportamento em Foco reúne capítulos derivados de trabalhos apresentados no XXX Encontro da Associação Brasileira de Ciências do Comportamento (ABPMC), realizado entre os dias 1 e 4 de setembro de 2021, em versão online. Neste volume, discussões teóricas e práticas sobre comportamento humano em contextos de saúde e educação, assim como considerações sobre as contribuições da Análise do Comportamento para pensar a sociedade, são destaques entre os escritos. Dentre as temáticas que integram os capítulos, estão o uso crítico de tecnologias comportamentais em instituições totais, no ensino superior e na educação ambiental, a atuação de analistas do comportamento com diferentes populações, como autistas, adolescentes em conflito com a lei, idosos/as e LGBTQIAP+, e as possibilidades de interpretação analítico-comportamental de fenômenos como coerção, agressão e arte.
... Discussion: The dura tion of conditions, the nature of employed stimulus and the research remote modality are examined, and follow-up experiments are quisadores e participantes não é possível (e.g., Krantz, 2021;Ratcli e et al., 2021). Nesses casos, ajustes metodológicos são necessários para ga rantir uma coleta de dados segura para ambas as partes (Reips, 2021) (Lane et al., 2020). É parcialmente baseada em investigações experimentais com humanos (Buss, 1961;Taylor, 1967) e não-humanos (Hutchinson, 1973;Ulrich, 1966) (Renfrew, 1997;Renfrew & Hutchinson, 1983;Skinner, 1969 Cherek et al. (1990) Nos casos acima citados, os pontos pro duzidos eram trocáveis por dinheiro, e bônus monetários também eram programados -uma estratégia efetiva para garantir prolongada par ticipação (Pilgrim, 1998). ...
Full-text available
Agressão humana tem sido investigada empregando o Paradigma de Agressão por Subtração de Pontos. Pesquisas indicaram que o comportamento agressivo mensurado pela tarefa é mantido por contingência de fuga, e não por suas consequências sociais. Objetivo: replicar e ampliar achados que especificam as variáveis de controle do responder na tarefa, em modalidade remota. Método: Onze participantes foram expostos às condições A e B, e 10 às condições C e B. Na condição A, a consequência para o responder era a retirada de cinco pontos de outro participante; a condição B reproduzia A, e um período de 30 s sem exposição à subtração de pontos também vigorava; em C, apenas o período mencio nado era consequência. Resultados: Não foi detectada variação sistemática do responder segundo as diferentes condições. Discussão: A duração das condições, a natureza dos es tímulos utilizados e a modalidade remota da pesquisa são examinados, e experimentos futuros são sugeridos.
... In the first article, Reips (2021) reviews different types of web-based research, including research that could not be done without the web. He highlights how some specific research areas, for example, on the rare disorder sexsomnia, have profited enormously from the advantages of the web, in this case the combination of reach and anonymity. ...
Full-text available
The two Bayesian adaptive psychometric methods named QUEST (Watson & Pelli, 1983) and QUEST+ (Watson, 2017) are widely used to estimate psychometric parameters, especially the threshold, in laboratory-based psychophysical experiments. Considering the increase of online psychophysical experiments in recent years, there is a growing need to have the QUEST and QUEST+ methods available online as well. We developed JavaScript libraries for both, with this article introducing one of them: jsQuestPlus. We offer integrations with online experimental tools such as jsPsych (de Leeuw, 2015), PsychoPy/JS (Peirce et al., 2019), and lab.js (Henninger et al., 2021). We measured the computation time required by jsQuestPlus under four conditions. Our simulations on 37 browser–computer combinations showed that the mean initialization time was 461.08 ms, 95% CI [328.29, 593.87], the mean computation time required to determine the stimulus parameters for the next trial was less than 1 ms, and the mean update time was 79.39 ms, 95% CI [46.22, 112.55] even in extremely demanding conditions. Additionally, psychometric parameters were estimated as accurately as the original QUEST+ method did. We conclude that jsQuestPlus is fast and accurate enough to conduct online psychophysical experiments despite the complexity of the matrix calculations. The latest version of jsQuestPlus can be downloaded freely from under the MIT license.
Full-text available
A tecnologia pode ser uma grande aliada da Análise do Comportamento. Discutimosos prós e contras dessa associação em três contextos: aplicação, pesquisa, e produção e divulgação cientí fica. Na aplicação, há uma abundância de suportes tecnológicos, como realidade virtual, robôs, chatbots, sensores, visão computacional e técnicas de inteligência artificial, que possibilitam aumento na eficiência de intervenções. Na pesquisa, recursos tecnológicos têm viabilizado a preparação de experimentos cada vez mais automatizados, e sumarizamos algumas caracterí sticas observadas em pesquisas sobre seleção cultural realizadas remotamente. Na produção e divulgação cientí fica, a modernização dos processos de submissão e editoração de periódicos científicos é um caminho inevitável, e novos cuidados são requeridos. Encorajamos estudantes, profissionais e pesquisadores em Aná lise do Comportamento a empregar a tecnologia como aliada de sua prática, assumindo a dianteira no desenvolvimento dessa ciência.
Full-text available
Web Survey Methodology guides the reader through the past fifteen years of research in web survey methodology. It both provides practical guidance on the latest techniques for collecting valid and reliable data and offers a comprehensive overview of research issues. Core topics from preparation to questionnaire design, recruitment testing to analysis and survey software are all covered in a systematic and insightful way. The reader will be exposed to key concepts and key findings in the literature, covering measurement, non-response, adjustments, paradata, and cost issues. The book also discusses the hottest research topics in survey research today, such as internet panels, virtual interviewing, mobile surveys and the integration with passive measurements, e-social sciences, mixed modes and business intelligence. The book is intended for students, practitioners, and researchers in fields such as survey and market research, psychological research, official statistics and customer satisfaction research. The book is now available as open access since at this Sage page in PDF and Epub version
Full-text available
The deception of research participants remains a controversial issue in the behavioral sciences. Current ethics codes consistently limit the use of deception to cases in which non-deceptive alternatives are unfeasible and, crucially, require that participants subjected to deception be debriefed correspondingly along with an option to withdraw their data after learning about the deception. These conditions pose a particular challenge in the context of web-based research because participants can typically discontinue a study unilaterally (i.e., dropout by simply closing the browser window) in which case full debriefing and an option to withdraw one’s data are no longer available. As a consequence, the study would no longer be compatible with ethical standards. Based on recent meta-analytical data, we provide an existence proof of this problem, showing that deception is used in web-based research with little to no indication of safeguards ensuring full debriefing and subsequent data withdrawal options. We close by revisiting recommendations for the (non-)use of deception in web-based research and offer solutions to implement such safeguards in case deception is truly unavoidable.
Full-text available
This article describes Advanced Training Institutes (ATIs) and workshops on psychological experimentation conducted via the Internet. These programs, conducted since 2002, presented instruction that evolved over the decades to reflect changes in Web-based methods and techniques. The need for instruction in the methods and methodology of Web-based research has increased over the years, as more researchers adopted these methods without necessarily learning the methodological lessons developed from theory and experience. Links to instructional materials created for the workshops are provided. From events and trends that played out over time, including the story of methods that were once state-of-the-art and are no longer functional today, lessons hard-won from the past can be used to anticipate and plan future directions in Web-based experimentation.
Full-text available
For 25 years, the web has been used for psychological research ( Krantz et al., 1997 ; Reips, 1997 ). While many areas of psychology have benefitted from the increased access to participants and other benefits of web-based research, one area of psychology has rarely taken advantage of the online format, that is, sensation and perception. Largely, sensation and perception research has not used the web because of the need for carefully calibrated equipment to successfully run their experiments. However, there may be classes of phenomena in our sensory processes that might be studied online where the equipment and stimuli vary. Suppose the critical feature of the stimulus is an abstraction of the physical stimulus that does not vary with different displays. In that case, these features can be successfully studied online, meaning that results from online studies will match those from controlled laboratories. This study will examine the Ebbinghaus illusion to illustrate the successful use of the web for perceptual research. The implications and some discussion of types of perceptual studies conducted on the web will be discussed.
Full-text available
This brief statement revisits some earlier observations on what makes web-based experiments, and especially citizen science using engaging games, an attractive alternative to laboratory-based setups. It suggests web-based experimenting to be a full-grown alternative to traditional laboratory-based experiments, especially in the field of music cognition, where sampling bias is a common problem and large amounts of empirical data are needed to characterize individual variability.
Hundreds of millions of people across the world use the Internet every day. Its functions vary, from shopping and banking to chatting and dating. From a psychological perspective, the Internet has become a major vehicle for interpersonal communication that can significantly affect people's decisions, behaviors, attitudes and emotions. Moreover, its existence has created a virtual social environment in which people can meet, negotiate, collaborate and exchange goods and information. Cyberspace is not just a technical device but a phenomenon which has reduced the world to a proverbial global village, fostering collaborations and international cooperations; thus reducing the barriers of geographical distance and indigenous cultures. Azy Barak and a team of prominent social scientists review a decade of scientific investigations into the social, behavioral and psychological aspects of cyberspace, collating state-of-the-art knowledge in each area. Together they develop emerging conceptualizations and envisage directions and applications for future research.
IN THIS STUDY, THE ROBUSTNESS OF AN ONLINE tool for objectively assessing singing ability was examined by: (1) determining the internal consistency and test-retest reliability of the tool; (2) comparing the task performance of web-based participants (n  285) with a group (n  52) completing the tool in a controlled laboratory setting, and then determining the convergent validity between settings, and (3) comparing participants' task performance with previous research using similar singing tasks and populations. Results indicated that the online singing tool exhibited high internal consistency (Cronbach's alpha  .92), and moderate-tohigh test-retest reliabilities (.65-.80) across an average 4.5-year-span. Task performance for web- A nd laboratory-based participants (n  82) matched on age, sex, and music training were not significantly different. Moderate-to-large correlations (|r| Â.31-.59) were found between self-rated singing ability and the various singing tasks, supporting convergent validity. Finally, task performance of the web-based sample was not significantly different to previously reported findings.Overall the findings support the robustness of the online tool for objectively measuring singing pitch accuracy beyond a controlled laboratory environment and its potential application in largescale investigations of singing and music ability.