No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
How long do respondents think online surveys should be? New evidence from two online panels in Germany
Abstract
In recent years, the number of surveys, especially online surveys, has increased dramatically. Due to the absence of interviewers in this survey mode (who can motivate the respondents to continue answering), some researchers and practitioners argue that online surveys should not be longer than 20 min. However, so far, there has been little research investigating how long respondents think that online surveys should or could be. In this study, we therefore asked respondents of two online panels in Germany (one probability-based panel and one nonprobability panel) about their opinions on the ideal and maximum lengths of surveys. We also investigated whether socio-demographic, personality-related, and survey-related variables were associated with the ideal and maximum lengths reported by respondents. Finally, we compared the stated and observed survey lengths to evaluate the extent to which respondents are able to accurately estimate survey length. Our results suggest that the ideal length of an online survey is between 10 and 15 min and the maximum length is between 20 and 28 min, depending on the measure of central tendency (mean or median) used and the panel. Moreover, we found significant effects of socio-demographics (gender, age, education, and number of persons in household), of personality traits, and survey-related questions (whether the respondents liked the survey, found it easy, and answered from a PC) on at least one of the dependent variables (ideal or maximum lengths). Finally, we found only small differences (less than two min) between stated and observed lengths.
... To quantitatively evaluate the psychometric properties of the assessment, we conducted a large-scale pilot test of version 2.0. Given the length and cognitive complexity of the assessment, we were concerned about the potential effect of survey fatigue on student attentiveness [22,40]. To keep testing time under the recommended 30-min time limit for most respondents [22,40], we reduced the length of the assessment by dividing version 2.0 into seven core questions and seven supplemental questions. ...
... Given the length and cognitive complexity of the assessment, we were concerned about the potential effect of survey fatigue on student attentiveness [22,40]. To keep testing time under the recommended 30-min time limit for most respondents [22,40], we reduced the length of the assessment by dividing version 2.0 into seven core questions and seven supplemental questions. When taking the assessment, all students received the seven core questions along with a random sample of five supplemental questions. ...
... To quantitatively evaluate the psychometric properties of the assessment, we conducted a large-scale pilot test of version 2.0. Given the length and cognitive complexity of the assessment, we were concerned about the potential effect of survey fatigue on student attentiveness [22,40]. To keep testing time under the recommended 30-min time limit for most respondents [22,40], we reduced the length of the assessment by dividing version 2.0 into seven core questions and seven supplemental questions. ...
... Given the length and cognitive complexity of the assessment, we were concerned about the potential effect of survey fatigue on student attentiveness [22,40]. To keep testing time under the recommended 30-min time limit for most respondents [22,40], we reduced the length of the assessment by dividing version 2.0 into seven core questions and seven supplemental questions. When taking the assessment, all students received the seven core questions along with a random sample of five supplemental questions. ...
Background and objectives
Universities throughout the USA increasingly offer undergraduate courses in evolutionary medicine (EvMed), which creates a need for pedagogical resources. Several resources offer course content (e.g. textbooks) and a previous study identified EvMed core principles to help instructors set learning goals. However, assessment tools are not yet available. In this study, we address this need by developing an assessment that measures students’ ability to apply EvMed core principles to various health-related scenarios.
Methodology
The EvMed Assessment (EMA) consists of questions containing a short description of a health-related scenario followed by several likely/unlikely items. We evaluated the assessment’s validity and reliability using a variety of qualitative (expert reviews and student interviews) and quantitative (Cronbach’s α and classical test theory) methods. We iteratively revised the assessment through several rounds of validation. We then administered the assessment to undergraduates in EvMed and Evolution courses at multiple institutions.
Results
We used results from the pilot to create the EMA final draft. After conducting quantitative validation, we deleted items that failed to meet performance criteria and revised items that exhibited borderline performance. The final version of the EMA consists of six core questions containing 25 items, and five supplemental questions containing 20 items.
Conclusions and implications
The EMA is a pedagogical tool supported by a wide range of validation evidence. Instructors can use it as a pre/post measure of student learning in an EvMed course to inform curriculum revision, or as a test bank to draw upon when developing in-class assessments, quizzes or exams.
... Die KI-gestützte Berechnungsmethode der Befragungsplattform prognostizierte eine etwas höhere Ausfüllzeit von zehn Minuten. Die ermittelten Werte entsprechen den Anforderungen an die ideale Medianlänge von Websurveys von zehn bis 15 Minuten (vgl.Revilla/Ochoa, 2017;Revilla/Höhne, 2020). ...
In einer Zeit, in der Nachhaltigkeit und Umweltschutz zu den zentralen Herausforderungen unserer Gesellschaft zählen, bietet diese Studie wertvolle Einblicke und praxisnahe Empfehlungen. Sie beleuchtet, wie Unternehmen Nachhaltigkeitsüberlegungen in ihre Projektmanagementprozesse integrieren und welche Rolle die SDGs dabei spielen. Mit einer detaillierten Analyse und fundierten Ergebnissen zeigt die Untersuchung auf, welche Fortschritte bereits erzielt wurden und wo noch Handlungsbedarf besteht. Die "Projektportfolio Sustainability Monitor 2024"-Studie untersucht den Status quo der Berücksichtigung von Nachhaltigkeit als Kriterium im Projektportfoliomanagement deutscher Unternehmen. Im Fokus steht dabei insbesondere die Integration der UN Sustainable Development Goals (SDGs). Ziel der Studie ist es zu erfassen, welche Bedeutung Nachhaltigkeitskriterien in der Projektselektion und -priorisierung haben und wie sich dies in der Einbindung der SDGs sowie in der Einhaltung relevanter Nachhaltigkeitsgesetze zeigt.
... age and interview experience), one for each track. The data from the pre-tests was omitted for the analyses, but the feedback led to some wording adjustments to avoid ambiguous phrases and verified that interviewees were able to complete the survey within 15-20 minutes, which is considered an appropriate length (Revilla & Höhne, 2020). ...
The paper investigates the signalling behaviour of digital native applicants in employment interviews and analyses how their reactions differ in face-to-face versus video-mediated contexts. The social presence within the interview setting and the possibility of employing impression management tactics are of particular interest to understanding the subjective acceptance and perceived fairness of the two types of selection procedures. The analyses of novel primary data from a German survey with 513 valid responses reveal that digital natives, similar to older applicants, appreciate signalling to lower information asymmetries. Regardless of interview mode, social presence and impression management are strong positive drivers of acceptance and perceived fairness. While members of the generational cohort still accept face-to-face interviews more than those mediated by videoconferencing technology, they perceive the former as less fair. This result, which may be explained by the specific characteristics of digital natives, contradicts the findings of studies that have investigated preceding generations. Hence, the paper complements the literature on applicant reactions by focusing on two younger generational cohorts, namely Generation Y and Z. Furthermore, the adoption of the signalling framework in this context suggests that the beneficial effects of signalling may stand vis-à-vis feelings of unfairness, which can be interpreted as additional psychological costs that are driven by moral considerations.
... Moreover, across the five respondents, the average completion time was 14 min and 17 s, which consolidated the efforts to design a sub-15 min survey. This timeframe was targeted as an appropriate duration for optimum engagement, reflecting the guidance of Revilla and Höhne [85], who reported that the ideal length of time for an online survey is between 10 and 15 min. Given the anticipated magnitude of ECEC practitioner datasets, the pilot was critical in verifying the internal functionality of the web-based survey platform and the associated data export operations. ...
Childhood trauma can exert a negative influence in the lives of young children. Yet, while Early Childhood Education and Care (ECEC) professionals are perfectly positioned to support children exposed to such trauma, extant research reports a scarcity of bespoke trauma awareness training for the ECEC profession. The aim of the current study served to explore the trauma awareness knowledge, strategies, and training of the ECEC profession in the Republic of Ireland. A comprehensive survey instrument, comprising 45 items across 5 Thematic Domains related to trauma knowledge and training, was disseminated to ECEC professionals nationwide. With a response rate of 1053 participants, key findings revealed (i) a fragmented understanding of what constitutes childhood trauma, and (ii) a significant association between lower levels of educational attainment and trauma education (Initial Practitioner Education, p = 0.000; Continuous Professional Development, p = 0.039). Further, 95% of participants called for context-specific, trauma awareness training, substantiating the voracious appetite for this crucial cog in the ECEC learning continuum, and thus reflecting the need for urgent reform to address and support the complexities of childhood trauma in ECEC discourse.
... The compensation offered by W&T was not only relatively low, but the task itself was very long. Online surveys should ideally be 5 to 15 min (Aguinis et al., 2021;Moss et al., 2023;Revilla & Höhne, 2020), with ~20 min being a reasonable maximum (Cape & Phillips, 2015;Chandler, 2023;Revilla & Ochoa, 2017). It would come as no surprise if many of W&T's participants abandoned the task or rushed to get a better hourly wage or avoid boredom (indeed, the lower bound of their reported completion times aligns with the maximum recommended time for a survey noted above). ...
A recent article in Perspectives on Psychological Science (Webb & Tangney, 2022) reported a study in which just 2.6% of participants recruited on Amazon’s Mechanical Turk (MTurk) were deemed “valid.” The authors highlighted some well-established limitations of MTurk, but their central claims—that MTurk is “too good to be true” and that it captured “only 14 human beings . . . [out of] N = 529”—are radically misleading, yet have been repeated widely. This commentary aims to (a) correct the record (i.e., by showing that Webb and Tangney’s approach to data collection led to unusually low data quality) and (b) offer a shift in perspective for running high-quality studies online. Negative attitudes toward MTurk sometimes reflect a fundamental misunderstanding of what the platform offers and how it should be used in research. Beyond pointing to research that details strategies for effective design and recruitment on MTurk, we stress that MTurk is not suitable for every study. Effective use requires specific expertise and design considerations. Like all tools used in research—from advanced hardware to specialist software—the tool itself places constraints on what one should use it for. Ultimately, high-quality data is the responsibility of the researcher, not the crowdsourcing platform.
... This duration could have led to response fatigue or reduced attention, potentially influencing the quality of the collected data, as longer surveys tend to decrease participant cooperation and completion rates (Sinkowitz-Cochran, 2013). Existing literature suggests that online surveys ideally fall within the 10 to 15-minute range for optimal participant engagement (Revilla & Höhne, 2020). However, as participation in our study was voluntary, it is assumed that most participants were interested in the topic of the questionnaire, decreasing the chance of a loss of motivation and/or attention. ...
... • In order not to take the results from participants whose attention may have been distracted by something having nothing to do with the experiment, all participants who had completed the whole task in more than 15 min (roughly speaking twice the median) were excluded from the analysis. This criterion was chosen in accordance with Revilla and Höhne (2020) recommendations on the ideal length of an online survey (i.e., between 10 and 20 min). We excluded 45 more participants because of this criterion. ...
The paper provides novel theoretical and experimental perspectives on the functioning of linguistic vagueness as an implicit persuasive strategy. It presents an operative definition of pragmatically marked vagueness, referring to vague expressions whose interpretation is not retrievable by recipients. The phenomenon is illustrated via numerous examples of its use in predominantly persuasive texts (i.e., advertising and political propaganda) in different languages. The psycholinguistic functioning of vague expressions is then illustrated by the results of a self-paced reading task experiment. Data showing shorter reading times associated with markedly vague expressions as compared to expressions that are either (a) lexically more precise or (b) made precise by the context suggest that the former are interpreted in a shallow way, without searching for and/or retrieving exact referents. These results support the validity of a differentiation between context-supported vs. non-supported vague expressions. Furthermore, validation of using marked vagueness as a persuasive implicit strategy which reduces epistemic vigilance is provided.
... We intended to provide the participating citizens with a short, manageable task to maximize engagement. We foresaw that an online task which took longer than 30 min would be off-putting for potential participants [43,44], and potentially reduce the response rate. We therefore chose ten surgical interventions with care to cover the spread of likely impacts of surgical delays, believing that evaluating the associated 20 health states in the Delphi study was realistic within this time limit, and would increase the representativeness of the sample. ...
Decision models can be used to support allocation of scarce surgical resources. These models incorporate health-related quality of life (HRQoL) values that can be determined using physician panels. The predominant opinion is that one should use values obtained from citizens. We investigated whether physicians give different HRQoL values to citizens and evaluate whether such differences impact decision model outcomes.
A two-round Delphi study was conducted. Citizens estimated HRQoL of pre- and post-operative health states for ten surgeries using a visual analogue scale. These values were compared using Bland–Altman analysis with HRQoL values previously obtained from physicians. Impact on decision model outcomes was evaluated by calculating the correlation between the rankings of surgeries established using the physicians’ and the citizens’ values.
A total of 71 citizens estimated HRQoL. Citizens’ values on the VAS scale were − 0.07 points (95% CI − 0.12 to − 0.01) lower than the physicians’ values. The correlation between the rankings of surgeries based on citizens’ and physicians’ values was 0.96 (p < 0.001).
Physicians put higher values on health states than citizens. However, these differences only result in switches between adjacent entries in the ranking. It would seem that HRQoL values obtained from physicians are adequate to inform decision models during crises.
... This is particularly the case because questionnaires for interviewer-administered surveys are often designed to be longer. In contrast, the ideal length of a web survey is between ten and 15 minutes, and the maximum length of a web survey is between 20 and 28 minutes (e.g., Revilla & Höhne, 2020;Revilla & Ochoa, 2017). ...
Large-scale surveys are increasingly moving from (face-to-face or telephone-based) interviewer-administered to self-administered online modes. To ensure high measurement quality and maximum comparability and equivalence between the source questionnaire and its adaptation⸺across modes as well as across survey waves⸺various aspects of question design must be considered and several decisions need to be made concerning question presentation and wording. This survey guideline summarizes good practices on how to transition questionnaires from interviewer- to self-administered web surveys and gives recommendations and examples for major adaptation issues as well as general questionnaire design elements relevant to web surveys. In this context, we focus on the switch from an interviewer-based to an online mode (although mixed-mode designs are also conceivable).
... This was of particular concern when designing a method to rate a large set of items because rating each one individually was likely to be a lengthy process. There is evidence that the ideal length of an online survey is between 10 and 15 minutes (65,66), and for this pairwise comparisonbased survey, the median time to complete was 14.1 minutes which sits in this bracket. ...
Quantitatively eliciting perspectives about a large number of similar entities (such as a list of competences) is a challenge for researchers in health professions education (HPE). Traditional survey methods may include using Likert items. However, a Likert item approach that generates absolute ratings of the entities may suffer from the "ceiling effect," as ratings cluster at one end of the scale. This impacts on researchers' ability to detect differences in ratings between the entities themselves and between respondent groups. This paper describes the use of pairwise comparison (this or that?) questions and a novel application of the Elo algorithm to generate relative ratings and rankings of a large number of entities, on a unidimensional scale. A study assessing the relative importance of 91 student "preparedness characteristics" for veterinary workplace clinical training (WCT) is presented as an example of this method in action. The Elo algorithm uses pairwise comparison responses to generate an importance rating for each preparedness characteristic on a scale from zero to one. This is continuous data with measurement variability which, by definition, spans an entire spectrum and is not susceptible to the ceiling effect. The output should allow for the detection of differences in perspectives between groups of survey respondents (such as students and workplace supervisors) which Likert ratings may be insensitive to. Additional advantages of the pairwise comparisons are their low susceptibility to systematic bias and measurement error, they can be quicker and arguably more engaging to complete than Likert items, and they should carry a low cognitive load for respondents. Methods for evaluating the validity and reliability of this survey design are also described. This paper presents a method that holds great potential for a diverse range of applications in HPE research. In the pursuit quantifying perspectives on survey items which are measured on a relative basis and a unidimensional scale (e.g., importance, priority, probability), this method is likely to be a valuable option.
... We split the German version of the 60-minute GGS questionnaire into three parts: a short recruitment wave (W1R) and two further subwaves (W1A three months after W1R and W1B four months after W1A). Each subwave should be close to the ideal length of a (web) survey of 10 to 15 minutes and below the maximum length of 30 minutes (Revilla & Höhne, 2020). There are a few studies on within-respondent modularization of questionnaires by splitting them into shorter parts and offering them to the respondents at several points in time. ...
... Laat tijdens het invullen de voortgang zien. Beloon respondenten als ze de vragenlijst hebben afgemaakt.Een vuistregel die door verschillende bureaus en experts wordt gehanteerd is dat het invullen van een online-vragenlijst niet veel langer dan 10 minuten mag duren (o.a.Revilla & Ochoa, 2017;Revilla & Höhne, 2020;Qualtrics, 2019;Wigmore, 2022). In dat geval mag de vragenlijst niet meer dan 30 vragen bevatten, waarbij het reageren op een stelling als één vraag telt. ...
The present report discusses how best to draw up the questionnaire survey on drug use in traffic. The Ministry of Infrastructure and Water Management would like to know what requirements the survey should meet to provide reliable and valid information for the risk indicator on drug use in traffic. Which types of drugs should be enquired about, over which period, in which way and in which sample?
To answer these questions, we used SWOV expertise, best practices in the field of questionnaire surveys, and the results of a brief literature review concerning drug use in general and during traffic participation in particular.
... Our survey was simple and designed to be short (Revilla and Hohne, 2020). Respondents took approximately 12 min to complete it (median), but it is possible that, if using their phones and having to go through multiple pages, respondents may have felt discouraged from completing it. ...
Our objective was to understand dairy employees' perceptions and educational needs at the onset of the COVID-19 pandemic. A bilingual (English and Spanish), anonymous survey targeted at dairy employees was circulated nationwide via university and allied industry media outlets. Responses (n = 63) from 11 states were received (May-Sep. 2020). Respondents worked in herds ranging from 50 to 40,000 animals in size. Dairy managers (33%) responded mostly to the English survey (52%), whereas entry-level workers (67%) chose the Spanish format (76%). Survey results highlighted different perspectives, educational needs, and preferred sources of information between English- and Spanish-speaking dairy workers. Overall, 83% of the respondents were somewhat concerned or very concerned about the COVID-19 pandemic. Respondents (51%) indicated that their main concern was "to bring the virus from work to home and make my family sick." Most dairy employees (83%) perceived that their employers were somewhat or very concerned about the pandemic. Respondents (65%) indicated that COVID-19 informative training was provided at the workplace, but training was more frequently undertaken among dairy managers (86%) than entry-level workers (53%). Most trainings (72%) were limited to posters on walls. The preferred means of information delivery was through in-person meetings at work (35%), with YouTube (29%) and on-demand videos (27%) as second and third options. The main source of information regarding the pandemic was social media (52%). Frequent handwashing (81%), limiting on-farm visits (70%), limiting agglomeration in break rooms (65%), hand sanitizer use (60%), and social distancing (60%) were the most common safety measures implemented at the workplace among the options given to respondents. Few respondents (38%) indicated that face-covering was required at work. Successful emergency plans on dairies should consider the outreach needs and preferences of dairy workers.
... Sayfa: 140-149 Sakallı A. O., Sakallı S., Akbaşak Ö. A., Erkut S. şekilde ayarlandı. 4 E-posta, WhatsApp (Meta Corp, USA) ve sosyal medya aracılığı ile katılımcılara ulaştırıldı. ...
Amaç: Bu çalışma SARS CoV-2 salgınının dental kliniklerde ne gibi değişiklikler yapılması gerektiğini ve bu değişikliklerin ne derecede uygulandığını anket yoluyla bulmayı amaçlamaktadır.Gereç ve Yöntem: Ankara Dişhekimleri Odası'na üye olan toplamda 385 diş hekimine SARS CoV-2 pandemisi sonucunda klinikte ne gibi değişiklikler yapıldığına dair sorular hazırlanıp online anket yolu ile ulaştırılmıştır.Bulgular: Diş hekimlerinin öğrencilik döneminden itibaren almış olduğu hastalıklardan korunma tedbirlerinin SARS CoV-2 pandemisine iyi uyum sağladığını göstermiştir.Sonuç: Anket sonuçlarına göre diş hekimleri olarak gelecekte benzeri yaşanılabilecek salgınlara karşı diş kliniklerinde salgın hastalıklara karşı etkin bir düzenleme standardı oluşturulmasının gelecekte yaşanılacak durumlara daha hızlı uyum sağlayabileceğimiz sonucu çıkarılabilir.
... We also piloted with a large number of users to get an accurate estimate of the time to complete the survey, so the advertised time was strongly aligned with the actual time taken by our participants. According to existing research [58,59], survey respondents get fatigued after 20 minutes, which might impact the data. In our case, however, there were two questionnaires with a break in between, in which participants had to use the tool under evaluation. ...
Data collection and aggregation by online services happens to an extent that is often beyond awareness and comprehension of its users. Transparency tools become crucial to inform people, though it is unclear how well they work. To investigate this matter, we conducted a user study focusing on Facebook, which has recently released the 'Off-Facebook Activity' transparency dashboard that informs about personal data collection from third parties. We exposed a group of n = 100 participants to the dashboard and surveyed their level of awareness and reactions to understand how transparency impacts users' privacy attitudes and intended behavior. Our participants were surprised about the massive amount of collected data, became significantly less comfortable with data collection, and more likely to take protective measures. Collaterally, we observed that current consent schemes are inadequate. Based on the survey findings, we make recommendations for more usable transparency and highlight the need to raise awareness about transparency tools and to provide easily actionable privacy controls.
... We need to address this concern in the future in case of future COVID-19-related lockdowns. A panel of expert researchers has suggested the maximum length of an online survey should range between 15 and 30 min [7], and we will therefore take this into consideration, in addition to the adaptation to the context of the country. Moreover, we should mention that we did not assess the impact of LQSFA on the transition of children back into class-based learning. ...
Background: The Lions Quest Skills for Adolescence (LQSFA) is an evidence-based social and emotional learning program for school students. It is implemented as a teacher-led extracurricular activity for children aged 10–15 years. From 2019 to 2022, the United Nations Office on Drugs and Crime, in collaboration with Lions Clubs International Foundation, implemented the LQSFA in 41 schools in Croatia. Due to the COVID-19 lockdown measures, the intervention was adjusted into a hybrid modality (in-class and online). We evaluated the experience that the teachers had with the LQSFA in a hybrid modality. Methods: We used a focus-group discussion approach to evaluate the experience of five LQSFA teachers. Results: Three themes emerged: (1) the appreciation of evidence-based programs by the teachers, (2) the benefit of the LQSFA on the parents, and (3) the length of the online version of the questionnaire tool that was used to assess pre- and post-LQSFA experiences among students was too long. These results indicate that the LQSFA is undergoing a scaling on a national level in Croatia, even when implemented in a hybrid setting. Conclusion: Using an evidence-based program such as the LQSFA was rewarding for teachers, despite the challenges in the administrative adjustments regarding the online and in-person class teaching. LQSFA filled an important gap during COVID19-related stress.
... On the other side, "learn-a-lot" was mentioned in the same way. marketing studies can be completed in several minutes and for an optimal study time as a trade-off between high attention level and annoying, a good range of 10-15 min until 20-28 min as a maximum could be identified, which rated our study time to the nearly possible limit (Revilla and Höhne, 2020). They concluded that the survey time was primarily influenced by socio-demographics, personality, and the survey difficulty. ...
Illumination preference models are usually defined in a static scenery, rating common-colored objects by a single scale or semantic differentials. Recently, it was reported that two to three illumination characteristics are necessary to define a high correlation in a bright office-like environment. However, white-light illumination preferences for vehicle-occupants in a dynamic semi- to full automated modern driving context are missing. Here we conducted a global free access online survey using VR engines to create 360° sRGB static in-vehicle sceneries. A total of 164 participants from China and Europe answered three levels in our self-hosted questionnaire by using mobile access devices. First, the absolute perceptional difference should be defined by a variation of CCT for 3,000, 4,500, and 6,000 K or combinations, and light distribution, either in a spot- or spatial way. Second, psychological light attributes should be associated with the same illumination and scenery settings. Finally, we created four driving environments with varying external levels of interest and time of the day. We identified three key results: (1) Four illumination groups could be classified by applying nMDS. (2) Combinations of mixed CCTs and spatial light distributions outperformed compared single light settings (p < 0.05), suggesting that also during daylight conditions artificial light supplements are necessary. (3) By an image transformation in the IPT and CAM16 color appearance space, comparing external and in-vehicle scenery, individual illumination working areas for each driving scenery could be identified, especially in the dimension of chroma-, partially following the Hunt-Effect, and lightness contrast, which synchronizes the internal and external brightness level. We classified our results as a starting point, which we intend to prove in a follow-up-controlled laboratory study with real object arrangements. Also, by applying novel methods to display high fidelity 360° rendered images on mobile access devices, our approach can be used in the future interdisciplinary research since high computational mobile devices with advanced equipped sensory systems are the new standard of our daily life.
... The median time it took to complete the questionnaire ranged between 21 and 24 minutes across the six countries in the sample. While longer than the optimal length for online panel research (10 to 15 minutes), this length still falls below the maximum recommended length of 20 to 28 minutes (Revilla & Höhne, 2020). We also note that our completion rates, which varied between 78.4% (UK) 2 and 91.3% (China), fell within the same ballpark (~85%) that Liu and Wronski (2018) noted based on a sample of 25,000 online surveys. ...
Research rationale: Market for wearable devices and applications designed for personal health and fitness has significantly increased in recent years and this segment is expected to continue growing. Furthermore, the boundaries between this segment and the smartphone and smartwatch segments are blurring as more and more smart devices offer functions previously only associated with standalone wearables. This makes it increasingly difficult to understand the full scope of when and how people use corresponding functions. Thorough knowledge about usage patterns will become important as the sensors and apps used for health and fitness services increase the granularity of behavior tracking. The data generated by these services will give valuable insights into people's physical and mental health. Policymakers will also need to take a nuanced approach when handling these issues to preserve fair conduct and competition as the data collected will be intimate in nature, and the data's impact on the health and pharmaceutical industry could be huge. Research objectives: Our paper sets out to (1) define a scope for the variety of devices, apps, and functions currently used by consumers to monitor and affect their health or fitness status. We further seek to (2) shed light on the actual usage patterns of these devices, apps, and functions, as well as what specific combinations of devices and apps consumers rely upon. Finally, we will (3) collect data on consumers' attitudes towards the helpfulness and accuracy of self-tracking, use cases for wearables, and the (potential) sharing of this collected data for medical diagnoses and treatment purposes. Methods: To fulfill these research objectives, we conducted representative online surveys relying on a stratified sampling approach in six countries (China, France, Germany, Italy, UK, and the US) featuring a sample of n>3,000 respondents in each country. A professional market research agency carried out the fieldwork and oversaw translation of the questionnaire into local languages. The fieldwork was carried out in late April 2022 and early May 2022. Results and implications: Our paper shows that: the omnipresent smartphone whose sensors can be accessed by various health and fitness apps remains the key platform for consumers when it comes to health and fitness; adoption rates for apps from within major digital ecosystems are similar to the ones we found for third-party apps; smartwatches constitute the second most relevant category of devices in the health and fitness space with substantial additional market potential; consumers are broadly open to using even niche and emerging wearables such as smart rings, skin patches, and PEMF headbands; both general and exercise specific tracking features are the most popular functions among consumers across age groups, while social and sharing functions do not resonate strongly with them; and that additional devices commonly also add a new (companion) app to the user portfolio of health and fitness apps. Based on these findings, we would expect a wider variety of apps to be used as the trend to consumer IoT unfolds. We have also found that consumers are positively inclined towards health and fitness technologies. Users of related devices and apps largely agree that these technologies help them achieve their exercise goals. They furthermore perceive the readings of their trackers to be accurate. However, we found more skeptical views when it comes to sharing the collected data with physicians or using collected data to detect illness early on. Our findings also underscore the importance of health and fitness technology for consumers and the likelihood of continued growth in this segment. So far, we did not find any indications of concerns related to digital ecosystem dominance. In fact, our results point instead to an increasing variety of apps being used as consumers adopt more devices. The low interest from respondents in sharing their records or collected data illustrates once more the value that consumers attach to self-determination when it comes to managing their online presence and the concern they have with keeping their data private.
... We also piloted with a large number of users to get an accurate estimate of the time to complete the survey, so the advertised time was strongly aligned with the actual time taken by our participants. According to existing research [59,60], survey respondents get fatigued after 20 minutes, which might impact the data. In our case, however, there were two questionnaires with a break in between, in which participants had to use the tool under evaluation. ...
Data collection and aggregation by online services happens to an extent that is often beyond awareness and comprehension of its users. Transparency tools become crucial to inform people, though it is unclear how well they work. To investigate this matter, we conducted a user study focusing on Facebook, which has recently released the "Off-Facebook Activity" transparency dashboard that informs about personal data collection from third parties. We exposed a group of n = 100 participants to the dashboard and surveyed their level of awareness and reactions to understand how transparency impacts users' privacy attitudes and intended behavior. Our participants were surprised about the massive amount of collected data, became significantly less comfortable with data collection, and more likely to take protective measures. Collaterally, we observed that current consent schemes are inadequate. Based on the survey findings, we make recommendations for more usable transparency and highlight the need to raise awareness about transparency tools and to provide easily actionable privacy controls.
... We decided to use only one video and to shorten it essentially for two reasons. First, according to Cheng et al. (2013), the most popular videos on YouTube are less than a minute long (corresponding to 20.0% of the videos on YouTube); second, the ideal length of an online survey is between 10 and 15 minutes (Revilla and Höhne, 2020). ...
In robotics, a field of research still populated by prototypes, much of the research is made through videos and pictures of robots. We study how the highly human-like robot Sophia is perceived through a YouTube video. Often researchers take for granted in their experiments that people perceive humanoids as such. With this study we wanted to understand to what extent a convenience sample of university students perceive Sophia’s human-likeness; second, we investigated which mental capabilities and emotions they attribute to her; and third, we explored the possible uses of Sophia they imagine. Our findings suggest that the morphological human-likeness of Sophia, through the video, is not salient in the Sophia’s representations of these participants. Only some mental functions are attributed to Sophia and no emotions. Finally, uses of Sophia turned out to be connected to the gender stereotypes that characterize stereotyped women’s professions and occupations but not completely.
... Frequency of sharing content on social media, trust in survey privacy, and survey safety were measured as in Revilla et al. (2019). Five questions were selected from the Spanish adaptation (Martínez Mompó, 2015) of the "Big 5" Personality Trait Short Questionnaire in adults (Morizot, 2014), similar to those already used by Revilla and Höhne (2020) to research survey length preferences by panelists. ...
Surveys are a fundamental tool of empirical research, but they suffer from errors: in particular, respondents can have difficulties recalling information of interest to researchers. Recent technological developments offer new opportunities to collect data passively (i.e., without participant’s intervention), avoiding recall errors. One of these opportunities is registering online behaviors (e.g., visited URLs) through tracking software (“meter”) voluntarily installed by a sample of individuals on their browsing devices. Nevertheless, metered data are also affected by errors and only cover part of the objective information, while subjective information is not directly observable. Asking participants about such missing information by means of web surveys conducted in the moment an event of interest is detected by the meter has the potential to fill the gap. However, this method requires participants to be willing to participate. This paper explores the willingness to participate in in-the-moment web surveys triggered by online activities recorded by a participant-installed meter. A conjoint experiment implemented in an opt-in metered panel in Spain reveals overall high levels of willingness to participate among panelists already sharing metered data, ranging from 69% to 95%. The main aspects affecting this willingness are related to the incentive levels offered. Limited differences across participants are observed, except for household size and education. Answers to open questions also confirm that the incentive is the key driver of the decision to participate, whereas other potential problematic aspects such as the limited time to participate, privacy concerns, and discomfort caused by being interrupted play a limited role.
... Consistent response quality may be attributed to participants' enthusiasm about the topic or that they found the topic important. Our average survey completion time is consistent with other completion times across multiple fields of study [79]. ...
Large language models such as GPT-3 and ChatGPT can mimic human-to-human conversation with unprecedented fidelity, which enables many applications such as conversational agents for education and non-player characters in video games. In this work, we investigate the underlying personality structure that a GPT-3-based chatbot expresses during conversations with a human. We conducted a user study to collect 147 chatbot personality descriptors from 86 participants while they interacted with the GPT-3-based chatbot for three weeks. Then, 425 new participants rated the 147 personality descriptors in an online survey. We conducted an exploratory factor analysis on the collected descriptors and show that, though overlapping, human personality models do not fully transfer to the chatbot's personality as perceived by humans. We also show that the perceived personality is significantly different from that of virtual personal assistants, where users focus rather on serviceability and functionality. We discuss the implications of ever-evolving large language models and the change they affect in users' perception of agent personalities.
This primer offers advice on survey best practices, with a specific focus on the beginnings of the survey creation process—from choosing a survey tool and writing survey questions, to articulating the purpose of a survey and accessibility considerations. These best practices will provide a foundation to help you collect reliable, quality data to understand your target audiences more deeply. Please note: This primer does not claim to be an all-encompassing source of survey knowledge; it is an introduction to 6 considerations when creating surveys.
Postulating a negative bias towards social ambiguity, we conducted cross-cultural online research to assess whether categorical discrepancies in the perception of androgynous faces were associated with the uncanny feeling and inferences of different morality. Across four studies, we found that androgynous faces were harder to classify into a binary sex category than sex-typical faces, but this difficulty did not influence social judgements of androgynous targets in a negative fashion. In Study 1 (Spanish-speaking sample, N = 76), we found that androgynous faces were rated as more trustworthy, less creepy, and less morally different than sex-typical faces. Study 2 replicated most of the findings from Study 1 in an Italian sample (N = 45). Positive bias towards androgyny was not replicated with a different set of stimuli featuring faces of diverse ethnic backgrounds (Study 3, Spanish-speaking sample, N = 140). However, results revealed a main effect of ethnicity in participants’ responses. When controlling for the effect of morphing procedures in stimuli selection, an overall positive bias towards androgynous targets arose, especially when compared to masculine targets (Study 4, Spanish-speaking sample, N = 85). These findings suggest that, at least in certain conditions, a positive social bias towards androgynous faces may emerge that does not depend on categorical uncertainty and facial attractiveness.
Research with children and young people is essential for understanding antisocial behavior and victimization. Likewise, learning about juvenile offenders’ needs and experiences has been a central issue in juvenile justice research. Therefore, criminology needs research with young people. In this chapter, we discuss the ethical and emotional issues in researching with and for children, based on our own experience in the field. We begin by emphasizing that the entire design of an investigation, in each of its phases, must be respectful and careful of children’s well-being, taking a child-centered approach. We pay special attention to how to encourage the voice of young people to be heard and attended to throughout the research process. Secondly, we reflect on how researchers can respond to ethical requirements, such as free participation and anonymity; also, we discuss about emotional work that researchers must carry out. Finally, we collect some lessons learned and good practices to carry out rigorous and respectful research.
The first of three volumes, the five sections of this book cover a variety of issues important in developing, designing, and analyzing data to produce high-quality research efforts and cultivate a productive research career. First, leading scholars from around the world provide a step-by-step guide to doing research in the social and behavioral sciences. After discussing some of the basics, the various authors next focus on the important building blocks of any study. In section three, various types of quantitative and qualitative research designs are discussed, and advice is provided regarding best practices of each. The volume then provides an introduction to a variety of important and cutting-edge statistical analyses. In the last section of the volume, nine chapters provide information related to what it takes to have a long and successful research career. Throughout the book, example and real-world research efforts from dozens of different disciplines are discussed.
Established face-to-face surveys encounter increasing pressures to move online. Such a mode switch is accompanied with methodological challenges, including the need to shorten the questionnaire that each respondent receives. Split Questionnaire Designs (SQDs) randomly assign respondents to different fractions of the full questionnaire (modules) and, subsequently, impute the data that are missing by design. Thereby, SQDs reduce the questionnaire length for each respondent. Although some researchers have studied the theoretical implications of SQDs, we still know little about their performance with real data, especially regarding potential approaches to constructing questionnaire modules. In a Monte Carlo study with real survey data, we simulate SQDs in three module-building approaches: random, same topic, and diverse topics. We find that SQDs introduce bias and variability in univariate and especially in bivariate distributions, particularly when modules are constructed with items of the same topic. However, single topic modules yield better estimates for correlations between variables of the same topic.
This article describes the history of the journal published by the UK Market research Society (MRS). The reasons for introducing a journal are discussed, and the competitive context, at that time and onwards. Changes in the format, title and content of the journal over time are described, and its evolution from an in-house production to its current status within the journal portfolio of a leading international academic publishing house. The changing nature of the two main communities with the most interest in the journal, academics and practitioners, are described and the impact on the journal are discussed. Finally, the article summarises some of the many key contributions that the journal has made to the body of knowledge and evidence in the fields of market and social research.
The primary aim of this paper is to draw practitioners’ attention to lesser-known risks of digital marketing research: while it enables quick and low-cost results, quality and reliability are not guaranteed. The paper also surfaces broader consequences of transitioning from traditional research, based on offline investigations and face-to-face interviews carried out by professionals, to digital research. The paper presents the results of a survey on a cohort of 200 freelance interviewers working for Italy’s main research institutions, conducted through a self-administered questionnaire. Recently online marketing research, especially through panels, has gained meaningful traction. As demand for traditional marketing research contracts, professional interviewers are experiencing a material drop in requests for their in-field services and a worsening working environment. In return, this affects the quality of on field research they can provide. This is the first study, to the best of the author’s knowledge, where issues and limitations of digital research are studied from the perspective of professional interviewers. This study enables managers and organisations that commission marketing research to make more informed decisions when facing the trade-offs between traditional and digital methods. Furthermore, it provides a view on how such choices may impact the future of professional interviewers and their services.
International migration originating from highly developed countries is a crucial component of global migration flows. There are, however, surprisingly little data about the international mobility of the populations of affluent countries. The German Emigration and Remigration Panel Study (GERPS) aims to provide a resource that enables the analysis of individual consequences of international migration as well as the socio-structural consequences for the country of origin. GERPS is based on an origin-based multistage probability sample using the German population registers as a sampling frame. The second wave yields a net sample of roughly 7,000 persons who recently moved abroad from Germany and persons who returned from Germany after having lived abroad. The study follows a multi-destination-country-design and enables the comparative analysis of migrants and non-migrants, who stayed in the country of origin. GERPS is a panel study with at least four waves during a period of at least 24 months. This documentation presents the methods and data of the second wave. It provides information for researchers and invites them to use the new data infrastructure for their own research.
Embedded Client Side Paradata (ECSP) is a tool that is licensed under the Creative Commons Attribution 4.0 International License (see http://creativecommons.org/licenses/by/4.0/). It is based on different program languages, such as JavaScript and HTML. In general, ECSP can be implemented in web-based survey software solutions that provide access to the source code. It enables researchers to passively collect different kinds of client-side paradata, such as response times and scrolling events, and data from built-in sensors, such as Global Positioning System (GPS) and acceleration data. This is irrespective of the Internet browser and operating system used and allows researchers to investigate respondents’ completion behavior with respect to web surveys. Paradata and sensor data are collected at the page-level and are stored together with the actual survey data (i.e., respondents’ answers) in the same dataset.
Embedded Client Side Paradata (ECSP) is a tool that is licensed under the Creative Commons Attribution 4.0 International License (see http://creativecommons.org/licenses/by/4.0/). It is based on different program languages, such as JavaScript and HTML. In general, ECSP can be implemented in web-based survey software solutions that provide access to the source code. It enables researchers to passively collect different kinds of client-side paradata, such as response times and scrolling events – irrespective of the Internet browser and operating system used – to investigate respondents’ completion behavior with respect to web surveys. Paradata are collected at the page-level and are stored together with the actual survey data (i.e., respondents’ answers) in the same dataset.
Certain survey characteristics proven to affect response rates, such as a survey's length and topic, are often under limited control of the researcher. Therefore, survey researchers sometimes seek to compensate for such undesired effects on response rates by employing countermeasures such as material or nonmaterial incentives. The scarce evidence on those factors' effects in web survey contexts is far from being conclusive. This study is aimed at filling this gap by examining the effects of four factors along with selected interactions presumed to affect response rates in web surveys. Requests to complete a web-based, self-administered survey were sent to 2,152 owners of personal websites. The 2 Ã 2 Ã 2 Ã 2 fully crossed factorial design encompassed the experimental conditions of (a) high versus low topic salience, (b) short versus long survey, (c) lottery incentive versus no incentive, and (d) no feedback and general feedback (study results) versus personal feedback (individual profile of results).
This study examines the effect of the timing of follow-ups, different incentives, length, and presentation of the questionnaire on the response rate and response quality in an online experimental setting. The results show that short questionnaires have a higher response rate, although long questionnaires still generate a surprisingly high response. Furthermore, vouchers seem to be the most effective incentive in long questionnaires, while lotteries are more efficient in short surveys. A follow-up study revealed that lotteries with small prizes, but a higher chance of winning are most effective in increasing the response rate. Enhancing questionnaires with visual elements, such as product images, lead to a higher response quality and generate interesting interaction effects with the length of the questionnaire and the incentives used. Finally, the timing of the follow-up has no significant influence on the response rate.
This paper aims to discover ‘How long can/should a survey be?’ by asking the question to respondents themselves in a web survey implemented by the Netquest fieldwork company in Mexico in 2016. It is a question that is frequently asked, even if research has already been carried out on the impact of survey length on non-response or data quality. The study conducted for this paper concluded that the ideal survey length is a median of 10 minutes and that the maximum survey length is 20 minutes. The reported lengths were significantly linked to the fact that respondents liked answering the survey and that they trusted that their data are treated in an anonymous way, but were not found to be linked to the socio-demographic variables tested, except age in the case of maximum length.
Web surveys appear to be attaining lower response rates than equivalent mail surveys. One reason may be that there is currently little information on effective strategies for increasing response to Internet-based surveys. Web users are becoming more impatient with high-burden Web interactions. The authors examined the decision to respond to a Web survey by embedding a series of experiments in a survey of students at the University of Michigan. A sample of over 4,500 students was sent an e-mail invitation to participate in a Web survey on affirmative action policies. Methodological experiments included using a progress indicator, automating password entry, varying the timing of reminder notices to nonrespondents, and using a prenotification report of the anticipated survey length. Each of these experiments was designed to vary the burden (perceived or real) of the survey request. Results of these experiments are presented.
This paper investigates how expected and actual questionnaire length affects cooperation rates and a variety of indicators
of data quality in web surveys. We hypothesized that the expected length of a web-based questionnaire is negatively related
to the initial willingness to participate. Moreover, the serial position of questions was predicted to influence four indicators
of data quality. We hypothesized that questions asked later in a web-based questionnaire will, compared to those asked earlier,
be associated with (a) shorter response times, (b) higher item-nonresponse rates, (c) shorter answers to open-ended questions,
and (d) less variability to items arranged in grids. To test these assumptions, we manipulated the stated length (10, 20,
and 30 minutes) and the position of questions in an online questionnaire consisting of randomly ordered blocks of thematically
related questions. As expected, the longer the stated length, the fewer respondents started and completed the questionnaire.
In addition, answers to questions positioned later in the questionnaire were faster, shorter, and more uniform than answers
to questions positioned near the beginning.
& SFB 884 “Political Economy of Reforms
- A G Blom
- M Fikel
- S Friedel
- J K Höhne
- U Krieger
- T Rettig