No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Conducting organizational survey and experimental research online: From convenient to ambitious in study designs, recruiting, and data quality
Abstract
Conducting organizational research via online surveys and experiments offers a host of advantages over traditional forms of data collection when it comes to sampling for more advanced study designs, while also ensuring data quality. To draw attention to these advantages and encourage researchers to fully leverage them, the present paper is structured into two parts. First, along a structure of commonly used research designs, we showcase select organizational psychology (OP) and organizational behavior (OB) research and explain how the Internet makes it feasible to conduct research not only with larger and more representative samples, but also with more complex research designs than circumstances usually allow in offline settings. Subsequently, because online data collections often also come with some data quality concerns, in the second section, we synthesize the methodological literature to outline three improvement areas and several accompanying strategies for bolstering data quality.
Plain Language Summary: These days, many theories from the fields of organizational psychology and organizational behavior are tested online simply because it is easier. The point of this paper is to illustrate the unique advantages of the Internet beyond mere convenience—specifically, how the related technologies offer more than simply the ability to mirror offline studies. Accordingly, our paper first guides readers through examples of more ambitious online survey and experimental research designs within the organizational domain. Second, we address the potential data quality drawbacks of these approaches by outlining three concrete areas of improvement. Each comes with specific recommendations that can ensure higher data quality when conducting organizational survey or experimental research online.
... Organizations may build productive and positive work environments through informed decisions, strategies, and interventions based on good methodology and objective data analysis. (Van, et al., 2022). ...
... Online organizational surveys and experiments can benefit from each of these suggestions for improving data quality. (Van, et al., 2022). In the context of organizational behavior, bibliometric data analysis is a useful approach to understanding developments and trends in a field of research. ...
Workload, organizational culture, work discipline, and job performance are some of the topics that this bibliometric review hopes to go into. The study finds trends, patterns, and gaps in the existing literature by using bibliometric analytic approaches. It utilizes co-citation and keyword analysis to uncover patterns of research collaboration and creates a publishing map according to year, author, journal, institution, and country. Researchers, practitioners, and policymakers can benefit from the analysis by better understanding the most recent breakthroughs and trends in organizational behavior, specifically related to employee performance. Furthermore, it addresses voids in the existing literature and pinpoints topics that require additional research. The results can help improve organizational policy initiatives and direct future research in the fields of human resource management and organizational behaviour. Job interview phrases and corporate culture were the subjects of an online poll that ran from 2019 through 2024. Beyond highlighting three ways these approaches could be enhanced to increase data quality, the research tries to demonstrate the benefits of online testing beyond its convenience. These recommendations for enhancing the quality of data can be applied to online organizational surveys and experiments.
... Given previous literature that has explored public safety professionals' views on substance use (e.g., Lowder et al., 2019; van Boekel et al., 2014), we found it somewhat surprising that some respondents from the justice system also expressed similar ideas in their open-ended answers, such as acknowledging the role of trauma and violence in relation to substance use. While we also found this result encouraging, we note that it may stem from self-selection bias (e.g., Nayak & Narayan, 2019;Van Quaquebeke et al., 2022). ...
... Given CPHA's professional relationships with organizations and individuals asked to support survey dissemination, as well as previously successful experiences with using a similar recruitment approach (e.g., CPHA, 2018b), we felt confident that respondents would be knowledgeable members of the survey's target audiences. That said, we also note that our approach raises the potential for self-selection biases (e.g., Nayak & Narayan, 2019;Van Quaquebeke et al., 2022), especially upon our reflection of the, overall, favourable perspectives in relation to a PHA to substance use collected from survey respondents across sectors. ...
... Finally, participants in our studies were recruited from an online panel (i.e., Prolific). Although this data source has several advantages (Van Quaquebeke et al., 2022), it also presents some limitations such as lack of control over participant attention. We followed several recommendations (e.g., prescreening, attention checks, preventing multiple submissions) to enhance our data quality (Keith et al., 2017;Van Quaquebeke et al., 2022), but future research could draw on alternative data sources. ...
... Although this data source has several advantages (Van Quaquebeke et al., 2022), it also presents some limitations such as lack of control over participant attention. We followed several recommendations (e.g., prescreening, attention checks, preventing multiple submissions) to enhance our data quality (Keith et al., 2017;Van Quaquebeke et al., 2022), but future research could draw on alternative data sources. ...
In response to calls for greater diversity, equity, and inclusion (DEI) in the workplace, many organizations have implemented a leadership role dedicated to advancing DEI. Although prior research has found that the traditional leader is associated with being White, anecdotal evidence suggests DEI leader roles are predominantly held by non-White individuals. To examine this contradiction, we draw on social role and role congruity theories to conduct three preregistered experimental studies (N = 1,913) and explore whether the DEI leader role diverges from the traditional leader role such that observers expect a DEI leader to be non-White (i.e., Black, Hispanic, or Asian). Our findings indicate that DEI leaders are generally presumed to be non-White (Study 1) and that observers perceive traits associated with non-White, rather than White, groups correspond more strongly with traits required for the DEI leader role (Study 2). We also explore the effects of congruity and find non-White candidates receive stronger leader evaluations for a DEI leader role and that this relationship is mediated by nontraditional, role-specific traits (i.e., commitment to social justice and suffered discrimination; Study 3). We conclude by discussing the implications of our work for DEI and leadership research as well as for work drawing on role theories.
... LegerOpinion ensures data quality through digital fingerprinting, background checks, and timestamps to flag careless responses. Similar online platforms provided evidence for the reliability and quality of the data collected [53]. We informed respondents of the purpose of the study, the voluntary nature of the survey, and the confidentiality of data treatment. ...
... Additionally, the validity of our findings may be limited by the fact that an external survey company collected all data. However, some researchers agree that survey firms produce data that displays good test-retest reliability, internal consistency, factorial stability, and relationships that are very similar to those found among data directly collected by researchers [53,75,76]. Further, our results are not necessarily generalizable, given that we collected data in Canada, where a federal program allowed many organizations to engage in procedures for furloughing workers. ...
During the COVID-19 lockdown period, several employers used furloughs, that is, temporary layoffs or unpaid leave, to sustain their businesses and retain their employees. While furloughs allow employers to reduce payroll costs, they are challenging for employees and increase voluntary turnover. This study uses a two-wave model (Time 1: n = 639/Time 2: n = 379) and confirms that furloughed employees’ perceived justice in furlough management and job insecurity (measured at Time 1) explain their decision to quit their employer (measured at Time 2). In addition, our results confirm that furloughed employees’ job embeddedness (measured at Time 1) has a positive mediator effect on the relationship between their perceived procedural justice in furlough management (measured at Time 1) and their turnover decision (Time 2). We discuss the contribution of this study to the fields of knowledge and practice related to turnover and furlough management to reduce their financial, human, and social costs.
... Three criteria were applied for exclusion of inattentive participants: First, all participants who stopped answering the survey before assignment of the experimental treatment (2116 participants). 4 Second, participants who answered the entire survey in less than 5 min with the expected response time being 12-15 min (104 participants). This decision was based on a cursory look at response time for outliers to detect careless responses (Huang et al., 2012;Van Quaquebeke et al., 2022). Third, participants who failed to answer more than half of the questions capturing preferences for leader facial dominance (56 participants). ...
... The pandemic has had a crucial impact on web survey responses due to the sudden transition to a fully digital work and life environment. This shift has led to a dramatic increase in the number of web surveys conducted, prompting researchers and organisations to implement diverse approaches to adapt to the new realities of data collection in a digital-first context [2,24,61]. Although many educational institutions considered MOOCs as an alternative to the traditional education system due to the flexibility of learning time during COVID-19, it contributed to high dropout rates and reduced user participation in learning activities such as forums and comments [44]. ...
The study explored the pedagogical practices employed by secondary school teachers in Tanzania during and after the COVID-19 pandemic, with a focus on Mbeya City Council. The study was guided by the Unified Theory of Acceptance and Use of Technology (UTAUT). The study aimed to explore the pedagogical practices used by teachers during and after the COVID-19 pandemic in Tanzania's secondary schools. A qualitative approach was utilised with a phenomenological research design. A total of 33 participants were involved, including one District Secondary Education Officer (DSEO), two Ward Education Officers (WEOs), six secondary school heads, and 24 teachers (four from each selected school). The data were analysed using thematic analysis strategies. The study revealed that teachers employed various methods, such as adaptive learning platforms, collaborative learning, interactive presentations, online discussions, and tools for assessment and feedback. However, these efforts were hindered by challenges such as unreliable internet access, insufficient infrastructure, and inadequate administrative and technical support. The study concludes that teachers' capacity for technology-enhanced pedagogy should be strengthened. The study recommends that the government and education stakeholders provide support to secondary schools in adopting digital tools for teaching and learning to ensure high-quality education that can compete in the global market.
... questionnaires were used for the purpose of the data collection. The time savings compared to in-person interviews make structured questionnaires an attractive option for data collection Van Quaquebeke et al.,2022). Three power stations in Saudi Arabia's were among the respondents in a recent study. ...
This study examines the impact of effect of AI Application on the quality of the photovoltaic system in the context of the KSA, focusing on AI application roles: forecasting and prediction, demand response management, grid optimization and automation, integration of distributed energy resources, Energy storage optimization and Enhanced security and resilience and their effects on photovoltaic system indicators, PV system performance, reliability and maintenance, component quality, environmental impact and durability, safety and compliance, overall Satisfaction. As AI Application roles increasingly reshapes worldwide industries, therefore understanding its specific benefits in Saudi Arabia’s context is vital for the KSA vision 2030.
A quantitative approach was adopted for this research, targeting employees working in three power plant in KSA in Saudi Arabia. Data was collected via a structured questionnaire, with 344 valid responses retained. A Multiple Linear Regression was employed to analyze the relationships between AI application roles dimensions and quality of photovoltaic system indicators. The findings confirmed that all AI application roles dimensions, have a significant positive impact on quality of photovoltaic system.
The study contributes to the limited research on AI application role in PV system quality within the context of Saudi Arabia, offering insights for stakeholders aiming to improve overall PV system quality. Recommendations include prioritizing the adoption of AI application roles that enhance PV system performance, reliability and maintenance, component quality, environmental impact and durability, safety and compliance, and overall satisfaction to achieve optimal PV system quality.
... Experimental research is still considered the preeminent way to test hypotheses in organizational research and beyond (Eden, 2017;Van Quaquebeke et al., 2022). The push for experimental research in organizations is based on (i) a clear understanding of organizational phenomena as processes emerging from precipitating events, producing changes, and somehow ending, and (ii) the need to develop and refine theories in a way that allow us to establish causal relationships (Eden, 2017;Aguinis & Vandenberg, 2014). ...
In a time where new research methods are constantly being developed and science is evolving, researchers must continually educate themselves on cutting-edge methods and best practices related to their field. The second of three volumes, this Handbook provides comprehensive and up-to-date coverage of a variety of issues important in developing, designing, and collecting data to produce high-quality research efforts. First, leading scholars from around the world provide an in depth explanation of various advanced methodological techniques. In section two, chapters cover general important methodological considerations across all types of data collection. In the third section, the chapters cover self-report and behavioral measures and their considerations for use. In the fourth section, various psychological measures are covered. The final section of the handbook covers issues that directly concern qualitative data collection approaches. Throughout the book, examples and real-world research efforts from dozens of different disciplines are discussed.
... An online survey was developed using Google Forms, which was randomly distributed to banking customers through various social media platforms. This approach was chosen to facilitate data collection, ensure a broad reach, and enable easy analysis of the collected data (Teitcher et al. 2015;Van Quaquebeke et al. 2022). Convenience sampling technique was used to recruit respondents for the study. ...
As digital technologies and AI systems increasingly pervade banking services, understanding customers’ digital
capabilities and awareness of AI ethics principles is paramount. This paper delves into these critical aspects among
banking clientele. Employing a survey research approach with a cross-sectional design, an online survey via Google
Forms was distributed to 500 banking customers, of whom 466 responded. The survey evaluated respondents’ digital
competencies and AI ethics awareness through multiple-choice questions and Likert scale items. Utilizing IBM SPSS
version 28 and Microsoft 365 analysis software, descriptive statistics and statistical analysis were conducted to discern
trends and patterns. The key finding emphasizes the urgency for financial institutions to invest in digital technologies
enhancing customer experiences and fostering ethical AI adoption. A significant policy implication underscores the
need for policymakers and regulatory bodies to collaborate with the banking sector, developing clear guidelines for
responsible AI practices, prioritizing customer privacy, data protection, and fairness in AI algorithms. This study also
introduces the CORE AI framework, concentrating on responsible AI adoption, customer-centricity, and strategic
optimization, promising positive outcomes for stakeholders. The original contribution of this research lies in its
comprehensive exploration of the nexus between digital competencies and AI ethics awareness.
... To obtain a diverse sample, over 130 Facebook pages linked to Greek tourism and destinations were contacted, with a total membership exceeding one million individuals. Even though the samples from online surveys may not be fully representative, they are regarded as more advantageous than traditional offline methods in terms of face validity and internal consistency (van Quaquebeke et al., 2022). Facebook is a valuable resource for conducting surveys on a large scale since it offers a distinctive digital observatory of human behavior and allows for extensive and adaptable targeting of the sample. ...
Despite its pervasive and harmful nature, sibling violence is often trivialised as a harmless and inconsequential form of familial aggression. The primary goal of the present study was to analyse the relationship between self-reported experiences of sibling violence during childhood to examine the dynamics of this issue. A sequential explanatory mixed-method design was used for data collection. The quantitative element of the study involved 595 emerging adults, while the qualitative element examined the dynamics of sibling violence during childhood. The results of the quantitative analysis, previous literature, and theoretical constructs informed the development of a focus group discussion guide. Data obtained from the survey were analysed using thematic analysis before integrating the quantitative and qualitative findings. The results revealed that the majority of respondents were female (73.9%), aged 20 to 21 years old (49.9%), and had three to four siblings (50.3%). Most respondents reported having experienced psychological violence (57.7%) and frequently had disputes with their younger sisters (52.8%), with an average age difference of three to four years (25.9%). Additionally, the analysis showed a significant moderate relationship between self-reported experiences of aggression and sibling violence (r = .615, p < 0.01) during childhood. Through a sequential explanatory mixed method approach, this study highlights the complexity of sibling violence and makes the case for increased awareness and targeted intervention to address this form of family conflict.
Purpose
It is increasingly recognised that knowledge transfer is essential for facilitating individual ambidexterity. However, there is a lack of research exploring the relationship between knowledge inflow and individual ambidexterity, as well as the factors that may moderate this relationship. This gap leaves the understanding of this relationship incomplete. This study, therefore, aims to address this gap by investigating the moderating role of job insecurity, knowledge rivalry and proactive personality in the relationship between knowledge inflow and employee ambidexterity.
Design/methodology/approach
This study uses a cross-sectional research approach and data were collected from a sample of 205 managerial and non-managerial employees across different industries in the United Arab Emirates (UAE). The data collection period spanned three months, and the partial least squares method algorithm and bootstrapping techniques were used to test this study’s hypotheses.
Findings
The results reveal that job insecurity and knowledge rivalry moderate the relationship between knowledge inflow and employee ambidexterity. The results further reveal that a proactive personality does not moderate the knowledge inflow–employees ambidexterity relationship.
Research limitations/implications
Drawing on a knowledge-based view, self-determination theory and conservation of resource theory, this study advances the existing literature by highlighting the critical role of internal rivalry and job security in influencing the effectiveness of knowledge inflow in stimulating employee ambidextrous behaviour. This research underscores the need to consider factors beyond individual personality traits, such as organisational culture and team dynamics when fostering employee ambidexterity.
Practical implications
By implication, the findings of this research suggest that organisations should strategically manage internal knowledge by fostering a collaborative culture that addresses competition-related concerns, ensuring knowledge is viewed as a collective asset. Also, managers should promote a balance between competition and collaboration through training programs, knowledge management systems and policies that enhance job security, fostering an environment conducive to knowledge sharing and individual ambidexterity.
Social implications
By implication, organisations need to address job insecurity and knowledge rivalry proactively, recognising their detrimental effects on ambidextrous behaviour.
Originality/value
To the best of the authors’ knowledge, this is the first study to investigate the moderating role of job insecurity and knowledge rivalry in the relationship between knowledge inflow and employee ambidexterity. Furthermore, this study is among early attempts for investigating the linkage of proactive personality with knowledge inflow and employee ambidexterity, thereby providing new insights into this relationship for researchers and practitioners.
The 21st century pedagogy triggers for myriad of challenges to teachers locally and internationally as it necessitates for innovation in pedagogy to cater to the curiosity and hunger for learning of English as Foreign Language (EFL) college students. The mixed method design and methodical analysis of data unveils that due to the introduction of dramatization approach, experimental group outpaced the control group of learners statistically and intellectually. Thus, this calls for the resourcefulness of EFL teachers internationally to always be inventive and proactive during teaching-learning process (TLP). EFL teachers should not just be confined with one or two teaching approaches since 21st century leaners are more explorative, inquisitive, and interested to discover the undiscovered opportunities. EFL teachers are then recommended to be always on the lookout for new schemes in teaching to keep pace with the learning styles of the new generation. In consideration of the foregoing findings, researchers generally endorse that dramatization be exploited during TLP to develop the total aspect of learners to be better individuals becoming the stronghold and hope of each country worldwide. Nevertheless, the study limits only on catching research-based findings to the RQs given. Henceforth, future inquiry using qualitative design is the pertinent study to be conducted.
As artificial intelligence (AI) becomes increasingly integrated in teams, understanding the factors that drive trust formation between human and AI teammates becomes crucial. Yet, the emergent literature has overlooked the impact of third parties on human‐AI teaming. Drawing from social cognitive theory and human‐AI teams research, we suggest that how much a human teammate perceives an AI teammate as trustworthy, and engages in trust behaviors toward the AI, determines a focal employee's trust perceptions and behavior toward this AI teammate. Additionally, we propose these effects hinge on an employee's perceptions of trustworthiness and trust in the human teammate . We test these predictions across two studies: (1) an online experiment comprising individuals with work experience that examines perceptions of disembodied AI trustworthiness, and (2) an incentivized observational study that investigates trust behaviors toward an embodied AI. Both studies reveal that a human teammate's perceived trustworthiness of, and trust in, the AI teammate strongly predict the employee's trustworthiness perceptions and behavioral trust in the AI teammate. Furthermore, this relationship vanishes when employees perceive their human teammates as less trustworthy. These results advance our understanding of third‐party effects in human‐AI trust formation, providing organizations with insights for managing social influences in human‐AI teams.
Isu-isu Kontemporari Muslim di Malaysia (ISKOM) is a course introduced in Universiti Kuala Lumpur for its students that provides knowledge related to contemporary issues surrounding the Muslim community in Malaysia. The history and development of Islam, as well as ideologies and beliefs that influence Muslims, are also explored in this course. Modern and contemporary issues related to consumerism, family and societal institutions, science and technology, along with the future of Islam and its implications, are clarified with reference to divine revelations and current realities are also discussed in this course as an extension to what undergraduates learned prior to their university lives. This research investigates the impact of the Isu-Isu Kontemporari Muslim di Malaysia (ISKOM) subject on medical and health sciences undergraduates and aims to assess how these subject influences student's understanding of societal, cultural, and ethical issues from an Islamic perspective, and how it informs their professional development in the medical and healthcare sectors. A total of 131 students participated in the research by completing a questionnaire designed to assess their perception of the subject's impact on their academic and professional development. This study is descriptive in nature and uses a questionnaire instrument with a four-point scale that caters to all topics in the course to measure the effects of the course towards the students as future medical practitioners. The results show that the course significantly contributes to students' awareness of how Islamic teachings can guide decision-making in medical and ethical contexts. Students' responses also reflect varying opinions on contemporary issues like vaccine acceptance and cosmetic procedures, revealing a range of interpretations of Islamic principles. By highlighting these intersections, the study provides deeper insights into the diverse cultural and religious landscape of Malaysia, fostering greater appreciation and respect for different perspectives within the Muslim community.
Purpose
In the evolving digital work landscape, where cyberloafing has become a notable challenge, this study aims to investigate the mechanisms through which organizations can effectively reduce such behaviors. Specifically, the research explores the role of employee adaptability in mitigating cyberloafing, taking into account the influences of temporal leadership, teamwork attitudes, and competitive work environments.
Design/methodology/approach
Utilizing the broaden-and-build theory and the job demands-resources (JD-R) model, we analyzed data from 245 employees through structural equation modeling (SEM) to investigate how various factors influence cyberloafing.
Findings
The results indicate that employee adaptability significantly mitigates cyberloafing and serves as a mediating factor between temporal leadership, teamwork attitudes, and the impact of competitive work environments on cyberloafing. Temporal leadership and teamwork attitudes positively correlate with increased adaptability, thereby reducing cyberloafing. Conversely, competitive work environments, while slightly enhancing adaptability, substantially increase cyberloafing.
Originality/value
The study contributes new insights into the dynamics of cyberloafing, emphasizing the critical roles of adaptability, teamwork attitudes, and temporal leadership in reducing such behaviors. It underscores the need for organizations to foster a supportive culture that minimizes competitive pressures and promotes teamwork and leadership strategies conducive to high productivity and minimal cyberloafing. This research offers practical implications for designing workplace strategies aimed at boosting productivity and curbing undesirable online behaviors during work hours.
This study explores the influence of consumer emotions, specifically pleasure and arousal, on the intention to use retail service robots. Through a survey of 323 people using an online snowball sampling technique and analysis via the PLS-SEM method, it finds that subjective norms, technological self-efficacy, habitual behavior, and attitudes significantly affect users' intentions. The research notably reveals that pleasure plays a more crucial role than arousal in mediating how consumers' perceptions impact their willingness to interact with these robots. By integrating the Theory of Reasoned Action, self-efficacy theory, and the PAD model (pleasure, arousal, dominance), the study enhances the understanding of human-robot interaction. It provides actionable insights for businesses , suggesting that emphasizing emotional aspects, particularly pleasure, can be a strategic approach to encourage the adoption of service robots. These insights are valuable for designing marketing strategies and service experiences that effectively increase consumer engagement with robotic technology.
The literature has widely discussed and supported the relationship between poverty and support for authoritarian leaders and regimes. However, there are different claims about the mediating mechanism and a lack of empirical tests. We hypothesize that the effect of poverty on support for authoritarianism is mediated by shame: People living in poverty frequently experience social exclusion and devaluation, which is reflected in feelings of shame. Such shame, in turn, is likely to increase support for authoritarianism, mainly due to the promise of social re-inclusion. We support our hypothesis in two controlled experiments and a large-scale field study while empirically ruling out the two main alternative explanations offered in the literature: stress and anxiety. Finally, we discuss how the present findings can support policymakers in efficiently addressing the negative political consequences of poverty.
A key observation in coalition formation is that bargainers who control many resources are often excluded from coalitions by bargainers who control few resources, the Strength‐is‐Weakness effect. We argue that this effect is contingent on whether resources provide a legitimate claim to be included in a coalition. Across three incentivized coalition experiments (n = 2745; 915 triads), three participants (player A had four resources, player B had three resources, player C had two resources) negotiated about a payoff of 90 monetary units. Depending on condition, these resources were obtained randomly, earned, or earned and proportionally linked to the payoff. Results showed player As were less included when resources were obtained randomly and more often included in coalitions when resources were earned and/or proportionally linked to the payoff. This provides evidence that the Strength‐is‐Weakness is contingent on the legitimacy of the resources.
In today's world, which has been administered by computers and artificial intelligence in many areas, online data gathering has become an inevitable way of collecting data. Many researchers have preferred online surveying, considering the advantages of this method over the classical ones. Hence, the factors that may affect the response rate of online surveying have become a prominent research topic. In line with the popularity of this issue, the purpose of this chapter was to clarify the concept of online surveys; give information about their types, advantages, and usage; and investigate the factors that affect the participants' response behaviors. Besides the discussions on the theoretical framework of online surveying, an online survey aiming to determine the factors affecting the participation in online surveying was administered to a group of people to investigate the response behaviors thoroughly. The findings revealed that rs might affect ants' response behaviors to online surveys in various ways radically.
Web Survey Methodology guides the reader through the past fifteen years of research in web survey methodology. It both provides practical guidance on the latest techniques for collecting valid and reliable data and offers a comprehensive overview of research issues. Core topics from preparation to questionnaire design, recruitment testing to analysis and survey software are all covered in a systematic and insightful way. The reader will be exposed to key concepts and key findings in the literature, covering measurement, non-response, adjustments, paradata, and cost issues. The book also discusses the hottest research topics in survey research today, such as internet panels, virtual interviewing, mobile surveys and the integration with passive measurements, e-social sciences, mixed modes and business intelligence. The book is intended for students, practitioners, and researchers in fields such as survey and market research, psychological research, official statistics and customer satisfaction research.
The book is now available as open access since at this Sage page in PDF and Epub version https://study.sagepub.com/web-survey-methodology
The deception of research participants remains a controversial issue in the behavioral sciences. Current ethics codes consistently limit the use of deception to cases in which non-deceptive alternatives are unfeasible and, crucially, require that participants subjected to deception be debriefed correspondingly along with an option to withdraw their data after learning about the deception. These conditions pose a particular challenge in the context of web-based research because participants can typically discontinue a study unilaterally (i.e., dropout by simply closing the browser window) in which case full debriefing and an option to withdraw one’s data are no longer available. As a consequence, the study would no longer be compatible with ethical standards. Based on recent meta-analytical data, we provide an existence proof of this problem, showing that deception is used in web-based research with little to no indication of safeguards ensuring full debriefing and subsequent data withdrawal options. We close by revisiting recommendations for the (non-)use of deception in web-based research and offer solutions to implement such safeguards in case deception is truly unavoidable.
This article describes Advanced Training Institutes (ATIs) and workshops on psychological experimentation conducted via the Internet. These programs, conducted since 2002, presented instruction that evolved over the decades to reflect changes in Web-based methods and techniques. The need for instruction in the methods and methodology of Web-based research has increased over the years, as more researchers adopted these methods without necessarily learning the methodological lessons developed from theory and experience. Links to instructional materials created for the workshops are provided. From events and trends that played out over time, including the story of methods that were once state-of-the-art and are no longer functional today, lessons hard-won from the past can be used to anticipate and plan future directions in Web-based experimentation.
Several recent studies have examined the prevention, causes, and consequences of insufficient effort responding (IER) to surveys. Scientific progress in this area, however, rests on the availability of construct valid IER measures. In the current paper we describe the potential merits of the page time index, which is computed by counting the number of questionnaire pages to which a participant has responded more quickly than two seconds per item (see Huang et al., 2012). We conducted three studies (total N = 1,056) to examine the page time index's construct validity. Across these studies , we found that page time converged highly with other IER indices, that it was sensitive to an experimental manipulation warning participants to respond carefully, and that it predicted the extent to which participants were unable to recognize item content. We also found that page time's validity was superior to that of total completion time and that the two-seconds-per-item rule yielded a construct-valid page time score for items of various word lengths. Given its apparent validity, we provide practical recommendations for the use of the page time index.
We used an internet-based survey platform to conduct a cross-sectional survey regarding the impact of COVID-19 on the LGBTQ + population in the United States. While this method of data collection was quick and inexpensive, the data collected required extensive cleaning due to the infiltration of bots. Based on this experience, we provide recommendations for ensuring data integrity. Recruitment conducted between May 7 and 8, 2020 resulted in an initial sample of 1251 responses. The Qualtrics survey was disseminated via social media and professional association listservs. After noticing data discrepancies, research staff developed a rigorous data cleaning protocol. A second wave of recruitment was conducted on June 11–12, 2020 using the original recruitment methods. The five-step data cleaning protocol led to the removal of 773 (61.8%) surveys from the initial dataset, resulting in a sample of 478 participants in the first wave of data collection. The protocol led to the removal of 46 (31.9%) surveys from the second two-day wave of data collection, resulting in a sample of 98 participants in the second wave of data collection. After verifying the two-day pilot process was effective at screening for bots, the survey was reopened for a third wave of data collection resulting in a total of 709 responses, which were identified as an additional 514 (72.5%) valid participants and led to the removal of an additional 194 (27.4%) possible bots. The final analytic sample consists of 1090 participants. Although a useful and efficient research tool, especially among hard-to-reach populations, internet-based research is vulnerable to bots and mischievous responders, despite survey platforms’ built-in protections. Beyond the depletion of research funds, bot infiltration threatens data integrity and may disproportionately harm research with marginalized populations. Based on our experience, we recommend the use of strategies such as qualitative questions, duplicate demographic questions, and incentive raffles to reduce likelihood of mischievous respondents. These protections can be undertaken to ensure data integrity and facilitate research on vulnerable populations.
While managers generally seem to enjoy better mental health than regular employees, there are also plenty of reports about them suffering from burnout. The present study explores this relationship between hierarchy level and burnout in more detail. In doing so, we not only investigate what impact managerial rank may have on burnout, but we also contrast two different theoretically meaningful mediators for the relationship: sense of power (feeling in control over people) and work-related self-efficacy (feeling in control over tasks). The results of two surveys—the first with 580 managers (single-source) and the second with 154 managers matched with ratings from close others (multi-source)—show a negative relationship between managers’ hierarchy level and burnout that is explained by both mediators independently. Additional analyses reveal that high sense of power and high self-efficacy are both necessary conditions for low levels of burnout. Such fine-grained analyses allow us to understand why managers at the top are less threatened by burnout, in contrast to what some media reports suggest.
Grounded in self-regulatory resources and conservation of resources theories, the current research examines poor self-regulatory capacity as a precursor to microbreaks and their possible outcomes at work. Full-time employees completed multiple online surveys for 10 (n1 = 779 daily observations) and 5 workdays (n2 = 1,024 daily observations). In Study 1, multilevel path analysis results showed that on days when employees had poorer recovery at home (i.e., poor sleep quality), they experienced higher fatigue in the next morning (low self-regulatory capacity) and thus took microbreaks more frequently at work. In turn, their engagement in microbreaks was related to higher work engagement during the day and lower end-of-work fatigue. Furthermore, perceived health climate was found to moderate the path from morning fatigue to microbreaks. In Study 2, we replicated and confirmed the serial mediation paths found in Study 1 (poor sleep quality à morning fatigue à microbreaks à work engagement and end-of-work fatigue). Building on Study 1, Study 2 also identified microbreak autonomy as a mechanism by which perceived health climate moderates the path between morning fatigue and microbreaks (i.e., mediated moderation effect). Exploratory analyses discovered intriguing patterns of socialization microbreaks versus other microbreaks, providing further implications for the theoretical perspective. Overall, the findings support the theoretical resource perspective of microbreaks as an effective energy management strategy while at work.
The Internet has been a popular source of data amongst academic researchers for many years, and for good reason. Online data collection is fast, provides access to hard-to-reach populations, and is often less expensive than in-lab recruitment. With these benefits also come risks, such as duplicate responses or participant inattention, which can significantly reduce data quality. Very recently, researchers have become aware of another concern associated with online data collection. Bots, also known as automatic survey-takers or fraudsters, have begun infiltrating scientific surveys, largely threatening the integrity of academic research conducted online. The aim of this paper is to warn researchers of the threat posed by bots and to highlight practical strategies that can be used to detect and prevent these bots. We first discuss strategies recommended in the literature that we implemented to identify bot responses from online survey data we collected in the past six months. We then share which strategies proved to be most and least effective in detecting bots. Finally, we discuss the implications of bot-generated data for the integrity of online research and the imminent future of bots in online data collection.
Online platforms such as Amazon's Mechanical Turk (MTurk) are increasingly used by researchers to collect survey and experimental data. Yet, such platforms often represent a tumultuous terrain for both researchers and reviewers. Researchers have to navigate the complexities of obtaining representative samples from online participant cohorts, ensuring data quality, ethically incentivizing participant engagement, and maintaining transparency. Reviewers, on the other hand, have to navigate the complexities of evaluating the efficacy of such data collection and execution efforts in answering important research questions. In order to provide clarity to these issues, this article provides researchers and reviewers with a series of recommendations for effectively executing and evaluating data collection via online platforms, respectively.
The use of Amazon’s Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019. Among scholars, though, there is a mixture of excitement about the practical and logistical benefits of using MTurk and skepticism about the validity of the data. Given that the practice is rapidly increasing but scholarly opinions diverge, the Journal of Management commissioned this review and consideration of best practices. We hope the recommendations provided here will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and related fields.
The purpose of this methodological insight is to analyze the foundations of multilevel research, answering two main questions: why this methodological approach is important for management research and how to conduct a multilevel study. We examine why multilevel research is relevant, emphasizing its potential, opportunities, and basic principles. Moreover, we point out the main theoretical, methodological, and analytic aspects to be considered for an appropriate application of multilevel research. The article refers throughout to the basic literature on multilevel research, reviewing conceptual, methodological, and empirical works.
JEL CLASSIFICATION: C18; M10
This study assesses whether varying contingent financial incentives in a web panel survey altered respondent behavior. A sample of US web panel participants were randomly assigned to cash incentive levels ranging from 3.00 awarded upon completion of an online survey gauging preferences for fresh strawberries. The analytical results indicate that increased contingent incentives had no clear association with the eligibility or break-off rates. Increased incentives reduced the incidence of trap question failure among a subset of qualified respondents. Responses of respondents flagged as unqualified to complete the survey, or qualified but inattentive, consistently differed from other qualified respondents on many survey items over a range of incentive levels. Overall, the results suggest increased contingent incentives yield limited improvements in the behavior of web panel respondents.
This study aimed to evaluate the prevalence of depression, anxiety, stress, and inadequate sleeping among the public in Egypt during the novel Corona Virus Disease (COVID-19) pandemic. An online snowball sampling approach was used to collect data from netizens in four Egyptian governorates between 16 and 30 April 2020. A semi-structured questionnaire was designed to assess sociodemographic characteristics, sleeping hours per day, and psychological disturbances (depression, anxiety, and stress) of participants using the Depression Anxiety Stress Scale-21. Out of 1629 participants, 48.1% were aged ≤ 30 years, 42.4% were men, and 20.0% were working in the health sector. The participants reported a high prevalence of depression (67.1%: mild to moderate 44.6% and severe to very severe 22.5%), anxiety (53.5%: mild to moderate 30.6% and severe to very severe 22.9%), stress (48.8%: mild to moderate 33.8% and severe to very severe 15.0%), and inadequate sleeping (< 6 h/day) 23.1%. Female sex, working in sectors other than the health sector, watching/reading COVID-19 news ≥ 2 h/day, and lack of emotional support from family and society were associated with a high prevalence of severe to very severe depression, anxiety, and stress. In conclusion, the psychological impacts of COVID-19 on the public in Egypt were enormous, therefore, providing psychological support and counsel is warranted.
The risk for female students in academia of sexual harassment by male faculty and staff remains a national crisis. This study examined effects of severity of coercive sexual harassment (CSH) by a male professor of a female student on peer bystander intervention responses. A total of 180 undergraduate and graduate college students who volunteered for an online survey were randomly assigned to one of three vignette conditions that varied severity of CSH. Following Bowes-Sperry’s ethical model of bystander behaviors, it was predicted that severity of CSH would affect cognitive appraisals and emotional reactions, which would further predict intentions for intervention behaviors. Predictions generally were supported. Logistic regression analyses indicated that severity of CSH, fears of consequences, and negative emotions toward the professor were significant predictors of direct confrontation of the professor by the student observer, whereas only severity and negative emotions toward the professor were predictors of the indirect intervention (finding an excuse to remove the victim) or delegation (going to get help) while the incident was occurring. Fears of personal consequences suppressed direct intervention in the moderate condition. Findings suggest ongoing need within academia for clarification of behaviors that constitute CSH, as well as safety for bystanders who intervene.
LIONESS Lab is a free web-based platform for interactive online experiments. An intuitive, user-friendly graphical interface enables researchers to develop, test, and share experiments online, with minimal need for programming experience. LIONESS Lab provides solutions for the methodological challenges of interactive online experimentation, including ways to reduce waiting time, form groups on-the-fly, and deal with participant dropout. We highlight key features of the software, and show how it meets the challenges of conducting interactive experiments online.
This study draws on previous studies to develop the Social Networking Sites Use Multi-Motive Grid Questionnaire (SNSU-MMG) and test its reliability and validity. The results show that social networking sites use motivation includes four factors: cognitive motivation, emotional motivation, leisure motivation and herding motivation. Confirmatory factor analysis, reliability and validity tests show that the questionnaire has good structural validity, internal consistency and split-half reliability. The SNSU-MMG can be used as a measurement tool for users' social networking sites use implicit motivation.
Recent years have seen a growing number of studies investigating the accuracy of nonprobability online panels; however, response quality in nonprobability online panels has not yet received much attention. To fill this gap, we investigate response quality in a comprehensive study of seven nonprobability online panels and three probability-based online panels with identical fieldwork periods and questionnaires in Germany. Three response quality indicators typically associated with survey satisficing are assessed: straight-lining in grid questions, item nonresponse, and midpoint selection in visual design experiments. Our results show that there is significantly more straight-lining in the nonprobability online panels than in the probability-based online panels. However, contrary to our expectations, there is no generalizable difference between nonprobability online panels and probability-based online panels with respect to item nonresponse. Finally, neither respondents in nonprobability online panels nor respondents in probability-based online panels are significantly affected by the visual design of the midpoint of the answer scale.
Amazon's Mechanical Turk is widely used for data collection; however, data quality may be declining due to the use of virtual private servers to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in the existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys.
Researchers have ample reasons to take their experimental studies out of the lab and into the online wilderness. For some, it is out of necessity, due to an unforeseen laboratory closure or difficulties in recruiting on-site participants. Others want to benefit from the large and diverse online population. However, the transition from in-lab to online data acquisition is not trivial and might seem overwhelming at first. To facilitate this transition, we present an overview of actively maintained solutions for the critical components of successful online data acquisition: creating, hosting and recruiting. Our aim is to provide a brief introductory resource and discuss important considerations for researchers who are taking their first steps towards online experimentation.
Traditional survey research faces declining response rates due to changing cultural habits and technological developments. Researchers have developed novel approaches to increase respondents’ likelihood of participating in web surveys. However, we lack information about whether these methods indeed increase response rates and, if so, whether they bias the resulting data. This article focuses on the use of nonmaterial incentives in the form of a video that provides the invitees with information tailored to their life situation. Analysis of our experimental data shows that instead of increasing respondents’ probability of starting the survey, the video treatments actually decrease it. We provide evidence that the lower salience of the intrinsic benefits of survey participation in the invitation email significantly contributes to this reduction. Additionally, the effect of the nonmaterial incentive differs across subgroups, affecting nonresponse biases in line with employment status, gender, and migration background. We therefore conclude that using additional information in the form of a video as a nonmaterial survey incentive is only suitable under specific conditions.
Despite recent concerns about data quality, various academic fields rely increasingly on crowdsourced samples. Thus, the goal of this study was to systematically assess carelessness in a crowdsourced sample (N = 394) by applying various measures and detection methods. A Latent Profile Analysis revealed that 45.9% of the participants showed some form of careless behavior. Excluding these participants increased the effect size in an experiment included in the survey. Based on our findings, several recommendations of easy to apply measures for assessing data quality are given.
The day reconstruction method (DRM) is an approach to measuring well-being that is designed to approximate the rich data that can be obtained from intensive repeated measures designs like those used in the experience sampling method (ESM). Although some preliminary tests of the validity of the DRM have been conducted, these typically focus on agreement between the 2 methods at very broad levels, rather than focusing on whether the 2 methods provide similar information about the exact same moments. This article reports 2 studies that use ESM and DRM to assess the same moments. Agreement between the 2 measures varied considerably depending on the focus of the analysis. For aggregate assessments of total time spent in situations and average affect in situations, agreement was high; for between-person differences in time use and experienced affect, agreement varied across situations; and for within-person differences in both situations and affect, agreement was quite low. In addition, we found preliminary evidence that the DRM may be more influenced by expectations regarding the pleasantness of situations as compared with ESM. These results suggest that for many common purposes, the DRM does not provide the same information as ESM. (PsycInfo Database Record (c) 2020 APA, all rights reserved).
Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to • Use the scientific method to evaluate hypotheses using controlled experiments • Define key metrics and ideally an Overall Evaluation Criterion • Test for trustworthiness of the results and alert experimenters to violated assumptions • Build a scalable platform that lowers the marginal cost of experiments close to zero • Avoid pitfalls like carryover effects and Twyman's law • Understand how statistical issues play out in practice.
Given the high cost of traditional survey administration (postal mail, phone) and the limits of convenience samples such as university students, online samples offer a much welcomed alternative. Amazon Mechanical Turk (AMT) has been especially popular among academics for conducting surveys and experiments. Prior research has shown that AMT samples are not representative of the general population along some dimensions, but evidence suggests that these differences may not undermine the validity of AMT research. The authors revisit this comparison by analyzing responses to identical survey questions administered to both a U.S. national sample and AMT participants at the same time. The authors compare the two samples on sociodemographic factors, online experiences, and prosociality. The authors show that the two samples are different not just demographically but also regarding their online behaviors and standard survey measures of prosocial behaviors and attitudes. The authors discuss the implications of these findings for data collected on AMT.
Side-hustles, income-generating work performed alongside full-time jobs, are increasingly common as the gig economy provides opportunities for employees to perform supplementary work. Although scholars suggest that side-hustles conflict with full-time work performance, we assert psychological empowerment from side-hustles enriches full-time work performance. We argue that side-hustle complexity—the motivating characteristics of side-hustles—positively relates to empowerment and that side-hustle motives moderate this relationship. A study of 337 employees supports these assertions. We then investigate spillover of side-hustle empowerment to full-time work performance in a 10-day experience-sampling method study of 80 employee-coworker dyads. We address an affective pathway in which daily side-hustle empowerment enriches full-time work performance through side-hustle engagement and positive affect at work. We also consider a cognitive pathway in which side-hustle empowerment distracts from full-time work performance through side-hustle engagement and attention residue—persistent cognitions about side-hustles during full-time work. Overall, performance enrichment from side-hustles was stronger than performance conflict. We also consider affective shift from full-time work to side-hustles, finding negative affect from full-time work strengthens the relationship between side-hustle empowerment and engagement. Combined, our two studies examine the source of side-hustle empowerment and how and why side-hustle empowerment influences affective and cognitive experiences during full-time work.
We theorize that anger incited by a social movement, which has a mobilizing effect among outsider activists, might immobilize collective action intentions for institutional insiders—those sympathetic to the movement and employed by its target. We conducted initial field surveys across a spectrum of social movements, including Occupy Wall Street and #metoo, as well as those related to business sustainability and gun control, which showed that institutional insiders are often just as angry as outsider activists. But the evidence from those surveys did not show that social movement anger translated into collective action intentions among institutional insiders. We tested our theory deductively with an experiment conducted with participants who were supportive of social movement issues in their organizations. Overall, our results show that anger about a social movement issue relates to greater collective action intentions among outsider activists but not among institutional insiders. Instead of anger emboldening institutional insiders to act despite the potential costs, anger triggers fear about the potential negative consequences of collective action in the workplace, which in turn results in withdrawal. While social movements often rely on anger frames to mobilize sympathizers, our work suggests that this practice may paradoxically cause fear that immobilizes those uniquely positioned to be able to influence organizations to change.
Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.
It is becoming increasingly popular and straightforward to collect data in cognitive psychology through web-based studies. In this paper, I review issues around web-based data collection for the purpose of numerical cognition research. Provided that the desired type of data can be collected through a web-browser, such online studies offer numerous advantages over traditional forms of physical lab-based data collection, such as gathering data from larger sample sizes in shorter time-windows and easier access to non-local populations. I then present results of two replication studies that employ classical paradigms in numerical cognition research: the number-size congruity paradigm and comparison to a given standard, which also included a masked priming manipulation. In both replications, reaction times and error rates were comparable to original, physical lab-based studies. Consistent with the results of original studies, a distance effect, a congruity effect, and a priming effect were observed. Data collected online thus offers a level of reliability comparable to data collected in a physical lab when it comes to questions in numerical cognition.
International humanitarian organizations (IHOs) always strive to improve their operational performance in the field. While anecdotes from practice suggest that IHO field office leadership plays a crucial role in this regard, these claims have not been deeply substantiated by primary data. In response, we collected survey data from 125 humanitarian workers, concentrated in disaster response and development programs, on the issues of field office leadership and operational performance. Building on the operations management and organizational behavior literature, we find that leaders who adopt an intergroup leadership style can significantly improve operational performance via enhancing cooperation between local and expatriate subgroups inside a field office. Notably, we find that the intergroup leadership style becomes more effective as humanitarian workers become more entrenched within cohesive subgroups. These results should help IHOs to better select and train their field office leaders and achieve higher operational performance. This article is protected by copyright. All rights reserved.
Using and extending career construction theory (CCT), this study disentangled parallel and attenuating relationships between career adaptability and proactive career behaviours on both the interindividual and intraindividual levels. Using a large heterogeneous sample of working adults, univariate and bivariate latent growth curve analyses were conducted (three measurement points, 9‐month time span). On the one hand, career adaptability and proactive career behaviours showed positive relations of initial interindividual levels. Furthermore, intraindividual changes in career adaptability and proactive career behaviours were positively related, pointing towards a parallel development. On the other hand, as predicted, the initial level of career adaptability was negatively related to the slope of developmental trajectories of proactive career behaviours. In other words, because the average growth rate for proactive career behaviours was negative, this means that individuals with higher initial career adaptability showed stronger decreases in proactive career behaviours. Finally, only the intraindividual developments of career adaptability, but not of proactive career behaviours, positively predicted higher levels of career satisfaction. These findings point towards diverse developmental relationships between career adaptability and proactive career behaviours. Results are discussed against the background of CCT and successful career development.
Practitioner Points
Organizational and individual career management should account for diverse interdependencies of career adaptability and proactive career behaviour developments.
Career counselling that fosters career adaptability goes along with proactive career behaviour enhancement, and vice versa, within the same clients.
Higher levels of initial career adaptability accelerate the decrease in proactive career behaviours within the same persons.
Career adaptability should be increased within clients and employees because this increase predicts higher levels of career satisfaction.
Amazon Mechanical Turk (AMT) offers a relatively low-cost alternative to traditional expensive survey samples, which likely explains its popularity among survey researchers. An important question about using such samples is whether they are representative of the larger Internet user population. Though prior research has addressed this question about demographic characteristics, little work has examined how AMT workers compare with others regarding their online activities—namely, social media experiences and online active engagement. This article analyzes survey data administered concurrently on an AMT and a national sample of U.S. adults to show that AMT workers are significantly more likely to use numerous social media, from Twitter to Pinterest and Reddit, as well as have significantly more experiences contributing their own online content, from posting videos to participating in various online forums and signing online petitions. The article discusses the implications of these findings for research that uses AMT as a sampling frame when examining questions related to social media use and active online engagement.
Background
The development of the web survey has led to significant strides in questionnaire survey methodology, including its potential to substantially increase sample sizes at minimal costs. Despite its advantages, web surveys typically achieve lower response rates from participants compared to more conventional survey methods.
Objective
The aim of this review was to evaluate strategies to increase the response rate to web surveys.
Methods
CINAHL (EBSCO), MEDLINE Complete, the Cochrane database of systematic reviews, the Cochrane central register of controlled trials and Psych Info were searched, from inception to the 24th of June, 2021. The Boolean search phrase (Ti: Web survey* OR online survey* OR internet survey*) AND (Ti: response rate* OR nonresponse* or participation rate*) was used. This was supplemented by a secondary search of the reference lists. To be eligible for inclusion in the review, papers had to evaluate one or more strategies to improve response rates to web surveys. Experimental and quasi experimental studies were included in the review.
Results
A total of 159 papers were identified. Following removal of duplicates, and further screening by two independent reviewers, 45 papers met the inclusion and exclusion criteria, and were included in the final review. The use of e-mail pre-notification, email invitation and two reminders were found to increase response rates to web surveys as do the use of a semi-automatic log-in, a simple design and a short survey which takes between 10 and 15 minutes to complete. Incentives, including entry into a prize draw with a cash prize considered to be of value to the participants also increase response rates.
Conclusion
Research studies are needed to explore whether the different strategies used by researchers with the intent to improve response rates are acceptable to potential participants and to evaluate the potential synergistic effect of combinations of several strategies identified in this review.
Tweetable abstract: email prenotification, email invitation, 2 reminders, simple 10 minute design and lottery incentives improve response rates to web surveys
The “credibility revolution” has fueled a number of initiatives to help bring scientific practices more in line with scientific ideals. These initiatives include increasing the sample size of studies, making data and materials publicly available, pre-registering data collection and analysis plans, publishing replication attempts, and publishing null results. To what extent have these practices become the norm in quantitative Organizational Behavior research? In the current study, using computer algorithms and human coders, we coded the reported use of several open science and reform practices in articles published in four prominent journals (Academy of Management Journal; Journal of Applied Psychology; Organizational Behavior and Human Decision Processes; and Organization Science) from 2011 through 2019. We found that although the vast majority of articles did not use any open science practices, some practices we coded were on the rise, especially in the last two to three years. While there is much room for improvement, these results suggest the field could be on the brink of important and sustained change.
Background:
Recruitment of people with disabilities often occurs through disability organizations, advocacy groups, service providers, and patient registries. Recruitment that relies exclusively on established relationships can produce samples that may miss important information. The MTurk online marketplace offers a convenient option for recruitment.
Objective:
The paper compares samples recruited through (1) conventional and (2) MTurk methods to better understand how these samples contrast with one another and with national estimates of people with disabilities.
Methods:
In 2019, researchers recruited 1374 participants through conventional methods and 758 through MTurk to complete the National Survey on Health and Disability (NSHD). We analyzed sample differences between recruitment groups with t-tests, Chi-square, and logistic regression.
Results:
With the exception of race/ethnicity, the conventional and MTurk samples were significantly different on several dimensions including age, gender, education level, marital status, children living at home, and sexual orientation. The MTurk sample was overrepresented in lower income brackets. A significantly higher percentage of the conventional sample received SSI, SSDI, or both, compared to the MTurk sample (36.2% vs 12.8%) and had significantly higher rates of insurance coverage. Comparisons with American Community Survey data show that the conventional and MTurk samples aligned more closely with the general population of people with disabilities on different characteristics.
Conclusions:
MTurk is a viable complement to conventional recruitment methods, but it should not be a replacement. A combination of strategies builds a more robust dataset that allows for more varied examination of issues relevant to people with disabilities.
Studies on abusive supervision typically posit that targets of abuse will either directly blame the perpetrating supervisor or indirectly blame the organization for allowing the abuse, and as a result reduce their cooperativeness at work. We pivot from this predominant logic and argue that, under certain circumstances, targets of abusive supervision may blame themselves, feel guilty, and then try to make it up to their abusive supervisors by helping them more. Drawing on the emotional process theory of abusive supervision and the more general socio-functional perspective of emotions, we specify that such a dynamic is more likely to ensue when subordinates otherwise experience the relationship with their supervisors as good (high LMX). Two studies—an experiment and a two-weeks bi-daily experience sampling study—provide support for our reasoning. As such, our study extends theorizing on the consequences of abusive supervision, which has typically found that it reduces cooperative behaviors. Moreover, it contributes to previous speculations that leaders may engage in abusive supervision because it has beneficial consequences for them.
Paradata are a valuable source of information to better understand the response process and to determine how the data collection processes can be improved. Currently, a debate is ongoing in academia and in the professional associations of market, opinion, and social research concerning the need to inform respondents about the collection and use of their web paradata. Unfortunately, research is lacking on the design of informed consent for web paradata use and its possible consequences for respondents’ willingness to share their paradata, completion of a survey, and response behavior. The aim of this study is to provide guidance to the survey practitioner on the optimal design of informed consent for web paradata use based on experimental evidence.
Employee surveys and privacy issues have become increasingly complex with the movement toward identified surveys, integrated databases, expanded analytics, the advancement of technology, and changing regulations. The purpose of this chapter is to review and provide recommendations on data privacy and ethical considerations related to employee attitude measurement, including new technologies. This is followed by a review of the ethical considerations related to employee surveys, including the ever-changing legal landscape. The authors then summarize select new technologies that collect employee attitude data and discuss the challenges they pose. They close with recommendations for the use of employee surveys and evolving technologies.
Header images are typically included in web surveys to make surveys more appealing for respondents. However, headers might also induce a systematic bias in response behavior. In ordertoexamineboththepotentialeffects(morespecifically,effectsonmotivationandcontext effects) of header images with respondents using different devices, an experiment embedded in a web survey on students’ time use and stress was conducted using a probability sample of 1,326 students at the University of Bonn. Respondents were presented either with a picture of an auditorium with students sitting in a class, a picture of leisure activities on campus, or no picture, respectively. To control for position effects, pictures were placed either in the upper rightorupperleftofthequestionnaire. Theresultsindicatethatheaderimagesattractattention in the beginning of a survey, but do not significantly increase motivation over the course of the survey. When faced with a header picture, respondents in the picture conditions evaluate their time in class differently compared to respondents in the control group. While the device providing the visibility makes no difference, effects are only significant when the picture is placed on the left side of the screen. In sum, the interaction of header placement and the content-related proximity of header content and question may alter response behavior.
We analyse the effect of different incentive structures on response rates for an online survey of New Zealand landowners who have previously not responded to an earlier wave of the survey. We find response rates are lowest for direct cash payment, lower even than for a control group with no incentive, which may be due to direct payment extinguishing any warm glow people receive from the charitable act of completing a survey. Lottery and charitable donation incentives do not increase response rates relative to a control group with no incentive.
Experience sampling method (ESM) and ecological momentary assessment (EMA) refer to a longitudinal research approach in which investigators collect self-reports and/or observational data at recurring intervals about study participants' everyday activities, affect, physical, and psychological states. Advances in Internet-enabled technology, along with the ubiquity of smartphone use among college populations, have vastly increased the feasibility of ESM/EMA research for higher education researchers. The chapter uses examples from higher education research to describe how existing and emerging technological affordances, including automated sensing, can be used to signal participant experience reports and to collect new forms of digital, written, visual, oral, and physiological data. The authors discuss problems and controversies in ESM/EMA research, present potential solutions to these issues, and consider the future of this evolving family of methods for higher education researchers.
Aims:
To examine prevalence of bullying among nurses and explore associations of organizational betrayal and support with well-being among nurses exposed to bullying.
Background:
Bullying is a problem in many nursing workplaces, and organizations have an obligation to support nurses who are bullied. Support or betrayal after bullying could affect nurse well-being, including burnout.
Methods:
In this cross-sectional study among U.S. nurses, data were collected in a survey using the Negative Acts Questionnaire-Revised for Nursing, the Institutional Betrayal Questionnaire for Health, the Well-Being Index, a job satisfaction scale and demographic questions.
Results:
Prevalence of weekly/daily bullying was 31% (N = 242). Among nurses exposed to any bullying (N = 173), organizational betrayal increased odds of burnout (OR 2.62, p=.02), job dissatisfaction (OR 2.97, p=.04), and absenteeism (OR 6.11, p<.001). Organizational support decreased odds of job dissatisfaction (OR 0.30, p=.001) and absenteeism (OR 0.50, p=.04).
Conclusion:
Analysis of study findings suggests organizational betrayal increases likelihood of burnout, job dissatisfaction, and absenteeism, and support decreases likelihood of dissatisfaction and absenteeism.
Implications for nursing management:
Nurse leaders should be aware of the issue of organizational betrayal and support in relation to well-being. Future studies can further explore the concepts of betrayal and support to provide additional evidence.
The aim of this article is to undertake a systematic literature review (SLR) of empirical research that uses multi‐source methods for collecting data about Ethical Leadership (EL). Research on this sensitive subject benefits from the inclusion of data from more than one source, in order to be better supported, and thus contribute to a deeper understanding of leadership and business ethics issues. The search strategy retrieved a total of 50 multi‐source empirical studies on the topic of EL, published until December 2017. This SLR shows that (a) research on EL has focused mostly on the perceptions of followers, possibly because they are the most accessible target of its outcomes, thus restricting the scope of this body of research; (b) EL is studied mainly through consideration of its consequences, a restriction that hinders explanation of the causal processes involved in ethical leadership, which remains a research arena in need of development. The systematic inclusion of other stakeholders in multi‐source methods is advanced as a way to further develop the field.
We develop and analyze empirical Bayes Stein-type estimators for use in the estimation of causal effects in large-scale online experiments. While online experiments are generally thought to be distinguished by their large sample size, we focus on the multiplicity of treatment groups. The typical analysis practice is to use simple differences-in-means (perhaps with covariate adjustment) as if all treatment arms were independent. In this work we develop consistent, small bias, shrinkage estimators for this setting. In addition to achieving lower mean squared error these estimators retain important frequentist properties such as coverage under most reasonable scenarios. Modern sequential methods of experimentation and optimization such as multi-armed bandit optimization (where treatment allocations adapt over time to prior responses) benefit from the use of our shrinkage estimators. Exploration under empirical Bayes focuses more efficiently on near-optimal arms, improving the resulting decisions made under uncertainty. We demonstrate these properties by examining seventeen routine experiments conducted on Facebook from April to June 2017.
The present study is the first to examine empirically whether required fields in online surveys impair reliability and response pattern, as participants forced to respond to all items may provide arbitrary answers. Two hundred and thirteen participants completed a survey consisting of six questionnaires testing personal and social issues and perceptions. They were randomly assigned to one of two versions of the survey: optional-fields (N = 104) or required-fields (N = 109). Comparison of the Cronbach’s alpha of the two versions revealed identical reliability values for all questionnaires, save for somatization, where a minor difference was found. Confirmatory factor analysis showed no difference in the factor structure of the two versions, and no differences were found by Bayesian t-test and Levene’s test for equality of variances. The findings suggest that required fields do not impair reliability or change the response pattern, and therefore can be used in online surveys to prevent missing data.