Cross-National Issues in Response Rates
Abstract and Figures
After decades of neglecting nonprobability sampling approaches (we still have no textbook on this widespread practice), in recent years there has finally been a breakthrough in academic attention to this approach. This includes formal professional acceptance (e.g. American Association for Public Opinion Research (AAPOR) code) and also increased scientific research attention. This chapter overviews the structure and trends of research conducted on response rates between 1990 and 2015. The overview indicates – mirroring the trend in existing published work – that nonresponse is rarely treated in a comprehensive and integrative manner. The chapter also highlights those research projects where the nonresponse rate, nonresponse bias, data quality and costs are examined simultaneously, using the European Social Survey as an example.
Figures - uploaded by Vasja Vehovar
Author content
All figure content in this area was uploaded by Vasja Vehovar
Content may be subject to copyright.
... Vehovar (2017) suggests a 50% response rate as acceptable in social science research, making this study's 70.73% rate highly reliable [13]. Wiseman (1984) quoted from the Australian Vice Chancellors' Committee & Graduated Careers Council of Australia (2001) which states that an acceptable response rate for the Course Experience Questionnaire (CEQ) is approximately 60% [14]. ...
Background. This research aims to enhance the health and well-being of the Universitas Negeri Yogyakarta (UNY) academic community during the transition period from Badan Layanan Umum (BLU) to Perguruan Tinggi Negeri Badan Hukum (PTNBH), a phase marked by significant changes. Recognizing the importance of this transition, the study focuses on developing an instrument to measure health and happiness at UNY. Methods. This study employs a descriptive quantitative approach using a survey method, incorporating the MEASURE and PLS-SEM model methodologies with the assistance of the SmartPLS 4 application. The MEASURE approach was selected due to its suitability for instrument development, while PLS-SEM was chosen for its flexibility in constructing diverse models. A total of 84 respondents participated in the study, comprising UNY officials and staff directly involved in the campus planning process. Results. The findings indicate that the UNY wellness matrix framework consists of four key aspects: social, economic, environmental, and knowledge management aspects. Conversely, the risk management aspect was found to have no significant impact on the UNY wellness matrix and was therefore excluded from the framework. Conclusion. The four independent variables identified in this study include social, economic, environmental, and knowledge management aspects. The dependent variable is the ultimate goal of the wellness matrix—enhancing health and well-being across the entire UNY academic community.
... This research can be classified as descriptive research which aims to explain the current set of circumstances, and cross-sectional research which concentrated on collecting data from research participants at a single point of time (Johnson, 2001;Moutinho, 2011). Most cross-sectional studies collect their data using questionnaires, and they could be distributed to the targeted respondents by mail, e-mail, or face-to-face and telephone contacts (Gray, Williamson, Karp, & Dalphin, 2007;Rowley, 2014;Vehovar & Beullens, 2018). This research used self-administrated mail questionnaire technique, where the questionnaire is mailed to a wide range of individuals and willing respondents can complete the survey at their leisure and send it back (Bhattacherjee, 2012). ...
Purpose - This paper researches export marketing strategy (EMS) archetypes of agri-food exporters, and organizational determinants that pose as their antecedent factors, using resource-based, dynamic capabilities, and contingency theories as theoretical framework in a multi-county research setting. A twofold objective is specified - to explore hitherto used EMS and to examine differences between agri-food exporters based on organizational determinants.
Design/methodology/approach - Comparative research design, quantitative methodology, an etic/emic approach, descriptive and causal data analyses were employed. EMS archetypes were portrayed on radial plots, while six hypotheses were tested using MANOVA.
Findings - The Tactical Coordinator was identified as a universal EMS archetype. Diversity of archetypes was found as results of the effect of organizational determinants, confirming their ambivalent impacts rooted in the resources, capabilities, and contingencies exporters have to face.
Research limitations/implications - Main limitations arise from the sample choice in international business, concentration only on organizational determinants, survey as a data collection technique, and reliance on self-report data from managers. Nevertheless, several theoretical and practical implications are defined.
Originality/value - To our knowledge, this paper is the first to apply the EMS archetype perspective to the agri-food industry in a developing country context and in an economic crisis context. Its multi-theory approach is supplemented with researched countries' national culture perspective and institutional background to extend understanding of agri-food firms' EMS archetypes.
https://www.emerald.com/insight/content/doi/10.1108/BFJ-03-2021-0217/full/html
Lower data collection costs make web surveys a promising alternative to conventional face-to-face and telephone surveys. A transition to the new mode has already been widely initiated in commercial research, but web surveys remains limited in academic and official research projects that typically require probability samples and high response rates. Various design approaches for coping with the problems of sampling frames, incomplete Internet use, and nonresponse in web surveys have been proposed. Mixed-mode designs and incentives are two common strategies to reach Internet non-users and increase the response rates in web surveys. However, such survey designs can substantially increase the costs, the complexity of administration and the possibility of uncontrolled measurement effects. This paper presents and demonstrates an approach to the evaluation of various survey designs with simultaneous consideration of the errors and costs. It focuses on the designs involving the web mode and discusses their potential to replace traditional modes for probability surveys of the general population. The main idea of this approach is that part of the cost savings enabled by the web mode can be allocated to incentives and complementary survey modes to compensate for the Internet non-coverage and the higher nonresponse. The described approach is demonstrated in an experimental case study that compares the performance of mixed-mode designs with the web mode and prepaid cash incentive with that of an official survey conducted using the face-to-face and telephone modes. The results show that the mixed-mode designs with the web mode and incentives can greatly increase the response rate, which even surpasses that of the conventional survey modes, but still offer substantial cost savings. However, the results also show that higher response rate does not necessary translate to higher data quality, especially when the main aim is to obtain estimates that are highly comparable with those of the reference survey.
This paper is focused on process and output aspects of the obtained sample and deals with the measurement of non-response, and the study of non-response bias from a viewpoint of comparative research in which the concept of “equivalence” in measurement is central (Jowell et al., 2007). The paper starts with a theoretical reflection on several designs for the detection of non-response bias: comparing sample statistics with population statistics; using information from reluctant respondents based on converted refusals; asking a small set of crucial questions at occasion of first contact (and refusal) or in a period after the main survey, and collecting observed information of the house and neighbourhood of the sampling units. Each of these methods are used in the past three round of ESS, but only the first and second approaches are fully documented for Rounds 1 and 2 till now. Problems related to each of these methods are considered, and the application of each of the procedures is (empirically) evaluated using information of past ESS surveys as far as the data are available. Methods that can be used for data based adjustment of the sample measures are considered.
Survey researchers increasingly use mixed-mode surveys for general population data collection because mixed-mode surveys are argued to provide lower selection error at constant budgets, or lower variable costs at constant selection error. Nevertheless, the advantage of lower selection error and variable costs might be counteracted by higher measurement error and higher fixed costs. This trade-off between selection error, measurement error, variable costs, and fixed costs has hardly been studied within the existing literature about mixed-mode surveys. A possible procedure for evaluating this trade-off is discussed in this paper by comparing the performance (mean squared error) of mixed-mode survey designs against single-mode survey designs. The procedure is further illustrated by real example data stemming from a mixed-mode mailface-toface survey. This illustration yields smaller errors for single-mode designs under low budgets but smaller errors for mixed-mode designs under large budgets or, alternatively, a budgetary advantage of single-mode designs when the allowed error is relatively high but a budgetary advantage of mixed-mode designs when the allowed error is relatively small. However, the validity of these results depend on several modeling assumptions which may be topics for future research.
The pursuit of high response rates to minimise the threat of nonresponse bias continues to dominate decisions about resource allocation in survey research. Yet a growing body of research has begun to question this practice because the cost of efforts to increase response rates is not always justified in terms of their impact on nonresponse bias. In this study, we assess the costs and benefits of different methods designed to increase response rates on the European Social Survey in Switzerland, by analysing data from a new sampling frame based on population registers to examine the changing composition of the respondent pool and the risk of bias, alongside the financial costs associated with additional fieldwork efforts. We compute an R-indicator to assess representativity with respect to the sampling register variables, and find little improvement in the sample composition as response rates increase. We then examine the impact of response rate increases on the risk of nonresponse bias based on Maximal Absolute Bias (MAB), and coefficients of variation between subgroup response rates, alongside the associated costs of different types of fieldwork effort. The results show that increases in response rate help to reduce MAB, while only small but important improvements to sample representativity are gained by varying the type of effort. The findings lend further support to research that has called into question the value of extensive investment in procedures aimed at reaching response rate targets and the need for more tailored fieldwork strategies aimed both at reducing survey costs and minimising the risk of bias.
One question that arises when discussing the usefulness of web-based surveys is whether they gain the same response rates compared to other modes of collecting survey data. A common perception exists that, in general, web survey response rates are considerably lower. However, such unsystematic anecdotal evidence could be misleading and does not provide any useful quantitative estimate. Metaanalytic procedures synthesising controlled experimental mode comparisons could give accurate answers but, to the best of the authors' knowledge, such research syntheses have so far not been conducted. To overcome this gap, the authors have conducted a meta-analysis of 45 published and unpublished experimental comparisons between web and other survey modes. On average, web surveys yield an 11% lower response rate compared to other modes (the 95% confidence interval is confined by 15% and 6% to the disadvantage of the web mode). This response rate difference to the disadvantage of the web mode is systematically influenced by the sample recruitment base (a smaller difference for panel members as compared to one-time respondents), the solicitation mode chosen for web surveys (a greater difference for postal mail solicitation compared to email) and the number of contacts (the more contacts, the larger the difference in response rates between modes). No significant influence on response rate differences can be revealed for the type of mode web surveys are compared to, the type of target population, the type of sponsorship, whether or not incentives were offered, and the year the studies were conducted. Practical implications are discussed.
Probability-based online panels are beginning to replace traditional survey modes for existing established surveys in Europe and the United States. In light of this, current standards for panel response rate calculations are herein reviewed. To populate these panels cost-effectively, more diverse recruitment methods, such as, mail, telephone, and recruitment modules added to existing surveys are being used, either alone or in combinations. This results in panel member cohorts from different modes complicating panel response rate calculations. Also, as a panel ages with inevitable attrition, multiple cohorts result from panel refreshment and growth strategies. Formulas are presented to illustrate how to handle multiple cohorts for panel metrics. Additionally, drawing on relevant metrics used for a panel response rate, we further demonstrate a computational tool to assist planners in building a probability-based panel. This provides a means to estimate the recruitment effort required to build a panel of a predetermined size.
This chapter focuses on the response rates achieved in the European Social Survey (ESS) and the prevalence of the basic types of nonresponse. In order to obtain such a final outcome code for every individual sampling unit in the ESS, the data documented in the contact forms had to be transformed in two respects. Firstly, the results of the various contact attempts (or calls) for each unit had to be combined to obtain a single final outcome code for each sampling unit. Secondly, in addition to combining the outcome codes from the different calls into one final code, the detailed coding scheme from the ESS contact forms also had to be recoded into the three broad categories of noncontact, refusal and ‘not able/other’. The relationship between efforts and response rates becomes even more equivocal if the results for ESS 2 and ESS 3 are taken into account. non-response bias
The paper reviews the growing literature on responsive and adaptive designs for surveys. These designs encompass various methods for managing data collection, including front loading potentially difficult cases, tailoring data collection strategies to different subgroups, prioritizing effort according to estimated response propensities, imposing stop rules for ending data collection, monitoring key survey estimates throughout the field period, using two-phase or multiphase sampling for following up non-respondents and calculating indicators of non-response bias (such as the R-indicator) other than response rates to monitor and guide fieldwork. We give particular attention to efforts to evaluate these strategies experimentally or via simulations. Although the field seems to have embraced these new tools, most of the evaluation studies suggest they produce marginal reductions in cost and non-response bias. It is clearly difficult to lower survey costs without reducing some aspect of survey quality. Other issues limiting the effectiveness of these designs include weakly predictive auxiliary variables, ineffective interventions and slippage in the implementation of interventions in the field. These problems are not, however, unique to responsive or adaptive design. We give recommendations for improving such designs and for improving the management of data collection efforts in the current difficult environment for surveys.
Increasing respondent contact problems and decreasing respondent willingness to cooperate have contributed to declining response rates in general population surveys, which has raised concerns of survey accuracy. To counteract nonresponse, several methods have been employed, including incentives, advanced letters, alternative survey modes for reluctant respondents, and increased field efforts to contact potential respondents. In particular, the number of contact attempts has been increased for many surveys. Even though more contact attempts increase survey costs, they are a reliable means for increasing response rates. However, the assumption that high response rates foster data quality and smaller nonresponse bias has been challenged. In this paper, we used contact data from the European Social Survey for Norway, Finland and Slovenia to see whether or not additional contact attempts resulting in a higher response rate can potentially reduce nonresponse bias.
