No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Respondent Incentives in a National Face-to-face Survey: Do They Affect Response Quality?
Abstract
Respondent incentives are a popular instrument to achieve higher response rates in surveys. However, the use of incentives is still a controversial topic in the methodological literature with regard to the possible reduction or increase in response quality. We conducted an experiment in a large-scale German face-to-face study in which the treatment group was promised a modest monetary incentive. We used different indicators of response quality and compared the incentivized group with the control group. Our results indicate that in general there are no systematic differences between the incentivized and the control group concerning response quality. We found some hints that specific subgroups react differently to incentives in terms of response behavior. While response quality usually tends to be lower for older respondents, we found that in the incentivized group the response quality is higher for older respondents as compared to younger ones regarding the level of item nonresponse.
... High quality is associated with answers reflecting the respondents' actual substantive positions with little to no distortion. Response tendencies such as acquiescence, extreme responding, non-differentiation, saying 'no opinion ', 'don't know' or refusing to answer altogether, as well as 'speeding' through the questionnaire and ignoring written instructions, in contrast, are interpreted as symptoms of low response quality (Grauenhorst et al. 2016;Kieruj and Moors 2013;Revilla 2016). ...
... Nevertheless, existing studies mainly assess one or several singular indicators of response quality (e.g. Deutskens et al. 2004;Grauenhorst et al. 2016), sometimes combined to create an index (e.g. Revilla 2016), thus assuming the existence of a singular underlying dimension of response quality. ...
When analyzing survey data, response quality has consequential implications for substantial conclusions. Differences in response quality are usually explained by personality, or socio-demographic or cognitive characteristics. Little, however, is known about how respondents’ political attitudes, values, and opinions impact on quality aspects. This is a striking analytical omission, as potential associations between political values and various forms of response biases and artefacts call into question surveys’ ability to represent ‘public opinion’. In this contribution, response quality is traced back to respondents’ political and ideological dispositions. For this purpose, a relational understanding of response quality is applied that takes into account different aspects of response behaviors, as well as the interrelations between these indicators. Using data from the US General Social Survey (2010–2014), an empirical typology of response quality is created via finite mixture analysis. The resulting classes are then related to positions in the US field of ideological dispositions, constructed via multiple correspondence analysis. The analyses reveal that there are (1) different combinations of response patterns and thus different empirical response types, and (2) that these types of response quality systematically vary with regard to the respondents’ political and ideological (dis)positions. Implications of the findings for public opinion surveys are discussed.
... Most responses were short with only a few cases being longer than 70 words. Previous research refers to the length and the interpretability of textual responses as an indicator of their quality (Grauenhorst et al., 2016;Mavletova, 2013;Schmidt et al., 2020). The response length in our study corresponds roughly to previous studies investigating response quality through response length (Meitinger et al., 2021;Schmidt et al., 2020). ...
The increasing use of algorithms in allocating resources and services in both private industry and public administration has sparked discussions about their consequences for inequality and fairness in contemporary societies. Previous research has shown that the use of automated decision-making (ADM) tools in high-stakes scenarios like the legal justice system might lead to adverse societal outcomes, such as systematic discrimination. Scholars have since proposed a variety of metrics to counteract and mitigate biases in ADM processes. While these metrics focus on technical fairness notions, they do not consider how members of the public, as most affected subjects by algorithmic decisions, perceive fairness in ADM. To shed light on subjective fairness perceptions of individuals, this study analyzes individuals’ answers to open-ended fairness questions about hypothetical ADM scenarios that were embedded in the German Internet Panel (Wave 54, July 2021), a probability-based longitudinal online survey. Respondents evaluated the fairness of vignettes describing the use of ADM tools across different contexts. Subsequently, they explained their fairness evaluation providing a textual answer. Using qualitative content analysis, we inductively coded those answers (N = 3697). Based on their individual understanding of fairness, respondents addressed a wide range of aspects related to fairness in ADM which is reflected in the 23 codes we identified. We subsumed those codes under four overarching themes: Human elements in decision-making, Shortcomings of the data, Social impact of AI, and Properties of AI. Our codes and themes provide a valuable resource for understanding which factors influence public fairness perceptions about ADM.
... Dla przykładu można wskazać opracowania metodologiczne stwierdzające istotny wpływ techniki badawczej na wskaźniki realizacji próby(Bethlehem, Cobben, Schouten 2011;Krzewińska, Grzeszkiewicz-Radulska 2013), czy też prace potwierdzające dodatnią korelację pomiędzy długością terenowej fazy badania oraz poziomem realizacji próby(Sztabinski, Sztabinski, Przybysz 2009;Jabkowski 2011;Fuchs, Bossert, Stukowski 2013). W literaturze podkreśla się także pozytywny wpływ wysyłania listów zapowiednich(Biemer, Lyberg 2003; Vonder Lippe, Schmich, Lange 2011), stosowania gratyfikacji(Grauenhorst, Blohm, Koch 2016), konwersji odmów (Sto-op i in. 2010), czy też procedur kontrolnych(Kohler 2007) oraz negatywne oddziaływanie dokonywania substytucji niedostępnych respondentów osobami o podobnych cechach(Elliot 1993;Lynn 2004;Menold 2014; Muñoz-Conde i in. ...
... The presence or absence of these fieldwork procedures can influence the quality of the data and the survey outcome rates, as shown by methodological studies discussing the mode effect (Bethlehem, Cobben, & Schouten, 2011), the negative correlation between the length of fieldwork and the fraction of nonresponse units (Vandenplas, Loosveldt, & Beullens, 2015), as well as the positive impact on the measurement quality of: advance letters (von der Lippe, Schmich, Lange, & Koch, 2011), incentives (Grauenhorst, Blohm, & Koch, 2016), refusal conversion (Stoop et al., 2010), back-checking procedures (Kohler, 2007) and the negative impact of fieldwork substitutions on nonresponse bias (Elliot, 1993). ...
This article addresses the comparability of sampling and fieldwork with an analysis of methodological data describing 1,537 national surveys from five major comparative cross-national survey projects in Europe carried out in the period from 1981 to 2017. We describe the variation in the quality of the survey documentation, and in the survey methodologies themselves, focusing on survey procedures with respect to: 1) sampling frames, 2) types of survey samples and sampling designs, 3) within-household selection of target persons in address-based samples, 4) fieldwork execution and 5) fieldwork outcome rates. Our results show substantial differences in sample designs and fieldwork procedures across survey projects, as well as changes within projects over time. This variation invites caution when selecting data for analysis. We conclude with recommendations regarding the use of information about the survey process to select existing survey data for comparative analyses.
... The reasons examined in this study support the proposition that Qatari self-reported cooperation rationales are consistent with participation theories typically cited in literature reviews. In countries where survey research has reached maturity or saturation which pushes down participation, experiments with the use of incentives to prompt participation have been reported as effective in boosting participation to a limited extent without significantly changing the composition of the sample (Grauenhorst, Blohm and Koch, 2015). This is a hopeful finding for the future should it become necessary. ...
In this paper, we assess respondent participation in academic public opinion surveys in the State of Qatar. Rates of participation in SESRI's face-to-face surveys remain at levels that far exceed those observed in Western and even other Middle Eastern countries and require no incentives. An examination of respondents' self-reported reasons for participation accords with major theories on participation in survey research. Subgroup analysis found the main distinction primarily on "national loyalty" as a participation motivator which was higher among older, better educated, and city-dwelling Qataris. Among younger Qataris, "expressing an opinion" was a significantly higher motivator than among older. At this time results indicate no reason to explore using incentives in the immediate future.
... There were 288 postcards returned as undeliverable yielding a response rate of 11.8%. Offering respondents a chance to win a small prize is likely to encourage rather than coerce participation, thus increasing the response rate with no significant negative effects on sample bias or response accuracy (Grauenhorst, Blohm, and Koch 2015;Pedersen and Nielsen 2016;Singer and Couper 2008). After eliminating respondents under the age of twenty-five, the final sample of US adults was 1,306. ...
We empirically examined citizen views on climate change in China and the United States, including an analysis of factors affecting public support for joining an international climate change treaty. We conducted face-to-face intercept surveys of Chinese adults (N=2,047) and online surveys of American adults (N=1,306) between September and November 2013. Overall, Chinese adults were more likely to accept the realities of anthropogenic climate change. Although a supermajority of respondents in both countries support an international climate treaty, support for such action was significantly greater among Chinese respondents compared to Americans. Additionally, a variety of variables reflecting climate change views, including awareness and understanding, were positively correlated with support for a climate treaty in both countries. All else being equal, we also found that more exposure to media content on climate change increased support for a treaty. Lastly, political affiliation was found to have an important influence on support for an international climate treaty in the United States.
... Despite MTurk's reputation as a commercial enterprise, designed to facilitate economic or market transactions between parties, higher financial incentives alone did not significantly increase response rates. These results mirror previous research that has explored how incentives influence other modes of survey response, like face-to-face, email, and telephone (Auspurg and Schneck, 2014;Boulianne, 2013;Grauenhorst et al., 2016;Singer et al., 2000), and extend to a new and increasingly popular means of data collection. Instead, this research suggests that granting bonus payments-which represent gestures of goodwill between participants and researchers-is more effective at increasing unit response, even when financial incentives are equal. ...
The ease and affordability of Amazon’s Mechanical Turk make it ripe for longitudinal, or panel, study designs in social science research. But the discipline has not yet investigated how incentives in this “online marketplace for work” may influence unit non-response over time. This study tests classic economic theory against social exchange theory and finds that despite Mechanical Turk’s transactional nature, expectations of reciprocity and social contracts are important determinants of participating in a study’s Part 2. Implications for future research are discussed.
The current literature on nonidealized media content (i.e., content that presents media models that do not follow traditional beauty standards) rarely includes novel positive body image outcomes, such as a broad conceptualization of beauty. Moreover, little is known about more positively oriented explanatory mechanisms, such as inspiration (i.e., feeling inspired by media to adopt a broader inner conception of beauty). Therefore, the current cross-sectional survey study among 458 women aged 19-30 years old (M = 22.76, SD = 2.64) investigated the links between exposure to nonidealized content on different media platforms, inspiration, and a broad conceptualization of beauty. Conditional process analysis, using Hayes' PROCESS tool, was performed. Our results indicated that exposure to nonidealized content through Instagram posts of influencers/celebrities as well as magazine articles was positively linked to women's broader conceptualization of beauty. Inspiration was also found to be a significant mediator in these relationships. Moreover, inspiration was found to fully mediate the relationship between exposure to nonidealized content through companies' Instagram posts and a broad conceptualization of beauty. No relationships were found for exposure to nonidealized content through television series or movies. These results highlight how nonidealized media models can optimally empower media users, as well as how some media platforms might have a greater potential to fulfill this empowering role than others. The findings inform practitioners of the relevance of including nonidealized media content in intervention programs.
Monetary respondent incentives are a means to counteract the trend toward declining response rates. This article summarizes the results of a series of experiments conducted in the past decade in the German General Social Survey. We found that prepaid monetary incentives led to a higher increase in response than promised incentives. There was no evidence that either promised or prepaid incentives had a systematic effect on sample composition. The incentives helped to reduce fieldwork efforts. Thus, the costs of incentives can—at least partially—be offset by a reduction in the number of contact attempts required to achieve a certain number of completed interviews. Our findings are highly credible due to the replicative design of the experiments.
This chapter presents a case study focusing on the comparison of a mixed‐mode design with a web component and a computer assisted personal interviewing‐only design and on the use of incentives. Outcome measures include response rate and other quality indicators. The chapter investigates these factors in a single wave of a longitudinal panel. It presents a critical review of the literature on randomized experiments to study the effects of incentives with special reference to different modes. The chapter then presents the case study based on experiments carried out in Wave 5 of the Understanding Society Innovation Panel in 2012. Outcomes from the experiments are presented using statistical tests and models. The case study provides an example of how to design such experiments and how to handle the analysis of such experiments from a statistical perspective, while also providing evidence of the impact of incentive strategies and modes.
W artykule scharakteryzowano procedury doboru oraz terenowej realizacji prób sondażowych w międzykrajowych badaniach porównawczych. Podstawą analiz była metodologiczna archiwizacja 1537 surveyów – zrealizowanych w latach 1981–2017 na reprezentatywnych próbach populacji obywateli państw europejskich. Pod uwagę wzięto pięć wiodących programów porównawczych, w tym: Eurobarometer & Candidate Countries Eurobarometer, European Quality of Life Survey, European Social Survey, European Values Study, International Social Survey Programme. Autor wyszedł od ogólnej charakterystyki projektów, by następnie przedstawić szczegółowy zakres archiwizacji krajowych surveyów oraz opisać międzyprojektowe oraz wzdłuż czasowe różnice w jakości dokumentacji procedur badawczych. Główną część analiz poświęcono opisowi między- oraz wewnątrzprojektowych różnic w zakresie: 1) sposobów definiowania populacji docelowych, 2) rodzajów operatów doboru prób badawczych, 3) typów prób sondażowych ischematów losowania jednostek, 4) procedur wewnątrzzespołowej selekcji respondentów docelowych w adresowych próbach osób, 5) procedur terenowych (w tym: czasu trwania badania, zastosowanych technik badawczych, stosowania substytucji jednostek niedostępnych innymi respondentami, listów zapowiednich, gratyfikacji respondentów, konwersji odmów oraz procedur kontrolnych), a także 6) raportowania o wskaźnikach terenowej realizacji prób badawczych.
Die Allgemeine Bevölkerungsumfrage der Sozialwissenschaften (ALLBUS) ist seit 1980 eine zentrale Datenquelle für die Untersuchung des gesellschaftlichen Wandels in Deutschland. Durch die regelmäßige Wiederholung thematischer Schwerpunktmodule können Veränderungen in den Einstellungen und Verhaltensweisen der deutschen Bevölkerung abgebildet und erklärt werden. Der ALLBUS hat dabei Standards für die Durchführung von bevölkerungsrepräsentativen Befragungen in Deutschland gesetzt. Die Methodenforschung im ALLBUS zielt darauf ab, die Qualität der Daten zu optimieren und der zurückgehenden Teilnahmebereitschaft an Umfragen zu begegnen. Seit 2017 ist der ALLBUS Teil der Integrierte Erhebungs- und Dateninfrastruktur (IEDI) bei GESIS. Durch die Bündelung von Umfragen sollen höhere Qualitätsstandards durchgesetzt und Daten für die Untersuchung sozialen und politischen Wandels in Deutschland der Forschung langfristig zur Verfügung stehen.
In survey interviews, "Don't know" (DK) responses are commonly treated as missing data. One way to reduce the rate of such responses is to probe initial DK answers with a follow-up question designed to encourage respondents to give substantive, non-DK responses. However, such probing can also reduce data quality by introducing additional or differential measurement error. We propose a latent variable model for analyzing the effects of probing on responses to survey questions. The model makes it possible to separate measurement effects of probing from true differences between respondents who do and do not require probing. We analyze new data from an experiment which compared responses to two multi-item batteries of questions with and without probing. In this study, probing reduced the rate of DK responses by around a half. However, it also had substantial measurement effects, in that probed answers were often weaker measures of constructs of interest than were unprobed answers. These effects were larger for questions on attitudes than for pseudo-knowledge questions on perceptions of external facts. The results provide evidence against the use of probing of "Don't know" responses, at least for the kinds of items and respondents considered in this study.
This chapter examines whether a change in incentives can lead to reductions of nonresponse bias, decreases in measurement error, and, through cost savings, decreases in sampling error. It first reviews the literature on incentives in face-to-face surveys, and then describes the survey, the administration of the incentive experiment, and the available data. The chapter also examines the impact of two different incentives on nonresponse error using several indirect measures, including response rates, composition of the interviewed groups, and differences in estimates of key statistics. It also reviews the relative costs of some treatments to see if the higher incentive can produce expected cost efficiencies, which can be translated into a larger number of interviews for a fixed budget. The chapter further explores the impact of incentives on consistency of reporting for questions administered by an interviewer using computer-assisted personal interviewing (CAPI) and the same self-administered using audio computer-assisted self-interviewing (ACASI).
Respondent incentives are increasingly used as a measure of combating falling response rates and resulting risks of nonresponse bias. Nonresponse in panel surveys is particularly problematic, since even low wave-on-wave nonresponse rates can lead to substantial cumulative losses; if nonresponse is differential, this may lead to increasing bias across waves. Although the effects of incentives have been studied extensively in cross-sectional contexts, little is known about cumulative effects across waves of a panel. We provide new evidence about the effects of continued incentive payments on attrition, bias and item nonresponse, using data from a large scale, multi-wave, mixed mode incentive experiment on a UK government panel survey of young people. In this study, incentives significantly reduced attrition, far outweighing negative effects on item response rates in terms of the amount of information collected by the survey per issued case. Incentives had proportionate effects on retention rates across a range of respondent characteristics and as a result did not reduce attrition bias in terms of those characteristics. The effects of incentives on retention rates were larger for unconditional than conditional incentives and larger in postal than telephone mode. Across waves, the effects on attrition decreased somewhat, although the effects on item nonresponse and the lack of effect on bias remained constant. The effects of incentives at later waves appeared to be independent of incentive treatments and mode of data collection at earlier waves.
We first present an overview of international figures on nonresponse and their trend over time, distinguishing between noncontacts and refusals. In the second part we model differences between countries, using background variables on survey design and fieldwork strategy...
Abstract Survey research has been witnessing declining response rates across a wide range of surveys. As a result, extensive resources have been invested on boosting response rates under the assumption that a higher response rate will lead to lower survey error. However, the extra attempts to reduce nonresponse are costly. Given limited budgets, survey researchers might be more confident about allocating a larger share of survey budgets to decreasing nonresponse,if they could be assured that an improved response rate does in fact lower survey error. In fact, the empirical evidence in this regard is
Nonresponse bias has been a long-standing issue in survey research (Brehm 1993; Dillman, Eltinge, Groves and Little 2002), with numerous studies seeking to identify factors that affect both item and unit response. To contribute to the broader goal of minimizing survey nonresponse, this study considers several factors that can impact survey nonresponse, using a 2007 Animal Welfare Survey Conducted in Ohio, USA. In particular, the paper examines the extent to which topic salience and incentives affect survey participation and item nonresponse, drawing on the leverage-saliency theory (Groves, Singer and Corning 2000). We find that participation in a survey is affected by its subject context (as this exerts either positive or negative leverage on sampled units) and prepaid incentives, which is consistent with the leverage-saliency theory. Our expectations are also confirmed by the finding that item nonresponse, our proxy for response quality, does vary by proximity to agriculture and the environment (residential location, knowledge about how food is grown, and views about the importance of animal welfare). However, the data suggests that item nonresponse does not vary according to whether or not a respondent received incentives.
In contingent valuation studies to assess the economic value of environmental goods, respondents are often given small presents or money amounts at the beginning of the interview to compensate them for their time spent on the survey and to encourage them to be conscientious when answering the questions. Yet, it is still an open question whether this practice biases contingent valuation survey responses, especially stated willingness to pay (WTP) for the respective environmental good. This study employs a set of field experiments to investigate the effect of respondent incentives in the form of monetary and in-kind gifts on responses in a contingent valuation survey. It is analysed how these different kinds of incentives affect (1) respondents' diligence when answering contingent valuation method questions, (2) the likelihood of a respondent to state a positive WTP and (3) the amount of stated WTP. Results show that with respect to raising respondents' diligence in the survey interview, a moderate monetary incentive is most effective. The results regarding the effect on WTP statements are less clear. While the likelihood to state a positive WTP is increased by most incentives, mean WTP estimates are virtually unaffected.
Self-determination theory (SDT) is an empirically based theory of human motivation, development, and wellness. The theory focuses on types, rather than just amount, of motivation, paying particular attention to autonomous motivation, controlled motivation, and amotivation as predictors of performance, relational, and well-being outcomes. It also addresses the social conditions that enhance versus diminish these types of motivation, proposing and finding that the degrees to which basic psychological needs for autonomy, competence, and relatedness are supported versus thwarted affect both the type and strength of motivation. SDT also examines people's life goals or aspirations, showing differential relations of intrinsic versus extrinsic life goals to performance and psychological health. In this introduction we also briefly discuss recent developments within SDT concerning mindfulness and vitality, and highlight the applicability of SDT within applied domains, including work, relationships, parenting, education, virtual environments, sport, sustainability, health care, and psychotherapy.
The quality of data collected in survey research is usually indicated by the response rate; the representativeness of the sample, and; the rate of completed questions (item-response). In attempting to improve a generally declining response rate in surveys considerable efforts are being made through follow-up mailings and various types of incentives. This study examines effects of including a scratch lottery ticket in the invitation letter to a survey.
Questionnaires concerning oral health were mailed to a random sample of 2,400 adults. A systematically selected half of the sample (1,200 adults) received a questionnaire including a scratch lottery ticket. One reminder without the incentive was sent.
The incentive increased the response rate and improved representativeness by reaching more respondents with lower education. Furthermore, it reduced item nonresponse. The initial incentive had no effect on the propensity to respond after the reminder.
When attempting to improve survey data, three issues become important: response rate, representativeness, and item-response. This study shows that including a scratch lottery ticket in the invitation letter performs well on all the three.
This study revisits the issue of monetary incentive effects utilizing data from a mail survey sent to a random sample of adults
across the United States regarding preferences for fuel ethanol. The results reported here are consistent with those found
in the literature regarding the effect of incentives on response rates: they improved them, with prepaid incentives performing
relatively better. We also found that state of residence was significantly correlated with choosing whether to respond to
a survey. Regarding the effect of incentives on sample composition, we found that incentives tended to bias the sample in
favor of less educated respondents, and tended to attract respondents less familiar with the survey subject. Finally, results
indicate that incentives had very little effect on item nonresponse. Instead, item nonresponse was driven by education level,
gender, and familiarity with the survey subject. However, combining the findings on sample composition with those of item
nonresponse, it appears that the use of incentives indirectly affects item nonresponse by recruiting relatively more respondents that are less educated and/or less familiar with the survey
topic, who are then less likely to respond to all questions.
The respondents’ susceptibility to item nonresponse is an important determinant for the quality of political science survey data. In particular, one has to expect biased samples when groups of respondents systematically differ in their nonresponse rates. Our study tests firstly whether and how strong four different aspects of political involvement which are frequently used in political studies affect the respondents’ susceptibility to nonresponse. Secondly, we tested the con-sequences of these differences in item nonresponse for the presence of sample-selection bias in general, and in particular for the comparability of the samples in international comparative political science research. The database of our study are Eurobarometer studies in the 10 new member states of the European Union. According to our first result, politically involved respondents are substantially less likely to refuse to answer questions and in particular less prone to “don’t know” responses. Our second result indicates that this causes a systematic over representation of involved respondents. The strength of this sample selection bias differs substantially between the analyzed countries.
Self-perception theory is applied to explain mail survey respondent behavior when incentives to respond are used. The results support the hypothesis that when external cues are present (i.e., incentives are used) the level of compliance, as measured by response quality, is reduced. A monetary and a comparable value nonmonetary incentive are used. The psychological value of money explanation is supported as the monetary incentive group yields a significantly higher response rate and faster response.
Self-determination Theory (SDT) is a motivational theory of personality, development, and social processes that examines how social contexts and individual differences facilitate different types of motivation, especially autonomous motivation and controlled motivation, and in turn predict learning, performance, experience, and psychological health. SDT proposes that all human beings have three basic psychological needs – the needs for competence, autonomy, and relatedness – the satisfaction of which are essential nutrients for effective functioning and wellness. Satisfaction of these basic needs promotes the optimal motivational traits and states of autonomous motivation and intrinsic aspirations, which facilitate psychological health and effective engagement with the world.
The Alcohol and Drug Services Study (ADSS) was conducted for the Substance Abuse and Mental Health Services Administration between 1996 and 1999 to assess the nation's substance abuse treatment system. The sample for ADSS was selected using a multi-stage stratified design. Clients were sampled from selected substance abuse treatment facilities as part of the Phase II sample. Client follow-up comprised interviews and urine specimen collection in Phase III of the survey. One component of ADSS examined the impact of different incentive payments on measures of response rate and response quality using ADSS Phase III client interview data. To test for the effects of four payment levels, several measures of response quality were used. The analysis of consistency involving respondent self-reports, abstracted record data, and urine test results showed no conclusive evidence that incentive payments have any positive or negative effect on data consistency. The analysis of item non-response rates showed that an increase in incentive payments was associated with a subtle decrease in item type non-response rates. Furthermore, mixed results were observed when analyzing the relationship between incentive payment and clients reporting more (or less) of certain types of behavior.
Declining response rates have led to extended efforts to convert reluctant respondents. If respondents with low motivation are likely to use cognitive shortcuts in survey responding (to engage in satisficing), bringing reluctant respondents into the respondent pool may result in an increase of measurement error. In a cross-national setting, the differential efforts of converting reluctant respondents across countries may result in differential data quality, potentially leading to non-equivalence. This study examines the relationship between reluctance and respondents' likelihood to engage in satisficing, controlling for cognitive ability in Belgium, Germany, the Netherlands, Norway, Spain and Sweden. The study is based on the European Social Survey (ESS), third round, conducted in 2006 and 2007. Latent class analysis is used to develop two main constructs of interest, including reluctance and satisficing. The results of the latent class analysis indicate a presence of a single satisficing factor for all the countries with the exception of Germany and Spain. Thus, Structural Equation Modeling is used for Belgium, the Netherlands, Norway and Sweden, and regression models are used for Germany and Spain. ^ Through an overview of the separately developed literature on reluctance and satisficing a number of common factors are identified. As hypothesized, a positive relationship between reluctance and satisficing is found in Belgium, the Netherlands, Norway and Sweden, and in a number of instances in Germany and Spain. Nevertheless, the results suggest that in Belgium, the Netherlands, Norway and Sweden the relationship between reluctance and satisficing is spurious, and is completely explained by cognitive ability. Similarly, in Germany and Spain, the effect of reluctance on satisficing decreases or becomes statistically nonsignificant after controlling for cognitive ability. The cross-country comparison suggests that the direction and the magnitude of the relationship among satisficing, reluctance, and cognitive ability are generally invariant across the countries. However, this does not guarantee the equivalence of estimates. Potential impacts of differences in cognitive ability, reluctance and in the overall likelihood to satisfice on equivalence are discussed.
Self-perception theory is applied to explain mail survey respondent behavior when incentives to respond are used. The results support the hypothesis that when external cues are present (i.e., incentives are used) the level of compliance, as measured by response quality, is reduced. A monetary and a comparable value nonmonetary incentive are used. The psychological value of money explanation is supported as the monetary incentive group yields a significantly higher response rate and faster response.
The manner in which the concept of reciprocity is implicated in functional theory is explored, enabling a reanalysis of the concepts of "survival" and "exploitation." The need to distinguish between the concepts of complementarity and reciprocity is stressed. Distinctions are also drawn between (1) reciprocity as a pattern of mutually contingent exchange of gratifications, (2) the existential or folk belief in reciprocity, and (3) the generalized moral norm of reciprocity. Reciprocity as a moral norm is analyzed; it is hypothesized that it is one of the universal "principal components" of moral codes. As Westermarck states, "To requite a benefit, or to be grateful to him who bestows it, is probably everywhere, at least under certain circumstances, regarded as a duty. This is a subject which in the present connection calls for special consideration." Ways in which the norm of reciprocity is implicated in the maintenance of stable social systems are examined.
To examine the effects of lotteries on response behavior, five experiments were conducted in a nonprofit online panel. The type of lottery offered for study participation was a cash lottery, a voucher lottery, or a lottery of surprise gifts of either known or unknown value. The control group was not offered a lottery. Dependent measures were response, retention, and several facets of response quality. None of the lotteries significantly increased response, retention, or response quality. However, looking at effect sizes reveals a pattern across the five experiments that can inform and refine the practice of employing lotteries in nonprofit online panels. This pattern suggests that lotteries are more effective with panelists low in motivation than high in motivation.
This article is intended to supplement rather than replace earlier reviews of research on survey incentives, especially those by Singer (2002); Singer and Kulka (2002); and Cantor, O’Hare, and O’Connor (2008). It is based on a systematic review of articles appearing since 2002 in major journals, supplemented by searches of the Proceedings of the American Statistical Association’s Section on Survey Methodology for unpublished papers. The article begins by drawing on responses to open-ended questions about why people are willing to participate in a hypothetical survey. It then lays out the theoretical justification for using monetary incentives and the conditions under which they are hypothesized to be particularly effective. Finally, it summarizes research on how incentives affect response rates in cross-sectional and longitudinal studies and, to the extent information is available, how they affect response quality, nonresponse error, and cost-effectiveness. A special section on incentives in Web surveys is included.
This research examines how the quality of survey measures varies with the age of the respondent. A structural modeling approach is applied to data from six surveys (five national surveys of American adults and one survey in a Canadian corporation—7,706 respondents in all) to generate measurement quality estimates for 106 survey measures. Results show that as respondent age increases (a) the percentage of true score variance in survey measures tends to decline, (b) the percentage of both random and correlated error variance tends to increase, and (c) people tend to have more interrelated—or less differentiated—views about their worlds. Most of these trends are highly replicable from survey to survey and are—except for the level of differentiation in views—not attributable to older respondents tending to have less education than younger people. The results suggest that data from older respondents tend to provide a somewhat less precise indication of the attitudes, behaviors, or other characteristics being measured than do data from younger respondents, but that relationships among survey measures will not necessarily be weaker.
External interventions have a significant and systematic effect on preferences: Under specific conditions they crowd an individual's intrinsic motivation in or out. Rewards given or regulations applied by a principle are more likely to crowd out an agent's inner preference for a certain task. The more personal the relation between the two actors is, the more interesting the agent finds the task and the more extensive an agent's participation possibilities are. Empirical evidence supports the claim that, in many cases, agents, indeed, react to an external motivation by reducing their effort to fulfill a certain duty. This points to new limits of pricing as well as regulating, even though the price mechanism does not destroy intrinsic motivation to the same extent because it is less restrictive than regulation.
Inspired by the positive effects of incentives on mail survey response rates, researchers have started using incentives to increase response rates to Web surveys. The established best practice of presending cash incentives is difficult to implement in Web surveys, and studies suggest that its presumed effects might not be witnessed in Web surveys. In contrast, several studies have found that lotteries can significantly increase Web survey response rates. Some authors have argued that this could reflect the fact that Internet users have come to expect Web surveys to be associated with lotteries. An experimental study among university students found that the lottery influences the Web survey response rates, but there are indications that different subgroups might be more influenced by this incentive than others. The observed differences are interpreted along the lines of possible differences in the degree to which different respondent groups expect incentives in return for their participation.
Preface. Chapter 1. The Evolution of Survey Process Quality. 1.1 The Concept of a Survey. 1.2 Types of Surveys. 1.3 Brief History of Survey Methodology. 1.4 The Quality Revolution. 1.5 Definitions of Quality and Quality in Statistical Organizations. 1.6 Measuring Quality. 1.7 Improving Quality. 1.8 Quality in a Nutshell. Chapter 2. The Survey Process and Data Quality. 2.1 Overview of the Survey Process. 2.2 Data Quality and Total Survey Error. 2.3 Decomposing Nonsampling Error into Its Component Parts. 2.4 Gauging the Magnitude of Total Survey Error. 2.5 Mean Squared Error. 2.6 An Illustration of the Concepts. Chapter 3. Coverage and Nonresponse Error. 3.1 Coverage Error. 3.2 Measures of Coverage Bias. 3.3 Reducing Coverage Bias. 3.4 Unit Nonresponse Error. 3.5 Calculating Response Rates. 3.6 Reducing Nonresponse Bias. Chapter 4. The Measurement Process and Its Implications for Questionnaire Design. 4.1Components of Measurement Error. 4.2 Errors Arising from the Questionnaire Design. 4.3 Understanding the Response Process. Chapter 5. Errors Due to Interviewers and Interviewing. 5.1 Role of the Interviewer. 5.2 Interviewer Variability. 5.3 Design Factors that Influence Interviewer Effects. 5.4 Evaluation of Interviewer Performance. Chapter 6. Data Collection Modes and Associated Errors. 6.1 Modes of Data Collection. 6.2 Decision Regarding Mode. 6.3 Some Examples of Mode Effects. Chapter 7. Data Processing: Errors and Their Control. 7.1 Overview of Data Processing Steps. 7.2 Nature of Data Processing Error. 7.3 Data Capture Errors. 7.4 Post-Data Capture Editing. 7.5 Coding. 7.6 File Preparation. 7.7 Applications of Continuous Quality Improvement: The Case of Coding. 7.8 Integration Activities. Chapter 8. Overview of Survey Error Evaluation Methods. 8.1 Purposes of Survey Error Evaluation. 8.2 Evaluation Methods for Designing and Pretesting Surveys. 8.3 Methods for Monitoring and Controlling Data Quality. 8.4 Postsurvey Evaluations. 8.5 Summary of Evaluation Methods. Chapter 9. Sampling Error. 9.1 Brief History of Sampling. 9.2 Nonrandom Sampling Methods. 9.3 Simple Random Sampling. 9.4 Statistical Inference in the Presence of Nonsampling Errors. 9.5 Other Methods of Random Sampling. 9.6 Concluding Remarks. Chapter 10.1 Practical Survey Design for Minimizing Total Survey Error. 10.1 Balance Between Cost, Survey Error, and Other Quality Features. 10.2 Planning a Survey for Optimal Quality. 10.3 Documenting Survey Quality. 10.4 Organizational Issues Related to Survey Quality. References. Index.
Behavior coding is one technique researchers use to detect problems in survey questions, but it has been primarily explored
as a practical tool rather than a source of insight into the theoretical understanding of the cognitive processes by which
respondents answer survey questions. The latter is the focus of the current investigation. Using data from a large study in
which face-to-face interviews were taped and extensive behavior coding was done, we tested whether sets of respondent behavior
codes could be used to distinguish respondent difficulties with comprehension of the question from difficulties associated
with mapping a judgment onto the response format provided, and whether characteristics of the survey questions and respondents
could be used to predict when and for whom such difficulties would occur. Sets of behavior codes were identified that reflected
comprehension and mapping difficulties, and these two types of difficulties were associated with different question and respondent
characteristics. This evidence suggests that behavior coding shows promise as a tool for researchers studying the cognitive
processes involved in answering survey questions.
Two incentive experiments were conducted in different online access panels. Experiment 1 was carried out in a commercial market research panel. It examined whether three different types of promised incentives (redeemable bonus points, money lottery and gift lottery), four different amounts of bonus points or raffled money, and two different denominations of raffled money influenced response quantity, sample composition, response quality and survey outcome. Type of incentive and number of bonus points mildly influenced dropout and sample composition. Moreover, response was higher with bonus points than with the two types of lotteries. Response quality and survey outcome were not affected. Experiment 2 was conducted in a non-profit panel, which holds one half self- selected and one half non-self-selected participants. Incentives were two different amounts of raffled money in two different denominations. Response, dropout, response quality, survey outcome and sample composition were not affected. Based on a cost-benefit analysis, recommendations for employing incentives in online access panels are given.
The abstract for this document is available on CSA Illumina.To view the Abstract, click the Abstract button above the document title.
Conducted an experiment with cash incentives in a personal interview survey in 3 Ontario, Canada, cities. 945 women and men served as Ss. One dollar prepaid yielded significantly higher response, sample-wide, and less sociodemographic bias than 10 promised at the conclusion of the interview. Item nonresponse did not significantly differ across the incentive conditions. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
This paper proposes that when optimally answering a survey question would require substantial cognitive effort, some repondents simply provide a satisfactory answer instead. This behaviour, called satisficing, can take the form of either (1) incomplete or biased information retrieval and/or information integration, or (2) no information retrieval or integration at all. Satisficing may lead respondents to employ a variety of response strategies, including choosing the first response alternative that seems to constitute a reasonable answer, agreeing with an assertion made by a question, endorsing the status quo instead of endorsing social change, failing to differentiate among a set of diverse objects in ratings, saying ‘don't know’ instead of reporting an opinion, and randomly choosing among the response alternatives offered. This paper specifies a wide range of factors that are likely to encourage satisficing, and reviews relevant evidence evaluating these speculations. Many useful directions for future research are suggested.
Cet article examine dans quelle mesure l'attribution d'incitations financieres comme prealables a la participation a des enquetes peut ameliorer la qualite des reponses en situation d'entretien en face-a-face. Une etude experimentale permet aux AA. de conclure a l'absence d'influence significative des incitations financieres sur la qualite des reponses des interviewes
Cross-sectional comparisons have consistently revealed that increased age is associated with lower levels of cognitive performance, even in the range from 18 to 60 years of age. However, the validity of cross-sectional comparisons of cognitive functioning in young and middle-aged adults has been questioned because of the discrepant age trends found in longitudinal and cross-sectional analyses. The results of the current project suggest that a major factor contributing to the discrepancy is the masking of age-related declines in longitudinal comparisons by large positive effects associated with prior test experience. Results from three methods of estimating retest effects in this project, together with results from studies comparing non-human animals raised in constant environments and from studies examining neurobiological variables not susceptible to retest effects, converge on a conclusion that some aspects of age-related cognitive decline begin in healthy educated adults when they are in their 20s and 30s.
In an effort to counter the problem of noncooperation, survey organizations are offering incentives to respondents with increasing
frequency, some at the outset of the survey, as has traditionally been done in mail surveys, and some only after the person
has refused, in an attempt to convert the refusal. This article reports on a series of experiments carried out over a period
of about 2 years with a monthly telephone survey, the Survey of Consumer Attitudes, in an effort to increase response rates
or reduce interviewer effort. We report on experiments with prepaid versus promised incentives; advance letters; and advance
letters with prepaid incentives; and we also report on the effects of incentives on response quality, sample composition,
response bias, interviewer and respondent expectations, and costs.
- Fricker S.
Beyond response rates: The effect of prepaid incentives on measurement error
- R L Medway
The use of respondent incentives in surveys
- G Nicolaas
- P Lynn
Paying respondents for survey participation In Studies of Welfare Populations: Data Collection and Research Issues
- E Singer
- R A Kulka
Examining the relationship between nonresponse propensity and data quality in two national household surveys
- S Fricker
- R Tourangeau
The use of incentives to reduce nonresponse in household surveys
- E Singer
- R M Groves
- D A Dillman
- J L Eltinge
- R J A Little
Respondent incentives in a national face-to-face survey: Effects on outcome rates, sample composition and fieldwork efforts
- M Blohm
- A Koch
The Silent Revolution: Changing Values and Political Styles in Advanced Industrial Society
- R Inglehart
Report on quality assessment of contact data files in Round 5: Final report 27 countries
- H Matsuo
- G Loosveldt