ArticlePDF Available

Setting Priorities: Spurious Differences in Response Rates

Authors:
Article

Setting Priorities: Spurious Differences in Response Rates

Abstract and Figures

Response rates are a key quality indicator of a survey. Thus, their comparability across surveys and countries is pivotal. The first round of the European Social Survey contains a natural experiment in the estimation of response rates. While all countries implemented the same standardized contact protocol to record the outcome of each contact attempt, no instructions were given as to how to code a final case disposition from these individual contact attempts and each country chose its own strategy. We demonstrate that the coding strategy chosen has a substantial impact on the response rates reported. In particular, contact and cooperation rates derived by means of different coding strategies are incomparable across countries when intensive refusal conversion efforts are in place.
Content may be subject to copyright.
SPURIOUS DIFFERENCES IN RESPONSE RATES 1
Setting Priorities: Spurious Differences in Response Rates
Annelies G. Blom
University of Mannheim
Abstract
Response rates are a key quality indicator of a survey. Thus, their comparability across
surveys and countries is pivotal. The first round of the European Social Survey contains a
natural experiment in the estimation of response rates. While all countries implemented the
same standardized contact protocol to record the outcome of each contact attempt, no
instructions were given as to how to code a final case disposition from these individual
contact attempts and each country chose its own strategy. We demonstrate that the coding
strategy chosen has a substantial impact on the response rates reported. In particular, contact
and cooperation rates derived by means of different coding strategies are incomparable across
countries when intensive refusal conversion efforts are in place.
Keywords: response rate, contact rate, cooperation rate, final disposition codes, priority
coding, most-recent coding
SPURIOUS DIFFERENCES IN RESPONSE RATES 2
Setting Priorities: Spurious Differences in Response Rates
Despite increasing concern about response rates being insufficient as sole indicators
of nonresponse bias and data quality (Groves & Peytcheva, 2008; Keeter, Miller, Kohut,
Groves & Presser, 2000), they are still the most prevalent measure for many journals, funding
agencies, and survey programs (for alternative indicators see Wagner 2012). In addition to
overall response rates, research has shown that contact and cooperation need to be considered
as separate processes associated with different sample unit characteristics and thus different
biases (e.g. Lynn & Clarke 2002). While non-contacted persons are more likely to be
employed and living an active life style, those who are contacted but do not participate in a
survey are more likely to be socially disengaged (Groves & Couper 1998, ch. 4-5).
Guidelines such as those of the American Association for Public Opinion Research (AAPOR)
thus standardize the calculation of contact and cooperation rates in addition to overall
response rates.
The standardized calculation and reporting of response rates has been a major survey
methodological achievement for comparing across surveys and countries various types of
response rates (i.e. overall response rates, contact rates, and cooperation rates). Nevertheless,
there is one important step that almost all surveys fail to make explicit when reporting
response rates: the coding of the sequence of call outcomes (i.e. contact attempts) at a sample
unit into a final disposition code for this sample unit.
The European Social Survey (ESS) has from its inception emphasized the importance
of high response rates across countries. In addition to aiming for an overall minimum
response rate of 60%, all countries also need to keep their non-contact rate at a maximum of
3%. Both are to be achieved through intensive fieldwork efforts. Such standards demonstrate
the importance of the comparability of response rates, especially in cross-national surveys.
SPURIOUS DIFFERENCES IN RESPONSE RATES 3
In 2002 a natural experiment occurred in the first round of the ESS. While all
countries implemented the same standardized contact protocol, which recorded each contact
attempt, no instructions were given as to how to determine a final disposition code from these
individual contact attempts. In this article we showcase the complexity of coding final
disposition codes and the impact that differential coding has on reported outcome rates (see
also McCarty (2003) for differences in overall response rates). In addition, the first round of
the ESS demonstrates that differential coding can lead to differential conclusions about
comparisons of response rates across surveys thereby jeopardizing cross-national quality
comparisons.
Standardization of Response Rates
The standardization of response rate calculations has been a methodological focus
from the 1970s onwards. In 1977 Kviz noted that “[t]he absence of a standard definition [of
response rates] has caused a great deal of confusion regarding the interpretation of reported
response rates and has frustrated methodological investigations because of a lack of
comparative data” (Kviz, 1977, p. 265). In the late 1990s, a committee of the AAPOR
“developed standard definitions for the final disposition of case codes and of various outcome
rates (e.g. response rates and cooperation rates) based on these codes” (Smith, 2002, p.30).
Since, AAPOR has regularly updated their case outcome and outcome rate definitions for
scholars and survey managers to adopt in their surveys (most recently in AAPOR, 2011).
In survey practice we typically distinguish three types of response outcomes
(Figure 1): First, in-office case outcomes are response outcomes that do not originate from the
field process, but instead are assigned to a sample unit in the office. In-office case outcomes
might be either assigned to cases that were never fielded or to cases that were fielded, but
where the final field outcome was superseded by an in-office case outcome (for example
when a respondent calls the survey agency to request their data to be deleted, i.e. an office
SPURIOUS DIFFERENCES IN RESPONSE RATES 4
refusal). Second, final call outcomes are call outcomes from the field, which by their very
nature will equal the final disposition code. By definition there can only be one call outcome
of this type for a case. The two most obvious examples of final call outcomes are when a full
interview is achieved and when a sample unit is found to be ineligible. Third, non-final call
outcomes do not automatically lead to a final disposition code. For sample units where the
only outcomes are a sequence of only non-final call outcomes, a final disposition code needs
to be derived, i.e. a derived final case outcome needs to be assigned. An example of this is the
increasingly common situation in which a sample unit is approached several times, yet is
either repeatedly busy (refusal) or not reached (non-contact). The disposition code will then
need to be derived from the sequence of these non-final call outcomes.
Figure 1: Deriving final disposition codes from contact data
Method
There are three main methods by which a final disposition code may be derived from
a sequence of non-final call outcomes: most-recent, priority, and subjective coding (Blom,
Jäckle, & Lynn, 2010). With most-recent coding, the outcome of the last call to a sample unit
is defined as the final disposition code (e.g. AAPOR, 2000). Priority coding, on the other
hand, involves arranging call outcomes according to a priority ranking, in which some
outcome codes take priority over others. For instance, one would define that achieving an
interview takes priority over a refusal, which in turn takes priority over a non-contact (Lynn,
Beerten, Laiho, & Martin, 2001). A situation in which an interviewer tries to convert an
initial refusal, yet never manages to make contact again, would be coded differently
according the two coding strategies. If the last call outcome defined the final disposition, this
would be a non-contact: with priority coding the final disposition would be a refusal. Finally,
subjective coding refers to situations where the rules for deriving a case outcome from a
sequence of call outcomes are not defined. Typically in such situations only descriptions of
SPURIOUS DIFFERENCES IN RESPONSE RATES 5
each final disposition code (which can vary in their precision) are provided. It is left to the
coder to decide how to allocate cases to outcomes. This kind of coding is perhaps most
common when interviewers are asked to return final disposition codes to the survey
organization, though it may also be used by survey organizations carrying out in-office
coding. Since subjective coding is not a standardized procedure even within a single survey,
it is unsuitable for comparative purposes.
Blom (2008) compares coding schemes with respect to ease of implementation in the
field and accuracy in reflecting the de facto response outcomes. She shows that most-recent
coding has clear advantages over priority coding in terms of ease of implementation (see also
McCarty, 2003). However, she further demonstrates that to reflect de facto response
outcomes as accurately as possible, priority coding is the method of choice. For most surveys,
response rates are used as quality indicators and should thus accurately reflect the fieldwork
processes. Thus it seems that response rates based on a priority coding, where the priority
ranking is standardized and published, should be preferred (p.32-36). In its Standard
Definitions AAPOR (2011) recommends a complex interplay of most recent and priority
coding (p.10-11). Unfortunately, the procedure is not described in much detail and, in
contrast to the calculation of response rates, only a recommendation rather than a
standardized process.
In our analyses we use the contact data of round 1 of the ESS, a cross-national face-
to-face survey conducted in 2002 (see www.europeansocialsurvey.org). We compare the
differential impact of a strict priority-ranked coding and most-recent coding for deriving final
disposition codes across 16 national ESS data collections
1
. For deriving case outcomes by
means of priority coding the call outcomes recorded on the standardized contact forms were
arranged according to the hierarchy in Table 1.
Table 1: Priority ranking of response outcomes
SPURIOUS DIFFERENCES IN RESPONSE RATES 6
Our analyses are composed of two parts. First, we use the ESS contact data to derive
final dispositions by means of most-recent coding and priority coding. We examine
differences across coding strategies in the occurrence of the aggregate final disposition codes
interview, ineligible, refusal, other contact, and non-contact. Second, we compare the final
dispositions that each country reported in round 1 (ESS 2003) with our derived final
dispositions. For each country we calculate overall response rates, contact rates, and
cooperation rates based on the respective coding strategy (country coding, most-recent
coding, and priority coding) and analyze the effect of the coding on reported response rates.
Results
In the following, we present the results of our analyses into the effect of different
coding strategies for coding sequences of non-final call outcomes on the distribution of final
dispositions and response rates.
Differences in Aggregate Final Disposition Codes
The rows in Table 2 display the frequency of case outcomes derived by means of the
last call outcome (i.e. most-recent coding); the columns contain the outcomes derived via the
priority ranking of call outcomes. The diagonal of Table 2 shows the number of cases where
most-recent and priority coding lead to the same aggregate final disposition code.
Table 2: Aggregate final disposition codes: most-recent versus priority coding
The most significant differences between the two coding strategies are found in the
refusal, non-contact, and other contact outcomes. Regarding refusals, priority coding yielded
10% more refusals than most-recent coding (13,613 cases priority coded compared with
12,367 cases most-recent coded). This is due to the fact that with priority coding any
unsuccessful attempt at converting refusals will be recorded as a refusal, while if the last call
outcome is taken, the final disposition might be another code (mostly non-contact or other
contact). Looking at the detailed outcome codes (not displayed) one further finds that most of
SPURIOUS DIFFERENCES IN RESPONSE RATES 7
these cases that were coded a refusal with priority coding were coded ‘non-contact, nobody at
home’ with most-recent coding (670 cases, 52%). Interesting are also cases that were coded
refusals with priority coding, but ‘other contact’ with most-recent coding. For 158 (12%) of
these the ‘other contact’ outcome was ‘unavailable temporarily’ or ‘unavailable during
fieldwork period’. These are possibly cases, where a household member claimed that the
target person was unavailable to avoid an interview. If this were true, the outcome of these
calls would be disguised refusals. Another 119 cases (9%) that were coded refusals with
priority coding were mentally or physically unable to do the interview at the last contact
attempt. Again, the question is, whether these cases were actually unable to do the interview
at the time of the call or whether this was just an easy way out.
With respect to non-contacts, on the other hand, most-recent coding derived 38%
more non-contact case outcomes than priority coding (4,678 most-recent coded compared
with 3,386 priority coded). Again, this can be attributed to repeated, yet unsuccessful call-
backs. In the priority ranking a non-contact takes very low priority. If in a sequence of calls
any contact is established with the household at any point and if this is then followed by non-
contacts in subsequent calls, priority coding assigns a contact outcome to the case, while
most-recent coding assigns a non-contact. In fact, most of these cases are refusal conversion
attempts; of the cases that were coded non-contact with most-recent coding 378 (29%) were a
refusal by the sample person when derived by means of priority coding, 298 (23%) were hard
or soft broken appointments, and another 188 (15%) were refusals by someone other than the
target person.
With priority coding there are also considerably fewer missing case outcomes than
with most-recent coding (106 compared to 128). The reason for this is that priority ranking
only codes a case as ‘missing’ if all call outcomes are missing.
SPURIOUS DIFFERENCES IN RESPONSE RATES 8
Finally, the proportion of cases coded ‘other contact no interview’ is approximately
the same in both coding strategies, although the specific cases assigned to this code differ
across the two coding strategies.
Differences in Contact, Cooperation and Overall Response Rates
In round 1 of the ESS a natural experiment occurred: While all country teams used the
same standardized contact forms, they were not given any instructions as to how they should
code call outcomes into final dispositions. As a consequence, with the contact form data we
can compare how different coding strategies (most-recent and priority coding) affect the
response rates in the ESS and which response rates the national teams arrived at with their
own coding strategy (using the officially reported final dispositions in ESS, 2003).
We calculated the response rates using the following AAPOR rates: RR1 for the
overall response rates, CON1 for the contact rates and COOP1 for the cooperation rates
(AAPOR 2011). Missing case outcomes were assumed to be eligible. The same response rate
formula was used, whether the final disposition codes were derived from the cross-national
contact forms by means of priority or most-recent coding or the final dispositions were
provided by the country teams (country coding).
Figure 2: Response rates across countries by coding strategy
Figure 2 shows differences in response rates across coding strategies in ESS countries.
The x-axis indicates the response rates calculated with the final dispositions provided by the
country teams. It is the baseline to which we compare the rates that we calculated by means
of the cross-national contact forms data.
+
denotes the difference in response rates between
the countries’ disposition codes and those we derived by means of priority coding, while X
denotes the difference in response rates between the countries’ disposition codes and those
derived by means of most-recent coding. Priority coded rates (
+
) and most-recent coded rates
SPURIOUS DIFFERENCES IN RESPONSE RATES 9
(X) are thus displayed as deviations from baseline of the rates based on the national final
disposition codes.
The differences in overall response rates between priority and most-recent coding are
marginal. Yet, this was to be expected, since response rates can only differ if the number of
ineligibles or interviews differs across coding strategies. Nevertheless, some agencies
reported final disposition codes for their cases that lead to higher overall response rates. The
largest difference is found in Germany, where the response rate is approximately
4 percentage points higher when Germany’s own distributions are used. This can be
explained by the German survey agency employing a different definition of eligibility in
round 1. Other countries that derived case outcomes leading to slightly higher response rates
were Greece, Italy, Luxembourg, Poland, and Spain. Conversely, the outcome codes provided
by Slovenia yield a 1 percentage point lower response rate.
While only small differences in rates are found for overall response, most-recent
coding yields a consistently lower contact rate, while priority coding reveals a consistently
lower cooperation rate across all countries examined. Interesting is the comparison of these
rates with those calculated by means of the case outcomes provided by the national survey
agencies. The agencies’ contact and cooperation rates tend to not correspond with the rates
calculated by means of either priority or most-recent coding. For example, the country
contact rates lie somewhere in between the priority and the most-recent coded contact rates in
seven countries, they are higher in four countries and lower than either most-recent or priority
coded rates in two countries. Only in Poland and the UK is the country contact rate almost the
same as the priority coded contact rate, and only in Greece is the country contact rate almost
the same as the contact rate calculated by means of the most-recent case outcomes. Similarly,
the cooperation rates vary across countries.
SPURIOUS DIFFERENCES IN RESPONSE RATES 10
In countries where considerable numbers of cases are re-issued the coding strategy
chosen has a sizeable impact on the rates reported. In an analysis of nonresponse bias in
round 1 of the ESS, Billiet, Philippens, Fitzgerald, and Stoop (2007) find large numbers of
refusal conversions in Austria, Germany, the UK, and the Netherlands. In all of these
countries but Austria, we also find sizeable differences in contact and cooperation rates
depending on the coding strategy chosen. This supports our finding that re-issued cases are
especially vulnerable to the strategy chosen to code non-final call outcomes into final
dispositions.
Discussion and Conclusion
Response rates play an important role in assessing the representativeness of a survey
sample. In addition to overall response rates, we typically report contact and cooperation
rates (or refusal rates) to document some of the traits of the achieved sample.
In this article we showed that contact and cooperation rates may to some extent be
artifacts of the strategy chosen to code the outcomes of sequences of contact attempts into
final dispositions. Especially, where considerable refusal conversion efforts are undertaken –
as is the case in many large-scale data collections – the coding has an impact on the outcome
rates.
Deriving of final case outcomes thus plays an important role when comparing contact
and cooperation rates, and to a lesser extent also overall response rates, across surveys. We
looked into the final dispositions reported on the first round of the ESS and found sizable
differences across coding strategies for several countries. For example, in Germany the
agency’s own coding strategy led to 4 percentage points higher overall response rate. In the
UK, a most-recent coding strategy would have yielded a 6 percentage points higher
cooperation rate than priority coded final dispositions. And while Greece and Norway had
quite comparable contact rates according to their own coding strategies (97%) the priority
SPURIOUS DIFFERENCES IN RESPONSE RATES 11
coded contact rates in the two countries differed by two percentage points (98% in Greece
and 96% in Norway). In surveys like the ESS, where maximum non-contact rates are set at
3%, differences in calculated contact rates of a few percentage points are important to the
survey organizations.
The reported natural experiment occurred in the first round of the ESS. The survey
methodologists working on the ESS quickly acknowledged the problem presented in this
paper. As a consequence, from round 2 onwards the ESS gave additional instructions to its
country teams on how to code the sequences of contact attempts into final dispositions (e.g.
Billiet & Pleysier, 2007; Matsuo, Billiet, Loosveldt & Malnar, 2010). Moreover, the ESS
makes available the full contact protocol data for all countries, thus enabling researchers to
derive final dispositions and conduct comparative nonresponse analyses (e.g. Kreuter &
Kohler, 2009; Stoop, Billiet, Koch, & Fitzgerald, 2010; Blom, de Leeuw, & Hox, 2011; Lipps
& Pollien, 2011; Blom, 2012). However, outside the ESS standards for deriving final
dispositions are far to be sought. The results presented exemplify how, in dearth of official
standards, in-house practices by survey organization affect the comparability of response
rates.
This article aims to draw attention to the importance of standardizing the coding of
call outcomes in order to achieve comparative response, contact and cooperation rates.
Currently, the AAPOR (2011) Standard Definitions provide little assistance regarding
standards for coding call outcomes and documentation. However, in times when outcome
rates play such an important role for journals and funding agencies we need to ensure that
differences in response rates are not simply artifacts of an undocumented coding strategy.
We recommend the adoption of priority coded final dispositions as a standard across
surveys and countries. In addition, it should be standard practice to document the adopted
coding strategy when reporting response rates. We vote in favor of priority coding instead of
SPURIOUS DIFFERENCES IN RESPONSE RATES 12
most-recent coding, because priority coded outcomes most accurately reflect the de facto
outcome of fieldwork and associated data quality aspects. The priority ranking depicted in
Table 1 is suitable for most face-to-face surveys of individuals. Other data collection modes
and surveys on other units of analysis (e.g. households), however, might need to adapt the
ranking for their purposes. Such standardization and documentation should ensure that
differences across surveys in contact and cooperation rates reflect actual differences in
fieldwork and sample composition.
SPURIOUS DIFFERENCES IN RESPONSE RATES 13
References
American Association for Public Opinion Research (2000). Standard Definitions: Final
Dispositions of Case Codes and Outcome Rates for Surveys, 2
nd
edition. Ann Arbor,
MI: American Association for Public Opinion Research.
American Association for Public Opinion Research (2011). Standard Definitions: Final
Dispositions of Case Codes and Outcome Rates for Surveys, 7
th
edition. American
Association for Public Opinion Research.
Billiet, J., Philippens, M., Fitzgerald, R., & Stoop, I. (2007). Estimation of Nonresponse Bias
in the European Social Survey: Using Information from Reluctant Respondents.
Journal of Official Statistics, 23(2), 135-162.
Billiet, J. & Pleysier, S. (2007). Response Based Quality Assessment in the ESS – Round 2. An
Update for 26 countries. London: Centre for Comparative Social Surveys, City
University.
Blom, A.G. (2008). Measuring Nonresponse Cross-Nationally. ISER Working Paper 2008-
41. Colchester: ISER.
Blom, A.G., Jäckle, A., & Lynn, P. (2010). The Use of Contact Data in UnderstandingCross-
national Differences in Unit Nonresponse. In J. Harkness, M. Braun, B. Edwards, T.
Johnson, L.E. Lyberg, & P.Ph. Mohler (eds.), Survey Methods in Multinational,
Multiregional, and Multicultural Contexts (pp. 335–354). New York, NY: John Wiley
& Sons, Inc. doi: 10.1002/9780470609927
Blom, A.G., de Leeuw, E.D., & Hox, J.J. (2011). Interviewer Effects on Nonresponse in the
European Social Survey. Journal of Official Statistics, 27(2).
Blom, A.G. (2012). Explaining cross-country differences in survey contact rates: application
of decomposition methods. Journal of the Royal Statistical Society: Series A (Statistics
in Society), 175(1). doi: 10.1111/j.1467-985X.2011.01006.x
SPURIOUS DIFFERENCES IN RESPONSE RATES 14
ESS (2003). ESS round 1: Documentation report. London: European Social Survey.
Available at http://ess.nsd.uib.no/ess/round1/surveydoc.html.
Groves, R. M., & Couper, M.P. (1998). Nonresponse in Household Interview Surveys. New
York, NY: John Wiley & Sons, Inc.
Groves, R.M. & Peytcheva, E. (2008). The Impact of Nonresponse Rates on Nonresponse
Bias: A Meta-Analysis. Public Opinion Quarterly, 72(2), 167–89.
doi:10.1093/poq/nfn011
Keeter, S., Miller, C., Kohut, A., Groves, R.M., & Presser, S. (2000). Consequences of
Reducing Nonresponse in a National Telephone Survey. Public Opinion Quarterly
64(2), 125–48. doi:10.1086/317759
Kreuter, F. & Kohler, U. (2009). Analyzing Contact Sequences in Call Record Data. Potential
and Limitations of Sequence Indicators for Nonresponse Adjustments in the European
Social Survey. Journal of Official Statistics, 25(2), 203-226.
Kviz, F. J. (1977). Toward a Standard Definition of Response Rate. Public Opinion
Quarterly, 41(2), 265-67. doi:10.1086/268382
Lipps, O. & Pollien, A. (2011). Effects of Interviewer Experience on Components of
Nonresponse in the European Social Survey. Field Methods, 23(2), 156-172. doi:
10.1177/1525822X10387770
Lynn, P., Beerten, R., Laiho, J., & Martin, J. (2001). Recommended Standard Final Outcome
Categories and Standard Definitions of Response Rate of Social Surveys. ISER
Working Paper 2001-23. Colchester: ISER.
Lynn, P. & Clarke P. (2002). Separating Refusal Bias and Non-Contact Bias: Evidence From
UK National Surveys. Journal of the Royal Statistical Society: Series D (The
Statistician), 51(3), 319-33. doi:10.1111/1467-9884.00321
SPURIOUS DIFFERENCES IN RESPONSE RATES 15
Matsuo, H., Billiet, J., Loosveldt, G., & Malnar, B. (2007). Response Based Quality
Assessment of ESS Round 4. Results for 30 Countries Based on Contact Files. Leuven:
Centrum voor Sociologisch Onderzoek (CeSO), KU Leuven.
McCarty, C. (2003). Differences in Response Using Most Recent Versus Final Dispositions in
Telephone Surveys. Public Opinion Quarterly, 67(3), 396-406. doi:10.1086/377243
Smith, T. (2002). Developing Nonresponse Standards. In Groves, R.M., Dillman, D. A.,
Eltinge, J. L. & Little, R. J. A. (eds.), Survey Nonresponse (pp. 27-40). New York,
NY: John Wiley & Sons, Inc.
Stoop, I., Billiet, J., Koch, A., & Fitzgerald, R. (2010). Improving Survey Response. Lessons
Learned from the European Social Survey. Chichester: John Wiley & Sons, Inc.
Wagner, J. (2012). A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.
Public Opinion Quarterly, 76(3), 555-575. doi:10.1093/poq/nfs032
SPURIOUS DIFFERENCES IN RESPONSE RATES 16
Table 1
Priority ranking of response outcomes
Call outcomes Final disposition codes
interview completed interview completed
interview broken off / incomplete tbc at later date refusal interview broken off / never completed
interview broken off / incomplete not tbc refusal interview broken off / never completed
interview undefined refusal interview undefined, no record in main
data
ineligible not residential / institution ineligible not residential / institution
ineligible not residential / business ineligible not residential / business
ineligible not yet built ineligible not yet built
ineligible derelict ineligible derelict
ineligible not occupied ineligible not occupied
ineligible not traceable ineligible not traceable
ineligible other ineligible other
ineligible undefined ineligible undefined
contact respondent deceased ineligible respondent deceased
contact respondent moved abroad ineligible respondent moved abroad
contact refusal by respondent refusal refusal by respondent
contact refusal by someone else refusal refusal by someone else
contact appointment made with respondent refusal appointment with respondent, never
realized
contact appointment made with someone else contact appointment with someone else, never
realized
non-contact broken appointment refusal broken appointment
contact mentally / physically unable contact mentally / physically unable
contact language barrier contact language barrier
contact unavailable throughout fieldwork period contact unavailable throughout fieldwork period
contact temporarily unavailable contact temporarily unavailable
contact other eligible contact no interview contact other eligible contact no interview
contact undefined contact undefined
contact respondent moved, still in country non-contact respondent moved, still in country
non-contact respondent / household moved, new
address non-contact respondent / household moved new address
non-contact at home but no answer non-contact at home but no answer
non-contact nobody at home non-contact nobody at home
non-contact no access to housing unit non-contact no access to housing unit
non-contact other non-contact other
non-contact undefined non-contact undefined
missing call outcome missing missing case outcome / all call outcomes missing
Notes.
For each sample unit the call outcome with the highest ranking on the hierarchy determines the final disposition code.
The first column ranks the priority of call outcomes, with the top category displaying the outcome with the highest priority and
the bottom category that with the lowest priority. Column two lists the respective final disposition codes.
In the process of deriving final disposition codes from call outcomes, the label of an outcome code might need to be changed.
For example a call-level outcome ‘appointment’ that is not followed up by an ‘interview’, becomes a ‘broken appointment’ (i.e.
an implicit refusal) in the final disposition code. The same change in labeling takes place when deriving case outcomes by
means of most-recent coding.
Ineligibles and interviews, which both have final call outcomes that can directly determine the case outcome, are
nevertheless included in the hierarchy and derived from the call-level outcomes. The reason for this is that the ESS contact
forms do not explicitly distinguish between final and non-final call outcomes and in-office case outcomes. As a
consequence, interviewers were able to register another contact attempt after a final call outcome (e.g. interview or
ineligible).
SPURIOUS DIFFERENCES IN RESPONSE RATES 17
Table 2
Aggregate final disposition codes: most-recent versus priority coding
Final disposition (priority coding)
Final disposition
(most-recent coding)
Interview Ineligible
Refusal Other
contact Non-
contact Missing Total
Interview 31,447 0 0 0 0 0 31,447
Ineligible 0 2,083 0 0 0 0 2,083
Refusal 0 35 12,332 0 0 0 12,367
Other contact 0 11 407 2,353 0 0 2,771
Non-contact 0 27 866 403 3,382 0 4,678
Missing 0 0 8 10 4 106 128
Total 31,447 2,156 13,613 2,766 3,386 106 53,474
Notes. Aggregate numbers for the disposition categories interview, ineligible, refusal, other contact, non-contact and missing
outcome across 16 countries. The counts were adjusted for cases where the interviewer failed to record an interview outcome
in the call-record data but interview data were available.
SPURIOUS DIFFERENCES IN RESPONSE RATES 18
Call outcomes Final disposition codes
In-office case outcome
(supersedes any call outcome)
Final call outcome
(with or without non-final call outcomes)
Final case outcome
Non-final call outcomes
(only)
Derived final case outcome
FIGURE 1. Deriving final disposition codes from contact data
SPURIOUS DIFFERENCES IN RESPONSE RATES 19
FIGURE 2. Differences in response rates across countries by coding strategy. Notes. The x-
axis denotes the outcome rates derived from the final dispositions provided by the countries.
+
country coding – priority coding, X country coding – most-recent coding.
SPURIOUS DIFFERENCES IN RESPONSE RATES 20
1
The 16 countries assessed are Belgium, Germany, Finland, Hungary, Norway,
Poland, Slovenia, Austria, Spain, UK, Greece, Ireland, Italy, Luxembourg, Netherlands, and
Portugal. Norway did not provide detailed outcomes of the interview, non-contact and
ineligible codes; the smaller differences between the priority and most-recent coding might be
due to this.

Supplementary resource (1)

... We also consider how robust the targeting is to misclassification of the outcome variable and to changes in the response propensity model. We consider misclassification by defining refusal in two ways: (i) as recorded by the interviewer and (ii) by including those cases (n ¼ 166) that were noncontacts at Wave 2 and refusals at Wave 3 as noncontact can sometimes be a hidden refusal (Blom 2014). ...
Article
Full-text available
We review two approaches for improving the response in longitudinal (birth cohort) studies based on response propensity models: strategies for sample maintenance in longitudinal studies and improving the representativeness of the respondents over time through interventions. Based on estimated response propensities, we examine the effectiveness of different re-issuing strategies using Representativity Indicators (R-indicators). We also combine information from the Receiver Operating Characteristic (ROC) curve with a cost function to determine an optimal cut point for the propensity not to respond in order to target interventions efficiently at cases least likely to respond. We use the first four waves of the UK Millennium Cohort Study to illustrate these methods. Our results suggest that it is worth re-issuing to the field nonresponding cases from previous waves although re-issuing refusals might not be the best use of resources. Adapting the sample to target subgroups for re-issuing from wave to wave will improve the representativeness of response. However, in situations where discrimination between respondents and nonrespondents is not strong, it is doubtful whether specific interventions to reduce nonresponse will be cost effective.
... All response rates presented in this paper were calculated using priority-coded final dispositions(Blom 2013). ...
Article
Full-text available
The paper looks into the processes and outcomes of setting-up and maintaining a probability-based longitudinal online survey, which is recruited face-to-face and representative of both the online and offline population aged 16 to 75 in Germany. This German Internet Panel (GIP) studies political and economic attitudes and reform preferences through bi-monthly longitudinal online interviews of individuals. The results presented demonstrate that a carefully designed and implemented online panel can produce high-quality data at lower marginal costs than existing panels that operate solely in face-to-face mode. Analyses into the representativeness of the online sample showed no major coverage or nonresponse biases. Finally, including offline households in the panel is important as it improves the representation of the older and female segments of the population.
Article
Research on nonresponse in face-to-face surveys in the United States has shown that nonresponse increases over time for most surveys, but there are also periods of fairly stable rates. Surprisingly, nonresponse for face-to-face surveys has not been as widely studied recently as compared to telephone surveys. The focus on telephone surveys may be due to the dramatic increase in nonresponse for these surveys or perhaps because face-to-face surveys still achieve relatively high levels of response. This paper updates nonresponse trends for face-to-face household surveys conducted in the United States since 2000. The review provides a comprehensive picture of the industry by including surveys conducted by government, private, and academic organizations. The relative role of refusals and noncontacts in the total nonresponse is also presented. We tie the trends in nonresponse with data on the level of effort for some of these surveys. Many researchers have suggested that extra effort has been needed to prevent response rates from falling even more precipitously but lacked the effort data to evaluate this hypothesis for face-to-face surveys. Some data on field effort are becoming available, so this question can be addressed for the first time across more than one survey. To complete the picture, we also look at loss over time for longitudinal face-to-face surveys.
Chapter
Das Robert Koch-Institut führt seit den frühen 2000er Jahren telefonische Gesundheitsssurveys durch. Kernthemen sind hierbei Einflussfaktoren auf die Gesundheit, die Inanspruchnahme von Leistungen des Gesundheitssystems sowie die Erfassung von chronischen Krankheiten. Hierfür wurde unter anderem ein eigenes CATI-Labor in der Abteilung Epidemiologie und Gesundheitsmonitoring aufgebaut.
Article
Full-text available
Inferential statistics teach us that we need a random probability sample to infer from a sample to the general population. In online survey research, however, volunteer access panels, in which respondents self-select themselves into the sample, dominate the landscape. Such panels are attractive due to their low costs. Nevertheless, recent years have seen increasing numbers of debates about the quality, in particular about errors in the representativeness and measurement, of such panels. In this article, we describe four probability-based online and mixed-mode panels for the general population, namely, the Longitudinal Internet Studies for the Social Sciences (LISS) Panel in the Netherlands, the German Internet Panel (GIP) and the GESIS Panel in Germany, and the Longitudinal Study by Internet for the Social Sciences (ELIPSS) Panel in France. We compare them in terms of sampling strategies, offline recruitment procedures, and panel characteristics. Our aim is to provide an overview to the scientific community of the availability of such data sources to demonstrate the potential strategies for recruiting and maintaining probability-based online panels to practitioners and to direct analysts of the comparative data collected across these panels to methodological differences that may affect comparative estimates.
Article
Full-text available
In face-to-face surveys interviewers play a crucial role in making contact with and gaining cooperation from sample units. While there are a few studies investigating the influence of interviewers on nonresponse, they are typically restricted to analyses within a single country. However, interviewer training, contacting and cooperation strategies as well as survey climates differ across countries thus influencing differential nonresponse processes and possibly nonresponse biases. Combining call-record data from the European Social Survey (ESS) with data from a detailed interviewer questionnaire on attitudes and doorstep behavior we analyze interviewer and country effects on nonresponse. Our findings show that there are systematic differences between countries in contacting and cooperation processes, which can in part be explained by differences in interviewer characteristics, such as contacting strategies and avowed doorstep behavior.
Article
Full-text available
The authors analyze interviewer-related nonresponse differences in face-to-face surveys, distinguishing three types of interviewers: those who have previous experience with the same high standard cross-sectional survey (‘‘experienced’’), those who were chosen by the survey agency to complete refusal conversions (‘‘seniors’’), and usual interviewers. The nonresponse components are obtaining household contact, target person contact, and target person cooperation. In addition, the authors consider whether interviewer homogeneity with respect to these components is different across the three interviewer groups. Data come from the European Social Survey (ESS) contact forms from four countries that participated in the rounds of 2002, 2004, and 2006 and used the same survey agency that, in turn, used the same interviewers to some extent. To analyze interviewer effects, the authors use discrete two-level models. The authors find some evidence of better performance by both senior and experienced interviewers and indications of greater homogeneity for nonresponse components, especially for those with room for improvement. Surprisingly, the senior interviewers do not outperform the experienced ones. The authors conclude that survey agencies should make more efforts to decrease the comparatively high interviewer turnover.
Article
Full-text available
Fifty-nine methodological studies were designed to estimate the magnitude of nonresponse bias in statistics of interest. These studies use a variety of designs: sampling frames with rich variables, data from administrative records matched to sample case, use of screening-interview data to describe nonrespondents to main interviews, followup of nonrespondents to initial phases of field effort, and measures of behavior intentions to respond to a survey. This permits exploration of which circumstances produce a relationship between nonresponse rates and nonresponse bias and which, do not. The predictors are design features of the surveys, characteristics of the sample, and attributes of the survey statistics computed in the surveys.
Article
An Introduction to Survey Participation. A Conceptual Framework for Survey Participation. Data Resources for Testing Theories of Survey Participation. Influences on the Likelihood of Contact. Influences of Household Characteristics on Survey Cooperation. Social Environmental Influences on Survey Participation. Influences of the Interviewers. When Interviewers Meet Householders: The Nature of Initial Interactions. Influences of Householder-Interviewer Interactions on Survey Cooperation. How Survey Design Features Affect Participation. Practical Survey Design Acknowledging Nonresponse. References. Index.
Article
In recent years more and more survey organizations have begun collecting data at every contact attempt; such call record data are increasingly and successfully used for fieldwork management purposes, often to enhance efforts to reduce nonresponse. These paradata are also seen as candidates for post-survey nonresponse adjustment because they are available for both respondents and nonrespondents. In the past, summary measures from contact data, for example the number of contact attempts or the occurrence of a soft refusal, have been used to study nonresponse. Such summary measures are not without problems. Here we will discuss an alternative set of indicators that take the sequence of contacts into account. We examine different characterizations of contact sequences. The richness of the European Social Survey data allows us to discuss reliability by replicating our analyses across three time points and fourteen countries.
Article
High response rates have traditionally been considered as one of the main indicators of survey quality. Obtaining high response rates is sometimes difficult and expensive, but clearly plays a beneficial role in terms of improving data quality. It is becoming increasingly clear, however, that simply boosting response to achieve a higher response rate will not in itself eradicate nonresponse bias. In this book the authors argue that high response rates should not be seen as a goal in themselves, but rather as part of an overall survey quality strategy based on random probability sampling and aimed at minimising nonresponse bias. Key features of Improving Survey Response: A detailed coverage of nonresponse issues, including a unique examination of cross-national survey nonresponse processes and outcomes. A discussion of the potential causes of nonresponse and practical strategies to combat it. A detailed examination of the impact of nonresponse and of techniques for adjusting for it once it has occurred. Examples of best practices and experiments drawn from 25 European countries. Supplemented by the European Social Survey (ESS) websites, containing materials for the measurement and analysis of nonresponse based on detailed country-level response process datasets. The book is designed to help survey researchers and those commissioning surveys by explaining how to prioritise the reduction of nonresponse bias rather than focusing on increasing the overall response rate. It shows substantive researchers how nonresponse can impact on substantive outcomes.
Book
Over the past two decades, the relevance of cross-national and cross-cultural methodologies has heightened across various fields of study. Responding to increasing cultural diversity and rapid changes in how research is conducted, Survey Methods in Multinational, Multiregional, and Multicultural Contexts addresses the need for refined tools and improved procedures in cross-cultural and cross-national studies worldwide. Based on research submitted to the International Conference on Multinational, Multicultural, and Multiregional Survey Methods (3MC), this book identifies important changes in comparative methodology approaches, outlines new findings, and provides insight into future developments in the field. Some of the world's leading survey researchers gather in this volume to address the need for a standard framework that promotes quality assurance and quality control in survey research, and its impact on various stages of the survey life cycle, including study design and organization, cross-national sampling, testing and pretesting, data collection, and input and output variable harmonization. Self-contained chapters feature coverage of various topics, such as: Question and questionnaire design, from both global and study-specific perspectives The construction and evaluation of survey translations and instrument adaptations The effects of cultural difference on the perception of question and response categories Non-response issues Analysis in comparative contexts, featuring discussion of polytomous item response theory, categorization problems, and Multi-Trait-Multi-Methods (MTMM) The significance of evolving methodologies for current international survey programs, including the European Social Survey, the International Social Survey Programme, and the Gallup World Poll Survey Methods in Multinational, Multiregional, and Multicultural Contexts is a valuable supplement for courses on comparative survey methods at the upper-undergraduate and graduate levels. It also serves as an insightful reference for professionals who design, implement, and analyze comparative research in the areas of business, public health, and the social and behavioral sciences.