Article

Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Linking paradata from a survey on labor market participation to administrative data, this article has a unique opportunity to examine nonresponse and measurement error jointly and provide estimates for bias, variance, and mean square error. We find that increased contact attempts resulted in significant reductions in nonresponse bias. Measurement error increased somewhat with increased effort, though total bias was nevertheless reduced. One of the key survey indicators revealed an interesting case of counteracting effects of nonresponse and measurement error whereby increased effort to recruit nonrespondents led to an increase in mean square error with increased level of effort despite a reduction in nonresponse bias and constant measurement error.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Such errors could offset gains in response accuracy achieved by the change to a self-administered mode. Our second motivation for using these items is because benefit receipt is commonly underreported and benefit recipients tend to be overrepresented in economic studies (Meyer, Mok, and Sullivan 2009;Kreuter, Müller, andTrappmann 2010, 2014;Bruckmeier, Müller, and Riphahn 2015). Thus, these items represent an interesting scenario in which nonresponse and measurement error biases are likely to counteract, which may have implications for the total bias of the estimates after the NRFU. ...
... These administrative variables have been found to be highly reliable as they are routinely used to track social security contributions and administer social services (Bender and Haas 2002;Jacobebbinghaus and Seth 2007). They have been used in other survey measurement error studies (Kreuter, Müller, andTrappmann 2010, 2014;Sakshaug and Kreuter 2012;Eckman et al. 2014). Thus, we are comfortable in using these records as a gold standard against which we evaluate measurement error bias in the survey estimates. ...
... It is apparent from the CATI survey that respondents and nonrespondents differ on most of the administrative variables: CATI respondents are significantly more likely than CATI nonrespondents to be older, female, of German nationality, ever employed as part-time or full-time, lower full-time but higher part-time monthly earners, and recipients of unemployment benefit and income assistance in 2010. The overrepresentation of benefit recipients is consistent with other survey nonresponse findings (Kreuter, Müller, and Trappmann 2010). All of these differences suggest that nonresponse bias is present in the CATI survey. ...
Article
To evaluate and adjust for nonresponse bias in household surveys, many social science studies conduct follow-up surveys with nonrespondents. By recruiting additional respondents, the goal of nonresponse follow-up (NRFU) surveys is to reduce nonresponse bias and make the respondent pool more representative of the characteristics of the sample as a whole. Often a change of data collection mode or a shorter questionnaire is implemented to increase response rates. However, whether these design features actually reduce nonresponse bias is usually unknown. What is also unknown is the effect of NRFU studies on measurement error, particularly when interviewer- and self-administered modes are used to administer sensitive questions susceptible to misreporting. Few studies have explicitly examined the joint impact of nonresponse and measurement error bias in NRFU surveys due to the lack of auxiliary validation data about the respondents and nonrespondents. We overcome this deficiency in an economic survey initially administered by telephone with mail nonresponse follow-up and administrative validation records available for the entire sample. This situation permits the estimation of both nonresponse and measurement error bias before and after the NRFU. We find that the NRFU survey succeeds in bringing in respondents who differ from the telephone respondents, but that these additional respondents are not always representative of the final nonrespondents. This results in reduced nonresponse bias for some items, but increased nonresponse bias for others. We also find that combining the mail NRFU respondents with the telephone respondents reduces measurement error bias for economic estimates. Lastly, we report a paradoxical finding in which adding NRFU respondents to the respondent pool produces greater total bias in some survey estimates despite reducing both nonresponse and measurement error bias separately. We conclude with a discussion of the practical implications of these findings and speculate on their possible causes. © The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved.
... These variables were chosen based on their common usage in labor market research and because they are linked to several surveys in Germany, including the aforementioned PASS, ALWA, and NEPS. These variables have also been used in several methodological research studies (Kreuter, Müller, and Trappmann 2010;West, Kreuter, and Jaenichen 2013;Eckman, Kreuter, Kirchner, J€ ackle, Tourangeau, et al. 2014;Sakshaug and Huber 2016). All numeric variables were categorized somewhat arbitrarily after inspecting their distributions. ...
... The largest discrepancy between the two groups occurs for the employment variable group, where 6 (out of 10) estimates exhibit meaningful biases in the CAPI group, compared to only 2 estimates in the mail/Web group. The CAPI biases indicate a strong underrepresentation of individuals consistently employed in the last 5 years (e.g., zero employment changes, most days employed in current establishment)-a result that is consistent with other findings in Germany showing that employed persons are underrepresented when a minimal number of interviewer contact attempts is carried out (Kreuter et al. 2010). ...
Article
Full-text available
Panel surveys are increasingly experimenting with the use of self-administered modes of data collection as alternatives to more expensive interviewer-administered modes. As data collection costs continue to rise, it is plausible that future panel surveys will forego interviewer administration entirely. We examine the implications of this scenario for recruitment bias in the first wave of a panel survey of employees in Germany. Using an experimental multi-mode design and detailed administrative record data available for the full sample, we investigate the magnitude of two sources of panel recruitment bias: nonresponse and panel consent (i.e., consent to follow-up interview). Across 29 administrative estimates, we find relative measures of aggregate nonresponse bias to be comparable between face-to-face and self-administered (mail/Web) recruitment modes, on average. Furthermore, we find the magnitude of panel consent bias to be more severe in self-administered surveys, but that implementing follow-up conversion procedures with the non-consenters diminishes panel consent bias to near-negligible levels. Lastly, we find the total recruitment bias (nonresponse and panel consent) to be similar in both mode groups—a reassuring result that is facilitated by the panel consent follow-up procedures. Implications of these findings for survey practice and suggestions for future research are provided in conclusion.
... For example, to assess measurement bias, external records can be used to assess the accuracy of respondents' self-reports (e.g.,Olson 2006;Kreuter et al. 2008;Sakshaug et al. 2010;Tourangeau et al. 2010). To assess nonresponse bias, these auxiliary data are needed for both respondents and nonrespondents (e.g.,Klausch et al. 2015a;Kreuter et al. 2010;Kappelhof 2013). As such data are rarely available to researchers, the potential utility of the MSE as a metric for evaluating the effects of different survey design features or for comparing whole survey systems (Biemer 1988) has not been fully exploited. ...
... However, these studies have focused mainly on between-mode comparisons rather than on the cumulative effects on measurement error of combining data from different modes. Similarly, efforts to compute the MSE of survey estimates (e.g.,Groves and Magilavy 1984;Peytchev et al. 2009) have tended to focus on single mode scenarios, even where mixed mode data were available (Kreuter et al. 2010;Olson 2006). These studies confirm that the MSE varies considerably by estimate, and changes as a result of efforts to reduce nonresponse. ...
Article
Full-text available
Mixed mode data collection designs are increasingly being adopted with the hope that they may reduce selection errors in single mode survey designs. Yet possible reductions in selection errors achieved by mixing modes may be offset by a potential increase in total survey error due to extra measurement error being introduced by the additional mode(s). Few studies have investigated this empirically, however. In the present study, we compute the Mean Squared Error (MSE) for a range of estimates using data from a mode comparison experiment. We compare two mixed mode designs (a sequential web plus mail survey, and a combined concurrent and sequential CATI plus mail survey) with a single mode mail survey. The availability of auxiliary data on the sampling frame allows us to estimate several components of MSE (sampling variance, non-coverage, nonresponse and measurement bias) for a number of sociodemographic and target variables. Overall, MSEs are lowest for the single mode survey, and highest for the CATI plus mail design, though this pattern is not consistent across all estimates. Mixing modes generally reduces total bias, but the relative contribution to total survey error from different sources varies by design and by variable type.
... While the first allows research into measurement error, the latter also allows research into non-response error. 33 For example, Kreuter et al. 33 have shown that initial non-response bias of means and proportions vanishes over the course of the fieldwork and that at the same time measurement error bias of these means and proportions does not increase. For welfare benefit receipt there is initially a substantial measurement error bias that decreases across time. ...
... While the first allows research into measurement error, the latter also allows research into non-response error. 33 For example, Kreuter et al. 33 have shown that initial non-response bias of means and proportions vanishes over the course of the fieldwork and that at the same time measurement error bias of these means and proportions does not increase. For welfare benefit receipt there is initially a substantial measurement error bias that decreases across time. ...
... Sturgis et al. (2017) have demonstrated that the reduction of potential nonresponse bias for these surveys is rather high during the first additional call attempts and subsides after about the fifth call attempt across the analyzed surveys. Kreuter, Müller, and Trappmann (2010) have reached similar conclusions regarding the significant reduction of nonresponse bias resulting from increased call attempts. Still other studies have focused on the post-survey evaluation of the productivity of fieldwork per unit in time-the so-called fieldwork power Vandenplas, Loosveldt, and Beullens 2017). ...
... The studies that focused on data quality provided by early versus late respondents have defined respondents as being early or late in terms of time passed since invitation (e.g., Wellman et al. 1980;Kruse et al. 2010;Sigman et al. 2014) or (non)response after certain fieldwork effort such as additional reminders (e.g. Kypri et al. 2011), additional contact attempts (e.g., Ullman and Newcomb 1998;D ıaz de Rada 2005;Rao and Pennington 2013;Kreuter et al. 2014), or other combinations of fieldwork efforts (e.g., Donald 1960). Yet, others have used combinations of the time and effort dimensions to connect data quality and response timing, basing the distinctions between early and late respondents on the distributions of completed interviews by date (e.g., Dalecki et al. 1993;Bates and Creighton 2000;Irani et al. 2004). ...
Article
Full-text available
Survey researchers are often confronted with the question of how long to set the length of the field period. Longer fielding time might lead to greater participation yet requires survey managers to devote more of their time to data collection efforts. With the aim of facilitating the decision about the length of the field period, we investigated whether a longer fielding time reduces the risk of nonresponse bias to judge whether field periods can be ended earlier without endangering the performance of the survey. By using data from six waves of a probability-based mixed-mode (online and mail) panel of the German population, we analyzed whether the risk of nonresponse bias decreases over the field period by investigating how day-by-day coefficients of variation develop during the field period. We then determined the optimal cut-off points for each mode after which data collection can be terminated without increasing the risk of nonresponse bias and found that the optimal cut-off points differ by mode. Our study complements prior research by shifting the perspective in the investigation of the risk of nonresponse bias to panel data as well as to mixed-mode surveys, in particular. Our proposed method of using coefficients of variation to assess whether the risk of nonresponse bias decreases significantly with each additional day of fieldwork can aid survey practitioners in finding the optimal field period for their mixed-mode surveys.
... These records can be aggregated to the person level and contain histories of employment, unemployment, job search and benefit receipt, as well as records of ALMP participation. Although the administrative data are not free of error, their overall quality is very high, and they are used by the German Government to calculate pension claims, to administer benefit claims and to make payments (Jacobebbinghaus and Seth, 2007;Köhler and Thomsen, 2009;Kreuter et al., 2010). In general, data on benefits, ALMP and job search are of the highest quality, because they are generated by activities of the Federal Employment Agency itself (Jacobebbinghaus and Seth, 2007). ...
... These people are non-compliers, i.e. they did not respond to the survey, their assigned treatment status. Non-response and attrition in surveys have many causes and can lead to bias or endogenous selection into treatment (Groves et al., 1992(Groves et al., , 2000Abadie, 2003;Kreuter et al., 2010). For example, people who agree to participate in three waves of a survey may be more compliant and thus more likely to participate in ALMPs, even without the treatment of the survey (e.g. ...
Article
Full-text available
Panel survey participation can bring about unintended changes in respondents’ behaviour and/or their reporting of behaviour. Using administrative data linked to a large panel survey, we analyse whether the survey brings about changes in respondents’ labour market behaviour. We estimate the causal effect of panel participation on the take‐up of federal labour market programmes by using instrumental variables. Results show that panel survey participation leads to an increase in respondents’ take‐up of these measures. These results suggest that panel survey participation not only affects the reporting of behaviour, as previous studies have demonstrated, but can also alter respondents’ actual behaviour.
... Closely related to this, such a norm might also make it socially undesirable to receive UB II. This could explain why some people misreport not to receive the benefits to the PASS survey (Kreuter, Müller andTrappmann 2010, Bruckmeier, Müller andRiphahn 2014). These workers probably suffer from receiving UB II the most as they are even willing to hide this circumstance from an anonymous survey. ...
... In spite of the relatively low response rates, several articles utilizing high-quality administrative data have shown that nonresponse bias is rather small for a range of variables such as benefit receipt, employment status, income, age, or disability (Kreuter, Mü ller, & Trappmann, 2010;Levenstein, 2010;Sakshaug & Kreuter, 2012). Foreign nationals have found to be considerably underrepresented (Kreuter et al., 2010), but weighting can adjust for this. Due to the oversampling of welfare benefit recipients, respondents to PASS Wave 11 are, before weights are applied, more likely to receive those benefits than the general population (30.0% ...
Article
The new European General Data Protection Regulation (GDPR) imposes enhanced requirements on digital data collection. This article reports from a 2018 German nationwide population-based probability app study in which participants were asked through a GDPR compliant consent process to share a series of digital trace data, including geolocation, accelerometer data, phone and text messaging logs, app usage, and access to their address books.With about 4,300 invitees and about 650 participants, we demonstrate (1) people were just as willing to share such extensive digital trace data as they were in studies with far more limited requests; (2) despite being provided more decision-related information, participants hardly differentiated between the different data requests made; and (3) once participants gave consent, they did not tend to revoke it. We also show (4) evidence for a widely-held belief that explanations regarding data collection and data usage are often not read carefully, at least not within the app itself, indicating the need for research and user experience improvement to adequately inform and protect participants. We close with suggestions to the field for creating a seal of approval from professional organizations to help the research community promote the safe use of data.
... Hence, studies designed to assess the presence of ME are essential to carry out adequate adjustments. A number of studies have explored the problems of measurement error affecting retrospectively reported work histories (Biemer, 2011;Kreuter et al., 2010;Manzoni et al., 2011Manzoni et al., , 2010Summers, 1984, 1995;Pyy-Martikainen and Rendtel, 2009). However, we are only aware of one study (Pyy-Martikainen and Rendtel, 2009) using a register as validation data to assess the different types of ME found in these types of questions, and even here we should note some important differences from our study. ...
Article
Full-text available
We use work histories retrospectively reported and matched to register data from the Swedish unemployment office to assess: 1) the prevalence of measurement error in reported spells of unemployment; 2) the impact of using such spells as the response variable of an exponential model; and 3) strategies for the adjustment of the measurement error. Due to the omission or misclassification of spells in work histories we cannot carry out typical adjustments for memory failures based on multiplicative models. Instead we suggest an adjustment method based on a mixture Bayesian model capable of differentiating between misdated spells and those for which the observed and true durations are unrelated. This adjustment is applied in two manners, one assuming access to a validation subsample and another relying on a strong prior for the mixture mechanism. Both solutions demonstrate a substantial reduction in the vast biases observed in the regression coefficients of the exponential model when survey data is used.
... Second, employed persons have a higher likelihood of having internet access at home than the general population (US Census Bureau 2019; German Federal Office of Statistics 2020a). Third, employees are more difficult to reach via interviewer-administration (Yan, Tourangeau, and Arens 2004;Asef and Riede 2006;Kreuter, Müller, and Trappmann 2010;Guzy 2015), as they are more likely to be occupied during the day or evenings (Knabe, R€ atzel, Schöb, and Weimann 2010). Given these factors, the employed population may benefit the most from being offered an online mode as opposed to an interviewer-administered one. ...
Article
Full-text available
Policy decisions in business and economic fields are often informed by surveys of employees. Many employee surveys use costly interviewer-administered modes to reach this special population. However, certain employee subgroups may be especially hard to reach using these modes. Thus, besides high administration costs, nonresponse bias is a concern. To reduce costs and potential nonresponse bias, some employee surveys have introduced web as part of a sequential mixed-mode design. However, the impact of introducing web on response rates, nonresponse bias, and costs in employee surveys is understudied. The present study addresses this research gap by analyzing a mode design experiment in which employees selected for a national survey in Germany were randomly assigned to a single-mode telephone design or a sequential web-telephone mixed-mode design. The study revealed four main findings. First, introducing the web mode significantly increased the response rate compared to the single-mode design. Second, despite the higher response rate, aggregate nonresponse bias was higher in the mixed-mode design than in the single-mode design. Third, the likelihood of web participation varied across certain employee subgroups, including occupation type and employment contract. Lastly, potential cost savings were evident under the mixed-mode design.
... Finally, we have extended the previous single mode design to a sequential mixed-mode design (Couper, 2011), with the implementation of computer-assisted telephone interviews (CATI) in addition to the online questionnaire (De Leeuw et al., 2008). This proceeding is considered to be more efficient to increase the cooperation of respondents and effective to convert noncooperating individuals for survey participation Hox et al., 2015;Kreuter et al., 2010) than the simultaneous offer of different modes (Dillman et al., 2009;Krug et al., 2017;Millar and Dillman, 2011). ...
Article
In this contribution, we evaluate the short- and long-term effects of a prepaid cash incentive on young people’s cooperation and response rate in the fourth and fifth wave of a panel with sequential mixed-mode design (online questionnaire, CATI). Analyses are based on a survey experiment of students from randomly selected school classes of equal shares, which have participated in the third wave. Findings show that a monetary incentive has a direct and positive effect on the response rate in the fourth but not in the subsequent wave. However, the effect of the incentive is not persistent, since the effect weakens and fades away during the field phase and cannot be directly transferred to the second survey mode. As emphasized in the tailored design method (TDM), a monetary incentive can contribute to a shorter field phase and hence lower costs, but it is an insufficient instrument against panel attrition and the optimization of the retention rate when other strategies are disregarded.
... Linkage of survey and administrative data appears to be an efficient way to obtain reliable health profile data (Hure et al., 2015;Gresham et al., 2015;Kreuter et al., 2010;Young et al., 2001;Sakshaug and Kreuter, 2012;Cullen et al., 2006). Our analysis draws on the strengths of two distinct yet complementary datasets frequently used in asthma epidemiology: a community-based survey and medical claims data. ...
Article
Full-text available
Community-level approaches for pediatric asthma management rely on locally collected information derived primarily from two sources: claims records and school-based surveys. We combined claims and school-based surveillance data, and examined the asthma-related risk patterns among adolescent students. Symptom data collected from school-based asthma surveys conducted in Oakland, CA were used for case identification and determination of severity levels for students (high and low). Survey data were matched to Medicaid claims data for all asthma-related health care encounters for the year prior to the survey. We then employed recursive partitioning to develop classification trees that identified patterns of demographics and healthcare utilization associated with severity. A total of 561 students had complete matched data; 86.1% were classified as high-severity, and 13.9% as low-severity asthma. The classification tree consisted of eight subsets: three indicating high severity and five indicating low severity. The risk subsets highlighted varying combinations of non-specific demographic and socioeconomic predictors of asthma prevalence, morbidity and severity. For example, the subset with the highest class-prior probability (92.1%) predicted high-severity asthma and consisted of students without prescribed rescue medication, but with at least one in-clinic nebulizer treatment. The predictive accuracy of the tree-based model was approximately 66.7%, with an estimated 91.1% of high-severity cases and 42.3% of low-severity cases correctly predicted. Our analysis draws on the strengths of two complementary datasets to provide community-level information on children with asthma, and demonstrates the utility of recursive partitioning methods to explore a combination of features that convey asthma severity.
... Second, many researchers advocated using callbacks to reduce the non-response rate. (Stoop, 2004, Kreuter et al., 2010, Olson, 2013 While the developing world has witnessed a massive expansion of mobile phone and broadband networks, such means of contacting sampled units remain practically infeasible in impoverished areas where telephones and computers are not affordable or in sparsely populated areas without easy access to such networks. Third, it is often difficult to rule out that non-response is non-informative. ...
Technical Report
Full-text available
Non-response is a commonly encountered problem in many population-based surveys. Broadly speaking, non-response can be due to refusal or failure to contact the sample units. Although both types of non-response may lead to bias, there is much evidence to indicate that it is much easier to reduce the proportion of non-contacts than to do the same with refusals. In this article, we use data collected from a nationally-representative survey under the Demographic and Health Surveys program to study non-response due to refusals to HIV testing in Malawi. We review existing estimation methods and propose novel approaches to the estimation of HIV prevalence that adjust for refusal behaviour. We then explain the data requirement and practical implications of the conventional and proposed approaches. Finally, we provide some general recommendations for handling non-response due to refusals and we highlight the challenges in working with Demographic and Health Surveys and explore dierent approaches to statistical estimation in the presence of refusals. Our results show that variation in the estimated HIV prevalence across dierent estimators is due largely to those who already know their HIV test results. In the case of Malawi, variations in the prevalence estimates due to refusals for women are larger than those for men.
... Remarkably, we know little about how these design choices affect total survey error, making it hard to allocate resources to limit attrition or measurement error (Lynn & Lugtig, 2017). Several validation studies in recent years have tried to study the relative contributions of nonresponse and measurement error for different questions and survey modes (Kreuter, Müller, & Trappmann, 2010, 2013. Felderer, Kirchner, and Kreuter (2013) for example use administrative records to study errors in social demographic variables, and benefit receipt in Germany in a randomized Web/Telephone study. ...
Article
Full-text available
This paper proposes a method to simultaneously estimate both measurement and nonresponse errors for attitudinal and behavioural questions in a longitudinal survey. The method uses a Multi-Trait Multi-Method (MTMM) approach, which is commonly used to estimate the reliability and validity of survey questions. The classic MTMM model is in this paper extended to include the e�ects of measurement bias and longitudinal nonresponse that occurs in longitudinal surveys. Measurement and nonresponse errors are expressed on a common metric in this model, so that their relative sizes can be assessed over the course of a panel study. Using an example about political trust from the Dutch LISS panel, we show that measurement problems lead to both small errors and small biases, that dropout in the panel study does not lead to errors or bias, and that therefore, measurement is a more important source of both error and bias than nonresponse.
... Wie bei fast jeder Befragung wird die Repräsentativität durch das Bias-Problem limitiert [88,89]. Es lassen sich leider keine Angaben dazu machen, warum Mitarbeiter an der durchgeführten Befragung teilgenommen haben oder nicht. ...
Article
Zusammenfassung Ziel der Studie Trotz zahlreich berichteter Defizite und einem zunehmend angespannten Arbeitsumfeld in deutschen Krankenhäusern wird die Arbeitszufriedenheit von den Mitarbeitern regelmäßig als hoch bis sehr hoch einschätzt. Damit fehlen wichtige Argumente zur nachhaltigen Verbesserung von Arbeitsbedingungen gegenüber den Vorständen. Aus dieser Diskrepanz zwischen Arbeitsbedingungen und subjektiver Zufriedenheit ergab sich die Motivation für die vorliegende Arbeit. Methodik Die Datenerhebung erfolgte mittels einer Mitarbeiterbefragung am Kinderzentrum des Universitätsklinikum Leipzig AöR. Unterteilt wurde nach ärztlichem, Pflege- und Funktionsdienst. Gemessen wurde die subjektive Arbeitszufriedenheit anhand der fast ausschließlich in Unternehmen verwendeten klassischen Globalurteile vs. qualitative Arbeitszufriedenheit nach dem weit moderneren kognitiv-emotionalen Konzept des „Schweizer Modells“. Darüber hinaus wurden Arbeitszeit, arbeitsbedingte psychische Belastungsfolgen, Kündigungsbereitschaft und reale Austritte in Relation gesetzt. Ergebnisse Die Auswertung der klassischen Globalurteile zeigt hohe Arbeitszufriedenheitsquoten. Im Vergleich dazu ergibt die qualitative Analyse, dass nur jeder vierte Mitarbeiter und bei den Ärzten sogar nur jeder zehnte real mit seinem Job zufrieden ist. Insgesamt resultieren 4 Zufriedenheitsgruppen. Deutliche Unterschiede zwischen den Berufsgruppen zeigen die altersstratifizierten Profile und die Auswertung der psychischen Belastungsfolgen. Konträr verhält sich die Kündigungsbereitschaft. Diskussion Bei den vielfach publizierten Ergebnissen hoher Arbeitszufriedenheit in Krankenhäusern handelt es sich in der Mehrzahl um resignativ zufriedene Mitarbeiter. Dies resultiert psychodynamisch aus einer Spannungskompensation. Mit den klassischen Globalurteilen ist dies nicht messbar. Ebenso undetektiert bleiben in der Regel die konstruktiv unzufriedenen Mitarbeiter, welche für Unternehmen eine oftmals unterschätzte Ressource darstellen. Deren Spannungsreduktion resultiert dann meist in einem Jobwechsel.
... Several surveys in Germany link to these data, including the PASS study, the National Educational Panel Study (NEPS), and the "Working and Learning in a Changing World (ALWA)" study. The high level of interest in these variables has also been exhibited in other survey methodological research (Kirchner, 2015;Kreuter, Müller, & Trappmann, 2010;West, Kreuter, & Jaenichen, 2013). 6 The Integrated Employment Biographies (IEB) data are sensitive administrative data which are available for the researchers at the Institute for Employment Research (IAB; http://www.iab.de) in Nuremberg, Germany. ...
Article
Full-text available
In an effort to reduce data collection costs survey organizations are considering more costeffective means of data collection. Such means include greater use of self-administered interview modes and acquiring substantive information from external administrative records conditional on respondent consent. Yet, little is known regarding the implications of requesting record linkage consent under self-administered survey modes with respect to consent rates and consent bias. To address this knowledge gap, we report results from a linkage consent study in which employees in an employment survey were randomly assigned to an intervieweradministered (face-to-face) or self-administered (mail/web) interview, which included a consent question to link to federal employment records. We observed a strikingly lower linkage consent rate in the self-administered (53.9 percent) versus the interviewer-administered (93.9 percent) survey mode. However, the impact of survey mode on linkage consent bias was much less severe: survey-measured correlates of linkage consent did not interact with mode and relative consent biases in the linked-administrative variables tended to be small (less than 6 percentage points) under both mode groups; though, linkage consent biases in the administrative variables were larger in the self-administered mode group compared to the intervieweradministered mode group, on average. We discuss the implications of these findings for survey practice and speculate on their possible causes.
... The primary rationale for using these administrative variables in our analysis of the NRFU survey is that they are commonly utilized in labor market research studies and are key measures used by the Federal Employment Agency to administer social benefits. The importance of these variables in other survey methodological work has also been demonstrated (Kirchner 2015;Kreuter, Müller, and Trappmann 2010;West, Kreuter, and Jaenichen 2013). Moreover, we chose these variables because they are very similar to items that were collected in the main CATI survey, including past employment activities, earnings, and receipt of unemployment benefit and income assistance. ...
Article
To mitigate the effects of low survey participation rates and possible nonresponse bias in survey estimates, survey organizations often try to collect auxiliary information with which to evaluate and possibly adjust for differences between respondents and nonrespondents. Call record data and other forms of paradata are commonly used for this purpose, but these data tend to be only weakly correlated with the survey items. Follow-up surveys conducted with nonrespondents try to get around this issue by asking a subset of key items selected from the original questionnaire. However, intensive follow-up procedures are expensive and have other known limitations. In this article, we explore an alternative follow-up procedure that simply asks nonrespondents for consent to use their administrative records in lieu of taking part in a telephone survey interview. Utilizing a unique study design with administrative records available for the overall study sample, we examine characteristics of telephone nonrespondents who consent to record use in a mail follow-up survey. Interestingly, we find that many telephone nonrespondents are willing to grant access to their administrative records. These consenting nonrespondents are similar to the remaining survey nonrespondents, yet different from the telephone survey respondents, which results in reduced nonresponse bias for some key economic items. We discuss the practical implications of these findings and offer some suggestions for incorporating the collected administrative data in nonresponse bias evaluation and adjustment procedures. © The Author 2016. Published by Oxford University Press on behalf of the American Association for Public Opinion Research.
... This provides us with a unique opportunity to disentangle error sources along both dimensions of the total survey error framework (Groves et al., 2004), namely the representation dimension and the measurement dimension, and their interaction by investigating the combined effects of nonresponse and measurement error comparing two important modes of survey administration. Existing validation studies comparing web and telephone that investigate the biasing effects of multiple error sources on estimates typically focus on the implications for univariate statistics, such as means or proportions (e.g., Kreuter et al., 2008Kreuter et al., , 2010aSakshaug et al., 2010). Research that does Investigate bias in regression coefficients typically focuses on the effe<:t of individual error sources on estimates, such as measurement error in wage or earnings regressions (e.g., Bound and Krueger, 1991;Kapteyn and Ypma, 2007;Kim and Tamborini, 2014;Rodgers and Herzog, 1987;Rodgers et al., 1993) instead of investigating the joint effects of different error sources. ...
Chapter
This chapter examines the differential effects of nonresponse and measurement error on regression coefficients by comparing a web survey to a telephone survey administration. It investigates the effects of these different error sources on a wage regression comparing survey and administrative data for a large-scale mixed-mode survey. Before investigating the potential biasing effects of nonresponse and measurement error on wage regressions, the chapter briefly reviews the types of bias encountered in mean statistics of the Work and Consumption in Germany (WCG) survey. The analyses for the WCG survey show that the biasing effects of nonresponse, measurement, and combined error highly depend on the variable that is investigated. The chapter compares three different survey respondent models adjusting for nonresponse to investigate whether the combined bias can be reduced and to assess which adjustment strategy is most useful.
... There is extensive literature on data quality problems in survey data (for an overview see Lyberg et al. 2012;Schnell 2011), but few studies analyze the quality of administrative data. Several studies have made use of them to validate survey data (Benitez-Silva et al. 2004;Gottschalk and Huynh 2010;Johansson and Skedinger 2009;Kapteyn and Ypma 2007;Kreuter et al. 2010) based on the assumption that these process-produced data are correct or at least highly reliable. However, some studies have detected inconsistencies or implausible sequences in administrative data (Bernhard et al. 2006;Bollinger and David 2005;Fitzenberger et al. 2006;Huber and Schmucker 2009;Jaenichen et al. 2005;Scioch 2010), and some studies have focused on how to address missing data (Büttner and Rässler 2008). ...
Technical Report
Full-text available
The use of process-produced data plays a large and growing role in empirical labor market research. To address data problems, previous research have developed deductive correction rules that make use of within-person information. We test data reliability and the effectiveness of different correction rules for information about educational degrees as reported in German register data. Therefore we use the unique dataset ALWA-ADIAB, which combines interview data and process-produced data from exactly the same individuals. This approach enables us to assess how effective the existing correction rules are and whether they manage to eliminate structural biases. In sum, we can state that simple editing rules based on logic assumptions are suitable for improving the quality of process-produced data, but they are not able to correct for structural biases.
... However, the benefits of employing a mixed-mode sequence in terms of reducing one or more bias sources should be weighed against the potential for increasing the total bias in survey estimates, particularly when the estimates are affected by counteracting biases. Such a backfiring effect is not uncommon and has been observed in other studies where nonresponse follow-up efforts achieve disproportionate reductions in individual bias components (e.g., Kreuter et al. 2010;Sakshaug and Eckman 2017). The present study was limited in multiple ways that could be addressed in future work. ...
Article
Mixing multiple modes of survey data collection has become standard practice in survey research. Mixed-mode surveys are faced with a slew of design decisions regarding which types of modes to administer and which sequence to administer them in. Such decisions are largely based on administrative objectives, such as minimizing costs and maximizing response rates. However, just as important to these mixed-mode decisions is their impact on nonresponse bias, measurement error bias, and total bias, which are understudied issues in the mixed-mode literature. In this article, we report on a sequential mixed-mode experiment of young adult drivers randomized to one of two mode sequences: an interviewer-administered (telephone) mode with self-administered (mail) follow-up, or the reverse sequence. Using a mix of direct and indirect bias estimation strategies, we find support for the notion that implementing a second mode of data collection can reduce nonresponse and measurement error bias, but the sequence in which the modes are administered makes a difference: the mail-telephone sequence minimizes bias to a greater extent than the telephone-mail sequence, relative to the starting mode and overall. However, a backfiring effect was found: despite reducing both nonresponse and measurement error bias, switching from mail to telephone increased the total bias in a key estimate of traffic accidents. A discussion of these findings and their implications for survey practice are provided in conclusion.
... Unlike previous studies, we thus do not have to fall back on twosample matching processes or the like, since survey responses and retrospective administrative information are already combined for the years 2008-2011. Studies on measurement errors usually also depend on consent to link survey responses to administrative records, which often leads to small sample sizes (Kreuter et al., 2010). In our study, between 95.6% (2008) and 99.4% (2011) of the respondents in the EU SILC survey could be identified with a PIN to assign the register information (Statistics Austria, 2014b). ...
Article
The paper analyses the sources of income measurement error in surveys with a unique data set. We use the Austrian 2008–2011 waves of the European Union ‘Statistics on income and living conditions' survey which provide individual information on wages, pensions and unemployment benefits from survey interviews and officially linked administrative records. Thus, we do not have to fall back on complex two‐sample matching procedures like related studies. We empirically investigate four sources of measurement error, namely social desirability, sociodemographic characteristics of the respondent, the survey design and the presence of learning effects. We find strong evidence for a social desirability bias in income reporting, whereas the presence of learning effects is mixed and depends on the type of income under consideration. An Owen value decomposition reveals that social desirability is a major explanation of misreporting in wages and pensions, whereas sociodemographic characteristics are most relevant for mismatches in unemployment benefits.
... Second, many researchers advocated using callbacks to reduce the non-response rate. (Stoop, 2004, Kreuter et al., 2010, Olson, 2013 While the developing world has witnessed a massive expansion of mobile phone and broadband networks, such means of contacting sampled units remain practically infeasible in impoverished areas where telephones and computers are not affordable or in sparsely populated areas without easy access to such networks. Third, it is often difficult to rule out that non-response is non-informative. ...
Article
Non-response is a commonly encountered problem in many population-based surveys. Broadly speaking, non-response can be due to refusal or failure to contact the sample units. Although both types of non-response may lead to bias, there is much evidence to indicate that it is much easier to reduce the proportion of non-contacts than to do the same with refusals. In this article, we use data collected from a nationally-representative survey under the Demographic and Health Surveys program to study non-response due to refusals to HIV testing in Malawi. We review existing estimation methods and propose novel approaches to the estimation of HIV prevalence that adjust for refusal behaviour. We then explain the data requirement and practical implications of the conventional and proposed approaches. Finally, we provide some general recommendations for handling non-response due to refusals and we highlight the challenges in working with Demographic and Health Surveys and explore di�erent approaches to statistical estimation in the presence of refusals. Our results show that variation in the estimated HIV prevalence across di�erent estimators is due largely to those who already know their HIV test results. In the case of Malawi, variations in the prevalence estimates due to refusals for women are larger than those for men.
... To date several studies have examined inconsistencies between survey and administrative data. In some instances administrative records have been used for the purpose of validating survey responses under the assumption that the records data are error-free and/or a suitable 'gold standard' against which survey responses can be compared (Davern et al. 2008;Kreuter et al. 2008Kreuter et al. , 2010Sakshaug et al. 2010a). Other studies, particularly, in the health survey literature, have questioned this assumption and made attempts to evaluate the quality of administrative data, in some cases finding that survey responses are more accurate than their administrative counterparts (Fowles et al. 1998;Hebert et al. 1999;Keating et al. 2003;Losina et al. 2003;Sakshaug et al. 2010b). ...
Article
Full-text available
Administrative records are increasingly being linked to survey records to highten the utility of the survey data. Respondent consent is usually needed to perform exact record linkage; however, not all respondents agree to this request and several studies have found significant differences between consenting and non-consenting respondents on the survey variables. To the extent that these survey variables are related to variables in the administrative data, the resulting administrative estimates can be biased due to non-consent. Estimating non-consent biases for linked administrative estimates is complicated by the fact that administrative records are typically not available for the non-consenting respondents. The present study can overcome this limitation by utilizing a unique data source, the German Panel Study "Labour Market and Social Security" (PASS), and linking the consent indicator to the administrative records (available for the entire sample). This situation permits the estimation of non-consent biases for administrative variables and avoids the need to link the survey responses. The impact of nonconsent bias can be assessed relative to other sources of bias (nonresponse, measurement) for several administrative estimates. The results show that non-consent biases are present for few estimates, but are generally small relative to other sources of bias.
... More information about the IEB can be found in vom Berge, Burghardt, and Trenkle (2013). The IEB has been used extensively in other survey methodological research projects (e.g., Kreuter, Müller, and Trappmann 2010;West, Kreuter, and Jaenichen 2013;Kirchner 2015). The IEB administrative records were made available for the entire WeLL sample, both respondents and nonrespondents. ...
Article
Surveys are susceptible to multiple error sources that threaten the validity of inferences drawn from them. While much of the survey methods literature has focused on identifying errors in cross-sectional surveys, errors in panel surveys have received less attention. Administrative records linked to the entire sample (respondents and nonrespondents) can be useful for studying various errors in panel surveys, including nonresponse, which tends to accumulate over multiple waves of the study. Record data can also be used to study errors due to linkage consent, which is commonly requested in panel surveys but not provided by all respondents. In this paper, we present bias estimates for both error sources from a panel survey in Germany. The bias estimates are derived from administrative data collected on a sample of employees who were invited to participate in the panel. We find evidence of increasing nonresponse bias over time for cross-sectional and longitudinal outcomes. The opposite pattern is observed for linkage consent bias, which decreases over time when respondents who do not provide consent in a prior wave are asked to reconsider their decision in subsequent waves. We conclude with a discussion of the practical implications of these findings and propose suggestions for future research.
... Under our assumption that true affect does not change over time, this suggests that VAR(J it ) = VAR(E Dit ) + VAR(E Rit ) (2) Within our theoretical framework, the effect of time on VAR(E Rit )is the opposite of its effect on VAR(E Dit ). While time-induced decrease in effort leads to decrease in VAR(E Dit ), it leads to an increase in VAR(E Rit ), because effort is negatively related to measurement error (Kreuter, Müller, & Trappmann, 2010). Therefore, a decrease in VAR(J it ) can be attributed only to a decrease in VAR(E Dit ). ...
Article
We study the effects of diary's serial day (the number of days from the beginning of the study) on participants’ (n = 2022) reports about positive and negative affect. We find that (1) the number of reported positive events and the number of reported negative events decrease with serial day; (2) positivity increases with serial day: Reported Positive Affect (PA) increases, and reported Negative Affect (NA) decreases; (3) emotional complexity – the tendency to differentiate between various types of emotions – decreases with serial day, both within- and between- affective dimensions. We attribute these effects to decrease in the effort exerted by participants in answering the diary questions, and suggest that these effects are consistent with the distinction between experienced and reported emotions and with a heuristic and biases perspective in which when effort decreases reported emotions regress to an easier to generate default response. This article is protected by copyright. All rights reserved
... Linked administrative-survey data have been used to examine substantive issues such as healthcare spending and economic planning (e.g. Hogan et al. 2001, Scholz et al. 2006, as well as methodological issues, such as survey measurement error (Kreuter et al. 2010, Olson 2006, Sakshaug et al. 2010. However, the literature on data linkage requests within a survey context is still relatively nascent. ...
... Not only does data linkage offer possible cost savings by reducing the length of the questionnaire, it offers attractive scientific possibilities for researchers interested in studying important substantive and methodological phenomena (Lillard and Farmer 1997;Calderwood and Lessof 2009). Studies utilizing linked survey and administrative data sources have made important contributions to our understanding of various substantive topics, including healthcare spending among older populations (Hogan et al. 2001;Lubitz et al. 2003;Peikes et al. 2009), lifetime earnings and retirement planning (Hurd and Zissimopoulos 2003;Gustman and Steinmeier 2005;Scholz et al. 2006), as well as methodological topics such as the accuracy of survey self-reports and the impact of nonresponse bias on survey estimates (Olson 2006;Kreuter et al. 2010;Sakshaug et al. 2010). ...
Article
Full-text available
Record linkage is becoming more important as survey budgets are tightening while at the same time demands for more statistical information are rising. Not all respondents consent to linking their survey answers to administrative records, threatening inferences made from linked data sets. So far, several studies have identified respondent-level attributes that are correlated with the likelihood of providing consent (e.g., age, education), but these factors are outside the control of the survey designer. In the present study three factors that are under the control of the survey designer are evaluated to assess whether they impact respondents' likelihood of linkage consent: 1) the wording of the consent question; 2) the placement of the consent question and; 3) interviewer attributes (e.g., attitudes toward data sharing and consent, experience, expectations). Data from an experiment were used to assess the impact of the first two and data from an interviewer survey that was administered prior to the start of data collection are used to examine the third. The results show that in a telephone setting: 1) indicating time savings in the wording of the consent question had no effect on the consent rate; 2) placement of the consent question at the beginning of the questionnaire achieved a higher consent rate than at the end and; 3) interviewers' who themselves would be willing to consent to data linkage requests were more likely to obtain linkage consent from respondents.
... Different studies have tried to disentangle representation and measurement effects using regression (J€ ackle, Roberts, and Lynn 2010), weighting (Hox, De Leeuw, and Zijlmans 2015), structural equation modeling (Heerwegh and Loosveldt 2011), experimental or quasi-experimental designs (Lugtig et al. 2011;Schouten et al. 2013;Cernat, Couper, and Ofstedal 2016;Klausch, Schouten, and Hox 2017), and record data as a gold standard to identify mode effects (Voogt and Saris 2005;Link and Mokdad 2006;Kreuter, Müller, and Trappmann 2010;Sakshaug, Yan, and Tourangeau 2010). ...
Article
With the increasing usage of dual-mode data collection, researchers of public opinion have shown considerable interest in understanding response differences across different interview modes. Are mode effects an outcome of representation or measurement differences across modes? We conducted a dual-mode survey (web and telephone) using Florida’s voter file as the sampling frame, randomly assigning registered voters into one mode versus the other. Having a priori information about the respondents allows us to gauge whether and how sample composition differences may be driven by mode effects, and whether mode affects estimated models of political behavior. Survey mode effects are still significant for issue voting even when sampling design is similar for both modes.
... Does the way in which error sources are linked vary across subgroups? Kreuter, Müller, and Trappmann (2010) provided evidence of variation in nonresponse and measurement error relationships across subgroups when looking at administrative data on welfare benefit receipt in Germany. How dependent is the link between these error sources on how questions themselves are constructed? ...
Article
Full-text available
This paper provides a discussion of the Tourangeau (2019) Morris Hansen Lecture paper. I address issues related to compounding errors in web surveys and the relationship between nonresponse and measurement errors. I provide a potential model for understanding when error sources in nonprobability web surveys may compound or counteract one other. I also provide three conceptual models that help explicate the joint relationship between nonresponse and measurement errors. Tourangeau’s paper provides two interesting case studies about the role of multiple error sources in survey data. The first case study is one in which errors occur at different stages of the representation process—errors first occur when creating a potential sample frame, then may be amplified when selecting sampled persons, possibly because of self-selection, and then are exacerbated with an individual’s decision to participate. The second case study has to do with situations where different error sources may influence each other and, in particular, the relationship between nonresponse error and various measurement error outcomes.
... Closely related to this, such a norm might also make it socially undesirable to receive UB II. This could explain why some people misreport not to receive the benefits (Kreuter, Müller, andTrappmann 2010, Bruckmeier, Müller, andRiphahn 2014). These workers probably suffer from receiving UB II the most as they are even willing to hide this circumstance from an anonymous survey. ...
Article
Using specific panel data of German welfare benefit recipients, we investigate the non-pecuniary life satisfaction effects of in-work benefits. Our empirical strategy combines difference-in-difference designs with synthetic control groups to analyse transitions of workers between unemployment, regular employment and employment accompanied by welfare receipt. Working makes people generally better off than being unemployed but employed welfare recipients do not reach the life satisfaction level of regular employees. This implies that welfare receipt entails non-compliance with the norm to make one's own living. Our findings allow us to draw cautious conclusions on employment subsidies paid as welfare benefits.
... Underrepresented groups include civil servants, the selfemployed, and homemakers, who are exempt from making social security contributions. 1 In the following analyses, we consider eight administrative variables: received UB II at least once since 2009, total number of employment spells in lifetime, currently employed, ever received regular unemployment benefit, age, sex, average daily wage, and foreign citizenship. These variables are commonly used in economic studies utilizing the IAB database (Boockmann, Ammermüller, Zwick, and Maier 2007;Kreuter, Müller, and Trappmann 2010;Baumgarten 2013;Burr, Rauch, Rose, Tisch, and Tophoven 2015). ...
Article
Full-text available
Survey researchers are actively seeking powerful auxiliary data sources capable of correcting for possible nonresponse bias in survey estimates of the general population. While several auxiliary data options exist, concerns about their usefulness for addressing nonresponse bias remain. One underutilized—but potentially rich—source of auxiliary data for nonresponse bias adjustment is federal administrative records. While federal records are routinely used to study nonresponse in countries where it is possible to directly link them (via a unique identifier) to population-based samples, such records are not widely used for this purpose in countries which lack a unique identifier to facilitate direct linkage. In this article, we examine the utility of indirectly linked administrative data from a federal employment database for nonresponse bias adjustment in a general population survey in Germany. In short, we find that the linked administrative variables have stronger correlations with the substantive survey variables than do standard paradata variables and that incorporating linked administrative data in nonresponse weighting adjustments reduces relative nonresponse bias to a greater extent than paradata-only weighting adjustments. However, for the majority of weighted survey estimates, including the administrative variables in the weighting adjustment procedure has minimal impact on the point estimates and their variances. We conclude with a general discussion of these findings and comment on the logistical issues associated with this type of linkage relevant to survey practice.
... The variables are sex (male), age (≥ 46 years), received non-university vocational training, currently employed, at least one employer change since 2008, average daily wage provided in Appendix Tables A1, A2 (telephone), A3, and A4 (Web). These variables, which have been extensively used in methodological studies using the BA data (Kreuter, Müller, and Trappmann 2010;West, Kreuter, and Jaenichen 2013;Kirchner 2015), are merged to all respondents with a 100 percent match rate using unique IDs from the sampling frame. Linkage consent bias is assessed by comparing the estimated proportion of the k th (= 1, 2, . . . ...
Article
Full-text available
Numerous surveys link interview data to administrative records, conditional on respondent consent, in order to explore new and innovative research questions. Optimizing the linkage consent rate is a critical step toward realizing the scientific advantages of record linkage and minimizing the risk of linkage consent bias. Linkage consent rates have been shown to be particularly sensitive to certain design features, such as where the consent question is placed in the questionnaire and how the question is framed. However, the interaction of these design features and their relative contributions to the linkage consent rate have never been jointly studied, raising the practical question of which design feature (or combination of features) should be prioritized from a consent rate perspective. We address this knowledge gap by reporting the results of a placement and framing experiment embedded within separate telephone and Web surveys. We find a significant interaction between placement and framing of the linkage consent question on the consent rate. The effect of placement was larger than the effect of framing in both surveys, and the effect of framing was only evident in the Web survey when the consent question was placed at the end of the questionnaire. Both design features had negligible impact on linkage consent bias for a series of administrative variables available for consenters and non-consenters. We conclude this research note with guidance on the optimal administration of the linkage consent question.
... Metcalfe et al. 2011;Chadi 2015;Goebel et al. 2015;Schueller 2016). 2 See e.g. the data documentation on panel attrition in the SOEP (Kroh , 2010(Kroh , 2012. 3 Only a few researchers so far use information on the number of contacts with potential respondents to discuss the role of difficult-to-reach survey participants in data collection. For instance, Kreuter et al. (2010) investigate data on benefit recipients and look at the role played by number of calls, and Heffetz and Reeves (2019) compare the responses of easy-and difficult-to-reach respondents in various governmental surveys. Heffetz and Rabin (2013) introduce the idea of using contact attempts to discuss potential nonresponse bias in happiness data. ...
Article
Full-text available
People with little motivation to participate in surveys can affect empirical research when they abstain from but also when they actually participate in interviews. This paper investigates whether happiness data are susceptible to such measurement bias. Evidence from the German Socio-Economic Panel Study (SOEP) reveals a strong relationship between self-reported life satisfaction and several indicators of respondent motivation, such as subsequent panel attrition. One explanation for this finding is that respondents on the margin of participation truly have lower life satisfaction. Alternatively, their low motivation may be the cause for an underreporting of life satisfaction. To learn more about this, an instrumental variable approach identifies future panel quitters with low motivation by using the occurrence of interviewer attrition in the year after the interview. The results of this analysis suggest that self-reported life satisfaction declines because of low respondent motivation. A discussion of the implications for analyses of happiness data underscores the potential importance of respondent motivation regardless of the explanation for why interviewees with low motivation report lower life satisfaction.
... Second, many researchers advocated using callbacks to reduce the nonresponse rate. [16][17][18] While the developing world has witnessed a massive expansion of mobile phone and broadband networks, such means of contacting sampled units remain practically infeasible in impoverished areas where telephones and computers are not affordable or in sparsely populated areas without easy access to such networks. Third, it is often difficult to rule out that non-response is non-informative. ...
Article
Non-response is a commonly encountered problem in many population-based surveys. Broadly speaking, non-response can be due to refusal or failure to contact the sample units. Although both types of non-response may lead to bias, there is much evidence to indicate that it is much easier to reduce the proportion of non-contacts than to do the same with refusals. In this article, we use data collected from a nationally representative survey under the Demographic and Health Surveys program to study non-response due to refusals to HIV testing in Malawi. We review existing estimation methods and propose novel approaches to the estimation of HIV prevalence that adjust for refusal behaviour. We then explain the data requirement and practical implications of the conventional and proposed approaches. Finally, we provide some general recommendations for handling non-response due to refusals and we highlight the challenges in working with Demographic and Health Surveys and explore different approaches to statistical estimation in the presence of refusals. Our results show that variation in the estimated HIV prevalence across different estimators is due largely to those who already know their HIV test results. In the case of Malawi, variations in the prevalence estimates due to refusals for women are larger than those for men.
Article
Background: The decline in participation rates in surveys, including epidemiological surveillance surveys, has become a real concern since it may increase nonresponse bias. The aim of this study is to estimate the contribution of a complementary survey among a subsample of nonrespondents, and the additional contribution of paradata in correcting for nonresponse bias in an occupational health surveillance survey. Methods: In 2010, 10,000 workers were randomly selected and sent a postal questionnaire. Sociodemographic data were available for the whole sample. After data collection of the questionnaires, a complementary survey among a random subsample of 500 nonrespondents was performed using a questionnaire administered by an interviewer. Paradata were collected for the complete subsample of the complementary survey. Nonresponse bias in the initial sample and in the combined samples were assessed using variables from administrative databases available for the whole sample, not subject to differential measurement errors. Corrected prevalences by reweighting technique were estimated by first using the initial survey alone and then the initial and complementary surveys combined, under several assumptions regarding the missing data process. Results were compared by computing relative errors. Results: The response rates of the initial and complementary surveys were 23.6% and 62.6%, respectively. For the initial and the combined surveys, the relative errors decreased after correction for nonresponse on sociodemographic variables. For the combined surveys without paradata, relative errors decreased compared with the initial survey. The contribution of the paradata was weak. Conclusion: When a complex descriptive survey has a low response rate, a short complementary survey among nonrespondents with a protocol which aims to maximize the response rates, is useful. The contribution of sociodemographic variables in correcting for nonresponse bias is important whereas the additional contribution of paradata in correcting for nonresponse bias is questionable.
Article
One of the most daunting challenges facing survey researchers is ensuring high response rates, which have been declining due to societal, political, and technological changes. This chapter analyzes whether investing resources into successfully interviewing difficult-to-obtain and reluctant respondents via nonresponse strategies (e.g., multiple call-back attempts and refusal conversions)-thereby reducing nonresponse bias-is counterproductive because the respondents obtained via these methods may not be fully engaged with the survey, and may therefore be inclined to satisfice. As such, nonresponse strategies may bring respondents into the sample who introduce more systematic and/or random measurement error than their easier-to-reach counterparts. We examine the relationship between reluctance and satisficing in three Internet surveys, and then examine whether survey mode moderates the impact of reluctance on satisficing by comparing Internet, telephone, and face-to-face surveys. We find that difficult-to-obtain Internet respondents are more likely to indicate a DK/NO response and are more likely to select the first reasonable response than their easy-to-obtain counterparts. In contrast, across the five indicators of satisficing examined in the face-to-face and phone modes (midpoint responding, DK/NO responding, non-differentiation, mental coin-flipping, and selecting the first reasonable response), the only one for which difficult-to-reach respondents were more likely to evidence satisficing (compared to their easy-to-reach counterparts) was middle responding (in the phone mode). The difference in effects between the Internet and phone mode varies in a theoretically predictable manner. Whereas reluctant respondents are more likely to select the middle option in telephone surveys when an interviewer is present (thus responding without expending much cognitive effort while maintaining the façade of compliance or competence), they are more likely to provide "don't know" responses when interviewed over the Internet (when the lack of an interviewer reduced concerns about appearing compliant or competent). Given that some forms of satisficing occurred more among reluctant respondents in the Internet mode, these new forms of survey interviewing present future challenges.
Article
Declining participation in voluntary establishment surveys poses a risk of increasing non‐response bias over time. In this paper, response rates and non‐response bias are examined for the 2010–2019 IAB Job Vacancy Survey. Using comprehensive administrative data, we formulate and test several theory‐driven hypotheses on survey participation and evaluate the potential of various machine learning algorithms for non‐response bias adjustment. The analysis revealed that while the response rate decreased during the decade, no concomitant increase in aggregate non‐response bias was observed. Several hypotheses of participation were at least partially supported. Lastly, the expanded use of administrative data reduced non‐response bias over the standard weighting variables, but only limited evidence was found for further non‐response bias reduction through the use of machine learning methods.
Research
Die FDZ-Methodenreporte befassen sich mit den methodischen Aspekten der Daten des FDZ und helfen somit Nutzerinnen und Nutzern bei der Analyse der Daten. Nutzerinnen und Nutzer können hierzu in dieser Reihe zitationsfähig publizieren und stellen sich der öffentlichen Diskussion. FDZ-Methodenreporte (FDZ method reports) deal with the methodical aspects of FDZ data and thus help users in the analysis of data. In addition, through this series users can publicise their results in a manner which is citable thus presenting them for public discussion.
Article
Full-text available
2011/4 жовтень-грудень.-с. 198-208 УДК 303.09, 303.8 Сидоров Микола Володимир-Станіславович Сидоров Николай Владимир-Станиславович Sydorov Mykola Volodymyr-Stanislav Використання параданих у соціологічних дослідженнях. Использование параданных в социологических исследованиях. Using of paradata in social researches. Анотація. У статті розглядаються парадані як особливий тип даних, зібраних під час опитування про сам процес збору інформації, а саме поведінку респондентів, характеристики їх умов проживання, реакції на запитання, час відповіді та ін. Наводяться приклади використання параданих у соціальних дослідженнях. Ключові слова: парадані, метадані, невідповіді. Аннотация. В статье рассматриваются параданные как особенный тип данных, собранных во время опроса о самом процессе сбора информации, таких как поведение респондентов, характеристики их условий проживания, реакции на вопросы, время ответа и др. Приводятся примеры использования параданных в социологических исследованиях. Ключевые слова: параданные, метаданные, неответы. Annotation. In the article paradata as a specific type of findings during survey about survey process such as respondents behaviour, accomodation environments, reaction on questions, answere time etc. There are some examples of using paradata in social surveys are presented.
Chapter
Administrative data obtained from government registers provide a wealth of potential for the social sciences. Administrative registers may contain considerable measurement errors, including definition, reporting, timing, processing, editing, linkage, and coverage errors. This chapter explores how to estimate the extent of measurement errors in both administrative register data and survey answers. It demonstrates one approach to doing so: latent variable modeling. Modeling these mode effects allows for the identification of both random classification errors and method effects without the need for multiple “traits” (true values). It could therefore be termed a “single-trait-multimethod” (STMM) approach. The chapter describes the data on neighborhood of residence obtained from a survey and an important Dutch official administrative register, and then details the latent class model built to estimate classification error rates in these measures. The costs of full-scale audits compared with latent variable modeling are gigantic, meaning that a small inaccuracy still leaves latent variable modeling an attractive alternative.
Chapter
This chapter discusses the efforts at the Institute for Employment Research (IAB) in Nuremberg, Germany, and shares a success story with regard to the establishment of research data centers that enable outside researchers to access IAB data (including administrative data sources). It also shares recent experiences of the US Census Bureau with respect to the use of administrative data for understanding total survey error (TSE) and demonstrates how these data can be used to improve the quality of survey products while also communicating important challenges that have been encountered. The chapter describes efforts by Statistics New Zealand to integrate various sources of “big data” and ultimately create a combined data source that will improve both survey operations and data products. It concludes with case studies illustrating the use of “big data” at the University of Michigan Survey Research Center, again focusing on the benefits and challenges of this enterprise.
Chapter
Die Nachfrage nach gut ausgebildeten DatenwissenschaftlerInnen, die sowohl die Fähigkeiten besitzen, Daten auf „traditionellem Weg“ zu erheben und auszuwerten und ebenso mit großen semi- oder gar unstrukturierten Datensätzen zu arbeiten, steigt kontinuierlich an. In diesem Beitrag beschreiben wir, welche Kompetenzen Sozial- und MarktforscherInnen heutzutage benötigen, um am Arbeitsmarkt erfolgreich zu sein. Wir diskutieren Herausforderungen und Chancen im Bereich der Lehre dieser neuen Inhalte und deren Potenzial, den steigenden Bedarf an Fachkräften im Bereich Datenerhebung und Datenanalyse in den kommenden Jahren zu decken.
Book
Full-text available
En översikt om metoder och strategier för att mäta snabba opinionsförändringar och undersöka opinioner/ händelser i sociala medier
Conference Paper
Full-text available
The ordinary least squares and ridge regression estimators lead to inefficient results when the joint problem of multicollinearity and y-direction outliers are present in the multiple linear regression models. In order to get precise estimates, ridge M-estimation method is usually used in such situations. The ridge parameter, k, plays a key role in achieving the optimal results. The estimation of ridge parameter is an important problem for many researchers. In this article, we considered some existing ridge M-estimators and developed some new estimators for k. Extensive Monte Carlo simulations are used to compare the performance of estimators through mean squared error criterion. The factors we choose to vary are multicollinearity, y-direction outliers, sample size, predictors and error distributions. Some of the new ridge M-estimators performed well compared to the ordinary least square, ridge regression and some existing popular ridge M-estimators. Finally, a numerical example is presented to illustrate the benefits of the new estimators.
Conference Paper
Student feedback is the output provided by the students related to their perception of teachings. This is offered to the students in every semester at the end in response to the various question asked about their activities, tasks and the environment they are engaged in the whole semester. Student’s achievements are a big contribution by the teachers. Teacher evaluation is focused these days to determine the teaching impact on student outcomes and is also use for their professional development. Different researches show different results for evaluation of teachers. This overview provides the information about the teachers practice and using it to improve their system. With existing empirical evidence mostly restricted to short-run outcomes. This study examines long-term academic outcomes of Teacher Evaluation for teachers by bachelor students at Air University. On the basis of statistical analysis the effectiveness of said dataset and relation towards teacher evaluation policies have been formulated.
Article
Full-text available
We investigate the general effect of the COVID-19 pandemic on subjective well-being and determine whether this effect differs between recipients of basic income support (BIS) and the rest of the working-age population in Germany. BIS recipients constitute one of the most disadvantaged groups in Germany and might lack resources for coping with the crisis. Thus, our analysis contributes to investigations of whether the pandemic exacerbates or equalises preexisting social inequality. Our analysis employs data from the panel survey “Labour Market and Social Security” (PASS). These data have the key advantage that the collection in 2020 started prior to implementation of the first COVID-19-related policies. This situation enables us to apply a difference-in-differences approach to investigate the causal change in subjective well-being. Our results suggest that well-being declined during the first phase of the COVID-19 pandemic. However, we find no difference in this decline between BIS recipients and other German residents. Thus, our results suggest that the first phase of the COVID-19 pandemic neither exacerbated nor equalised pre-existing inequalities.
Article
Full-text available
This study tests the idea that features of the presentation of a survey to potential respondents can affect nonresponse error, measurement error, and the relation between the two. A few weeks after they had completed one web survey, we asked members of two opt-in web panels to take part in a second web study and systematically varied our description of that survey. The description varied both the purported topic and sponsor of the second survey. The members of the sample were not aware of the connection between the two surveys. We found little evidence that the survey presentation affected response rates to the second survey or the make-up of the sample on the variables we had collected in the initial questionnaire. There were indications, however, that some answers to the questions in the second survey were affected by the framing of the survey request. For example, respondents were less favorable to gun control than they had been in the initial survey when we described the sponsor of the second survey as the "The National Coalition of Gun Owners" rather than "The National Coalition for Victims of Gun Violence" or "The National Center for the Study of Crime." We argue that the description of the survey can affect how respondents interpret the questions and what they see as a useful response. We also found evidence that attitudes toward the survey sponsor and interest in the topic were related to carelessness in completing the questions. Respondents were, for example, less likely to give the same answer to every item in a grid if they had favorable attitudes toward the sponsor than if their attitudes toward the sponsor were unfavorable.
Article
Full-text available
Fifty-nine methodological studies were designed to estimate the magnitude of nonresponse bias in statistics of interest. These studies use a variety of designs: sampling frames with rich variables, data from administrative records matched to sample case, use of screening-interview data to describe nonrespondents to main interviews, followup of nonrespondents to initial phases of field effort, and measures of behavior intentions to respond to a survey. This permits exploration of which circumstances produce a relationship between nonresponse rates and nonresponse bias and which, do not. The predictors are design features of the surveys, characteristics of the sample, and attributes of the survey statistics computed in the surveys.
Article
Full-text available
1 Abstract2 Using survey and contact attempt history data collected with the 2005 National Health Interview Survey (NHIS), a multi- purpose health survey conducted by the National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC), we set out to explore the impact of participant concerns/reluctance on data quality, as measured by rates of partially complete interviews and item nonresponse. Overall, results show that respondents from households where some type of concern or reluctance (e.g., "too busy," "not interested") was expressed produced higher rates of partially complete interviews and item nonresponse than respondents from households where concern/reluctance was not expressed. Differences by type of concern were also identified.
Conference Paper
Full-text available
While the individual components of total survey error have been well documented in the literature, relatively little is known about the intersection of these error sources. In particular, there is scant empirical work on the interplay between nonresponse error and measurement error - despite the potentially significant implications for data quality as well as techniques used to recruit respondents. In this paper we investigate the connection between these two error sources using data from a survey of University of Maryland alumni. The availability of administrative records for seven items on the survey instrument (donations, membership in the alumni association, and multiple measures of academic performance) make this dataset particularly well-suited for this type of analysis. We evaluate several causal models related to the nonresponse / measurement error nexus. These models predict differential effects for particular subgroups of the population: recent versus older graduates and alumni who demonstrated low versus high academic achievement.
Article
Full-text available
A common hypothesis about practices to reduce survey nonresponse is that those persons brought into the respondent pool through persuasive efforts may provide data filled with measurement error. Two questions flow from this hypothesis. First, does the mean square error of a statistic increase when sample persons who are less likely to be contacted or cooperate are incorporated into the respondent pool? Second, do nonresponse bias estimates made on the respondents, using survey reports instead of records, provide accurate information about nonresponse bias? Using a unique data set, the Wisconsin Divorce Study, with divorce records as the frame and questions about the frame information included in the questionnaire, this article takes a first look into these two issues. We find that the relationship between nonresponse bias, measurement error bias, and response propensity is statistic- specific and specific to the type of nonresponse. Total bias tends to be lower on estimates calculated using all respondents, compared with those with only the highest contact and cooperation propensities, and nonresponse bias analyses based on respondents yield conclusions simi- lar to those based on records. Finally, we find that error properties of statistics may differ from error properties of the individual variables used to calculate the statistics.
Article
Full-text available
Declining contact and cooperation rates in random digit dial (RDD) national telephone surveys raise serious concems about the validity of estimates drawn from such research. While research in the 1990s indicated that nonresponse bias was relatively small, response rates have continued to fall since then. The current study replicates a 1997 methodological experiment that compared results from a "Standard" 5-day survey employing the Pew Research Center's usual methodology with results from a "Rigorous" survey conducted over a much longer field period and achieving a significantly higher response rate. As with the 1997 study, there is little to suggest that unit nonre-sponse within the range of response rates obtained seriously threatens the quality of survey estimates. In 77 out of 84 comparable items, the two surveys yielded results that were statistically indistinguishable. While the "Rigorous" study respondents tended to be somewhat less politically engaged, they did not report consistently different behaviors or attitudes on other kinds of questions. With respect to sample composition, the Standard survey was closely aligned with estimates from the U.S. Census and other large govemment surveys on most vari-ables. We extend our analysis of nonresponse to include comparisons with the hardest-to-reach respondents and with respondents who termi-nated the interview prior to completion.
Article
Full-text available
Previous research has documented effects of the order in which response choices are offered to respondents using closed-ended survey items, but no theory of the psychological sources of these effects has yet been proposed. This paper offers such a theory drawn from a variety of psychological research. Using data from a split-ballot experiment in the 1984 General Social Survey involving a variant of Kohn's parental values measure, we test some predictions made by the theory about what kind of response order effect would be expected (a primacy effect) and among which respondents it should be strongest (those low in cognitive sophistication). These predictions are confirmed. We also test the “form-resistant correlation” hypothesis. Although correlations between items are altered by changes m response order, the presence and nature of the latent value dimension underlying these responses is essentially unaffected.
Article
Full-text available
The authors argue that both the large variability in survey estimates of volunteering and the fact that survey estimates do not show the secular decline common to other social capital measures are caused by the greater propensity of those who do volunteer work to respond to surveys. Analyses of the American Time Use Survey (ATUS)--the sample for which is drawn from the Current Population Survey (CPS)--together with the CPS volunteering supplement show that CPS respondents who become ATUS respondents report much more volunteering in the CPS than those who become ATUS nonrespondents. This difference is replicated within subgroups. Consequently, conventional adjustments for nonresponse cannot correct the bias. Although nonresponse leads to estimates of volunteer activity that are too high, it generally does not affect inferences about the characteristics of volunteers.
Article
Full-text available
Psychologists have worried about the distortions introduced into standardized personality measures by social desirability bias. Survey researchers have had similar concerns about the accuracy of survey reports about such topics as illicit drug use, abortion, and sexual behavior. The article reviews the research done by survey methodologists on reporting errors in surveys on sensitive topics, noting parallels and differences from the psychological literature on social desirability. The findings from the survey studies suggest that misreporting about sensitive topics is quite common and that it is largely situational. The extent of misreporting depends on whether the respondent has anything embarrassing to report and on design features of the survey. The survey evidence also indicates that misreporting on sensitive topics is a more or less motivated process in which respondents edit the information they report to avoid embarrassing themselves in the presence of an interviewer or to avoid repercussions from third parties.
Article
Full-text available
We use a new and exceptionally rich administrative data set for Germany to evaluate the employment effects of a variety of public sponsored training programs in the early 2000s. Building on the work of Sianesi (2003, 2004), we employ propensity score matching methods in a dynamic, multiple treatment framework in order to address program heterogeneity and dynamic selection into programs. Our results suggest that in West Germany both short-term and medium-term programs show considerable employment effects for certain population subgroups but in some cases the effects are zero in the medium run. Short-term programs are surprisingly effective when compared to the traditional and more expensive longer-term programs. With a few exceptions, we find little evidence for significant positive treatment effects in East Germany. There is some evidence that the employment effects decline for older workers and for low-skilled workers.
Article
An Introduction to Survey Participation. A Conceptual Framework for Survey Participation. Data Resources for Testing Theories of Survey Participation. Influences on the Likelihood of Contact. Influences of Household Characteristics on Survey Cooperation. Social Environmental Influences on Survey Participation. Influences of the Interviewers. When Interviewers Meet Householders: The Nature of Initial Interactions. Influences of Householder-Interviewer Interactions on Survey Cooperation. How Survey Design Features Affect Participation. Practical Survey Design Acknowledging Nonresponse. References. Index.
Article
»Datenintegration und Datenkonsolidierung von administrativen Daten aus unterschiedlichen Quellen am Beispiel von Daten zum deutschen Arbeitsmarkt«. This article introduces the data integration and consolidation process of the research data base of the Institute for Employment Research. The data are process generated data and stem from various, autonomous administrative processes. This fact implies that there are manifold inconsistencies between the data from the different data sources. This opens up the methodological problem of a successful consolidation of inconsistencies. Two contrarian strategies to handle this methodological problem are discussed and the solution in the IAB-data base is presented.
Article
An extension of the Wilcoxon rank-sum test is developed to handle the situation in which a variable is measured for individuals in three or more (ordered) groups and a non-parametric test for trend across these groups is desired. The uses of the test are illustrated by two examples from cancer research.
Article
Error in survey data originates from failure to contact the sample and from false answers to verifiable questions. These errors may be systematic and associated with uncooperative or unreliable respondents. Zabel modeled attrition in the Survey of Income and Program Participation and found systematic demographic and design effects. Bollinger and David modeled response error and identified correlations to income per capita. In this analysis, we link missing interviews in a panel and response error through a trivariate probit analysis. Robustness of the correlation between attrition and response error is examined by comparing variants of the model. The joint model of response error and attrition becomes the first stage of a pseudolikelihood estimate of a model of food-stamp participation. The model is significantly different from naive probit on the survey data.
Article
Fellegi's (1974) improved method for estimating the interviewer component of correlated response variance is extended to l groups of k interviewer assignments for general multistage survey designs. Using a linear models approach suggestive of Hartley and Rao's (1978), the independence of the two estimators of interviewer variance is established and the forms of the variances of the estimators are derived. Then using 1980 census data to compute terms in the estimator variances, (a) the optimal design of interviewer variance studies is considered, (b) the improvement of the composite estimator is demonstrated, and (c) some principles of efficient study design are developed.
Article
An important mark of professional competence is a sophisticated and critical attitude toward the procedures that are used in the performance of professional functions. The best examples of social research have increasingly exhibited this attitude both in the reports of particular projects and in special research inquiries aimed primarily at testing and improving the research procedures that are in common use. It would seem that the greatest progress has been made in the development of scales of measurement and sampling procedures, but important progress has also been made in testa of the validity of the data produced by surveys and other research inquiries. Here is a unique study of the validity of two procedures for obtaining data in surveys. Charles F. Cannell is Program Director and Director of the Field Staff at the Survey Research Center of the University of Michigan and co-author with Robert L. Kahn of a book, The Dynamics of Interviewing. Floyd J . Fowler is an Assistant Study Director at the Survey Research Center and is enrolled in the Doctoral Program in Social Psychology at the University of Michigan.
Article
Preface. Chapter 1. The Evolution of Survey Process Quality. 1.1 The Concept of a Survey. 1.2 Types of Surveys. 1.3 Brief History of Survey Methodology. 1.4 The Quality Revolution. 1.5 Definitions of Quality and Quality in Statistical Organizations. 1.6 Measuring Quality. 1.7 Improving Quality. 1.8 Quality in a Nutshell. Chapter 2. The Survey Process and Data Quality. 2.1 Overview of the Survey Process. 2.2 Data Quality and Total Survey Error. 2.3 Decomposing Nonsampling Error into Its Component Parts. 2.4 Gauging the Magnitude of Total Survey Error. 2.5 Mean Squared Error. 2.6 An Illustration of the Concepts. Chapter 3. Coverage and Nonresponse Error. 3.1 Coverage Error. 3.2 Measures of Coverage Bias. 3.3 Reducing Coverage Bias. 3.4 Unit Nonresponse Error. 3.5 Calculating Response Rates. 3.6 Reducing Nonresponse Bias. Chapter 4. The Measurement Process and Its Implications for Questionnaire Design. 4.1Components of Measurement Error. 4.2 Errors Arising from the Questionnaire Design. 4.3 Understanding the Response Process. Chapter 5. Errors Due to Interviewers and Interviewing. 5.1 Role of the Interviewer. 5.2 Interviewer Variability. 5.3 Design Factors that Influence Interviewer Effects. 5.4 Evaluation of Interviewer Performance. Chapter 6. Data Collection Modes and Associated Errors. 6.1 Modes of Data Collection. 6.2 Decision Regarding Mode. 6.3 Some Examples of Mode Effects. Chapter 7. Data Processing: Errors and Their Control. 7.1 Overview of Data Processing Steps. 7.2 Nature of Data Processing Error. 7.3 Data Capture Errors. 7.4 Post-Data Capture Editing. 7.5 Coding. 7.6 File Preparation. 7.7 Applications of Continuous Quality Improvement: The Case of Coding. 7.8 Integration Activities. Chapter 8. Overview of Survey Error Evaluation Methods. 8.1 Purposes of Survey Error Evaluation. 8.2 Evaluation Methods for Designing and Pretesting Surveys. 8.3 Methods for Monitoring and Controlling Data Quality. 8.4 Postsurvey Evaluations. 8.5 Summary of Evaluation Methods. Chapter 9. Sampling Error. 9.1 Brief History of Sampling. 9.2 Nonrandom Sampling Methods. 9.3 Simple Random Sampling. 9.4 Statistical Inference in the Presence of Nonsampling Errors. 9.5 Other Methods of Random Sampling. 9.6 Concluding Remarks. Chapter 10.1 Practical Survey Design for Minimizing Total Survey Error. 10.1 Balance Between Cost, Survey Error, and Other Quality Features. 10.2 Planning a Survey for Optimal Quality. 10.3 Documenting Survey Quality. 10.4 Organizational Issues Related to Survey Quality. References. Index.
Article
This chapter includes the following topics: Rationale for a Joint Concern about Costs and ErrorsUse of Cost and Error Models in Sample DesignCriticisms of Cost-Error Modeling to Guide Survey DecisionsNonlinear Cost Models Often Apply to Practical Survey AdministrationSurvey Cost Models are Inherently DiscontinuousCost Models Often Have Stochastic FeaturesDomains of Applicability of Cost Models Must be SpecifiedSimulation Studies Might Best be Suited to Design DecisionsIs Time Money?Summary: Cost Models and Survey Errors
Article
Many surveys of the U.S. household population are experiencing higher refusal rates. Nonresponse can, but need not, induce nonresponse bias in survey estimates. Recent empirical findings illustrate cases when the linkage between nonresponse rates and nonresponse biases is absent. Despite this, professional standards continue to urge high response rates. Statistical expressions of nonresponse bias can be translated into causal models to guide hypotheses about when nonresponse causes bias. Alternative designs to measure nonresponse bias exist, providing different but incomplete information about the nature of the bias. A synthesis of research studies estimating nonresponse bias shows the bias often present. A logical question at this moment in history is what advantage probability sample surveys have if they suffer from high nonresponse rates. Since postsurvey adjustment for nonresponse requires auxiliary variables, the answer depends on the nature of the design and the quality of the auxiliary variables.
Article
Nonexperimental and experimental studies have shown a lack of association between survey effort and nonresponse bias. This does not necessarily mean, however, that additional effort could not reduce nonresponse bias. Theories on nonresponse would suggest the use of different recruiting methods for additional survey effort in order to address nonresponse bias. This study looks at changes in survey estimates as a function of making additional calls under the same protocol and additional calls under a different protocol. Respondents who were interviewed as a result of more than five call attempts were not significantly different on any of the key survey variables than those interviewed with fewer than five calls. Those interviewed under a different survey protocol, however, were different on 5 of 12 measures. Additional interviews under both the same and different protocols contributed to the reduction of total nonresponse error. In sum, the use of multiple protocols for part of the survey effort increased the response rate, changed point estimates, and achieved lower total nonresponse error. Future work is needed on optimizing survey designs that implement multiple survey protocols.
Article
"The paper introduces the general design features and particularities of a new largescale panel study for research on recipients of benefits for the long-term unemployed (the so called Unemployment Benefit II) in Germany that combines a sample of 6000 recipient households with an equally large sample of the general population. Particular focus is on the sampling procedure for the general population, where a commercial database was used to draw a sample stratified by status." (author's abstract, IAB-Doku) ((en))
Article
Critics of public opinion polls often claim that methodological shortcuts taken to collect timely data produce biased results. This study compares two random digit dial national telephone surveys that used identical questionnaires but very different levels of effort: a "Standard" survey conducted over a 5-day period that used a sample of adults who were home when the interviewer called, and a "Rigorous" survey conducted over an 8-week period that used random selection from among all adult household members. Response rates, computed according to AAPOR guidelines, were 60.6 percent for the Rigorous and 36.0 percent for the Standard study. Nonetheless, the two surveys produced similar results. Across 91 comparisons, no difference exceeded 9 percentage points, and the average difference was about 2 percentage points. Most of the statistically significant differences were among demographic items. Very few significant differences were found on attention to media and engagement in politics, social trust and connectedness, and most social and political attitudes, including even those toward surveys.
Article
From 1979 to 1996, the Survey of Consumer Attitudes response rate remained roughly 70 percent. But number of calls to complete an interview and proportion of interviews requiring refusal conversion doubled. Using call-record histories, we explore what the consequences of lower response rates would have been if these additional efforts had not been undertaken. Both number of calls and initially cooperating (vs. initially refusing) are related to the Index of Consumer Sentiment (ICS), but only number of calls survives a control for demographic characteristics. We assess the impact of excluding respondents who required refusal conversion (which reduces the response rate 5–10 percentage points), respondents who required more than five calls to complete the interview (reducing the response rate about 25 percentage points), and those who required more than two calls (a reduction of about 50 percentage points). We found no effect of excluding any of these respondent groups on cross-sectional estimates of the ICS using monthly samples of hundreds of cases. For yearly estimates, based on thousands of cases, the exclusion of respondents who required more calls (though not of initial refusers) had an effect, but a very small one. One of the exclusions generally affected estimates of change over time in the ICS, irrespective of sample size.
Article
A basic estimation strategy in sample surveys is to weight units inversely proportional to the probability of selection and response. Response weights in this method are usually estimated by the inverse of the sample-weighted response rate in an adjustment cell, that is, the ratio of the sum of the sampling weights of respondents in a cell to the sum of the sampling weights for respondents and non-respondents in that cell. We show by simulations that weighting the response rates by the sampling weights to adjust for design variables is either incorrect or unnecessary. It is incorrect, in the sense of yielding biased estimates of population quantities, if the design variables are related to survey non-response; it is unnecessary if the design variables are unrelated to survey non-response. The correct approach is to model non-response as a function of the adjustment cell and design variables, and to estimate the response weight as the inverse of the estimated response probability from this model. This approach can be implemented by creating adjustment cells that include design variables in the cross-classification, if the number of cells created in this way is not too large. Otherwise, response propensity weighting can be applied.
Article
Latent class analysis has been used to model measurement error, to identify flawed survey questions and to estimate mode effects. Using data from a survey of University of Maryland alumni together with alumni records, we evaluate this technique to determine its usefulness for detecting bad questions in the survey context. Two sets of latent class analysis models are applied in this evaluation: latent class models with three indicators and latent class models with two indicators under different assumptions about prevalence and error rates. Our results indicated that the latent class analysis approach produced good qualitative results for the latent class models-the item that the model deemed the worst was the worst according to the true scores. However, the approach yielded weaker quantitative estimates of the error rates for a given item. Copyright (c) 2008 Royal Statistical Society.
Article
Error in survey data originates from failure to contact the sample and from false answers to verifiable questions. These errors may be systematic and associated with uncooperative or unreliable respondents. Zabel modeled attrition in the Survey of Income and Program Participation and found systematic demographic and design effects. Bollinger and David modeled response error and identified correlations to income per capita. In this analysis, we link missing interviews in a panel and response error through a trivariate probit analysis. Robustness of the correlation between attrition and response error is examined by comparing variants of the model. The joint model of response error and attrition becomes the first stage of a pseudolikelihood estimate of a model of food-stamp participation. The model is significantly different from naive probit on the survey data.
Article
Nonresponse weighting is a common method for handling unit nonresponse in surveys. A widespread view is that the weighting method is aimed at reducing nonresponse bias, at the expense of an increase in variance. Hence, the efficacy of weighting adjustments becomes a bias-variance trade-off. This note suggests that this view is an oversimplification -- nonresponse weighting can in fact lead to a reduction in variance as well as bias. A covariate for a weighting adjustment must have two characteristics to reduce nonresponse bias - it needs to be related to the probability of response, and it needs to be related to the survey outcome. If the latter is true, then weighting can reduce, not increase, sampling variance. A detailed analysis of bias and variance is provided in the setting of weighting for an estimate of a survey mean based on adjustment cells. The analysis suggests that the most important feature of variables for inclusion in weighting adjustments is that they are predictive of survey outcomes; prediction of the propensity to respond is a secondary, though useful, goal. Empirical estimates of root mean squared error for assessing when weighting is effective are proposed and evaluated in a simulation study.
Befragung von Arbeitslosengeld-II-Beziehern: Wege aus der Grundsicherung
  • Juliane Achatz
  • Mark Trappmann
Achatz, Juliane, and Mark Trappmann. 2009. Befragung von Arbeitslosengeld-II-Beziehern: Wege aus der Grundsicherung. IAB-Kurzbericht 28/2009. Nürnberg: Institut für Arbeitsmarkt-und Berufsforschung.
Erwerbstätige Leistungsbezieher im SGB II: Aufstocker -bedürftig trotz Arbeit
  • Kerstin Bruckmeier
  • Tobias Graf
  • Helmut Rudolph
Bruckmeier, Kerstin, Tobias Graf, and Helmut Rudolph 2007. Erwerbstätige Leistungsbezieher im SGB II: Aufstocker -bedürftig trotz Arbeit. IAB-Kurzbericht 05/2009. Nürnberg: Institut für Arbeitsmarkt-und Berufsforschung.
Reluctant Respondents: Differences between Early, Late, and Nonrespondents to a Mail Survey
  • Kathy E Green
Green, Kathy E. 1991. ''Reluctant Respondents: Differences between Early, Late, and Nonrespondents to a Mail Survey.'' Journal of Experimental Education 59:268-76.
The Causes of No-opinion Responses to Attitude Measures in Surveys: They Are Rarely What They Appear to Be
  • Jon A Krosnick
Krosnick, Jon A. 2002. ''The Causes of No-opinion Responses to Attitude Measures in Surveys: They Are Rarely What They Appear to Be.'' In Survey Nonresponse, ed. Robert M. Groves, Don A. Dillman, John L. Eltinge, and Roderick J. A. Little. New York: John Wiley & Sons, 87-100.
An Experimental Study of Ways to Increase Exit Poll Response Rates and Reduce Survey Error
  • Daniel Merkle
  • Murray Edelman
  • Kathy Dykeman
  • Chris Brogan
Merkle, Daniel, Murray Edelman, Kathy Dykeman, and Chris Brogan 1998. ''An Experimental Study of Ways to Increase Exit Poll Response Rates and Reduce Survey Error.'' Paper presented at the Annual Meeting of the American Association for Public Opinion Research, St. Louis, Missouri.
The Hunt for the Last Respondent. The Hague: Social and Cultural Planning Office (SCP)
  • Ineke A Stoop
Stoop, Ineke A. L 2005. The Hunt for the Last Respondent. The Hague: Social and Cultural Planning Office (SCP).
Initial Cooperators vs. Converted Refusers: Are There Response Behavior Differences?'' Paper presented at the Annual Conference of the American Association for Public Opinion Research
  • Timothy Triplett
  • Johnny Blair
  • Teresa Hamilton
  • Yun Chiao Kang
Triplett, Timothy, Johnny Blair, Teresa Hamilton, and Yun Chiao Kang 1996. ''Initial Cooperators vs. Converted Refusers: Are There Response Behavior Differences?'' Paper presented at the Annual Conference of the American Association for Public Opinion Research, Salt Lake City: Utah.
When Less Is More: Are Reluctant Respondents Poor Reporters?
  • Ting Yan
  • Roger Tourangeau
Yan, Ting, Roger Tourangeau, and Zac Arens 2004. ''When Less Is More: Are Reluctant Respondents Poor Reporters?'' In Proceedings of the Survey Research Methods Section Toronto, Canada, 4632-51.
When Do Nonresponse Follow-ups Improve or Reduce Data Quality? A Meta-analysis and Review of the Existing Literature Paper presented at the International Workshop on Total Survey Error. Research Triangle Park, NC
  • Kristen Olson
  • Chun Feng
  • Lindsey Witt
Olson, Kristen, Chun Feng, and Lindsey Witt. 2008. ''When Do Nonresponse Follow-ups Improve or Reduce Data Quality? A Meta-analysis and Review of the Existing Literature.'' Paper presented at the International Workshop on Total Survey Error. Research Triangle Park, NC. http://www.niss.org/sites/default/files/OlsonTSEWorkshopNRMEReview080108.pdf (accessed 9/24/10).
Erwerbstätige Leistungsbezieher im SGB II: Aufstocker-bedürftig trotz Arbeit. IAB-Kurzbericht 05
  • Kerstin Bruckmeier
  • Tobias Graf
  • Helmut Rudolph
Bruckmeier, Kerstin, Tobias Graf, and Helmut Rudolph 2007. Erwerbstätige Leistungsbezieher im SGB II: Aufstocker-bedürftig trotz Arbeit. IAB-Kurzbericht 05/2009. Nürnberg: Institut für Arbeitsmarkt-und Berufsforschung.