Measuring Attitudes towards Surveys:
A Validation Study
Relevance & Motivation
General attitudes towards surveys …
… are part of respondents’ motivation for survey participation
… predict participants’ willingness to perform supportively during (online)
surveys (e.g. de Leeuw et al., 2017; Stocké, 2006)
Ulrike Schwabe (firstname.lastname@example.org) | Thorsten Euler (email@example.com) |
Isabelle Fiedler (firstname.lastname@example.org) | Niklas Jungermann (email@example.com)
De Leeuw, E., Hox, J., Lugtig, P., Scherpenzeel, C. V., Goritz, A., & Bartsch, S. (2010): Measuring and comparing survey attitude among new and repeat
respondents cross-culturally. In 63rd Annual Conference World Association for Public Opinion Research (WAPOR), Chicago.
De Leeuw, E., Hox, J., & Rosche, B. (2017): Survey attitude, nonresponse and attrition in a probability-based online panel. In International Workshop
on Household Survey Nonresponse.
De Leeuw, E., Hox, J., Silber, H., Struminskaya, B., & Vis, C. (2019): Development of an international survey attitude scale: Measurement equivalence,
reliability, and predictive validity. Measurement Instruments for the Social Sciences, 1(1), 1-10.
Gengler, J., Tessler, M., Lucas, R., & Forney, J. (2021). ‘Why Do You Ask?’ The Nature and Impacts of Attitudes towards Public Opinion Surveys in the
Arab World. British Journal of Political Science, 51(1), 115-136.
Loosveldt, G., & Storms, V. (2008). Measuring public opinions about surveys. International Journal of Public Opinion Research, 20(1), 74-89.
Stocké, V. (2006). Attitudes toward surveys, attitude accessibility and the effect on respondents’ susceptibility to nonresponse”. Quality & Quantity,
Stocké, V., & Langfeldt, B. (2003). Umfrageeinstellung und Umfrageerfahrung : die relative Bedeutung unterschiedlicher Aspekte der
Interviewerfahrung für die generalisierte Umfrageeinstellung. ZUMA Nachrichten, 27(52), 55-90.
Struminskaya, B., Bosnjak, M., de Leeuw, E., Lugtig, P., & Toepoel, V. (2015): GESIS Panel Core Study Module - Panel Survey Participation Evaluation &
Mode 65 Preferences. In: GESIS (Ed.): GESIS Panel Study Descriptions. Related to ZA5664 and ZA5665. Version 10.0.0. 127–129. Mannheim.
Summary of Results
(1) CFA results indicate replicability of the proposed scales with our sample. However,
the SAS model performs less accurate than the alternative model (ATS), yet two out
of three SAS dimensions demonstrate higher reliability scores (see Table 1 and 2).
(2) As expected, we find high correlations between those items which intend to
measure similar dimensions such as survey value and survey reliability or survey
burden and survey privacy (see Table 3).
(3) EFA results do not support alternative dimensions in our data that are superior to
existing ones (results not reported, but available on request).
The recommended scales are promising to measure attitudes towards surveys. Despite
some weaknesses, the SAS instrument is valid and efficient. As our validation study is
based on a sample of highly qualified graduates, generalizing our results to the entire
population should be taken with caution.
Schwabe, U., Euler, T., Fiedler, I., Jungermann, N. (2023): Measuring Attitudes towards Surveys: A Validation Study. General Online Research (GOR) Conference: Kassel.
Survey Attitude Scale (SAS): Nine-Item Model
(de Leeuw et al. 2010, 2019; own illustration; unstandardized factor loadings)
The Survey Attitude Scale (SAS)
differentiates between three dimensions
(de Leeuw et al., 2010, 2019; Struminskaya et al. 2015):
i. survey enjoyment (se)
ii. survey value (sv)
iii. survey burden (sb)
Other research proposes additional
dimensions (Gengler et al., 2021;
Looseveldt & Storms, 2008):
i. survey reliability (sr)
ii. survey privacy (sp)
iii. survey intentions (si)
se1 I enjoy answering questionnaires sent to me by mail or web. Survey
⍺ ≈ 0.81
⍵ ≈ 0.82
⍺ ≈ 0.80
⍵ ≈ 0.79
se2 I enjoy being interviewed for surveys.
se3 I find surveys generally interesting.
sv1 In my opinion, surveys are important for society. Survey
⍺ ≈ 0.85
⍵ ≈ 0.86
sv2 I think, useful information can be obtained from surveys.
sv3 In my opinion, participation in surveys is a waste of time.
sb1 I am asked way too often to participate in a survey. Survey
⍺ ≈ 0.55
⍵ ≈ 0.57
sb2 I perceive opinion surveys as an intrusion in my privacy.
sb3 I find it annoying to answer many questions in an interview.
se1 se2 se3
sv1 sv2 sv3
sb1 sb2 sb3
Attitudes Towards Surveys (ATS): Alternative Dimensions
(Gengler et al., 2021, Looseveldt & Storms, 2008; own illustration; unstandardized factor loadings)
sr1 sr2 sr3
sp1 sp2 sp3
si1 si2 si3
sr1 Participants in surveys do their best to give truthful answers to the
questions they are asked. (Gengler et al., 2021) Survey
⍺ ≈ 0.58
⍵ ≈ 0.58
⍺ ≈ 0.78
⍵ ≈ 0.76
sr2 Most institutions that conduct public opinion surveys work hard to
make their surveys as accurate as possible. (Gengler et al., 2021)
sr3 If a survey is well done, it will give very accurate information about the
views of the people surveyed. (Gengler et al., 2021)
sp1 I sometimes hesitate about taking part in a survey because I do not
know what will happen with my replies. (Loosveldt & Storms, 2008) Survey
⍺ ≈ 0.79
⍵ ≈ 0.82
sp2 Surveys tend to include questions that are too personal. (Loosveldt &
sp3 Surveys often ask something that is no one's business. (Stocké & Langfeldt,
2003, own translation)
si1 The results of surveys are usually heavily influenced by the personal
interests of political preferences of the people conducting the
research. (Gengler et al., 2021) Survey
⍺ ≈ 0.69
⍵ ≈ 0.70
si2 Surveys are frequently used to manipulate and mislead people. (Gengler
et al., 2021)
si3 I have the impression that surveys are always designed to produce the
desired results. (self-development, own translation)
1 0.93 0.50 0.43 1 0.97 -0.50 0.57 1 1.17 0.93
1 1.48 0.87 1 1.14 1.13 1 1.13 1.15
●Different items from the literature included in a modularized German survey for
higher education graduates in 2019.
●Respondents in the module “survey attitudes” answered…
… items capturing survey enjoyment, survey value and survey burden (SAS)
(n=1,369, >1% item non-response).
… items capturing survey reliability, survey privacy and survey intentions (ATS)
(n=1370, about 2-3% item non-response).
1) Replication of the original scale(s) applying confirmatory factor analyses (CFA),
reliability using Cronbach’s Alpha and McDonald’s Omega.
2) Convergent and discriminant validity by correlating different dimensions.
3) Exploratory factor analysis (EFA) using the entire item set (results not reported).
Data: DZHW graduate panel 2009 (3rd wave)
●panel of higher education graduates
from German HEI, established in
●module “survey attitudes”
measured in 2019.
●gross sample n = 3,345 participants
(response rate: 66.5 %).
●analytical sample for …
… SAS: n = 1,304
… ATS: n = 1,287.
Survey Value 0.31 1.00
-0.54 -0.58 1.00
0.39 0.78 -0.58 1.00
-0.29 -0.43 0.86 -0.48 1.00
-0.29 -0.54 0.72 -0.77 0.71 1.00
Table 1: Survey Attitudes Scale (de Leeuw et al., 2010, 2019): Item wording and reliability
scores (Cronbach’s Alpha and McDonald’s Omega) for three dimensions. Own calculations.
Table 2: Attitudes Towards Surveys: Item wording and reliability scores (Cronbach’s Alpha and
McDonald’s Omega) for three dimensions. Own calculations.
Table 3: Pearson correlations between the different dimensions of the SAS and the ATS (all
significant at p < 0.001). Bold: correlations > 0.7. Own calculations.