Mick P. Couper’s research while affiliated with Concordia University Ann Arbor and other places

With the rise of smartphone use in web surveys, voice or oral answers have become a promising methodology for collecting rich data. Voice answers not only facilitate broader and more detailed narratives but also include additional metadata, such as voice amplitude and pitch, to assess respondent engagement. Despite these advantages, challenges persist, including high item non-response rates, mixed respondent preferences for voice input, and labor-intensive, manual answer transcription and coding. This study addresses these last two challenges by evaluating two critical aspects of processing voice answers. First, it compares the transcription performance of three leading Automatic Speech Recognition (ASR) tools—Google Cloud Speech-to-Text API, OpenAI Whisper, and Vosk—using voice answers collected from an open-ended question on nursing home transparency that was administered in an opt-in online panel in Spain. Second, it evaluates the efficiency and quality of coding these transcriptions using human coders and GPT-4o, a Large Language Model (LLM) developed by OpenAI. We found that each of the ASR tools has distinct merits and limits. Google sometimes fails to provide transcriptions, Whisper produces hallucinations (false transcriptions), and Vosk has clarity issues and high rates of incorrect words. Human and LLM-based coding also differ significantly. Thus, we recommend using several ASR tools for voice answer transcription and implementing human as well as LLM-based coding, as the latter offers additional information at minimal added cost.

The evaluation of innovative web-based data collection methods that are convenient for the general public and that yield high-quality scientific information for demographic researchers has become critical. Web-based methods are crucial for researchers with nationally representative research objectives but without the resources of larger organizations. The web mode is appealing because it is inexpensive relative to in-person and telephone modes, and it affords a high level of privacy. We evaluate a sequential mixed-mode web/mail data collection, conducted with a national probability sample of U.S. adults from 2020 to 2022. The survey topics focus on reproductive health and family formation. We compare estimates from this survey to those obtained from a face-to-face national survey of population reproductive health: the 2017–2019 National Survey of Family Growth (NSFG). This comparison allows for maximum design complexity, including a complex household screening operation (to identify households with persons aged 18–49). We evaluate the ability of this national web/mail data collection approach to (1) recruit a representative sample of U.S. persons aged 18–49; (2) replicate key survey estimates based on the NSFG, considering expected effects of the COVID-19 pandemic lockdowns and the alternative modes on the estimates; (3) reduce complex sample design effects relative to the NSFG; and (4) reduce the costs per completed survey.

Objective Retention to complete follow-up surveys in extensive longitudinal epidemiological cohort studies is vital yet challenging. All of Us developed pilot interventions to improve response rates for follow-up surveys. Study design and setting The pilot interventions occurred from April 27, 2020, to August 3, 2020. The three arms were: (1) telephone appointment [staff members calling participants offering appointments to complete surveys over phone] (2) postal [mail reminder to complete surveys through U.S. Postal Service], and (3) combination of telephone appointment and postal. Controls received digital-only reminders [program-level digital recontact via email or through the participant portal]. Study sites chose their study arm and participants were not randomized. Results A total of 50 sites piloted interventions with 17,593 participants, while 47,832 participants comprised controls during the same period. Of all participants, 6,828 (10.4%) completed any follow-up surveys (1448: telephone; 522: postal; 486: combination; 4372: controls). Follow-up survey completions were 24% higher in the telephone appointment arm than in controls in bivariate analyses. When controlling for confounders, telephone appointment and combination arms increased rates of completion similarly compared to controls, while the postal arm had no significant effect (odds ratio [95% Confidence Interval], telephone appointment:2.01[1.81–2.23]; combination:1.91[1.66–2.20]; postal:0.92[0.79–1.07]). Although the effects of the telephone appointment and combination arms were similar, differential effects were observed across sub-populations. Conclusion Telephone appointments appeared to be the most successful intervention in our study. Lessons learned about retention interventions, and improvement in follow-up survey completion rates provide generalizable knowledge for similar cohort studies and demonstrate the potential value of precision reminders and engagement with sub-populations of a cohort.

The most important national surveys of the general population for creating official statistics or public policymaking in many countries, including South Korea, are still conducted using face-to-face interviews with household members. Recently face-to-face surveys have faced threats to data quality from decreasing response rates and rising costs of in-person visits. The COVID-19 pandemic and associated lockdown exacerbated the situation for face-to-face surveys. Survey organizations suspended fieldwork or began to explore alternate means of collecting data. One alternative was a shift to telephone surveys; however, telephone interviews have encountered similar difficulties with declining response rates and increasing costs. Could a self-administered web survey be a viable alternative to interviewer-administered modes such as telephone interviews (CATI) or face-to-face interviews (CAPI)? Smartphones may offer opportunities not offered by other modes. We conducted a smartphone web survey using SMS invitations where a sample of cell phone numbers was selected by random digit dialing (RDD) and compared it with a large-scale national face-to-face survey (CAPI) where a sample of households was selected by stratified cluster sampling. The two surveys were conducted during the COVID pandemic in the second half of 2020. The coverage and sample representation of the smartphone web survey were comparable to that of the face-to-face survey. Despite the relatively small number of respondents, the quality of the smartphone web survey was sufficient to provide accurate data and compared favorably with the CAPI survey. The smartphone web survey yielded more reports of socially undesirable attitudes and behavior than the CAPI survey. The findings will guide researchers to explore new opportunities in establishing a web survey methodology that obtains data more conveniently, efficiently, accurately, and with less cost.

In this case study, we examine a novel aspect of data collected in a typical probability and a typical nonprobability panel: mobile app data. The data were collected in Great Britain in 2018, using the Innovation Panel of the UK Household Longitudinal Study and the Lightspeed online access panel. Respondents in each panel were invited to participate in a month-long study, reporting all their daily expenditures in the app. In line with most of the research on nonprobability and probability-based panel data, our results indicate differences in the data gathered from these data sources. For example, more female, middle-aged, and highly educated people with higher digital skills and a greater interest in their finances participated in the nonprobability app study. Our findings also show that resulting differences in the app spending data are difficult to eliminate by weighting. The only data quality aspect for which we do not find evidence of differences between the nonprobability and probability-based panel is behavior in using the spending app. This finding is contrary to the argument that nonprobability online panel participants try to maximize their monetary incentive at the expense of data quality. However, this finding is in line with some of the scarce existing literature on response behavior in surveys, which is inconclusive regarding the question of whether nonprobability online panel participants answer questions less conscientiously than probability-based panel respondents. Since the two panels in our case study differ in more aspects than the sample selection procedure, more research in different contexts is necessary to establish generalizability and causality.

We report on an experiment in a supplemental web survey as part of a longitudinal study in the United Kingdom where we ask survey respondents to consent to two forms of data linkage to health records and to consent to be mailed a serology kit. We varied the placement (early, early in context, or late in the survey) and order (linkage first or serology first) of the consent requests. We also examine reasons for consent or non-consent. We find that order of the requests does not make much difference, but making the requests early in the survey significantly increases consent rates over asking them after a series of content-related questions (by 3.4 percentage points) or later in the survey (by 7.2 percentage points). This is consistent with previous research showing that early requests for consent in a survey have a positive effect. The main reason chosen for not consenting related to the personal nature of the information requested.

Understanding Society, the UK Household Longitudinal Study has a programme of research and development that underpins innovations in data collection methods. One of our current focuses is on using mobile applications to collect additional data that supplement data collected in annual interviews. To date, we have used mobile apps to collect data on consumer expenditure, well‐being, anthropometrics and cognition. In this paper, we review the potential barriers to data collection using mobile apps and experimental evidence collected with the Understanding Society Innovation Panel, on what can be done to reduce these barriers.

The grid question refers to a table layout for a series of survey question items (i.e., sub-questions) with the same introduction and identical response categories. Because of their complexity, concerns have already been raised about grids in web surveys on PCs, and these concerns have heightened regarding mobile devices. Some studies suggest decomposing grids into item-by-item layouts, while others argue that this is unnecessary. To address this challenge, this paper provides a comprehensive evaluation of the grid layout and four item-by-item alternatives, using 10 response quality indicators and 20 survey estimates. Results from the experimental web survey (n = 4644) suggest that item-by-item layouts (unfolding or scrolling) should be used instead of grids, not only on mobile devices but also on PCs. While the former justifies the already increasing use of item-by-item layouts on mobile devices in survey practice, the latter implies that the prevailing routine of using grids on PCs should be reconsidered.

The American Family Health Study (AFHS) collected family health and fertility data from a national probability sample of persons aged 18–49 between September 2021 and May 2022, using web and mail exclusively. In July 2022, we surveyed AFHS respondents and gauged their willingness to become part of a national web panel that would create novel longitudinal data on these topics. We focus on predictors of willingness to participate, identifying the potential selection bias that this type of approach may introduce. We found that efforts of this type to create a national web panel may introduce potential selection bias in estimates based on the panel respondents, with individuals having higher socioeconomic status being more cooperative. Thus, alternative recruitment strategies and re-weighting of the subsample may be needed to further reduce selection bias. We present methodological implications of our results, limitations of our approach, and suggestions for further research on this topic.

The American Family Health Study (AFHS) collected family health and fertility data from a national probability sample of persons aged 18–49 between September 2021 and May 2022, using web and mail exclusively. In July 2022, we surveyed AFHS respondents and gauged their willingness to become part of a national web panel that would create novel longitudinal data on these topics. We focus on predictors of willingness to participate, identifying the potential selection bias that this type of approach may introduce. We found that efforts of this type to create a national web panel may introduce potential selection bias in estimates based on the panel respondents, with individuals having higher socioeconomic status being more cooperative. Thus, alternative recruitment strategies and re-weighting of the subsample may be needed to further reduce selection bias. We present methodological implications of our results, limitations of our approach, and suggestions for further research on this topic.