Article

Presidential Address: Applying a Total Error Perspective for Improving Research Quality in the Social, Behavioral, and Marketing Sciences

Authors:
To read the full-text of this research, you can request a copy directly from the author.

Abstract

The topic about which I am speaking today is very special to me, and I have been looking forward to making a presentation such as this for a long time. But please know that the ideas about which I will be speaking continue to evolve for me, and what I would have said about this topic twenty years ago, or even ten years ago, would not have been as �developed� as it is for me today (cf. Lavrakas 2012, 2013). I would like to begin by noting the three major premises that underlie the views I will be expressing. These come from my nearly forty years as a researcher, during which I have encountered a great many and wide variety of social, behavioral, and marketing research studies, both quantitative and qualitative in nature. First, I believe that many of these studies were conceptualized poorly, executed poorly, and/or interpreted poorly. Second, I believe that the quality of most of these studies could have been improved with few, if any, cost implications. And third, I believe that using the Total Error framework, about which I am speaking today, can help bring about a meaningful improvement in research quality. Many in the audience already are familiar with the Total Survey Error (TSE) approach (cf. Groves 1989; Fuchs 2008). But I sense that many more are not familiar with it. Furthermore, from what I have observed in the past twenty-plus years, few appear to apply the approach broadly to the diverse realms of social, behavioral, and marketing research. And yet, it is what I call the �Total Error� perspective that underlies the TSE perspective. I am not sure why this is the case, but my goal today is to demonstrate why thinking broadly about a Total Error (TE) approach, not merely �

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the author.

... Online pre-workshop surveys can provide information to design risk communications for both traditional written products and engagement opportunities, such as meetings with stakeholders. Pre-and post-surveys might have better response rates if they are implemented according to survey research practices described in Dillman (2000) and discussed in Lavrakas (2013). Post-workshop surveys may be useful, but the experiences here suggest that they require very careful implementation, with incentives and reminders such as those described in Dillman (2000) to encourage a representative sample of workshop participants to respond. ...
... Lavrakas' schematic for the Total Error Framework, fromLavrakas (2013). ...
Article
Full-text available
This study applies a mental models survey approach to assess public thinking about oil spills and oil spill response. Based on prior interdisciplinary oil spill response research, the study first applies qualitative interview results and a response risk decision model to the design of a survey instrument. The decision model considers controlled burning, public health, and seafood safety. Surveying U.S. coastal residents (36,978 pairs of responses) through Google Insights identifies beliefs and gaps in understanding as well as related values and preferences about oil spills, and oil spill responses. A majority of respondents are concerned about economic impacts of major oil spills, and tend to see ocean ecosystems as fragile. They tend to see information about chemical dispersants as more important than ecological baseline information, and dispersants as toxic, persistent, and less effective than other response options. Although respondents regard laboratory studies as predictive of the effects of oil and of controlled burning, they are less confident that scientists agree on the toxicity and effectiveness of dispersants. The results illustrate opportunities to reframe discussions of oil spill response in terms of tradeoffs between response options, and new possibilities for assessing public opinions and beliefs during events.
... Online pre-workshop surveys can provide information to design risk communications for both traditional written products and engagement opportunities, such as meetings with stakeholders. Pre-and post-surveys might have better response rates if they are implemented according to survey research practices described in Dillman (2000) and discussed in Lavrakas (2013). Post-workshop surveys may be useful, but the experiences here suggest that they require very careful implementation, with incentives and reminders such as those described in Dillman (2000) to encourage a representative sample of workshop participants to respond. ...
... Lavrakas' schematic for the Total Error Framework, fromLavrakas (2013). ...
Article
Full-text available
Oil spill stakeholders, including decision makers and other groups, have expressed concerns about and questioned the use of dispersants and other non-mechanical response options for years. Concerns in past decades were primarily ecological, but during the Deepwater Horizon oil spill some individuals and communities in the Gulf of Mexico states also articulated perceptions of public health risks associated with the use of dispersants. Effective risk communication is essential to manage the potential risks associated with oil spills. Stakeholders concerned about risks want or need information in the form of communications products, such as guides or briefs. Because people process new information within the context of their existing beliefs, such communication products are likely to be more effective and useful for their intended audiences if they are designed to: (1) take into account the communication recipients' existing beliefs; and (2) directly address the decisions/judgments faced by recipients by providing them with the information they want and need to make those decisions. Stakeholder engagement is essential to learn about risk perceptions, to learn about what information stakeholders want and need to make decisions effectively, and to develop communication products to that end. This paper builds upon a mental models approach to dispersant risk communications research from the 1990s. It describes and presents results from an industry-government collaborative project to develop risk-based tools designed to communicate the subject of dispersants to local level stakeholders among others. This project includes an expanded science-informed dispersant decision model, two stakeholder open houses, and two surveys (interactive and online) to gather data-driven insights about local stakeholder knowledge and understanding of dispersants, as well as their perceptions of the risks and benefits associated with dispersant use during a spill in relation to other response options. The surveys were distributed at two open houses for local stakeholders on the Eastern Shore of Virginia (Wallops Is.) and the Pacific Northwest (Port Townsend, WA). Both open houses were co-sponsored with USCG-led Area Committees. The Virginia workshop was also co-sponsored by The Nature Conservancy and the Virginia Shore Keeper. It is expected that the surveys may be incorporated into future meetings and open houses involving stakeholders at any level, e.g., local, state, regional or national. The data from the surveys can guide the improvement of future communication efforts about dispersants, as well as provide support for enhanced stakeholder engagement during preparedness and response.
... In Poland, the participants were recruited by a professional research firm, which used the CATI survey procedure [109]. There were two inclusion criteria. ...
Article
Full-text available
In the world of university research, although the figure of project manager is not formally foreseen, the principal researcher (PR) is, at many times, the last responsible the project results, schedule, and cost. The study aimed to investigate, in the light of the literature and through a cross-cultural study conducted in Italy and Poland, the relationship between soft skills (empowering leadership style, self-efficacy beliefs, and collective efficacy) of the principal researcher (PR) and the perceived success of research projects and satisfaction with the project, taking into account cross-cultural differences. A total of 67 PRs of complex projects in public universities (28 in Italy and 39 in Poland) participated in the study, completing a self-report questionnaire. Data were analyzed using descriptive and correlational analyses. The results showed a significantly higher mean value for team management self-efficacy in a Polish sample and a higher satisfaction with projects in Italian sample. All the soft skills included in the study were related to project success and satisfaction with the project. The results could be used to identify possible ways of intervention to establish a more mature project culture in public research organizations.
... Other typical limitations that all surveys face are also important issues for tobacco surveys. According to survey error models, one must account for possible errors like sampling, coverage, nonresponse and inferential errors (Lavrakas, 2013). In this context, several errors could be decreased by focusing surveys on total consumption rather than illicit consumption. ...
Article
Full-text available
Social structures embed technologies, which people use in the implementation of national single window digital platforms. We argue that stakeholder interests determine digital transformation – with people being the key to understanding how digital platforms change or do not change the environments into which they are introduced. In our multicase studies based in the port of Tema, Ghana, stakeholders have divergent opinions about technology, which causes conflicts. Our empirical findings reflect the interpretive flexibility that moderates the traditional dichotomy between technological determinism and social constructivism. By employing the theory of sociotechnical systems, we identify the frictions and interlinkages of non‑technological factors.
... This section documents the data and methods we used, the decisions and assumptions we made in the data collection and analysis process, and ways we mitigated the limitations. This transparent documentation approach was guided by the quality standards for qualitative research described in Lavrakas (2013) and Roller and Lavrakas (2015). ...
Chapter
Full-text available
Focus groups are increasingly used in a range of disciplines to obtain data from culturally and linguistically diverse populations. However, there is a scarcity of research on how conversational styles impact focus group discussions due to cultural differences in communication norms. Scholars in the field of sociolinguistics have long pointed out that systematic, observable differences in communication styles are evident across different languages. In this study, we used a coding scheme developed on the basis of sociolinguistic theory to analyze the linguistic behavior of speakers of five languages (English, Chinese, Korean, Spanish, and Vietnamese) to compare their participatory pattern in the focus group discussion of survey research materials. A systematic analysis of quantitative and qualitative data demonstrates the interrelatedness of cultural norms of communication and participatory patterns. Results show that speakers of Western languages (English and Spanish) demonstrate similar high-involvement styles or strong participatory patterns. Speakers of Asian languages (Chinese and Vietnamese) shared similar low-involvement styles or weak participatory patterns, but the moderator’s subject-matter expertise and methodological design to achieve homogenous group composition could minimize the differences in participatory patterns between Asian (Korean) and Western languages. These results suggest that while there are notable differences in focus group interactions across cultural and linguistic groups, focus groups can be still a valid research method across different cultures and languages. Its efficiency as a research method will increase if linguistic and cultural characteristics are properly considered in the research design.
... This section documents the data and methods we used, the decisions and assumptions we made in the data collection and analysis process, and ways we mitigated the limitations. This transparent documentation approach was guided by the quality standards for qualitative research described in Lavrakas (2013) and Roller and Lavrakas (2015). ...
Chapter
Full-text available
In recent years, substantial advances have been made in the field of multinational, multiregional, and multicultural research (commonly referred to as 3MC survey research; Johnson, Pennell, Stoop, & Dorer, 2018). This research magnifies challenges associated with monocultural studies and poses unique ones at both the organizational and methodological levels. Because cross-cultural surveys seek to make comparative estimates across populations, the data must be valid and reliable for each specific group, as well as comparable across them. Even when questionnaires are carefully translated and adapted, groups may systematically differ in the way they interpret certain questions or respond to them, posing a threat to the validity of the comparisons. In this context, pretesting becomes particularly beneficial to identify potential problems in survey questions and to assess comparability (Willis, 2015). This chapter introduces the concept and importance of pretesting in cross-cultural survey research. The most common methods used to pretest 3MC surveys are described, highlighting recent applications and developments, as well as current challenges. These methods include cross-cultural cognitive interviewing, online probing, vignettes, and behavior coding. Next, reference is made to the combination of multiple pretesting methods to assess and improve cross-cultural surveys. In the last section of this chapter, the main challenges and opportunities of pretesting in comparative contexts are discussed.
... This section documents the data and methods we used, the decisions and assumptions we made in the data collection and analysis process, and ways we mitigated the limitations. This transparent documentation approach was guided by the quality standards for qualitative research described in Lavrakas (2013) and Roller and Lavrakas (2015). ...
... Strategies have been put in place to encourage participation, including a personalized approach by PBRN and accreditation of training for physicians and nurses. There may also be a selection bias [55] which will be taken into account in the statistical analyses. First, during the data collection process, we will follow up on respondents closely to develop customized strategies with the PBRNs to encourage professionals' participation in the survey. ...
Article
Full-text available
Background: Timely access in primary health care is one of the key issues facing health systems. Among many interventions developed around the world, advanced access is the most highly recommended intervention designed specifically to improve timely access in primary care settings. Based on greater accessibility linked with patients' relational continuity and informational continuity with a primary care professional or team, this organizational model aims to ensure that patients obtain access to healthcare services at a time and date convenient for them when needed regardless of urgency of demand. Its implementation requires a major organizational change based on reorganizing the practices of all the administrative staff and health professionals. In recent years, advanced access has largely been implemented in primary care organizations. However, despite its wide dissemination, we observe considerable variation in the implementation of the five guiding principles of this model across organizations, as well as among professionals working within the same organization. The main objective of this study is to assess the variation in the implementation of the five guiding principles of advanced access in teaching primary healthcare clinics across Quebec and to better understand the influence of the contextual factors on this variation and on outcomes. Methods: This study will be based on an explanatory sequential design that includes 1) a quantitative survey conducted in 47 teaching primary healthcare clinics, and 2) a multiple case study using mixed data, contrasted cases (n = 4), representing various implementation profiles and geographical contexts. For each case, semi-structured interviews and focus group will be conducted with professionals and patients. Impact analyses will also be conducted in the four selected clinics using data retrieved from the electronic medical records. Discussion: This study is important in social and political context marked by accessibility issues to primary care services. This research is highly relevant in a context of massive media coverage on timely access to primary healthcare and a large-scale implementation of advanced access across Quebec. This study will likely generate useful lessons and support evidence-based practices to refine and adapt the advanced access model to ensure successful implementation in various clinical contexts facing different challenges.
... El enfoque del error total de la encuesta, marco teórico dominante en el momento actual (entre otros, Lavrakas, 2013;Lyberg, 2012;Groves y Lyberg, 2010), clasifica en dos los tipos de errores de la técnica de la encuesta: los causados por una incorrecta representación y los originados por defectos de medición. El primero agrupa al error de muestreo, cobertura y no respuesta; mientras que los errores de medición pueden tener su origen en los entrevistadores, los entrevistados, el cuestionario y el modo de recogida de información. ...
Article
Full-text available
In this work we make a comparison between a face-to-face and landline CATI survey using two equivalent samples. The questionnaire has forced-choice questions with vertical disposition of the answers, and grid questions. In the first part we compare the presence of answer effects (primacy and recency) and the number of answers “not know”. The telephone survey has fewer partial nonresponse, and presence of recency effect in questions with vertical disposition with 4 and 5 categories. In the battery questions we verify if the telephone surveys have a smaller degree of differentiation in the scales, a greater election of extreme answers, as well as a greater assent. This phenomenon has been detected in the questions of 5 and 10 categories of answer, not detecting effect primacy in questions administered with card.
... This model successfully organized decades of empirical research within a single unifying theoretical framework. An expanded elaboration of the TSE model has been more recently presented by Lavrakas [9], in which he identifies two general classes of errors, measurement and representation, and then explores multiple subclasses of errors within each. Table 1 lists the various elements of the Lavrakas TSE model. ...
Article
Full-text available
Population-based estimates of substance use patterns have been regularly reported now for several decades. Concerns with the quality of the survey methodologies employed to produce those estimates date back almost as far. Those concerns have led to a considerable body of research specifically focused on understanding the nature and consequences of survey-based errors in substance use epidemiology. This paper reviews and summarizes that empirical research by organizing it within a total survey error model framework that considers multiple types of representation and measurement errors. Gaps in our knowledge of error sources in substance use surveys and areas needing future research are also identified.
... Теоретическая рамка В подходе, развиваемом представителями Мичиганской методологической школы, совокупная ошибка исследования делится на две больших части: ошибки опросного инструмента и ошибки репрезентации [9,11]. Первые имеют отношение к коммуникативной ситуации, восприятию и регистрации мнений опрашиваемых, вторые -к организации отбора, неточностям построения выборочного дизайна и нерелевантному с точки зрения выборочных требований поведению отобранных людей (рис. ...
Chapter
Population-based estimates of substance use patterns have been regularly reported for several decades. Concerns with the quality of the survey methodologies employed to produce these estimates, however, date back almost as far. These concerns have led to a considerable body of research specifically focused on understanding the nature and consequences of survey-based errors in substance use epidemiology. This chapter reviews and summarizes that empirical research by organizing it within a total survey error model framework that considers multiple types of representation and measurement errors. Gaps in knowledge regarding sources of error in substance use surveys and areas needing future research are also identified.
Article
Full-text available
Household surveys are still the most popular tool for gathering sociological data, while they allow to get the most comprehensive and proper information. However, the quality of information obtained through personal interviews, is largely determined by the quality of the interviewer’s work. It is reflected in his or her conscious desire to fulfill all the instructions and avoid substandard information during a fieldwork, which appears due to questions’ distortion, violation of their priority, etc. Often a large number of field errors come down to the interviewers’ fraud caused, in turn, by lack of prior training and coaching. The authors of this article joined one of the leading polling companies as freelancers in attempt to see how the field phase of the research based on a household survey technology and the interviewers’ work are organized. The study found that the methodical organization of interviewers’ work (including group coaching and training) is practically nonexistent and is expressed only in a few rules emerging from case to case. Interviewers are not included in the process of the company functioning and are forced to act on a whim, orienting by themselves and considering work optimization tactics on their own initiative in a vast field of research. The authors see firsthand through their experience of conducting surveys that the lack of regular interaction with colleagues and the leadership encourages interviewers to create their own practices and standards of conduct based on falsification and fabrication.
Article
This chapter includes the following topics: Rationale for a Joint Concern about Costs and ErrorsUse of Cost and Error Models in Sample DesignCriticisms of Cost-Error Modeling to Guide Survey DecisionsNonlinear Cost Models Often Apply to Practical Survey AdministrationSurvey Cost Models are Inherently DiscontinuousCost Models Often Have Stochastic FeaturesDomains of Applicability of Cost Models Must be SpecifiedSimulation Studies Might Best be Suited to Design DecisionsIs Time Money?Summary: Cost Models and Survey Errors
Measurement Dependency: A Metric for Response Representativeness and Better Sample Design
  • Fan
  • David
Fan, David P. 2013. " Measurement Dependency: A Metric for Response Representativeness and Better Sample Design. " Survey Practice 6(2), http://www.surveypractice.org/index.php/ SurveyPractice/article/view/43.