Article

Do Respondents Really "Mark All That Apply" On Self-Administered Questions?

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

An experiment was conducted to assess the effect of using “mark all that apply” question instructions on survey reporting as part of the field test for the Second Follow-up of the National Education Longitudinal Study of 1988 Eighth Graders (NELS:88). Mark-all-that-apply instructions were compared with instructions asking respondents to indicate “yes” or “no” to each response option on responses to three items dispersed throughout the questionnaire and consisting of different topics and numbers of response options. For the three items, significantly fewer response options were selected with the mark-all-that-apply instructions than with the yes/no instructions, but because external validity criteria were not available, overreporting to the yes/no instructions cannot be ruled out. Instructiondependent primacy effects, predicted under the hypothesis that respondents would engage in more superficial processing when given the mark-all-that-apply instructions, were not found.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... When the respondents are forced to select between a yes/no answer (or even an agree/disagree choice) they tend to express more positive opinions rather when they have a "selectall-that-apply" option (Dillman et al. 2003); hence, multiple-answer questions allow the respondents to express better their opinion. Rasinski et al. (1994) show that it is possible that respondents do not really mark "all-that-apply" when answering to this kind of question. Furthermore, there is evidence that the first given answers receive more selections than the other ones (Dillman et al. 2003;Rasinski et al. 1994). ...
... Rasinski et al. (1994) show that it is possible that respondents do not really mark "all-that-apply" when answering to this kind of question. Furthermore, there is evidence that the first given answers receive more selections than the other ones (Dillman et al. 2003;Rasinski et al. 1994). We conjecture that, by giving a fixed amount of answers to select, we induce the respondents to read all the possible answers. ...
... As shown in the previous section, some limits of closed questions still remain, in particular the possible different interpretation of the terms used in the answer options (for instance, in the analysis of the third cluster). However, we can observe that the choice of allowing the selection of only three options resulted in avoiding one of the usual limits of multiple-answer question: differently than observed in previous research (Dillman et al. 2003;Rasinski et al. 1994), the respondents did not show a preference for the first options in the list, indeed options A and C were among the less selected. This observation suggests that while answering to this kind of questions, respondents pay more attention to the selection of those options that mainly represent their opinion. ...
Chapter
In this chapter, we argue for a conceptualisation of thinking, doing and feeling as inseparable. We argue that affect is not a property of an individual, but rather a distributed and relational flow among participants and objects. We also draw on the notion of ritualisation, within the teaching and learning of mathematics, as pointing towards activity which may be mute but is far from unthinking. Using empirical data from a series of one-to-one teaching sessions, we propose that it is possible to analyse affective flows, for example, in affective aligning, or misaligning. We have tentative evidence that ritualisation activity supports affective aligning and hence has a potentially powerful role to play in learning mathematics.
... Although some researchers have shown that both the CAS and CATA question formats provide similar results in terms of outcomes, time, and survey satisfaction, other researchers [12,23] disagree and suggest otherwise. Fundamentally, the CAS seeks a response (e.g., yes or no) for each item, while the CATA question format requires that respondents only check those that they believe apply (the "yes" response) [24]. Sudman [1] and Smyth et al. [10] suggest that respondents pay more attention, read all the items, and provide more thoughtful responses for CAS than CATA questions. ...
... Sudman [1] and Smyth et al. [10] suggest that respondents pay more attention, read all the items, and provide more thoughtful responses for CAS than CATA questions. CAS has also been shown to result in more detailed responses in terms of a mean number of affirmative checked ("agree" or "apply") responses per respondent as compared to the CATA format [10,17,24]. This finding is also consistent with behavior survey data conducted in different languages and countries of residence [25]. ...
... Data for proteins, dairy, fruit, and sweet/dessert foods are presented in the (Tables A1-A4). The CAS format amassed a higher percentage of "agree" responses than the CATA question format, which is similar to that mentioned or found by other authors [1, [10][11][12]23,24]. For example, baked potatoes (a starchy food) in the USA (Table 4) received more "agree" responses for all of the eating motivation constructs when CAS was used as compared to when the CATA question format was used. ...
Article
Full-text available
Check All That Apply (CATA) has become a popular type of questionnaire response in sensory/consumer research in recent years. However, some authors have pointed out potential problems with the method. An online survey using either a Check-All-That-Apply (CATA) or Check-All-Statements (CAS) format for questions was conducted to provide a deeper understanding of the response data using the two question formats. With CATA, respondents select all terms or statements that apply from a given list, while, with CAS, respondents must respond (e.g., yes/no or agree/disagree) to each term or statement to show that it applies or does not apply. Respondents from five countries (Brazil, China, India, Spain, and the USA) were randomly assigned one of the two question formats (N = 200 per country per method). Motivations for eating items that belong to five food groups (starchy, protein, dairy, fruits, and desserts) were assessed. Results showed that CAS had higher percentages of “agree” responses than CATA. Also, the response ratio of CAS and CATA data was different, suggesting that interpretations of the data from each response type would also be different. Respondents in the USA, China, and Spain took longer to complete the CAS questionnaire, while respondents in Brazil and India had similar time durations for the two question formats. Overall, the CATA format was liked slightly more than the CAS format and fewer respondents dropped out of the survey when using the CATA response type. These findings suggest that the CATA format is quick and relatively easy for consumers to complete. However, it provokes fewer “apply” responses, which some psychologists suggest underestimates applicable terms or statements and CATA provides a different interpretation of data than the CAS format that requires consumers to respond to each term or statement. Further, CAS may overestimate the applicable terms. Consumer insights collected using CATA and CAS can lead to different decisions due to differences in data interpretation by researchers (e.g., marketers, nutritionists, product developers, and sensory scientists). More investigation is needed for the CATA and CAS question formats.
... It also is beneficial to product developers, sensory scientists, and marketing researchers, as it guides them in producing and promoting food products that meet the needs of consumers. Various formats of questions have been used in consumer surveys to collect food product characterizations based on perceptions, opinions, beliefs, and attitudes of target group consumers [1][2][3][4][5][6][7][8]. Some commonly used ones include Check-All-That-Apply (CATA), Check-All-Statements (CAS), Rate-All-That-Apply (RATA), and Rate-All-Statements (RAS). ...
... Some commonly used ones include Check-All-That-Apply (CATA), Check-All-Statements (CAS), Rate-All-That-Apply (RATA), and Rate-All-Statements (RAS). These question formats are commonly used in consumer central location studies [2,7,[9][10][11][12], phone interview surveys [13,14], selfadministered studies (home-use tests [15,16] and on-line surveys [17,18]), and printed surveys [8] for a number of different types of studies related to consumer perception. ...
... With CATA, consumers are asked to check all items that are of importance from a list of options [8,19]. The items provided are usually product sensory characteristics [9,20] and physiological and psychographic variables [16]. ...
Article
Full-text available
Question formats are critical to the collection of consumer health attitudes, food product characterizations, and perceptions. The information from those surveys provides important insights in the product development process. Four formats based on the same concept have been used for prior studies: Check-All-That-Apply (CATA), Check-All-Statements (CAS), Rate-All-That-Apply (RATA), and Rate-All-Statements (RAS). Data can vary depending on what question format is used in the research, and this can affect the interpretation of the findings and subsequent decisions. This survey protocol compares the four question formats. Using a modified version of the Eating Motivation Survey (EMS) to test consumer eating motivations for five food items, each question format was translated and randomly assigned to respondents (N = 200 per country per format) from Brazil (Portuguese), China (Mandarin Chinese), India (Hindi or English), Spain (Spanish), and the USA (English). The results of this survey should provide more understanding of the differences and similarities in distribution of data for the four scale formats. Also, the translations and findings of this survey can guide marketers, sensory scientists, product developers, dieticians, and nutritionists when designing future consumer studies that will use these question formats.
... These alternatives require the respondent to look at each item individually and Sudman and Bradburn (1982) believed these alternatives could remove or reduce list order effects. Several experimental studies have shown that for the same item the percentage of "yes" responses in the "yes/no" format is higher than the percentage choosing the item in the "mark all that apply" format (Rasinski, Mingay, & Bradburn, 1994;Smyth, Dillman, Christian, & Stern, 2006;Thomas & Klein, 2006). This is true for comparisons of individual items (Thomas & Klein, 2006) as well as for comparisons of the mean number of endorsements (Rasinski et al., 1994;Smyth et al., 2006). ...
... Several experimental studies have shown that for the same item the percentage of "yes" responses in the "yes/no" format is higher than the percentage choosing the item in the "mark all that apply" format (Rasinski, Mingay, & Bradburn, 1994;Smyth, Dillman, Christian, & Stern, 2006;Thomas & Klein, 2006). This is true for comparisons of individual items (Thomas & Klein, 2006) as well as for comparisons of the mean number of endorsements (Rasinski et al., 1994;Smyth et al., 2006). This finding has been replicated across various behavioral topics, languages, countries of residence (Thomas & Klein, 2006) and opinion-based items (Smyth et al., 2006). ...
... Hypotheses 1 and 2: Question format effects. We found that overall the number of response options selected was higher in the "yes/no" format compared to the "mark all that apply" format which is consistent with the research of Smyth et al. (2006) (as well as Rasinski et al. (1994) and Thomas and Klein (2006)). We found that respondents took more time to answer the "yes/no" questions compared to the "mark all that apply" questions. ...
Article
Full-text available
The work of Smyth, Dillman, Christian, and Stern (2006) and Smyth, Christian, and Dillman (2008) compares “yes/no” questions to “check all that apply” questions. They conclude that the “yes/no” format is preferable as it reflects deeper processing of survey questions. Smyth et al. (2008) found that the “yes/no” format performed similarly across telephone and web modes. In this paper we replicate their research and extend it by including a comparison with face-toface in addition to telephone and web and by using probability samples of the general adult population. A cognitive interviewing follow-up was used to explore the quantitative findings. Our results suggest there are times when the “yes/no” format may not perform similarly across modes and that there may be factors which limit the quality of answers.
... It would be possible to replace CATA questions by a series of forced-choice Yes/No questions as recommended by Rasinski, Mingay, and Bradburn (1994). If one question per attribute is used, the increased time required to complete the longer questionnaire is a key drawback. ...
... Studies of web-based survey questionnaires have shown relationships between the manner in which questions are presented and subsequent responses. Survey researchers have found that if CATA terms are presented in a series of forced-choice Yes/No questions, responses will contain more yes responses than are obtained from the same question posed in a CATA format (Rasinski et al., 1994;Smyth, Dillman, Christian, & Stern, 2006). At the same time, Rasinski et al. (1994) did not observe differences between the study formats with regard to primacy effects, suggesting that sequential Yes/No questions might reduce the impact of satisficing but not primacy. ...
... Survey researchers have found that if CATA terms are presented in a series of forced-choice Yes/No questions, responses will contain more yes responses than are obtained from the same question posed in a CATA format (Rasinski et al., 1994;Smyth, Dillman, Christian, & Stern, 2006). At the same time, Rasinski et al. (1994) did not observe differences between the study formats with regard to primacy effects, suggesting that sequential Yes/No questions might reduce the impact of satisficing but not primacy. However, in case of a long attribute list, respondents might find the Yes/No format tedious. ...
Article
Full-text available
A check-all-that-apply (CATA, often also known as choose-all-that-apply) question is a question format that has been used in recent years to obtain rapid product profiles from consumers. Consumers are presented with a list of attributes and asked to indicate which words or phrases appropriately describe their experience with the sample being evaluated. The terms might include sensory attributes, as well as hedonic responses, emotional responses, purchase intentions, potential applications, product positioning, or other terms that the consumer might associate with the sample.
... A check-all-that-apply format requires the respondent to check or mark the items of interest, with the implicit assumption that the checked items are a 'yes' and the non-checked items are a 'no'. Check-all-that-apply is also called 'mark all that apply' (Rasinski et al. 1994;Nicolaas et al. 2011), 'multiple response format' (Thomas & Klein 2006) or, simply, 'check all' (Smyth et al. 2006;Dykema et al. 2011). An alternative method of asking the same question is to have the respondent explicitly provide a 'yes' or 'no' answer (or similarly dichotomous options: applies to me, does not apply to me; describes me, does not describe me; etc.) for each item in the list. ...
... They found that items presented later in the list are endorsed less, but there was no significant response format interaction. These results echo the paper-and-pencil experiment by Rasinski et al. (1994), where the order of the items was reversed for half the respondents. The authors found a significant order effect (items presented early were selected more often) but no interaction with the format of the question. ...
... In addition to reliability, researchers should also assess measurement validity in evaluating the quality of a response format (Rasinski et al. 1994;Smyth et al. 2008;Thomas & Klein 2006). In their camping study, Dyck and Moore (2008) obtained validation data for the camping shelter types via parking operators who observed and recorded the type of camping shelter each camping party used (e.g. ...
Article
Full-text available
When writing questions with dichotomous response options, those administering surveys on the web or on paper can choose from a variety of formats, including a check-all-that-apply or a forced-choice format (e.g. yes–no) in self-administered questionnaires. These two formats have been compared and evaluated in many experimental studies. In this paper, we conduct a systematic review and a few meta-analyses of different aspects of the available research that compares these two formats. We find that endorsement levels increase by a factor of 1.42 when questions are posed in a forced-choice rather than check-all format. However, when comparing across a battery of questions, the rank order of endorsement rates remains the same for both formats. While most authors hypothesise that respondents endorse more alternatives presented in a forced-choice (versus check-all-that-apply) format because they process that format at a deeper cognitive level, we introduce the acquiescence bias hypothesis as an alternative and complementary explanation. Further research is required to identify which format elicits answers closer to the ‘true level’ of endorsement, since the few validation studies have proved inconclusive. © M. Callegaro, M. H. Murakami, Z. Tepman, and V. Henderson, 2015
... Using short lists of terms can encourage consumers to use them all, decreasing their ability to discriminate among samples. On the other hand, large lists of terms can encourage consumers to use satisfi cing strategies, making them choose the fi rst alternatives from the list, without thinking carefully about the product's sensory characteristics (Krosnick and Alwin, 1987;Rasinski et al ., 1994). ...
... Hence, the order in which the terms are presented has been consistently reported to bias responses to CATA questions in marketing and survey research (Rasinski et al .,1994;Smyth et al ., 2006) and also for the application of CATA questions for sensory characterization of food products (Ares and Jaeger, 2013;Castura, 2009;Lee et al ., 2013). ...
... When used in marketing and survey research, one of the alternatives that have been reported to discourage satisfi cing response strategies and increase consumer attention is to ask them to answer yes/no to each one of the attributes included in the CATA question (Rasinski et al .,1994;Smyth et al ., 2006), as shown in Fig. 11.6. ...
Article
Check-all-that-apply (CATA) questions are versatile multiple choice questions which are being increasingly used for product sensory characterization with consumers. The methodology has been reported to be a simple and reliable approach for sensory product characterization of a wide range of products, providing similar results to descriptive analysis with trained assessors. However, several methodological issues related to how CATA questions are implemented can affect the results. This chapter presents the method and discusses recommendations for implementation and data analysis.
... When using the forced-choice Yes/No format, consumers must answer 'Yes' or 'No' to all attributes, thus reducing the satisficing bias. Satisfaction bias is the subject's bias in not selecting all the attributes applicable to the product because they are satisfied with the answer after selecting some answers (Krosnick, 1991(Krosnick, , 1999Rasinski, Mingay, & Bradburn, 1994). Second, this utilizes a two-step rating procedure in order to improve product discrimination and stabilize consumers' evaluative criteria during the testing. ...
... The CATA question has long been used in marketing and consumer research (Rasinski et al., 1994), and for the last 15 years, it has been popular for consumer perception studies of sensory product characterization due to its simplicity of test procedure (Antúnez, Vidal, de Saldamando, Giménez, & Ares, 2017;Ares, Barreiro, Deliza, Giménez, & Gambaro, 2010;Dooley et al., 2010;Parente, Manzoni, & Ares, 2011;Adams et al., 2007). But, its simplicity can also induce low product discrimination because of a lack of response options Kim et al., 2017a;Reinbach, Giacalone, Ribeiro, Bredie, & Frøst, 2014;Varela & Ares, 2012) and above-mentioned response biases such as satisficing and acquiescence bias. ...
Article
For successful product development and marketing in the food/beverage industry, it is vital to study the product attributes perceived by consumers effectively and efficiently. Recently, a two-step rating-based ‘double-faced applicability’ (DFA) test was developed to study consumer perception and evaluation of product attributes in a quantitative way using d-prime affect magnitude (d'A) based on signal detection theory. This study assessed the product discrimination and characterization performance and interpretationability of d'A output measures of the DFA approach in comparison with the check-all-that-apply (CATA) question. An independent samples design (each N=108) was used to evaluate eighteen ready-to-drink (RTD) latte products with eighteen attributes. A cluster analysis was also performed, and based on this the relationships of the two output statistics - d'A and total citation frequency were examined. Product attributes were generated based on literature reviews on coffee sensory and emotion studies and sensory/consumer research methodology to encompass consumer perception generally applicable to beverage, and for each attribute, a pair of ‘double-faced’ descriptors in three categories – ‘general weak-strong’, ‘sensory’, and ‘sentimental consequence’ were used for the DFA test. The d'A of the DFA test provided similar product characterization to citation frequency of the CATA question, yet provided better product discrimination and reflected the consumer affective perception for each product independently in a better interpretable way, in valence. It showed a sigmoid relationship with the citation frequency due to its nature of reflecting valence. Overall, the results contribute support for the usefulness of the DFA approach for consumer product testing and suggest the potential of developing a generally applicable DFA test format for studying integrated attribute perception.
... The Yes/No forced-choice question format (vs. check-all-that-apply) encourages deeper thinking/attention to the question format (Rasinski, Mingay, & Bradburn, 1994;Smyth, Dillman, Christian, & Stern, 2006) and instructions given to participants could possibly be further used to encourage deliberation. ...
... These data are consistent with the findings on sensory CATA data and its variants, as reported above, and suggest that for emotion tasks, rating versus CATA scaling may not be equal in terms of the potential to bias hedonic responses. There may be a number of reasons for this, but it may well be due to the characteristics of the CATA question format which does not require deep cognitive processing (Rasinski et al. 1994;Strack, 1992;Sudman & Bradburn, 1982). ...
Chapter
Within academia and industry, product-focused emotion research has been attracting considerable attention for more than a decade. In this chapter, focus is directed to the use of emotion word questionnaires for product emotion research with an emphasis on methodological issues. By creating awareness of important empirical considerations in product emotion research, we seek to provide guidance for existing/new investigators in this area of research. The chapter has been updated and a new section added regarding the single-response circumplex-inspired emotion questionnaire.
... In addition, some respondents did not even look at latter response alternatives indicating that at least in some cases they are completely ignored (Galesic, Tourangeau, Couper, & Conrad, 2008). In either case, response alternatives near the top of a list of response alternatives are more likely to be selected than those closer to the bottom which has been found for paper-based surveys (Krosnick & Alwin, 1987;Rasinski, Mingay, & Bradburn, 1994) as well as for web-based surveys (Galesic et al., 2008;Smyth, Christian, & Dillman, 2008;Thomas & Klein, 2006). Moreover, this kind of premature termination of the question-answer processing in check-all-that-apply questions may result in the selection of fewer response alternatives than actually apply to the respondents. ...
... Besides, it seems reasonable that the cognitive costs of optimizing in check-all-that-apply questions increase with the length of the list of response alternatives. Thus, the respondents' tendency to satisfice in a check-all-that-apply question is even more likely when respondents are confronted with longer lists of response alternatives (Ares et al., 2013;Galesic et al., 2008;Krosnick & Alwin, 1987;Rasinski et al., 1994). ...
Article
Check-all-that-apply questions are one of the most commonly used question formats in self-administered surveys. They are especially valuable because they allow respondents to select several responses from a list of alternatives that they consider applicable. In this study, we assessed the effectiveness of different types of instructions requesting a specific number of responses to a check-all-that-apply question in a web survey. We compared “static” instructions that are always visible together with the question stem, “dynamic” instructions that instantly appear once respondents start answering the question, and “combined” instructions taking advantage of both static and dynamic instructions. Findings showed that in view of respondent compliance with the instruction, the combination of a static and dynamic instruction is most effective. However, findings also revealed that the specific number of responses requested in the instruction has to be taken into account as a decisive factor influencing the response selection process and ultimately data quality.
... Having to provide a "yes" or "no" answer for each specific question, rather than selecting all relevant conditions that apply from a list, could result in higher estimates in the comparison surveys relative to NSDUH. Survey experiments using self-administered paper-and-pencil or Web questionnaires have indicated that respondents may not choose all applicable responses when presented with a list and instructed to choose all that apply (Rasinsky, Mingay, & Bradburn, 1994;Smith, Dillman, Christian, & Stern, 2006). Furthermore, respondents who choose applicable response options closer to the top of a list may consider their task in answering the question to be finished, (Galesic, Tourangeau, Couper, & Conrad, 2008;Rasinsky et al., 1994). ...
... Survey experiments using self-administered paper-and-pencil or Web questionnaires have indicated that respondents may not choose all applicable responses when presented with a list and instructed to choose all that apply (Rasinsky, Mingay, & Bradburn, 1994;Smith, Dillman, Christian, & Stern, 2006). Furthermore, respondents who choose applicable response options closer to the top of a list may consider their task in answering the question to be finished, (Galesic, Tourangeau, Couper, & Conrad, 2008;Rasinsky et al., 1994). Thus, if instructing respondents to choose all applicable health conditions from a list yields lower estimates in NSDUH than in surveys that use separate yes/no questions, differences between NSDUH and other surveys might be expected to be more pronounced for health conditions in NSDUH that are closer to the bottom of the list. ...
Article
Full-text available
methodological differences that could cause differences in estimates are discussed, including type and mode of data collection; weighting and representativeness of the sample; question placement, wording, and format; and use of proxy reporting for adolescents. There were no differences between the lifetime estimate of diabetes among adults from NSDUH (7.7 percent) and the estimates from NHIS, NHANES, BRFSS, and MEPS. The lifetime estimate of asthma among adults from NSDUH (10.7 percent) was similar to the estimate from NHIS (11.0 percent); estimates from other sources ranged from 9.6 percent to 14.2 percent. The lifetime estimates of stroke and high blood pressure among adults from NSDUH were both lower than estimates from NHIS, NHANES, and MEPS, and there was considerable variation between surveys in the rate of lifetime heart disease. Estimates of past year inpatient hospitalization among adults did not differ significantly between NSDUH and NHANES, but NSDUH was significantly higher than the estimates derived from NHIS and MEPS. For both adults and adolescents, the NSDUH estimates of receiving treatment in an ER in the past year were higher than estimates from other surveys. Demographic differences in the prevalence of chronic health conditions and health care utilization were similar across multiple surveys. Given all of the methodological differences among these data sources, the similarities among estimates are noteworthy.
... That is a response style that is characterized by a lack of care ful consideration to the question. The Yes/No forced-choice question format (vs check all that apply) encourages deeper thinking/attention to the question format (Rasinski, Mingay, & Bradburn, 1994;Smyth, Dillman, Christian, & Stern, 2006) and instruc tions given to participants could possibly be further used to encourage deliberation. In a study using images of kiwifruit (Jaeger, unpublished data), two instructions/ data elicitation approaches were compared using a between-subjects design with 59 consumers in each group. ...
... These data are consistent with the findings on sensory CATA data and its variants, as reported above, and suggest that for emotion tasks, rating versus CATA scaling may not be equal in terms of the potential to bias hedonic responses. There may be a number of reasons for this, but it may well be due to the characteristics of the CATA question format which does not require deep cognitive processing (Rasinski et al. 1994;Strack, 1992;Sudman & Bradburn, 1982). ...
Chapter
Full-text available
Within academia and industry product-focused emotion research is currently attracting considerable attention. In this chapter, focus is directed to the use of questionnaires for product emotion research and methodological issues are discussed. By creating awareness of important empirical considerations in product emotion research, we seek to provide guidance for existing/new investigators in this area of research.
... Sudman and Bradburn (1982) Mitofsky and Edelman (1993) mention that the 'yes/no' format produced a higher endorsement of behaviours than 'mark all that apply' in their 1993 AAPOR conference paper, but this was never published. Rasinski, Mingay, and Bradburn (1994) conducted the first published experimental test on the topic. They compared 'yes/no' versus 'mark all that apply' formats with a paper questionnaire given to high school seniors as part of a field test for round three of the National Education ...
... Using Washington State University students as participants, Smyth, Dillman, Christian and Stern (2006) expanded on the work of Rasinski et al (1994) by using web as well as paper self-completion and attitudes as well as behaviours. Recruiting participants from the same population, Smyth, Christian and Dillman (2008) extended the research further to include telephone surveys. ...
... The bulk of the research comparing 'forced-choice' and 'check all that apply' questions shows that respondents are more likely to choose from the options presented at the beginning of a list, irrespective of the survey mode and medium (paper/electronic). Early theories (Krosnick & Alwin, 1987) attributing the phenomenon to such questions requiring deeper cognitive processing were later supplemented with interpretations alluding to the primacy effect (Rasinski et al., 1994) and to acquiescence response bias (Smyth et al., 2006), i.e., the tendency to agree with a statement whatever it says (Holbrook et al., 2003). This paper develops a simple and appropriate data coding and analysis proposal with great potential for achieving maximum data output from multiple-response questions. ...
Article
Full-text available
Multiple-response questions are a common feature of survey-based research, as its advantages of information collection are well known. Statistical analysis of the responses, however, tends to be either one or two-dimensional, with the restrictions that this entails. This paper presents a multidimensional analysis protocol that provides the researcher with tools to identify more and better profiles about ‘who says what’. The strategy begins by coding the response options as a set of metric binary variables. The ideal methodological duo for the exploration of the resulting data is Principal Component Analysis coupled with an Ascending Hierarchical Cluster Analysis, incorporating, in addition, supplementary variables. When applied to the analysis of three different multiple-response questions included in a Spanish National Survey, this proposal provides evidence not only of the interpretation potential of the coding/analysis protocol but also of the limitations of some multiple-response question formats.
... One potential problem that has been identified with the forced-choice format is the phenomenon of respondents marking answers only in the affirmative column and leaving the negative column blank, essentially treating the item as a check-all question. When this happens, it is unclear if the missing items were overlooked (i.e., truly missing) or intended to be "not affirmative" responses (note that this confusion about missing items is always the case in the check-all format - Rasinski, Mingay, and Bradburn 1994). For example, the 2016 NSCH methodology report indicates that the items with the highest missing data rates were forcedchoice items about reasons needed health care was not received, sources of health insurance, and reasons for not having health insurance, and that the primary reason for the missing data was respondents only using the affirmative response option (i.e., treated the item as a check-all) (U.S. Census Bureau 2018b). ...
Article
Full-text available
Telephone surveys have been a ubiquitous method of collecting survey data, but the environment for telephone surveys is changing. Many surveys are transitioning from telephone to self-administration or combinations of modes for both recruitment and survey administration. Survey organizations are conducting these transitions from telephone to mixed modes with only limited guidance from existing empirical literature and best practices. This article summarizes findings by an AAPOR Task Force on how these transitions have occurred for surveys and research organizations in general. We find that transitions from a telephone to a self-administered or mixed-mode survey are motivated by a desire to control costs, to maintain or improve data quality, or both. The most common mode to recruit respondents when transitioning is mail, but recent mixed-mode studies use only web or mail and web together as survey administration modes. Although early studies found that telephone response rates met or exceeded response rates to the self-administered or mixed modes, after about 2013, response rates to the self-administered or mixed modes tended to exceed those for the telephone mode, largely because of a decline in the telephone mode response rates. Transitioning offers opportunities related to improved frame coverage and geographic targeting, delivery of incentives, visual design of an instrument, and cost savings, but challenges exist related to selecting a respondent within a household, length of a questionnaire, differences across modes in use of computerization to facilitate skip patterns and other questionnaire design features, and lack of an interviewer for respondent motivation and clarification. Other challenges related to surveying youth, conducting surveys in multiple languages, collecting nonsurvey data such as biomeasures or consent to link to administrative data, and estimation with multiple modes are also prominent.
... One of the alternative methods that have gained popularity is check all that apply (CATA), originally used in marketing (Rasinski et al., 2002) and subsequently proposed as an alternative method in the food industry to gather information about consumers' perceptions (Adams et al., 2007). CATA is a rapid sensory-profiling technique that uses a questionnaire consisting of a list of attributes (in the form of words or phrases), from which trained or untrained panellists can select all the descriptors they consider appropriate to characterise each sample (Valentin et al., 2012). ...
Article
Full-text available
There are many ways of manipulating the components of grape berries and one of these is the implementation of a specific trellising system. This will affect light exposure in the bunch zone, air flow through the canopy, crop load, etc., and consequently the primary metabolites that contribute to the production of secondary compounds in wine. The aim of the present study was to characterise the grape musts and wines of Chenin blanc made from grapes of different trellising systems, using chemical fingerprinting of the samples and the check-all-that-apply method, combined with a quality scoring test. The results indicate that, while the wines could not be separated according to treatment from an aroma point of view, the grapes produced by different trellis systems resulted in wines characterised by differences in taste and mouthfeel. The present study shows that trellising systems can influence amino acids, yeast assimilable nitrogen, phenolic content and aroma compounds, as well as sensory characteristics. In this case study, wine quality was not affected by the trellising systems, with one exception. Fingerprinting using high-resolution mass spectrometry proved to be a successful tool to separate the samples according to the systems.
... This feature, called Tick diary in 2018, was sparingly used by Tick App users , but we anticipate that these daily assessments can provide more accurate estimates of prevention method use in relation to outdoor activities when a more robust sample is acquired. Lastly, although we tried to avoid "check all that apply" question structures where participants are more likely to select the first options (Rasinski et al., 1994), the personal preventative behavior question was structured this way. Over-reporting of the first option (i.e. ...
Article
The dynamics of zoonotic vector-borne diseases are determined by a complex set of parameters including human behavior that may vary with socio-ecological contexts. Lyme disease is the most common vector-borne disease in the United States. The Northeast and upper Midwest are the regions most affected - two areas with differing levels of urbanization and differing sociocultural settings. The probability of being infected with Lyme disease is related to the risk of encounters with Ixodes scapularis ticks infected with Borrelia burgdorferi sensu lato, which reflects both the environmental tick hazard and human behaviors. Herein, we compare behavioral and peridomestic risk factors perceived to influence the risk for human-tick encounters between two high-incidence states in the Northeast (New York and New Jersey) and one high-incidence state in the Midwest (Wisconsin). We used a smartphone application, The Tick App, as a novel survey tool, during spring and summer of 2018. Adaptive human behavior was identified in the relationship between outdoor activities and the use of methods to prevent tick bites. More frequent recreational outdoor activities and gardening (a peridomestic activity) were associated with a 1.4–2.3 times increased likelihood of using personal protective measures to prevent tick bites, when accounting for demographics and previous Lyme diagnosis. Most outdoor activities were more frequently reported by participants from the Midwest (n = 697), representing an older demographic, than the Northeast (n = 396). Participants from the Northeast were less likely to report use of personal protective measures to prevent tick bites, but a larger proportion of participants from the Northeast reported application of environmental pesticides targeting ticks or mosquitoes or other insects on their property (34 % of 279 versus 22 % of 616 participants) and interventions to reduce the presence of peridomestic deer compared to participants from the Midwest (e.g. 20 % of 278 versus 7% of 615 participants reported having a deer proof fence). Participants from the Midwest were more likely to kill rodents on their property (28 % versus 13 %). These differences illustrate the need for further assessment of personal behavior and tick exposure in these two Lyme disease-endemic regions to aid in targeted public health messaging to reduce tick-borne diseases.
... The sensory descriptors were selected based on the description of variables for tempe in previous studies (Jeleń, Majcher, Ginja, & Kuligowski, 2013;Stan, 2013) and through the discussion with ten respondents after the pilot study. For each descriptor, a yes/no box approach was applied (Ennis & Ennis, 2013;Reinbach, Giacalone, Ribeiro, Bredie, & Frøst, 2014;Jaeger et al., 2014;Krosnick, 1991;Rasinski, Mingay, & Bradburn, 1994) to enhance the likelihood that the consumer actually read through the whole list, reducing the behaviour known as satisficing. ...
Article
Nine different tempe (five traditional and four modernised versions of tempe) were tested by 165 millennial consumers (63.6% women) in blind and informed sessions. The effects of information on hedonic response, sensory perception, collative properties and the product-elicited emotions were evaluated. The provided information related to the raw materials (bean type), origin (local or imported) and the production methods (traditional inoculum usar, with starter culture inoculum ragi and use of stainless steel factory processing equipment hygienis). The millennial consumers’ hedonic response and other parts of their perception were highly affected by the product information. The five traditional tempe were more liked when the product information was provided. In addition, significantly differences in collative properties were observed when product information is provided. Particularly the collative properties Authentic and Traditional. The elicited emotion that were most affected were Proudness. Surprisingly, a number of sensory properties were also affected significantly by the provided information. However, the interaction is not systematic neither for bean nor production type. Segmentation of respondents based on their level of Food Neophobia and Attitudes Towards Traditional Foods (ATTF), showed profound differences. ATTF segmentation in particular, as it affected both hedonic response, elicited emotions, as well as sensory and collative properties. Pro-traditional consumers reacted more positively to the information than other segments. The results demonstrated the powerful effect of transparency regarding the products’ provenience and product methods. The results show the importance of the millennial consumers’ education and information to preserve or maintain the diversity of food we eat.
... When using CATA questions, participants are presented with a list of terms that can be used to describe a sample, and their task is to select those that apply . The question format, which is easy to implement, has been extensively used in marketing research (Rasinski, Mingay, & Bradburn, 1994), and Adams, Williams, Lancaster, and Foley (2007) brought awareness of the potential application of CATA questions for sensory product characterisation with consumers. CATA questions were soon adopted, and uptake is likely to have been further boosted by extensive methodological research to identify pros/cons and develop guidelines for implementation Ares & Varela, 2018). ...
Article
The use of check-all-that-apply (CATA) questions in product-focused consumer research on foods and beverages is now common, and the method is known to provide valid sensory product characterisations. Extensive methodological research has been conducted and has supported uptake, but understanding of how consumers complete CATA questions is incomplete, particularly with regard to their decision to select or not a term to describe the sensory properties of products. The present research was situated within this gap, and using open-ended questions participants (n = 636) were asked to describe how they perceived a pair of samples with regard to an attribute and link this to CATA term selection. The results, obtained for taste (‘sweet’ and ‘sour/acidic’) and flavour (‘cinnamon’ and ‘smoky’) confirmed consumers’ ability to accurately perform sensory characterisation tasks. In particular, it was found that: i) the great majority of the consumers accurately used the CATA terms for describing the sensory characteristics they perceived in a sample, ii) when a term was not selected for describing samples, the majority of the consumers indicated that the corresponding sensory attribute was not perceived, iii) when a term was selected for describing only one of the samples in a pair, consumers reported to have perceived a difference in attribute intensity between the samples. Thus, CATA questions remain a desirable option for sensory product characterisation tasks with consumers, but should be selected with thought as they may not always be able to achieve desired sample discrimination due to the binary nature of the responses.
... Some indirect evidence supporting the "does not apply" method can also be found in studies comparing what happens when a respondent is presented with a list of items and is requested to either (a) "select all that apply" or (b) answer either "no" or "yes" to each item on the list. Such studies find that more items are endorsed with the latter procedure, which suggests that it forces people to pay greater attention to each item (Rasinski et al. 1994;Smyth et al. 2006;Thomas and Klein, 2006). ...
... These different outcomes can be attributed to the level of cognitive processing required for each different test method. Namely, it can be thought that the CATA questions and forcedchoice Yes/No questions do not encourage consumers to engage in deep processing due to their simpler and more intuitive task, whereas RATA questions increase consumers' engagement and attention to the task (Sudman & Bradburn, 1992;Rasinski, Mingay & Bradburn, 1994;Smyth, Dillman, Christian, & Stern, 2006;. The DFA test in this study used the forced-choice Yes/No questions, with the list of the 'double-faced' attributes. ...
Article
In the fast moving consumer goods industry, measuring consumer acceptance toward products is crucial for product development and marketing. Consumers are generally considered hedonists and, thus momentary hedonic scores are assumed to represent consumer acceptance. Yet for many product types, such as household care products, consumers might be considered utilitarian and their usage experience with the product might be equally important for consumer acceptance. To quantify consumer holistic product usage experience, a two-step signal detection rating-based satisfaction measure was used such that an independent signal detection theory index termed d’A (d-prime affect magnitude) could be computed for each product to represent consumer satisfaction with the usage experience and with the product itself. The objective of the present study was to investigate the effect of simultaneous attribute evaluation using the ‘double-faced applicability’ (DFA) test on product discrimination of this satisfaction measure. The conventional 10-point hedonic ratings with and without the DFA test were used as control methods for comparison. Results showed that significant product discriminations were observed only in the group who performed the satisfaction measure with the DFA test. Also, significant discriminations in quality attributes of the DFA test questionnaire were more frequently observed in the group that performed the satisfaction test than in those who performed the hedonic test. These results indicate that compared to using hedonic scores, the satisfaction test with the DFA has the potential to improve research on the quality predictors of household products.
... Check-all-that-apply is sometimes equated to a multiple choice question however; one can select as many choices as possible (Valera and Ares, 2012). The CATA method has been widely used in marketing research and has gained popularity because it reduces participant response burden (Rasinski et al., 1994;Smyth et al., 2006;Adams et al., 2007;Lancaster and Foley, 2007;Ares et al., 2010). Most animal production studies (Mahanjana and Cronjé, 2000;Sebei et al., 2004;Rumosa-Gwaze, 2009;Mapiliyao et al., 2012;Slayi et al., 2014) have used open ended and forced-choice questionnaires which are more "cumbersome" compared to check-all-that-apply questions. ...
Article
The check-all-that-apply (CATA) technique was used to determine lamb management practices and some constraints faced by resource-limited communal sheep farmers in the Sour veld and Sweet veld regions. A total of 107 respondents were involved in the study. Data generated from CATA was analysed using the XLSTAT 2016 software. Results showed that the majority of the households were headed by men (75.70%) and consisted of the elderly (˃50 years, 79.44%) who had attended primary school (67.29%) and were not employed (82.24%). Wool production was ranked the most important reason for keeping sheep in the two regions. The CATA revealed that the management systems between the two communities differed significantly (P ˂0.05) in terms of assisting sheep at lambing, attending to twin births, castration, monitoring lambing and dipping of mature lambs. Farmers’ age was associated with the type of management performed (P ˂0.05). None of the farmers in the regions performed navel dipping. Constraints faced in the two regions also differed significantly (P ˂0.05). The extent of theft, predation, cold, heat, foot rot, starvation, drought and lack of knowledge differed significantly (P ˂0.05) in the two regions. The CATA identified vaccination and dipping as the major missing practices in both regions. In conclusion, CATA was able to gather information on management practices and constraints faced by farmers based on their perceptions. CATA was also able to measure how farmers were managing sheep compared to what is expected in an “ideal” farm; at the same time revealing farmers’ preferences for certain management practices; making it an interesting method to use in developing animal improvement programmes.
... Istraživač treba prilikom dizajniranja upitnika dobro razmisliti o tome koju formu odabrati za određeno pitanje i na koji način definirati svojstva odabrane forme. Objavljena su razna istraživanja o utjecaju osobina i vrsti ponuđenih formi na odgovore ispitanika kao, na primjer, o utjecaju osobina polja za unos teksta na dobivene odgovore (Couper et al., 2001); o utjecaju vrste forme (na primjer, o izboru između polja za unos teksta i izbornih gumba (Couper et al., 2001) ili o izboru između izbornih gumba i padajućih izbornika Heerwegh, Loosveldt, 2002)); o osobinama okvira za izbor (Krosnick, 1991;Rasinski et al., 1994;Smyth et al., 2006). Neki autori bavili su se i vizualnim značajkama ponuđenih formi za odgovore (razmak između ponuđenih odgovora, pozicioniranje ponuđenih odgovora, redoslijed ponuđenih odgovora i slično) koje također mogu imati utjecaj na rezultate ispitivanja (Tourangeau et al., 2004). ...
Article
Web-ispitivanja kod kojih se podaci od ispitanika prikupljaju web-upitnikom danas postaju sve popularnija. No sve više ljudi za pristup internetu koristi zaslone raznih formata, pametne telefone, tablete itd., pa su posebno važne mogućnosti raznih prikaza web-upitnika, odnosno fleksibilnost u njihovu dizajniranju. S druge strane, dizajn web-upitnika može utjecati na podatke prikupljene tim ispitivanjem. Dosadašnja istraživanja o utjecaju vizualne orijentacije skale za odgovore i broja stranica web-upitnika na rezultate daju dosta nejasne, a ponekad i oprečne nalaze. Osim toga, postoji premali broj istraživanja koja se bave ovakvim pitanjima. U članku je opisano istraživanje koje se bavi utjecajem navedenih osobina web-upitnika na rezultate ispitivanja na uzorku od 207 studentica i studenata Filozofskog fakulteta u Rijeci. Cilj istraživanja bio je provjeriti utječu li vizualna orijentacija skale za odgovaranje i broj stranica web-upitnika na rezultate ispitivanja. U kontekstu ovog istraživanja pod rezultatima ispitivanja misli se posebno na vrijeme potrebno za ispunjavanje web-upitnika i posebno na sadržaj odgovora. Primijenjen je web-upitnik sa skalom stavova o računalima i sa skalom stavova o internetu. Korištene su tri inačice web-upitnika. U prvoj je korišten straničeni dizajn, u drugoj vertikalni klizaći, a u trećoj horizontalni klizaći dizajn. Kruskal-Wallisovom analizom ustanovljeno je da nema značajnih razlika u brzini rješavanja web-upitnika među skupinama ispitanika sa straničenim dizajnom, vertikalnim klizaćim i horizontalnim klizaćim dizajnom. Također, analizom varijance nije ustanovljena značajna razlika u sadržaju odgovora među tim skupinama. Možemo zaključiti da dizajn web-upitnika u smislu broja stranica i vizualne orijentacije skale za odgovaranje nije utjecao na rezultate ispitivanja.
... In RQ3a, CATA questions (EC3) were compared to forced yes/no questions (EC4). The results (Table 3) showed a large increase in frequency of emoji use with forced yes/no questions, which fitted expectations since this question format encourages more attention (Rasinski, Mingay, & Bradburn, 1994;Smyth, Dillman, Christian, & Stern, 2006). However, there was no accompanying increase in sample discrimination relative to CATA questions. ...
Article
Product insights beyond hedonic responses are increasingly sought and include emotional associations. Various word-based questionnaires for direct measurement exist and an emoji variant was recently proposed. Herein, emotion words are replaced with emoji conveying a range of emotions. Further assessment of emoji questionnaires is needed to establish their relevance in food-related consumer research. Methodological research contributes hereto and in the present research the effects of question wording and response format are considered. Specifically, a web study was conducted with Chinese consumers (n = 750) using four seafood names as stimuli (mussels, lobster, squid and abalone). Emotional associations were elicited using 33 facial emoji. Explicit reference to “how would you feel?” in the question wording changed product emoji profiles minimally. Consumers selected only a few emoji per stimulus when using CATA (check-all-that-apply) questions, and layout of the CATA question had only a small impact on responses. A comparison of CATA questions with forced yes/no questions and RATA (rate-all-that-apply) questions revealed an increase in frequency of emoji use for yes/no questions, but not a corresponding improvement in sample discrimination. For the stimuli in this research, which elicited similar emotional associations, RATA was probably the best methodological choice, with 8.5 emoji being used per stimulus, on average, and increased sample discrimination relative to CATA (12% vs. 6–8%). The research provided additional support for the potential of emoji surveys as a method for measurement of emotional associations to foods and beverages and began contributing to development of guidelines for implementation.
... strongly focus their attention on each of the sensory attributes included on the list of terms (Sudman & Bradburn, 1992;Strack, 1992;Rasinski, Mingay, & Bradburn, 1994), which could potentially minimize their influence on overall liking scores. However, RATA questions do involve some form of intensity scaling, suggesting that scale format is not reason for the observed differences, and this is further supported by Popper et al. (2004) who reported that intensity scales did not have a biasing effect. ...
Article
In line with research in non-Western countries becoming main-stream, the need to validate existing research methods with consumers from these populations increase. The present research contributes hereto by quasi-replicating with Korean and Chinese consumers previous research concerning the risk of hedonic product responses being biased by co-elicitation of CATA/RATA questions for sensory product characterisation. Using consumers in several Western countries it was previously reported that bias could occur, but was unlikely to. Eleven studies involving ∼1000 East Asian consumers confirmed this conclusion. The studies were conducted with diversified populations and across multiple product categories. Across 7 studies, there were no instances where CATA co-elicitation was found to bias hedonic scores. However, in one of four studies where RATA responses were co-elicited bias did occur, and hedonic scores were, on average, lower when RATA responses were co-elicited. It is recommended that the research be replicated with consumers residing in their home countries and extended to other East and South-East Asian counties.
... Conversely, with CATA questions consumers only select those attributes that they consider as appropriate for describing the sample. In this sense, several studies have shown that consumers usually engage in satisficing response strategies when answering CATA questions, which encourages not to spend too much effort attending to all the terms that apply to describe a focal sample (Rasinski, Mingay, & Bradburn, 1994;Smyth, Dillman, Melani Christian, & Stern, 2006). Besides, in the case of the CATA questions with hedonic/intensity connotation terms, the fact that only the extreme anchors of the JAR scales were used as CATA terms might have encouraged participants not to select these terms when they perceived only slight deviations from their ideal. ...
Article
Methodologies that identify ways products differ from consumers’ ideal are commonly used to guide innovation. In this research the use of CATA questions for this purpose was compared to JAR scales, which are well established for use in product optimization efforts. Two CATA variants were considered: (i) CATA questions including terms with hedonic-intensity connotations (e.g., not enough sweet, much too sweet), and (ii) CATA questions pertaining to both the tasted and the ideal products. In six consumer studies (n = 939), spanning multiple product categories and consumer populations (Uruguay, New Zealand and USA), it was found that CATA questions and JAR scales provided similar insights regarding the most relevant deviations from ideal. However, several differences were also identified. In particular, CATA questions tended to identify fewer deviations than did JAR scales, especially when terms with hedonic-intensity connotations were used and when differences between samples were small. This difference is likely linked to facets of the two methodologies: only applicable terms are selected when using CATA questions, whereas responses must be provided for each JAR scale included in a study. Besides, the fact that only the extreme anchors of the JAR scale were included in the CATA question could have encouraged consumers not to indicate deviations from the ideal. Penalty analysis, performed using Partial-Least Squares (PLS) regression identified several significant deviations from the ideal. While the two methodologies established the same main differences, JAR scales identified more significant deviations from the ideal than CATA questions. Although results confirmed the potential for the use of CATA questions in product optimization research, careful consideration of purpose of the research and attention to terms included in the CATA questions is recommended.
... Similar mechanisms may link interviewers' HOW INTERVIEWERS ANSWER SURVEY QUESTIONS 6 multiple response, differentiation, and interview speed to their respondents' multiple response, differentiation, and interview speed, respectively. Interviewers who consider all the response options to "all that apply" questions are more likely to select more than one answer in the self-administered questionnaire (Rasinski, Mingay, and Bradburn 1994) and may consistently probe for additional answers, resulting in more answers given on average. Interviewers who consider different items in a grid separately, are less likely to straight line and are expected to put more effort into motivating respondents to do the same. ...
Article
Full-text available
Previous research shows that interviewers to some extent fail to expend the effort that is needed to collect high-quality survey data. We extend the idea of interviewer satisficing to a related task, in which the interviewers themselves answer survey questions. We hypothesize that interviewers who self-administer the questionnaire in a careless manner will also not apply themselves fully to the task of administering survey interviews. Based on interviewer and respondent data from the sixth round of the European Social Survey in Belgium, we find support for some of the hypothesized associations between (suboptimal) response characteristics of interviewers in the “task as respondent” and the same (suboptimal) response characteristics recorded for their respondents, specifically with regard to interview speed, multiple response, and item nonresponse to the household income question.
... CATA is increasingly applied in sensory research (Dooley, Lee, & Meullenet, 2010;Lado, Vicente, Manzzioni, & Ares, 2010;Plaehn, 2012) and is commonly used in marketing research (Rasinski, Mingay, & Bradburn, 1994). In CATA, respondents should select among a series of words or phrases that they consider applicable to the product they are evaluating from a list of predefined multiple choice questions (words or phrases). ...
Article
Understanding the interaction of sensory and extrinsic product attributes in consumer preferences has been identified as one of the key pillars for raising the likelihood of food products’ success in the market. Over the course of the last decade there has been increased attention on research emphasizing a combination of these food-choice driving parameters. This paper discusses progress made in the field focusing on three groups of methods: (i) conjoint hedonic methods (ii) “classic” hedonic testing and (iii) alternative descriptive approaches. For each method a description of the methodology in question, its objectives, advantages, drawbacks and applications are examined. Industrial challenges and future research needs are discussed.
... Patterns of primacy bias have been reported when CATA questions are used in marketing/survey research and for sensory product characterization. Terms located at the beginning of the list are more frequently selected than those located at the end Castura, 2009;Lee, Findlay, & Meullenet, 2013;Rasinski, Mingay, & Bradburn, 1994). This bias has been reported to affect conclusions regarding similarities and differences among samples . ...
Chapter
Check-all-that-apply (CATA) questions have become the most common tool for product sensory characterization with consumers and the question format is increasingly being used in emotion questionnaires. This is because CATA questions are simple for research participants, deliver reproducible results, and reliably elicit emotional associations to products and other food-related stimuli. This protocol explains how to implement questionnaires with emotion words as the CATA terms and how to analyze the generated data. Drawing on the more extensive literature on CATA questions in sensory product characterizations with consumers, methodological issues related to how CATA question implementation can influence the results are also covered.Key wordsCheck-all-that-apply questionsCATAEmotionsConsumer researchQuestionnaires
Article
Check-All-That-Applies (CATA) is a method commonly used for sensory and consumer tests. Within FrieslandCampina(FC), CATA is used to supplement quantitative descriptive methods for products’ characterization. Typically for quantitative descriptive methods, ANOVA is commonly used for panel performance checks to ensure the good quality of data (ISO 2012). However, data generated from CATA experiments are binary and the assumptions for ANOVA are violated, meaning alternatives should be considered. However, most studies in literature for the analysis of CATA data involve consumers, and we investigate whether the statistical treatment recommended could also be used for panel performance checks. 4 separate datasets on dairy products profiled by the Singapore panel in FC were used. Each dataset had at least 9 panellists with one duplicate. The panel was evaluated on an overall level for the ability to discriminate between products using 3 different approaches; Non-parametric analysis using Cochran’s Q Test, Generalized Linear Mixed Effect Models(GLMEM) using ANOVA, and GLMEM using Logistic Regression. The p-values and the test statistics were compared. Cochran’s Q test provides good results but is sensitive to low elicitation rates. ANOVA is stable in results and interaction effects penalizes p-value. Logistic Regression has convergence issues and provides unstable results.
Chapter
In the last years, research has paid strong attention to pre-service primary teachers’ views of mathematics. Interviews and questionnaires to pre-service teachers during their academic studies are the mainly used tools for collecting data. Qualitative and quantitative approaches may give different insights. In this paper, after a review of the different methods used in the literature to face the topic of pre-service primary teachers’ views of mathematics, we propose a new method. A clustering technique is applied to data collected with multiple-answer questions about pre-service primary teachers’ views of mathematical ability. Obtained clusters are interpreted and compared.
Article
Existing research shows that response options are endorsed at a higher rate when presented in forced-choice format (FC, yes-no grids) than in check-all-that-apply format (CATA). Information Processing Theory explains this contrast with differential effects on the level of cognitive reflection. The study tests this hypothesis with eye movement data collected in a randomized laboratory experiment in which 131 respondents participated in a web survey with four treatment questions. In one condition (CATA) the questions were presented in check-all format, in the other (FC) in forced-choice structure. I find higher levels of affirmative responses and longer completion times in FC compared to CATA in three of the questions. With all four questions, respondents invested more cognitive effort—measured by fixation counts and times—in FC than in CATA when considering the question in total. I find no differences when considering only the list of response options, however. This indicates that the longer fixation times did not result from a more careful evaluation of the response options. Other possible causes and practical implications are discussed.
Article
When writing questions with dichotomous response options, those administering surveys on the web or on paper can choose from a variety of formats, including a check-all-that-apply or a forced-choice format (e.g. yes-no) in self-administered questionnaires. These two formats have been compared and evaluated in many experimental studies. In this paper, we conduct a systematic review and a few meta-analyses of different aspects of the available research that compares these two formats. We find that endorsement levels increase by a factor of 1.42 when questions are posed in a forced-choice rather than check-all format. However, when comparing across a battery of questions, the rank order of endorsement rates remains the same for both formats. While most authors hypothesise that respondents endorse more alternatives presented in a forced-choice (versus check-all-that-apply) format because they process that format at a deeper cognitive level, we introduce the acquiescence bias hypothesis as an alternative and complementary explanation. Further research is required to identify which format elicits answers closer to the ‘true level’ of endorsement, since the few validation studies have proved inconclusive.
Article
Full-text available
The present study investigated the drivers of liking ethnic sauces in a cross-cultural context. Experiments were conducted to understand the acceptance of salad dressings and dipping sauces developed from Korean fermented seasonings among consumers with different ethnic backgrounds, including: South Korean, United Arab Emirates (UAE), and American. The samples of interest included four types of salad dressings made from fermented soybean paste (doenjang) and four types of spicy dipping sauces made from fermented chili pepper paste (gochujang). The salad dressings were preferred by Korean and US consumers. Koreans liked the nutty-flavored salad dressings, whereas UAE and American consumers commonly liked the spicy type. There was a stronger cross-cultural agreement in liking dipping sauces rather than salad dressings. Both Korean and American consumers liked spicy dipping sauces that elicited a sweet taste. UAE consumers tended to prefer the less spicy dipping sauce samples. Consumers in all three countries generally liked spicy dipping sauces more than salad dressings. Cultural differences were observed between the responses depending on the presence and level of spiciness in the two different food types. For product development with ethnic fermented flavors or chili spices, the contextual appropriateness and consumer familiarity with the corresponding flavor should be taken into account.
Chapter
The urgent need to reformulate driven by recent and future regulation requires product manufactures to conform to regulatory requirements that meet sensory quality and consumer expectations while also making a profit. In order to achieve these goals, producers require sensory, consumer-driven methods that can assist them in getting their products to market most efficiently. The major methods discussed in this chapter include ranking, flash profiling, ranking descriptive analysis, free sorting, projective mapping, napping, polarized sensory positioning, the optimized sensory profile method, polarized projective mapping, ranked-scaling, check-all-that-apply, temporal dominance of sensations, the ideal profile method, and just-about-right scales. The application as well as the advantages and disadvantages of each will be discussed.
Article
Many social identities (e.g., race, ethnicity) are measured using mark-all-that-apply (MATA) questions because they allow survey respondents to account for the multiple, nonexclusive ways in which they identify themselves. We test the use of MATA measures of sexual orientation and gender identity and compare them with forced choice (FC), an alternative format using a series of yes-or-no questions. Respondents, including an oversample of lesbian, gay, bisexual, and queer (LGBQ) individuals, participated in a 2 × 3 factorial survey experiment. For the first factor, we hypothesize that respondents randomly assigned to FC will report a higher count of identities than those assigned to MATA. For the second factor, we hypothesize that increased topic salience will help LGBQ respondents in particular to overcome poor question design. Findings suggest that MATA and FC measure comparably when question writing best practices are followed, but topic salience can yield higher data quality when poorly formatted questions are used.
Article
This study examined the inhibiting factors of satisficing response on multiple-answer format (MA) in volunteer panel Web surveys. Data were based on two Web surveys in the Tokyo metropolitan area (one of 2,257 Japanese adults in March 2014, and the other of 519 Japanese adults in October 2017). The results revealed the following: (a) respondents endorsed fewer options and took less time to answer MA than forced choice format (FC); (b) an acquiescence tendency was not found for almost all the items in FC; and (c) satisficing responses were not inhibited by the number or properties of items. Results (a) and (b) suggest that MA may encourage satisficing response strategies. Furthermore, these tendencies can be seen not only in attitudinal (judgment-type) questions, but also behavioral (recall-type) questions. These results indicate the strength of satisficing response behaviors, so the use of FC is recommended in volunteer panel Web surveys. © 2018 Japanese Psychological Association. All rights reserved.
Article
Elite Malaysian athletes (N = 179) from integrated and segregated sports rated the perceived importance of eight psychological strategies for improving performance using two different response format methods, a Likert rating scale and forced-choice. A forced-choice procedure produced better discrimination among the skills than a Likert rating scale procedure. We also found that the ratings of importance differed as a function of sport type and gender. Specifically, athletes in integrated sports placed more importance on setting team goals and clarifying roles/responsibilities compared to athletes in segregated sports. At the same time, participants in segregated sports viewed setting personal goals, psych-up strategies, and imagery as more important for performance than those in integrated sports. Significant interaction effects indicated that, within segregated sports, females rated positive self-talk higher than males, but communication skills were rated higher by males than by females.
Article
Previous research has shown that check-all-that-apply (CATA) and forced-choice (FC) question formats do not produce comparable results. The cognitive processes underlying respondents’ answers to both types of formats still require clarification. This study contributes to filling this gap by using eye-tracking data. Both formats are compared by analyzing attention processes and the cognitive effort respondents spend while answering one factual and one opinion question, respectively. No difference in cognitive effort spent on the factual question was found, whereas for the opinion question, respondents invested more cognitive effort in the FC than in the CATA condition. The findings indicate that higher endorsement in FC questions cannot only be explained by question format. Other possible causes are discussed.
Article
The purpose of this study is to test whether multiple-answer formats (MA) and forced-choice formats (FC) produce similar results in Web surveys. Data were based on a Web survey of 1,559 Japanese adults in the Tokyo metropolitan area in March 2010. The results revealed the following: (1) Respondents endorse fewer options and take less time to answer in MA than in FC. (2) For MA respondents, options are more likely to be endorsed when they appear in the first half of a list than in the second half. These findings suggest that MA may encourage weak satisficing response strategies. In addition, these tendencies can be seen not only in attitudinal questions (judgment-type questions), but also behavioral questions (recall-type questions). However, the differences between FC and MA are greater in attitudinal questions than in behavioral questions.
Article
Due to its simplicity and efficiency for evaluating product attributes, check-all-that-apply (CATA) questions have been widely used. Yet, CATA questions have been reported to lack discriminability of subtle product differences and suffer from the problems of response bias from satisficing and acquiescence behaviors. In the present paper, a novel two-step rating-based ‘double-faced applicability’ test was developed as an extended response format of CATA, to improve its sensitivity for product discrimination and for stabilizing subjects’ evaluative criteria. In the ‘double-faced applicability’ test, each attribute was ‘double-faced’, meaning that two descriptors (a pair of semantic-differential descriptors) are separately presented in the questionnaires representing both sides of each attribute. For performing the two-step rating on each attribute in the questionnaire, subjects are instructed to first respond ‘Yes (does apply)’ or ‘No (does not apply)’ and then to answer a 3-point sureness rating (how sure they were about their Yes or No response). The performance of the two-step ratings in this new method was compared to a simple one-step applicability rating test method as well as the forced-choice Yes/No questions without the sureness rating in terms of sensitivity in sample discrimination. The results showed that the ‘double-faced applicability’ test provided better product discrimination and showed the potential to reduce acquiescence response bias when using existing variants of CATA response formats.
Article
The measurement of decisions requiring a comparison between alternatives could be improved for researchers because limitations exist with the more traditional survey techniques. To address this concern, the purpose of this review centered on discussing the merits of the forced-choice certainty method against those offered by single-stimulus Likert scale and forced-choice survey instruments. Few reviews have used the forced-choice certainty method to test topics which involve comparison and to gather accurate information on consumers, commercial products and services, and other important issues of public debate. This has occurred due to some negative literature on forced-choice surveys and preferences shown for the various reliability and validity statistics that can be easily produced with single-stimulus Likert-scale instruments. Ultimately, this work attempts to help researchers better understand the contribution that the forced-choice certainty method can make and showcase it as a product resulting from the merger of both forced-choice and Likert-scale instruments.
Article
Using newly available paradata, this article explores the use of "doorstep concerns" to predict interim and final refusals in the National Health Interview Survey (NHIS). Using ten weeks of automated contact history records, we analyze privacy and burden concerns but also examine other verbal and nonverbal interactions recorded by interviewers during contact with households. We conduct a multi-model multinomial logit analysis starting with a social environmental model (e.g., region, urbanicity), followed by the addition of process variables (e.g., number of noncontacts, mode of contact), and finally include the new household-level doorstep concerns (e.g., privacy concerns, too busy). The study found that the doorstep concerns greatly improved models predicting nonresponse relative to models including only environmental variables and basic contact history measures. Privacy concerns were significant in predicting interim refusals, but not final refusals. The effects of burden differed depending upon the particular doorstep concern used as an indicator of burden.
Article
Most social surveys in Spain used face-to-face in order to collect data. However, in market research telephone survey is the most prevalent mode. But the cost of achieving a survey using both traditional modes is increasing and, at the same time, the response rates decrease. Because each mode has its own weaknesses and strengths, the idea of mixing modes of data collection is becoming more and more popular. Nevertheless, combining different modes of data collection may be problematic if people answer differently depending on the mode. This paper presents findings from an experimental study carried out in Spain to assess the impact a change in data collection mode from the current face-to-face interviewing to telephone might have on data quality and to study the likely causes of any observed mode effects.
Conference Paper
Statistics teaching in higher education has a number of challenges. An adaptive training, tracking and teaching tool in a gaming environment aims to address problems inherent in statistics teaching. This paper discusses the implementation of this tool in a large first year university programme and considers its uses and effects. It finds that such a tool has students practice with statistics problems frequently and that success rate of the statistics course may increase.
Chapter
This chapter describes the application of consumer methodologies to the study of extra virgin olive oil (EVOO). Due to the multivariate nature of consumer behavior, both qualitative and quantitative methods were employed to provide a more holistic view of the consumption behavior. A focus group technique showed that the diversity of the participants’ experiences with olive oil resulted in differences in existing perceptions regarding what constitutes an EVOO and the meaning of ‘extra virgin’ and determined how the combination of considered factors influenced purchase and usage motivations. A two-stage sorting task was conducted to identify American consumers’ opinions of 25 EVOOs based on visual assessments of the bottles. The majority of the consumers perceived the EVOO bottles similarly; however the two-state sorting task allowed consumers to provide additional criteria of their perception of the products. Means-end chain analysis on the interview data revealed common grounds for consumption and buying motivations with three different consumer segments. As part of the quantitative research methods, survey research was employed to identify consumer preferences and attitudes regarding EVOO. Univariate and multivariate approaches were employed to understand how hedonic scores are related to descriptive analysis measurements. Three segments were identified using cluster analysis; the three segments agreed in the rejection of bitterness and pungency. In general, the positive drivers of liking are nutty, tea, green fruit, and green tomato. Some consumers are less sensitive to the presence of defects in EVOO and tend to like defective oils.
Article
Full-text available
Previous research has documented effects of the order in which response choices are offered to respondents using closed-ended survey items, but no theory of the psychological sources of these effects has yet been proposed. This paper offers such a theory drawn from a variety of psychological research. Using data from a split-ballot experiment in the 1984 General Social Survey involving a variant of Kohn's parental values measure, we test some predictions made by the theory about what kind of response order effect would be expected (a primacy effect) and among which respondents it should be strongest (those low in cognitive sophistication). These predictions are confirmed. We also test the “form-resistant correlation” hypothesis. Although correlations between items are altered by changes m response order, the presence and nature of the latent value dimension underlying these responses is essentially unaffected.
Article
Analyses of question wording experiments on the General Social Survey spending items showed consistent wording effects for several issues across three years. An examination of types of wording change indicate that even minor changes can affect responses. However, an examination of interactions with respondent individual differences showed no consistent pattern.
Article
Responses to survey questions are dependent on the words used in the questions. Sometimes the alteration of words can completely change the response distribution without obviously changing the meaning or intent of the question. This situation occurs when “welfare” is used instead of “poor.” In all contexts examined “welfare” produced much more negative and less generous responses than “poor.” In addition the two terms appear to tap slightly different dimensions with “welfare” accessing notions of waste and bureaucracy that are untapped or tapped much less by “poor.”
Article
Social values are most commonly measured using ranking techniques, but there is a scarcity of systematic comparisons between rankings and other approaches to measuring values in survey research. On the basis of data from the 1980 General Social Survey, this article evaluates the comparability of results obtained using rankings and ratings of valued qualities. The comparison focuses on (1) the ordering of aggregate value preferences and (2) the measurement of individual differences in latent value preferences. The two methods are judged to be similar with respect to ordering the aggregate preferences of the sample, but dissimilar with regard to the latent variable structure underlying the measures
Article
This paper proposes that when optimally answering a survey question would require substantial cognitive effort, some repondents simply provide a satisfactory answer instead. This behaviour, called satisficing, can take the form of either (1) incomplete or biased information retrieval and/or information integration, or (2) no information retrieval or integration at all. Satisficing may lead respondents to employ a variety of response strategies, including choosing the first response alternative that seems to constitute a reasonable answer, agreeing with an assertion made by a question, endorsing the status quo instead of endorsing social change, failing to differentiate among a set of diverse objects in ratings, saying ‘don't know’ instead of reporting an opinion, and randomly choosing among the response alternatives offered. This paper specifies a wide range of factors that are likely to encourage satisficing, and reviews relevant evidence evaluating these speculations. Many useful directions for future research are suggested.