Content uploaded by Hunter Gehlbach
Author content
All content in this area was uploaded by Hunter Gehlbach on Mar 25, 2016
Content may be subject to copyright.
A preview of the PDF is not available
Seven Survey Sins
Abstract and Figures
The coming years are likely to manifest increased pressure to administer surveys to early adolescents. Yet, because their cognitive capacities are still developing, these youth pose particular challenges for self-administered questionnaires. Thus, the biased response patterns often found in poorly constructed survey measures may be particularly acute among early adolescents. This article identifies seven common, non-obvious, consequential “survey sins.” After describing each one, strategies for addressing each problem are presented. Through these low-cost strategies, scholars can reduce the error in their measures and improve the validity of the inferences that they make.
Figures - uploaded by Hunter Gehlbach
Author content
All figure content in this area was uploaded by Hunter Gehlbach
Content may be subject to copyright.
Objective
to cross-culturally adapt and assess the content validity evidence of the Cognitive Symptom Checklist-Work-21 for the Brazilian context.
Method
a psychometric study of cross-cultural adaptation, covering the stages of translation, reconciliation, back-translation, intercultural equivalence assessment and content validity evidence analysis, considering Content Validity Ratio parameters in breast cancer survivors.
Results
the translations were equivalent to the original version. Colloquial expressions were modified, tense, verbal adjusted, and two items containing multiple commands were separated. The final version now contains 22 items, presenting semantic, conceptual, idiomatic and experimental equivalences. The pre-test indicated good understanding and ease in the response process.
Conclusion
the final version was defined as “Lista de verificação de sintomas cognitivos relacionados ao trabalho - 22 itens”, showing good linguistic equivalence and strong evidence of content validity in the Brazilian context.
Descriptors:
Validation Study; Cross-Cultural Comparison; Breast Neoplasms; Cancer Survivors; Return to Work
Objective
to cross-culturally adapt and assess the content validity evidence of the Cognitive Symptom Checklist-Work-21 for the Brazilian context.
Method
a psychometric study of cross-cultural adaptation, covering the stages of translation, reconciliation, back-translation, intercultural equivalence assessment and content validity evidence analysis, considering Content Validity Ratio parameters in breast cancer survivors.
Results
the translations were equivalent to the original version. Colloquial expressions were modified, tense, verbal adjusted, and two items containing multiple commands were separated. The final version now contains 22 items, presenting semantic, conceptual, idiomatic and experimental equivalences. The pre-test indicated good understanding and ease in the response process.
Conclusion
the final version was defined as “Lista de verificação de sintomas cognitivos relacionados ao trabalho - 22 itens”, showing good linguistic equivalence and strong evidence of content validity in the Brazilian context.
Descriptors:
Validation Study; Cross-Cultural Comparison; Breast Neoplasms; Cancer Survivors; Return to Work
Audit firms rely on audit teams where memberships are frequently shared, shifted, and dissolved. In practice, this means that many auditors are part of multiple engagement teams for a given period of time. This paper examines why and when such multiple team memberships (MTMs) may lead auditors to engage in audit quality-threatening behaviors. We analyze data from a survey of 202 auditors—ranging from assistants to partners—working at Dutch audit firms. Our findings demonstrate that serving on MTMs can undermine auditor learning and in so doing leads auditors to engage in audit quality-threatening behaviors. Analyses show that less resilient auditors—those who are less able to bounce back from experienced difficulties—appear most susceptible to these deleterious effects. In addition, exploratory analyses suggest that the negative effect of serving on many MTMs appears to be more pronounced for field-level auditors than for management-level auditors.
When formulating questionnaire items, generally accepted rules include: Keeping the wording as simple as possible and avoiding double-barreled items. However, the empirical basis for these rules is sparse. The present study aimed to systematically investigate in an experimental design whether simplifying items of a personality scale and avoiding double-barreled items (i.e., items that contain multiple stimuli) markedly increases psychometric quality. Specifically, we compared the original items of the Big Five Inventory-2 – most of which are either double-barreled or can be regarded as complexly formulated – with simplified versions of the items. We tested the two versions using a large, heterogeneous sample ( N = 2,234). The simplified versions did not possess better psychometric quality than their original counterparts; rather, they showed weaker factorial validity. Regarding item characteristics, reliability, and criterion validity, no substantial differences were identified between the original and simplified versions. These findings were also replicated for the subsample of lower-educated respondents, who are considered more sensitive to complex item formulations. Our study thus suggests that simplifying item wording and avoiding double-barreled items in a personality inventory does not improve the quality of a questionnaire; rather, using simpler (and consequently more vague) item formulations may even decrease factorial validity.
In college, students often encounter situations in which they struggle to meet their academic goals in difficult courses. We integrate the Motivational Theory of Life-Span Development and Situated Expectancy-Value Theory to investigate how motivational beliefs and experiences in a difficult course predict the use of goal engagement oriented and goal adjustment oriented control strategies that can help students stay engaged in challenging courses. We used survey data collected in two academic quarters at a public university in the U.S. ( N = 231). Students who perceived their midterm exam as more difficult than expected and students with higher course-specific subjective task values reported using more goal engagement oriented and goal adjustment oriented control strategies. Students with higher course-specific ability beliefs were less likely to use goal adjustment strategies. Results further showed that students planned to use control strategies depending on their experienced setbacks or success in exams. Findings provide important insights into how motivational orientations and course experiences relate to adaptive and goal-oriented behavior in college courses.
Based on the dichotomous classroom goal structure model and TARGET framework, this study developed and validated a new Dichotomous Classroom Goal Structure Questionnaire (DCGSQ). A total of 339 seventh-grade students in Beijing completed the two-phase survey in Study 1. Exploratory factor analysis and reliability analysis were used to assess the psychometric quality of the questionnaires in Study 1. The revised DCGSQ was further validated in Study 2 among 583 seventh-grade students in Beijing. The results of confirmatory factor analysis indicated that an 8-factor model best fit the data. The criterion validity of the DCGSQ was supported by the relations between its eight instructional dimensions and student self-efficacy.
The hobo syndrome (i.e., the wanderlust someone posits to frequently change employers) has a behavioural (i.e., frequent job-quitting behaviour) and an attitudinal dimension (i.e., attitudes towards frequent job-quitting). Across two studies, we examine both dimensions across 348 career starters. By doing so, we expand our understanding of Ghiselli’s hobo syndrome in two ways: (a) we explore the effect of both ‘bright’- and ‘dark’-side personality traits on each dimension of the hobo syndrome, and (b) using longitudinal research, we shed light on the role of each dimension of the hobo syndrome in predicting actual job-quitting behaviour. Data for both studies were gathered through a survey and LinkedIn. Results of regression analyses show that psychopathy is associated with both dimensions of the hobo syndrome. Conversely, openness to experience is only associated with the attitudinal dimension, while agreeableness and extraversion are only associated with the behavioural dimension. Finally, we find that only the behavioural dimension is associated with the length of tenure with the first employer, suggesting that one’s intentions to frequent job-quitting are more important in predicting one’s actual job-quitting behaviour than one’s attitudes towards frequent job-quitting.
Krupat weighs in on the longstanding debate about the best ways to write items for surveys by exploring what is meant by "Likert item".
This study applies multi-level analysis to student reports of effective teacher-student interactions in 50 upper elementary school classrooms (N = 594 fourth- and fifth-grade students). Observational studies suggest that teacher-student interactions fall into three domains: Emotional Support, Classroom Organization, and Instructional Support. Results of multi-level confirmatory factor analyses indicated that a three-factor model fits between- and within-classroom variability in students’ reports reasonably well. Multi-level regressions provide some evidence of criterion validity, with student reports at the classroom level related to parallel observations. Both classroom- and student-level student report data were associated with students’ reading proficiency and disciplinary referrals. Findings are discussed in terms of implications for future research on student reports of classroom interactions and their practical utility in teacher evaluation and feedback systems.
Value-added (VA) models measure teacher contributions to student learning and are increasingly employed in educational reform efforts. Using data from 35 seventh-grade teachers and 2,026 students across seven schools, we employ VA methods to measure teacher contributions to students’ motivational orientations (mastery and performance achievement goals) and their mathematics performance. The analysis suggests less variation in teachers’ contributions to students’ achievement goals than mathematics achievement. However, during a time when most students’ mastery motivation is declining sharply, a one standard deviation increase in teacher contributions to student mastery orientation is associated with a 40% smaller decline in student mastery goals. Teacher mastery contributions are also associated with gains in a student’s seventh-grade mathematics achievement (d =.11). In addition to using VA measures to focus on improving student achievement, these measures can be used to orient teachers, schools, and districts on the enhancement of students’ motivation to learn.
Evaluative processes refer to the operations by which organisms discriminate threatening from nurturant environments. Low activation of positive and negative evaluative processes by a stimulus reflects neutrality, whereas high activation of such processes reflects maximal conflict. Attitudes, an important class of manifestations of evaluative processes, have traditionally been conceptualized as falling along a bipolar dimension, and the positive and negative evaluative processes underlying attitudes have been conceptualized as being reciprocally activated, making the bipolar rating scale the measure of choice. Research is reviewed suggesting that this bipolar dimension is insufficient to portray comprehensively positive and negative evaluative processes and that the question is not whether such processes are reciprocally activated but under what conditions they are reciprocally, nonreciprocally, or independently activated. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
A theory of ironic processes of mental control is proposed to account for the intentional and counterintentional effects that result from efforts at self-control of mental states. The theory holds that an attempt to control the mind introduces 2 processes: (a) an operating process that promotes the intended change by searching for mental contents consistent with the intended state and (b) a monitoring process that tests whether the operating process is needed by searching for mental contents inconsistent with the intended state. The operating process requires greater cognitive capacity and normally has more pronounced cognitive effects than the monitoring process, and the 2 working together thus promote whatever degree of mental control is enjoyed. Under conditions that reduce capacity, however, the monitoring process may supersede the operating process and thus enhance the person's sensitivity to mental contents that are the ironic opposite of those that are intended.
Classrooms are unique and complex work settings in which teachers and students both participate in and contribute to classroom processes. This article describes the measurement phase of a study that examined the social ecology of urban classrooms. Informed by the dimensions and items of an established measure of organizational climate, we designed the Student Climate Survey (n = 53 items) to assess student psychological climate in third through eighth grades. We administered the survey to 621 students at three time points within one school year in 69 classrooms within eight urban schools. A multidimensional item response theory (IRT) analysis based on a full-information item bifactor model revealed 18 items that loaded on a primary factor and demonstrated good criterion and predictive validity. Opportunities for the Student Climate Survey to advance our contextual understanding of urban classrooms and inform intervention are discussed.
Introduction Evaluating Data Quality Number of Scale Points Labeling Scale Points No-Opinion Filters Epilogue
Examines the psychological processes involved in answering different types of survey questions. The book proposes a theory about how respondents answer questions in surveys, reviews the relevant psychological and survey literatures, and traces out the implications of the theories and findings for survey practice. Individual chapters cover the comprehension of questions, recall of autobiographical memories, event dating, questions about behavioral frequency, retrieval and judgment for attitude questions, the translation of judgments into responses, special processes relevant to the questions about sensitive topics, and models of data collection. The text is intended for: (1) social psychologists, political scientists, and others who study public opinion or who use data from public opinion surveys; (2) cognitive psychologists and other researchers who are interested in everyday memory and judgment processes; and (3) survey researchers, methodologists, and statisticians who are involved in designing and carrying out surveys. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Peers become increasingly influential in children’s development during late childhood and early adolescence. A large body of research has documented children’s proclivity for forming friendships with peers who share similar attributes to themselves, a phenomenon termed homophily. Researchers have used multiple procedures to operationalize homophily. However, a measure of homophily that accounts for the diversity of children in classrooms has yet to be validated in elementary and middle school settings. Using social network data collected from a racially diverse sample of 553 elementary and middle school students (53 classrooms; 7 schools), this study aims to validate a measure of friendship homophily that accounts for differential opportunity structures in classroom contexts. Using race and gender as grouping attributes, we find evidence for the reliability, as well as the convergent and predictive validity, of the friendship homophily measure. Implications for researchers and educators are discussed.