Article

Anchoring Vignettes: Can They Make Adolescent Self-Reports of Social-Emotional Skills More Reliable, Discriminant, and Criterion-Valid?

Article

Anchoring Vignettes: Can They Make Adolescent Self-Reports of Social-Emotional Skills More Reliable, Discriminant, and Criterion-Valid?

If you want to read the PDF, try requesting it from the authors.

Abstract

Individuals differ in the way they use rating scales to describe themselves, and these differences are particularly pronounced in children and early adolescents. One promising remedy is to correct (or "anchor") an individual's responses according to the way they use the scale when they rate an anchoring vignette (a set of hypothetical targets differing on the attribute of interest). Studying adolescents' self-reports of their socio-emotional attributes, we compared traditional self-report scores with vignette-corrected scores in terms of reliability (internal consistency), discriminant validity (scale intercorrelations), and criterion validity (predicting achievement test scores in language and math). A large and representative sample of 12th grade Brazilian students (N = 8,582, 62% female, mean age 18.2) were administered a Portugueselanguage self-report inventory assessing social-emotional skills related to the Big Five personality dimensions. Correcting scores according to vignette ratings led to increases in the reliability of scales measuring Conscientiousness and Openness, but discriminant validity and criterion validity increased only when each scale was corrected using its own corresponding vignette set. Moreover, accuracy in rating the vignettes was correlated with language achievement test scores, suggesting that verbal factors play a role in providing both normative vignette ratings of others and self-reports that are reliable and valid.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Moreover, as is well known, the responses given to this type of self-reported scale are influenced by the subject's response style, which can lead to bias in the research results that use them (Primi et al., 2016). One threat is the acquiescence, also known as the "yea-saying" effect, which concept is related to the inclination to endorse positive categories of a Likert scale despite the item content. ...
... We control the response bias and acquiescence using four inverted items (two related to the task and two to the context) besides anchoring vignettes. A study by Primi et al. (2016) indicated that the anchoring vignettes help in the control of the self-style response. Thus, for this study, three vignettes were created, describing examples of individuals with high, medium, and low job performance. ...
... Hence, an item with a raw score lower than this threshold should be coding as 1 (value 1 is the lowest score on the new scale, and indicates a lower score than the behaviors described in the first vignette). Considering the original Likert scale has five categories, the raw scores were transformed into seven points: 1 = if raw score was below the first vignette (or first threshold); 2 = raw score equal to the first vignette; 3 = raw score between the first and the second vignettes; 4 = raw score equal to the second vignette; 5 = raw score between the second and third vignettes; 6 = raw score equal to the third vignette; 7 = raw score above the third vignette (further details on the procedure are available in Primi et al. (2016)). ...
Article
Este artículo tiene como objetivo reducir la Escala de autoevaluación de Desempeño en el trabajo, como también controlar el direccionamiento de respuesta y aprobación, utilizando la técnica de viñetas y elementos invertidos. Para la reducción de la escala, se utilizó el banco de datos de la escala original, compuesta por 20 elementos divididos en dos factores: tarea y contexto. Para la reducción, fueron elegidos los 10 elementos con mayores cargas factoriales y thresholds. La escala reducida fue estimada por un factor general y dos dimensiones específicas: Tarea y contexto, representando un modelo bifactor, con indicadores de ajustes adecuados (RMSEA = 0,05; TLI = 0,98). Para controlar el direccionamiento de respuesta y aprobación, fue realizada una colecta de datos, en la cual las respuestas fueron recodificadas y realizados análisis factoriales con la finalidad de realizar una comparación de los resultados con y sin la utilización de viñetas y elementos invertidos. Los resultados apuntaron a que las viñetas mejoraron las cargas factoriales de los análisis, y que los elementos invertidos no tuvieron mejores resultados además de las viñetas. This paper aims to reduce the job performance self-assessment scale as well as control the response and acquiescence bias using vignettes anchors and inverted items. For the reduction of the scale, the original scale database was used, composed of 20 items divided into two factors: task and context. For the reduction, the 10 items with higher factor loads and thresholds were chosen. The reduced scale was estimated by a general factor and two specific dimensions: task and context, representing a bifactor model, with adequate adjustment indicators (RMSEA = 0.05; TLI = 0.98). To control response bias and acquiescence, a second study was carried out, in which the responses were recoded and factor analyses were performed in order to make a comparison of the results with and without the use of the vignettes and inverted items. The results indicated that the vignettes improved the factorial loads, however, the inverted items did not perform better than the vignettes.
... Moreover, as is well known, the responses given to this type of self-reported scale are influenced by the subject's response style, which can lead to bias in the research results that use them (Primi et al., 2016). One threat is the acquiescence, also known as the "yea-saying" effect, which concept is related to the inclination to endorse positive categories of a Likert scale despite the item content. ...
... We control the response bias and acquiescence using four inverted items (two related to the task and two to the context) besides anchoring vignettes. A study by Primi et al. (2016) indicated that the anchoring vignettes help in the control of the self-style response. Thus, for this study, three vignettes were created, describing examples of individuals with high, medium, and low job performance. ...
... Hence, an item with a raw score lower than this threshold should be coding as 1 (value 1 is the lowest score on the new scale, and indicates a lower score than the behaviors described in the first vignette). Considering the original Likert scale has five categories, the raw scores were transformed into seven points: 1 = if raw score was below the first vignette (or first threshold); 2 = raw score equal to the first vignette; 3 = raw score between the first and the second vignettes; 4 = raw score equal to the second vignette; 5 = raw score between the second and third vignettes; 6 = raw score equal to the third vignette; 7 = raw score above the third vignette (further details on the procedure are available in Primi et al. (2016)). ...
Article
Full-text available
This paper aims to reduce the job performance self-assessment scale as well as control the response bias and acquiescence bias using vignettes anchors and inverted items. The original scale database was composed of 20 items divided into two factors: task and context. For the reduction, the ten items with higher factor loads and thresholds were chosen. The reduced scale was estimated by a general factor and two specific dimensions: task and context, representing a bifactor model, with adequate adjustment indicators (RMSEA = .05; TLI = .98). To control response bias and acquiescence, a second study was carried out, in which the responses were recoded and factor analyses were performed in order to make a comparison of the results with and without the use of the vignettes and inverted items. The results indicated that the vignettes improved the factorial loads; however, the reversed items did not perform better than the vignettes.
... AVs have been shown to increase the reliability of scales assessing Conscientiousness and Openness in a representative study of 12th grade students in Brazil (N = 8,582) (Primi et al., 2016). The study applied a set of three vignettes for Conscientiousness and Openness. ...
... Hence, it is possible that the participants applied different standards in answering the AVs and the self-report personality questionnaires and hence violated the assumption of Response Consistency (discussed in detail below). Likewise, Primi et al. (2016) as well as Mõttus et al. (2012) do not report whether or not they tested any measurement assumptions (i.e., Response Consistency and Vignette Equivalence, which are described in detail below), which must be fulfilled in order to use AVs. He et al. (2017) compared different methods and procedures to improve the comparability between cultures including a vignette set with two levels of conscientiousness (N = 3,560 university students in 16 countries). ...
... The order violations within Openness are probably due to the conceptualization of the Openness factor and the corresponding AV-set. Primi et al. (2016) discovered similar results and suggested that this is because Openness is different from the other domains and is not as easy to rate as it mostly includes not observable behaviors compared to the other domains, such as Conscientiousness, which has observable behaviors. Likewise, given response consistency is traditionally difficult to confirm or disconfirm, we proposed a novel statistical solution and tested it with the present data set. ...
Article
Full-text available
Data from self-report tools cannot be readily compared between cultures due to culturally specific ways of using a response scale. As such, anchoring vignettes have been proposed as a suitable methodology for correcting against this difference. We developed anchoring vignettes for the Big Five Inventory-44 (BFI-44) to supplement its Likert-type response options. Based on two samples (Rwanda: n = 423; Philippines: n = 143), we evaluated the psychometric properties of the measure both before and after applying the anchoring vignette adjustment. Results show that adjusted scores had better measurement properties, including improved reliability and a more orthogonal correlational structure, relative to scores based on the original Likert scale. Correlations of the Big Five Personality Factors with life satisfaction were essentially unchanged after the vignette-adjustment while correlations with counterproductive were noticeably lower. Overall, these changed findings suggest that the use of anchoring vignette methodology improves the cross-cultural comparability of self-reported personality, a finding of potential interest to the field of global workforce research and development as well as educational policymakers.
... Anchoring vignettes serve the goal of taking individuals' differential interpretation and use of rating scales into account, and may help correct various response styles such as acquiescence, extreme responding, or differential use of the scale (Primi, Zanon, Santos, De Fruyt, & John, 2016). Using the anchoring vignette technique, individuals rate the behavior or traits of a person in a hypothetical scenario using the same response format and on the same rating scale as they later rate their own behavior or traits (for an example see Primi, Zanon, et al., 2016). ...
... Anchoring vignettes serve the goal of taking individuals' differential interpretation and use of rating scales into account, and may help correct various response styles such as acquiescence, extreme responding, or differential use of the scale (Primi, Zanon, Santos, De Fruyt, & John, 2016). Using the anchoring vignette technique, individuals rate the behavior or traits of a person in a hypothetical scenario using the same response format and on the same rating scale as they later rate their own behavior or traits (for an example see Primi, Zanon, et al., 2016). Because all individuals rate the same vignette, variability among individuals in their re-This document is copyrighted by the American Psychological Association or one of its allied publishers. ...
... The above-mentioned methods show different ways to correct the response tendencies associated with Likert scales, traditionally used for assessing social-emotional skills. Lessons learnt from the correction of individual differences in acquiescent responding in personality reports (e.g., Soto et al., 2008) and the promising evidence on the use of anchoring vignettes to rescale and correct self-report measures of social-emotional skills (Primi, Zanon, et al., 2016), constitute a great impulse to enhance the assessment of social-emotional constructs, especially in children and youth. Moreover, new methods of assessment have been proposed, some of them with more accumulated evidence than others in the area of social-emotional assessment (e.g., forced choice response format vs. biodata). ...
Article
Full-text available
The development and promotion of social-emotional skills in childhood and adolescence contributes to subsequent well-being and positive life outcomes. However, the assessment of these skills is associated with conceptual and methodological challenges. This review discusses how social-emotional skill measurement in youth could be improved in terms of skills' conceptualization and classification, and in terms of assessment techniques and methodologies. The first part of the review discusses various conceptualizations of social-emotional skills, demonstrates their overlap with related constructs such as emotional intelligence and the Big Five personality dimensions, and proposes an integrative set of social-emotional skill domains that has been developed recently. Next, methodological approaches that are innovative and may improve social-emotional assessments are presented, illustrated by concrete examples. We discuss how these innovations could advance social-emotional assessments, and demonstrate links to similar issues in related fields. We conclude the review by providing several concrete assessment recommendations that follow from this discussion. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
... For example, children who have peers with relatively high noncognitive skills might rate themselves relatively lower because they compare themselves with their peers. As a sensitivity analysis, our study uses anchoring vignettes, a survey method that adjusts the child reports to account for reference bias (21)(22)(23)(24)(25). ...
... To account for reference bias in children's reports, we included anchoring vignettes, which provide a way to adjust self-reported measures by "anchoring" the items of interest using the respondents' assessment of the behavior of hypothetical children (21)(22)(23)(24)(25). The instructions ask children to rate the behaviors of three hypothetical children in terms of each of the Big Five using the same response categories that we used to assess the Big Five. ...
Article
Full-text available
Significance Recent evidence has shown that noncognitive or socioemotional skills (e.g., persistence and self-control) are predictive of success in life and can be shaped through interventions. Accordingly, policy makers and researchers have increasingly measured children’s noncognitive skills, typically relying on surveys in which children rate their own skills or adults rate the skills of children. Such ratings are often collected from multiple respondent types. This study demonstrates that, compared with child and guardian reports of noncognitive skills, teacher reports are more predictive of children’s cognitive and behavioral outcomes in school. Child and guardian reports add minimal predictive power beyond teacher reports. These findings suggest that policy makers and researchers should prioritize teacher reports above those of children and guardians.
... However, those assumptions would be violated if (a) vignette questions cannot be objectively judged by respondents, or do not behave as expected, for example if ties or reversals are observed as opposed to eliciting only strictly ordered responses, and if (b) vignettes invoke other types of biases in some respondents (e.g., preference for the names or language used in the vignettes or desire for a respondent to represent themselves in a certain way when judging vignettes). As examples, tied or reversed responses to vignette questions have been observed, either due to poorly written or translated vignettes, or due to respondents' lack of attention or ability to understand the vignettes (e.g., Kyllonen & Bertling, 2014;Mõttus, Allik, Realo, Pullman, et al., 2012;Primi, Zanon, Santos, De Fruyt, & John, 2016). ...
... Several empirical studies revealed that this is not always true: On average, 25% of respondents in the PISA (Kyllonen & Bertling, 2014), 18% to 32% of respondents on the Big Five (Primi et al., 2016), and 8% to 35% of respondents in a cross-national study of conscientiousness (Mõttus et al., 2012) did not provide strictly ordered vignette scores. Moreover, the adjustment is conducted at the respondent level and applied to all original response variables in the same way, which induces effects on reliability and validity estimates by increasing correlations among transformed variables. ...
Article
Full-text available
The research presented in this article combines mathematical derivations and empirical results to investigate effects of the nonparametric anchoring vignette approach proposed by King, Murray, Salomon, and Tandon on the reliability and validity of rating data. The anchoring vignette approach aims to correct rating data for response styles to improve comparability across individuals and groups. Vignettes are used to adjust self-assessment responses on the respondent level but entail significant assumptions: They are supposed to be invariant across respondents, and the responses to vignette prompts are supposed to be without error and strictly ordered. This article shows that these assumptions are not always met and that the anchoring vignette approach leads to higher Cronbach’s alpha values and increased correlations among adjusted variables regardless of whether the assumptions of the approach are met or violated. Results suggest that the underlying assumptions and effects of the anchoring vignette approach should be carefully examined as the increased correlations and reliability estimates can be observed even for response variables that are independent random draws and uncorrelated with any other variable.
... The concept of grit was introduced by Duckworth and colleagues (Duckworth et al. 2007;Duckworth and Quinn 2009;Eskreis-Winkler et al. 2014), who defined it as perseverance and passion for long-term goals. With reference to the Big Five framework (e.g., Soto and John 2017), grit can be seen as an incremental facet of conscientiousness (e.g., Credé, Tynan, and Harms 2017;Primi et al. 2016) that emphasises the long-term aspect of goal pursuit. Accordingly, persons with a high level of grit should be more motivated and work harder, they should be able to cope with setbacks, remain focused on their goals, and thus be more successful as a result. ...
Article
Full-text available
This paper presents one of the first cross-nationally comparative studies on the economic and social returns to grit (perseverance and passion for long-term goals). Specifically, we investigated whether grit incrementally predicted objective (income) and subjective (job satisfaction) career success over and above cognitive skills and educational attainment; whether grit’s association with career success was moderated by individual characteristics; whether grit’s associations with career success varied across countries; and whether this variation could be explained by labour market conditions (unemployment rates). Multi-group structural equation models with data from 6,738 respondents from 19 countries revealed that grit was incrementally associated with income and job satisfaction in some but not all countries. Grit’s effect sizes were mostly modest, yet there were some indications for variations by education and labour market conditions. Overall, these findings suggest that the associations between grit and career success are not uniform but vary across individuals and contexts.
... That said, the distribution of many of the positively-phrased noncognitive questions is highly skewed to the right, suggesting it remains a potential concern. 37 The use of anchoring vignettes may provide a possible promising avenue to address different types of response patterns (Hopkins and King, 2010;Primi et al, 2016) and more research is needed to test this approach in large rural household surveys. ...
... Another recent anchoring study by Primi et al (2016) was based on Brazilian adolescents. The work focused on personality measures of conscientiousness and openness. ...
Article
Full-text available
We examine construct validity of the anchoring method used with 12 noncognitive scales from the Programme for International Student Assessment (PISA) 2012 project. This method combines individuals’ responses to vignettes and self-rated scores based on Likert-type items. It has been reported that the use of anchoring vignettes can reverse country-level correlations between academic achievement scores and noncognitive measures from negative to positive, and therefore align them with the typically reported individual-level correlations. Using the PISA 2012 data, we show that construct validity of this approach may be open to question because the anchored scales produce a different set of latent dimensions in comparison with nonanchored scales, even though both scales were created from the same set of individual responses. We also demonstrate that only one of three vignettes may be responsible for the resolution of the “paradox” highlighting that the choice of vignettes may be more important than what was previously reported.
... In this case, the deviance scores (but not the low-and high-trait scores) showed an acceptable metric invariance, suggesting the stability of the structure and factor loadings of the vignettes correction factor across domains, yet some uniform item bias continued to be present. This finding again speaks to the recommendation not to use one set of vignettes to rescale various domains not targeted in the vignettes (Primi, Zanon, Santos, De Fruyt, & John, 2016). ...
Article
Full-text available
This study aims to evaluate a number of procedures that have been proposed to enhance cross-cultural comparability of personality and value data. A priori procedures (anchoring vignettes and direct measures of response styles (i.e. acquiescence, extremity, midpoint responding, and social desirability), a posteriori procedures focusing on data transformations prior to analysis (ipsatization and item parcelling), and two data modelling procedures (treating data as continuous vs as ordered categories) were compared using data collected from university students in 16 countries. We found that (i) anchoring vignettes showed lack of invariance, so they were not bias-free; (ii) anchoring vignettes showed higher internal consistencies than raw scores where all other correction procedures, notably ipsatization, showed lower internal consistencies; (iii) in measurement invariance testing, no procedure yielded scalar invariance; anchoring vignettes and item parcelling slightly improved comparability, response style correction did not affect it, and ipsatization resulted in lower comparability; (iv) treating Likert-scale data as categorical resulted in higher levels of comparability; (v) factor scores of scales extracted from different procedures showed similar correlational patterning; and (vi) response style correction was the only procedure that suggested improvement in external validity of country-level conscientiousness. We conclude that, although no procedure resolves all comparability issues, anchoring vignettes, parcelling, and treating data as ordered categories seem promising to alleviate incomparability. We advise caution in uncritically applying any of these procedures. Copyright © 2017 European Association of Personality Psychology
... Užívání škály je tedy do určité míry podobné v obou zkoumaných oblastech. Pro korektní očištění o odlišné užívání škály lze však doporučit užívat viněty, které popisují stejný koncept jako hodnoticí otázky (více Primi et al., 2016;Voňková et al., 2017). (Voňková, 2016). ...
... The exception occurs in the locus of control scale that means external beliefs ("Do you usually feel that it's almost useless to try in school because most other children are just plain smarter than you are?") and internal beliefs ("Most of the time, do you feel that you can change what might happen tomorrow by what you do today?"). A series of studies has already investigated its psychometric qualities (PANCORBO; LAROS, 2017;PRIMI et al., 2019c;PRIMI, et al., 2016a;PRIMI et al., 2016b). ...
Article
Full-text available
RESUMO No contexto educacional, cada vez mais, uma série de habilidades, além das cognitivas, vêm sendo valorizadas. Dentre elas, os construtos de criatividade e competências socioemocionais, foco do estudo. Tendo como objetivo investigar a relação entre os dois construtos, a amostra foi composta por 362 estudantes do 3o(n = 168) e 5o ano (n = 194) do Ensino Fundamental, idades entre 8 e 15 anos (M = 10,3 anos; DP = 1,33), sendo 180 do gênero feminino, provenientes de 13 diferentes escolas públicas municipais localizadas no estado de Pernambuco, Brasil. Os estudantes responderam dois testes de criatividade (figural e verbal) e um instrumento que avalia seis competências socioemocionais (abertura a novas experiências, conscienciosidade, extroversão, amabilidade, lócus de controle externo e neuroticismo). Os resultados apontaram para a existência de correlações positivas significativas entre a criatividade figural e quatro das competências socioemocionais (conscienciosidade, amabilidade, abertura a experiências e lócus de controle externo) e da criatividade verbal com duas dimensões socioemocionais (conscienciosidade e amabilidade). Ainda que os construtos tenham apresentado alguma comunalidade, a importância de se estimular ambos no contexto educacional se faz presente, dados os resultados positivos que têm sido relacionados a essas habilidades, consideradas essenciais no século XXI.
... Kapteyn et al. 2007). Currently it is also being applied in the education sciences, specifically in research into teachers' educational support of students and teachers' classroom management (Kyllonen and Bertling 2013;Vonkova et al. 2018), students' motivation for learning and personal characteristics (He and van de Vijver 2016;Primi et al. 2016), and students' dishonest behavior in school (Vonkova et al. 2017) and students' ICT skills (Vonkova and Hrabak 2015). In general, studies using the anchoring vignette method show that the comparison of different countries or socioeconomic groups of respondents is (significantly) influenced by heterogeneity in reporting behavior and presenting results without adjusting for this heterogeneity may be misleading. ...
Article
Full-text available
The valid measurement of children’s life satisfaction is key for subsequent analysis and policy recommendations. It has been demonstrated that individuals use different scales in (self-)reports which leads to misleading results. In this study we focus on the analysis of life satisfaction self-reports among children with differences in the following characteristics: family, school, and free-time activities. Using the anchoring vignette method, we analyze differences in response scale usage among the groups and the impact of these differences on the comparison of life satisfaction among the groups. Our sample (N = 3737) is a representative sample of 5th graders (11 year-olds) in the Czech Republic. After adjustment for differential scale usage, children’s life satisfaction is significantly positively related to being female, having a father at home, having good school grades, spending time with friends and the level of education they expect to attain. It is significantly negatively linked to preparing for hard admission exams and time spent playing computer games. The adjustment for response scale differences substantially changes the comparison of different groups. The most significant change is for gender – after correction girls’ life satisfaction is significantly higher while before correction it is the opposite. Before adjustment the differences between some groups are underestimated – children with excellent grades, living with their father and spending at least some time with friends have a higher level of life satisfaction after adjustment in comparison to other children. We recommend examining the differences in scale usage among different cultures, countries and groups in children’s life satisfaction research.
... The Big Five Model (McCrae & John, 1992) has served as a theoretical framework for the evaluation of the 21 st Century Skills of children and adolescents (Kyllonen, Lipnevich, Burrus, & Roberts, 2008;Primi, Zanon, Santos, de Fruyt, & John, 2016). These skills are positively related to a series of desirable outcomes in school, including school engagement (Duckworth & Seligman, 2005;Fredricks & McColskey, 2012;Zanon, Bastianello, Pacico, & Hutz, 2014) and healthy and well-succeeded development (Kautz, Heckman, Diris, Weel, &Borghans, 2014;Saklofske et al., 2012). ...
Article
Full-text available
The Big Five has been used as a theoretical framework for the evaluation of the 21stcentury skills and is associated with desirable outcomes. This study evaluates the extent to which the Big Five relates to students’ adaptation to the academic environment. Participants were 845 undergraduate students (60.7% female), aged from 17 to 31 years who answered the Factorial Personality Battery and the Academic Experience Questionnaire. Data reduction analysis at the facet level recovered the Big Five structure and indicated that Extraversion, Neuroticism, and Conscientiousness are, respectively, associated with higher personal, interpersonal, and study related academic adaptation dimensions. Agreeableness and Openness did not cluster with academic dimensions, suggesting lower relevance for academic adaptation in higher education.***A Relevância do Modelo dos Cinco Grandes Para a Adaptação dos Estudantes ao Ambiente Acadêmico***O Big Five tem sido usado como referencial teórico para a avaliação das habilidades do século 21 e está associado a resultados desejáveis. Este estudo avalia o quanto os Cinco Grandes Fatores se relacionam com a adaptação dos alunos ao ambiente acadêmico. Participaram 845 estudantes de graduação (60,7% do sexo feminino), com idades entre 17 e 31 anos, que responderam a Bateria de Personalidade Fatorial e o Questionário de Experiência Acadêmica. A análise de redução de dados no nível de faceta recuperou a estrutura dos cinco grandes fatores e indicou que Extroversão, Neuroticismo e Conscienciosidade estão, respectivamente, associados a dimensões de adaptação acadêmica pessoais, interpessoais e relacionadas ao estudo. A Amabilidade e a Abertura não se agruparam com as dimensões acadêmicas, sugerindo menor relevância para a adaptação acadêmica no ensino superior.
... A razão para o uso de escala de escolha forçada se refere aos vieses comuns às respostas de escala tipo Likert(Brown & Maydeu-Olivares, 2013), especialmente no caso de construtos que são mais sujeitos à as respostas dos participantes foram recodificadas de tal sorte que quem apresentou uma avaliação mais ou menos severa teve seus escores corrigidos nesta respectiva direção. Se a resposta do participante ficasse abaixo do padrão da vinheta, era atribuída uma nota mais elevada e o oposto ocorria quando o participante tinha respostas acima do parâmetro da vinheta.Assim, as respostas aos questionários dadas pelos participantes foram recodificadas, considerando o grau de exigência demonstrado por cada um na avaliação das vinhetas âncora(Primi et al., 2016). Os casos em que as respostas às vinhetas foram aleatórias/incoerentes foram desconsiderados da amostra, em função da baixa confiabilidade das respostas aos demais itens do questionário. ...
Article
O processo de desenvolvimento profissional ocorre de formas distintas para as pessoas. Esta pesquisa objetivou comparar as percepções de tal desenvolvimento e de seus propulsores em trabalhadores com e sem pós-graduação. A pesquisa com 388 trabalhadores de diferentes ocupações e profissões (75,7% mulheres, idade média de 43,7 anos) usou escala de escolha forçada, para priorização dos elementos que influenciam o desenvolvimento profissional. Também foram utilizadas vinhetas âncoras para ajuste das respostas à escala de julgamento. Os resultados indicaram mais similitudes do que diferenças nos propulsores do desenvolvimento profissional dos participantes com e sem pós-graduação, mas há alguns propulsores distintos entre eles. Os achados podem contribuir para a avaliação e a tomada de decisões de carreira de diferentes trabalhadores.
... [14] use anchoring vignettes to compare measures of Conscientiousness across 21 countries and show that country rankings of self-reported Conscientiousness to some degree result from differences in response styles. [15] show that the reliability of scales assessing Conscientiousness and Openness to Experience increases when using anchoring vignettes in a study of 12th-grade students in Brazil. [16] employ anchoring vignettes for the Big Five Inventory in Rwanda and the Philippines. ...
Article
Full-text available
In this study, we investigate whether Conscientiousness, Emotional Stability and Risk Preference relate to student performance in higher education. We employ anchoring vignettes to correct for heterogeneous scale use in these non-cognitive skills. Our data are gathered among first-year students at a Dutch university. The results show that Conscientiousness is positively related to student performance, but the estimates are strongly biased upward if we use the uncorrected variables. We do not find significant relationships for Emotional Stability but find that the point estimates are larger when using the uncorrected variables. Measured Risk Preference is negatively related to student performance, yet this is fully explained by heterogeneous scale use. These results indicate the importance of using more objective measurements of personality traits.
... An additional burden is that the anchoring vignettes themselves have to be developed, which requires expertise, time, and resources. However, if these hurdles are overcome, anchoring vignettes can help to increase the reliability and validity of measures (e.g., Primi, Zanon, Santos, De Fruit, & John, 2016). ...
Article
The anchoring vignettes method as an alternative to the questionnaire using the Likert scale was newly applied to the student survey in the 2012 Programme for International Student Assessment (PISA). The three levels of vignettes (high, medium, and low levels) were included in the student survey related to ‘Teacher Support’ and ‘Classroom Management’ in mathematics, the main domain in the PISA 2012. Based on responses to the vignette questions, the PISA provided adjusted scaled scores for 12 student survey constructs. This study re-visited the responses of the vignettes as well as the relevant survey constructs, particularly focusing on the Korean students. The study findings can provide meaningful implications for understanding the response patterns for the Korean students on attitudes toward mathematics and developing survey tools using vignettes.
Article
Full-text available
PurposeAnchoring vignettes (AVs) are a promising measurement technique to reduce bias in patient-reported outcome (PRO) measures by helping researchers understand differences in how individuals and groups interpret response options. However, little attention has been paid to ensure quality development of AVs, and their performance has not been well assessed in pediatric populations. In this study, we explore the application of a rigorous development process for AVs based upon current standards for PROs, as well as feasibility of AVs when administered to children and adolescents.Methods We developed AVs using a rigorous, patient-centered mixed methods process including three phases: (1) development, (2) a pilot study, and (3) a field test. Our proposed process included the generation of a conceptual framework based on the PRO, the Localized Scleroderma Quality of Life Instrument, and numerous vignette-specific considerations. We qualitatively explored readability and comprehension of the AVs (pilot study) and then analyzed ranking patterns within vignette sets (field test).ResultsFour sets of four vignettes were developed. Revisions were suggested at each phase of development. The pilot study demonstrated that children ≥ 10 years had no trouble indicating understanding of the AVs. In the field test, although appropriate rankings of vignettes were generally demonstrated by participants, the percentage of tied rankings was higher than expected in this pediatric group.Conclusions This work supports the need for rigorous developmental standards for AVs, as each stage of development suggested revisions. Additionally, AVs showed initial promise for use with pediatric populations; general feasibility and understanding were supported.
Article
We investigate whether Anchoring Vignettes (AV) improve intercultural comparability of non-cognitive student-directed factors (e.g., procrastination). So far, correlation analyses for anchored and non-anchored scores with a criterion have been used to demonstrate the effectiveness of AV in improving data quality. However, correlation analyses are often used to investigate external validity of a scale. Nonetheless, before testing for validity, the reliability of the measurement of a construct should be examined. In the present study, we tested for measurement invariance across countries and languages and compared anchored and non-anchored student-directed self-reports that are highly relevant for the students’ self and their behaviour and performance. In addition, we apply further criteria for testing reliability. The results indicate that the data quality for some of the constructs can – in fact – be improved slightly by anchoring; whereas, for other self-reports, anchoring is less successful than was hoped. We discuss with regard to possible consequences for research methodology.
Chapter
When analyzed via item response theory, Likert-type items are modeled by estimating a set of thresholds (i.e., parameters that inform on the latent trait level required for endorsing a given scale option) that are assumed to be invariant across the population of individuals. If persons vary in response styles this assumption may not hold. This is called person differential functioning (PDIF). Anchoring vignettes offer an approach to learn how individuals translate the latent trait into Likert responses, and a method to assess potential variability in item thresholds across individuals. A vignette presents hypothetical persons differing on the attribute of interest (usually low, medium and high), and asks respondents to rate the hypothetical persons in the same Likert scale used in self-assessment. This can then be used to resolve PDIF, potentially producing measures that are more comparable. We investigated if the patters of responses to vignettes have a developmental trend and if they are related to cognitive capacity, using data from a large-scale educational assessment. We then investigated if anchor-adjusted scores produce more reliable and valid measures.
Thesis
Full-text available
The job performance is an important factor for the work psychology, as it relates to the fundamental characteristics not only for the individual but also for the organization. This research aims to improve an instrument for performance evaluation, as well as to test a network model for the job performance. The dissertation is composed of two studies. The first proposes to reduce the Self-Assessment Scale of Performance at work, as well as to control the response bias, by using vignettes anchors and inverted items. To reduce the scale, it was used the original scale validation database, composed of 1,617 participants. The original scale consists of 20 items divided into two factors: task performance, focused on the technical part of the company; And context performance, focused on the social and psychological support, in which organizational goals are sought. For the reduction, 10 items with higher factor loadings and varied thresholds were chosen. In the final structure, it was estimated a general factor and two specific dimensions related to context and task. Therefore, the reduced scale version was structured by a bifactor model, which presented good fit indices (RMSEA = 0.05, TLI = 0.98). In order to control response bias, a new data collect was carried out, in which 303 workers from different professions participated. The responses were re-coded and confirmatory factorial analyzes were performed to compare the results with and without the use of the vignettes and the inverted items. The results indicated that the anchors improved the factor loads of analyzes, however, the inverted items did not perform better than the vignettes. The goal of the second study is to analyze a theoretical model setting relations among the intelligence, learning support and job performance, as well as the mediation role of job crafting and the moderation role of the subjective complexity of the task (i.e. a moderated mediation model). 250 employees of public and private companies participated in this study. Participants answered questionnaires for the purpose of measuring intelligence, job performance, job crafting, learning support and the subjective complexity of the task. The results showed that there was no direct relation between intelligence and performance for the analyzed database. However, positive relations between support learning and performance were found. The job crafting showed a mediation effect on the relationship between the learning support and the job performance. And the task complexity showed a moderation effect on the relation between the learning support and job crafting.
Article
With the introduction of school choice as a viable option of school reform, researchers have begun to look beyond academic achievement to the effects of choice on alternative outcomes. Recently, non-cognitive skills and behavior stand at the front of those alternative outcomes. From the most up to date relevant literature it appears that school choice has adverse effects on non-cognitive skills but positive effects on behavior. The potential presence of reference group bias highlights the possibility that the negative effects of choice on non-cognitive skills may not be real.
Thesis
This dissertation examines the role that cognitive and non-cognitive skills play in developing countries along three axes: measurement of these skills, wage returns to them, and as determinants of levels of trust. The first chapter uses panel data from two cohorts of a skills training program in Mozambique to contrast two types of skill measurement: self-assessments and observational exercises. The chapter shows that self-assessments capture underlying traits, while observational exercises are better suited for program evaluations. The second chapter is based on a novel matched employer-employee data set representing the formal sector in Bangladesh and provides evidence of both the relative importance of cognitive and non-cognitive skills in this part of the labor market and the interplay between skills and hiring channels in determining wages. Results demonstrate positive returns to non-cognitive skills, varying by hiring channel. This chapter also offers suggestive evidence that employers might use hiring channels differently, depending on which skills they deem important. The third chapter makes use of the demonetization policy of November 2016 in India and a newly collected data set to causally identify the determinants of trust in a rural setting, controlling for a variety of individual characteristics, such as non-cognitive skills and cognitive ability, that could influence network formation and trust. We find that social interactions determine trust, though this result holds only among men and varies along the lines of caste membership.
Chapter
Scholars have presented much evidence that supports a crucial role of human agency in understanding human life. Yet agency is still one of the most “slippery” and elusive concepts that bring confusions to social science research. Building on recent attempts to build theoretical and empirical clarity of agency as a sociological concept, this chapter sheds light on additional facet of agency: a behavioural dimension of agency beyond subjective agentic beliefs and future-oriented expectations. This chapter introduces relevant concepts, such as grit, that tap into the behavioural facet of agency and shows why it is important to incorporate this additional component into our understanding of agency in an individual’s life course, particularly in the contexts of educational and occupational achievement.
Article
The aim of the current research was to provide recommendations to facilitate the development and use of anchoring vignettes (AVs) for cross-cultural comparisons in education. Study 1 identified six factors leading to order violations and ties in AV responses based on cognitive interviews with 15-year-old students. The factors were categorized into three domains: varying levels of AV format familiarity, differential interpretations of content, and individual differences in processing information. To inform the most appropriate approach to treat order violations and re-scaling method, Study 2 conducted Monte Carlo simulations with the manipulation of three factors and incorporation of five responses styles. Study 2 found that the AV approach improved accuracy in score estimation by successfully controlling for response styles. The results also revealed the reordering approach to treat order violations produced the most accurate estimation combined with the re-scaling method assigning the middle value among possible scores. Along with strategies to develop AVs, strengths and limitations of the implemented nonparametric AV approach were discussed in comparison to item response theory modeling for response styles.
Article
This research aimed to examine the enhancement of soft skills in public education in Peru through analysing non-profit organisations’ techniques at the school level. The objectives were (a)To understand how the quality of education in Peruvian public schools assures the development of soft skills, (b)To explore the potential of non-profit organisations in supporting the development of soft skills in public schools. Interviews with three non-profit organisations and a government official, alongside reports, were analysed based on an OECD model. The findings include government initiatives that indirectly develop soft skills, but management issues might lead to teacher-centred rather than student-centred methodologies, deterring skills development. Unfriendly environments characterise vulnerable areas for skills development; non-profits become supportive disruptive spaces to cultivate values, ethics, and skills. The discourse analysis concludes that, despite their small scope, they mainly develop collaboration and task performance skills, impacting career decisions, families and communities. Non-profits employ a playful and concrete methodology that fosters the development of soft skills, but which is difficult to implement in public schools due to a lack of resources and school management. Further research should explore non-profits’ impact on skills development. This study is a pioneer in this field in LATAM.
Article
Anchoring vignettes are item batteries especially designed for correcting responses that might be affected by incomparability. This article investigates the effects of anchoring vignettes on the validity of student self-report data in 64 cultures. Using secondary data analysis from the 2012 Programme for International Student Assessment (PISA), we checked the validity of ratings on vignette questions, and investigated how rescaled item responses of two student scales, Teacher Support and Classroom Management, enhanced comparability and predictive validity. The main findings include that (a) responses to vignette questions represent valid individual and cultural differences; in particular, violations in these responses (i.e., misorderings) are related to low socioeconomic status and low cognitive sophistication; (b) the rescaled responses tend to show higher levels of comparability; and (c) the associations of rescaled Teacher Support and Classroom Management with math achievement, Student-Oriented Instruction, and Teacher-Directed Instruction are slightly different from raw scores of the two target constructs, and the associations with rescaled scores seem to be more in line with the literature. Namely, the associations among all self-report Likert-type scales are weaker with rescaled scores, presumably reducing common method variance, and both rescaled scale scores are more positively related to math achievement. The country ranking also changes substantially; in particular, Asian cultures top the ranking on Teacher Support after rescaling. However, anchoring vignettes are not a cure-all in solving measurement bias in cross-cultural surveys; we discuss the technicality and directions for further research on this technique.
Article
Full-text available
Neubert, Mainert, Kretzschmar, and Greiff (2015) plea to integrate the 21st century skills of complex problem solving (CPS) and collaborative problem solving (ColPS) in the assessment and development suite of industrial and organizational (I-O) psychologists, given the expected increase in nonroutine and interactive tasks in the new workplace. At the same time, they promote new ways of assessing these skills using computer-based microworlds, enabling the systematic variation of problem features in assessment. Neubert and colleagues’ (2015) suggestions are a valuable step in connecting differential psychologists’ models of human differences and functioning with human resources professionals’ interest in understanding and predicting behavior at work. We concur that CPS and ColPS are important transversal skills, useful for I-O psychologists, but these are only two babies of a single family, and the domain of 21st century skills includes other families of a different kind that are also with utility for I-O psychologists. The current contribution is meant to broaden this interesting discussion in two important ways. We clarify that CPS and ColPS need to be considered in the context of a wider set of 21st century skills with an origin in the education domain, and we highlight a number of crucial steps that still need to be taken before “getting started” (Neubert et al., 2015, p. last page of the discussion) with this taxonomic framework. But first, we feel the need to slightly reframe the relevance of considering 21st century skills in I-O psychology by shifting the attention from narrow task-related skills to the broader domain of career management competencies.
Article
Full-text available
In two studies, we examined the shared and unique meaning of acquiescent, extreme, midpoint, and socially desirable responding in association with the Occupational Personality Questionnaire (OPQ32), a forced-choice format personality measure designed to be less affected by these response styles, compared with personality inventories with Likert-type scales. Country-level response style indexes were derived from six waves of the International Social Survey Programme and from a meta-analysis of a social desirability scale. In the country-level correlational analysis, the four response styles formed a general response style (GRS) factor which was positively associated with (a) dominance (vs. submission) in interpersonal relationships, (b) competitive (vs. modest and democratic) feelings and emotions, and (c) data rational thinking. In a multilevel analysis, age showed a positive and education a negative effect on the individual-level GRS. Negative effects of country-level socioeconomic development and individualism and positive effects of competitiveness and data rational thinking on the individual-level response style were found. We conclude that country-level response styles are systematically associated with country personality measured by the OPQ32, suggesting that they can be viewed as having substantive meaning (i.e., culturally influenced response amplification vs. moderation). Implications are discussed.
Article
Full-text available
This study proposes and tests an alternative methodology to conceptualize and assess aberrant personality tendencies at work beyond the dark triad. A sample of college alumni (N= 247) were administered the NEO PI‐R prior to entering the labor market and 15 years later when their professional careers had unfolded. Drawing on the dimensional perspective on personality functioning, 6 five‐factor model (FFM) aberrant compounds were computed as indicators of aberrant personality tendencies. As expected, FFM aberrant personality tendencies were highly stable across time, with test–retest correlations ranging from .61 (Narcissistic) to .73 (avoidant). With regard to predictive validity, borderline, schizotypal, and avoidant tendencies were negatively associated with extrinsic and intrinsic career outcomes. The obsessive‐compulsive tendency was largely unrelated to career outcomes, whereas individuals with antisocial and narcissistic characteristics tended toward higher hierarchical and financial attainment. In addition, relative importance analyses indicated that (a) FFM aberrant personality tendencies showed incremental validity in the prediction of career outcomes beyond FFM general traits, and that (b) both FFM general and FFM aberrant personality tendencies are important predictors when considered jointly. It is concluded that FFM aberrant personality tendencies suggest interesting avenues for personnel psychologists to form new linear combinations of FFM facets, complementing FFM general domains.
Article
Full-text available
In cross-cultural studies with social variables such as values or attitudes, it is often assumed that differences in scores can be compared at face value. However, response styles like acquiescence and extreme response style may affect answers, particularly on rating scales. In three sets of data from marketing studies, each with representative samples fromat least threeoutofsix countries(Greece,Italy, Spain,France,Germany, andthe United Kingdom), these two response styles were found to be more present in the Mediterranean than in Northwestern Europe. Evidence for response style effects was not only found in response distributions on rating scales, but also in discrepancies of these distributions with national consumer statistics and self-reported actual behavior.
Article
Full-text available
Studies of attitudes across countries generally rely on a comparison of aggregated mean scores to Likert-scale questions. This presupposes that when people complete a questionnaire, their answers are based on the substantive meaning of the items to which they respond. However, people's responses are also influenced by their response style. Hence, the studies we conduct might simply reflect differences in the way people respond to surveys, rather than picking up real differences in management phenomena across countries. Our 26-country study shows that there are major differences in response styles between countries that both confirm and extend earlier research. Country-level characteristics such as power distance, collectivism, uncertainty avoidance and extraversion all significantly influence response styles such as acquiescence and extreme response styles. Further, English-language questionnaires are shown to elicit a higher level of middle responses, while questionnaires in a respondent's native language result in more extreme response styles. Finally, English-language competence is positively related to extreme response styles and negatively related to middle response styles. We close by discussing implications for cross-national research.
Article
Full-text available
In cross-national studies, mean levels of self-reported phenomena are often not congruent with more objective criteria. One prominent explanation for such findings is that people make self-report judgements in relation to culture-specific standards (often called the reference group effect), thereby undermining the cross-cultural comparability of the judgements. We employed a simple method called anchoring vignettes in order to test whether people from 21 different countries have varying standards for Conscientiousness, a Big Five personality trait that has repeatedly shown unexpected nation-level relationships with external criteria. Participants rated their own Conscientiousness and that of 30 hypothetical persons portrayed in short vignettes. The latter type of ratings was expected to reveal individual differences in standards of Conscientiousness. The vignettes were rated relatively similarly in all countries, suggesting no substantial culture-related differences in standards for Conscientiousness. Controlling for the small differences in standards did not substantially change the rankings of countries on mean self-ratings or the predictive validities of these rankings for objective criteria. These findings are not consistent with mean self-rated Conscientiousness scores being influenced by culture-specific standards. The technique of anchoring vignettes can be used in various types of studies to assess the potentially confounding effects of reference levels. Copyright © 2011 John Wiley & Sons, Ltd.
Article
Full-text available
Scales were developed to assess 10 specific facet traits within the broad Big Five personality domains from the item pool of the Big Five Inventory (BFI). In two independent samples, the BFI facet scales dem-onstrated substantial (a) reliability, (b) convergence with self-reports on the Revised NEO Personality Inventory and peer-reports on the BFI, and (c) discriminant validity. These brief scales offer new oppor-tunities for researchers who wish to assess specific personality characteristics within an overarching Big Five framework. for serving as judges for the correlate-matching task.
Article
Full-text available
Rankings of countries on mean levels of self-reported Conscientiousness continue to puzzle researchers. Based on the hypothesis that cross-cultural differences in the tendency to prefer extreme response categories of ordinal rating scales over moderate categories can influence the comparability of self-reports, this study investigated possible effects of response style on the mean levels of self-reported Conscientiousness in 22 samples from 20 countries. Extreme and neutral responding were estimated based on respondents' ratings of 30 hypothetical people described in short vignettes. In the vignette ratings, clear cross-sample differences in extreme and neutral responding emerged. These responding style differences were correlated with mean self-reported Conscientiousness scores. Correcting self-reports for extreme and neutral responding changed sample rankings of Conscientiousness, as well as the predictive validities of these rankings for external criteria. The findings suggest that the puzzling country rankings of self-reported Conscientiousness may to some extent result from differences in response styles.
Article
Full-text available
When respondents use the ordinal response categories of standard survey questions in different ways, the validity of analyses based on the resulting data can be biased. Anchoring vignettes is a survey design technique intended to correct for some of these problems. The anchors package in R includes methods for evaluating and choosing anchoring vignettes, and for analyzing the resulting data.
Article
Full-text available
Cultural differences in questionnaire response styles have been reported in a number of studies. Compared to those of European-heritage, responses from individuals of East-Asian heritage tend to be more ambivalent and moderate. These stylistic differences warrant attention because they may contaminate substantive conclusions about cultural differences. One possible source of these stylistic differences is the East-Asian tendency toward dialectical thinking, that is, a tolerance of contradictory beliefs. In Study 1 (N = 4835), we exemplify the difficulty of demonstrating cultural differences in response styles using the Rosenberg self-esteem scale: the problem is that ambivalence and moderacy indices are inextricably confounded with mean levels of self-esteem. In Study 2 (N = 185), we use a broader set of personality measures to show that cultural differences in ambivalence and moderacy can be established independently of mean levels. We go further to show that dialectical thinking mediates the cultural differences in moderacy and ambivalence. Together, these studies demonstrate the importance of considering dialectical thinking when interpreting results from cross-cultural surveys.
Article
Full-text available
Incl. bibl., index.
Article
Full-text available
This article reports a meta-analysis of personality-academic performance relationships, based on the 5-factor model, in which cumulative sample sizes ranged to over 70,000. Most analyzed studies came from the tertiary level of education, but there were similar aggregate samples from secondary and tertiary education. There was a comparatively smaller sample derived from studies at the primary level. Academic performance was found to correlate significantly with Agreeableness, Conscientiousness, and Openness. Where tested, correlations between Conscientiousness and academic performance were largely independent of intelligence. When secondary academic performance was controlled for, Conscientiousness added as much to the prediction of tertiary academic performance as did intelligence. Strong evidence was found for moderators of correlations. Academic level (primary, secondary, or tertiary), average age of participant, and the interaction between academic level and age significantly moderated correlations with academic performance. Possible explanations for these moderator effects are discussed, and recommendations for future research are provided.
Article
Full-text available
We examined several determinants of interjudge agreement on personality traits. The findings, which were cross-validated in two samples, suggest that agreement is a function of four factors: which Big Five content domain the trait represents, how observable relevant behaviors are, how evaluative the trait is, and whether the self is one of the judges. Agreement was highest for traits related to Extraversion and lowest for traits related to Agreeableness. More observable and less evaluative traits elicited higher interjudge agreement. On average, self-peer agreement was lower than peer-peer agreement. However, this effect was limited to evaluative traits; for neutral traits, self-peer agreement was as high as peer-peer agreement. These findings suggest that self- and peer perception proceed through similar processes for neutral traits but not for highly evaluative traits, raising the possibility that self-perceptions become distorted when the trait is affectively charged.
Article
Full-text available
Personality profiles of cultures can be operationalized as the mean trait levels of culture members. College students from 51 cultures rated an individual from their country whom they knew well (N=12,156). Aggregate scores on Revised NEO Personality Inventory (NEO-PI-R) scales generalized across age and sex groups, approximated the individual-level 5-factor model, and correlated with aggregate self-report personality scores and other culture-level variables. Results were not attributable to national differences in economic development or to acquiescence. Geographical differences in scale variances and mean levels were replicated, with Europeans and Americans generally scoring higher in Extraversion than Asians and Africans. Findings support the rough scalar equivalence of NEO-PI-R factors and facets across cultures and suggest that aggregate personality profiles provide insight into cultural differences.
Article
Full-text available
This study examines 5 types of personality continuity--structural, mean-level, individual-level, differential, and ipsative--in a representative population (N=498) and a twin and sibling sample (N=548) of children and adolescents. Parents described their children on 2 successive occasions with a 36-month interval using the Hierarchical Personality Inventory for Children (I. Mervielde & F. De Fruyt, 1999). There was evidence for structural continuity in the 2 samples, and personality was shown to be largely differentially stable. A large percentage had a stable trait profile indicative of ipsative stability, and mean-level personality changes were generally small in magnitude. Continuity findings were explained mainly by genetic and nonshared environmental factors.
Article
Full-text available
How do youths' personality reports differ from those of adults? To identify the year-by-year timing of developmental trends from late childhood (age 10) to early adulthood (age 20), the authors examined Big Five self-report data from a large and diverse Internet sample. At younger ages within this range, there were large individual differences in acquiescent responding, and acquiescence variability had pronounced effects on psychometric characteristics. Beyond the effects of acquiescence, self-reports generally became more coherent within domains, and better differentiated across domains, at older ages. Importantly, however, different Big Five domains showed different developmental trends. Extraversion showed especially pronounced age gains in coherence but no gains in differentiation. In contrast, Agreeableness and Conscientiousness showed large age gains in differentiation but only trivial gains in coherence. Neuroticism and Openness showed moderate gains in both coherence and differentiation. Comparisons of items that were relatively easy versus difficult to comprehend indicated that these patterns were not simply due to verbal comprehension. These findings have important implications for the study of personality characteristics and other psychological attributes in childhood and adolescence.
Article
Whereas the structure of individual differences in personal attributes is well understood in adults, much less work has been done in children and adolescents. On the assessment side, numerous instruments are in use for children but they measure discordant attributes, ranging from one single factor (self-esteem; grit) to three factors (social, emotional, and academic self-efficacy) to five factors (strength and difficulties; Big Five traits). To construct a comprehensive measure for large-scale studies in Brazilian schools, we selected the eight most promising instruments and studied their structure at the item level (Study 1; N = 3,023). The resulting six-factor structure captures the major domains of child differences represented in these instruments and resembles the well-known Big Five personality dimensions plus a negative self-evaluation factor. In a large representative sample in Rio de Janeiro State (Study 2; N = 24,605), we tested a self-report inventory (SENNA1.0) assessing these six dimensions of socio-emotional skills with less than 100 items and found a robust and replicable structure and measurement invariance across grades, demonstrating feasibility for large-scale assessments across diverse student groups in Brazil. Discussion focuses on the contribution to socio-emotional research in education and its measurement as well as on limitations and suggestions for future research.
Article
This article concerns how noncognitive constructs—personality and motivation—can be assessed and developed to increase students' readiness for college. We propose a general framework to account for personality and motivational differences between students. We review numerous studies showing that personality and motivational factors are related to educational outcomes, from early childhood to adulthood. We discuss various methods for assessing noncognitive factors, ranging from self-assessments to performance tests. We consider data showing that personality and motivation change over time and find that particular interventions have proven successful in changing particular personality facets, leading to increased achievement. In a final section we propose a strategy for implementing a comprehensive psychosocial skills assessment in middle and high school, which would include setting proficiency standards and providing remedial instruction.
Article
Secondary analyses were carried out with four large data sets that included responses by Hispanics (N = 1,908) and by non-Hispanic Whites (N = 14,425). Results are fairly consistent in showing that Hispanics prefer extreme responses to a greater extent than non-Hispanic Whites. In general, Hispanics prefer to agree with a given item more than non-Hispanic Whites. Two significant variables seem to affect the rate at which these response styles are chosen. First, the level of acculturation among Hispanics affects the level of extreme and acquiescent responses so that as Hispanics acculturate they tend to choose these types of response less frequently. The less educated respondents (less than 12 years of formal education) tend to make more extreme choices than the more highly educated (those with at least a secondary education). Gender does not affect these response sets in a consistent fashion.
Article
Measuring the health state of individuals is important for the evaluation of health interventions, monitoring individual health progress, and as a critical step in measuring the health of populations. Self-report responses in household survey data are widely used for assessing the non-fatal health status of populations. These data typically take the form of ordered
Article
A multidimensional item response theory model that accounts for response style factors is presented. The model, a multidimensional extension of Bock's nominal response model, is shown to allow for the study and control of response style effects in ordered rating scale data so as to reduce bias in measurement of the intended trait. In the current application, the model is also used to investigate response style as an underlying cause of differential item functioning. The approach is illustrated using the item responses of cigarette smokers to the Wisconsin Inventory of Smoking Dependence Motives, a self-report measure of tobacco dependence.
Article
The tendency for some survey respondents to be unreasonably agreeable on attitude measures with an agree-disagree format is typically termed acquiescence, or acquiescent responding (AR). This paper suggests an alternative operational definition of AR plus a statistical test for identifying AR subjects. If positively and negatively phrased items are included in attitude scales to balance the effect of AR, then traditional methods of measuring AR often require computing the difference between the sum of responses to positive items and the sum of (reverse scored) responses to negative items. Large differences are an indication of more AR. Differential person functioning (DPF) can be determined, and AR can be defined as statistically significant DPF between positively and negatively worded item groups. The Theoretical Orientation Scale for Clinicians (TOSC), an inventory of principles of a new therapeutic approach known as solution-focused brief therapy, was completed by 284 counselors (175 usable inventories completed). Thirty-two subjects were identified as "yea sayers," and 8 were identified as "nay sayers." Removing these 40 DPF subjects yielded somewhat improved reliability and factor structure for the scale. The implications of removing DPF subjects from analysis are discussed. (Contains 2 figures and 19 references.) (SLD)
Article
Cognitive or intellectual investment theories propose that the development of intelligence is partially influenced by personality traits, in particular by so-called investment traits that determine when, where, and how people invest their time and effort in their intellect. This investment, in turn, is thought to contribute to individual differences in cognitive growth and the accumulation of knowledge across the life span. We reviewed the psychological literature and identified 34 trait constructs and corresponding scales that refer to intellectual investment. The dispositional constructs were further classified into 8 related trait categories that span the construct space of intellectual investment. Subsequently, we sought to estimate the association between the identified investment traits and indicators of adult intellect, including measures of crystallized intelligence, academic performance (e.g., grade point average), college entry tests, and acquired knowledge. A meta-analysis of 112 studies with 236 coefficients and an overall sample of 60,097 participants indicated that investment traits were mostly positively associated with adult intellect markers. Meta-analytic coefficients ranged considerably, from 0 to .58, with an average estimate of .30. We concluded that investment traits are overall positively related to adult intellect; the strength of investment-intellect associations differs across trait scales and markers of intellect; and investment traits have a diverse, multifaceted nature. The meta-analysis also identified areas of inquiry that are currently lacking in empirical research. Limitations, implications, and future directions are discussed. (PsycINFO Database Record (c) 2012 APA, all rights reserved).
Article
Mixed Rasch modelling allows the existence of sub-groups who differ in their use of questionnaire response scales to be investigated. This is relevant to personality measurement, as individual differences in scale use are a source of error in calculating trait scores. The objectives of the analysis reported here were to investigate the use of the mixed Rasch model on personality (NEO-FFI) data and to examine the issue of response set contributions to scale-level NEO correlations. Modelling led to two-class solutions for Neurot-icism, Agreeableness and Conscientiousness, which could be interpreted in terms of respondents showing a preference for the extremes or the middle of the scale. Correlations between all pairs of class assignments were positive and highly significant, demonstrating consistency in individual response preference across traits. Plots of item difficulty parameters suggested that both groups interpreted the items similarly. 'Extreme responders' had significantly higher scores on Extraversion and Conscientiousness, and females and younger people were more likely than males and older people to show a preference for extreme responding. For Extraversion and Openness, model fitting gave more complex results, but a two-class solu-tion was considered to also be the most appropriate for Extraversion. The results of this study confirm pre-vious findings on the existence of distinct response classes in self-report data and on personality correlates of individual differences in response scale use and also allow an estimate of response set effects on scale-level correlations.
Article
ABSTRACT We examined several determinants of interjudge agreement on personality traits. The findings, which were cross-validated in two samples, suggest that agreement is a function of four factors: which Big Five content domain the trait represents, how observable relevant behaviors are, how evaluative the trait is, and whether the self is one of the judges. Agreement was highest for traits related to Extraversion and lowest for traits related to Agreeableness. More observable and less evaluative traits elicited higher interjudge agreement. On average, self-peer agreement was lower than peer-peer agreement. However, this effect was limited to evaluative traits; for neutral traits, self-peer agreement was as high as peer-peer agreement. These findings suggest that self- and peer perception proceed through similar processes for neutral traits but not for highly evaluative traits, raising the possibility that self-perceptions become distorted when the trait is affectively charged.
Article
The California Child Q-set (CCQ) was used to explore the structure of personality in early adolescence and to develop scales to measure the “Big Five” dimensions: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience. Mothers provided Q-sorts of 350 ethnically diverse boys between 12 and 13 years old. Analyses of the construct validity of the scales provided a nomological network relating the Big Five to theoretically and socially important criterion variables, such as juvenile delinquency, Externalizing and Internalizing disorders of childhood psychopathology, school performance, IQ, SES, and race. These effects were obtained using diverse methods, including self-reports from the boys, ratings by their mothers and their teachers, and objective-test data. In addition to the Big Five, analyses also suggested 2 possibly age-specific dimensions of personality in early adolescence. Discussion is focused on the changing manifestations of personality traits throughout development.
Article
Social scientists often estimate models from correlational data, where the independent variable has not been exogenously manipulated; they also make implicit or explicit causal claims based on these models. When can these claims be made? We answer this question by first discussing design and estimation conditions under which model estimates can be interpreted, using the randomized experiment as the gold standard. We show how endogeneity – which includes omitted variables, omitted selection, simultaneity, common-method variance, and measurement error – renders estimates causally uninterpretable. Second, we present methods that allow researchers to test causal claims in situations where randomization is not possible or when causal interpretation could be confounded; these methods include fixed-effects panel, sample selection, instrumental variable, regression discontinuity, and difference-in-differences models. Third, we take stock of the methodological rigor with which causal claims are being made in a social sciences discipline by reviewing a representative sample of 110 articles on leadership published in the previous 10 years in top-tier journals. Our key finding is that researchers fail to address at least 66% and up to 90% of design and estimation conditions that make causal claims invalid. We conclude by offering 10 suggestions on how to improve non-experimental research.
Article
We address two long-standing survey research problems: measuring complicated concepts, such as political freedom and efficacy, that researchers define best with reference to examples; and what to do when respondents interpret identical questions in different ways. Scholars have long addressed these problems with approaches to reduce incomparability, such as writing more concrete questions—with uneven success. Our alternative is to measure directly response category incomparability and to correct for it. We measure incomparability via respondents’ assessments, on the same scale as the self-assessments to be corrected, of hypothetical individuals described in short vignettes. Because the actual (but not necessarily reported) levels of the vignettes are invariant over respondents, variability in vignette answers reveals incomparability. Our corrections require either simple recodes or a statistical model designed to save survey administration costs. With analysis, simulations, and cross-national surveys, we show how response incomparability can drastically mislead survey researchers and how our approach can alleviate this problem. Government Version of Record
Article
When respondents use the ordinal response categories of standard survey questions in different ways, the validity of analyses based on the resulting data can be biased. Anchoring vignettes is a survey design technique, introduced by King et al. (2004, Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review 94 [February]: 191–205), intended to correct for some of these problems. We develop new methods both for evaluating and choosing anchoring vignettes and for analyzing the resulting data. With surveys on a diverse range of topics in a range of countries, we illustrate how our proposed methods can improve the ability of anchoring vignettes to extract information from survey data, as well as saving in survey administration costs.
Article
The five-factor model of personality is a hierarchical organization of personality traits in terms of five basic dimensions: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience. Research using both natural language adjectives and theoretically based personality questionnaires supports the comprehensiveness of the model and its applicability across observers and cultures. This article summarizes the history of the model and its supporting evidence; discusses conceptions of the nature of the factors; and outlines an agenda for theorizing about the origins and operation of the factors. We argue that the model should prove useful both for individual assessment and for the elucidation of a number of topics of interest to personality psychologists.
Article
RECENT CRITICISMS OF RESPONSE STYLE RESEARCH HAVE IGNORED THE LITERATURE OF THE EXTREME RESPONSE STYLE (ERS). THESE CRITICISMS ARE QUESTIONED BY A SIZABLE LITERATURE SUGGESTING THE POTENTIAL USEFULNESS OF ERS AS AN INDICATOR OF CERTAIN PERSONALITY ATTRIBUTES. THERE IS CONSIDERABLE EVIDENCE FOR THE RELIABILITY OF ERS. STUDIES OF GROUP DIFFERENCES IN ERS TENDENCIES YIELD EVIDENCE FOR RELATIONSHIPS BETWEEN ERS AND PERSONALITY VARIABLES. CORRELATIONAL AND FACTOR-ANALYTIC STUDIES OFFER FURTHER SUPPORT, ALTHOUGH THE VARIABLES UNDERLYING ERS HAVE NOT BEEN CLEARLY IDENTIFIED. (2 P. REF.)
Article
The California Child Q-set (CCQ) was used to explore the structure of personality in early adolescence and to develop scales to measure the "Big Five" dimensions: Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience. Mothers provided Q-sorts of 350 ethnically diverse boys between 12 and 13 years old. Analyses of the construct validity of the scales provided a nomological network relating the Big Five to theoretically and socially important criterion variables, such as juvenile delinquency, Externalizing and Internalizing disorders of childhood psychopathology, school performance, IQ, SES, and race. These effects were obtained using diverse methods, including self-reports from the boys, ratings by their mothers and their teachers, and objective-test data. In addition to the Big Five, analyses also suggested 2 possibly age-specific dimensions of personality in early adolescence. Discussion is focused on the changing manifestations of personality traits throughout development.
Article
An important purpose in pooling time-series and cross-section data is to control for individual-specific unobservable effects which may be correlated with other explanatory variables, e.g. latent ability in measuring returns to schooling in earnings equations or managerial ability in measuring returns to scale in firm cost functions. Using instrumental variables and the time-invariant characteristics of the latent variable, we derive: 1. (1) a test for the presence of this effect and for the over-identifying restriction we use;2. (2) necessary and sufficient conditions for identification of all the parameters in the model; and3. (3) the asymptotically efficient instrumental variables estimator and conditions under which it differs from the within-groups estimator. We calculate efficient estimates of a wage equation from the Michigan income dynamics data which indicate substantial differences from within-groups and Balestra-Nerlove estimates — particularly a significantly higher estimate of the returns to schooling.
Social and emotional development and school learning: a measurement proposal in support of public policy Technical report for Organization for Economic Cooperation and Development (OCDE) Rio de Janeiro State Education Department (SEEDUC) and Ayrton Senna Institute
  • D Santos
  • R Primi
Santos, D., & Primi, R. (2014). Social and emotional development and school learning: a measurement proposal in support of public policy Technical report for Organization for Economic Cooperation and Development (OCDE) Rio de Janeiro State Education Department (SEEDUC) and Ayrton Senna Institute. São Paulo, Brazil: Ayrton Senna Institute.
Innovative questionnaire assessment methods to increase cross-country comparability Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis Boca Raton
  • P C Kyllonen
  • J P Bertling
Kyllonen, P. C., & Bertling, J. P. (2014a). Innovative questionnaire assessment methods to increase cross-country comparability. In L Rutkowski, M. Von Davier, & D. Rutkowski (Eds.), Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp. 277–286). Boca Raton: Taylor & Francis.
psych: Procedures for personality and psychological research
  • W Revelle
Revelle, W. (2014). psych: Procedures for personality and psychological research. Evanston, IL, USA: Northwestern University.
Acquiescence as a factor in the measurement of personality
  • T F Lentz
Lentz, T. F. (1938). Acquiescence as a factor in the measurement of personality. Psychological Bulletin, 35, 659.
The development of a nationwide inventory assessing social and emotional skills in Brazilian youth R: A language and environment for statistical computing
  • R Primi
  • D Santos
  • O P John
  • F De Fruyt
Primi, R., Santos, D., John, O. P., & De Fruyt, F. (2016). The development of a nationwide inventory assessing social and emotional skills in Brazilian youth. European Journal of Psychological Assessment, 32, 5–16. doi: 10.1027/1015- 5759/a000343 R Development Core Team. (2014). R: A language and environment for statistical computing, Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http:// www.R-project.org.
Employability in the 21 st century: Complex (interactive) problem solving and other essential skills. Industrial and Organizational Psychology: Perspectives on Science and Practice
  • De Fruyt
  • F Wille
  • B John
De Fruyt, F., Wille, B., & John, O. P. (2015). Employability in the 21 st century: Complex (interactive) problem solving and other essential skills. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 276–281.
Statistical models for enhancing cross-population comparability In Health systems performance assessment: Debates, methods and empiricism Geneva: World Health Organization. Retrieved from http 21 st century skills: Learning for life in our times
  • A Tandon
  • C J Murray
  • J A Salomon
  • G B King
  • C Fadel
Tandon, A., Murray, C. J., Salomon, J. A., & King, G. (2003). Statistical models for enhancing cross-population comparability. In Health systems performance assessment: Debates, methods and empiricism (pp. 727–746). Geneva: World Health Organization. Retrieved from http://www.who.int/ healthinfo/paper42.pdf Trilling, B., & Fadel, C. (2009). 21 st century skills: Learning for life in our times. San Francisco, CA: Jossey-Bass.
21 st century skills and competences for New Millennium Learners in OECD countries
  • K Ananiadou
  • M Claro
Ananiadou, K., & Claro, M. (2009). 21 st century skills and competences for New Millennium Learners in OECD countries. Paris, France: Centre for Educational Research and Innovation (CERI) -New Millennium Learners.
Response styles and personality traits: A multilevel analysis
  • J He
  • D Bartram
  • I Inceoglu
  • Van De
  • F J R Vijver
He, J., Bartram, D., Inceoglu, I., & van de Vijver, F. J. R. (2014). Response styles and personality traits: A multilevel analysis. Journal of Cross-Cultural Psychology, 45, 1028-1045. doi: 10.1177/0022022114534773
Acquiescence as differential person functioning. Assessment & Evaluation in Higher Education
  • G A Johanson
  • C J Osborn
Johanson, G. A., & Osborn, C. J. (2004). Acquiescence as differential person functioning. Assessment & Evaluation in Higher Education, 29, 535-548. doi: 10.1080/ 02602930410001689126