ArticlePDF Available

Abstract and Figures

Some disciplines in the social sciences rely heavily on collecting survey responses to detect empirical relationships among variables. We explored whether these relationships were a priori predictable from the semantic properties of the survey items, using language processing algorithms which are now available as new research methods. Language processing algorithms were used to calculate the semantic similarity among all items in state-of-the-art surveys from Organisational Behaviour research. These surveys covered areas such as transformational leadership, work motivation and work outcomes. This information was used to explain and predict the response patterns from real subjects. Semantic algorithms explained 60–86% of the variance in the response patterns and allowed remarkably precise prediction of survey responses from humans, except in a personality test. Even the relationships between independent and their purported dependent variables were accurately predicted. This raises concern about the empirical nature of data collected through some surveys if results are already given a priori through the way subjects are being asked. Survey response patterns seem heavily determined by semantics. Language algorithms may suggest these prior to administering a survey. This study suggests that semantic algorithms are becoming new tools for the social sciences, opening perspectives on survey responses that prevalent psychometric theory cannot explain.
Content may be subject to copyright.
A preview of the PDF is not available
... The approach depends on a bag-of-words representation where each paragraph's word order is abandoned and frequently used terms downweighed before the termdocument matrix is subjected to a singular value decomposition (SVD) as described in Larsen and Monarchi (2004). In general practice, 300-500 dimensions are retained (Arnulf et al., 2014). In the creation of this specific semantic space a 398-dimension space was created. ...
... Much work has gone into understanding how LSA works relative to the human mind, and Landauer (2007, p. 31) even argued that LSA "demonstrates a computational method by which a major component of language learning and use can be achieved." The applicability of LSA to partially replicate through text analysis survey responses by people seems to support this contention (e.g., Arnulf et al., 2014Arnulf et al., , 2018 (2007)], we use LSA to address a specific question in a way that is mathematically rigorous and that can be replicated by anyone with an understanding of statistical methods. ...
... Readers interested in this process are referred to one of many detailed descriptions, ranging from mathematical introductions (e.g., Larsen and Monarchi, 2004;Martin and Berry, 2007) to conceptual explanations (e.g. Evangelopoulos et al., 2012;Arnulf et al., 2014). ...
... We will start with showing how the semantic algorithms can find patterns that will be similar to all people, predicting 86% of the variation in respondent correlation matrices (Arnulf, Larsen, Martinsen, & Bong, 2014). We argue that this happens because the semantic relationships in language are not "out there" in the world but rather aspects of semantic processing by the Most speakers will adhere perfectly to the linguistic rule that a Thursday precedes a Friday, which in turn is succeeded by Saturday. ...
... Strong semantic similarities will create internal coherence in scales, emerging as high alpha reliabilities, while weaker but still notable semantic relationships will link the scales. The stronger semantic relationships will cluster as factors and the weaker relationships will appear as correlations between factors (Arnulf, Larsen, & Dysvik, 2018;Arnulf et al., 2014). To construct an example from the ones previously used: ...
... However, a semantic overlap between the core constructs being studied has been seen as unwanted autocorrelations yielding inflated statistics, and so techniques like rotated factor analysis have been applied to ensure the independence of the studied constructs (Abdi, 2003). Our studies show that factor analysis is not sufficient to prevent semantic influences from seeping into the statistics (Arnulf et al., 2014). ...
Chapter
This chapter reviews the person-situation dimension in behavior prediction through the semantic theory of survey responses (STSR). This theory proposes that the most likely source of variation in correlations between scores on Likert-scale items is overlap in meaning. We review and explain a growing number of empirical studies that support this: Up to 86% of the variation in correlation matrices may be explained using text algorithms. Also, semantics seem to predetermine the relationships between different scales, including those cast as “predictors” and “outcomes” of one another. The studies seek to establish semantic properties on population, group, and individual levels, showing that comparisons of score levels across groups are affected by predictable differences in their interpretation of items. The findings relativize the importance of data collected by semantically influenced surveys. On the bright side, they open new ways of matching individual and group level characteristics to the general population.
... Notably, climate change was explicitly mentioned in the description of the adaptation items, which could have increased the semantic similarity between these items and the climate change perception items. This can lead to artificially inflated correlations because people may respond similarly to items that both contain the word 'climate', which is also known as 'common method variance' (Arnulf et al. 2014). Similarly, filling out the items about climate change perceptions at the beginning of the questionnaire may have made climate change more salient. ...
... Overall, these findings are consistent with previous studies, which also generally found a positive relationship between climate change perceptions and policy support (Bateman and O'Connor 2016;Mildenberger et al. 2019). Expanding the previous findings, we demonstrated that these relationships hold even when climate change perceptions and policy support are measured one week apart, indicating that these findings are not likely to be the result of common method variance (Arnulf et al. 2014). More generally, our findings further confirm previous studies that have shown that climate change perceptions are associated with support for climate policies in general (see also Hornsey et al. 2016). ...
Article
Full-text available
People are increasingly exposed to climate-related hazards, including floods, droughts, and vector-borne diseases. A broad repertoire of adaptation actions is needed to adapt to these various hazards. It is therefore important to identify general psychological antecedents that motivate people to engage in many different adaptation actions, in response to different hazards, and in different contexts. We examined if people’s climate change perceptions act as such general antecedents. Questionnaire studies in the Netherlands (n = 3,546) and the UK (n = 803) revealed that the more people perceive climate change as real, human-caused, and having negative consequences, the more likely they are to support adaptation policy and to seek information about local climate impacts and ways to adapt. These relationships were stronger and more consistent when the information and policies were introduced as measures to adapt to risks of climate change specifically. However, the three types of climate change perceptions were inconsistently associated with intentions to implement adaptation behaviours (e.g. installing a green roof). This suggests that climate change perceptions can be an important gateway for adaptation actions, especially policy support and information seeking, but that it may be necessary to address additional barriers in order to fully harness the potential of climate change perceptions to promote widespread adaptation behaviour.
... It is therefore reasonable to believe that there will be persons in a healthy group that are over the clinical cutoff. Reasons for misclassification could be, for example, that people semantically interpret questions differently (Arnulf et al., 2014). ...
Article
The present study examined levels of emotional exhaustion, a key symptom of burnout, in Swedish professional and semiprofessional sport coaches in comparison to the normative values specified in the Maslach Burnout Inventory manual, and to the clinical cutoffs developed by Kleijweg, Verbraak, and Van Dijk. The sample contained 318 Swedish coaches ( M age = 42.7 years, 12% female) working at least 50% full time away from both team (60%) and individual (40%) sports. Our study shows that, in general, coaches in this sample experience lower average levels of exhaustion than normative samples both regarding the Maslach Burnout Inventory and clinical cutoffs. Two groups of coaches did, however, stand out. Coaches living in single households as well as coaches working part time had higher risk of severe levels of emotional exhaustion. These results place coach exhaustion levels in relation to other occupations and highlight that in this sample, the coaching profession does not stand out as more emotionally exhausting than other occupations.
... These limitations are mitigated to some extent by our use of multi-source data from supervisors and subordinates to represent dependent and independent variables, respectively, Frontiers in Psychology | www.frontiersin.org which ensured that the key relationships of interest were not affected by response set consistencies common to singlesource data (Arnulf et al., 2014). Moreover, the consideration of five dimensions of employability in this study provided a holistic view of potential age stereotyping, which is rare in the scholarly literature so far. ...
Article
Full-text available
Status incongruence resulting from a supervisor who is younger than their subordinate potentially leads to age stereotyping of employees. This article investigates the relationship between age difference and supervisory ratings of five competence-based measures of subordinate employability (Occupational Expertise, Anticipation/Optimisation, Personal Flexibility, Corporate Sense, and Balance). In addition, we consider the buffering role of a supportive learning context which allows older workers access to learning resources. Learning context is represented by duration of the supervisory relationship, perceived organizational learning climate and participation in, and application of, training and development. Using 295 dyads of employees and their direct supervisors in a Dutch building company, findings show that age dissimilarity reflecting status incongruence was related to lower supervisory ratings of Occupational Expertise (job-related competence) and Corporate Sense (social/organizational competence) regardless of learning context. Longer duration relationships exacerbated, rather than buffered, the age difference effect on some types of supervisory ratings. The implications of these findings for age stereotyping with regard to employability are considered.
... Indeed, experiment of Section 4 showed that word2vec data reflect implicit affective information allowing for extraction and analysis. As other methods of language semantics in psychology and behavioral science Arnulf (2020); Arnulf, Larsen, Martinsen, and Bong (2014); Hollis and Westbury (2016); Jackson et al. (2019), this method suggests valuable complement for traditional experimental approaches in terms of scope, statistical reliability, and scalability. ...
Preprint
Full-text available
The paper describes model of human affect based on quantum theory of semantics. The model considers emotion as subjective representation of behavioral context relative to a basis binary choice, organized by cyclical process structure and an orthogonal evaluation axis. The resulting spherical space, generalizing well-known circumplex models, accommodates basic emotions in specific angular domains. Predicted process-semantic structure of affect is observed in the word2vec data, as well as in the previously obtained spaces of emotion concepts. The established quantum-theoretic structure of affective space connects emotion science with quantum models of cognition and behavior, opening perspective for synergetic progress in these fields.
... Moreover, future research may use J o u r n a l P r e -p r o o f NATURAL LANGUAGE PROCESSING 9 semantic similarity as a guide in hypothesis generation. Empirical correlations between constructs are predicted, in part, by the degree of semantic overlap of scale measures [28,6]. ...
Article
Full-text available
Prosociality (measured with economic games) is correlated with individual differences in psychological constructs (measured with self-report scales). We review how methods from Natural Language Processing (NLP), a subfield of computer science focused on processing natural text, can be applied to understand the semantic content of scales measuring psychological constructs correlated with prosociality. Methods for clustering language and assessing similarity between text documents can be used to assess the novelty (or redundancy) of new scales; to understand the overlap among different psychological constructs; and to compare different measures of the same construct. These examples illustrate how NLP methods can augment traditional survey- and game-based approaches to studying individual differences in prosociality.
... Since then, researchers have expanded the use of semantic similarity of scale items to explore survey responses in a number of ways. Studies have shown that semantics may predict survey responses in organizational behavior (Arnulf et al., 2014(Arnulf et al., , 2018c, leadership Larsen, 2015, Arnulf et al., 2018b,d), employee engagement (Nimon et al., 2016), technology acceptance (Gefen and Larsen, 2017), and intrinsic motivation (Arnulf et al., 2018a). ...
Article
Full-text available
The methodological tools available for psychological and organizational assessment are rapidly advancing through natural language processing (NLP). Computerized analyses of texts are increasingly available as extensions of traditional psychometric approaches. The present Research Topic is recognizing the contributions but also the challenges in publishing such inter-disciplinary research. We therefore sought to provide an open-access avenue for cutting-edge research to introduce and illustrate the various applications of semantics in the assessment of attitudes and personality. The result is a collection of empirical contributions spanning from assessment of psychological states through methodological biases to construct identity detection.
... We are simply left to have faith that the researchers know what they are doing, and that reported "validity" correlations are not driven by these types of tautologies. Unfortunately, we see little reason to grant this trust to behavioral scientists at this point in time, given the frequency with which widely trumpeted associations linking scales to outcomes are later found to be driven by these types of uninteresting tautologies (Arnulf, Larsen, Martinsen, & Bong, 2014;Möttus, 2016;van Knippenberg & Sitkin, 2013; The result of all this is that, at the individual study level, we can rarely even be sure of what has been found. ...
Article
The study of emotional intelligence (EI) in the field of leadership, and in the organizational sciences in general, has often been characterized by controversy and criticism. But the study of EI has nonetheless persisted by developing new measures and models to address these concerns. In a prior letter exchange by Antonakis, Ashkanasy, and Dasborough (2009), two author teams debated the role of EI in the leadership literature, but also set an agenda for research and reconciliation for the future. The present exchange revisits these arguments using evidence accumulated over the past decade. Specifically, the authors debate not only the evidence for the predictive power of EI for workplace outcomes, but also the validity of EI as a construct, the measurement of EI, and the appropriateness of analytical tests for establishing the value of EI. Although the author teams agree on the value of the study of emotions and the need for rigorous research in this area, they nonetheless propose alternative agendas and priorities for the future. Further, they conclude that the issues identified in this exchange are not unique to the study of EI; but should also serve to inform the study of other personality factors and leadership more broadly.
Research
Full-text available
Update and extension of the 1996 meta-analysis of the transformational leadership literature by Lowe, Kroeck and Sivasubramaniam (1996) published in The Leadership Quarterly.
Article
Drawing from recent developments in social cognition, cognitive psychology, and behavioral decision theory, we analyzed when and how the act of measuring beliefs, attitudes, intentions, and behaviors affects observed correlations among them. Belief, attitude, or intention can be created by measurement if the measured constructs do not already exist in long-term memory. The responses thus created can have directive effects on answers to other questions that follow in the survey. But even when counterparts to the beliefs, attitudes, and intentions measured already exist in memory, the structure of the survey researcher's questionnaire can affect observed correlations among them. The respondent may use retrieved answers to earlier survey questions as inputs to response generation to later questions. We present a simple theory predicting that an earlier response will be used as a basis for another, subsequent response if the former is accessible and if it is perceived to be more diagnostic than other accessible inputs. We outline the factors that determine both the perceived diagnosticity of a potential input, the likelihood that it will be retrieved, and the likelihood that some alternative (and potentially more diagnostic) inputs will be retrieved.
Article
Drawing from recent developments in social cognition, cognitive psychology, and behavioral decision theory, we analyzed when and how the act of measuring beliefs, attitudes, intentions, and behaviors affects observed correlations among them. Belief, attitude, or intention can be created by measurement if the measured constructs do not already exist in long-term memory. The responses thus created can have directive effects on answers to other questions that follow in the survey. But even when counterparts to the beliefs, attitudes, and intentions measured already exist in memory, the structure of the survey researcher's questionnaire can affect observed correlations among them. The respondent may use retrieved answers to earlier survey questions as inputs to response generation to later questions. We present a simple theory predicting that an earlier response will be used as a basis for another, subsequent response if the former is accessible and if it is perceived to be more diagnostic than other accessible inputs. We outline the factors that determine both the perceived diagnosticity of a potential input, the likelihood that it will be retrieved, and the likelihood that some alternative (and potentially more diagnostic) inputs will be retrieved. This article examines the effects of measurement operations on revealed correlations among survey measures of belief, attitude, intention, and behavior. The potential reactivity of measurement has long been of concern in psychology. Methodologists (e.g., Campbell & Stanley, 1966; Cook & Campbell, 1979; Runkel & McGrath, 1972) warn of measurement-induced distortions relating to social desirability, evaluation apprehension, and sensitization to experimental treatments. Although it is true that the problem of reactivity of measurement affects work in both the social and physical sciences, in the physical sciences, measurement effects are expressed in terms of substantive theory. For instance, Heisenberg's uncertainty principle links basic
Article
In this article, we review the history of the social scientific study of leadership and the prevailing theories of leadership that enjoy empirical support. We demonstrate that the development of knowledge concerning leadership phenomena has been truly cumulative and that much is currently known about leadership. We identify the contributions of the trait, behavioral, contingency and neocharismatic paradigms and the results of empirical research on prevailing theories. Issues that warrant research in each of the paradigms and theories are described. Ten additional topics for further investigation are discussed and specific recommendations are made with regard to future research on each of these topics.