ArticlePDF Available

Predicting Survey Responses: How and Why Semantics Shape Survey Statistics on Organizational Behaviour

PLOS
PLOS One
Authors:
Jan Ketil Arnulf at BI Norwegian Business School
Jan Ketil Arnulf
Kai R. Larsen at University of Colorado Boulder
Kai R. Larsen
Øyvind L. Martinsen at BI Norwegian Business School
Øyvind L. Martinsen
Chih How Bong at University of Malaysia, Sarawak
Chih How Bong
Download full-text PDFDownload full-text PDF
Read full-text
Download citation

Abstract and Figures

Some disciplines in the social sciences rely heavily on collecting survey responses to detect empirical relationships among variables. We explored whether these relationships were a priori predictable from the semantic properties of the survey items, using language processing algorithms which are now available as new research methods. Language processing algorithms were used to calculate the semantic similarity among all items in state-of-the-art surveys from Organisational Behaviour research. These surveys covered areas such as transformational leadership, work motivation and work outcomes. This information was used to explain and predict the response patterns from real subjects. Semantic algorithms explained 60–86% of the variance in the response patterns and allowed remarkably precise prediction of survey responses from humans, except in a personality test. Even the relationships between independent and their purported dependent variables were accurately predicted. This raises concern about the empirical nature of data collected through some surveys if results are already given a priori through the way subjects are being asked. Survey response patterns seem heavily determined by semantics. Language algorithms may suggest these prior to administering a survey. This study suggests that semantic algorithms are becoming new tools for the social sciences, opening perspectives on survey responses that prevalent psychometric theory cannot explain.
Figures - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Arnulf Larsen Martinsen Bong PLOS ONE 20
14.pdf
Content uploaded by Kai R. Larsen
Author content
All content in this area was uploaded by Kai R. Larsen on Sep 04, 2014
Content may be subject to copyright.
Predicting Survey Responses: How and Why Semantics Shape Survey Statistics on Organizational Behavio
ur.pdf
Available via license: CC BY 4.0
Content may be subject to copyright.
A preview of the PDF is not available
... Semantic similarity between questionnaire statements has been recognized as a significant source of covariance [4], [5]. Statements are semantically similar when they have similar meanings, even if they use different words or sentence structures. ...
... Arnulf et al. [4] proposed the semantic theory of survey responding (STSR) which essentially states that a substantial part of the covariance between the questionnaire items can be explained by their semantic similarity. This means that the factor structure and the relationships between the factors found in human response data may not only reflect the implied constructs (e.g., attitudes, beliefs or behaviours), but they partly rise from the way how humans use language to represent concepts [16]. ...
... Previous studies on semantic similarity and survey responding have focused on the semantic similarity of the questionnaire statements. Arnulf et al. [4] showed that in the Multifactor Leadership Questionnaire the semantic similarity of the questionnaire statements replicated the intended factor structure and explained up to 86% of the covariance between the factors in the human data. Gefen et al. [17] showed that lexical closeness could replicate the measurement model of the Technology Acceptance Model (TAM) [18] using two independent semantic spaces derived from business journal articles and news articles. ...
Revealing the Influence of Semantic Similarity on Survey Responses: A Synthetic Data Generation Approach
Article
Full-text available
  • Jan 2025
Questionnaires are essential for measuring self-reported attitudes, beliefs, and behaviour in many research fields. Semantic similarity of the questions is recognized as a source of covariance in the human data, implying that response patterns partly arise from the questionnaire itself. A practical method to assess the influence of semantic similarity could significantly facilitate the design of questionnaires and the interpretation of their results. The current study presents a novel method for estimating the influence of semantic similarity for questionnaires with Likert-scale responses. The method represents responses as natural language sentences combining the statement and the response option and uses the Sentence-BERT algorithm to estimate a semantic similarity matrix between them. Synthetic response data are generated using the semantic similarity matrix and a noise parameter as input. Synthetic data can then be analysed using the same tools as human survey data, making the comparison straightforward. The method was tested with a questionnaire measuring the acceptance of automated driving. Synthetic data explained 40correlations in the human response data. This means that semantic similarity substantially influenced responses. Using synthetic data, it was possible to identify the same factor structure as in the human data and to identify relationships between factors that might have been inflated by semantic similarity. Semantically generated synthetic data could help in designing multi-factor questionnaires and correctly interpreting the found relationships between factors.
View
... Previous work has shown that natural language processing methods can be used to help clarify the relations between constructs and their measures (for example, refs. 11,14,15). Crucially, some approaches have been shown to identify jingle-jangle fallacies in an automated fashion across a large swathe of constructs 11 . However, these previous efforts have used techniques, such as latent semantic analysis, that have now been superseded by more powerful language models that promise to capture several aspects of human psychology even better (for example, refs. ...
... 'security' (17,16.3%) and 'anger' (15,14.4%). The majority of candidate jangle fallacies can be explained at least partially by shared items. ...
Semantic embeddings reveal and address taxonomic incommensurability in psychological measurement
Article
Full-text available
  • Mar 2025
  • Nat. Hum. Behav.
Taxonomic incommensurability denotes the difficulty in comparing scientific theories due to different uses of concepts and operationalizations. To tackle this problem in psychology, here we use language models to obtain semantic embeddings representing psychometric items, scales and construct labels in a vector space. This approach allows us to analyse different datasets (for example, the International Personality Item Pool) spanning thousands of items and hundreds of scales and constructs and show that embeddings can be used to predict empirical relations between measures, automatically detect taxonomic fallacies and suggest more parsimonious taxonomies. These findings suggest that semantic embeddings constitute a powerful tool for tackling taxonomic incommensurability in the psychological sciences.
View
... The ability of such novel data sources to complement traditional data collection techniques such as household surveys and focus groups is clear [33]. The data is collected passively without the need for costly and potentially dangerous active data collection, which also avoids inaccuracies due to human error, bias [3] or dishonesty. *Corresponding author: desislava.hristova@cl.cam.ac.uk ...
The International Postal Network and Other Global Flows As Proxies for National Wellbeing
Preprint
  • Jan 2016
The digital exhaust left by flows of physical and digital commodities provides a rich measure of the nature, strength and significance of relationships between countries in the global network. With this work, we examine how these traces and the network structure can reveal the socioeconomic profile of different countries. We take into account multiple international networks of physical and digital flows, including the previously unexplored international postal network. By measuring the position of each country in the Trade, Postal, Migration, International Flights, IP and Digital Communications networks, we are able to build proxies for a number of crucial socioeconomic indicators such as GDP per capita and the Human Development Index ranking along with twelve other indicators used as benchmarks of national wellbeing by the United Nations and other international organisations. In this context, we have also proposed and evaluated a global connectivity degree measure applying multiplex theory across the six networks that accounts for the strength of relationships between countries. We conclude with a multiplex community analysis of the global flow networks, showing how countries with shared community membership over multiple networks have similar socioeconomic profiles. Combining multiple flow data sources into global multiplex networks can help understand the forces which drive economic activity on a global level. Such an ability to infer proxy indicators in a context of incomplete information is extremely timely in light of recent discussions on measurement of indicators relevant to the Sustainable Development Goals.
View
... Self-report questionnaires can have favorable factor structures and correlational patterns even if they were deliberately constructed to measure a nonexistent concept (Maul, 2017). On the contrary, empirical evidence indicates that classically constructed questionnaires are extremely noisy since response patterns can be predicted almost entirely from the semantic characteristics of the items alone (Arnulf et al., 2014). This suggests that questionnaire scores depend only little on interindividual variations in the latent variable, but heavily on how items were phrased in the first place. ...
A Computational Method to Reveal Psychological Constructs From Text Data
Article
Full-text available
  • Sep 2024
When starting to formalize psychological constructs, researchers traditionally rely on two distinct approaches: the quantitative approach, which defines constructs as part of a testable theory based on prior research and domain knowledge often deploying self-report questionnaires, or the qualitative approach, which gathers data mostly in the form of text and bases construct definitions on exploratory analyses. Quantitative research might lead to an incomplete understanding of the construct, while qualitative research is limited due to challenges in the systematic data processing, especially at large scale. We present a new computational method that combines the comprehensiveness of qualitative research and the scalability of quantitative analyses to define psychological constructs from semistructured text data. Based on structured questions, participants are prompted to generate sentences reflecting instances of the construct of interest. We apply computational methods to calculate embeddings as numerical representations of the sentences, which we then run through a clustering algorithm to arrive at groupings of sentences as psychologically relevant classes. The method includes steps for the measurement and correction of bias introduced by the data generation, and the assessment of cluster validity according to human judgment. We demonstrate the applicability of our method on an example from emotion regulation. Based on short descriptions of emotion regulation attempts collected through an open-ended situational judgment test, we use our method to derive classes of emotion regulation strategies. Our approach shows how machine learning and psychology can be combined to provide new perspectives on the conceptualization of psychological processes.
View
... Such differences in semantics are relevant, because they can influence survey outcomes. For instance, language processing algorithms reveal that a substantial part of the variation in survey outcomes in organisational behaviour research may stem from semantic differences, and not from fundamental differences in the latent variable 36 . This might mean that regions differ in their SCI values not because of fundamental differences in corruption levels, but rather as a result of differences in formulation of the questions. ...
The Subnational Corruption Database: Grand and petty corruption in 1,473 regions of 178 countries, 1995–2022
Article
Full-text available
  • Jun 2024
This data descriptor presents the Subnational Corruption Database (SCD), which provides data on corruption in 1,473 subnational areas of 178 countries. The SCD includes a comprehensive overall corruption index, the Subnational Corruption Index (SCI), and its two components: the Subnational Grand Corruption Index (SGCI) and Subnational Petty Corruption Index (SPCI). The SCD is constructed by combining data of 807 surveys held in the period 1995–2022 and includes the corruption experiences and perceptions of 1,326,656 respondents along 19 separate dimensions. The data are available for multiple years, allowing longitudinal analyses. At the national level, the SCI correlates strongly with established corruption indices, like the Transparency International Corruption Perceptions Index (CPI) and the World Bank Control of Corruption Index (CCI). We create subnational estimates of the CPI and CCI by superimposing the subnational variation of the SCI around the national averages of these indices. The presentation of subnational data in the SCD and the separation between grand and petty corruption significantly broaden the global knowledge base in the field of corruption.
View
... The quantitative computational linguistic analysis of text data is a new measurement method available to science and practice. Semantic analysis can predict and understand psychological constructs from textual data (Arnulf et al., 2014), with open-ended questions demonstrating equal or higher validity and reliability compared to closed-ended numerical rating scales (Kjell et al., 2019). Grunenberg et al. (2024) outperformed traditional recruiter judgements in assessing applicants' Big Five personality traits based on CVs and short text responses. ...
View
View
Semantic meaning means a lot: Exploring the role of semantics in the development of a Big Five taxonomy
Article
  • Apr 2025
Developing a Big Five adjective taxonomy in Brazilian Portuguese, we explored the effects of linguistic properties in our classification processes. The first two studies implement top-down (expert ratings) and bottom-up (self-ratings from a community sample; N = 500) strategies for taxonomy classification and validation. We identified a clear five-factor structure with 171 adjectives supporting the Big Five. Study 3 correlated frequency of use and the semantic dimensions of valence, arousal, and dominance to Big Five measures for each adjective. We found weak effects of frequency, but systematic effects of semantic dimensions with expert ratings and component loadings, that were congruent with differences and overlaps between the five traits. We discuss the potential role of linguistic effects on personality structure and assessment.
View
Maskiner som snakker med ledere
Chapter
  • Aug 2024
I dette kapitlet beskrives kommunikasjon i skjæringspunktet mellom biologisk og kunstig intelligens med vekt på organisasjoner. Hva betyr det at maskiner snakker, og hva skjer når mennesker snakker med maskiner? Kapitlet ser på hva digitale språkmodeller kan fortelle oss om menneskets kommunikasjonsevner, og hvordan teknologien bak dem ligger til grunn for vår forståelse av kommunikasjon i organisasjoner. Denne teknologien har preget vårt forhold til kommunikasjon i 70 år til nå, og vil helt sikkert endre vårt syn på kommunikasjon i de nærmeste årene. Et hovedpoeng i kapitlet er at mennesket alltid har bygget teknologi inn i sin egen mentale fungering. Språk-roboter er best forstått som mentale proteser som kan gjøre oss mer intelligente, men bare hvis vi tar oss bryderiet med å forstå både kunstig og biologisk intelligens.
View
Kartlegging av foreldrestil: et spørreskjema for empatibasert, følelsesbevisst foreldrekompetanse
Article
Full-text available
  • Mar 2024
The present study describes the construction of a new inventory that measures parenting style along five independent dimensions: 1) empathic communication; 2) authoritarian leadership; 3) valuation/compassion; 4) avoidant leadership and 5) mentalization. Existing models and inventories do not capture the same breadth and complexity of parenting skills as the present form. This inventory could potentially provide a more nuanced picture of the individual’s parenting style and a more accurate indication of relevant areas of development. The results from our exploratory and confirmatory factor analysis with two different parent samples (n = 592 individual parents) revealed that the five-factor structure demonstrated a significant model fit of the five parental dimensions. The results of the project so far provide a good foundation for further standardisation and investigation of reliability and validity. Keywords: inventory, emotion-focused, parenting style, factor analysis
View
A meta-analysis of transformational and transactional leadership correlates of effectiveness and satisfaction: An update and extension
Research
Full-text available
  • Aug 2015
Update and extension of the 1996 meta-analysis of the transformational leadership literature by Lowe, Kroeck and Sivasubramaniam (1996) published in The Leadership Quarterly.
View
Cognition and Action
Chapter
  • Oct 2008
In the last decade, there has been a tremendous surge of research on the mechanisms of human action. This volume brings together this new knowledge in a single, concise source, covering most if not all of the basic questions regarding human action: what are the mechanisms by which action plans are acquired, mentally represented, activated, selected, and expressed? The chapters provide up-to-date summaries of the published research on this question, with an emphasis on underlying mechanisms. This ‘bible’ of action research brings together the current thinking of eminent researchers in the domains of motor control, behavioural and cognitive neuroscience, psycholinguistics, biology, as well as cognitive, developmental, social, and motivational psychology. It represents a determined multidisciplinary effort, spanning across various areas of science as well as national boundaries.
View
View
View
View
View
View
View
View
Self-Generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior
Article
  • Aug 1988
  • J PERS SOC PSYCHOL
Drawing from recent developments in social cognition, cognitive psychology, and behavioral decision theory, we analyzed when and how the act of measuring beliefs, attitudes, intentions, and behaviors affects observed correlations among them. Belief, attitude, or intention can be created by measurement if the measured constructs do not already exist in long-term memory. The responses thus created can have directive effects on answers to other questions that follow in the survey. But even when counterparts to the beliefs, attitudes, and intentions measured already exist in memory, the structure of the survey researcher's questionnaire can affect observed correlations among them. The respondent may use retrieved answers to earlier survey questions as inputs to response generation to later questions. We present a simple theory predicting that an earlier response will be used as a basis for another, subsequent response if the former is accessible and if it is perceived to be more diagnostic than other accessible inputs. We outline the factors that determine both the perceived diagnosticity of a potential input, the likelihood that it will be retrieved, and the likelihood that some alternative (and potentially more diagnostic) inputs will be retrieved.
View