Predicting Survey Responses: How and Why Semantics Shape Survey Statistics on Organizational Behaviour

Article (PDF Available)inPLoS ONE 9(9):1-13 · September 2014with 299 Reads
DOI: 10.1371/journal.pone.0106361
Cite this publication
Abstract
Some disciplines in the social sciences rely heavily on collecting survey responses to detect empirical relationships among variables. We explored whether these relationships were a priori predictable from the semantic properties of the survey items, using language processing algorithms which are now available as new research methods. Language processing algorithms were used to calculate the semantic similarity among all items in state-of-the-art surveys from Organisational Behaviour research. These surveys covered areas such as transformational leadership, work motivation and work outcomes. This information was used to explain and predict the response patterns from real subjects. Semantic algorithms explained 60–86% of the variance in the response patterns and allowed remarkably precise prediction of survey responses from humans, except in a personality test. Even the relationships between independent and their purported dependent variables were accurately predicted. This raises concern about the empirical nature of data collected through some surveys if results are already given a priori through the way subjects are being asked. Survey response patterns seem heavily determined by semantics. Language algorithms may suggest these prior to administering a survey. This study suggests that semantic algorithms are becoming new tools for the social sciences, opening perspectives on survey responses that prevalent psychometric theory cannot explain.
A preview of the PDF is not available
  • Article
    Likert-scale surveys are frequently used in cross-cultural studies on leadership. Recent publications using digital text algorithms raise doubt about the source of variation in statistics from such studies to the extent that they are semantically driven. The Semantic Theory of Survey Response (STSR) predicts that in the case of semantically determined answers, the response patterns may also be predictable across languages. The Multifactor Leadership Questionnaire (MLQ) was applied to 11 different ethnic samples in English, Norwegian, German, Urdu and Chinese. Semantic algorithms predicted responses significantly across all conditions, although to varying degree. Comparisons of Norwegian, German, Urdu and Chinese samples in native versus English language versions suggest that observed differences are not culturally dependent but caused by different translations and understanding. The maximum variance attributable to culture was a 5% unique overlap of variation in the two Chinese samples. These findings question the capability of traditional surveys to detect cultural differences. It also indicates that cross-cultural leadership research may risk lack of practical relevance.
  • Article
    Full-text available
    Research on sensemaking in organisations and on linguistic relativity suggests that speakers of the same language may use this language in different ways to construct social realities at work. We apply a semantic theory of survey response (STSR) to explore such differences in quantitative survey research. Using text analysis algorithms, we have studied how language from three media domains–the business press, PR Newswire and general newspapers–has differential explanatory value for analysing survey responses in leadership research. We projected well-known surveys measuring leadership, motivation and outcomes into large text samples from these three media domains significantly different impacts on survey responses. Business press language was best in explaining leadership-related items, PR language best at explaining organizational results and “ordinary” newspaper language seemed to explain the relationship among motivation items. These findings shed light on how different public arenas construct organizational realities in different ways, and how these differences have consequences on methodology in research on leadership.
  • Article
    Full-text available
    In the context of public service delivery, public professionals nowadays intensively collaborate with citizens. The joint, sometimes mandatorily, efforts of citizens and professionals to provide public services have become known as “coproduction.” Although coproduction directly affects professionals’ work environment, professionals’ attitudes toward coproduction are hardly studied. This article explains variation in professionals’ engagement in coproduction from characteristics of their work environment, specifically their perceived level of autonomy, perceived organizational support for coproduction, and perceived red tape associated with coproduction activities. Survey research was conducted to question managers of Dutch organizations for elderly care about their interaction with client councils, an example of coproduction activities in the domain of health care. The results show that perceived autonomy in coproduction, red tape associated with coproduction, and organizational support affect professionals’ engagement. Organizational support moreover reinforces the effect of work autonomy on professionals’ perception on the importance of coproduction. These findings add to the study of coproduction and can help support public organizations to improve coproduction.
  • Article
    Job resources are essential for superior performance. Focusing on innovation, we examine how and for whom the job resource of time control is related to the implementation of novel ideas in the workplace. Drawing on the job demands‐resources model and affect‐as‐information theory, we propose that positive affect explains how the association of time control with innovation unfolds and how this psychological process depends on the extent to which employees work under problem‐solving demands. Using a survey study, conducted with 198 employees from diverse organizations, we supported a mediation mechanism in which time control was positively associated with positive affect, which is turn was positively related to innovation. Furthermore, the strength of this mediation process depended on problem‐solving demands, such that positive affect and innovation resulting from time control was stronger for those working in complex environments. These results corroborate and expand knowledge on the job demands‐resources, affect and innovation literatures, showing that job resources matter for performance, particularly in conjunction with job complexity denoting challenging performance. Thus, organizations should be aware of opportunities to manage the degree of job control and complexity if they aim to foster employee innovation. This article is protected by copyright. All rights reserved.
  • Article
    Meaning depends on context. This applies in obvious cases like deictics or sarcasm as well as more subtle situations like framing or persuasion. One key aspect of this is the identity of the participants in an interaction. Our interpretation of an utterance shifts based on a variety of factors, including personal history, background knowledge, and our relationship to the source. While obviously an incomplete model of individual differences, demographic factors provide a useful starting point and allow us to capture some of this variance. However, the relevance of specific demographic factors varies between situations—where age might be the key factor in one context, ideology might dominate in another. To address this challenge, we introduce a method for combining demographics and context into situated demographic embeddings—mapping representations into a continuous geometric space appropriate for the given domain, showing the resulting representations to be functional and interpretable. We further demonstrate how to make use of related external data so as to apply this approach in low‐resource situations. Finally, we show how these representations can be incorporated into improve modeling of real‐world natural language understanding tasks, improving model performance and helping with issues of data sparsity.
  • Article
    Full-text available
    Over the past century, various value models have been proposed. To determine which value model best predicts prosocial behavior, mental health, and pro-environmental behavior, we subjected seven value models to a hierarchical regression analysis. A sample of University students (N = 271) completed the Portrait Value Questionnaire (Schwartz et al., 2012), the Basic Value Survey (Gouveia et al., 2008), and the Social Value Orientation scale (Van Lange et al., 1997). Additionally, they completed the Values Survey Module (Hofstede and Minkov, 2013), Inglehart’s (1977) materialism–postmaterialism items, the Study of Values, fourth edition (Allport et al., 1960; Kopelman et al., 2003), and the Rokeach (1973) Value Survey. However, because the reliability of the latter measures was low, only the PVQ-RR, the BVS, and the SVO where entered into our analysis. Our results provide empirical evidence that the PVQ-RR is the strongest predictor of all three outcome variables, explaining variance above and beyond the other two instruments in almost all cases. The BVS significantly predicted prosocial and pro-environmental behavior, while the SVO only explained variance in pro-environmental behavior.
  • Preprint
    Full-text available
    Many experts consider target leakage one of the most insidious problems of automated machine learning. In this book, the term "target leakage" (aka. data leakage) will be defined in a broader scope than usual, as this provides an opportunity to discuss related issues of importance. Our definition of target leakage considers a target leak to have occurred when the model performance metrics are better than what is possible in a real-world scenario due to the inclusion of illegitimate features in the model. This definition is expansive and goes beyond the conventional description, which focuses only on features that are not available at the time of prediction. It is worth noting that feature illegitimacy often depends on both the context of model usage and the reporting of model performance. The clearest example of target leakage comes from the use of the target itself to predict the target. Take, as an example of this, a party game: you claim to be a great prognosticator of people's ages. You require only one feature to do so. When you ask them how old they were on this date a year ago, no one is impressed that you can add one to their answer, and the impressiveness or validity of your prediction disappears. Here are seven types of target leak: 1. The model contains one or more features that are not available to the model at the time of prediction. For example, when building a model of whether a patient who is about to be released from the hospital is likely to be readmitted, the inclusion of their blood-sugar level during their next visit would be predictive. No such test results would be available for patients never again readmitted.
  • Article
    I am a critical post-Positivist ? and hold it as a badge of honor. To be a Positivist means that you are humbly willing to put your theory to a test that can falsify it ? that is, to show that it is wrong despite all that you have put into it. To be a post-Positivist means believing in that standard but also acknowledging the limitations of Positivism, including limitations of measurement and its theory-dependence as well as acknowledging that there are social processes in science too. A post- Positivist does not claim that other epistemologies are necessarily wrong. It is just that researchers who adhere to the falsification principle are willing to put their money where their claims are, rather than allowing in the temptation to endlessly massage theories so that they are constantly updated to be consistent with new facts. In that vein, Positivism, despite its limitations, can by virtue of the falsification principle put a dent in confirmation bias and the kind of heuristics suggested by Tversky and Kahneman that prejudice our seeing patterns even when they do not exist. It is in that spirit that this paper is written.
  • Article
    Full-text available
    Although initially studies examining human resource management (HRM)–outcome relationships only used management‐based ratings of HR practices, arguments have been advanced in favour of using employee‐based ratings. To examine this, a systematic analysis of HRM–outcome studies published between 2000 and 2017 is performed, which shows that over time studies have indeed increasingly made use of employees as respondents to measures of human resource (HR) practices. An in‐depth analysis of these measures of perceived HR practices revealed that various problems and issues can be identified when critically reviewing these measures. It is observed that considerable idiosyncrasy exists in measures of perceived HR practices, coupled with a lack of transparency in how these measures are often reported in existing studies. Also, a mixture of evaluative and descriptive items creates concerns about jingle fallacies in extant research and in turn about the validity of HRM–outcome results. Recommendations are provided to further advance the measurement and conceptualisation of this core construct.
  • Article
    While previous studies indicate that female consumers display less trust for online shopping than males do, there is little research to addresses the reasons behind this difference. Our study applies a combination of both self‐report and the Implicit Association Test (IAT) to compare the cognitive and affective components of attitudes in men and women toward online shopping in China. Although female participants showed no significant difference from male participants in affective attitudes toward online shopping in the self‐report condition, females associated online shopping more frequently with unpleasant adjectives and off‐line shopping with pleasant adjectives in the indirect IAT condition. The opposite pattern was found for the male group. This finding indicates a wanting but disliking attitude toward online shopping among the female consumers in China, which provides a unique theoretical contribution to consumer behavior theory and helps to enhance e‐marketers’ market targeting and segmentation effectiveness in China.
  • Article
    Drawing from recent developments in social cognition, cognitive psychology, and behavioral decision theory, we analyzed when and how the act of measuring beliefs, attitudes, intentions, and behaviors affects observed correlations among them. Belief, attitude, or intention can be created by measurement if the measured constructs do not already exist in long-term memory. The responses thus created can have directive effects on answers to other questions that follow in the survey. But even when counterparts to the beliefs, attitudes, and intentions measured already exist in memory, the structure of the survey researcher's questionnaire can affect observed correlations among them. The respondent may use retrieved answers to earlier survey questions as inputs to response generation to later questions. We present a simple theory predicting that an earlier response will be used as a basis for another, subsequent response if the former is accessible and if it is perceived to be more diagnostic than other accessible inputs. We outline the factors that determine both the perceived diagnosticity of a potential input, the likelihood that it will be retrieved, and the likelihood that some alternative (and potentially more diagnostic) inputs will be retrieved.
  • Research
    Full-text available
    Update and extension of the 1996 meta-analysis of the transformational leadership literature by Lowe, Kroeck and Sivasubramaniam (1996) published in The Leadership Quarterly.
  • Article
    In this article, we review the history of the social scientific study of leadership and the prevailing theories of leadership that enjoy empirical support. We demonstrate that the development of knowledge concerning leadership phenomena has been truly cumulative and that much is currently known about leadership. We identify the contributions of the trait, behavioral, contingency and neocharismatic paradigms and the results of empirical research on prevailing theories. Issues that warrant research in each of the paradigms and theories are described. Ten additional topics for further investigation are discussed and specific recommendations are made with regard to future research on each of these topics.
  • Book
    Ontology was once understood to be the philosophical inquiry into the structure of reality: the analysis and categorization of 'what there is'. Recently, however, a field called 'ontology' has become part of the rapidly growing research industry in information technology. The two fields have more in common than just their name. Theory and Applications of Ontology is a two-volume anthology that aims to further an informed discussion about the relationship between ontology in philosophy and ontology in information technology. It fills an important lacuna in cutting-edge research on ontology in both fields, supplying stage-setting overview articles on history and method, presenting directions of current research in either field, and highlighting areas of productive interdisciplinary contact. Theory and Applications of Ontology: Computer Applications presents ontology in ways that philosophers are not likely to find elsewhere. The volume offers an overview of current research in ontology, distinguishing basic conceptual issues, domain applications, general frameworks, and mathematical formalisms. It introduces the reader to current research on frameworks and applications in information technology in ways that are sure to invite reflection and constructive responses from ontologists in philosophy. © Springer Science+Business Media B.V. 2010. All rights reserved.
  • Article
    There is a widely shared consensus that charismatic–transformational leadership is a particularly effective form of leadership. In a critical assessment of the state-of-the-science in this area of research, we question the validity of that conclusion. We identify four problems with theory and research in charismatic–transformational leadership. First, a clear conceptual definition of charismatic–transformational leadership is lacking. Current theories advance multi-dimensional conceptualizations of charismatic–transformational leadership without specifying how these different dimensions combine to form charismatic–transformational leadership, or how dimensions are selected for inclusion or exclusion. Second, theories fail to sufficiently specify the causal model capturing how each dimension has a distinct influence on mediating processes and outcomes and how this is contingent on moderating influences. Third, conceptualization and operationalization confounds charismatic–transformational leadership with its effects. Fourth, the most frequently used measurement tools are invalid in that they fail to reproduce the dimensional structure specified by theory and fail to achieve empirical distinctiveness from other aspects of leadership. Given that these problems are fundamental and inherent in the approaches analyzed, it is recommended that current approaches be abandoned, and that the field forego the label of charismatic–transformational leadership in favor of the study of more clearly defined and empirically distinct aspects of leadership.
  • Article
    Full-text available
    Despite conceptual overlap between the transformational–transactional model of leadership and the Ohio State two-factor model (i.e., Consideration and Initiating Structure), no systematic research examines correspondence among these behaviors or estimates their relative validities across a common set of outcomes. The current studies a) examine the factor structure of five key dimensions of these two models (transformational, contingent reward, laissez faire, Initiating Structure, and Consideration) and b) estimate relative validities with respect to two organizational outcomes: employee job satisfaction and perceptions of leadership effectiveness. Although results of a meta-analysis show that transformational leadership is significantly related to both Consideration (ρ = .74) and Initiating Structure (ρ = .50), results of two primary studies provide support for the independence of these leadership dimensions. Moreover, dominance analyses (Budescu, 1993) reveal that Consideration and transformational leadership are the most important predictors of employee job satisfaction and ratings of leadership effectiveness, and each had incremental validity when controlling for the effects of the other. Overall, results suggest that dimensions from both models are important predictors of employee outcomes.