Conference PaperPDF Available

Neurophysiological Correlates of Information Systems Commonly Used Self-Reported Measures: A Multitrait Multimethod Study

Authors:

Abstract

Given the importance and criticality of instrument validation in IS research, the objective of this study is to provide a systematic assessment of IS construct validity via a multitrait multimethod (MTMM). In doing so, this paper uses structurally different methods -- neurophysiological and self-reported scales - to measure three commonly used IS constructs: engagement, arousal and cognitive load. The study's results generally support MTMM expectations and shed light on the complexity of detecting the nature of mono-method bias. More specifically, the study's results show that primitive perceptual IS constructs such as arousal are unlikely to suffer from mono-method bias, whereas more complex perceptual constructs such as engagement or cognitive load have higher within method correlations. There are two alternative explanations for the within method correlations: a) a method bias, or b) a combination between trait and method.
Gmunden Retreat on NeuroIS 2012
©
NeuroIS.org
21
Neurophysiological Correlates of Information Systems Commonly Used Self-
Reported Measures: A Multitrait Multimethod Study
Ana Ortiz de Guinea, Ryad Titah, Pierre-Majorique Léger
Corresponding author: pierre-majorique.leger@hec.ca
Given the importance and criticality of instrument
validation in IS research, the main objective of this
study is to provide a systematic assessment of IS
construct validity via a multitrait multimethod
(MTMM). To do so, the paper uses structurally
different methods neurophysiological and self-
reported scales - to measure three commonly used IS
constructs: engagement, arousal and cognitive load in
two different experimental settings. The experiments
involved seventeen (17) and twenty four (24)
participants respectively and consisted in using
different IS to execute a set of both instrumental and
hedonic tasks. The neurophysiological measures were
taken while participants performed their tasks, while
the psychometric measures were taken immediately
after the tasks were completed. The study’s results
generally support MTMM expectations and shed light
on the complexity of detecting the nature of mono-
method bias. More specifically, the study’s results
show that primitive perceptual IS constructs such as
arousal seem to be less affected by mono-method bias,
whereas more complex perceptual constructs such as
engagement or cognitive load have higher within
method correlations. There are two alternative
explanations for the within method correlations: a) a
method bias, or b) a combination between trait and
method.
Finally, the study’s results prompt a series of
potentially important questions for our field and for
the scientific research process in general: What are the
implications if no correlation is found between
neurophysiological and self-reported measures? Are
actual methods for detecting and/or controlling for
mono-method bias reliable? Are we under or over
estimating the impact of mono-method bias on
research results?
R
EFERENCES
Bagozzi, R. P. 2011. “Measurement and meaning in Information
Systems and organizational research: Methodological and
philosophical foundations,” MIS Quarterly, (35:2), pp. 261-292.
Brannick, M. T., Chan, D., Conway, J. M., Lance, C. E. and Spector,
P. E. 2010. “What Is Method Variance and How Can We Cope With
It? A Panel Discussion,” Organizational Research Methods, (13:3),
pp. 407-420.
Burton-Jones, A. 2009. “Minimizing Method Bias through
Programmatic Research,” MIS Quarterly, (33:3), pp. 445-471.
Cameron, A. F. 2007 “Juggling Multiple Conversations with
Communication Technology: Towards a Theory of Multi-
Communicating Impacts in the Workplace,” Queen's University,
Kingston.
Campbell, D. T. and Fiske, D. W. 1959. “Convergent and
Discriminant Validation by the Multitrait-Multimethod Matrix,”
Psychological Bulletin, (56:2), pp. 81-105.
Churchill, G. A. 1986. “A Paradigm for Developing Better Measures,”
Journal of Marketing Research, XVI, pp.64-73.
Dimoka, A. 2010 “What Does the Brain Tell Us About Trust and
Distrust? Evidence from a Functional Neuroimaging Study,” MIS
Quarterly, (34:2), pp. 373-396.
Dimoka, A., Banker, R. D., Benbasat, I., Davis, F. D., Dennis, A. R.,
Gefen, D., Gupta, A., Ischebeck, A., Kenning, P., Pavlou, P. A.,
Müller-Putz, G., Riedl, R., vom Brocke, J. and Weber, B. “On the use
of neurophysiological tools in IS research: Developing a research
agenda for NeuroIS, MIS Quarterly forthcoming.
Doty, D. H. and Glick, W. H. 1998. Common Method Bias: Does
Common Methods Variance Really Bias Results?” Organizational
Research Methods, (1:4), pp. 374-406.
Fiske, D. W. and Campbell, D. T. 1992. “Citations do not Solve
Problems,” Psychological Bulletin, (112:3), pp. 393-395.
Riedl, R., Banker, R. D., Benbasat, I., Davis, F. D., Dennis, A. R.,
Dimoka, A., Gefen, D., Gupta, A., Ischebeck, A., Kenning, P., Pavlou,
P. A., Müller-Putz, G., Straub, D., vom Brocke, J. and Weber, B.
2009. “On the foundations of NeuroIS: Reflections on the Gmundem
retreat 2009, Communication of the Association for Information
Systems, (27:15), pp. 243-264.
Spector, P. E. 2006. “Method Variance in Organizational Research:
Truth or Urban Legend?” Organizational Research Methods, (9:2),
pp. 221-232.
... Neuroscience measures that are linked to one of the three systems in emotional experiences should be seen as complementing measures that are linked to one of the other systems, rather than replacing them. This view diverges from one in which NeuroIS methods are seen primarily as a way of increasing research rigor by providing more objective measures than subjective self-report measures (e.g., see [21]). The viewpoint that we advance here suggests that NeuroIS and self-report measures tap into different aspects of an emotional experience and hence give different insights; thus, there are advantages in using them in combination. ...
Article
Human emotions' role in phenomena related to information systems ( IS) is increasingly of interest to research and practice, and is now informed by a burgeoning literature in neuroscience. This study develops a nomological network with an overarching view of relationships among emotions and other constructs of interest in IS research. The resulting 3-emotion systems' nomological network includes three interacting emotion systems: language, physiology, and behavior. Two laboratory experiments were conducted to test the nomological network, with six online travel service Web pages used as stimuli. The first study used paper-based self-report measures and qualitative comments, whereas the second included both self-reports and electroencephalography (EEG) measures. An outcome measure of e-loyalty was included in each study. The results of both studies showed positive and negative emotion-inducing stimuli were related to positive and negative emotions when viewing the Web sites as indicated by both self-reports and EEG data. In turn, positive and negative emotions as measured by both self-reports and EEG measures were linked to e-loyalty to some degree. This research is novel and significant because it is possibly the first in-depth study to link the study of emotions in IS with a sound theory base and multiple measurement approaches, including neuroscience measures. It shows that an EEG measure has some predictive power for an outcome such as e-loyalty. Implications of the research are that IS studies should distinguish between the different emotion systems of language and physiology, choose emotion measures carefully, and also recognize the intertwining of the emotion systems and cognitive processing.
... This makes this approach better suited to analyses in which system-generated epochs are aggregated over longer task episodes, as can be seen from an accumulating body of studies. This approach has been used in the context of serious games for learning (Charland, Allaire-Duquette, Léger, & Perret, 2012), IT-based learning simulations in the context of business management (Ortiz de Guinea, Titah, Léger, & Michenau, 2012), military operations (Stevens et al., 2011) and in various work environments. It could be used productively with almost any population of learners in authentic learning situations, such as classroom lectures, autonomous learning tasks, and computer-based learning environments. ...
Article
Full-text available
Psychophysiological correlates of learning and performance are currently viewed as an important source of information for the improvement of many spheres of human activity, including the field of education. Studies in neuroeducation are likely to benefit from the concomitant analysis of different levels in the cognitive architecture (psychophysiological, cognitive, and social cognitive). The case study reported herein is concerned with the psychophysiological and cognitive levels. The objective is to test a methodology permitting, on the one hand, to relate data from EEG and data from a thinkaloud procedure collected simultaneously during the performance of a problemsolving task in a semantically-complex domain and, on the other hand, to test hypotheses regarding functional relationships between those levels. During a two-hour experiment, a single participant planned remedial reading activities on the basis of a description of a student having reading difficulties. The procedure for data preparation is presented. Then, as an illustration of the methodology, hypotheses involving both levels are tested. Results show that the methodology can establish correlational relations between targeted levels in terms of cooccurrence of cognitive and psychophysiological states. They also show that an EEG trace transformed into 1-second intervals characterizing brain states is not sufficiently fine-grained to test causal relationships based on the temporality of the data. This study stresses the need to explore the phenomenon of rapidlytransitioning psychophysiological microstates in the context of neuroeducational research. http://www.neuroeducationquebec.org/storage/revue/Mercier2012.pdf
Chapter
Cognitive load is one of the most studied constructs in NeuroIS [1]. Not surprisingly, we have identified 27 papers presented at NeuroIS retreats between 2012 and 2020 which included measurement of cognitive load or related constructs. This paper reviews terminology used to refer to cognitive load, mental workload and its variations, as well as their operationalizations and measurements. All 27 papers employed physiological NeuroIS measures, while six of them additionally used subjective self-ratings. The wide range of measurements prompts us to question if we are measuring the same construct. We provide an overview and a summary of cognitive load terminology and measurement used in these 27 papers and conclude with recommendations for future research.
Article
Full-text available
NeuroIS is a field in Information Systems (IS) that makes use of neuroscience and neurophysiological tools and knowledge to better understand the development, adoption, and impact of information and communication technologies. The fact that NeuroIS now exists for more than a decade motivated us to comprehensively review the academic literature. Investigation of the field's development provides insights into the status of NeuroIS, thereby contributing to identity development in the NeuroIS field. Based on a review of N=200 papers published in 55 journals and 13 conference proceedings in the period 2008-2017, we addressed the following four research questions: Which NeuroIS topics were investigated? What kind of NeuroIS research was published? How was the empirical NeuroIS research conducted? Who published NeuroIS research? Based on a discussion of the findings and their implications for future research, which considers results of a recent NeuroIS survey (N=60 NeuroIS scholars), we conclude that today NeuroIS can be considered an established research field in the IS discipline. However, our review also indicates that further efforts are necessary to advance the field, both from a theoretical and methodological perspective.
Chapter
This chapter provides an overview of methodological approaches and current work in the evaluation of user engagement (UE). Using a series of propositions about the nature of engagement, I review a selection of recent research that utilizes varied methodological approaches to study UE in various human-computer interaction settings. The propositions and the reviewed literature are used to propose a methodological framework to guide decision making and reflection regarding how UE will be evaluated in a given context. The chapter concludes with reflections on broader issues related to how researchers’ methodological stances influence the evaluation of UE. Overall, the chapter argues that UE should be measured using a thoughtful mix of qualitative and quantitative methods, considering the particulars of the use context, and balancing established and emerging subjective and objective metrics.
Conference Paper
Full-text available
NeuroIS is a field in Information Systems (IS) that makes use of neuroscience and neurophysiological tools and knowledge to better understand the development, adoption, and impact of information and communication technologies. The fact that NeuroIS now exists for approximately ten years motivated us to comprehensively review the academic literature. Investigation of the field’s development provides insights into the status of NeuroIS, thereby contributing to identity development in the NeuroIS field. Based on a review of N = 164 papers published in 55 journals and 11 conference proceedings, we addressed the following four research questions: Who published NeuroIS research? What kind of NeuroIS research was published? Which major thematic orientation was chosen by NeuroIS researchers? How was the empirical NeuroIS research conducted? Based on a discussion of the findings and their implications for future research, we conclude that today NeuroIS can be considered as established research field in the IS discipline.
Thesis
Full-text available
Information and communication services have become ubiquitous in our everyday life and, in turn, research on Ubiquitous Information Systems (UIS) has received increasing attention. UIS services can elicit both negative and positive emotions, which are not necessarily perceived consciously by individuals but which may still have an impact on predictors and outcomes of UIS service use. Due to the limitations of psychological self-reports in uncovering these automatic cognitive processes, the current work investigates emotional stimuli of UIS services with neurophysiological data. In particular, we choose electrodermal activity as an indicator of physiological arousal and assess its utility for the design and use of UIS services. To account for the neurophysiological nature of electrodermal activity and to investigate its value in relation to established self-report instruments, we integrate the stimulus-organism-response paradigm with a two-systems view of cognitive processing. Against the background of this theoretical framework, we hypothesise relationships between breakdown events of UIS services (the emotional stimuli), physiological arousal and perceived ease of use (manifestations of the organism's automatic and inferential cognitive processes), and task performance (the response of the organism). We also consider physiological learning processes related to generalisation effects. In order to test the hypotheses, we use empirical data from two studies. Results indicate that electrodermal activity is a useful measure for the design and use of UIS services, even though generalisation effects can reduce its reliability. Moreover, we demonstrate that electrodermal activity is related to perceived ease of use and task performance. We finally discuss the theoretical and practical implications of our results, examine the limitations of the current work and outline future research.
Article
Appeal refers to the positive emotional response to an aesthetic, beautiful, or in another way desirable stimulus. It is a recurring topic in information systems (IS) research, and is important for understanding many phenomena of user behavior and decision-making. While past IS research on appeal has relied predominantly on subjective self-rating scales, this research-in-progress paper proposes complementary objective measurement for appeal. We start by reviewing the linkages between the theoretical constructs related to appeal and their neurophysiological correlates. We then review past approaches to measuring appeal and discuss their characteristics. Finally, we arrive at a recommendation that builds on a combination of psychophysiology (EDA, facial EMG) and brain imaging (fNIRS).
Article
Full-text available
Behavioral beliefs – perceived usefulness and perceived ease of use – have been identified as the most influential antecedents of individuals’ information systems use intentions and behaviors within the technology acceptance model. However, little research has been aimed at investigating the implicit (automatic or unconscious) determinants of such cognitive beliefs, and more importantly, the potential nonlinear relationships of such antecedents with explicit (perceptual) ones. As such, this paper theorizes that implicit neurophysiological states – memory load and distraction - and explicit – engagement and frustration - antecedents interact in the formation of perceived usefulness and perceived ease of use. In order to test the study’s hypotheses, we conducted an experiment that measured neurophysiological states while individuals worked on instrumental and hedonic tasks using technology. The results show that, as theorized, implicit and explicit constructs interact together, and thus, have a nonlinear effect on behavioral beliefs. Specifically, when engagement is high, neurophysiological distraction does not affect perceived usefulness, whereas when engagement is low, neurophysiological distraction has a negative and significant effect on usefulness. The results also show that when frustration is high, neurophysiological memory load has a negative effect on perceived ease of use, whereas when it is low, neurophysiological memory load has a positive effect on perceived ease of use. This study makes several contributions, including the demonstration of the importance of emotional perceptions for moderating the effects of neurophysiological states on behavioral beliefs.
Chapter
Full-text available
The chapter discusses the role of the manner of attitude formation. It focuses on the development of an attitude through direct behavioral experience with the attitude object and examines whether such attitudes better predict subsequent behavior than attitudes formed without behavioral experience. The chapter provides an overview of the attitude-behavior consistency problem and describes the effect of the manner of attitude formation through the “housing” study, the “puzzle” experiment, and the “subject pool” study. The prior-to-later behavior relation is also discussed in the chapter, wherein it has described the self-perception of past religious behaviors, attitudes and self-reports of subsequent behavior, an individual difference perspective, and a partial correlation analysis. The chapter discusses attitudinal qualities—namely, confidence and clarity, the persistence of the attitude, and resistance to attack. The reasons for the differential strength are also explored in the chapter—namely, the amount of information available, information processing, and attitude accessibility. The chapter briefly describes the attitude-behavior relationship, personality traits, and behavior.
Article
Full-text available
One of the most welcome recent developments in Information Systems scholarship has been the growing interest in individuals’ continuing use of information technology well after initial adoption, known in the literature as IT usage, IT continuance, and post-adoptive IT usage. In this essay, we explore the theoretical underpinnings of IS research on continuing IT use. Although the IS literature on continuing IT use emphasizes the role of habitual behavior that does not require conscious behavioral intention, it does so in a way that largely remains faithful to the theoretical tradition of planned behavior and reasoned action. However, a close reading of reference literatures on automatic behavior (behavior that is not consciously controlled) and the influences of emotion on behavior suggests that planned behavior and reasoned action may not provide the best theoretical foundation for the study of continuing IT use. As a result, we call for empirical research that directly compares and contrasts the consensus theory of continuing IT use with rival theories that place much greater emphasis on unplanned and unreasoned action.
Article
A critical element in the evolution of a fundamental body of knowledge in marketing, as well as for improved marketing practice, is the development of better measures of the variables with which marketers work. In this article an approach is outlined by which this goal can be achieved and portions of the approach are illustrated in terms of a job satisfaction measure.
Article
Researchers have long known that research methods influence construct measurements and that this influence, or method bias, can lead to false conclusions. Despite much work in the methodological literature on specific aspects of method bias, such as common method bias and self-report bias, the meaning of method bias remains unclear, and there is no comprehensive approach for dealing with it. This paper offers a clear definition of method bias, proposes a more comprehensive approach for dealing with it, and describes a demonstration exercise applying the approach in an empirical study of how individual system use and task performance relate. The demonstration suggests that the approach is feasible and illustrates how it can help researchers test theories and identify new research opportunities.
Article
Campbell and Fiske's 1959 article made the now-obvious statement that measures of the same variable made by different methods should agree (converge) and certainly should agree better than measures of different variables made by those several methods. Almost everyone accepted that statement, yet the multitrait-multimethod matrices published today show little or no improvement over the published examples cited in 1959. The article raised the conceptual problem of how a variable and its measurement should be linked and the methodological problem of clarifying the nature of methods and their effects. Given these unresolved problems, it is perhaps not surprising that the question of the appropriate statistical analysis of these matrices has also no consensual answer. Perhaps methods and traits or contents are so thoroughly intertwined that their interaction cannot be adequately analyzed.
Article
This paper presents a meta-analysis-based technique to estimate the effect of common method variance on the validity of individual theories. The technique explains between-study variance in observed correlations as a function of the susceptibility to common method variance of the methods employed in individual studies. The technique extends to mono-method studies the concept of method variability underpinning the classic multitrait-multimethod technique. The application of the technique is demonstrated by analyzing the effect of common method variance on the observed correlations between perceived usefulness and usage in the technology acceptance model literature. Implications of the technique and the findings for future research are discussed.
Article
This article reflects on the discussions of the fifteen participants (co authors) of a retreat on the "Foundations of NeuroIS" that took place in Gmunden (Austria) in September 2009. In particular, this article offers initial answers to a set of research questions which are important for the foundations of NeuroIS, an emerging subfield within the IS discipline. The key questions discussed during the retreat that are addressed in this article are: (1) What is NeuroIS, and how does it relate to sister disciplines, such as neuroscience, neuroeconomics, and neuromarketing? (2) Which neuroscience tools are relevant for IS research? (3) What can IS researchers learn from the neuroscience literature, and what do we already know about brain activity? (4) What are possible IS research topics that can be examined with neuroscience tools, and what are some promising research areas for NeuroIS? (5) How can NeuroIS be established as a new subfield in the IS literature, and what are the current challenges for NeuroIS? The article concludes by offering the participants' outlook on the future of NeuroIS.