Conference PaperPDF Available

A neuroscientific perspective for assessing student engagement in e-learning


Abstract and Figures

This paper explores the potential of collecting neurophysiological data to assess learner engagement in order to further understand e-learning experiences. The experimental setup involves collecting electroencephalographic signal (EEG) to infer users’ engagement state while they played to an educational video game involving problem solving in mechanics. The paper will finally present and discuss preliminary results that suggest this neuroscientific perspective is promising for research in e-learning.
No caption available
No caption available
Content may be subject to copyright.
Patrick Charland
Department of didactics
Université du Québec à Montréal
Geneviève Allaire-Duquette
Université du Québec à Montréal
Pierre-Majorique Léger
Department of Information Technology
HEC Montreal
Geneviève Gingras
Université du Québec à Montreal
This paper explores the potential of collecting neurophysiological data to assess learner engagement in
order to further understand e-learning experiences. The experimental setup involves collecting
electroencephalographic signal (EEG) to infer users’ engagement state while they played to an
educational video game involving problem solving in mechanics. The paper will finally present and
discuss preliminary results that suggest this neuroscientific perspective is promising for research in e-
According ton many studies, engagement plays a crucial role in learning. For Clark & Mayer (2011, p.16),
regardless of delivery media, “all learning requires engagement”. Zhang & al. (2006) also remind that increased student
engagement can improve learning outcome, such as problem solving and critical thinking skills. Precisely, studies in e-
learning have suggested that learner engagement is higher with interactive than passive multimedia instruction. Higher
interactivity can lead to higher learner engagement (Chapman, Selvarajah &Webster, 1999) and better learning outcome
(Chou, 2001; Hill & Schneider, 2006).
Generally, engagement is seen as a real time indicator of learner motivation (Ainley, 2004). However,
Fredricks, Blumenfeld & Paris (2004) reviewed several studies and claimed that engagement must be defined by its
multidimensional nature. We therefore adopt their multifaceted definition of this concept:
Behavioural engagement draws on the idea of participation; it includes involvement in academic
and social or extracurricular activities. (…) Emotional engagement encompasses positive and
negative reactions to teachers, classmates, academics, and school and is presumed to create ties to an
institution and influence willingness to do the work. Finally, cognitive engagement draws on the idea
of investment; it incorporates thoughtfulness and willingness to exert the effort necessary to
comprehend complex ideas and master difficult skills.” (p.61)
Preview version of this paper. Content and pagination may change prior to final publication.
The measures of cognitive engagement, conceptualized as a psychological investment in learning (Fredricks,
Blumenfeld & Paris, 2004), are limited, because cognition is not readily observable. It must be either inferred from
behaviour or assessed from self-report measures. Also, from an external perspective, as a student work, it is difficult to
discern by observation whether they are trying to get the work done as quickly and easily as possible or whether they are
using deep-level learning strategies to master the content. Thus, Fredricks, Blumenfeld & Paris (2004) were unable to
find any published studies using measures of cognitive engagement.
Recently, disciplines, such as economics, psychology and marketing, have been using neurophysiological
measurements (Freeman & al., 1999; Glimcher & Rustichini, 2004; Lieberman, 2007; Pope, Bogart & Bartolome, 1995)
in order to assess learners’ cognitive state, such as cognitive engagement (Mandryk & Inkpen, 2004; Nacke & Lindley,
2008). These neurophysiological measures, coupled with efficient analysis algorithms, are much less obstrusive than
traditional self-reported scales, and can allow research protocols in more authentic learning or environments. These tools
include equipment to monitor heart rate, breath rate, blood pressure, body temperature, pupil diameter, electrodermal
activity, electroencephalography (EEG), etc. The remaining of this paper focuses solely on this later technique.
General EEG body of knowledge states that i) increases in brainwave beta activity are associated to a higher
level of engagement related to a task, and ii), increases related to alpha and/or theta activity would reflect less alertness
and task engagement due to decreased information processing (Freeman & al., 1999). Research on task engagement is
builded on vigilance and attention studies, conducted in early 90’s (Makeig & Jung, 1995; Sterman & Mann, 1995).
Thus, Pope & al. (1995) developed and validated an Engagement Index (EI) by computing the respective power of
“beta / (alpha+theta)” bands from various scalp sites (Cz, Pz, P3, P4).
Since Pope’s and his colleague work, new data collection systems (e.g. EMOTIV
) and
algorithms (Stevens & al., 2010) are now able to quantify, in real time, the cognitive states (sleepiness; distraction; low
engagement; high engagement) within a specific task. Initially developed for neuroergonomics, these systems were
designed to investigate drivers’ or plane pilots’ levels of engagement. Researchers in related fields have already started
to adapt EEG based constructs to study engagement in information technology (Ortiz & al., 2013) or in educational
videogame (Charland, Allaire-Duquette & Léger, 2012). We hypothesize that these systems/algorithms could also be
useful in educational research, precisely in e-learning.
The Study
This article aims to present data collected from an exploratory study in science education. The research goal is
to investigate if cognitive engagement, analyzed with recent neuroergonomics algorithms, could be useful to characterize
the learning experience of students resolving physics problems in an online educational video game - Mecanika. Using
an experimental approach, a within-subject comparison was performed to determine cognitive state of learners during the
game. Mecanika is a game designed to learn qualitative conceptual physics: it doesn’t require any calculation or use of
formulas. The goal of the game is to place robots in a 2D futuristic galactic environment and to throw scouts in order to
collect stars on a specific trajectory. The learner has to visualize the trajectory, and modify it consequently with robots,
so that scouts will light a path of stars. Robot placement is central in the game: it involves predictions based on the
comprehension of active forces. If the robots are correctly placed, scouts will collect every star and the student will have
completed the level. If the scouts fail to light every star, the student is asked to try again by placing the robots differently.
Thus, understanding, or problem solving, can be assessed in observing robot placement and/or quantifying scout
throwing. Figure 1 shows an example where the scouts fail to pass on the third star because robots were not placed
correctly. The original game consists of 50 levels designed in coherence with the Force Concept Inventory (FCI), a
robust validated questionnaire (Hestenes, Wells & Swackhamer, 1992) developed in science education research field to
assess students’ qualitative understanding of basic concepts in Newtonian physics. Each level of Mecanika triggers a
specific misconception identified in the FCI. In this pilot study, five (5) levels were extracted from the original game.
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
Figure 1. Mecanika game screen
EEG signal of twenty-four (24) subjects was gathered during an approximately 15 minutes playing session.
During this session, the subjects had to complete the most they could in the game. All subjects were right-handed male
adults, students in a business school IT program. None of the subjects had previous experience related to the game. Signal
was gathered using wireless B-Alert headset system, developed by Advanced Brain Monitoring, Inc. The B-Alert sensor
headset (Berka & al., 2007; Johnson & al., 2011) was used to acquire the EEG data from three referential channels (Fz,
Cz, and POz) and two bipolar channels (Fz-POz and Cz-POz). The sampling rate was 256 units for all channels.
Proprietary data acquisition software receives Wi-Fi data and stores it on the host computer. A cleaning algorithm (Berka
& al., 2007) automatically detect and remove artefacts in the time-domain EEG signal, including spikes caused by tapping
or bumping of the sensors, amplifier saturation, eye blinks and excessive muscle activities (Berka & al., 2007). From the
decontaminated EEG recording, signal is then analyzed on a second-by-second basis, and averaged on a one second time
frame. In other words, the B-Alert algorithm classifies every seconds the subject’s probability of being in one of the four
cognitive states (sleep onset, distraction, low engagement, and high engagement) (Stikic & al., 2011).
Installation of B-Alert headset is a short procedure, which requires less than 10 minutes. In order to make sure
that all the electrodes are in place, the software starts the protocol with a measurement of their impedance. Then, each
subject has to perform a standardized cognitive task to individually calibrate the B-Alert engagement metric. Results of
EEG measurements during the game involve a comparison with this calibration. Also, in-game usage data (time spent on
levels, number of scouts thrown) were collected for each subject.
Graphic 1 shows the general mean of probability of being in a cognitive state, for all 24 users playing Mecanika.
We can state that a 15 minutes spent in Mecanika is a rather engaging task, with a lower probability of being sleepy or
distracted during the game.
Graphic 1. Mean of overall cognitive state probability during the game (all levels; n=24)
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
On an individual analysis, graphic 2 is helpful in identifying sleepy or distracted users. On the x axis, we can
observe each subject probability (y axis) of being in one of the four cognitive states. For example, Subject 4 was highly
engaged in more than 80% of the game, but subject 17 seemed to be sleepy, and subject 23 was quite distracted.
Legend : sleepy (blue)-distracted (red)-low engaged (green) - high engaged (purple)
Graphic 2. Mean of the overall probability of cognitive states for each subjects
On a game level analysis, building upon the recent work of Senecal and his colleagues (2012), an EEG cognitive state
index (CSI) was also computed using the probability of the subject being in one of the four cognitive states:
CSI = (Prob (high engagement) + Prob (Low Engagement))/ (Prob (Distraction) + Prob (Sleepy))
The higher the cognitive state index, the more cognitively engaged was the subject during the task. The EEG
index was normalized using Level 1 of the game as a baseline. Graphic 3 report the variation of this index from the
baseline. We observe that the EEG index increased by 87,6 % between the level 1 and 2 of the game meaning that the
subjects had to mobilize their cognitive resources in order to solve the level 2. This variation from baseline significantly
reduces (Wilcoxon test, p=0.03) between level 2 and level 3, with a 42,3 % difference with baseline. In level 4, subjects
were almost back to their baseline level with a modest 17.4 % difference.
Graphic 3. Level by level variation of EEG Index from baseline (in %)
These last results suggest that, as the game progresses, the learner become more skilled and required to mobilize
less cognitive resources to solve the problems. This is in line with work in cognitive neuroscience (Camerer, Loewenstein,
& Prelec, 2005; Satpute & Lieberman, 2006), which states that a dual process governs human decision and behaviour: a
more controlled system that is conscious, serial and slow, and a more automatic, which is unconscious, paralleled and
effortless. As an expertise in a task is developed, one moves from a controlled to an automatic perspective, thus changing
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
the pattern of neuronal activations (Hill & Schneider, 2006).
The objective of this exploratory study was primarily to determine if EEG analysis algorithms developed in the
field of neuroergonomics could be transferred in educational research.
We believe that e-learning research field could benefit from these recent methodological developments. The wide
variance of cognitive states throughout the subjects and the timeframe in which we can observe variation of engagement is
quite encouraging in the idea of using these neuroergonomics algorithms in this field. Specifically, quantifying learner
engagement over time can be useful to assess program and course developments. Engagement index could be a powerful
tool to inform course designers. For example, as Martens, Gulikers, & Bastiaens (2004, p.374) observe, “quite often,
developers tend to add multimedia add-ons, simulations, and so on, mainly because technology makes it possible, even
though they are not based on careful educational analysis and design”. Therefore, neurophysiological data could inform
designers if a specific add-on is valuable, in terms of learner engagement, as an indicator of motivation.
Also, real-time assessment of cognitive state and learner engagement opens up the possibilities for adaptive e-
learning environments. We can imagine a learner, wearing a lightweight EEG helmet, being prompt for a pause by the
system when its engagement level is lowering. A fair amount of research and development are currently conducted in this
innovative field of Brain-Computer Interface (BCI).
Finally, we need to conclude this paper by raising the fact that the small sample size of the current study limits
the possibilities of generalization among our findings. This research was also limited to cognitive states classification and
we cannot report on the actual learning from the subjects during the task, which will be part of eventual scaled up studies.
Ainley, M. (2004). What do we know about student motivation and engagement?. Communication presented at the Annual
meeting of the Australian Association for Research in Education, Melbourne, Australia.
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Daivis, G., Zivkovic, V., Olmstead, R. E., Tremoulet, P. D., &
Craven, P. L. (2007). EEG Correlates of Task Engagement and Mental Workload in Vigilence, Learning and Memory
Tasks. Aviation, Space and Environmental Medecine, Volume 78, Supplement 1, May 2007 , pp. B231-B244.
Berka, C., Levendowski, D. J., Westbrook, P., Davis, G., Lumicao, M. N., Olmstead, R. E., Popovic, M., Zivkovic V. T.,
& Ramsey, C. K. (2005). EEG quantification of alertness: methods for early identification of individuals most
susceptible to sleep deprivation: Biomonitoring for Physiological and Cognitive Performance during Military
Operations. Proceedings of the International Society for Optical Engineering, p. 78 - 89.
Camerer, C., Loewenstein, G., & Prelec, D. (2005). Neuroeconomics: How Neuroscience Can Inform Economics. Journal
of Economic Literature, 43, 9-64.
Chapman, P., Selvarajah, S., & Webster, J. (1999, mois). Engagement in multimedia training systems. Communication
presented at the 32nd Hawaii International Conference on System Sciences, Maui, HI, USA.
Charland, P., Allaire-Duquette, G., & Léger, P.-M. (2012). Collecting neurophysiological data to investigate users
cognitive states during game play. Journal on Computing, 2(3), 20-24.
Chou, C. (2001). Student interaction in a collaborative distance-learning environment: a model of learner-centered
computer mediated interaction (unpublished Ph.D. thesis). University of Hawaii.
Clark, R. C., & Mayer, R. E. (2011). E-learning and the Science of Instruction. Pfeiffer : San Francisco.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School Engagement: Potential of the Concept, State of the
Evidence. Review of Educational Research, 74(1), 59–109.
Freeman, F. G., Mikulka, P. J., Prinzel, L. J., & Scerbo, M. W. (1999). Evaluation of an adaptive automation system using
three EEG indices with a visual tracking task. Biological psychology, 50, 61-76.
Gevins, A., & Smith, M. E. (2005). Neurophysiologic measures for neuroergonomics. Communication presented at the
HCI International 2005, Las Vegas, USA.
Glimcher, P., & Rustichini, A. (2004). Neuroeconomics: The Consilience of Brain and Decision, Science, 51, 447-452.
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30, 141-158.
Hill N.M., & Schneider, W. (2006). Brain Changes in the Development of Expertise: Neuroanatomical and
Neurophysiological Evidence about Skill-based Adaptations. In K.A. Ericsson, N. Charness, P. Feltovich, and R.
Hoffman (eds.), Cambridge Handbook of Expertise and Expert Performance, pp. 653-682. England : Cambridge
University Press.
Johnson, R. R., Popovic, D. P., Olmstead, R. E., Stikic, M., Levendowski, D. J., & Berka, C. (2011). Drowsiness/alertness
algorithm development and validation using synchronized EEG and cognitive performance to individualize a
generalized model. Biological psychology, 87, 241-250.
Lieberman, M. D. (2007). Social Cognitive Neuroscience: A Review of Core Processes. Annual Review of Psychology, 58,
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
Makeig, S., & Jung, T. P. (1995). Changes in altertness are a principal component of variance in the EEG spectrum.
NeuroReport, 7, 213-217.
Mandryk, R., & Inkpen, K. (2004). Physiological indicators for the evaluation of co-located collaborative play. Paper
presented at ACM Conference.
Martens, R. L., Gulikers, J., & Bastiaens, T. (2004). The impact of intrinsic motivation on e-learningin authentic computer
tasks. Journal of Computer Assisted learning, 20, 368–376.
Nacke, L., & Lindley, C. (2008). Flow and immersion in first-person shooters: measuring the player's gameplay
experience. Paper presented at the Conference on Future Play: Research, Play, Share, Lieu.
Ortiz de Guinea, A., Titah, R., & Léger, P.-M. (2013). Measure for Measure: A two study multi-trait multi-method
investigation of construct validity in IS research. Computers in Human Behavior,29 (3), 833–844.
Pope, A. T., & Bogart, E. H. (1994). Method of encouraging attention by correlating video game difficulty with attention
level. United States Patent #5377100, December 1994.
Pope, A. T., Bogart, E. H., & Bartolome, D. S. (1995). Biocybernetic system evaluation indices of operator engagement in
automated task. Biological psychology, 40, 187-195.
Satpute A. B., & Lieberman, M. D. (2006). Integrating automatic and controlled processing into neurocognitive models of
social cognition. Brain Research, 1079, 86-9.
Senecal, S., Léger, P. M., Fredette, M., & Riedl, R. (2012, December). Consumers' Online Cognitive Scripts: A
Neurophysiological Approach. Paper presented at the 12th International Conference on Information Systems (ICIS
Sterman, M. B., & Mann, C. A. (1995). Concepts and applications of EEG analysis in aviation performance evaluation.
Biological psychology, 40, 115-130.
Stikic, M., Johnson, R. R., Levendowski, D. J., Popovic, D. P., Olmstead, R. E., & Berka, C. (2011). EEG-derived
estimators of present and future cognitive performance. Frontiers in Human Neuroscience, 5(August), 3-4.
Stevens, R., Galloway, T., Berka, C., & Behneman, A. (2010). A Neurophysiologic Approach For Studying Team
Cognition. Communication presented at Interservice/Industry Training, Simulation, and Education Conference
Zhang, D., Zhou, L. Briggs, R. O., & Nunamaker , J. F. (2006). Instructional video in e-learning: Assessing the impact
of interactive video on learning effectiveness. Information & Management, 43(1), 15-27.
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
Interaction research in distance education has focused mostly on learner -teacher interaction in a learning environment based on a behaviorist curriculum. This presentation focuses on factors contributing to learner-learner interaction in a distance learning course based on learner -centered and collaborative instructional design. The proposed model, which resulted from research on patterns of learner interaction in both synchronous and asynchronous computer-mediated communication modes, examines factors contributing to interaction in the areas of learner characteristics, technology attributes, and learning activities. Introduction Interaction research provides important information on student behaviors in distance learning environments to educators, researchers, and instructional designers. The current state of interaction research has focused mostly on the quantitative results of inter-connected messages in computer-mediated communication (CMC) conferences. Contributing factors to interaction such as theoretical principles of course design and learning contexts are largely ignored. While various virtual learning environments and course management systems are being introduced to the distance learning community, it is easy to loose sight on the pedagogical application of the learning systems. Teachers are rushed to learn various state-of-art instructional technologies but not given instructional examples or time to develop well-designed instructional materials for conducting distance learning courses. The issues faced by the educators are similar to that of a novice pilot being rushed to drive a commercial airplane without going through appropriate training via flight simulation. More research on how these systems can enhance student learning and examples of best practices on instructional design in various disciplines are needed for the success of distance education. As interaction has been identified as the key to the success of online learning by researchers (Gunawardena et al., 1997), this study examines patterns of online interaction and the types of instructional design that would enhance online interaction via both synchronous and asynchronous communication. Learner-centered computer-mediated interaction in this study is defined as reciprocal communication among participants of computer-mediated learning environments that emphasizes learner developments in cognition, motivation, and social advancement for the purpose of knowl edge construction and community building. Two theoretical principles that are highly relevant to such interaction are constructivism and learner-centered principles (LCPs). A constructivist distance learning environment places emphases on knowledge construction through interaction with the physical environment and through the appropriation of culturally relevant activities. In other words, knowledge is co-constructed with peers or experts and through the immersion in a social context (Bonk & Cunningham, 1998). The Learner-Centered Principles were developed by the American Psychological Association (APA, 1997) as a framework for the new educational approaches that stress the integration of the needs, skills, interests, and backgrounds of the students into the curriculum planing. The following section on literature review examines the connection between these principles and interaction.
Full-text available
The objective is to combine simultaneous neurophysiologic signals from team members to develop pattern categories, called neurophysiologic synchronies (NS), that can be related to the second-by-second activities of teams. Neurophysiologic synchronies are a low level data stream that can be collected and analyzed in real time and in realistic settings. If the expression of different NS patterns is sensitive to changes in the behavior of teams over time they may be a useful tool for studying team cognition. EEG-derived measures of engagement from team members were normalized and pattern classified by self-organizing artificial neural networks and hidden Markov models. The temporal expression of these patterns was mapped onto team events. Across multiple teamwork tasks NS expression was shown to be non-random and sensitive to changes in the task and the activities of team members. These studies suggest that neurophysiologic indicators measured by EEG may be useful for studying team behavior not only at the milliseconds level, but at more extended time frames. ABOUT THE AUTHORS RON STEVENS, PH.D. is Professor and a member of the Brain Research Institute at the UCLA School of Medicine. He directs the internet-based IMMEX problem solving project which has engaged over 150,000 students and teachers in computational education and professional development activities that span elementary school through medical school. Recently Dr. Stevens received the 'Foundations of Augmented Cognition' award from the Augmented Cognition Society. His interests include using machine learning tools and electroencephalography (EEG) to model the acquisition of scientific problem solving skills. CHRIS BERKA, CEO and Co-Founder of Advanced Brain Monitoring has over 25 years experience managing clinical research and developing and commercializing new technologies. She is co-inventor of seven patented and seven patent-pending technologies and is the principal investigator or co-investigator for grants awarded by the National Institutes of Health, DARPA, ONR and NSF. She has 10 years experience as a research scientist with publications on the analysis of the EEG correlates of cognition in healthy subjects and patients with sleep and neurological disorders. TRYSHA GALLOWAY directs the EEG studies for The Interactive Multi Media Exercises (IMMEX) laboratory and is co-author on eight peer reviewed studies. Her research interests blend the population-based advantages of probabilistic performance modeling with the detection of neurophysiologic signals to help personalize the learning process in complex education and training activities. ADRIENNE BEHNEMAN is a Project Manager at Advanced Brain Monitoring. Since 2007, she has played a key leadership role in the Accelerated Learning, RAPID and ANITA projects. She is interested in the development of neuroscience-based tools to enhance training and education. Her current focus is on researching the psychophysiology of expertise in domains including marksmanship, deadly force decision making and team function, as part of the Accelerated Learning project.
Full-text available
Interactive video in an e-learning system allows proactive and random access to video content. Our empirical study examined the influence of interactive video on learning outcome and learner satisfaction in e-learning environments. Four different settings were studied: three were e-learning environments—with interactive video, with non-interactive video, and without video. The fourth was the traditional classroom environment. Results of the experiment showed that the value of video for learning effectiveness was contingent upon the provision of interactivity. Students in the e-learning environment that provided interactive video achieved significantly better learning performance and a higher level of learner satisfaction than those in other settings. However, students who used the e-learning environment that provided non-interactive video did not improve either. The findings suggest that it may be important to integrate interactive instructional video into e-learning systems.
Conference Paper
Full-text available
Researching experiential phenomena is a challenging undertaking, given the sheer variety of experiences that are described by gamers and missing a formal taxonomy: flow, immersion, boredom, excitement, challenge, and fun. These informal terms require scientific explanation, which amounts to providing measurable criteria for different experiential states. This paper reports the results of an experimental psychophysiological study investigating different traits of gameplay experience using subjective and objective measures. Participants played three Half-Life 2 game modifications while being measured with electroencephalography, electrocardiography, electromyography, galvanic skin response and eye tracking equipment. In addition, questionnaire responses were collected after each play session. A level designed for combat-oriented flow experience demonstrated measurable high-arousal positive affect emotions. The positive correlation between subjective and objective indicators of gameplay experience shows the great potential of the method presented here for providing real-time emotional profiles of gameplay that may be correlated with self-reported subjective descriptions.
The concept of school engagement has attracted increasing attention as representing a possible antidote to declining academic motivation and achievement. Engagement is presumed to be malleable, responsive to contextual features, and amenable to environmental change. Researchers describe behavioral, emotional, and cognitive engagement and recommend studying engagement as a multifaceted construct. This article reviews definitions, measures, precursors, and outcomes of engagement; discusses limitations in the existing research; and suggests improvements. The authors conclude that, although much has been learned, the potential contribution of the concept of school engagement to research on student experience has yet to be realized. They call for richer characterizations of how students behave, feel, and think—research that could aid in the development of finely tuned interventions
Conference Paper
A cognitive script is a predetermined sequence of actions that define a well-known situation. Building on neuroscience literature, the objectives of this research-in-progress are to verify and validate that consumers activate cognitive scripts when shopping online, understand how cognitive scripts are formed by consumers over multiple online shopping trips, and investigate how consumers activating different cognitive scripts respond when facing a novel shopping environment. Twenty-one novice participants (i.e., no digital music purchase experience) were assigned to either an “intrascript” condition (multiple visits to a single website) or an “interscript” condition (single visits to multiple websites). Using psychometric and neurophysiological measures, our results suggest that intrascript consumers appear to use more automatic processing, while interscript consumers use more controlled processing. In addition, when visiting a new website, interscript consumers perceive this website as easier to use than intrascript consumers. Theoretical and practical implications of these results are discussed.
Given the importance and criticality of instrument validation in IS research, the main objective of this study is to provide a systematic assessment of IS construct validity via multi-trait multi-method (MTMM) matrix. To do so, the paper uses structurally different methods – neurophysiological and self-reported instruments – to measure three important and commonly used IS constructs: engagement, arousal and cognitive load in two different experimental settings. The experiments involved seventeen (17) and twenty-four (24) participants respectively and consisted in using different IS to execute a set of both instrumental and hedonic tasks. The results generally support MTMM matrix expectations and shed light on the complexity of detecting the nature of mono-method bias. Specifically, the results show that prim- itive perceptual IS constructs such as arousal seem to be less affected by mono-method bias, whereas more complex perceptual constructs such as engagement or cognitive load have higher within method correlations. There are two complementary explanations for the within method correlations: (a) a com- bination between complexity of trait and method and (b) method effects that are congeneric.