Content uploaded by Pierre-Majorique Léger
Author content
All content in this area was uploaded by Pierre-Majorique Léger
Content may be subject to copyright.
A NEUROSCIENTIFIC PERSPECTIVE FOR ASSESSING
STUDENT ENGAGEMENT IN E-LEARNING
Patrick Charland
Department of didactics
Université du Québec à Montréal
Canada
charland.patrick@uqam.ca
Geneviève Allaire-Duquette
Université du Québec à Montréal
Canada
allaire-duquette.genevieve@gmail.com
Pierre-Majorique Léger
Department of Information Technology
HEC Montreal
Canada
pml@hec.ca
Geneviève Gingras
Université du Québec à Montreal
Canada
gingras.genevieve.4@uqam.ca
This paper explores the potential of collecting neurophysiological data to assess learner engagement in
order to further understand e-learning experiences. The experimental setup involves collecting
electroencephalographic signal (EEG) to infer users’ engagement state while they played to an
educational video game involving problem solving in mechanics. The paper will finally present and
discuss preliminary results that suggest this neuroscientific perspective is promising for research in e-
learning.
Introduction
According ton many studies, engagement plays a crucial role in learning. For Clark & Mayer (2011, p.16),
regardless of delivery media, “all learning requires engagement”. Zhang & al. (2006) also remind that increased student
engagement can improve learning outcome, such as problem solving and critical thinking skills. Precisely, studies in e-
learning have suggested that learner engagement is higher with interactive than passive multimedia instruction. Higher
interactivity can lead to higher learner engagement (Chapman, Selvarajah &Webster, 1999) and better learning outcome
(Chou, 2001; Hill & Schneider, 2006).
Generally, engagement is seen as a real time indicator of learner motivation (Ainley, 2004). However,
Fredricks, Blumenfeld & Paris (2004) reviewed several studies and claimed that engagement must be defined by its
multidimensional nature. We therefore adopt their multifaceted definition of this concept:
“Behavioural engagement draws on the idea of participation; it includes involvement in academic
and social or extracurricular activities. (…) Emotional engagement encompasses positive and
negative reactions to teachers, classmates, academics, and school and is presumed to create ties to an
institution and influence willingness to do the work. Finally, cognitive engagement draws on the idea
of investment; it incorporates thoughtfulness and willingness to exert the effort necessary to
comprehend complex ideas and master difficult skills.” (p.61)
Preview version of this paper. Content and pagination may change prior to final publication.
The measures of cognitive engagement, conceptualized as a psychological investment in learning (Fredricks,
Blumenfeld & Paris, 2004), are limited, because cognition is not readily observable. It must be either inferred from
behaviour or assessed from self-report measures. Also, from an external perspective, as a student work, it is difficult to
discern by observation whether they are trying to get the work done as quickly and easily as possible or whether they are
using deep-level learning strategies to master the content. Thus, Fredricks, Blumenfeld & Paris (2004) were unable to
find any published studies using measures of cognitive engagement.
Recently, disciplines, such as economics, psychology and marketing, have been using neurophysiological
measurements (Freeman & al., 1999; Glimcher & Rustichini, 2004; Lieberman, 2007; Pope, Bogart & Bartolome, 1995)
in order to assess learners’ cognitive state, such as cognitive engagement (Mandryk & Inkpen, 2004; Nacke & Lindley,
2008). These neurophysiological measures, coupled with efficient analysis algorithms, are much less obstrusive than
traditional self-reported scales, and can allow research protocols in more authentic learning or environments. These tools
include equipment to monitor heart rate, breath rate, blood pressure, body temperature, pupil diameter, electrodermal
activity, electroencephalography (EEG), etc. The remaining of this paper focuses solely on this later technique.
General EEG body of knowledge states that i) increases in brainwave beta activity are associated to a higher
level of engagement related to a task, and ii), increases related to alpha and/or theta activity would reflect less alertness
and task engagement due to decreased information processing (Freeman & al., 1999). Research on task engagement is
builded on vigilance and attention studies, conducted in early 90’s (Makeig & Jung, 1995; Sterman & Mann, 1995).
Thus, Pope & al. (1995) developed and validated an Engagement Index (EI) by computing the respective power of
“beta / (alpha+theta)” bands from various scalp sites (Cz, Pz, P3, P4).
Since Pope’s and his colleague work, new data collection systems (e.g. EMOTIV
®
or B-ALERT
®
) and
algorithms (Stevens & al., 2010) are now able to quantify, in real time, the cognitive states (sleepiness; distraction; low
engagement; high engagement) within a specific task. Initially developed for neuroergonomics, these systems were
designed to investigate drivers’ or plane pilots’ levels of engagement. Researchers in related fields have already started
to adapt EEG based constructs to study engagement in information technology (Ortiz & al., 2013) or in educational
videogame (Charland, Allaire-Duquette & Léger, 2012). We hypothesize that these systems/algorithms could also be
useful in educational research, precisely in e-learning.
The Study
This article aims to present data collected from an exploratory study in science education. The research goal is
to investigate if cognitive engagement, analyzed with recent neuroergonomics algorithms, could be useful to characterize
the learning experience of students resolving physics problems in an online educational video game - Mecanika. Using
an experimental approach, a within-subject comparison was performed to determine cognitive state of learners during the
game. Mecanika is a game designed to learn qualitative conceptual physics: it doesn’t require any calculation or use of
formulas. The goal of the game is to place robots in a 2D futuristic galactic environment and to throw scouts in order to
collect stars on a specific trajectory. The learner has to visualize the trajectory, and modify it consequently with robots,
so that scouts will light a path of stars. Robot placement is central in the game: it involves predictions based on the
comprehension of active forces. If the robots are correctly placed, scouts will collect every star and the student will have
completed the level. If the scouts fail to light every star, the student is asked to try again by placing the robots differently.
Thus, understanding, or problem solving, can be assessed in observing robot placement and/or quantifying scout
throwing. Figure 1 shows an example where the scouts fail to pass on the third star because robots were not placed
correctly. The original game consists of 50 levels designed in coherence with the Force Concept Inventory (FCI), a
robust validated questionnaire (Hestenes, Wells & Swackhamer, 1992) developed in science education research field to
assess students’ qualitative understanding of basic concepts in Newtonian physics. Each level of Mecanika triggers a
specific misconception identified in the FCI. In this pilot study, five (5) levels were extracted from the original game.
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
Figure 1. Mecanika game screen
EEG signal of twenty-four (24) subjects was gathered during an approximately 15 minutes playing session.
During this session, the subjects had to complete the most they could in the game. All subjects were right-handed male
adults, students in a business school IT program. None of the subjects had previous experience related to the game. Signal
was gathered using wireless B-Alert headset system, developed by Advanced Brain Monitoring, Inc. The B-Alert sensor
headset (Berka & al., 2007; Johnson & al., 2011) was used to acquire the EEG data from three referential channels (Fz,
Cz, and POz) and two bipolar channels (Fz-POz and Cz-POz). The sampling rate was 256 units for all channels.
Proprietary data acquisition software receives Wi-Fi data and stores it on the host computer. A cleaning algorithm (Berka
& al., 2007) automatically detect and remove artefacts in the time-domain EEG signal, including spikes caused by tapping
or bumping of the sensors, amplifier saturation, eye blinks and excessive muscle activities (Berka & al., 2007). From the
decontaminated EEG recording, signal is then analyzed on a second-by-second basis, and averaged on a one second time
frame. In other words, the B-Alert algorithm classifies every seconds the subject’s probability of being in one of the four
cognitive states (sleep onset, distraction, low engagement, and high engagement) (Stikic & al., 2011).
Installation of B-Alert headset is a short procedure, which requires less than 10 minutes. In order to make sure
that all the electrodes are in place, the software starts the protocol with a measurement of their impedance. Then, each
subject has to perform a standardized cognitive task to individually calibrate the B-Alert engagement metric. Results of
EEG measurements during the game involve a comparison with this calibration. Also, in-game usage data (time spent on
levels, number of scouts thrown) were collected for each subject.
Findings
Graphic 1 shows the general mean of probability of being in a cognitive state, for all 24 users playing Mecanika.
We can state that a 15 minutes spent in Mecanika is a rather engaging task, with a lower probability of being sleepy or
distracted during the game.
Graphic 1. Mean of overall cognitive state probability during the game (all levels; n=24)
Thrown
Scout
Positionned
robots
Scout
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
On an individual analysis, graphic 2 is helpful in identifying sleepy or distracted users. On the x axis, we can
observe each subject probability (y axis) of being in one of the four cognitive states. For example, Subject 4 was highly
engaged in more than 80% of the game, but subject 17 seemed to be sleepy, and subject 23 was quite distracted.
Legend : sleepy (blue)-distracted (red)-low engaged (green) - high engaged (purple)
Graphic 2. Mean of the overall probability of cognitive states for each subjects
On a game level analysis, building upon the recent work of Senecal and his colleagues (2012), an EEG cognitive state
index (CSI) was also computed using the probability of the subject being in one of the four cognitive states:
CSI = (Prob (high engagement) + Prob (Low Engagement))/ (Prob (Distraction) + Prob (Sleepy))
The higher the cognitive state index, the more cognitively engaged was the subject during the task. The EEG
index was normalized using Level 1 of the game as a baseline. Graphic 3 report the variation of this index from the
baseline. We observe that the EEG index increased by 87,6 % between the level 1 and 2 of the game meaning that the
subjects had to mobilize their cognitive resources in order to solve the level 2. This variation from baseline significantly
reduces (Wilcoxon test, p=0.03) between level 2 and level 3, with a 42,3 % difference with baseline. In level 4, subjects
were almost back to their baseline level with a modest 17.4 % difference.
Graphic 3. Level by level variation of EEG Index from baseline (in %)
These last results suggest that, as the game progresses, the learner become more skilled and required to mobilize
less cognitive resources to solve the problems. This is in line with work in cognitive neuroscience (Camerer, Loewenstein,
& Prelec, 2005; Satpute & Lieberman, 2006), which states that a dual process governs human decision and behaviour: a
more controlled system that is conscious, serial and slow, and a more automatic, which is unconscious, paralleled and
effortless. As an expertise in a task is developed, one moves from a controlled to an automatic perspective, thus changing
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
the pattern of neuronal activations (Hill & Schneider, 2006).
Conclusion
The objective of this exploratory study was primarily to determine if EEG analysis algorithms developed in the
field of neuroergonomics could be transferred in educational research.
We believe that e-learning research field could benefit from these recent methodological developments. The wide
variance of cognitive states throughout the subjects and the timeframe in which we can observe variation of engagement is
quite encouraging in the idea of using these neuroergonomics algorithms in this field. Specifically, quantifying learner
engagement over time can be useful to assess program and course developments. Engagement index could be a powerful
tool to inform course designers. For example, as Martens, Gulikers, & Bastiaens (2004, p.374) observe, “quite often,
developers tend to add multimedia add-ons, simulations, and so on, mainly because technology makes it possible, even
though they are not based on careful educational analysis and design”. Therefore, neurophysiological data could inform
designers if a specific add-on is valuable, in terms of learner engagement, as an indicator of motivation.
Also, real-time assessment of cognitive state and learner engagement opens up the possibilities for adaptive e-
learning environments. We can imagine a learner, wearing a lightweight EEG helmet, being prompt for a pause by the
system when its engagement level is lowering. A fair amount of research and development are currently conducted in this
innovative field of Brain-Computer Interface (BCI).
Finally, we need to conclude this paper by raising the fact that the small sample size of the current study limits
the possibilities of generalization among our findings. This research was also limited to cognitive states classification and
we cannot report on the actual learning from the subjects during the task, which will be part of eventual scaled up studies.
References
Ainley, M. (2004). What do we know about student motivation and engagement?. Communication presented at the Annual
meeting of the Australian Association for Research in Education, Melbourne, Australia.
Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Daivis, G., Zivkovic, V., Olmstead, R. E., Tremoulet, P. D., &
Craven, P. L. (2007). EEG Correlates of Task Engagement and Mental Workload in Vigilence, Learning and Memory
Tasks. Aviation, Space and Environmental Medecine, Volume 78, Supplement 1, May 2007 , pp. B231-B244.
Berka, C., Levendowski, D. J., Westbrook, P., Davis, G., Lumicao, M. N., Olmstead, R. E., Popovic, M., Zivkovic V. T.,
& Ramsey, C. K. (2005). EEG quantification of alertness: methods for early identification of individuals most
susceptible to sleep deprivation: Biomonitoring for Physiological and Cognitive Performance during Military
Operations. Proceedings of the International Society for Optical Engineering, p. 78 - 89.
Camerer, C., Loewenstein, G., & Prelec, D. (2005). Neuroeconomics: How Neuroscience Can Inform Economics. Journal
of Economic Literature, 43, 9-64.
Chapman, P., Selvarajah, S., & Webster, J. (1999, mois). Engagement in multimedia training systems. Communication
presented at the 32nd Hawaii International Conference on System Sciences, Maui, HI, USA.
Charland, P., Allaire-Duquette, G., & Léger, P.-M. (2012). Collecting neurophysiological data to investigate users
cognitive states during game play. Journal on Computing, 2(3), 20-24.
Chou, C. (2001). Student interaction in a collaborative distance-learning environment: a model of learner-centered
computer mediated interaction (unpublished Ph.D. thesis). University of Hawaii.
Clark, R. C., & Mayer, R. E. (2011). E-learning and the Science of Instruction. Pfeiffer : San Francisco.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School Engagement: Potential of the Concept, State of the
Evidence. Review of Educational Research, 74(1), 59–109.
Freeman, F. G., Mikulka, P. J., Prinzel, L. J., & Scerbo, M. W. (1999). Evaluation of an adaptive automation system using
three EEG indices with a visual tracking task. Biological psychology, 50, 61-76.
Gevins, A., & Smith, M. E. (2005). Neurophysiologic measures for neuroergonomics. Communication presented at the
HCI International 2005, Las Vegas, USA.
Glimcher, P., & Rustichini, A. (2004). Neuroeconomics: The Consilience of Brain and Decision, Science, 51, 447-452.
Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30, 141-158.
Hill N.M., & Schneider, W. (2006). Brain Changes in the Development of Expertise: Neuroanatomical and
Neurophysiological Evidence about Skill-based Adaptations. In K.A. Ericsson, N. Charness, P. Feltovich, and R.
Hoffman (eds.), Cambridge Handbook of Expertise and Expert Performance, pp. 653-682. England : Cambridge
University Press.
Johnson, R. R., Popovic, D. P., Olmstead, R. E., Stikic, M., Levendowski, D. J., & Berka, C. (2011). Drowsiness/alertness
algorithm development and validation using synchronized EEG and cognitive performance to individualize a
generalized model. Biological psychology, 87, 241-250.
Lieberman, M. D. (2007). Social Cognitive Neuroscience: A Review of Core Processes. Annual Review of Psychology, 58,
259-289.
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
Makeig, S., & Jung, T. P. (1995). Changes in altertness are a principal component of variance in the EEG spectrum.
NeuroReport, 7, 213-217.
Mandryk, R., & Inkpen, K. (2004). Physiological indicators for the evaluation of co-located collaborative play. Paper
presented at ACM Conference.
Martens, R. L., Gulikers, J., & Bastiaens, T. (2004). The impact of intrinsic motivation on e-learningin authentic computer
tasks. Journal of Computer Assisted learning, 20, 368–376.
Nacke, L., & Lindley, C. (2008). Flow and immersion in first-person shooters: measuring the player's gameplay
experience. Paper presented at the Conference on Future Play: Research, Play, Share, Lieu.
Ortiz de Guinea, A., Titah, R., & Léger, P.-M. (2013). Measure for Measure: A two study multi-trait multi-method
investigation of construct validity in IS research. Computers in Human Behavior,29 (3), 833–844.
Pope, A. T., & Bogart, E. H. (1994). Method of encouraging attention by correlating video game difficulty with attention
level. United States Patent #5377100, December 1994.
Pope, A. T., Bogart, E. H., & Bartolome, D. S. (1995). Biocybernetic system evaluation indices of operator engagement in
automated task. Biological psychology, 40, 187-195.
Satpute A. B., & Lieberman, M. D. (2006). Integrating automatic and controlled processing into neurocognitive models of
social cognition. Brain Research, 1079, 86-9.
Senecal, S., Léger, P. M., Fredette, M., & Riedl, R. (2012, December). Consumers' Online Cognitive Scripts: A
Neurophysiological Approach. Paper presented at the 12th International Conference on Information Systems (ICIS
2012).
Sterman, M. B., & Mann, C. A. (1995). Concepts and applications of EEG analysis in aviation performance evaluation.
Biological psychology, 40, 115-130.
Stikic, M., Johnson, R. R., Levendowski, D. J., Popovic, D. P., Olmstead, R. E., & Berka, C. (2011). EEG-derived
estimators of present and future cognitive performance. Frontiers in Human Neuroscience, 5(August), 3-4.
Stevens, R., Galloway, T., Berka, C., & Behneman, A. (2010). A Neurophysiologic Approach For Studying Team
Cognition. Communication presented at Interservice/Industry Training, Simulation, and Education Conference
I/ITSEC.
Zhang, D., Zhou, L. Briggs, R. O., & Nunamaker , J. F. (2006). Instructional video in e-learning: Assessing the impact
of interactive video on learning effectiveness. Information & Management, 43(1), 15-27.
Preview version of this paper. Content and pagination may change prior to final publication.
E-Learn 2013
