Augmented Metacognition: Exploring Pupil Dilation Sonification to Elicit Metacognitive Awareness

Conference Paper (PDF Available) · March 2018with 57 Reads
DOI: 10.1145/3173225.3173265
Conference: the Twelfth International Conference
Abstract
Metacognitive awareness enables people to make conscious decisions about their own cognitions, and adapt to meet task performance goals. Despite the role of metacognition in task performance, technologies that effectively augment metacognition are scarce. We explore a novel approach to augment metacognition based on making the eye's pupil dilations, which associate with a variety of cognitions, audible via sonification in real-time. In this exploratory study, we investigated whether pupil dilation sonification can elicit metacognitive awareness. Our findings suggest that correlations between a variety of cognitions, e.g., attentional focus and depth of thinking, and sounds generated by the sonification can emerge spontaneously and by instruction. This justifies further research into the use of pupil dilation sonification as a means to augment metacognitive abilities.
Augmented Metacognition: Exploring
Pupil Dilation Sonification to Elicit
Metacognitive Awareness
Abstract
Metacognitive awareness enables people to make
conscious decisions about their own cognitions, and
adapt to meet task performance goals. Despite the role
of metacognition in task performance, technologies that
effectively augment metacognition are scarce. We
explore a novel approach to augment metacognition
based on making the eye’s pupil dilations, which
associate with a variety of cognitions, audible via
sonification in real-time. In this exploratory study, we
investigated whether pupil dilation sonification can elicit
metacognitive awareness. Our findings suggest that
correlations between a variety of cognitions, e.g.,
attentional focus and depth of thinking, and sounds
generated by the sonification can emerge
spontaneously and by instruction. This justifies further
research into the use of pupil dilation sonification as a
means to augment metacognitive abilities.
Author Keywords
Augmented Metacognition; Biofeedback; Embodied
Information; Metacognition; Pupil Dilation; Sonification.
Introduction
Awareness about your own thinking, or metacognitive
awareness, enables you to adapt cognitions to meet
Permission to make digital or hard copies of part or all of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for third-party components of this work must be honored. For all other
uses, contact the Owner/Author.
TEI '18, March 1821, 2018, Stockholm, Sweden
© 2018 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-5568-1/18/03.
https://doi.org/10.1145/3173225.3173265
Alwin de Rooij
Tilburg University
Warandelaan 2, Tilburg, NL
alwinderooij@uvt.nl
Hanna Schraffenberger
Tilburg University
Warandelaan 2, Tilburg, NL
h.k.schraffenberger@uvt.nl
Mathijs Bontje
Tilburg University
Warandelaan 2, Tilburg, NL
mathijsbontje@gmail.com
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
237
task demands [12]. It can help compensate for lower
IQ [31], improve learning [33], and benefit self-
confidence [18]. However, technologies that augment
metacognition are scarce [16]. We propose that by
making pupil dilations audible in real-time with a pupil
dilation sonification [cf. 35], correlations can emerge
between the sounds generated by the sonification, and
the cognitions that associate with changes in pupil
dilation [8, 21]. When such correlations emerge,
changes in the sound can bring changes in cognition
into awareness, i.e., elicit metacognitive awareness. In
this paper, we explore the potential of pupil dilation
sonification to elicit metacognitive awareness; as a first
step toward a novel way to augment metacognition.
Metacognition and its augmentation
Metacognition can be defined as the awareness and
control of one’s own cognitive processes [1]. When
changes in cognitive processes enter awareness,
decision making about one’s cognitive behaviour, e.g.,
for task performance goals, can take place. For
example, awareness about sustained attentional focus
can aid in the appraisal of what kind of task requires
such focus [cf. 36]. This, in turn, can be used to exert
control over one’s own cognition in order to effectively
adapt to changing task demands [11].
However, technologies that effectively augment
metacognition are scarce [16]. Typically, closed-loop
systems that classify cognitions based on physiological
data [23] are used to select and present feedback to
support metacognitive awareness and control [25]; and
thus, in turn, affect cognition. However, such
classification models are error prone [14], and
generalise poorly to situations not contained in the data
these models are developed from [22]. Even if
cognitions could be classified accurately [14], other
information that is necessary to enable metacognition
[12], such as task performance appraisals, may not be
available to the technology [cf. 5, 6, 7]. Overcoming
these issues may be necessary to truly enable
metacognitive augmentation. This study presents a first
step toward that aim. We build on the fact that people
themselves have access to the necessary information,
as it resides in their own mental environment; and
focus on bringing this information into people’s
awareness; empowering them to adapt and control
their cognitive processes.
The pupil dilation-cognition link
Technologically augmented metacognition could be
based on the eye’s pupil dilations. The pupil is a circular
opening that dilates and constricts [3] to regulate the
amount of light that falls onto the retina [36]. Pupil
dilation also associates with the type and amount of
allocation of cognitive resources [26] and tracks
changes in cognition rapidly [24]. Task-evoked pupil
dilation associates with the allocation of cognitive
resources to a task, e.g., attentional focus [15] and
mental effort [8]; task-independent pupil dilation
associates with the allocation of cognitive resources to
task-irrelevant information, e.g., distractibility [15] and
mind wandering [28]. Similarly, the association
between pupil dilation and depth of thinking [2] and
cognitive load [17] can be explained in this way.
Being able to perceive one’s own physiological changes
can serve as embodied information about the
cognitions these changes associate with [30]. For
example, mechano- and thermoreceptors in the skin
enable us to sense when we are sweating [10], and
when engaging in a difficult exam, this may elicit
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
238
metacognitive awareness about the difficulty of the
exam [29]. As pupil dilation tracks changes in a variety
of cognitions rapidly [8, 24], it could be a rich source of
embodied information [11]. However, one can normally
not perceive one’s own pupil dilations. Therefore, it
could be that a technology that makes a person’s own
pupil dilation perceivable can be used as a basis for a
new way to augment metacognition.
Using pupil dilation sonification to augment
metacognition
We propose a novel way to augment metacognition via
pupil dilation sonification. We conjecture that perceiving
one’s own pupil dilations can elicit metacognitive
awareness. As cognitions change, associated pupil
dilations rapidly follow [24]. When these changes are
perceived via audible sound, it might enable one to
learn correlations between one’s own pupil dilations and
related cognitions [cf. 21]. These correlations then
potentially can function as embodied information about
if, when, and what cognitions change during a task [cf.
30], i.e., help elicit metacognitive awareness [11].
In this study, we take the first steps to explore
whether, what, and when correlations emerge between
sound generated based on pupil dilation and related
cognitions spontaneously; and we check whether
people can experience instructed relations between the
sound and pupil dilation related cognitive processes.
Method
An exploratory study was conducted to investigate the
conjectures developed in the above.
Participants
Twenty people participated (Mage=22.68, SDage=5.11,
Rangeage=18-38, 16, 4). Participation required (past)
involvement in higher education. They were Dutch.
Participants self-reported mathematical expertise was
above average (M=6.58, SD=1.29, Range=2-7).
Course credit was offered where appropriate.
Pupil dilation sonification
Sonification [35] was used because of the high
temporal resolution of the auditory sense [20] and
because it allows us to make pupil dilation changes
perceivable in detail [19]. The Eye Tribe eye-tracker
streamed the mean pupil size of both eyes at 60 Hz
[32] to a custom sonification program made in Cycling
’74 Max 7 [4]. To scale the streaming data to a 0-1
range a 15 micrometre minimum pupil size was
assumed and a maximum was captured while
darkening the room. Signal loss (e.g., due to blinking)
was handled by retaining the last data point before
signal loss until new data was received. The data
stream was resampled at 44.1 KHz and interpolated.
See Figure 1 for the setup used.
The pre-processed pupil dilation data controlled the
amplitude of a sine wave (ν=440Hz, Figure 2). Pupil
dilations linearly increased, whereas constrictions
linearly decreased the volume of the sine wave. Initial
pilot testing suggested that this yielded more clearly
discernable results and was less intrusive than other
combinations that varied in type, i.e., mapping to
volume and frequency, and in wave form, i.e., noise,
sine, saw, and square waves. The sounds were
presented through headphones. Dimmed lighting
conditions (room, screen) were used to minimize
interference from the pupil’s light reflex [36].
Figure 1 Setup used in the
experiment. Participants solve a
mathematics problems presented
on a computer screen; while
listening to a sonification of their
own pupil dilation data that is
captured and streamed by an
eye-tracker in real-time.
Figure 2 Illustration of the pupil
dilation-to-sound mapping. The
size of the pupil (A1) is mapped in
real-time to the amplitude (A2) of
a continuously playing sine wave
(ν=440Hz), and presented to the
user during a problem solving
task.
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
239
Problem solving task
A mathematics problem solving task was developed
during which the sonification was used. Mathematics
involves cognitions that associate with pupil dilation,
e.g., attentional focus [8]; and can be manipulated by
varying the mathematics problems difficulty [13].
Participation required (past) involvement in higher
education so that three difficulty levels could be based
on educational norms: easy, problems people can solve
after primary school; medium, problems people can
solve after high school; hard, problems at a behavioural
sciences undergraduate level. The difficulty of the
problems was gradually increased, and interspersed
with easy problems (Table 1). Pilot tests suggested this
affected the pupil dilation-cognition link more
effectively compared to e.g., random presentation.
The first part of the task consisted of twelve
mathematics problems (Table 1 and cognition. , T1). No
information was given about the link between the
sounds heard and cognition. This was done to explore if
and what correlations between the sound and cognitive
processes emerge spontaneously. The second part was
preceded by the instruction that there exists a link
between “…the amount of cognitive resources allocated
to the task and the changes in volume of the sound”.
Participants were then instructed to try to solve an
additional five problems (Table 1, T2). This was done to
explore whether instruction enabled experiencing a
given link between the sounds and cognition.
Questionnaire
To explore whether changes in the sounds generated
by the pupil dilation sonification elicited awareness
about related cognitions spontaneously, we asked
participants to elaborate on whether, what, and when
correlations between their behaviour and the sounds
emerged after the first part of the task (T1). Responses
were clustered into categories based on (a) type of
response (e.g., deep thinking’, ‘when staring’), and (b)
whether the responses related to cognitions (e.g.,
‘concentrating’), oculomotor behaviours (e.g.,
‘blinking’), or no associations. After the second part
(T2) we explored whether changes in the sounds
brought changes in the instructed relationship between
the sound and cognition into awareness, by inquiring
about the participant’s ability to perceive the instructed
correlation. The questions are presented in Table 2.
The qualities of the sound itself may also affect the
participants. We thus asked to describe the sound
itself. Responses were clustered by (a) type (e.g.,
‘annoying’), and (b) valence (e.g., ‘negative’). This was
done to gather data to help improve future versions of
the sonification. To gain some insight into the socio-
demographics of the sample, participants were asked to
report their age, gender, and education level, but also
mathematical ability (very poor=1, very good=10).
Procedure
Participants were seated behind a computer screen and
introduced to the study. Information that could reveal
the purpose of the study was withheld. Participants
reported their socio-demographics and signed informed
consent. Task instructions followed, the eye-tracker (9-
point calibration) and sonification were calibrated.
Participants put on headphones and adjusted loudness
to comfortable levels. They then engaged in part one
and two of the mathematics problem solving task while
hearing the sounds generated by the pupil dilation
sonification. After part one and two they filled in the
questionnaires, were debriefed, and sent on their way.
T
1
1
2
3
4
5
6
7
8
9
E
M
M
M
E
M
M
H
E
10
11
12
H
H
E
T
2
1
2
3
4
5
E
M
H
H
E
Table 1 Presentation order of
mathematics problems in part
one (T1) and part two (T2) of the
task. Presentation index
(numbers) and difficulty level
(E=easy, M=medium, H=hard).
T1
Did you notice anything
particular about the sound
If so, what behaviours
correlated with the sound
When did you notice a
correlation between the
sound and your behaviour
T2
Were you able to correlate
the change in sound with
your state of mind now
that you know that it was
directly influenced by it
Table 2 Questions asked after
part one (T1) and part two (T2) of
the Presentation order of
mathematics problem solving
task.
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
240
Results
The results suggest that people can correlate sounds
generated based on pupil dilation with their cognitions.
Figure 3 Correlations reported between the sounds generated
by the pupil dilation sonification and behaviours.
Eighteen participants (90%) reported they noticed a
link between the sound and their behaviour, two (10%)
did not. Eleven (55%) suggested a correlation with a
cognitive processes, including thinking, concentration,
focus, and dealing with complex questions (Figure 3);
five (25%) correlated the sound to oculomotor
behaviours, such as looking away, staring, and moving
the eyes quickly; whereas four participants (20%) did
not uncover such correlations spontaneously. Relatedly,
fifteen participants (78%) reported that the sounds
changed when there was a change in cognition or
related behaviour (Figure 4); whereas two participants
(11%) thought the sound changed due to oculomotor
activity only; and two (11%) did not think the sound
changed at any particular moment. When instructed
that the sounds correlate with the amount of cognitive
resources allocated, fifteen participants (75%)
experienced this correlation, whereas five (25%) did
not. These findings indicate that some people are likely
to correlate the sounds to oculomotor behaviours,
whereas for some it may not be possible to experience
such correlations at all. However, most people are able
to correlate the pupil dilation sonification sounds with
cognitive processes spontaneously and by instruction,
which suggests pupil dilation sonification has the
potential to help elicit metacognitive awareness.
Figure 4 Indication of when the sounds generated by the pupil
dilation sonification changed during the problem solving task.
However, there were issues with the quality of the
sounds generated by the pupil dilation sonification
(Figure 5). Fifteen negative responses (68%), e.g., that
the sound was annoying or distracting, seven neutral
responses, e.g., that it sounds like a radar (32%), and
no positive responses (0%) were reported. Despite its
apparent ability help elicit metacognitive awareness,
the findings also suggest issues with the use of (our)
pupil dilation sonification, which requires further study.
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
241
Figure 5 Experiences and attribution to the sound itself.
Discussion and conclusion
In the present study we explored the potential of pupil
dilation sonification for eliciting metacognitive
awareness. The results showed that most people are
able to correlate the pupil dilation sonification sounds
with cognitive processes spontaneously and by
instruction. However, some tend to correlate the
sounds with oculomotor behaviours, whereas a small
minority was not able to perceive a correlation at all.
This preliminary evidence suggests that pupil dilation
sonification could potentially be a novel way to
augment metacognition, but more work is needed.
There are of course limitations. For example, the full
range of possible correlations between the pupil dilation
sonification sounds and cognition has not yet been
explored because a) the mathematics task mostly
manipulated task-evoked rather than task-independent
pupil dilations [8], and b) light can cause larger
dilations than cognition, biasing correlations between
sound and oculomotor behaviour [3]. Second, negative
responses to the sound itself may have influenced
metacognitive awareness (Figure 5), introducing
uncertainty about the validity of our approach. Third,
caution about any conclusions about using pupil dilation
sonification to eliciting metacognitive awareness and
augment metacognition is needed as no control group
was used and task performance was not measured.
Future work will address these limitations. First, we will
explore how to mitigate negative responses to the
initial version of the sonification, by a) exploring
sparsity, e.g., by sonifying change in pupil dilation, b)
using non-intrusive sounds, e.g., café-sounds or
waves; but c) it might also require translating pupil
dilations to other senses, e.g., haptics or vision [9].
Second, we will explore how to improve the elicitation
of metacognitive awareness, by a) developing a
sonification that emphasizes differences in task-evoked
and task-independent pupil dilations, which signify
different cognitions [8]; and (b) test these in a wider
range of tasks fully the range correlations between
pupil dilation and cognition. Third, replication is advised
with a control (fake sonification) to further confirm the
theoretical assumptions that underlie our approach.
The contribution of this study is thus preliminary
evidence that making pupil dilation audible can elicit
metacognitive awareness. Given the role of such
awareness in task performance, it may have potential
to develop a novel technological approach that can
augment metacognition to help people to get the most
out of their own cognitive capabilities.
References
1. Baker, L. (2010). Metacognition. International
encyclopedia of education, 204-210.
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
242
2. Beatty, J., & Wagoner, B. L. (1977). Pupillometric
signs of brain activation vary with level of cognitive
processing (No. TR-9). University of California.
3. Campbell, F. W., & Gubisch, R. W. (1966). Optical
quality of the human eye. The Journal of
Physiology, 186(3), 558-578.
4. Cycling ’74 (2017). Retrieved at 26-07-2017 from
https://cycling74.com/products/max
5. de Rooij, A., Corr, P. J., & Jones, S. (2015).
Emotion and creativity: Hacking into cognitive
appraisal processes to augment creative ideation.
In Proceedings of the 2015 ACM SIGCHI
Conference on Creativity and Cognition, 265-274.
6. de Rooij, A., & Jones, S. (2015). (E) motion and
creativity: Hacking the function of motor
expressions in emotion regulation to augment
creativity. In Proceedings of the Ninth International
Conference on Tangible, Embedded, and Embodied
Interaction, 145-152.
7. de Rooij, A., van Dartel, M., Ruhl, A.,
Schraffenberger, H., van Melick, B., Bontje, M.,
Daams, M., & Witter, M. (2017). Sensory
Augmentation: Toward a Dialogue between the Arts
and Sciences. In Proceedings of the 6th EAI
International Conference: ArtsIT, Interactivity &
Game Creation.
8. Eckstein, M. K., Guerra-Carrillo, B., Singley, A. T.
M., & Bunge, S. A. (2017). Beyond eye gaze: What
else can eyetracking reveal about cognition and
cognitive development?. Developmental cognitive
neuroscience, 25, 69-91.
9. Ehlers, J., Strauch, C., Georgi, J., & Huckauf, A.
(2016). Pupil size changes as an active information
channel for biofeedback applications. Applied
psychophysiology and biofeedback, 41(3), 331-
339.
10. Filingeri, D., Fournet, D., Hodder, S., & Havenith,
G. (2014). Why wet feels wet? A neurophysiological
model of human cutaneous wetness sensitivity.
Journal of neurophysiology, 112(6), 1457-1469.
11. Flavell, J. H. (1979). Metacognition and cognitive
monitoring: A new area of cognitivedevelopmental
inquiry. American psychologist, 34(10), 906.
12. Fleming, S. M., & Frith, C. D. (Eds.). (2014). The
cognitive neuroscience of metacognition. Springer
Science & Business Media.
13. Fuchs, L. S., Compton, D. L., Fuchs, D., Paulsen,
K., Bryant, J. D., & Hamlett, C. L. (2005). The
prevention, identification, and cognitive
determinants of math difficulty. Journal of
Educational Psychology, 97(3), 493
14. Gerjets, P., Walter, C., Rosenstiel, W., Bogdan, M.,
& Zander, T. O. (2014). Cognitive state monitoring
and the design of adaptive instruction in digital
environments: lessons learned from cognitive
workload assessment using a passive brain-
computer interface approach. Frontiers in
neuroscience, 8, 385.
15. Gilzenrat, M. S., Nieuwenhuis, S., Jepma, M., &
Cohen, J. D. (2010). Pupil diameter tracks changes
in control state predicted by the adaptive gain
theory of locus coeruleus function. Cognitive,
Affective, & Behavioral Neuroscience, 10(2), 252-
269.
16. Goldberg et al., (2014). Enhancing self-regulated
learning through metacognitively-aware intelligent
tutoring systems. In Proceedings of the 11th
International Conference of the Learning Sciences.
17. Kahneman, D., & Beatty, J. (1966). Pupil diameter
and load on memory. Science, 154(3756), 1583-
1585.
18. Kleitman, S., & Stankov, L. (2007). Self-confidence
and metacognitive processes. Learning and
Individual Differences, 17(2), 161-173.
19. McLaren, J. W., Erie, J. C., & Brubaker, R. F.
(1992). Computerized analysis of pupillograms in
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
243
studies of alertness. Investigative ophthalmology &
visual science, 33(3), 671-676.
20. Moore, B. C. (2012). An introduction to the
psychology of hearing. Brill.
21. Novich, N. (2015). Sound-to-touch sensory
substitution and beyond. Doctoral thesis. Rice
University.
22. Ocumpaugh, J., Baker, R., Gowda, S., Heffernan,
N., & Heffernan, C. (2014). Population validity for
Educational Data Mining models: A case study in
affect detection. British Journal of Educational
Technology, 45(3), 487-501.
23. Omurtag, A., Aghajani, H., & Keles, H. O. (2017).
Decoding human mental states by whole-head
EEG+FNIRS during category fluency task
performance. Journal of neural engineering, pre-
print.
24. Reimer, J., McGinley, M. J., Liu, Y., Rodenkirch, C.,
Wang, Q., McCormick, D. A., & Tolias, A. S. (2016).
Pupil fluctuations track rapid changes in adrenergic
and cholinergic activity in cortex. Nature
communications, 7, 13289.
25. Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K.
R. (2011). Improving students’ help-seeking skills
using metacognitive feedback in an intelligent
tutoring system. Learning and instruction, 21(2),
267-280.
26. Rondeel, E. W., Van Steenbergen, H., Holland, R.
W., & van Knippenberg, A. (2015). A closer look at
cognitive control: differences in resource allocation
during updating, inhibition and switching as
revealed by pupillometry. Frontiers in human
neuroscience, 9, 494.
27. Schraw, G. (1998). Promoting general
metacognitive awareness. Instructional Science.
26, 113-125.
28. Smallwood, J., Brown, K. S., Tipper, C., Giesbrecht,
B., Franklin, M. S., Mrazek, M. D., & Schooler, J. W.
(2011). Pupillometric evidence for the decoupling of
attention from perceptual input during offline
thought. PloS one, 6(3), e18298.
29. Spielberger, C. D., Anton, W. D., & Bedell, J.
(2015). The nature and treatment of test anxiety.
Emotions and anxiety: New concepts, methods,
and applications, 317-344.
30. Storbeck, J., & Clore, G. L. (2008). Affective
arousal as information: How affective arousal
influences judgments, learning, and memory.
Social and personality psychology compass, 2(5),
1824-1843.
31. Swanson, H. L. (1992). The relationship between
metacognition and problem solving in gifted
children. Roeper Review, 15(1), 43-48.
32. The Eye Tribe (2017). Retrieved at 26-07-2017
from http://www.theeyetribe.com
33. Thiede, K. W., Anderson, M., & Therriault, D.
(2003). Accuracy of metacognitive monitoring
affects learning of texts. Journal of educational
psychology, 95(1), 66.
34. Walcher, S., Körner, C., & Benedek, M. (2017).
Looking for ideas: Eye behavior during goal-
directed internally focused cognition.
Consciousness and Cognition, 53, 165-175.
35. Walker, B. N., & Nees, M. A. (2011). Theory of
sonification. The sonification handbook, 9-39.
36. Woodhouse, J. M., & Campbell, F. W. (1975). The
role of the pupil light reflex in aiding adaptation to
the dark. Vision research, 15(6), 649-653.
Work-in-Progress
TEI 2018, March 18–21, 2018, Stockholm, Sweden
244
de_rooij_tei2018.pdf
423.78 KB
  • Article
    Full-text available
    Emotions can augment or diminish creativity. This presents an opportunity for designing interactive systems that augment creativity. However, how to design such systems is still an open problem. In this position paper, this problem is addressed theoretically. Systems should regulate, rather than cause, emotional responses during the idea generation process; which can be achieved by embedding features of the emotion components that underlie the emotion-creativity link directly into the user-system interactions. Two case studies are discussed that demonstrate successful regulation via technology of a link between positive emotion and generating original ideas. Although these verify the theory's usefulness, further debate needs to uncover how to make practical application possible.
  • Conference Paper
    Full-text available
    People sense the world by exploiting correlations between their physical actions and the changing sensory input that results from those actions. Interfaces that translate non-human sensor data to signals that are compatible with the human senses can therefore augment our abilities to make sense of the world. This insight has recently sparked an increase in projects that explore sensemaking and the creation of novel human experiences across scientific and artistic disciplines. However, there currently exists no constructive dialogue between artists and scientists that conduct research on this topic. In this position paper, we identify the theory and practice of sensory augmentation as a domain that could benefit from such a dialogue. We argue that artistic and scientific methods can complement each other within research on sensory augmentation and identify six thematic starting points for a dialogue between the arts and sciences. We conducted a case study to explore these conjectures, in which we instigated such a dialogue on a small scale. The case study revealed that the six themes we identified as relevant for a dialogue on sensory augmentation emerge rather spontaneously in such a dialogue and that such an exchange may facilitate progress on questions that are central to the theory and practice of sensory augmentation. Overall, this position paper contributes preliminary evidence for the potential of, and a starting point for, a dialogue between the arts and sciences that advances our understanding of sensory augmentation and the development of applications that involve it.
  • Article
    Full-text available
    Objective: Concurrent scalp electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS), which we refer to as EEG+fNIRS, promises greater accuracy than the individual modalities while remaining nearly as convenient as EEG. We sought to quantify the hybrid system's ability to decode mental states and compare it with its unimodal components. Approach. We recorded from healthy volunteers taking the category fluency test and applied machine learning techniques to the data. Main results. EEG+fNIRS's decoding accuracy was greater than that of its subsystems, partly due to the new type of neurovascular features made available by hybrid data. Significance. Availability of an accurate and practical decoding method has potential implications for medical diagnosis, brain computer interface design, and neuroergonomics.
  • Article
    Full-text available
    Humans have a highly developed visual system, yet we spend a high proportion of our time awake ignoring the visual world and attending to our own thoughts. The present study examined eye movement characteristics of goal-directed internally focused cognition. Deliberate internally focused cognition was induced by an idea generation task. A letter-by-letter reading task served as external task. Idea generation (vs. reading) was associated with more and longer blinks and fewer microsaccades indicating an attenuation of visual input. Idea generation was further associated with more and shorter fixations, more saccades and saccades with higher amplitudes as well as heightened stimulus-independent variation of eye vergence. The latter results suggest a coupling of eye behavior to internally generated information and associated cognitive processes, i.e. searching for ideas. Our results support eye behavior patterns as indicators of goal-directed internally focused cognition through mechanisms of attenuation of visual input and coupling of eye behavior to internally generated information.
  • Article
    Full-text available
    Rapid variations in cortical state during wakefulness have a strong influence on neural and behavioural responses and are tightly coupled to changes in pupil size across species. However, the physiological processes linking cortical state and pupil variations are largely unknown. Here we demonstrate that these rapid variations, during both quiet waking and locomotion, are highly correlated with fluctuations in the activity of corticopetal noradrenergic and cholinergic projections. Rapid dilations of the pupil are tightly associated with phasic activity in noradrenergic axons, whereas longer-lasting dilations of the pupil, such as during locomotion, are accompanied by sustained activity in cholinergic axons. Thus, the pupil can be used to sensitively track the activity in multiple neuromodulatory transmitter systems as they control the state of the waking brain.
  • Article
    Full-text available
    This review provides an introduction to two eyetracking measures that can be used to study cognitive development and plasticity: pupil dilation and spontaneous blink rate. We begin by outlining the rich history of gaze analysis, which can reveal the current focus of attention as well as cognitive strategies. We then turn to the two lesser-utilized ocular measures. Pupil dilation is modulated by the brain’s locus coeruleus-norepinephrine system, which controls physiological arousal and attention, and has been used as a measure of subjective task difficulty, mental effort, and neural gain. Spontaneous eyeblink rate correlates with levels of dopamine in the central nervous system, and can reveal processes underlying learning and goal-directed behavior. Taken together, gaze, pupil dilation, and blink rate are three non-invasive and complementary measures of cognition with high temporal resolution and well-understood neural foundations. Here we review the neural foundations of pupil dilation and blink rate, provide examples of their usage, describe analytic methods and methodological considerations, and discuss their potential for research on learning, cognitive development, and plasticity.
  • Article
    Pupil size is usually regarded as a passive information channel that provides insight into cognitive and affective states but defies any further control. However, in a recent study (Ehlers et al. 2015) we demonstrate that sympathetic activity indexed by pupil dynamics allows strategic interference by means of simple cognitive techniques. Utilizing positive/negative imaginings, subjects were able to expand pupil diameter beyond baseline variations; albeit with varying degrees of success and only over brief periods. The current study provides a comprehensive replication on the basis of considerable changes to the experimental set-up. Results show that stricter methodological conditions (controlled baseline settings and specified user instructions) strengthen the reported effect, whereas overall performance increases by one standard deviation. Effects are thereby not restricted to pupillary level. Parallel recordings of skin conductance changes prove a general enhancement of induced autonomic arousal. Considering the stability of the results across studies, we conclude that pupil size information exceeds affective monitoring and may constitute an active input channel in human-computer interaction. Furthermore, since variations in pupil diameter reliably display self-induced changes in sympathetic arousal, the relevance of this parameter is strongly indicated for future approaches in clinical biofeedback.
  • Article
    Metacognition is the awareness and control of one's own cognition. The construct of metacognition has been useful to researchers and educators seeking an explanation for why some students fare better in school than others. The consistent finding in over 30 years of research is that more-successful students exhibit higher levels of metacognitive knowledge about a given domain and are more skilled at regulating their cognitive processes than less-successful students. This article addresses issues regarding definitions, origins, measurement, and intervention. It highlights prominent research findings in the academic domains of reading, writing, mathematics, and science.
  • Article
    Full-text available
    Metacognitive monitoring affects regulation of study, and this affects overall learning. The authors created differences in monitoring accuracy by instructing participants to generate a list of 5 keywords that captured the essence of each text. Accuracy was greater for a group that wrote keywords after a delay (delayed-keyword group) than for a group that wrote keywords immediately after reading (immediate-keyword group) and a group that did not write keywords (no-keyword group). The superior monitoring accuracy produced more effective regulation of study. Differences in monitoring accuracy and regulation of study, in turn, produced greater overall test performance (reading comprehension) for the delayed-keyword group versus the other groups. The results are framed in the context of a discrepancy-reduction model of self-regulated study. Many models of self-regulated learning can be classified as discrepancy-reduction models (e.