Eye Tracking for Evaluating an AR-based Learning System on Monocotyledons/Dicotyledons

Abstract

Eye movement tracking and augmented reality (AR) technologies are regarded of more and more importance nowadays. Due to the continually improvement of measurement of eye movement, it is applied to many research areas such as the processes of reading and cognition. However, the study on eye tracking when manipulating AR systems is still in infancy. Therefore, this work aims at combing AR and eye movement evaluation. In the experiment, the participants were divided into two groups: domain experts group and IT skilled group, with each group 30 persons. All participants manipulated AR teaching aids on performing tasks about learning monocotyledons/dicotyledons recognition. This study used eye tracking to measure the operation of the AR task time and number of operating times. Hope to have a better understanding of the use of the AR. This study analyzed the relationship among the eye movement conditions, workload level, and the system usability via the use of scales through the system scale (SUS), the Task Load Index NASA-TLX scale, and several types of eye movement evaluation.

Full text

T. Hirashima et al. (Eds.) (2011). Proceedings of the 19th International Conference on Computers in
Education. Chiang Mai, Thailand: Asia-Pacific Society for Computers in Education
Eye Tracking for Evaluating an AR-based
Learning System on
Monocotyledons/Dicotyledons
Ching-Ju CHAOab*, Hao-Chiang Koong LINa, Cheng-Hung WANGa, Min-Chai HSIEHa
a Dept. of Information and Learning Technology, National University of Tainan, Taiwan
b Tung Fang Design University, Taiwan
*chingju@mail.tf.edu.tw
Abstract: Eye movement tracking and augmented reality (AR) technologies are regarded of
more and more importance nowadays. Due to the continually improvement of measurement of
eye movement, it is applied to many research areas such as the processes of reading and
cognition. However, the study on eye tracking when manipulating AR systems is still in
infancy. Therefore, this work aims at combing AR and eye movement evaluation. In the
experiment, the participants were divided into two groups: domain experts group and IT skilled
group, with each group 30 persons. All participants manipulated AR teaching aids on
performing tasks about learning monocotyledons/dicotyledons recognition. This study used
eye tracking to measure the operation of the AR task time and number of operating times.
Hope to have a better understanding of the use of the AR. This study analyzed the relationship
among the eye movement conditions, workload level, and the system usability via the use of
scales through the system scale (SUS), the Task Load Index NASA-TLX scale, and several
types of eye movement evaluation.
Keywords: Augmented Reality (AR), Eye Movement Tracking
1. Introduction
The oculomotor measure is applied in reading research generally. As the progress of
technology, new instruments have more subtle and precise tracks on the eyes. There’re many
research infers reading and other trace of cognition process from the experiment of
oculomotor.[1][2] Since AR is regarded of importance and widely applied to many areas
nowadays, this work attempts to explore the user behaviors when manipulating AR interfaces
so that the designers can establish a better AR system. Besides, the study on eye tracking
when manipulating AR systems is still in infancy. Therefore, this work aims at combing AR
and eye movement evaluation[3][4]. This work will construct AR teaching aids on learning
monocotyledons/dicotyledons recognition. this study will analyze the relationship among the
eye movement conditions, workload level, and the system usability via the use of scales
through the system scale (SUS), the Task Load Index NASA-TLX scale, and several types of
eye movement evaluation.
2. The Method of Research
The structure of this research is showed in picture 1. This paper addresses the
following research questions:Q1:Is there a significant difference in eye movement between
users with various learning backgrounds when manipulating AR systems? Q2:Is there a
significant difference in task loads between users with various learning backgrounds when
using AR to learn monocotyledon/dicotyledon recognition? Q3:Is there a significant

T. Hirashima et al. (Eds.) (2011). Proceedings of the 19th International Conference on Computers in
Education. Chiang Mai, Thailand: Asia-Pacific Society for Computers in Education
difference in perceived system usability between users with various learning backgrounds
when using AR to learn monocotyledon/dicotyledon recognition?
Figure 1. Research Framework.
3. Experimental Results
In this study, the eye tracker was used to measure the eye movements of subjects when
they were performing the experiment tasks. The system diagram is shown as in Figure 2.
Figure2. System diagram
When viewing marker-1 and marker-2, the average fixation duration of the IT skilled
group is significantly longer than that of the expert domain group, as shown in Table 1. In
Table 2, the average fixation count of the IT skilled group is significantly higher than that of
the expert domain group. In Table3, when viewing these four makers, the values of average
fixation duration of the IT skilled group are closer. It conveys that the IT skilled group takes
the similar strategies when observing these four cards. When the IT skilled group views these
markers, the eyes move with saccadic jumps; the eye movements direct at the target direction.
Table 4 indicates that the saccade counts decrease in both groups when viewing these four
cards, where the saccade count of the domain expert group is less than that of the IT skilled
group. Besides, there are significant differences in saccade counts between two groups when
viewing marker-1and marker-2.
As to the task loads, Table 5 indicates that the average values of mental load,
physiological load, time load, and frustration level of the domain expert group are higher than
the IT skilled group; whilst the energy consumption and performance show reverse results.
The SUS result is shown in Table 6. It indicates the IT skilled group obtains a higher
SUS score than the domain expert group. However, there is no significant difference. As to
the skewness and kurtosis, the domain expert group has right-skewed normal peak, whilst the
IT skilled group has left-skewed normal peak.
Table 1. T-test on Fixation Duration
IT Skilled
Group
Domain Expert
Group
Variables
Average
Average
P Value
Marker-1
56566.80
40247.77
.000*
Marker-2
38806.27
24841.03
.000*
Marker-3
24153.67
23844.63
.946
Marker-4
18447.17
19680.00
.663
*p<0.05
Table 2. T-test on Fixation Count
IT Skilled
Group
Domain Expert
Group
Variables
Average (ms)
Average (ms)
P Value
Marker-1
1468.73
650.47
.000*
Marker-2
957.53
406.03
.000*
Marker-3
283.47
231.50
.324
Marker-4
248.10
180.47
.196
*p<0.05
Q2
Q1
Q3
Augmented Reality
Subjects
Attention
Eye Movement Data
NASA-Task Load
Index (NASA-TLX)
Research Group
IT skilled group
System Usability Scale
(SUS)

T. Hirashima et al. (Eds.) (2011). Proceedings of the 19th International Conference on Computers in
Education. Chiang Mai, Thailand: Asia-Pacific Society for Computers in Education
Table 3. T-test on Average Fixation Duration
IT Skilled
Group
Domain Expert
Group
Variables
Average (ms)
Average (ms)
P Value
Marker-1
33.30
90.63
.169
Marker-2
33.43
417.27
.113
Marker-3
43.50
38.33
.534
Marker-4
35.80
340.43
.113
*p<0.05
Table 4. T-test on saccade count
IT Skilled
Group
Domain Expert
Group
Variables
Average
Average
P Value
Marker-1
1765.33
1052.00
.000*
Marker-2
1137.20
686.41
.000*
Marker-3
671.23
686.16
.910
Marker-4
519.67
467.54
.533
*p<0.05
Table 5. Descriptive statistics and t-test on average task loads
IT skilled group
Domain expert group
Variables
Average
Skewness
Kurtosis
Average
Skewness
Kurtosis
p Value
Mental load
2.20
.293
.261
2.83
-.132
2.150
.002*
Physiological load
2.03
.763
.018
2.33
.749
.058
.249
Time load
2.30
.555
-.212
2.56
.001
-.214
.278
Energy consumption
2.16
.232
-.786
2.06
.338
-.170
.651
Performance
3.63
.692
-.699
3.13
1.217
3.711
.004*
Frustration level
2.06
.543
-.140
2.33
.226
-.498
.243
*p<0.05
Table 6. SUS results and t-test
Average
Max
Min
Skewness
Kurtosis
p value
IT skilled group
66.91
87.5
27.5
-1.353
-.157
.196
domain expert group
62.91
85
40
2.406
-.165
*p<0.05
4. Conclusion and Discussion
In this study, because the research is designed to match the operation of remote eye
tracker, so the design of expandable virtual reality experiment is more easier. So we could
suggest that the expandable virtual reality could add more Interaction. Now we have two
groups of subjects of this research, IT skilled group and domain expert group. And it have a
possibility to add the third group, like about the academic group in the high school, to
increase the difference between the subjects. Or we could classify the different subjects
according to learning or recognition style. Besides, the 3D plant model is in an acceptable
range for subjects, but it’s not so subtle enough. So we could cooperate with art talented
person to show a visual effect subjects expect to add the interest to the experiment.
References
[1] Radach, R., & Kennedy, A. (2004). Theoretical perspectives on eye movements in reading: Past
controversies, current issues, an agenda for future research. European Journal of Cognitive Psychology,
16, 3-26.
[2] Rayner, K., & Juhasz, B. J. (2004). Eye movement in reading: Old questions and new directions. European
Journal of Cognitive Psychology,16, 340-352.
[3] Lin, H. C. K. & Li, Francis (2008), Employing Max/MSP/Jitter and sobel operations to create digital art
works based on the interaction among images, sounds, and MIDI music, JSTS, Journal of Scientific and
Technological Studies, pp.15-28, Volume 42, Number 2, Oct.
[4] Lin, H. C. K. & Li, Francis (2009), The combination of information technologies and digital arts - from the
interaction between images and sounds to the “empathy” status, Proc. of ITIA09, the 2009 Conference
on Information Technology and Industrial Application, in Technology and Science Institute of Northern
Taiwan, Beitou, Taipei, Taiwan, June 13.