DESIGN AND EVALUATION OF TWO MOBILE COMPUTER-ASSISTED PRONUNCIATION TRAINING TOOLS TO FAVOR AUTONOMOUS PRONUNCIATION TRAINING OF ENGLISH AS A FOREIGN LANGUAGE

Full text

DESIGN AND EVALUATION OF TWO MOBILE COMPUTER-
ASSISTED PRONUNCIATION TRAINING TOOLS TO FAVOR
AUTONOMOUS PRONUNCIATION TRAINING OF ENGLISH AS A
FOREIGN LANGUAGE
C. Tejedor-García, V. Cardeñoso-Payo, D. Escudero-Mancebo
University of Valladolid, ECA-SIMM Research Group (SPAIN)
Abstract
Foreign language (L2) learning systems usually address the development of skills related to lexical or
grammatical knowledge and proficiency. Pronunciation training, although it is a key condition for fluent
communication in an L2, has not received the same amount of attention. The constant quality
improvement of automatic speech recognition (ASR) and text-to-speech (TTS) has given way to a
substantial increase in the number of applications which include some form of training based on the use
of these technologies. There are not many contributions on the experimental evaluation of the learning
effectiveness and degree of usability and user experience of computer-assisted pronunciation training
(CAPT) tools using ASR or TTS or both. Also, there is still some debate on the feasibility of these tools
as learning aids, since they are still seen as an imperfect and unfair alternative to the human expert
teacher. In this work, we present the design and qualitative evaluation of two different versions of a
mobile CAPT tool designed to help pronunciation training of English as L2 for native Spanish students.
The tool includes exposure, perception, and production tasks on a set of minimal pairs. In particular,
175 university-level native Spanish learners of English as L2 tested took part in the experimentation.
The interaction with the CAPT tool has been automatically recorded by file logs. Questionnaires and
focus groups were also used to gather data. Results show a significant pronunciation improvement
among the learners at the end of the experimentation, as well as a positive experience of the CAPT tool
by learners, teachers, and instructors. The large dataset gathered will help researchers, teachers, and
instructors to design future improved versions of the tool.
Keywords: computer-assisted pronunciation training (CAPT), autonomous learning, mobile learning
tools, learning environments
1 INTRODUCTION
Current globalization leads to a growing demand in second language (L2) acquisition needs. Even
though one-to-one tutoring and in-classroom lessons are still the first choice for L2 learners, computer-
assisted language learning (CALL) is becoming a very valuable resource as language and speech
processing technologies progress [1]. Training L2 pronunciation not only enhances speech
comprehensibility and intelligibility, but also helps to acquire a native-like pronunciation [2], [3]. In
particular, computer-assisted pronunciation training (CAPT) is a relevant sub-area of CALL focused on
L2 pronunciation teaching, which offers a number of benefits to supplement classroom-based
pronunciation training, integrates speech technology, and can be included in software applications for
smart devices [2].
Nowadays, speech technology can contribute to the teaching and assessment of L2 pronunciation [4].
Although it has fueled much debate, it has led to a high number of interesting practical proposals [3],
[5], [6]. In particular, current automatic speech recognition (ASR) and text-to-speech (TTS) systems
have reached high quality levels in terms of usability, affordability and performance. For instance,
Google ASR has achieved a word accuracy rate of 95% for the English language, reaching the threshold
of human accuracy [7], and Google TTS has reached natural speech [8]. In short, both quality and
availability are clearly on the side of those interested in applying ASR and TTS to L2 teaching in CAPT.
Learning games engage and motivate users to train [9], [10]. Well-designed games deploy techniques
that encourage players to achieve and maintain a state of intense concentration and full involvement,
especially when challenges are closely geared to ability [11]. The potential of games to create effective
social practices provides means for users to participate in interpretive communities for learning [12],
[13]. Learning games include gamification, which refers to the “use of game design elements, such as
points, leaderboards, or badges within non-game contexts” [14]. In the particular case of language

learning, Duolingo marked a turning point in the industry by introducing gamification elements with a
clear intention to enrich the users’ learning experience [15]. Before Duolingo, commercial language
learning tools, such as Sanako or Rosetta Stone developed strategies based on course content
selection and the design of interfaces that targeted either academic institutions or self-training.
In previous works, we have reported on the results about the motivation, performance, and learning
outcomes of two versions of a novel CAPT tool. First, we focused on a gamified version of the tool [16].
That version included game elements and a social competition which resulted in a higher game intensity
and greater playing regularity than the original version of the game [17]. The most active players also
achieved better pronunciation results. In the second version [18], we focused on guiding learners
through a common goal varying the activities to perform depending on their results under a predefined
protocol. We also eliminated the possibility of users selecting their own training activities and the game
elements. In this case, results showed significant pronunciation improvement among the learners who
used the CAPT tool, as well as a correlation between human rater’s assessment of post-tests and the
automatic CAPT assessment of users. In addition to this previous experimentation, in this work we
complement our previous results by reporting a high degree of usability and user experience by means
of questionnaires and focus group sessions in both versions of the CAPT tool.
2 METHODOLOGY
2.1 Participant Enrollment
A total of 165 native Spanish undergraduate students from the University of Valladolid (Spain)
participated in the game version of the CAPT tool. 99 (60.0%) were female and 66 (40.0%) were male.
The average age was 21 years (M = 21.3, SD = 1.96). All 165 participants took a pre-test, played with
the tool for 24 days, and took a final post-test in which were included two questionnaires about
pronunciation level self-concept and user experience. Besides, four focus group sessions with 16
different learners in each one were carried out after concluding the experiment (see specific details in
Tejedor-García et al. [16]).
On the other hand, 10 native Spanish undergraduate students from the same course of English as L2
of the University of Valladolid (Spain) participated in the guided version of the CAPT tool. 2 (20.0%)
were female and 8 (80.0%) were male. The average age was 23 years (M = 23.6, SD = 2.06). After
taking a pre-test in one session, working with the CAPT tool in another three sessions of 60 minutes in
three different days each one, and taking a post-test in another session, learners filled an online
questionnaire about their user experience. Finally a focus group session was carried out with the 10
participants (see specific details in Tejedor-García et al. [18]).
2.2 CAPT Tools Description
Both versions of the CAPT tool allowed learners to train the pronunciation of English sounds. Together
with stress and rhythm in connected speech, and prominence and intonation in discourse, these two
aspects are the main components of the sound system to be learned in L2 acquisition [19]. Non-accurate
production and discrimination of vocalic and consonant sounds could become a fossilized issue for non-
native speakers, which is difficult to improve without specific training [20]. Our previous results have
shown that the use of ASR and TTS technology, providing immediate audio and visual feedback,
contributes to improving the pronunciation competences of users when they are integrated into
appropriate methodologies [16], [18].
Our CAPT tools rely on the use of a set of minimal pairs, that is, two words frequently monosyllabic, that
are identical except for one sound, which changes their meanings completely. They naturally can be
integrated in teaching pronunciation methods [21], [22]. The same activities are presented in both
versions of the CAPT tool, based on the cycle exposure-perception-production [23], [24]. First, exposure
activities with minimal pairs, synthesizing both words in the pair, each repetition slower, and allowing
the learner to directly experience the perceptive differences between two particular sounds. Second,
perception activities in which the student must decide to which word of the pair a given synthesized
audio corresponds. Finally, production activities, in which learners must pronounce the words of the
minimal pair correctly so that the ASR system will accept them.
In the game version of the CAPT tool, users challenge other participants in order to get high scores and
climb up a leaderboard. They also can train individually specific sounds without obtaining points for the
leaderboard [16]. On the other hand, in the guided version users train with the CAPT tool with no
interaction with other participants. The CAPT tool protocol imposes the phonemes to be practiced and

the order in which they are confronted in different sessions. It also adapts to the user's performance
since it incorporates an element of progress control and monitoring [18].
2.3 Instruments and Variables
In the present study, four different data sources have been used to gather information:
1. Registration forms. They are used to get learner’s demographic information: age, gender, native
language (L1), academic level, and informed consent to analyze all gathered data.
2. Online questionnaires. Related to the usability, user experience, and pronunciation level self-
concept aspects affecting the use of the CAPT tools. This data is collected and saved into a secure web
server. The System Usability Scale (SUS) [25] measures the usability of a system and consists of 10
items which are evaluated on a 5-point Likert scale ranging from 1 "strongly disagree" to 5 "strongly
agree". Table 1 represents the 10 questions applied to this study. Note that half the statements are
positively worded and half are negatively worded. The results are distributed on a specific scale ranging
from 0 for "worst imaginable" to 100 for "best imaginable" [26]. SUS is an appropriate and robust usability
measure with easy application for the user [27]. It is often applied in design studies evaluating the
graphical user interfaces [28].
Table 1. SUS questionnaire [25] for the two versions of the CAPT tool.
Question
1
I think that I would like to use this tool frequently.
2
I found the tool unnecessarily complex.
3
I thought the tool was easy to use.
4
I think that I would need the support of a technical person to be able to use this tool.
5
I found the various functions in this tool were well integrated.
6
I thought there was too much inconsistency in this tool.
7
I would imagine that most people would learn to use this tool very quickly.
8
I found the tool very awkward to use.
9
I felt very confident using the tool.
10
I needed to learn a lot of things before I could get going with this tool.
In addition to the SUS questionnaire, five extra questions related to specific game dynamics were
provided to the participants in both versions of the CAPT tool on a 5-point Likert scale ranging from 1
"strongly disagree" to 5 "strongly agree".
3. Focus group sessions. The audio of the session is recorded and the most important quotes and
requests of the participants are written by a member of the research team by taking notes. This data is
carefully collected and saved into digital video and text files.
4. User’s interaction log files. The CAPT tool gathers data associated with all low level interaction
events and monitors all users’ results. This data is automatically saved into local log files and periodically
uploaded to a web server. Thorough experimental studies on pronunciation improvement and level of
activity extracted from these log-files have already been published in Tejedor-Garcia et al. [16] and
Tejedor-Garcia et al. [18], for the game and guided version of the CAPT tool, respectively.
3 RESULTS AND DISCUSSION
3.1 Game Version of the CAPT Tool
3.1.1 User Experience Questionnaire
Participants evaluated the usability of the CAPT tool (SUS questionnaire, see Table 1) with an overall
score of 78.3 (Standard Deviation, SD 13.9) out of 100, which suggests good usability (almost excellent).

In order to complement the SUS questionnaire, learners filled an extra questionnaire with five Likert-
scale questions related to the specific game dynamics of the game version of the CAPT tool (Table 2).
Table 2. User experience questionnaire of the game version of the CAPT tool. Me and Mo refer to
Median and Mode, respectively.
Question
Answers (%)
Me
Mo
1
2
3
4
5
EXTRA1
Game dynamics are well understood. It
does not take long to understand how to
play.
2.4
4.8
15.2
42.4
35.2
4.0
4
EXTRA2
Game dynamics are enjoyable.
3.6
15.2
27.3
40.6
13.3
4.0
4
EXTRA3
The challenge dynamics are easy to
understand.
2.4
8.5
17.6
37.0
34.5
4.0
4
EXTRA4
Access to the leaderboards is easy.
1.8
5.5
9.1
33.3
50.3
5.0
5
EXTRA5
Access to the training activities is easy.
0.0
3.0
18.2
41.8
37.0
4.0
4
Users found the game elements and their dynamics easy to understand and enjoy, and the access to
the game contents is also highly valued since the median and mode values of the five extra questions
are 4, except for the EXTRA4 question in which are 5. Although more than 53% of the users agree that
the game dynamics are enjoyable, these dynamics can be improved in future versions to obtain better
agreement values.
3.1.2 Pronunciation Level Self-Concept Questionnaire
The objective learning improvement related to perception and production skills reported in Tejedor-
Garcia et al. [16] also had a projection in the self-concept about the pronunciation level of the 165 users
who answered the pre-test and post-test (Table 3). The players who declared a lower level (Likert-values
from 1 to 3 in rows), declared a higher level after the experiment. In particular, there are statistically
significant differences in those who declared a level 3 in the post-test after having declared a level 2 in
the pre-test, 20.6% vs 67.6%, (p-value < 0.001, Chi-square test) and those who declared a level 4 after
having declared a previous level 3, 20.6% vs. 67.6% (p-value = 0.006, Chi-square test).
Although most users who initially declared a higher level (levels 4 or 5) maintained their opinion (80.
4%), 15.7% of those who declared level 4 in the pre-test declared a lower level in the post-test (p-value
< 0.001, Chi-square test). On the other hand, two out of the three learners who initially declared the
highest possible level (5) declared the same level in the post-test (reaching the final positions 1st and
6th in the leaderboard, respectively) and the participant who finally declared a level 4, finished in position
67 in the leaderboard.
Before carrying out the study, we hypothesized that it was possible that leaderboard classification and
the rest of game elements included in the application would help players to achieve a more objective,
precise, and well-founded perception of their true level of pronunciation. We considered the possibility
that most of the participants might have an overly optimistic or pessimistic view of their true skills in the
absence of such game elements, which would lead to an obvious contribution of these game elements.
The results of Table 3 clearly show that a high percentage of participants expressed a different
perception of their own level after 24 days of competitive practice.
Our recommendation is to use the leaderboard position as an element of motivation, but not as an
element of assessment. The leaderboard does not constitute an exact measure of the actual skill
developed by the player. The evaluation of a teacher who implements gamified techniques through
CAPT would have to be completed with other procedures such as official tests. However, the ability to
self-diagnose skill level, and the ability to reflect on one's own achievements and difficulties in the field
of learning are highly appreciated in the current drive by educators to discover new ways to motivate
students to encourage effective uptake. It would be useful in future research to continue exploring ways
in which a competition could facilitate self-diagnosis.

Table 3. Pronunciation level self-concept questionnaire of the game version of the CAPT tool.
Post-test
1
2
3
4
5
Total
Pre-
Test
1
0,0%
50,0%
50,0%
0,0%
0,0%
2
2
2,9%
20,6%*
67,6%*
8,8%
0,0%
33
3
0,0%
2,6%
59,2%*
36,8%*
1,3%
76
4
0,0%
0,0%
15,7%*
80,4%*
3,9%
51
5
0,0%
0,0%
0,0%
33,3%
66,7%
3
Total
1
9
77
73
5
165
Note. The symbol * refers to statistically significant differences between the pre-test
and post-test values (Chi-square test, 95% confidence level).
3.1.3 Focus Group Sessions
Finally, four focus group sessions were carried out one week after taking the post-test. The most relevant
opinions of the participants are described in Table 4.
Table 4. Most important notes extracted from the focus group sessions of the game version.
Topic
Notes
Game’s
strategy
"I ended the competition challenging the same group of players".
"When I saw myself in low positions on the leaderboard, I quit playing".
"I only challenged users above my position on the leaderboard".
"Climbing up the leaderboard was very motivating".
"I played because I enjoyed the game".
"If your challenges were ignored you could not go forward".
"I tried to challenge users above myself in the leaderboard".
"I ignored most challenges from users who were below myself in the leaderboard".
"If someone ignored my challenges I did not accept her/his challenges".
"There were more possibilities to be ignored in challenges with more than two players".
"I stopped playing because I saw that it was impossible to win".
Training
"I think people did not train enough because it does not reward points".
"I found the Training mode very interesting".
"I trained words that I did not utter correctly".
"I did not train because I did not get points".
"I preferred playing than training".
"I felt bored training".
"I only trained when there were words I did not understand well".
"I needed to train for climbing up to the first position on the leaderboard".
"I trained because I could listen to the words I found impossible to pronounce correctly".
"I liked to be able to choose specific phonemes to practice in the Training mode".
"I trained a little, only when I had problems with some words"

English
level
"I thought I had better English pronunciation skills".
"I realized the differences between similar words".
"I think I am now capable of distinguishing more English vowels".
"The phonetic transcription helped me so much".
"I think I have improved my pronunciation".
"At the end of the day I could produce words better".
"This game allows me to improve my pronunciation".
"I appreciate pronunciation oriented educational apps".
"I thought I was not able to produce so many different sounds, but this app helped me".
"It is very useful to become aware of some sounds".
"I find it very positive to learn the pronunciation of English in general".
3.2 Guided Version of the CAPT Tool
3.2.1 Usability Questionnaire
Participants evaluated the usability of the CAPT tool (SUS questionnaire, see Table 1) with an overall
score of 74.3 (Standard Deviation, SD 11.6) out of 100, which suggests good usability. In order to
complement the SUS questionnaire, learners filled an extra questionnaire with five Likert-scale
questions related to the specific game dynamics of the guided version of the CAPT tool (Table 5).
Table 5. User experience questionnaire of the guided version of the CAPT tool. Me and Mo refer to
Median and Mode, respectively.
Question
Answers (%)
Me
Mo
1
2
3
4
5
EXTRA1
The order in which the exercises are
presented seems appropriate to me.
0
0
10
70
20
4.0
4
EXTRA2
I would like to continue training with the rest
of English vowels and consonants.
10
0
30
50
10
4.0
4
EXTRA3
The videos of each lesson were useful and
adequate.
0
0
40
20
40
4.0
3-5
EXTRA4
I have gained a clear idea of how the
English vowels are produced.
0
10
40
30
20
3.5
3
EXTRA5
I think I have improved my pronunciation.
0
10
30
40
20
4.0
4
Although an objective improvement in pronunciation skills was reported [18] and high rates of agreement
are presented in Table 5, providing more feedback is necessary to improve user experience since the
median and mode values of the EXTRA4 question are the lowest ones (3.5 and 3, respectively).
3.2.2 Focus group session
Finally, a single focus group session with the 10 participants was carried out after completing all previous
stages. The most relevant opinions of the students are described in Table 6. These opinions are in line
with the positive results of the questionnaires since participants found the activities appropriate and easy
to understand. Students would also continue using the tool for more sounds. However, some learners
declared being frustrated with production activities on some occasions, as the answers of the EXTRA4
question of Table 5. We believe that, for any technological complement to be truly effective, it must be
subordinated to carefully-designed methodological activities that also include human interaction and
feedback in the case learners find difficulties performing the activities. Therefore, CAPT tools can be
seen as complementary resources for English classes, since they can be used in the classroom and
outside.

Table 6. Most important notes extracted from the focus group session of the guided version.
Topic
Notes
Training
"I think the activities were appropriated to improve my pronunciation".
"The tool was easy to use".
"The activities were easy to understand".
"Perception activities were my favorites".
"I would include a scheme of how each sound is pronounced".
"I would like to see what the tool has understood when I mispronounce".
"I felt a bit of frustration in the production activities".
"Repeating the same word multiple times is frustrating".
English
level
"I would use the tool as a complement to English support classes with my teacher".
"I find the tool useful for learning new sounds".
"I find the tool useful for university students".
"I see great potential in the tool and I would use it for training more sounds".
"I would use the tool in my English classes".
"I would like to train more consonant sounds".
"I would use the toll to reinforce phonetic learning".
4 CONCLUSIONS
L2 learning tools, via CAPT with ASR and TTS technology, are effective ways to integrate out-of-class
activities. The use of these technologies is beneficious not only for students who want to improve L2
pronunciation proficiency, but also for instructors who can provide learners a way to be autonomous in
their learning [2]. Integrating technology-mediated activities is an added value to the L2 pronunciation
teaching and learning processes. As seen with this study, we have reported on the degree of usability
and user experience of two versions of a novel CAPT tool for smart devices by university students of
English as L2, and in which language teachers, instructors, and researchers have collaborated in the
final design [6]. Good usability results are suggested in both versions of the CAPT tool, and the user
experience opinions gathered from the focus group sessions consolidate this affirmation. This qualitative
data should be taken into account for future versions of the CAPT tool. Therefore, guided CAPT tools
can result in an appropriate reinforcement of classroom-based pronunciation training sessions, whereas
CAPT tools with game elements can encourage greater participation and level of play in non-academic
contexts.
Recording all user interaction with the CAPT tool via log files, and gathering subjective information by
means of questionnaires and focus group sessions, has proven to be a very effective means to gather
huge amount of experimental data. A considerable amount of data has already been gathered, and new
data is being generated continuously. However, comparing and contrasting experiences with learners
of English as L2 from other different universities would be helpful to re-envision even more CAPT
opportunities.
ACKNOWLEDGEMENTS
This work was supported in part by Ministerio de Economía y Empresa (MINECO) and the European
Regional Development Fund FEDER (TIN2014-59852-R), by Consejería de Educación de la Junta de
Castilla y León (VA050G18), and by University of Valladolid (Ph.D. Research Grant 2015 and
MOVILIDAD DOCTORANDOS UVa 2019).
REFERENCES
[1] A. T. Dina and S. I. Ciornei, “The advantages and disadvantages of computer assisted language
learning and teaching for foreign languages,”. Procedia–Social and Behavioral Sciences, vol. 76,
pp. 248–252, 2013. doi: 10.1016/j.sbspro.2013.04.107

[2] M. G. O'Brien, T. M. Derwing, C. Cucchiarini, D. M. Hardison, H. Mixdorff, R. I. Thomson, H. Strik,
J. M. Levis, M. J. Munro, J. A. Foote, and G. M. Levis, “Directions for the future of technology in
pronunciation research and teaching,” Journal of Second Language Pronunciation, vol. 4, no. 2,
pp. 182–207, 2018. doi: 10.1075/jslp.17001.obr
[3] R. I. Thomson and T. M. Derwing, “The effectiveness of L2 pronunciation instruction: A narrative
review,” Applied Linguistics, vol. 36, no. 3, pp. 326–344, 2015. doi: 10.1093/applin/amu076
[4] D. Litman, H. Strik, and G. S. Lim, “Speech technologies and the assessment of second language
speaking: Approaches, challenges, and opportunities,” Language Assessment Quarterly, vol. 15,
no. 3, pp. 294–309, 2018. doi: 10.1080/15434303.2018.1472265
[5] A. Kukulska-Hulme, “Mobile-assisted language learning,” in The Encyclopedia of Applied
Linguistics, C. Chapelle, Ed. New York, NY, USA: Wiley, pp. 3701–3709, 2012. doi:
10.1002/9781405198431.wbeal0768
[6] C. Agarwal and P. Chakraborty, “A review of tools and techniques for computer aided
pronunciation training (CAPT) in English,” in Education and Information Technologies, vol. 24, no.
6, pp. 3731–3743, 2019. doi: 10.1007/s10639-019-09955-7
[7] Mary Meeker, Internet trends 2017, May 2017, Kleiner Perkins, Los Angeles, CA, USA, Rep.
[Online]. Retrieved from https://www.bondcap.com/report/it17
[8] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y.Wang, R.
Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu, "Natural TTS synthesis by
conditioning WaveNet on MEL spectrogram predictions," in Proceedings of the International
Conference on Acoustics, Speech, and Signal Processing 2018. ICASSP2018, pp. 4779–4783,
2018.
[9] S. Sepehr and M. Head, “Understanding the role of competition in video gameplay satisfaction,”
Information & Management, vol. 55, no. 4, pp. 407–421, 2018. doi: 10.1016/j.im.2017.09.007
[10] D. B. Clark, E. E. Tanner-Smith, and S. S. Killingsworth, “Digital Games, Design, and Learning: A
Systematic Review and Meta-Analysis,” Review of Educational Research, vol. 86, no. 1, pp. 79–
122, 2016. https://doi.org/10.3102/0034654315582065
[11] M. Prensky, Digital Game-Based Learning. New York, NY, USA: McGraw-Hill, 2004. doi:
10.1145/950566.950596
[12] M. M. Elaish, N. A. Ghani, L. Shuib, and A. M. Al-Haiqi, “Mobile Games for Language Learning,”
In S. Paiva (Ed.), Mobile Applications and Solutions for Social Inclusion (pp. 137–156). Hershey,
PA: IGI Global, 2018. doi:10.4018/978-1-5225-5270-3.ch006
[13] K. D. Squire, “Video games and education: Designing learning systems for an interactive age,”
Education and Technology, vol. 48, no. 2, pp. 17, 2008.
[14] S. Deterding, M. Sicart, L. Nacke, K. O'Hara, and D. Dixon, “Gamification. Using game-design
elements in non-gaming contexts,” in Proceedings of the 2011 Annual Conference Extended
Abstracts on Human Factors in Computing Systems. CHI EA’11. New York, NY, USA, pp. 2425–
2428, 2011. doi: 10.1145/1979742.1979575
[15] D. Huynh and H. Iida, “An analysis of winning Streak's effects in language course of ‘Duolingo’,”
Asia-Pacific Journal of Information Technology and Multimedia, vol. 6, no. 2, pp. 23–29, 2017.
[16] C. Tejedor-García, D. Escudero-Mancebo, V. Cardeñoso-Payo, and C. González-Ferreras, “Using
Challenges to Enhance a Learning Game for Pronunciation Training of English as a Second
Language,” in IEEE Access, vol. 8, pp. 74250–74266, 2020. doi: 10.1109/ACCESS.2020.2988406
[17] C. Tejedor-García, V. Cardeñoso-Payo, E. Cámara-Arenas, C. González-Ferreras, and D.
Escudero-Mancebo, “Measuring pronunciation improvement in users of CAPT tool TipTopTalk!,” in
Proceedings of Interspeech 2016, San Francisco, CA, USA, pp. 1178–1179, 2016.
[18] C. Tejedor-García, D. Escudero-Mancebo, E. Camára-Arenas, C. González-Ferreras, and V.
Cardeñoso-Payo, “Assessing pronunciation improvement in students of English using a controlled
computer-assisted pronunciation tool,” IEEE Transactions on Learning Technologies, early
access, 2020. doi: 10.1109/TLT.2020.2980261
[19] M. Celce-Murcia and J. M. Goodwin, Teaching Pronunciation. London, U.K.: Thomson Learning,
2014.

[20] M. Shormani, “Fossilization and Plateau Effect in Second Language Acquisition,” Language in
India. vol. 13, no. 2, pp. 763–784, 2013.
[21] A. Baker and S. Goldstein, Pronunciation Pairs Teacher's Book. Cambridge, U.K.: Cambridge
University Press, 2008.
[22] A. Baker, Ship Or Sheep? Student's Book: An Intermediate Pronunciation Course, vol. 1. Stuttgart,
Germany: Ernst Klett Sprachen, 2006.
[23] E. Cámara-Arenas, Native Cardinality: On Teaching American English Vowels to Spanish
Students (Historia y Sociedad). Valladolid, Spain: Ediciones Universidad de Valladolid, 2013.
[24] C. Nagle, “Perception, Production, and Perception-Production: Research findings and implications
for language pedagogy” World Languages and Cultures Publications, 2018. Retrieved from
https://lib.dr.iastate.edu/language_pubs/171
[25] J. Brooke, “SUS: a quick and dirty usability scale”. Taylor Francis: London, UK, 1996.
[26] A. Bangor, P. T. Kortum, and Miller J. T., “Determining what individual SUS scores mean: Adding
an adjective rating scale,” Journal of Usability Studies, vol. 15, no. 3, pp. 294–309, 2009.
[27] A. Bangor, P. T. Kortum, and Miller J. T., “An empirical evaluation of the system usability scale,”
International Journal of Human–Computer Interaction, vol. 24, pp. 574–594, 2008.
[28] D. Raptis, N. Tselios, J. Kjeldskov, and M.B. Skov, “Does size matter?: investigating the impact of
mobile phone screen size on users’ perceived usability, effectiveness and efficiency,” in
Proceedings of the 15th international conference on Human-computer interaction with mobile
devices and services. MobileHCI’13, pp. 127–136, 2013.