ChapterPDF Available

E-assessment in the TRADILEX project.

Authors:

Abstract and Figures

TRADILEX is a project which aims at studying how Didactic Audiovisual Translation fosters language learning. It has been implemented in a virtual course through a series of Lesson Plans which contain AVT tasks, and two language skills tests. Assessing the tasks and the language skills in a virtual and asynchronous environment was a paramount challenge. There was a need to have all the data centralised in a concrete platform because making use of different applications was confusing for both, the teachers, and the students in the course. Therefore, two key elements were taken into account: the importance of feedback and the importance of scaffolding. That is why the team chose Moodle because it facilitates both, the task of providing students with instant feedback and the inclusion of scaffolding in the form of Didactic-Mediated Dialogue in the virtual course through HTML5 code. The resources of the virtual platform allowed us to create tasks with e-rubrics which facilitate assessment and foster communication through specific applications. Besides, teachers have their own virtual space with video tutorials about the FAQs and fora to share their thoughts on the process of assessment. E-assessment is thus a keydimension of the TRADILEX project.
Content may be subject to copyright.
Designing
Online Assessment
Solutions that are rigorous,
trusted, flexible and scalable
Klaus-Dieter Rossade, José Janssen,
Carlton Wood & George Ubachs
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
EADTU Special interest Group on Online Assessment
European Association of Distance Teaching Universities (EADTU) | Netherlands
George Ubachs (Chair, author)
Stefan Meuleman (Coordination, layout)
Piet Henderikx
Theodora Boskou
Open University (OUUK) | United Kingdom
Klaus-Dieter Rossade (Author)
Carlton Wood (Author)
Open Universiteit (OUNL) | Netherlands
José Janssen (Author)
FernUniversität in Hagen | Germany
Andreas Kempka
Stefan Stürmer
Open University | Israel
Michal Ben Shaul
University of Ljubljana | Slovenia
Rasa Urbas
Marko Papic
CALED | Ecuador
Mary Elizabeth Morocho Quezada
Universidade Aberta | Portugal
Diogo Casanova
Lúcia da Graça Cruz Domingues Amante
Universitat Oberta de Catalunya (UOC) | Spain
Ana Guerrero
Nadja Gmelch
Loles González
UniDistance | Switzerland
Damien Carron
Henrietta Carbonel
Tampere University of Applied Sciences (TAMK)| Finland
Kirsi Saarinen
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
Thomas More University College | Belgium
Quinten Verdonck
Artevelde University of Applied Sciences | Belgium
Soetkin Lootens
Olaf Spittaels
Eef Vervalle
Hellenic Open University | Greece
Efthimios Zervas
Dimitris Kalles
Universidad Nacional de Educación a Distancia (UNED)| Spain
Miguel Santamaria
Ángeles Sánchez-Elvira Paniagua
José L. Aznarte
National Center for Distance Education | Czech Republic
Jan Beseda
Jyväskylä University | Finland
Pessi Lyyra
Tomi Waselius
Johannes Kepler Universität (JKU) | Austria
Thomas Fischer
Ursula Niederländer
Open University (OUC) | Cyprus
Erato Ioanna Sarri
Michalis Epiphaniou
Further contributions
Universidad Nacional de Educación a Distancia (UNED) | Spain
Elena Martín-Monje, M. Dolores Castrillo, Paz Díez-Arcón, Llanos Tobarra Abad,
Jesús González Boticario, M Ángeles Escobar Álvarez, Ana Ibáñez Moreno, Noa
Talaván and Antonio Tinedo-Rodríguez
Universidade Aberda | Portugal
L. Amante, M. C. Pinto and I. Oliveira
Open Universiteit (OUNL) | Netherlands
Ekaterina Muravyeva
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
Universidad Técnica Particular de Loja (UTPL) | Ecuador
Luis Moncada Mora, Duval Quezada Chávez and María José Rubio Gómez
Czech Association of Distance Teaching Universities (ČADUV) | Czech Republic
Lucie Rohlíková
Published by
European Association of Distance Teaching Universities | The Netherlands
Correspondence
European Association of Distance Teaching Universities (EADTU) att George Ubachs,
Managing Director Parkweg 27, 6212 XN Maastricht, The Netherlands Tel: +31 43 311
87 12 | E-mail: secretariat@eadtu.eu www.eadtu.eu | empower.eadtu.eu
Suggested citation
Rossade, K. D., Janssen, J., Wood, C., & Ubachs, G. (2022). Designing Online
Assessment - Solutions that are Rigorous, Trusted, Flexible and Scalable. Maastricht,
The Netherlands: EADTU.
License used
This work is licensed under a Creative Commons Attribution ShareAlike 4.0
International License: https://creativecommons.org/licenses/by/4.0/ This license lets
others distribute, remix, tweak, and build upon this work (even commercially) as long
as credit is provided for the original creation. This is among the most accommodating
of CC licenses offered, and recommended for maximum dissemination and use of
licensed materials.
Disclaimer
The work of EADTU is supported by the European Commission, DG EAC, under the
Erasmus+ Programme. The European Commission supports for the production of this
publication and does not constitute an endorsement of the contents which reflects the
views only of the authors, and the Commission cannot be held responsible for any use
which may be made of the information contained therein.
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
Contents
Introduction
7
Chapter 1: Assessment Design
13
Chapter 2: Trust, privacy & ethics
17
Chapter 3: Operational processes, technology and support
24
Final remarks
31
References
34
Annex I: Good Practices
36
1: Assessment approaches at the UAb: the present and the future
37
Diogo Casa Nova
Universidade Aberta | Portugal
2: El modelo de ‘evaluación en línea controladade la Universidad Técnica Particular de Loja
41
Luis Moncada Mora, Duval Quezada Chávez and María José Rubio Gómez
Universidad Técnica Particular De Loja (UTPL)| Ecuador
3: Examples of Digital Tools and Support at Tampere University of Applied Sciences
46
Kirsi Saarinen
Tampere University of Applied Sciences (TAMK) | Finland
4: Continuous Assessment Method on Engineering Physics Courses at TAMK
48
Kirsi Saarinen
Tampere University of Applied Sciences (TAMK) | Finland
5: Online Proctoring
Ekaterina Muravyeva, José Janssen
50
Open Universiteit (OUNL) | Netherlands
6: Moodle Test Activity Pilot project
56
L. Amante; M. C. Pinto; I. Oliveira; D. Casa Nova
Universidade Aberta | Portugal
7: Data-driven learning to improve writing skills in foreign languages
59
Elena Martín-Monje, M. Dolores Castrillo
Universidad Nacional de Educación a Distancia (UNED) | Spain
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
8: Pros and cons of online learning and assessment methods in higher education
62
Pessi Lyyra & Tomi Waselius
University of Jyväskyla (JYU) | Finland
9: Using MS TEAMS for online peer assessment: an innovative educational proposal
66
Paz Díez-Arcón, Elena Martín-Monje & M. Dolores Castrillo
Universidad Nacional de Educación a Distancia (UNED) | Spain
10: Open book assessment
69
Lucie Rohlíková
University of West Bohemia | Czech Republic
11: Agile, collaborative academic decision-making at scale and speed
72
Klaus-Dieter Rossade
The Open University (OUUK)| United Kingdom
12: Avoiding e-proctoring, ensuring trust: UNED's AvEx
75
José L. Aznarte, Ángeles Sánchez-Elvira, Miguel Santamaría, Llanos Tobarra Abad, Jesús
González Boticario
Universidad Nacional de Educación a Distancia (UNED) | Spain
13: Improving engagement and student performance through e-continuous assessment
78
Miguel Santamaría Lancho
Universidad Nacional de Educación a Distancia (UNED) | Spain
14: Continuous assessment in the subject of English II for Tourism: development of a
methodological proposal for the UNED 60/40 course
83
M Ángeles Escobar Álvarez & Ana Ibáñez Moreno
Universidad Nacional de Educación a Distancia (UNED) | Spain
15: E-assessment in the TRADILEX project
87
Noa Talaván, Antonio Tinedo-Rodríguez
Universidad Nacional de Educación a Distancia (UNED) | Spain
Annex II: Assessment and recognition of MOOC courses and MOOC pathways
91
Assessment
Design 01
Annex I: Good Practices
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
GP15
E-assessment in the TRADILEX project
Noa Talaván, Antonio Tinedo-Rodríguez
Universidad Nacional de Educación a Distancia (UNED) | Spain
Introduction
Assessing in virtual environments is a challenge and a need for the 21st century education.
Assessment goes beyond accountability because, according to Camilloni, Celman, Litwin, & Palou de
Maté (1998), it is an essential component of a planning for it provides both, students, and teachers,
with relevant information on the process of learning and on the process of teaching. E-assessment
has the advantage of providing students with immediate feedback and it can be implemented in
different ways such as a web-based system with assessment tools (Alruwais, Wills, & Wald, 2018,
p.34).
Assessment is crucial for the TRADILEX project because after doing each activity in the course which
aims at fostering language learning through Audio-visual Translation (AVT) tasks, students receive
their feedback within a maximum span of time of seven days. Fernández Abascal et al. (2010)
highlighted the importance of extrinsic motivation, which is a type of operant conditioning, when it
comes to achieving a goal or completing a task, because you can alter the level of motivation by
provoking certain stimuli. Burgers, Eden, van Engelenburg, & Buningh, (2015) highlighted how
positive feedback is a booster for motivation.
When designing the TRADILEX project, the washback effect of the tests and the activities included on
it was something crucial because as Harris & McCan (1994) affirm, the washback effect, which is the
influence of a test in the course, plays an essential role since it is linked to motivation. According to
Harries & McCan (1994, p. 27), in the field of language learning, a good test is that test which
TRADILEX is a project which aims at studying how Didactic Audiovisual Translation fosters
language learning. It has been implemented in a virtual course through a series of Lesson Plans
which contain AVT tasks, and two language skills tests. Assessing the tasks and the language skills
in a virtual and asynchronous environment was a paramount challenge. There was a need to have
all the data centralised in a concrete platform because making use of different applications was
confusing for both, the teachers, and the students in the course. Therefore, two key elements
were taken into account: the importance of feedback and the importance of scaffolding. That is
why the team chose Moodle because it facilitates both, the task of providing students with
instant feedback and the inclusion of scaffolding in the form of Didactic-Mediated Dialogue in the
virtual course through HTML5 code. The resources of the virtual platform allowed us to create
tasks with e-rubrics which facilitate assessment and foster communication through specific
applications. Besides, teachers have their own virtual space with video tutorials about the FAQs
and fora to share their thoughts on the process of assessment. E-assessment is thus a key
dimension of the TRADILEX project.
87
Annex I: Good Practices
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
contains authentic and real-life examples of tasks learners will have to perform in the future. That is
why the initial and the final tests for B1 and B2 used in TRADILEX were thoroughly designed by
members of the team. The names of these tests were ITIS (Initial Test of Integrated Skills) and FITIS
(Final Test of Integrated Skills) (Couto-Cantero, Sabaté-Carrové, & Gómez-Pérez, 2021). Even though
the tests were originally designed for online environments, its implementation was a challenge that
has been improved progressively according to the perceptions of both teachers and students in pre-
piloting experiences.
Furthermore, the e-assessment dimension of the TRADILEX project goes beyond testing the
language skills at the beginning and at the end of the project. García-Aretio (2014) focused on the
importance of formative assessment in the design of e-learning courses, and it impregnates the
TRADILEX project because every single task is assessed through an ad hoc interactive rubric.
Technological challenges
A project such as TRADILEX, which is fully developed in online environments, necessarily faces
technological challenges such as:
Choosing an appropriate virtual platform.
Didactic-Mediated Dialogue (DMD).
Its asynchronous nature.
Dropout rate.
Choosing an appropriate virtual platform that centralizes the data is crucial when it comes to
designing the e-assessment dimension of a project which is fully implemented online. Making use of
multiple platforms could be discouraging for both teachers and students. In this case, the research
team took the decision of making use of Moodle. The main advantages of Moodle are the following
ones:
It is a free and open-source Learning Management System (LMS).
It includes apps for assessment and for learning: fora, chats, blogs, collaborative glossaries,
testing tools, etc. Besides, more apps by developers may be installed to meet specific needs
of the project.
It is based on a constructivist approach of learning in which social communication is at the
very core.
It is constantly updated.
Following García-Aretio (2014), DMD plays an essential role when it comes to designing virtual
courses because it somehow replaces the role of the teacher by providing students with specific and
clear instructions on how to follow the course.
88
Annex I: Good Practices
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
Figure 1. DMD in the virtual course
In the TRADILEX project, DMD has taken a prominent role and it is available in English and in Spanish.
It has been embedded in the course through HTML and there is also a PDF guide available for
students. Besides, each lesson plan has its own guidelines to facilitate the process.
The main objective of this thorough design is to avoid dropout rate. Besides, taking the
asynchronous nature of the course into account, different fora were created for specific purposes so
that students and teachers could easily communicate with each other.
Figure 2. Fora in the virtual course
Good Practices
The good practices consist of the development of a system which aims at facilitating the
communication between teachers and students through a series of e-rubrics for tasks and for tests
that allows students to know their marks and their feedback when the teacher has just marked them
as it is displayed in Figure 3.
89
Annex I: Good Practices
Designing online assessment - solutions that are rigorous, trusted, flexible and scalable
Figure 3. One of the rubrics used in TRADILEX
The inclusion of communication tools as shown in Figure 2 and the thorough design of the DMD as
displayed in Figure 3 are also good practices of the project. Students and teachers involved in
TRADILEX show their satisfaction and express their positive feedback on the process of virtualisation
and centralisation of e-assessment. In fact, the very nature of Moodle fosters ubiquity so teachers
can assess anytime anywhere, by providing them with the flexibility they need. Besides, a specific
space has been created for teachers so that they can share their thoughts on the process. This space
contains video tutorials with the FAQs to facilitate the assessment task.
References
Alruwais, N., Wills, G., & Wald, M. (2018). Advantages and challenges of using e-assessment. International
Journal of Information and Education Technology, 8(1), 34-37.
Burgers, C., Eden, A., van Engelenburg, M. D., & Buningh, S. (2015). How feedback boosts motivation and play
in a brain-training game. Computers in Human Behavior, 48, 94-103.
Camilloni, A., Celman, S., Litwin, E., & Palou de Maté, M. d. (1998). La evaluación de los aprendizajes en el
debate didáctico contemporáneo. Paidós.
Couto-Cantero, P., Sabaté-Carrové, M., & Gómez-Pérez, M. d. (2021). Preliminary design of an Initial Test of
Integrated Skills within TRADILEX: an ongoing project on the validity of audiovisual translation tools in teaching
English. Research in Education and Learning Innovation Archives, 73-88.
Fernández Abascal, E. G., García Rodríguez, B., Jiménez Sánchez, M. P., Martín Díaz, M. D., & Domínguez
Sánchez, F. J. (2010). Psicología de la motivación. Ramón Areces.
García-Aretio, L. (2014). Bases, mediaciones y futuro de la educación a distancia en la sociedad digital. Síntesis.
Harris, M., & McCan, P. (1994). Handbooks for the English Classroom: Assessment. McMillan.
90
Article
Full-text available
This paper reports a mixed-method study on the factors shaping language learning through didactic audiovisual translation. The aim is to analyse the determinants influencing the effective implementation of an integrative project, revealing the strengths and weaknesses of the learning process. Data from 172 students were gathered through initial and final questionnaires, which offered insights into their perceptions of didactic audiovisual translation. The results demonstrate a noticeable shift in learners’ attitudes towards audiovisual translation as a language learning tool during the tasks. This transformation led to the recognition of potential benefits, not only in conventional modes like subtitling and dubbing but also in less recognised modes such as audio-description and subtitling for the deaf and hard of hearing.
Article
Full-text available
This article shows the preliminary design process for a tailor-made Initial Test of Integrated Skills (ITIS) specially adapted to the needs of TRADILEX, a project (2020-2023) funded by the Spanish Ministry of Science, Innovation and Universities to study the scope and validity of using audiovisual translation (AVT) tools to develop communicative competence in a foreign language. We also design a complete virtual AVT-based teaching platform that will help to promote faster and better EFL learning. In this paper we present one of the project’s initial stages, i.e. the ITIS design and the methodological framework for the design process from draft stage to final product. The draft pilot ITIS was tested by specialists, who produced a series of reports and anticipated several potential problems that occurred when the test was carried out. We pay special attention to the changes and improvements made to the draft ITIS as well as its three-stage process and remedial measures. The validity of the ITIS is currently being tested with a pilot group. The ITIS will be complemented with a Final Test of Integrated Skills (FITIS), which is currently underway.
Article
Full-text available
Information and communication technology (ICT) has been involved in different sectors. ICT has an impact to improve learning environment and process. With the increasing number of students and the limitation of placement, a new innovation in learning is required. Thus, E-learning and E-assessment have been introduced, to offer online courses and electronic assessment. E-assessment enables the students live in remote areas to have the test in their homes, and helps the teacher to correct the exam and release the marks in short time. This paper explains the meaning of E-assessment from different opinions and illustrates the cycle of E-assessment process. Moreover, this paper discusses the advantages and obstacles of using E-assessment in learning for different domains: student, teacher, institution, and in education aims.
Article
Full-text available
Games are important vehicles for learning and behavior change as long as players are motivated to continue playing. We study the impact of verbal feedback in stimulating player motivation and future play in a brain-training game. We conducted a 2 (feedback valence: positive vs. negative) × 3 (feedback type: descriptive, comparative, evaluative) between-subjects experiment (N = 157, 69.4% female, Mage = 32.07). After playing a brain-training game and receiving feedback, we tapped players’ need satisfaction, motivation and intention to play the game again. Results demonstrate that evaluative feedback increases, while comparative feedback decreases future game play. Furthermore, negative feedback decreases players’ feeling of competence, but also increases immediate game play. Positive feedback, in contrast, satisfies competence and autonomy needs, thereby boosting intrinsic motivation. Negative feedback thus motivates players to repair poor short-term performances, while positive feedback is more powerful in fostering long-term motivation and play.
Book
Full-text available
Bases, historia, principios, calidad y teorías de la educación a distancia. Los medios y recursos tecnológicos que la hacen posible. La docencia y la tutoría, las tendencias y la planificación de la EaD.
Handbooks for the English Classroom: Assessment
  • M Harris
  • P Mccan
Harris, M., & McCan, P. (1994). Handbooks for the English Classroom: Assessment. McMillan.