Conference PaperPDF Available

Assessment for/as Learning: Integrated Automatic Assessment in Complex Learning Resources for Self-Directed Learning

Authors:

Abstract

In the so-called ‘New Culture for Assessment’ assessment has become a tool for Learning. Assessment is no more considered to be isolated from the learning process and provided as embedded assessment forms. Nevertheless, students have more responsibility in the learning process in general and in assessment activities in particular. They become more engaged in: developing assessment criteria, participating in self, peer-assessments, reflecting on their own learning, monitoring their performance, and utilizing feedback to adapt their knowledge, skills, and assessment tools have emerged from being stand-alone represented by monolithic assessment tools to more flexible and interoperable generation by adopting the service-oriented architecture and modern learning specifications and standards. The new generation holds great promise when it comes to having interoperable learning services and tools within more personalized and adaptive e-learning platforms. In this paper, integrated automated assessment forms provided through flexible and SOA-based tools are discussed. Moreover, it presents a show case of how these forms have been integrated with a Complex Learning Resource (CLR) and used for self-directed learning. The results of the study show, that the developed tool for self- directed learning supports students in their learning process.
Assessment for/as Learning
Integrated Automatic Assessment in Complex Learning Resources for Self-directed Learning
Mohammad AL-Smadi and
Gudrun Wesiak
Graz University of Technology,
Graz, Austria
msmadi@iicm.edu;
gudrun.wesiak@uni-graz.at
Christian Guetl
Graz University of Technology,
Curtin University of Technology,
Perth, WA.
Graz, Austria
cguetl@iicm.edu
Andreas Holzinger
Graz University of Technology,
Medical University Graz
Graz, Austria
andreas.holzinger@medunigraz.at
Abstract— In the so-called ‘New Culture for Assessment’
assessment has become a tool for Learning. Assessment is no
more considered to be isolated from the learning process and
provided as embedded assessment forms. Nevertheless,
students have more responsibility in the learning process in
general and in assessment activities in particular. They become
more engaged in: developing assessment criteria, participating
in self, peer-assessments, reflecting on their own learning,
monitoring their performance, and utilizing feedback to adapt
their knowledge, skills, and behavior. Consequently,
assessment tools have emerged from being stand-alone
represented by monolithic systems through modular
assessment tools to more flexible and interoperable generation
by adopting the service-oriented architecture and modern
learning specifications and standards. The new generation
holds great promise when it comes to having interoperable
learning services and tools within more personalized and
adaptive e-learning platforms. In this paper, integrated
automated assessment forms provided through flexible and
SOA-based tools are discussed. Moreover, it presents a show
case of how these forms have been integrated with a Complex
Learning Resource (CLR) and used for self-directed learning.
The results of the study show, that the developed tool for self-
directed learning supports students in their learning process.
Keywords-component; Automatic Assessment; Complex
Learning Resource; e-Assessment; Self-directed Learning;
Service-Oriented Architecture.
I. INTRODUCTION
Learners grow up with technology dominating most of
their life activities. They use technology anywhere, anytime,
and they are faced with the challenge of needing to be
engaged and motivated in their learning [1]. The emergence
of Web 2.0 and the influence of Information and
Communication Technology (ICT) have fostered e-learning
to be more interactive, challenging, and situated. As a result,
learners feel empowered when they are engaged in
collaborative learning activities and self-directed learning.
The learners are also provided with e-learning systems that
maintain their social identity and situated learning
experience. Given the different learning styles of students,
educators are faced with the challenge of having to develop
assessment tools which are required to appraise the students’
learning process. Assessment forms provided in current e-
learning activities have to be adapted so that they can foster
effective types of learning such as reflective-learning,
experiential-learning, and socio-cognitive learning [2].
To this end, this research investigates the following
goals: (G1) the applicability and usability of flexible and
interoperable educational tools in one complex learning
resource, moreover (G2) students’ perception towards the
use of a complex learning resource integrated with automatic
e-Assessment during self-directed learning activities, finally,
(G3) the relation between students’ motivation and their
preferred learning style when it comes to use complex
learning objects (CLO) enriched with automatic assessment
for self-directed learning.
The remaining part of this paper is organized as follows:
Section II explains the notion of CLR and discusses the
requirements and architecture for the developed tools,
Section III explains the study design and analysis, and
Section IV discusses the results and reflects on the research
goals and hypotheses.
II. COMPLEX LEARNING RESOURCE FOR SELF-DIRECTED
LEARNING
According to [3] the Atomic learning object is defined as
“the smallest unit of reuse for LOs that may or may not be
associated to one or more multimedia contents”, whereas a
Complex LO (CLO) is defined as “an LO whose
instructional material is an aggregation of Learning Objects.
Being an LO, a Complex LO can be treated exactly as any
other LO”. Accordingly we define a CLR as a composite
didactic resource consists of one or multiple learning objects
(either atomic or complex). Accordingly, CLR inherits the
features of LO of reusability and interoperability provided by
the standards and specifications used to represent LOs.
The CLR developed for this study is a composite didactic
resource for self-directed learning that consists of learning
materials represented in SCORM [4], enriched with IMS
QTI [5] compliant test items automatically created from
learning material. Moreover IEEE Learning Object Metadata
(LOM) [6] is used to annotate the CLR. The CLR is
provided by the Intelligent Web Teacher (IWT) [7] - a
learning management system allowing the definition and
execution of personalized e-learning experience tailored on
the basis of learners’ cognitive status and learning
preferences - based on fully integrated tools and services.
A. Functional Requirements and Architectural Design
The study aims at developing a CLR for self-directed
learning enriched with automatically created formative
assessments based on textual learning material. The tools
have been designed to consider the following functional
requirements: (1) formative assessments are created
automatically using an automatic question creator tool based
on extracted concepts from textual learning material, (2)
Standard compliance by using IMS QTI to annotate created
questions, (3) a variety of test items – i.e. multiple choice,
true/false, fill-in-the-blank, and open ended questions, (4) the
creation and fruition of a self-directed course where students
are provided with search features to find and select learning
material, and (5) a semi to fully automatic interactive
approach by which students are allowed to tag learning
content and extract concepts, and get automatically created
tests for those concepts.
Assessment tools have emerged from being stand-alone
represented by monolithic systems through modular
assessment tools to a more flexible and interoperable
generation by adopting the service-oriented architecture and
modern learning specifications and standards. The new
generation holds great promise when it comes to having
interoperable learning services and tools within more
personalized and adaptive e-learning platforms [8]. This
generation highly depends on service-oriented architectures
(SOA) where its services support federated exchange
(information and control), various levels of interoperability
(intra-domain and inter-domain), and service composition
(orchestration and choreography). The CLR has been
developed with respect to the architecture proposed in [9] for
self-directed learning with automatically created tests – using
automatic question creator tool (AQC) [10] - based on the e-
assessment framework discussed in [8] [11]. For the sake of
flexibility a web service has been developed in order to
interpret, validate, and create QTI-based assessment items
and tests. The web service is developed as part of the
suggested middleware for tools interoperability and flexible
e-assessment system [8]. The web service is then used based
on a service-oriented framework for assessment (SOFA) [11]
to support modules of items and tests authoring/viewing, and
extend the learning platform with the features provided by
AQC as discussed in this study.
The developed CLR for self-directed learning has the
following competitive advantages:
Advanced tool supporting the creation of four
different test item types: multiple choice
questions, true/false exercises, fill-in-the-blank
exercises, and open ended questions.
Learning setting dependent operating modes
support fully-automatic test item creation and
interactive process types taking into account student
or teacher input.
Domain knowledge independent methods allow
test item creation of unseen textual content by
applying statistical, semantic, and structural
analyses.
Language dependent data flow and process chain
design provide multilingual test item creation,
currently English and German, and support the easy
extension to other languages.
Flexible design supports an easy integration or
exchange of modules in the system to offer
improved processing tasks or even new features [9].
Easy integration into other systems and service
provision by a standard-conform web service
interface [8] [11].
IMS QTI Standard compliance [5] enables an
easy export and reuse of test items created by the
tool [9].
III. STUDY DESCRIPTION AND FINDINGS
The study has been conducted as part of a scientific
research course, in which a CLR enriched with automatic
assessment has been used to provide a self-directed learning
course. The course has been delivered in distance learning
settings and participants got to know their partners within the
study activities.
A. Method
The entire study comprised three phases, with one
questionnaire provided for each phase. Since this article is on
self-directing learning, we will only report the method and
results from phases 1 and 2.
1) Participants
In this study 12 students had participated, for 5 of them
the course was mandatory, 7 participated as life-long
learners. Eight participants are male and four female with
their age ranging between 22 and 41 years old (M= 32, SD=
6.53). With respect to education level, three students hold a
Bachelor degree, eight hold a Master degree, and 1 one has a
PhD.
Only six students finished the entire study as the course
was mandatory for five of them. One student participated in
all the three phases but s/he did not finish the requirements
of phase 3. Two students finished phases 1 and 2 and three
students only participated in phase 1.
2) Apparatus and Stimuli
The course material and tests have been provided online
using IWT as a learning management system allowing the
definition and execution of personalized e-learning
experience tailored on the basis on learners’ cognitive status
and learning preferences based on fully integrated tools and
services. Nevertheless, the LimeSurvey 1 deployed on our
campus server has been used to deliver three questionnaires -
1 [http://www.limesurvey.org/]
one for each phase of the study - to investigate aspects such
as, motivation and attitudes, emotions, preferable learning
style, and usability.
a) Pre-questionnaire
This questionnaire was provided at the beginning of the
study and investigated information on demographic data and
previous experience, and motivational aspects towards using
CLR enriched with automatic assessment for self-directed
learning.
In order to investigate participants’ motivation towards
the course in general and the study phases in particular, a
section adapted from [12] has been added based on the
following three motivation scales: Intrinsic Goal Orientation
Scale measures the students’ intrinsic motivation regarding
the course, for instance: “I prefer course material that arouses
my curiosity, even if it is difficult to learn”. A high value on
this scale would mean that the students are doing the course
for reasons such as challenges and curiosity. The Extrinsic
Goal Orientation Scale deals with the extrinsic motivation of
students, e.g. “Getting a good grade is the most satisfying
thing for me right now” A student is extrinsically motivated
when s/he is rather interested in rewards or good a grade than
in the task itself., Finally, the Task Value Scale is about the
learning task itself, i.e. how important, interesting, and useful
the task and the task material are for the students. More
interest in the task should lead to more involvement in one’s
learning. To give an example, one item out of this scale is: “I
think I will be able to use what I learn in this course in other
courses”. Answers were given on a 5-point Likert scale, so
that students could state their level of agreement or
disagreement. The rating scale ranged from “I strongly
disagree” (1), “I disagree” (2), “neither/nor” (3) to “I agree”
(4), “I strongly agree” (5).
b) Post-questionnaire
This questionnaire was provided at the end of the study
on self-directed learning with automatic formative
assessment (see procedure section for more details) to
investigate aspects such as, quality of written content, quality
and frequency of received tests, preferred learning style,
emotional aspects, and tools’ usability. Regarding the quality
of written content and test questions a scale from “very bad”
(1), “bad” (2), “ok” (3), “good” (4) up to “very good” (5) has
been used. Where students were asked how often they had
taken a test the used scale ranged from “never” (1), “seldom”
(2), “sometimes” (3), to “often” (4).
Regarding the “usability of the learning scenario” we
used the System Usability Scale (SUS) [13] which contains
10 items and a 5-point Likert scale to state the level of
agreement or disagreement (e.g. “I think that I would like to
use this system frequently”).
The learning style of ‘elaborating’ or ‘repeating’ has
been investigated in order to find out if the students’ learning
process is rather superficial or aims at a deeper
understanding. For this section, items developed by [14]
have been translated into English (e.g. item regarding the
elaborating learning style: “In my mind I try to connect what
I have learned with already known issues concerning the
same topic”, item regarding the repeating learning style: “I
try to learn the content of scripts or other notes by heart”).
The answers were also given on a 5-point Likert scale.
To assess the participant’s emotional state during the
second phase, the emotional scale developed by [16] has
been used. This scale includes 12 items describing four
emotions related to learning new computer software as
follows: Happiness (“When I used the tool, I felt
satisfied/excited/curious.”), Sadness (“When I used the tool,
I felt disheartened/dispirited”), Anxiety (“When I used the
tool, I felt anxious/insecure/helpless/nervous”), and Anger
(“When I used the tool, I felt irritable/frustrated/angry”). For
this section answers followed a scale from “None of the
time”, “Some of the time”, “Most of the time” or “All of the
time”.
Moreover, an open comment section has been added to
this questionnaire to get additional comments and
suggestions from the students.
Finally, a section named motivational aspects has been
provided to investigate the participants’ motivation during
the study. For instance students were asked “How motivated
were you according to the following tasks?”, Reading the
contents, working with the self-directed tool, testing myself
with questions, and filling in the evaluation questionnaires.
The following scale has been used to get the participants
answers: “absolutely unmotivated” (1), “unmotivated” (2),
“motivated” (3), and “very motivated” (4).
3) Procedure
After learning a content, provided by the developed
system, on scientific working and taking a test on this
content (Phase 1), students have been grouped by the
instructor into 6 groups – two members each - based on their
interest in the course (i.e. mandatory of 3 groups and
volunteer of 3 groups). In this second Phase of the study,
students should deepen their knowledge in two main
categories: experimentation design and experimentation
analysis. For each of them, 6 articles have been delivered.
Each group member has been requested to select one article
from both categories different than the ones selected by his
peer within the same group. In order to avoid members from
the same group selecting similar articles they have been
asked to use the discussion forum to agree on their
selections. Moreover, participants introduced each other
using the forum, and selected their articles based on their
interest.
Furthermore the self-directed learning course supported
students with the ability to test themselves before, during
(after sections), and after reading the article. A “TestMe”
button has been added to the course player by which the
provided learning content is used to automatically create
tests based on the students’ preferences. Those created tests
could be taken several times in a formative way to get
formative feedback about their current knowledge state with
respect to the learning material.
At the end of the study students have been asked to
answer the post-questionnaire.
B. Evaluation Methods and First Findings
This section reports the results derived from students’
answers on the two questionnaires and tests the study
hypotheses as follows:
1) H1: the use of the tools is easy even if the user is a
non-expert
In order to test this hypothesis, the following evaluation
criteria and metrics have been used:
C1.1: To evaluate the user’s level of
satisfaction towards the tools,
C1.2: To identify possible improvements for
the tool based on comments and suggestions,
M1.1: Ratings for functionality/usability of the
tool itself, and frequency of use (post-
questionnaire)
M1.2: Ratings for emotional aspects while
using the tools (post-questionnaire)
M1.3: Suggestions and comments based on
open questions. (post-questionnaire)
A. With respect to M1.1:
results have shown that 7 out of 8 students who
completed the first two phases of the study have taken
formative tests during the self-directed learning in phase 2,
and one student said that s/he has never took a test because
s/he did not have time. Counting the tests which the students
took optionally during phase 2, 30 tests were taken in total.
Regarding the three different types of tests the students
stated on a 4-point rating scale that they seldom took a test
before, during, or after reading the topic (pre-test: M = 2.13,
SD = 0.64; sub-sections test: M = 2.25, SD = 0.71,; and post-
test: M = 2.25, SD = 0.87). However, looking at the actual
data, the students called the AQC 6 times for a pre-test and a
post-test (maximal twice per person), and 18 times for the
sub-sections tests (between 0 and 8 times per person).
With respect to the tool usability, the average SUS score
based on eight students’ responses is “66.88”, where the SUS
scale gives a score within a range of “0” and “100”.
According to [16] “The average SUS score from all 500
studies is a 68. A SUS score above a 68 would be considered
above average and anything below 68 is below average”.
The reference provides a calculator to convert the SUS score
into a percentile rank through a process called normalizing.
A score above an 80.3 is considered an A (the top 10% of
scores). Scoring at the mean score of 68 gets you a C and
anything below a 51 is an F (putting you in the bottom 15%).
The calculator “takes raw SUS scores and generates
percentile ranks and letter-grades (from A+ to F) for eight
different application types”. The score 66.88 the CLR
achieved indicates that the tool has higher perceived usability
than (40% - 50%) of all products that have been tested, and it
can be interpreted as a grade C. According to [17] this score
can be considered as “OK” having the complexity of the
learning scenario and the use of multiple tools in a flexible
and interoperable way within the same learning scenario.
B. With respect to M1.2:
Concerning students’ emotions during working with the
self-directed learning tool, a comparison of the mean values
indicate that the students felt equally happy (M = 1.88, SD =
0.80), sad (M = 1.5, SD = 0.60), anxious (M = 1.41, SD =
0.65), and angry (M = 1.54, SD = 0.31). Multiple dependent
t-tests also show that with a significance level of p > .05
there was no difference between the emotions happiness
(tsadness (7) = 0.98; tanxiety (7) = 1.44 tanger (7) = 1) sadness
(tanxiety (7) = 0.31; tanger (7) = 0.17), anxiety (tanger (5) = 0.57)
and anger. By interpreting the mean values, it can be
assumed that the students seldom felt consciously happy,
sad, anxious, or angry.
Linking the emotional state with the tool frequency of
use form last section, we can assume that despite the unclear
emotional state during the self-directed learning activity,
they frequently requested an automatic test with a rate of
(twice per student) on pre, and post-tests, as well as (between
0 - 8 times per student) on sub-sections tests.
C. With respect to M1.3:
Regarding what the students liked about the tool, students
stated that they were in favor of the simplicity of the tool and
the division of the content into meaningful modules.
Furthermore the students liked the consistency and the
possibility to have an overview of the learning progress and
their own test results. They mentioned that the course was
well organized and they appreciated that the course was
online, so that they could work from anywhere. Also, the
content itself was described very well and was precise and
useful. On the other hand, students did not like that they
were logged out after a short period of time (Session time-
out was short). Some also complained about the slow
interface. Regarding the Test Module within the self-
regulated tool, some students criticized the difficulty to
navigate to different questions. Regarding comments and
suggestions for improvement, they would like to download
content and print it directly as a handout. Besides, some
students suggested a layout for a better overview. The
students would improve the text structure and recommended
clearer instructions for the assessment parts.
Moreover, the students were asked about “what they like
about the three types of tests”. Results have shown that the
different types of questions helped them getting an overview
about the topics. Furthermore, they were in favor of the
division of the learning material into small modules. Some
students also stated that the sub-section and post-tests
supported them in observing their learning progress. In the
opposite, they were asked “what they did not like”. First of
all, the tests were criticized, as in particular they focus on
factual knowledge. Additionally, the multiple choice
questions were criticized due to the possibility of having low
quality distractors.
1) H2: Using the tools has a positive impact on the
users’ motivation concerning their learning activities
In order to test this hypothesis, the following evaluation
criteria and metrics have been used:
C2.1: To evaluate students’ motivation
concerning their learning activities.
C2.2: To identify preferable learning styles of
the students, and the impact of students’
motivation on their preferable learning style.
M2.1: Ratings of students’ extrinsic and
intrinsic motivation regarding the course and its
tasks before using the tool (pre-questionnaire).
M2.2: Ratings regarding students’ learning
styles. (post-questionnaire)
A. With respect to M2.1:
The results of student’s motivation regarding the course
and its tasks shows that their intrinsic motivation (M = 3.94,
SD = 0.53) was significantly higher than their extrinsic
motivation (M = 2.83, SD = 0.79; t (11) = 3.43, p<.01). This
means that they are interested in the course for reasons such
as curiosity and challenge, whereas high grades or rewards
were not so important for them. These findings are supported
by the results of the task value scale. A mean value of 3.83
(SD = 0.74) shows that the students were really interested in
the task itself. The task material was also very useful and
important for them. Due to their high interest, it can be
assumed that this also leads to more involvement in their
learning activities.
In general, questions regarding students’ motivation
concerning their learning activities during the three phases
revealed that they were motivated up to very motivated over
the course of the study. Table 1 shows the mean ratings as
well as the respective medians in order to take account of
extreme values.
TABLE I. MEAN RATINGS OF MOTIVATION DURING THE COURSE
Motivatio n while … M (SD) Md
r
eading the content 3.5 (0.55) 3.5
w
orking with the too
l
2.67 (0.52) 3.0
testin
g
themselves with questions 2,5 (0.84) 3.0
f
illing in the questionnaire 3.0 (0.0) 3.0
Note: ratings were given on a 4-pt. scale
B. With respect to M2.2:
A comparison of the mean values derived from eight and
seven questions for the elaborating and repeating learning
style, respectively, shows that there is a significant difference
between the elaborating (M = 4.05, SD = 0.56) and the
repeating learning style (M = 3.04, SD = 0.82; t (7) = 2.71,
p<.05). The students prefer the elaborating learning style,
which means that their learning process aims at deeper
understanding and is less superficial. Concerning
elaborating, for instance the students stated that they try to
link new terms or new theories to familiar terms and theories
(M = 4.38, SD = 0.52). In contrast to that, students said that
they do not learn the content of scripts or other notes by heart
(M = 2, SD = 1.07) which would indicate a repeating
learning style.
From M2.1 and M2.2 results, we can deduce a relation
between elaborating learning style and deep learning based
on intrinsic motivation to participate in the learning activity.
The results from M2.1 show that students were intrinsically
motivated at the beginning of the study. Due to their learning
style preference, it can be assumed that the students were
still intrinsically motivated during the self-directed learning
activity. Thus, the students answered the questions out of
pleasure with the aim to deepen their knowledge.
In addition, the students stated that if they learn
something, testing themselves with questions often helps
them (M = 3.63, SD = 1.50). This result is in line with the
results discussed above. So it can be assumed that providing
self-directed learning courses with the ability to create
automatic tests supported the students to achieve their
learning goals. However, the findings are in line with
literature. The research of [18] shows that there is an
evidence of the influence of intrinsic motivation on learners
engagement that leads to ‘deep’ learning through higher level
thinking skills and the conceptual understanding. Moreover,
the author of [19] highlights the problems associated with
extrinsic motivation as it leads to ‘shallow’ instead of ‘deep’
learning.
IV. CONCLUSION AND OUTLOOK
With respect to the study goals, this section concludes the
findings accordingly and provides some look ahead for
future work. To this end, summarizing (G1), it can be
assumed that the tools developed to integrate assessment
forms with complex didactic resources are user friendly.
First, we can be satisfied with the functionality because of
the satisfactory SUS score the tools have reached. Moreover,
the students were in favor of the various functions of the
tools and their simplicity. Second, they stated that the tools
gave them a good overview of their learning progress. For
further improvement, a closer look on the “fill in the blank”
question type, which was not easy to solve and work on a
faster interface should be considered. Moreover, the study
shows the applicability of combining interoperable and
flexible learning tools in one complex learning scenario.
Regarding students’ motivation (G2), the results show
that the students were intrinsically motivated at the
beginning of the course. So they were really interested in the
course and its tasks, which lead also to more involvement in
their learning activities. At the end of the course, the students
were asked about their motivation concerning different
learning activities. According to the results, students’
motivation was high during reading content, and filling in the
questionnaires. In addition, testing themselves with
automatically created tests – i.e. pre-, intermediate based on
subsection, and post-tests - and working with the self-
directed learning tool also motivated them.
By investigating students’ learning styles, we found out
that the students’ learning process aims at deeper
understanding and is less superficial. This result is in line
with the results discussed above, as intrinsic motivation has a
great influence on learner engagement and their learning
style. Thus, it can be assumed that students took tests out of
pleasure with the aim to deepen their knowledge. Besides,
students also stated that testing themselves often supported
them in their learning process (G3).
ACKNOWLEDGMENT
This research is partially supported by the European
Commission under the Collaborative Project ALICE
"Adaptive Learning via Intuitive/Interactive, Collaborative
and Emotional System", VII Framework Program, Theme
ICT-2009.4.2 (Technology-Enhanced Learning), Grant
Agreement n. 257639. We are grateful to Dominik Kowald
for his support in developing the CLR as well as Margit
Höfler and Isabella Pichlmair for the support during the
planning and analysis of this study.
REFERENCES
[1] M. Prensky, “Digital Natives, Digital Immigrants”. In On the
Horizon, October 2001, 9(5), NCB University Press.
[2] F. J.Dochy & L., McDowell “Introduction Assessment as a tool for
learning.” Studies in Educational Evaluation, 23 (4), 279-298, 1997.
[3] Di Nitto, E., Mainetti, L., Monga, M., Sbattella, L. & Tedesco, R.
Supporting Interoperability and Reusability of Learning Objects: The
Virtual Campus Approach. Educational Technology & Society, 2006,
9 (2), 33-50.
[4] SCORM. ADL Sharable Courseware Object Reference Model. Last
retrieved on March 3rd, 2012 from
http://www.adlnet.gov/scorm/index.aspx
[5] IMS QTI. IMS Question & Test Interoperability Specification,
Version 2.0 - Final Specification. Last retrieved March 3rd, 2012 from
http://www.imsglobal.org/question/index.html.
[6] IEEE LOM. IEEE Standard for Learning Object Metadata. Last
retrieved on March 3rd, 2012 from
http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf.
[7] N. Capuano, S. Miranda, and F. Orciuoli, “IWT: A Semantic Web-
based Educational System”, IV Workshop of the Working Group on
“AI & E-Learning” 2009, pp. 11–16.
[8] M. AL-Smadi, and C. Guetl, “Service-oriented flexible and
interoperable assessment: towards a standardised e-assessment
system”, Int. J. Continuing Engineering Education and Life-Long
Learning, 2011, Vol. 21, No. 4, pp.289–307.
[9] M. AL-Smadi, & C.Guetl “Supporting Self-Regulated Learners with
Formative Assessments using Automatically created QTI-Questions”,
Proceedings of IEEE EDUCON Education Engineering 2011 - The
Future of Global Learning Engineering Education, April 2 - 6, 2011,
Amman, Jordan.
[10] C. Gütl, K. Lankmayr, and J. Weinhofer. “Enhanced approach of
automatic creation of test items to foster modern learning setting.” In
Proc. of the 9th European Conference on e-Learning, Porto, Portugal,
4-5 November 2010, 225-234.
[11] M. AL-Smadi, C. Guetl, D. Helic, “Towards a standardized e-
assessment system: motivations, challenges and first findings”,
International Journal of Emerging Technologies in Learning (iJET),
Volume 4, Special Issue 2: "IMCL2009" , 2009.
[12] P.R. Pintrich, , D.A.F. Smith, , T., Garcia, & W.J. McKeachie, “A
Manual for the Use of the Motivated Strategies for Learning
Questionnaire (MSLQ)”. Technical Report,1991, 91, 7-17.
[13] J. Brooke,. “SUS: A “quick and dirty” usability scale”. In Usability
evaluation in industry, 1996, London: Taylor & Francis.
[14] K.-P.. Wild, Lernstrategien im Studium. Strukturen und
Bedingungen. Münster: 2000, Waxmann.
[15] R.H., Kay, & S. Loverock, “Assessing emotions related to learning
new software: The computer emotion scale”. Computers in Human
Behavior. 2008, 24, 1605-1623.
[16] J., Sauro, “Measuring Usability with the System Usability Scale
(SUS)”, retrieved on March 3rd, 2012 from
http://www.measuringusability.com/sus.php
[17] A. Bangor, P. T. Kortum & J. T. Miller ”An Empirical Evaluation of
the System Usability Scale”, International Journal of Human-
Computer Interaction, 2008, 24:6, 574-594.
[18] T., Kellaghan, G. F., Madaus, & A. Raczek, The Use of External
Examinations to Improve Student Motivation. 1996, Washington,
DC:AERA
[19] T.J. Crooks, “The impact of classroom evaluation practices on
students”, Review of Educational Research, 1988, vol. 58, no. 4, pp.
438-481.
... To investigate whether students could benefit from the learning environment, questionnaires covering task awareness, motivational and emotional aspects, and usability were sent to the students at three points during the study. A detailed description of the study can be found in [32]. ...
Article
Full-text available
Today’s e-learning systems meet the challenge to provide interactive, personalized environments that support self-regulated learning as well as social collaboration and simulation. At the same time assessment procedures have to be adapted to the new learning environments by moving from isolated summative assessments to integrated assess- ment forms. Therefore, learning experiences enriched with complex didactic resources - such as virtualized collabora- tions and serious games - have emerged. In this extension of [1] an integrated model for e-assessment (IMA) is outlined, which incorporates complex learning resources and assess- ment forms as main components for the development of an enriched learning experience. For a validation the IMA was presented to a group of experts from the fields of cognitive science, pedagogy, and e-learning. The findings from the validation lead to several refinements of the model, which mainly concern the component forms of assessment and the integration of social aspects. Both aspects are accounted for in the revised model, the former by providing a detailed sub-model for assessment forms.
... Öğrenme çıktılarının niteliğini tam olarak belirleyebilmek için öğrencileri değerlendirmede çeşitli ölçme araç ve teknikleri işe koşulmalıdır (Popham, 2001). Ayrıca, öğrencilerin farklı öğrenme biçimlerine sahip olmaları ve değerlendirmenin farklı amaçlarla yapılması da öğrencilerin başarı durumlarının belirlenmesinde farklı ölçme araç ve tekniklerinin kullanılmasını gerektirmektedir (AL-Smadi, Wesiak, Guetl, Holzinger, 2012;Grover, 2009;Huba ve Freed, 2000). Amerikan Yüksek Öğretim Derneğinin (American Association for Higher Education-AAHE) öğrencilerin değerlendirilmesi süreciyle ilgili belirlemiş olduğu uygulama ilkelerinin birinde öğrenmenin karmaşık bir süreç olduğu, bilgi ve becerinin yanında değer, tutum ve alışkanlıkları da içerdiğinden hareketle, öğrencilerin eğitim deneyimlerini geliştirmek için farklı ölçme yöntemlerinin kullanılması gerektiğine vurgu yapılmaktadır (Astin ve diğ.,1996). ...
Article
Full-text available
Different kinds of multiple choice questions are used in exams in open and distant education systems. These types are multiple choice, information dependent, completion and matching type questions. It is important to determine whether one of these types are harder than the other. Besides, it is important to determine whether one of these question types is more discriminating in terms of exam validation. In this study, we tried to determine whether difficulty and discrimination value differ based on question type. To this end, the difficulty and discrimination values of traditional multiple choice, information dependent, completion and macthing type questions of 9296 questions used in the final exam in Anadolu University Open Education Faculty 2017-2018 fall semester. It is found out that completion type of questions are easier and more dicriminating.
... To investigate whether students could benefit from the learning environment, questionnaires covering task awareness, motivational and emotional aspects, and usability were sent to the students at three points during the study. A detailed description of the study can be found in [15]. ...
Conference Paper
Full-text available
Today's e-learning systems meet the challenge to provide interactive, personalized environments that support self-regulated learning as well as social collaboration and simulation. At the same time assessment procedures have to be adapted to the new learning environments by moving from isolated summative assessments to integrated assessment forms. In this paper an integrated model for assessment (IMA) is outlined, which incorporates complex learning resources and assessment forms as main components for the development of an enriched learning experience. For a validation the IMA was presented to a round of experts from the fields of cognitive science, pedagogy, and e-learning. The findings from the validation lead to several refinements of the model, which mainly concern the component forms of assessment and the integration of social aspects. Both aspects are accounted for in the revised model, the former by providing a detailed sub-model for assessment forms.
Article
Full-text available
It was aimed to determine the effect of "Web 2.0 supported measurement and evaluation training" (WSMAT) on the content and form properties of the questions developed by pre-service science, mathematics and social studies teachers. For this purpose, pre-service teachers were received a total of 40 hours training for 5-days to utilize web 2.0 tools used for the measurement and evaluation purposes. Pre-service teachers were asked to develop exam questions consisting of 30 questions by using traditional methods before the training and web 2.0 tools after the web 2.0 tools training. The questions prepared by the pre-service teachers were evaluated by two field experts from each branches (science, mathematics, social studies) by using the “evaluation scale” developed in line with the opinions and suggestions of the field experts. The questions developed by using web 2.0 tools for measurement-evaluation purposes were more qualified in terms of both content and form features than the questions developed by traditional methods. The reason for this is that web 2.0 tools are easy to produce content, easy to develop different question types with visual support, are independent of time and place, have an user-friendly interface and support teachers' attention and motivation by reducing the burden on them.
Article
Full-text available
Bu çalışmada, dünyada açık ve uzaktan eğitim hizmeti sunan mega üniversitelerden biri olan Anadolu Üniversitesi Açıköğretim Sistemi'nde ölçme ve değerlendirme uygulamalarının nasıl geliştirilebileceği araştırılmıştır. Bu kapsamda öğrenenlerin biçimlendirici ve düzey belirleyici değerlendirme etkinliklerine ilişkin görüşleri incelenerek Sistem'de başka hangi ölçme araç ve tekniklerinin kullanılabileceği belirlenmeye çalışılmıştır. Karma yöntem araştırma desenlerinden keşfedici yöntemin (NİTEL → nicel) benimsendiği çalışmada; nitel veriler yarı yapılandırılmış görüşmeler yoluyla, nicel veriler ise çevrimiçi uygulanan "Ölçme Araçları Değerlendirme ve Tercih Anketi"yle toplanmıştır. Yarı yapılandırılmış görüşmelere 12 öğrenen, ankete ise 13.739 öğrenen katılmıştır. Nitel verilerin çözümlenmesinde betimsel analiz yöntemi kullanılırken nicel verilerin çözümlenmesinde yüzde-frekans tabloları, çoklu uyum analizi, Mann-Whitney U ve Kruskal-Wallis testlerinden yararlanılmıştır. Çalışma sonunda; öğrenenleri değerlendirmede kullanılan ölçme araçlarının çeşitlendirilmesi, çevrimiçi sınav olanaklarının artırılması, öğrenenlere öğrenme ve değerlendirme süreciyle ilgili daha fazla geribildirim verilmesi ve öğrenenlere öğrenim gördükleri alanlarda daha fazla uygulama yapma fırsatı tanınması gerektiği ortaya çıkmıştır.
Article
Background Endoscopic examination is a popular and routine procedure for the diagnosis and treatment of gastrointestinal (GI) diseases. Skilled endoscopists are in great demand in clinical practice, but the training process for beginners to become endoscopy specialists is fairly long. Convenience and a self-paced, learner-centered approach make electronic learning (e-learning) an excellent instructional prospect. Objective This study aimed to develop and apply an e-learning system in gastroscopy teaching and learning and to evaluate its effectiveness and user satisfaction. Methods The e-learning software Gastroscope Roaming System was developed for primary learners. The system simulates the real structure of the upper gastrointestinal (UGI) tract to teach the main characteristics of gastroscopy under both normal conditions and conditions of common UGI tract diseases. A randomized controlled trial was conducted. Participants were randomly allocated to an e-learning group (EG)or a non–e-learning control group after a pretest. On completing the training, participants undertook a posttest and gastroscopy examination. In addition, the EG completed a satisfaction questionnaire. ResultsOf the 44 volunteers, 41 (93%) completed the gastroscopy learning and testing components. No significant pretest differences were found between the intervention and control groups (mean 50.86, SD 6.12 vs mean 50.76, SD 6.88; P=.96). After 1 month of learning, the EG’s posttest scores were higher (mean 83.70, SD 5.99 vs mean 78.76, SD 7.58; P=.03) and improved more (P=.01) than those of the control group, with better performance in the gastroscopy examination (mean 91.05, SD 4.58 vs mean 84.38, SD 5.19; P
Chapter
Full-text available
The number of educational or serious games (SGs) available to educators has increased in recent years as the cost of game development has been reduced. A benefit of SGs is that they employ not only lesson content but also knowledge contexts where learners can connect information to its context of use with active participation and engagement. This, in turn, improves learners’ ability to recall, integrate, and apply what they learn. Much of the research on game analytics has examined learner in-game trails to build predictive models that identify negative learner actions (e.g., systematic guessing after the fact). However, analytics can also be used in the game design and development phases. Drawing on evidence-centered design (ECD), the chapter outlines ways that analytics can drive the development of scenarios and activities in a game and thus allows SGs to function as contextual apprenticeships, providing robust assessment opportunities. We describe how ECD theory was applied in a project to develop and test a SG that trains people to reduce their reliance on cognitive biases. We describe instances during the design process where our team encountered obstacles due to differing psychological and learning/teaching orientations, a topic rarely explored in the SG or ECD literature. Furthermore, we describe the final analytics-based game design features. We propose an additional element (persona) and how we anticipate incorporating that ECD extension into future projects.
Article
Full-text available
In most educational programs, a substantial proportion of teacher and student time is devoted to activities which involve (or lead directly to) evaluation by the teacher of student products or behavior. This review summarizes results from 14 specific fields of research that cast light on the relationships between classroom evaluation practices and student outcomes. Particular attention is given to outcomes involving learning strategies, motivation, and achievement. Where possible, mechanisms are suggested that could account for the reported effects. The conclusions derived from the individual fields are then merged to produce an integrated summary with clear implications for effective educational practice. The primary conclusion is that classroom evaluation has powerful direct and indirect impacts, which may be positive or negative, and thus deserves very thoughtful planning and implementation.
Conference Paper
Full-text available
Recently, a new age of information has appeared where information and communication technology plays a major role in education and learning society. Consequently, modern learning settings represented by emerging learning strategies have been used in higher education. A primary goal for higher education is to support students and lifelong learners to be independent, self-motivated, and self-regulated. As a result, students should be provided with enhanced approaches of learning and assessment that support them to set their learning goals effectively, to plan and use effective strategies in order to achieve their goals, to manage resources, to monitor their understanding, and to assess their progress towards their goals. This paper proposes an enhanced approach of e-assessment where automatically created formative assessments are provided to support self-regulated learners.
Article
Full-text available
Free-text answers assessment has been a field of interest during the last 50 years. Several free-text answers assessment tools underpinned by different techniques have been developed. In most cases, the complexity of the underpinned techniques has caused those tools to be designed and developed as stand-alone tools. The rationales behind using computers to assist learning assessment are mainly to save time and cost, as well as to reduce staff workload. However, utilising free-text answers assessment tools separately form the learning environment may increase the staff workload and increase the complexity of the assessment process. Therefore, free-text answers scorers have to have a flexible design to be integrated within the context of the e-assessment system architectures taking advantages of software-as-a-service architecture. Moreover, flexible and interoperable e-assessment architecture has to be utilised in order to facilitate this integration. This paper discusses the importance of flexible and interoperable e-assessment. Moreover, it proposes a service-oriented flexible and interoperable architecture for futuristic e-assessment systems. Nevertheless, it shows how such architecture can foster the e-assessment process in general and the free-text answers assessment in particular.
Article
Assessment has to be seen as an integrated and important activity in the learning process. In particular modern educational approaches -such as self-directed or exemplary learning -and per-sonalized learning activities cause a tremendous effort or make it even impossible to prepare appro-priate and individualized test items, assess them and provide feedback. This situation has motivated the Advanced Educational Media Technologies (AEMT) Group at Graz University of Technology to initiate a research program on e-assessment to cover the entire life cycle of the assessment process by semi-automated and automated approaches. In this paper we focus on the automated process to create different types of test items out of textual learning content, more precisely to create single choice, multiple-choice, completion exercises and open ended questions. Started in previous re-search activities by applying statistic approaches to summarize learning content and to identify most important words, our most recent approach applies a combination of statistic and natural language processing as well as semantic methods to identify most important concepts on an abstracted level. In our enhanced approach, identified concepts and differently related concepts represent the underpin-ning input for the test item creation. The implemented prototype can process learning content stored in various file formats, extracts most important content and related concepts, creates different types of test items and reference answers, and supports online assessments as well as exports the test items in QTI format. In this paper we cover in particular the following aspects: the motivation for automated test item creation, related work, requirement and design of the enhanced tool, implementation and usage viewpoints. Furthermore, we outline a study on the evaluation of the prototype, which suggests that the quality of extracted concepts and created test items is comparable with such ones provided by humans.
Article
Part one of this paper highlights how students today think and process information fundamentally differently from their predecessors, as a result of being surrounded by new technology. The author compares these “digital natives” with the older generation who are learning and adopting new technology naming them “digital immigrants”.