Conference PaperPDF Available

Effectively Assessing the Mastery of Learning Outcomes in ODL Courses

Authors:

Abstract

The constant emphasis on maintaining a high level of quality in the Open Distance Learning (ODL) self directed course modules developed and delivered at Wawasan Open University (WOU) demands the accurate assessment of learner centric Outcome Based Education (OBE). The summative and formative assessment components of each course module are, in theory, designed to effectively measure the mastery of a set of course learning outcomes (CLOs) by an adult learner in the ODL environment. However, in reality, there is no real scrutiny of the assessments to ascertain whether a learner has achieved the expected mastery of a CLO other than the assumption that assessment marks are an indication of the mastery of a particular set of CLOs measured in a specific assessment component. With the aim of identifying whether assessment marks are indeed reflective of the mastery of a set of CLOs, a pilot project was initiated at WOU in collaboration with Pearson eCollege to implement the eCollege Learning Outcome Manager (LOM), a web based platform, which facilitates the effective measurement of mastery of CLOs in a course module. TCC123/05 Visual Programming, a lower level technical course module equivalent to a first year course module in a conventional university, was identified as a suitable test case with an enrolment of seventy one learners distributed in four geographic locations across Malaysia. The evaluation of the mastery of CLOs by learners was conducted for this course on the LOM as a postmortem exercise which superimposed the mastery of CLOs on the assessment marks. This paper discusses the methodology and findings of this pilot project with respect to identifying a potential correlation between the mastery of CLOs and the assessment marks.
1
Effectively Assessing the Mastery of Learning Outcomes in ODL
Courses
Ishan Sudeera Abeywardena
Senior Lecturer, School of Science and Technology, Wawasan Open University
54, Jalan Sultan Ahmad Shah, 10050 Penang, Malaysia
Tel : +604-2180 484 Fax: +6042297 323 E-mail: ishansa@wou.edu.my
Abstract
The constant emphasis on maintaining a high level of quality in the Open Distance Learning (ODL)
self directed course modules developed and delivered at Wawasan Open University (WOU) demands
the accurate assessment of learner centric Outcome Based Education (OBE). The summative and
formative assessment components of each course module are, in theory, designed to effectively
measure the mastery of a set of course learning outcomes (CLOs) by an adult learner in the ODL
environment. However, in reality, there is no real scrutiny of the assessments to ascertain whether a
learner has achieved the expected mastery of a CLO other than the assumption that assessment marks
are an indication of the mastery of a particular set of CLOs measured in a specific assessment
component.
With the aim of identifying whether assessment marks are indeed reflective of the mastery of a set of
CLOs, a pilot project was initiated at WOU in collaboration with Pearson eCollege to implement the
eCollege Learning Outcome Manager (LOM), a web based platform, which facilitates the effective
measurement of mastery of CLOs in a course module. TCC123/05 Visual Programming, a lower level
technical course module equivalent to a first year course module in a conventional university, was
identified as a suitable test case with an enrolment of seventy one learners distributed in four
geographic locations across Malaysia. The evaluation of the mastery of CLOs by learners was
conducted for this course on the LOM as a postmortem exercise which superimposed the mastery of
CLOs on the assessment marks.
This paper discusses the methodology and findings of this pilot project with respect to identifying a
potential correlation between the mastery of CLOs and the assessment marks.
Keywords: Learning Outcomes, Outcome Based Education, OBE, Open Distance Learning,
ODL, Learning Outcome Measurement, Assessment, Pearson, eCollege, Learning Outcome
Manager, LOM
2
1.0 Introduction
Being an Open Distance Learning (ODL) institution, Wawasan Open University (WOU)
adopts a blended method for course module delivery to its adult learners which includes self
directed course materials supplemented by brief face to face tutorial sessions and online
student support. Due to the lifelong learning aspect of the business where the focus on
learning outcomes is considered crucial (Hussey & Smith, 2003), WOU invests heavily in the
development and continuous improvement of the self directed course material which undergo
a comprehensive instructional design process that makes them suitable for learner centric
Outcome Based Education (OBE) (Mc Neir, 1993). Each course module is specifically
designed to promote mastery of a particular set of course learning outcomes (CLOs) by a
learner. The CLOs are further subdivided into unit learning outcomes (ULOs) which allow
learners to self assess their mastery using formative assessment components. Despite the fact
that the formative assessment component providing valuable insights into the learners’
mastery of CLOs (Black & Wiliam, 1998) the lack of class room based activities in ODL
makes the summative assessment component key in assessing the mastery of CLOs.
However, the summative assessment component, which comprises of tutor marked
assignments (TMAs) acting as continuous assessments and the final proctored exam, is not
scrutinised in detail to assess the mastery of the CLOs by the learners. It is generally assumed
that the assessment marks are reflective of the learners’ mastery of the CLOs. i.e the higher
the marks the better the mastery of the CLOs. Even though there have been studies conducted
such as identifying the correlation between the mastery of CLOs and the GPA (Gijbels et al.,
2005) to prove this assumption, no study has been conducted to identify the validity of this
assumption in the context of WOU.
2.0 Methodology
A pilot project was initiated at WOU in collaboration with Pearson eColloege to identify the
correlation between assessment marks and mastery of CLOs in an adult ODL environment
using the eCollege Learning Outcome Manager (LOM) platform which is a comprehensive
web based software application. An independent academy was created for WOU on the LOM
which allowed the detailed tracking of progress for each course module assessed using the
platform. TCC123/05 Visual Programming, a five credit hour lower level technical course
module equivalent to a first year course module of a conventional university, specialising in
Visual Basic.Net (VB.Net) was used as the test case for the pilot project taking into account
the total of seventy one learners enrolled in the course module from learning centers located
in the regions of Penang, Ipoh, Kuala Lumpur and Johor Bharu distributed across Malaysia.
The assessment for this particular course comprised of (i) three TMAs which are a
combination of theory and laboratory exercises contributing 50% to the final score; and (ii) a
proctored examination contributing the remaining 50% to the final score. The learners were
required to obtain a minimum mark of 40% for both components to pass the course module.
As the LOM allows for the tracking of mastery of CLOs in real-time, the study was
conducted as a postmortem exercises to avoid any implications the real-time tracking and
feedback would have on the marking of assessments. i.e. the LOM was setup to superimpose
the mastery of CLOs on top of the assessment marks which were awarded as an independent
exercise conducted prior to the study. Four classes were created representing each of the
learning centers and LOM accounts were created for each student enrolled in a particular
class. In a real-world scenario, the students would be able to track their progress through the
3
LOM but this feature was disabled for the purposes of this pilot study as it was done as
postmortem. Only the Student ID was used to identify the student in the system to protect
anonymity and to ensure unbiased evaluation.
To effectively measure the mastery of the CLOs (i) each CLO was granulated into smaller
ULOs; (ii) a rubric comprising of a three point Likert scale was identified which measured
whether the learner needs improvement, meets the requirement or exceeds the requirement
with respect to mastery of ULOs; and (iii) learning statements were identified which
measured the mastery of a ULO against the rubric. This assessment rubric was then setup on
the LOM and each ULO was mapped against the TMAs and exam assessment components
which measured the mastery of individual ULOs. An assumption was made from the outset
that the average of the mastery of each ULO will determine the overall mastery of the
corresponding CLO. i.e. an average mastery of 2/3 would be deemed as meeting the
requirement with respect to the mastery of the CLO.
After the initial setup of the system, the population of data was done by the academic in
charge of the course module. The answer scripts of the exam as well as the TMAs were re-
evaluated to determine the mastery of the ULOs by the learners and were marked against the
assessment rubric on the LOM. The assessment marks as well as the letter grades were also
recorded in the LOM for further analysis. Once the re-evaluation was complete, detailed
reports were generated using the LOM in MS Excel format to be further analysed. The
Pearson product-moment correlation coefficient (r) was used to determine the correlation
between the assessment marks and the mastery of CLOs. The statistical analysis was
conducted using the MS Excel software application.
3.0 Results
The mastery of the CLOs was mapped against the assessment marks for the exam and the
three TMAs as shown in Figure 1.1. The Pearson product-moment correlation coefficient (r)
between mastery of CLOs and assessment marks was calculated for each assessment as
shown in Table 1.1.
Assessment Number of Learners (r)
Exam 48 0.903748
TMA1 66 0.768821343
TMA2 61 0.843116626
TMA3 54 0.816461787
Table 1.1 Pearson product-moment correlation coefficient (r) between mastery of CLOs and
assessment marks
4
Figure 1.1 Mastery of CLOs Vs. Assessment Marks
4.0 Discussion
To statistically determine whether there was a correlation between the mastery of the CLOs
and the assessment marks, the exam marks and the marks for the three TMAs were analysed
using the Pearson product-moment correlation coefficient (r). According to the results of this
statistical test, as shown in Table 1.1, there is a strong positive correlation between the
mastery of the CLOs and the assessment marks. This provides some indication that the marks
of an assessment can be used to ascertain mastery of the CLOs. However, the correlation
does not indicate which range of marks would be representative of meeting the requirement
with respect to the mastery of the CLOs. To identify the possible correspondence between the
assessment marks and the CLO assessment rubric, the assessment marks for both the exam as
well as the TMA component were plotted against the mastery of the CLOs in a scatter plot as
shown in Figure 1.1. By analysing the scatter plot, it was identified that the marks ranging
from 60% to 90% approximately indicate that the learner has met the requirement with
respect to mastery of the CLOs. It was further approximated from Figure 1.1 that marks less
than 40% indicate the need for improvement in terms of mastering the CLOs. This bears a
resemblance to the passing mark of an assessment at WOU which is set at 40%. Assessment
0.00
0.50
1.00
1.50
2.00
2.50
3.00
0 10 20 30 40 50 60 70 80 90 100
Mastery of CLOs
Assessment Marks %
Exam
TMA1
TMA2
TMA3
5
marks beyond 80% were approximately representative of the learner exceeding the
requirement as far as the mastery of CLOs is concerned.
Despite the fact that the aforementioned statistical analysis is indicative of a correlation
between the assessment marks and the mastery of the CLOs, the correspondence between the
CLO assessment rubric and the assessment marks is only an approximation. This is mainly
due to the breadth of the three point Likert scale which was used in the CLO assessment
rubric. Therefore, it can be concluded that a more granular CLO assessment rubric is needed
for effectively identifying a potential correspondence between the assessment marks and the
mastery of CLOs.
During the course of the pilot study, it was realised that the implementation of the LOM
needs to be done holistically from inception to conclusion of a course module which include
the design, development, delivery, assessment, feedback and revision stages. The strengths as
well as the weaknesses of the present assessment mechanisms at WOU were also made
apparent during the course of the study especially with respect to assessing the CLOs.
Another point to note was that not every CLO needed to be assessed in each assessment. Also
the marking schemes would need to take into consideration the CLO assessment rubric when
allocating marks.
Overall, the LOM was found to be a more comprehensive as well as a more effective method
of assessing the mastery of the CLOs which allows academics as well as learners the ability
to track the progress of the mastery of smaller ULOs in addition to the larger CLOs. This in
turn provides the key stakeholders the ability to tailor learning activities as well as
assessments to ensure the mastery of most of the CLOs by the learners. The LOM provides a
usable yardstick for the accurate assessment of OBE in ODL environments which acts as a
valuable enabler especially in accreditation exercises.
5.0 Conclusion
A pilot project was initiated at WOU in collaboration with Pearson eCollege to identify a
possible correlation between assessment marks and the mastery of CLOs in a self directed
adult ODL environment. A lower level technical course module was used as the test case for
this project which superimposed the mastery of the CLOs on top of the assessment marks
using the eCollege Learning Outcome Manager (LOM) web based software application.
Using the Pearson product-moment correlation coefficient (r), it was statistically identified
that there was a strong positive correlation between the two. A scatter plot of the mastery of
CLOs against the assessment marks provided an approximation of the range of assessment
marks which is indicative of the mastery of CLOs. However it was made apparent during
course of the study that the assessment marks cannot be used as an effective measure of the
CLOs due to the breadth of the range. Thus the LOM was found to be a more effective as
well as comprehensive method of assessing the mastery of the smaller ULOs in addition to
the CLOs in a self directed adult ODL environment which enables the key stakeholders to
keep track of the progress. It was also found to be an effective yardstick for the accurate
assessment of OBE in ODL which acts as a valuable enabler in accreditation exercises.
6
Acknowledgments
The author wishes to acknowledge the support provided by Professor Dr Wong Tat Meng,
Vice Chancellor and Professor Dato’ Dr Ho Sinn Chye, Deputy Vice Chancellor (Academic
Affairs) of Wawasan Open University for the guidance and advice provided with respect to
steering this pilot project.
References
Black, P. & Wiliam, D. (1998). Assesment and Classroom Learning. Assessment in
Education: Principles, Policy & Practice, 5(1)
Gijbels, D., Van de Watering, G., Dochy, F., & Van den Bossche, P. (2005). The relationship
between students’ approaches to learning and the assessment of learning outcomes. European
Journal of Psychology of Education, 20(4), 327-341.
Hussey, T. & Smith, P. (2003) The uses of learning outcomes, Teaching in Higher Education,
8(3), 357–368.
Mc Neir, G. (1993). Outcome-based Education. ERIC Digest, 85, 3-5
... At the undergraduate level, the learners undertake courses of five credit hours in self directed mode using purpose designed course materials supplemented by brief face to face tutorial sessions and an online learning management system. Due to the lifelong learning aspect of the business where the focus on learning outcomes is considered crucial (Hussey & Smith, 2003), WOU invests heavily in the development and continuous quality improvement of the self directed course materials which undergo a comprehensive instructional design process (Abeywardena, 2013). This, in turn, makes the materials suitable for learner centric Outcome Based Education (OBE) (McNeir, 1993). ...
... A preliminary version of this paper was presented during the 25 th Asian Association of Open Universities Annual Conference, Penang, Malaysia, 2011 (Abeywardena, 2011). ...
Article
Full-text available
... The author acknowledges the support provided by Pearson eCollege with respect to free access to the eCollege Learning Outcome Manager (LOM) software platform and user training. A preliminary version of this paper was presented during the 25 th Asian Association of Open Universities Annual Conference, Penang, Malaysia, 2011 (Abeywardena, 2011). ...
Article
Full-text available
The constant emphasis on maintaining a high level of quality in the Open Distance Learning (ODL) self di-rected courses delivered at Wawasan Open University (WOU) demands the accurate assessment of course learning outcomes (CLO). The summative and formative assessment components of each course module are, in theory, designed to effectively measure the mastery of a set of CLO by an adult learner. However, in reality, there is no real scrutiny of the assessments at WOU to ascertain whether a learner has achieved the expected mastery of a CLO. The general assumption is that the assessment marks are an indication of the mastery of a particular set of CLO measured in a specific assessment component. With the aim of identifying whether assessment marks are indeed reflective of the mastery of a set of CLO, a study was conducted to implement the "Person eCollege Learning Outcome Manager" software application. This paper discusses the methodology and findings of this study and provides several suggestions for the smooth implementation of the Learning Outcome Manager.
Article
Full-text available
The purpose of the present study is to gain more insight into the relationship between students' approaches to learning and students' quantitative learning outcomes, as a function of the different components of problem-solving that are measured within the assessment. Data were obtained from two sources: the revised two factor study process questionnaire (R-SPQ-2F) and students' scores in their final multiple-choice exam. Using a model of cognitive components of problem-solving translated into specifications for assessment, the multiple-choice questions were divided into three categories. Three aspects of the knowledge structure that can be targeted by assessment of problem-solving were used as the distinguishing categories. These were: understanding of concepts; understanding of the principles that link concepts; and linking of concepts and principles to application conditions and procedures. The 133 second year law school students in our sample had slightly higher scores for the deep approach than for the surface approach to learning. Plotting students' approaches to learning indicated that many students had low scores for both deep and surface approaches to learning. Correlational analysis showed no relationship between students' approaches to learning and the components of problem-solving being measured within the multiple choice assessment. Several explanations are discussed.
Article
This paper argues that learning outcomes need to be reclaimed from their current use as devices for monitoring and audit, and returned to their proper use in aiding good teaching and learning. We require a broader, flexible and more realistic understanding of learning outcomes, better suited to the realities of the classroom and of practical use to those teachers who wish to respond to the enthusiasm of their students. To this end, a new model is produced that starts from the idea of an articulated curriculum, and embraces both intended and emergent learning outcomes. The model employs the distinction between predicted and unpredicted learning outcomes, together with the distinction between those that are desirable and those that are undesirable. The resulting account is intended to aid understanding of the nature and proper use of learning outcomes in teaching and learning.
Article
This article is a review of the literature on classroom formative assessment. Several studies show firm evidence that innovations designed to strengthen the frequent feedback that students receive about their learning yield substantial learning gains. The perceptions of students and their role in self‐assessment are considered alongside analysis of the strategies used by teachers and the formative strategies incorporated in such systemic approaches as mastery learning. There follows a more detailed and theoretical analysis of the nature of feedback, which provides a basis for a discussion of the development of theoretical models for formative assessment and of the prospects for the improvement of practice.
Article
The move to outcome-based education has been one of the most important trends in health-profession education in recent years. This paper defines outcomes and outcome-based education, describes the development of outcome-based education, identifies several different ways that outcomes have been presented, and discusses the advantages and disadvantages of the outcome-based educational approach. The implementation of outcome-based education at the University of Dundee medical school, Scotland, UK is described as a case study for curriculum planners in veterinary medical education. The lessons learned in Dundee from six years' experience with outcome-based education are identified to aid veterinary medical educators wishing to implement the approach.